Human behaviour sensing and profiling in the wild

<p>With advances in sensors, wearables and the Internet of Things, it has become more and more convenient to gather information from human daily life, which has promoted the development of human behavioural sensing technology. In general, heterogeneous sensing data (e.g. behavioural, environmental and physiological sensing signals) may come from different sources (e.g. mobile phones, buildings, weather stations and wearables). From this information, it is possible to infer multiple human behaviours and psychological states, such as personality, thermal comfort and learning engagement. Sensing and profiling human behaviours has many advantages, such as supporting medical diagnosis, improving self-awareness, creating supportive study/work environments and taking timely measures to promote human wellbeing.</p> <p>However, human behaviour sensing is a complex task with some key challenges: (1) Limited sources of sensing data: Previous research has primarily explored one type or a limited number of types of sensing data (e.g. accelerometer data and heart rate signals) to build predictive models rather than incorporating sensing data from multiple sources. (2) Lab-based settings: Most studies have been conducted in environments specifically designed for research; however, field experiments are more likely to reflect real-world human behavioural patterns due to the authenticity of natural settings. (3) Difficulty in validating the ground truth: Self-report surveys are generally considered to be measures of ground truth in human-based research but may be prone to subjectivity and various types of response bias. (4) Difficulty in depicting dynamic behaviours: Human behaviours are dynamic and complex in heterogeneous environments, making it difficult to accurately depict them. (5) Shortage of annotations: Traditional self-report surveys are the most popular way to understand human behaviours, but they are both time consuming and labour-intensive,  resulting in insufficient annotations and difficulty in creating effective models. In this thesis, we address the above challenges and make the following contributions.</p> <p>First, we address the challenge of the limited sources of sensing data in the wild. Using wearable sensors to log physiological data and daily surveys to query the participants’ thermal comfort, learning engagement, emotions and seating behaviours, we will collect data from 23 high school students and six teachers participating in 11 courses (144 classes) over a four-week period. We will then explore the validity of the collected data to ensure that we can reliably profile human behaviours using heterogeneous sensing data collected in the wild.</p> <p>Second, we will explore wearable and environmental sensing data to understand learning engagement in the wild. With the data previously collected in the wild, we will create a classroom sensing system to automatically measure the multidimensional engagement (i.e. behavioural, emotional and cognitive engagement) of high school students during classes. In particular, we will combine physiological signals, physical activities and indoor environmental data to estimate changes in student engagement levels. To the best of our knowledge, this will be the first system for detecting multidimensional engagement from multiple sensors in the wild.</p> <p>Third, we will investigate group behaviours to understand social relationships using physiological sensors. We will explore how the group-wise seating experience relates to student engagement by examining the participants' physiological arousal and synchrony. We will investigate whether students sitting close together are more likely to have similar learning engagements and greater physiological synchrony than students sitting far apart. This research has the potential to assist in maximising student engagement by providing more flexible and intelligent seating arrangements in the future.</p> <p>Fourth, we will employ unobtrusive mobile sensing for dynamic user behavioural modelling. Two real-world tasks (modelling Big Five personality traits and notification response behaviours) will be explored based on the participants' mobile phone usage behaviours. A comprehensive study on a real-world dataset will demonstrate whether it is possible to utilise smartphone usage behaviours to predict users' Big Five personality traits. In order to estimate response times, we will investigate whether the established regression model can accurately predict the response time to notifications using the user’s mood and physiological signals. Our research will shed light on the future intelligent notification management system for mobile users.</p> <p>Finally, we will model aggregated behaviour (i.e. thermal comfort) using environmental sensing with limited annotations by transferring knowledge from multiple locations to another domain. We will build a transfer learning framework and confirm that thermal comfort sensor data from multiple cities in the same climate zone can be used to improve the small thermal comfort dataset of a target building that has insufficient training data. Extensive experimental results will show that the proposed models outperform the state-of-the-art algorithms for thermal comfort prediction and can be implemented in any building, even if adequate thermal comfort labelled data are not available.</p> <p>In summary, this thesis provides several contributions to profiling and modelling human behaviours in the wild. This research will exploit various types of sensing data from multiple sources in different real-world tasks to address common challenges in the area of human behavioural modelling. We will also publish the largest and most diverse dataset collected in the wild to better understand participants' behaviour, engagement, emotion and comfort using heterogeneous sensors and wearables. This will benefit building scientists, behavioural psychologists and ubiquitous computing researchers in the future. Overall, we believe this research will provide a significant contribution to human-based sensing and behavioural profiling in the wild that will make  researchers, managers and policymakers more aware of occupants/users and more able to adapt to their needs.</p>