Fusing Social Media and Sensor Data for Rich Personalization

Fusing Social Media and Sensor Data for Rich Personalization

Saiph Savage  Tobias Hollerer

We are living in a data explosion, fueled by the information captured by governments, businesses, researchers, social media, and mobile devices. The main discussion of  this research is that by using pattern recognition techniques,  this big  data can be used to connect what is happening in the digital, social and physical worlds, with which  data becomes more valuable, practical, and alive for the end user and her community. Below we describe a few of our research projects related to this theme:

I'm Feeling LoCo:
A Location Based Context Aware Recommendation System

Research in ubiquitous location recommendation systems has focused on automatically inferring a user's preferences while little attention has been devoted to the recommendation algorithms. Location recommendation systems with a focus on recommendation algorithms generally require the user to complete complicated and time consuming surveys and rarely consider the users current context. The purpose of this investigation is to design a more complete ubiquitous location based recommendation algorithm that by inferring users preferences and considering time geography and similarity measurements automatically, betters the user experience.

Our system learns user preferences by mining a persons social network profile. The physical constraints are delimited by a users location, and form of transportation, which is automatically detected through the use of a decision tree followed by a discrete Hidden Markov Model. We defined a decision-making model, which considers the learned preferences, physical constraints and how the individual is currently feeling. Our recommendation algorithm is based on a text classification problem. The detection of the form of transportation and the user interface was implemented on the Nokia N900 phone, the recommendation algorithm was implemented on a server which communicates with the phone. The novelty of our approach relies on the fusion of information inferred from a users social network profile and his/her mobile phones sensors for place discovery.


Seems Familiar: An Algorithm For Inferring Spatial Familiarity Automatically

Our level of spatial familiarity with a particular place determines our navigation and interaction around that space. For example it defines a user's degree of exploration: in familiar environments, users have shown to have a greater initiative in visiting new places, additionally the type of route a user prefers to take highly depends on his/her familiarity with the area. Therefore, when a user is utilizing a wayfinding tool or a Place recommendation system, these should present information and interactions according to the person's current level of spatial familiarity. The lack of integration of a user’s spatial familiarity in location based services may be due to scarce work in automatic detection of a user’s spatial familiarity.

In this work, following related studies in both psychology and geography, we propose a system that, given the user’s current location, can use the sensor data from a user's mobile devices, along with information from the person's social media accounts to automatically infer the person’s level of spatial familiarity around the area he/she is currently in. Our model considers not only the number of visits made to a particular location, but also information about the spatial layout associated with the place, the relationships between the user and the place, such as, authoritative constraints: a person under 21 cannot enter bars, as well as attributes related with the person's profession: a computer science major has visited numerous computer laboratories in the past, and therefore although they may currently be in a never before visited computer laboratory, to some degree, they are familiar with this particular environment. Finally the user's behavior is also considered for classification, in particular, a user’s movement pattern, such as taking loops, shortcuts, or immediate reverse path (indicating that user took a wrong direction), and movement speed)


Enchantment Under The Sea: An Intelligent Social Environment For Music Mixing

Disc Jokeys (DJs) generally mix music in a confined isolated space. This can make the DJ have depression sentiments and it can also difficult the DJ's understanding of his public. We present Enchantment Under The Sea: a new intelligent environment that allows the disc jokey to roam freely, interact directly with his audience, receive informative feedback about the public's social interactions, while also respecting the DJs privacy concerns. The music interface is controlled using Microsoft’s wireless touch mouse with ubiquitous gestures that resemble dance moves. The music mixing interface is displayed on the walls of the event, where two different display modalities are enabled: open interface, in which the public can observe all of the DJ’s decisions with the music mixing interface and also actively give music suggestions to the DJ. And a closed interface, where all the music controllers are mapped to sea animals, that only the DJ knows the mapping to, thus providing privacy to the DJ’s work. The public's social interactions are measured with sonar sensors whose data is provided to the DJ through the musical interface. We report results of a controlled usability inspection.

Mmmmmm: A Multi-Modal Musical Mobile Mixer

Mmmmm stands for Multimodal Musical Mobile Mixer and itis a new musical DJ application that has been designed for the Nokia n900 phone. This software enables a new type of interaction between the DJ and the crowd. Mmmmm has streaming via bluetooth of the music generated in the phone to wireless speakers, which allows the DJ to move about the environment and get familiarized with the crowd, turning the experience of DJing into an interactive and audience engaging process.It also has Eyes Free mode enabled and it is carried out through hand gestures and haptic feedback, in this way the DJ can focus on the public and have a better intuition of what they are feeling. Mmmmm also has a "Party Detection" mode. By using the phone camera, the party scene is scanned and then a certain type of music as well as a series of songs is suggested. This mode helps novice DJs to instantly have a much better music repertoire, creating the illusion that an expert DJ is selecting the music the crowd is dancing to. Eyes free mixing can be enabled at the wish of the user


Relevant Publications