ºÝºÝߣ

Editor's Notes

1. I am Miriam and I am a vision researcher. What happens when you put together a computational social scientist, 2 psychologists from UT Austin, a computer vision researcher? We thought of a machine vision system that can read face images under an new, unexpected point of view.
2. Computer vision researchers, we love to put the sentence "an image is worth a thousand words" pretty much everywhere in our presentations.
3. But what we often forget to say is that, when a picture is a portrait of a person, that face image is worth an entire book of words.
4. A face image is a book telling the biography of the person depicted, his history, his personality traits, his emotions. Computer vision systems have been devloped in the recent years to automatically infer some aspects of this biography, such as the identity for example. But in this work, we go beyond this intuitive dimensions, building an algorithm able to estimate the AMBIANCE of the place where people go based on their profile pictures.
5. So for example, our algorithm could see a group of profile pictures and say: ¡®oh those guys might like geek places, or party places¡¯
6. We are inspired by a study that was presented four years ago by our co-authors at this same conference. In this study, they selected 49 FS venues in Austin, and asked a group of students to physically go to each of those places and rate their AMBIANCEs, according to 72 AMBIANCE types such as ¡®party¡¯ or creative. They then collected from FS the profile pictures of 25 users that frequently go to those venues. They showed these profile images to another team of students, and asked them to guess the preferences of the subjects in the pictures. And, by correlating such face-driven scores with the on-the-spot ratings, they showed that actually the students were able to correctly assess the AMBIANCE of the place where people would go, by just looking at their profile pictures.
7. We are inspired by a study that was presented four years ago by our co-authors at this same conference. In this study, they selected 49 FS venues in Austin, and asked a group of students to physically go to each of those places and rate their AMBIANCEs, according to 72 AMBIANCE types such as ¡®party¡¯ or creative. They then collected from FS the profile pictures of 25 users that frequently go to those venues. They showed these profile images to another team of students, and asked them to guess the preferences of the subjects in the pictures. And, by correlating such face-driven scores with the on-the-spot ratings, they showed that actually the students were able to correctly assess the AMBIANCE of the place where people would go, by just looking at their profile pictures.
8. The natural question that came to our mind when looking at this study was: can we use computer vision techniques to build a machine that is able to do the same as the students were doing in this study, and automatically guess the AMBIANCE preferences given the profile pictures of patrons. And if such a machine existed, what are the visual attributes that the machine evaluates when predicting the AMBIANCE of a place, and how does it differ from humans when performing this task?
9. Let¡¯s go step by step. To build such a system, we need a learning algorithm that is able to associate AMBIANCE ratings with attributes of the profile pictures. To do so, we resort to the dataset collected by our co-authors in their study. For each of the 49 FS venues, we get 25 profile pictures and we compute a set of visual features. Visual features, based on signal processing techniques can tell us (and the machine) something about the properties of the picture and its subject. We then associate a set of AMBIANCE labels to each place. These AMBIANCE labels correspond to the on-the-spot AMBIANCE ratings that the students gave when visitng the venues. We then feed a regression algorithm with both features and ratings, and make it learn how to associate these 2 dimensions.
10. Let¡¯s take a look at the visual features we design for AMBIANCE prediction.
11. The first group of visual features is inspired by our previous work in compuational aesthetic. Aesthetic features tell us something about the image composition and its quality. For example, we extract some brightness, contrast and saturation metrics, some structural information such as the symmetry or the number of circles, - here we have ... one.
12. We then look at the color distribution, since visual perception theory tells us that image colors relate to some specific feelings or atmosphere depicted in the scene. For example, yellow is associated with cheerfulness
13. We then use computer vision to understand the facial expressions of a subject. So that we can know whether the subject is sad, happy, or whether she is having a sugar shock for example!
14. Based Face ++ Face analysis software we can record the demographic traits of the subject, age gender and race. For example, this is a picture of a white mail when he was still a young researcher ;)
15. Lastly, in this work we are not dealing with simple portraits. Ours are profile pictures from online social networks, and we want to understand the appearance choices that users make when presenting themselves to the online society thorough their profile pictures. We look at the presence of glasses, whether the image shows a real face or not, the position/orientation of the face, and we compute a uniqueness score based on spectral analysis, that can tell us how original the image composition is.
16. Now we build an algorithm that combines these features in a variety of ways. What is itsaccuracy?
17. And it turns out that it works! When we saw the framework's performances we were actually surprised! This plot shows the MSE between the real AMBIANCE scores and the AMBIANCE scores predicted by the machine in a leave-one-out fashion. The error is always lower than 12%! Sometimes, the machine finds it harder to guess the AMBIANCE of places, especially for AMBIANCEs such as creative and party, where the corresponding profile pictures are very diverse, introducing more noise in the system.
18. We then compared the scores given by our system when ¡°looking¡± at the pictures versus the scores given by the students when looking at the pictures, and, we see that, for 10 out of 18 AMBIANCEs, our algorithm is better than the students in guessing AMBIANCE preferences! How is this possible! Our intuition is that the machine directly learns how to associate faces with AMBIANCEs, without any cultural mediation. On the other hand, humans when looking at faces necessarily attach values influenced by their cultural background and stereotypes. And indeed, when we look at the importance of features that machines and humans evaluate when performing this task, we find that, on one hand, the algorithm tends to look more at signal statistics and structural element of the picture, while humans are much more influenced by color attributes and demographic traits.
19. Also, the machine at times is able to overcome basic human gender stereotypes. In the study the students would associate the presence of women with romantic and pick-up places. However, the algorithm simply says that people going to romantic places tend to put warm colors in their pictures, and that you can find both genders in pick-up places, which makes sense.
20. Sometimes, humans and machine agree in their stereotpyes. As expected, friendly AMBIANCEs are associated with people that smile. Patrons of strange places seem to choose yellow to represent themselves, and, at least for the machine, they don't want to show their faces too much. And finally, nerds going to reading places like to wear glasses, and the machine confirms it!
21. faces tell us stories. And we have some powerful machine vision techniques to read those stories. We made the choice of selecting features that are interpretable [PAUSE]. That¡¯s the only way to counter algorithmic stereotyping. But one of the most take-home messages from this study is that Visual features can help us understanding not only what faces say, but also how humans perceive them. Visual features are glass-boxes through which we are able to detect stereotypes. And if we know which visual patterns relate to sterotypes, we might also know how to modify such patterns, encourage relativism and discourage stereotyping.