ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

A Subjunctive Exploratory Search Interface to Support Media Studies Researchers

?
?
S
Sound and Vision R&D

Presentation by Marc Bron at the 2012 SIGIR Conference in Portland, USA. Paper by Marc Bron (University of Amsterdam) Jasmijn van Gorp (Utrecht University) Andrei Vishneuski (University of Amsterdam) Frank Nack (University of Amsterdam) Sonja de Leeuw (Utrecht University) Maarten de Rijke (Universiteit van Amsterdam) can be found at http://dl.acm.org/citation.cfm?id=2348283.2348342

1 of 38
A SUBJUNCTIVE EXPLORATORY SEARCH INTERFACE TO SUPPORT MEDIA STUDIES RESEARCHERS M. Bron1, J. van Gorp2, F. Nack1, M. de Rijke1, A. Vishneuski1, S. de Leeuw2 (15-08-2012) (1) University of Amsterdam (2) Utrecht University vrijdag 10 augustus 2012
INTRODUCTION TO MEDIA STUDIES RESEARCH de?nition: media studies concerns the study of production, content and/or reception of various types of media, e.g., newspapers, ?lms, and television programs vrijdag 10 augustus 2012
INTRODUCTION TO MEDIA STUDIES RESEARCH de?nition: media studies concerns the study of production, content and/or reception of various types of media, e.g., newspapers, ?lms, and television programs example: how does the representation of women smokers change in magazines and ?ction shows between 1900 and 1950? vrijdag 10 augustus 2012
INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE how can we support media studies researchers in gathering material in electronic archives? methodology: interviews; 30 mins; topic list pick a recent research project? where did you get the idea? how did you search for data? did you encounter any problems? did your research questions change? can you describe the various stages during the project? can you write them down on index cards and order them chronologically? vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE exploration literature study explore domain data collection developing the research question determining available material literature broad gathering of material vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE exploration contextualization literature study literature study explore analysis domain data collection data collection analyzing material re?ning the research question ?nding context ?nding additional material vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE exploration contextualization presentation literature study literature study explore analysis writing report domain data collection data collection vrijdag 10 augustus 2012
ANALYSIS OF THE RESEARCH CYCLE exploration contextualization presentation literature study literature study explore analysis writing report domain data collection data collection analysis of data, e.g., watching and transcribing ?lms costs a lot of time, money, and effort. vrijdag 10 augustus 2012
DEVELOPMENT OF AN INTERFACE our example research question: how does the representation of women smokers change in magazines and ?ction shows between 1900 and 1950? requirements of the material: time and genre constraints, e.g., ?ction shows a speci?c group, e.g., women smokers is it possible to contrast ?ndings in the data, e.g., a change vrijdag 10 augustus 2012
DEVELOPMENT OF AN INTERFACE our example research question: how does the representation of women smokers change in magazines and ?ction shows between 1900 and 1950? requirements of the material: time and genre constraints, e.g., ?ction shows a speci?c group, e.g., women smokers is it possible to contrast ?ndings in the data, e.g., a change Major problem: whether the material satis?es all these criteria is only known after everything has been analyzed! vrijdag 10 augustus 2012
vrijdag 10 augustus 2012
vrijdag 10 augustus 2012
vrijdag 10 augustus 2012
DEVELOPMENT OF AN INTERFACE key question: does the option to compare statistics of multiple results sets support media studies researchers in ?nding material that satis?es the contrast requirement in their research questions? vrijdag 10 augustus 2012
vrijdag 10 augustus 2012
vrijdag 10 augustus 2012
vrijdag 10 augustus 2012
EXPERIMENTAL SETUP EXPERIMENTAL DESIGN Total of 36 media studies researchers and MSc. students (experience MD = 3; IQR = 0-6.5); between subject design; remote user study; data is 1.2 M tv programs with metadata from audio visual archive. pre search task survey a complex search task (30 mins): Collect material relevant to develop and answer a research question on the topic of how minorities are described in the Dutch media over the last 40 years. After collection, formulate a research question. post search task: write down the research question developed during searching exit survey vrijdag 10 augustus 2012
EXPERIMENTAL SETUP EVALUATION subjunctive interface interaction statistics: more query formulations more documents bookmarked wider variety of documents bookmarked (average pairwise similarity) research questions formulated with the subjunctive interface: a more re?ned scope usability of the interface usefulness for research; interesting dif?cult to use; intuitive vrijdag 10 augustus 2012
RESULTS INTERACTION STATISTICS showing median and (inter quartile ranges) for the query formulations for the number of documents bookmarked interaction baseline subjunctive queries 3 (2-6) 5 (3-8)* bookmarks 9 (6-22) 9 (3-13) for wider variety of documents bookmarked indicated by the average pairwise cosine similarity interaction baseline subjunctive bookmark variety .62 (.56-.69) .52 (.43-.63)* * signi?cant at the alpha = 0.05 level using Wilcoxon rank sum test vrijdag 10 augustus 2012
ANALYSIS INTERACTION PATTERNS Query -> Inspect -> Query -> Inspect is more often seen with the subjunctive interface Query -> Inspect -> Filter -> Inspect is seen more often with the standard exploratory search interface Bookmark -> Paginate -> Bookmark -> Paginate is a frequent pattern when using the baseline interface; pagination does not occur in the top interaction patterns with the subjunctive interface vrijdag 10 augustus 2012
ANALYSIS INTERACTION PATTERNS Query -> Inspect -> Query -> Inspect is more often seen with the subjunctive interface Query -> Inspect -> Filter -> Inspect is seen more often with the standard exploratory search interface Bookmark -> Paginate -> Bookmark -> Paginate is a frequent pattern when using the baseline interface; pagination does not occur in the top interaction patterns with the subjunctive interface comparison visualizations invite more queries bookmarking form two result lists i.s.o. single long list promotes more diverse documents vrijdag 10 augustus 2012
RESULTS RESEARCH QUESTIONS Do the research questions formulated with the subjunctive interface have a more re?ned scope determined by a contrast or comparison? three judges; experts in the ?eld of media and migration assessed the research questions showing median and (inter quartile ranges)/ 5 point Likert scale 1 is very bad; 5 is very good judge 1 judge 2 judge 3 baseline subjunctive baseline subjunctive baseline subjunctive 3 (3-4) 4 (3-4) 2 (2-4) 2 (2-3) 4 (3-4) 4 (2-4) It is hard too judge whether the scope of research questions is more or less re?ned what if we look at the research questions? vrijdag 10 augustus 2012
ANALYSIS RESEARCH QUESTIONS analysis of two research questions... Investigation of the evolution of the term integration in news and human interest programs broadcasted between 1992 and 2012 In 1987 we observe a diminishing in the use of the term migrant worker and a rise in the use of the term immigrant. Is it possible to identify a cause in the broadcasting schedule of that time? interface genre group topics period technical baseline 11 14 8 10 2 subjunctive 10 17 12 9 3 vrijdag 10 augustus 2012
ANALYSIS RESEARCH QUESTIONS analysis of two research questions... Investigation of the evolution of the term integration in news and human interest programs broadcasted between 1992 and 2012 In 1987 we observe a diminishing in the use of the term migrant worker and a rise in the use of the term immigrant. Is it possible to identify a cause in the broadcasting schedule of that time? interface genre group topics period technical baseline 11 14 8 10 2 subjunctive 10 17 12 9 3 In research questions formulated with the subjunctive interface the contrast is more concrete. vrijdag 10 augustus 2012
RESULTS USABILITY usability of the interfaces median and (inter quartile ranges)/ 5 point Likert scale 1 is absolutely not; 5 is very much interaction baseline subjunctive intuitive 4 (4-4) 4 (3-4) interesting 4 (4-5) 4 (4-5) useful 4 (4-5) 5 (4-5) dif?cult to use 2 (2-2) 2 (2-4) both interfaces are found to be (very) useful for media studies research surprisingly: the subjunctive interface is not considered to be dif?cult to use vrijdag 10 augustus 2012
CONCLUSIONS the media studies research process is a dynamic and iterative process, in which it is dif?cult to make research questions concrete until material has been collected and analyzed a subjunctive exploratory interface supports media researchers in exploration: it invites more query formulations and more diverse documents to be gathered formulating research questions in which a contrast is considered based on patterns in the material observed during exploration moreover the subjunctive interface, while more complex, is not considered to be dif?cult to use and is considered very useful for media studies research vrijdag 10 augustus 2012
CONCLUSIONS the media studies research process is a dynamic and iterative process, in which it is dif?cult to make research questions concrete until material has been collected and analyzed a subjunctive exploratory interface supports media researchers in exploration: it invites more query formulations and more diverse documents to be gathered formulating research questions in which a contrast is considered based on patterns in the material observed during exploration moreover the subjunctive interface, while more complex, is not considered to be dif?cult to use and is considered very useful for media studies research Thank you; are there any questions? vrijdag 10 augustus 2012

More Related Content

A Subjunctive Exploratory Search Interface to Support Media Studies Researchers

  • 1. A SUBJUNCTIVE EXPLORATORY SEARCH INTERFACE TO SUPPORT MEDIA STUDIES RESEARCHERS M. Bron1, J. van Gorp2, F. Nack1, M. de Rijke1, A. Vishneuski1, S. de Leeuw2 (15-08-2012) (1) University of Amsterdam (2) Utrecht University vrijdag 10 augustus 2012
  • 2. INTRODUCTION TO MEDIA STUDIES RESEARCH de?nition: media studies concerns the study of production, content and/or reception of various types of media, e.g., newspapers, ?lms, and television programs vrijdag 10 augustus 2012
  • 3. INTRODUCTION TO MEDIA STUDIES RESEARCH de?nition: media studies concerns the study of production, content and/or reception of various types of media, e.g., newspapers, ?lms, and television programs example: how does the representation of women smokers change in magazines and ?ction shows between 1900 and 1950? vrijdag 10 augustus 2012
  • 4. INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
  • 5. INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
  • 6. INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
  • 7. INTRODUCTION TO MEDIA STUDIES RESEARCH vrijdag 10 augustus 2012
  • 8. ANALYSIS OF THE RESEARCH CYCLE how can we support media studies researchers in gathering material in electronic archives? methodology: interviews; 30 mins; topic list pick a recent research project? where did you get the idea? how did you search for data? did you encounter any problems? did your research questions change? can you describe the various stages during the project? can you write them down on index cards and order them chronologically? vrijdag 10 augustus 2012
  • 9. ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
  • 10. ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
  • 11. ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
  • 12. ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
  • 13. ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
  • 14. ANALYSIS OF THE RESEARCH CYCLE explore literature data analysis writing report domain study collection vrijdag 10 augustus 2012
  • 15. ANALYSIS OF THE RESEARCH CYCLE exploration literature study explore domain data collection developing the research question determining available material literature broad gathering of material vrijdag 10 augustus 2012
  • 16. ANALYSIS OF THE RESEARCH CYCLE exploration contextualization literature study literature study explore analysis domain data collection data collection analyzing material re?ning the research question ?nding context ?nding additional material vrijdag 10 augustus 2012
  • 17. ANALYSIS OF THE RESEARCH CYCLE exploration contextualization presentation literature study literature study explore analysis writing report domain data collection data collection vrijdag 10 augustus 2012
  • 18. ANALYSIS OF THE RESEARCH CYCLE exploration contextualization presentation literature study literature study explore analysis writing report domain data collection data collection analysis of data, e.g., watching and transcribing ?lms costs a lot of time, money, and effort. vrijdag 10 augustus 2012
  • 19. DEVELOPMENT OF AN INTERFACE our example research question: how does the representation of women smokers change in magazines and ?ction shows between 1900 and 1950? requirements of the material: time and genre constraints, e.g., ?ction shows a speci?c group, e.g., women smokers is it possible to contrast ?ndings in the data, e.g., a change vrijdag 10 augustus 2012
  • 20. DEVELOPMENT OF AN INTERFACE our example research question: how does the representation of women smokers change in magazines and ?ction shows between 1900 and 1950? requirements of the material: time and genre constraints, e.g., ?ction shows a speci?c group, e.g., women smokers is it possible to contrast ?ndings in the data, e.g., a change Major problem: whether the material satis?es all these criteria is only known after everything has been analyzed! vrijdag 10 augustus 2012
  • 24. DEVELOPMENT OF AN INTERFACE key question: does the option to compare statistics of multiple results sets support media studies researchers in ?nding material that satis?es the contrast requirement in their research questions? vrijdag 10 augustus 2012
  • 28. EXPERIMENTAL SETUP EXPERIMENTAL DESIGN Total of 36 media studies researchers and MSc. students (experience MD = 3; IQR = 0-6.5); between subject design; remote user study; data is 1.2 M tv programs with metadata from audio visual archive. pre search task survey a complex search task (30 mins): Collect material relevant to develop and answer a research question on the topic of how minorities are described in the Dutch media over the last 40 years. After collection, formulate a research question. post search task: write down the research question developed during searching exit survey vrijdag 10 augustus 2012
  • 29. EXPERIMENTAL SETUP EVALUATION subjunctive interface interaction statistics: more query formulations more documents bookmarked wider variety of documents bookmarked (average pairwise similarity) research questions formulated with the subjunctive interface: a more re?ned scope usability of the interface usefulness for research; interesting dif?cult to use; intuitive vrijdag 10 augustus 2012
  • 30. RESULTS INTERACTION STATISTICS showing median and (inter quartile ranges) for the query formulations for the number of documents bookmarked interaction baseline subjunctive queries 3 (2-6) 5 (3-8)* bookmarks 9 (6-22) 9 (3-13) for wider variety of documents bookmarked indicated by the average pairwise cosine similarity interaction baseline subjunctive bookmark variety .62 (.56-.69) .52 (.43-.63)* * signi?cant at the alpha = 0.05 level using Wilcoxon rank sum test vrijdag 10 augustus 2012
  • 31. ANALYSIS INTERACTION PATTERNS Query -> Inspect -> Query -> Inspect is more often seen with the subjunctive interface Query -> Inspect -> Filter -> Inspect is seen more often with the standard exploratory search interface Bookmark -> Paginate -> Bookmark -> Paginate is a frequent pattern when using the baseline interface; pagination does not occur in the top interaction patterns with the subjunctive interface vrijdag 10 augustus 2012
  • 32. ANALYSIS INTERACTION PATTERNS Query -> Inspect -> Query -> Inspect is more often seen with the subjunctive interface Query -> Inspect -> Filter -> Inspect is seen more often with the standard exploratory search interface Bookmark -> Paginate -> Bookmark -> Paginate is a frequent pattern when using the baseline interface; pagination does not occur in the top interaction patterns with the subjunctive interface comparison visualizations invite more queries bookmarking form two result lists i.s.o. single long list promotes more diverse documents vrijdag 10 augustus 2012
  • 33. RESULTS RESEARCH QUESTIONS Do the research questions formulated with the subjunctive interface have a more re?ned scope determined by a contrast or comparison? three judges; experts in the ?eld of media and migration assessed the research questions showing median and (inter quartile ranges)/ 5 point Likert scale 1 is very bad; 5 is very good judge 1 judge 2 judge 3 baseline subjunctive baseline subjunctive baseline subjunctive 3 (3-4) 4 (3-4) 2 (2-4) 2 (2-3) 4 (3-4) 4 (2-4) It is hard too judge whether the scope of research questions is more or less re?ned what if we look at the research questions? vrijdag 10 augustus 2012
  • 34. ANALYSIS RESEARCH QUESTIONS analysis of two research questions... Investigation of the evolution of the term integration in news and human interest programs broadcasted between 1992 and 2012 In 1987 we observe a diminishing in the use of the term migrant worker and a rise in the use of the term immigrant. Is it possible to identify a cause in the broadcasting schedule of that time? interface genre group topics period technical baseline 11 14 8 10 2 subjunctive 10 17 12 9 3 vrijdag 10 augustus 2012
  • 35. ANALYSIS RESEARCH QUESTIONS analysis of two research questions... Investigation of the evolution of the term integration in news and human interest programs broadcasted between 1992 and 2012 In 1987 we observe a diminishing in the use of the term migrant worker and a rise in the use of the term immigrant. Is it possible to identify a cause in the broadcasting schedule of that time? interface genre group topics period technical baseline 11 14 8 10 2 subjunctive 10 17 12 9 3 In research questions formulated with the subjunctive interface the contrast is more concrete. vrijdag 10 augustus 2012
  • 36. RESULTS USABILITY usability of the interfaces median and (inter quartile ranges)/ 5 point Likert scale 1 is absolutely not; 5 is very much interaction baseline subjunctive intuitive 4 (4-4) 4 (3-4) interesting 4 (4-5) 4 (4-5) useful 4 (4-5) 5 (4-5) dif?cult to use 2 (2-2) 2 (2-4) both interfaces are found to be (very) useful for media studies research surprisingly: the subjunctive interface is not considered to be dif?cult to use vrijdag 10 augustus 2012
  • 37. CONCLUSIONS the media studies research process is a dynamic and iterative process, in which it is dif?cult to make research questions concrete until material has been collected and analyzed a subjunctive exploratory interface supports media researchers in exploration: it invites more query formulations and more diverse documents to be gathered formulating research questions in which a contrast is considered based on patterns in the material observed during exploration moreover the subjunctive interface, while more complex, is not considered to be dif?cult to use and is considered very useful for media studies research vrijdag 10 augustus 2012
  • 38. CONCLUSIONS the media studies research process is a dynamic and iterative process, in which it is dif?cult to make research questions concrete until material has been collected and analyzed a subjunctive exploratory interface supports media researchers in exploration: it invites more query formulations and more diverse documents to be gathered formulating research questions in which a contrast is considered based on patterns in the material observed during exploration moreover the subjunctive interface, while more complex, is not considered to be dif?cult to use and is considered very useful for media studies research Thank you; are there any questions? vrijdag 10 augustus 2012