ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

2007 Ala

?Download as PPT, PDF?
?
S
Sarah Maximiek

Presentation given for RSS RUSA at the 2007 ALA Annual Conference on the transcript analysis project at Binghamton University Libraries.

1 of 24
Downloaded 43 times
Evaluating the Quality of Instant Messaging Reference Service through Transcript Analysis 13th Annual Reference Research Forum Sunday, June 24th Sarah Maximiek [email_address] Erin Rushton [email_address] . edu Binghamton University
Binghamton University FTE 14,000 + 769 faculty Two libraries: Bartle Library (Humanities, Social Sciences); Science Library (Sciences, Psychology, Nursing) Undergraduate through Doctorate
Background Fall 2003 ¨C Launched virtual reference service using QuestionPoint Fall 2004 ¨C Migrated to Docutek VRLPlus March 2005 ¨C Digital Reference Committee (DRC) formed June 2005 ¨C IM pilot service launched using Trillian
Background 6 IM accounts: 2 usernames across 3 services (Yahoo, MSN, AOL) BuMainLib: Bartle Library BuSciLib: Science Library Service mostly staffed at Reference service point Service staffed either at 1 or 2 libraries Staffed by librarians, staff, and grad students
Transcript Analysis Objectives Evaluate quality of the service and recommend improvements Produce data that describes the use and users of the service Recommend changes for library services, web design and collections based on needs of virtual users
Methodology Reviewed literature to determine what qualitative and quantitative factors to measure and evaluate Printed 284 transcripts from June 2005 to June 2006 Data from transcripts was collected on an Access form
Methodology Transcripts except simple catalogue look-ups and directional questions were double analyzed Coding key help avoid ambiguity and data was cross-checked by the DRC DRC analyzed data
Demographics 31% of users were identified as students
Session Length
Sessions by Day
Sessions by Day and Time A.M. P.M.
Sessions by Month
Types of Reference Questions *More than one question type could be selected per transcript
Instructional Questions 23% of questions demonstrated instruction Instruction challenges: No co-browsing Multitasking multiple reference services (in-person, phone, email, and IM) Sometimes easier to just give the patron the answer Tutorials and guides may help when its not possible to provide instruction For further discussion see: Stephanie Graves and Christina M. Desai. Instruction via chat reference: does co-browse help? Reference Services Review, Vol. 34 No. 3, 2006, pp. 340-357
Research/Quick Reference 22% research/quick reference Patrons feel comfortable asking in-depth research questions Need time, staffing, resources and expertise to answer research questions through IM
Resources Used to Answer Question *More than one question type could be selected per transcript
RUSA Guidelines for Behavioral Performance friendly greeting cordial, and encouraging uses a tone of voice and/or written language appropriate maintains or re-establishes "word contact" allows the patrons to state fully their information need rephrases the question or request and asks for confirmation uses open-ended questioning techniques finds out what patrons have already tried, and encourages patrons to contribute ideas explains the search strategy and sequence to the user
Correctness and Completeness Not complete : incomplete reference interview OR no referral/follow-up Not correct : used for incorrect answers Model adapted from ¡°Evaluating the Quality of a Chat Service. Julie Arnold. Neal Kaske. portal: Libraries and the Academy 5.2 (2005) 177-193
Next Steps Further analysis to determine: Staffing levels FAQ¡¯s Quality of service especially when provided by grad students Referrals Improvements to RUSA Guidelines for Behavioral Performance Recommendations to Head of Research, Information & Instructional Services User satisfaction Solicit feedback from colleagues Provide additional/ongoing training Revise transcript analysis strategy and continue analyzing transcripts
Findings 76% of sessions answered in under 1 minute Mean session length was 1h and 9 minutes, mode was 2 minutes Over 85% of questions were answered correctly IM traffic patterns mirrors in-person reference traffic Web navigation questions are most common Variety of questions from basic reference to in-depth research questions RUSA Behaviors were demonstrated
Bibliography Arnold, Julie, and Neal K. Kaske. ¡°Evaluating the Quality of a Chat Service.¡± portal: Libraries and the Academy . 5.2 (2005): 177-193. 22 November 2006 <http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v005/5.2arnold.html> Diamond, and Pease. ¡°Digital reference: a case study of question types in an academic library.¡± Reference Services Review . 29.3 (2001): 210-219. 22 November 2006 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400290305.html>. Graves, Stephanie J., and Christina M. Desai. ¡°Instruction Via Chat Reference: Does Co-browse Help?¡± Reference Services Review . 34.3 (2006): 340 -357. 1 June 2007 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400340303.html>. Houlson, Van, Kate McCready, and Carla Stienberg Pfahl. ¡°A Window into Our Patron's Needs: Analyzing Data from Chat Transcripts.¡± Internet Reference Services Quarterly . 11.4 (2006): 19 - 39. Luo, Lili. ¡°Chat Reference Competencies: Identification from a Literature Review and Librarian Interviews.¡± Reference Services Review . 35.2 (2007): 195-209. McClure, Charles, Lankes, R. David, Gross, Melissa, and Choltco-Devlin, Beverly. Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures. . 22 November 2006 <http://eric.ed.gov/ERICWebPortal/Home.portal?_nfpb=true&_pageLabel=RecordDetails&ERICExtSearch_SearchValue_0=ED472588&ERICExtSearch_SearchType_0=eric_accno&objectId=0900000b8017a71e>. Paster, amy, and Kathy Fescemyer. ¡°Assessing Reference: Using the Wisconsin-Ohio Reference Evaluation Program in an Academic Science Library.¡± Issues in Science and Technology Librarianship . (2006). Sears, JoAnn. ¡°Chat Reference Service: An Analysis of one Semester's Data.¡± Issues in Science and Technology Librarianship . (2001). 22 November 2006 <http://scholar.google.com/scholar?hl=en&lr=&q=%22chat+reference+service+an+analysis%22&btnG=Search>. Smyth, Joanne. ¡°Virtual Reference Transcript Analysis: A Few Models.¡± Searcher . 11.3 (2003). 22 November 2006 <http://referenc.lib.binghamton.edu:2510/ehost/detail?vid=2&hid=1&sid=a85713a4-9ac9-4d51-84cf-9f6420ea2d7a%40sessionmgr2>. Ward, David. ¡°Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study.¡± Reference and User Services Quarterly . 44.1 (2004): 46-56.
Sample Transcript [09:59] hello [09:59] i have a quick question [09:59] would like to know where i can find the JSTOR database [09:59] BUmainlib: ok what can i hlep you with [09:59] BUmainlib: oh ok [10:00] thx alot [10:00] BUmainlib: go to METALINK on the library homepage [10:00] BUmainlib: and in the find database search box type in jstor [10:00] BUmainlib: the link will appear [10:01] oooo [10:01] BUmainlib: let me know if you find it [10:01] ooo lol that was easy [10:01] i found it!!:) [10:01] BUmainlib: oh good! [10:01] BUmainlib: any other questions for me? [10:01] no thats it [10:01] thx so much [10:01] BUmainlib: ok have a good morning then! you are welcome! [10:01] BUmainlib: bye [10:02] u 2 thanx again
Sample Transcript [15:11] hello [15:11] hello [15:12] can you help me i am trying to locate information on &quot; normalizing constant factors&quot; in Bio Stats? [15:13] What type of information do you need? Is Bio Stats a textbook? [15:13] its a class biostatistics , i need to see how it is expressed mathamaticly [15:15] So you just need the formula? Let me look a few places [15:15] outstanding :-) [15:17] I found this article in Wikipedia. Wikipedia is always 100 percent accurate, but on something like this, you can probably trust it. (Link: http://en.wikipedia.org/wiki/Normalization_factor)http://en.wikipedia.org/wiki/Normalization_factor [15:20] Is that what you were looking for? [15:20] yes the formula [15:20] Any other questions? [15:20] :-) thats it [15:20] why that wasnt hard enough lol [15:21] have a good day. bye
[12:29] hello, is this the right screen name for help with psychology research? [12:30] yes - what are you working on? [12:30] i am working on a research paper focusing on the effects noise has on memory [12:31] where have you started your research? [12:32] i looked online with psycArticles and psycINFO [12:32] do you need articles, books, either, both? [12:32] im mainly looking for articles [12:32] ok [12:33] I think you were on the right track with PsycINFO [12:33] I'm giving it a try... [12:33] ok...thank you [12:34] i found that a lot of the articles i found were very loosely related to the topic, they didnt have enough information to be one of my actual references [12:34] are you looking for studies on humans or animals? [12:34] humans [12:34] ok [12:38] I'm trying a search of the descriptors of (auditory stimulation or distraction) and memory [12:39] I'd like to email you a couple of results to know if this is on the right track [12:39] ok my email is XXXXX [12:39] ok, hang on [12:39] unfortunately i have to go to a class but i will check the articles [12:39] ok [12:40] thank you very much

More Related Content

2007 Ala

  • 1. Evaluating the Quality of Instant Messaging Reference Service through Transcript Analysis 13th Annual Reference Research Forum Sunday, June 24th Sarah Maximiek [email_address] Erin Rushton [email_address] . edu Binghamton University
  • 2. Binghamton University FTE 14,000 + 769 faculty Two libraries: Bartle Library (Humanities, Social Sciences); Science Library (Sciences, Psychology, Nursing) Undergraduate through Doctorate
  • 3. Background Fall 2003 ¨C Launched virtual reference service using QuestionPoint Fall 2004 ¨C Migrated to Docutek VRLPlus March 2005 ¨C Digital Reference Committee (DRC) formed June 2005 ¨C IM pilot service launched using Trillian
  • 4. Background 6 IM accounts: 2 usernames across 3 services (Yahoo, MSN, AOL) BuMainLib: Bartle Library BuSciLib: Science Library Service mostly staffed at Reference service point Service staffed either at 1 or 2 libraries Staffed by librarians, staff, and grad students
  • 5. Transcript Analysis Objectives Evaluate quality of the service and recommend improvements Produce data that describes the use and users of the service Recommend changes for library services, web design and collections based on needs of virtual users
  • 6. Methodology Reviewed literature to determine what qualitative and quantitative factors to measure and evaluate Printed 284 transcripts from June 2005 to June 2006 Data from transcripts was collected on an Access form
  • 7. Methodology Transcripts except simple catalogue look-ups and directional questions were double analyzed Coding key help avoid ambiguity and data was cross-checked by the DRC DRC analyzed data
  • 8. Demographics 31% of users were identified as students
  • 11. Sessions by Day and Time A.M. P.M.
  • 13. Types of Reference Questions *More than one question type could be selected per transcript
  • 14. Instructional Questions 23% of questions demonstrated instruction Instruction challenges: No co-browsing Multitasking multiple reference services (in-person, phone, email, and IM) Sometimes easier to just give the patron the answer Tutorials and guides may help when its not possible to provide instruction For further discussion see: Stephanie Graves and Christina M. Desai. Instruction via chat reference: does co-browse help? Reference Services Review, Vol. 34 No. 3, 2006, pp. 340-357
  • 15. Research/Quick Reference 22% research/quick reference Patrons feel comfortable asking in-depth research questions Need time, staffing, resources and expertise to answer research questions through IM
  • 16. Resources Used to Answer Question *More than one question type could be selected per transcript
  • 17. RUSA Guidelines for Behavioral Performance friendly greeting cordial, and encouraging uses a tone of voice and/or written language appropriate maintains or re-establishes &quot;word contact&quot; allows the patrons to state fully their information need rephrases the question or request and asks for confirmation uses open-ended questioning techniques finds out what patrons have already tried, and encourages patrons to contribute ideas explains the search strategy and sequence to the user
  • 18. Correctness and Completeness Not complete : incomplete reference interview OR no referral/follow-up Not correct : used for incorrect answers Model adapted from ¡°Evaluating the Quality of a Chat Service. Julie Arnold. Neal Kaske. portal: Libraries and the Academy 5.2 (2005) 177-193
  • 19. Next Steps Further analysis to determine: Staffing levels FAQ¡¯s Quality of service especially when provided by grad students Referrals Improvements to RUSA Guidelines for Behavioral Performance Recommendations to Head of Research, Information & Instructional Services User satisfaction Solicit feedback from colleagues Provide additional/ongoing training Revise transcript analysis strategy and continue analyzing transcripts
  • 20. Findings 76% of sessions answered in under 1 minute Mean session length was 1h and 9 minutes, mode was 2 minutes Over 85% of questions were answered correctly IM traffic patterns mirrors in-person reference traffic Web navigation questions are most common Variety of questions from basic reference to in-depth research questions RUSA Behaviors were demonstrated
  • 21. Bibliography Arnold, Julie, and Neal K. Kaske. ¡°Evaluating the Quality of a Chat Service.¡± portal: Libraries and the Academy . 5.2 (2005): 177-193. 22 November 2006 <http://muse.jhu.edu/journals/portal_libraries_and_the_academy/v005/5.2arnold.html> Diamond, and Pease. ¡°Digital reference: a case study of question types in an academic library.¡± Reference Services Review . 29.3 (2001): 210-219. 22 November 2006 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400290305.html>. Graves, Stephanie J., and Christina M. Desai. ¡°Instruction Via Chat Reference: Does Co-browse Help?¡± Reference Services Review . 34.3 (2006): 340 -357. 1 June 2007 <http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/EmeraldFullTextArticle/Articles/2400340303.html>. Houlson, Van, Kate McCready, and Carla Stienberg Pfahl. ¡°A Window into Our Patron's Needs: Analyzing Data from Chat Transcripts.¡± Internet Reference Services Quarterly . 11.4 (2006): 19 - 39. Luo, Lili. ¡°Chat Reference Competencies: Identification from a Literature Review and Librarian Interviews.¡± Reference Services Review . 35.2 (2007): 195-209. McClure, Charles, Lankes, R. David, Gross, Melissa, and Choltco-Devlin, Beverly. Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures. . 22 November 2006 <http://eric.ed.gov/ERICWebPortal/Home.portal?_nfpb=true&_pageLabel=RecordDetails&ERICExtSearch_SearchValue_0=ED472588&ERICExtSearch_SearchType_0=eric_accno&objectId=0900000b8017a71e>. Paster, amy, and Kathy Fescemyer. ¡°Assessing Reference: Using the Wisconsin-Ohio Reference Evaluation Program in an Academic Science Library.¡± Issues in Science and Technology Librarianship . (2006). Sears, JoAnn. ¡°Chat Reference Service: An Analysis of one Semester's Data.¡± Issues in Science and Technology Librarianship . (2001). 22 November 2006 <http://scholar.google.com/scholar?hl=en&lr=&q=%22chat+reference+service+an+analysis%22&btnG=Search>. Smyth, Joanne. ¡°Virtual Reference Transcript Analysis: A Few Models.¡± Searcher . 11.3 (2003). 22 November 2006 <http://referenc.lib.binghamton.edu:2510/ehost/detail?vid=2&hid=1&sid=a85713a4-9ac9-4d51-84cf-9f6420ea2d7a%40sessionmgr2>. Ward, David. ¡°Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study.¡± Reference and User Services Quarterly . 44.1 (2004): 46-56.
  • 22. Sample Transcript [09:59] hello [09:59] i have a quick question [09:59] would like to know where i can find the JSTOR database [09:59] BUmainlib: ok what can i hlep you with [09:59] BUmainlib: oh ok [10:00] thx alot [10:00] BUmainlib: go to METALINK on the library homepage [10:00] BUmainlib: and in the find database search box type in jstor [10:00] BUmainlib: the link will appear [10:01] oooo [10:01] BUmainlib: let me know if you find it [10:01] ooo lol that was easy [10:01] i found it!!:) [10:01] BUmainlib: oh good! [10:01] BUmainlib: any other questions for me? [10:01] no thats it [10:01] thx so much [10:01] BUmainlib: ok have a good morning then! you are welcome! [10:01] BUmainlib: bye [10:02] u 2 thanx again
  • 23. Sample Transcript [15:11] hello [15:11] hello [15:12] can you help me i am trying to locate information on &quot; normalizing constant factors&quot; in Bio Stats? [15:13] What type of information do you need? Is Bio Stats a textbook? [15:13] its a class biostatistics , i need to see how it is expressed mathamaticly [15:15] So you just need the formula? Let me look a few places [15:15] outstanding :-) [15:17] I found this article in Wikipedia. Wikipedia is always 100 percent accurate, but on something like this, you can probably trust it. (Link: http://en.wikipedia.org/wiki/Normalization_factor)http://en.wikipedia.org/wiki/Normalization_factor [15:20] Is that what you were looking for? [15:20] yes the formula [15:20] Any other questions? [15:20] :-) thats it [15:20] why that wasnt hard enough lol [15:21] have a good day. bye
  • 24. [12:29] hello, is this the right screen name for help with psychology research? [12:30] yes - what are you working on? [12:30] i am working on a research paper focusing on the effects noise has on memory [12:31] where have you started your research? [12:32] i looked online with psycArticles and psycINFO [12:32] do you need articles, books, either, both? [12:32] im mainly looking for articles [12:32] ok [12:33] I think you were on the right track with PsycINFO [12:33] I'm giving it a try... [12:33] ok...thank you [12:34] i found that a lot of the articles i found were very loosely related to the topic, they didnt have enough information to be one of my actual references [12:34] are you looking for studies on humans or animals? [12:34] humans [12:34] ok [12:38] I'm trying a search of the descriptors of (auditory stimulation or distraction) and memory [12:39] I'd like to email you a couple of results to know if this is on the right track [12:39] ok my email is XXXXX [12:39] ok, hang on [12:39] unfortunately i have to go to a class but i will check the articles [12:39] ok [12:40] thank you very much