際際滷

際際滷Share a Scribd company logo

Measuring the Wind: Metrics in Technical Services

M
Morgan McCune

The document discusses the importance of metrics in technical services departments in libraries. It begins by outlining different types of metrics that can be used to measure individual, departmental, library, and university performance. These include quantitative metrics like number of records processed, as well as qualitative metrics like accuracy rates. The document then discusses how metrics can be used for training, performance evaluations, benchmarking, planning, and assessing how well the library is meeting its mission and supporting the university's goals. Overall, the document advocates for the implementation of comprehensive metrics programs in technical services to facilitate accountability, continuous improvement, and effective communication of the department's value and contributions.

1 of 50
Downloaded 92 times
Metrics in Technical Services Morgan O.H. McCune Pittsburg State University, Pittsburg, Kansas 8th Annual Brick and Click Libraries, Nov. 7, 2008 11/7/2008
Beaufort Scale 00 0Calm 1Light Air 2Light Breeze 3Gentle Breeze 4Moderate Breeze 5Fresh Breeze 6Strong Breeze 7Near Gale 8Fresh Gale 9Strong Gale 10Whole Gale 11Violent Storm 11/7/2008 12Hurricane
The dogmas of the quiet past are inadequate to the stormy present. The occasion is piled high with difficulty, and we must rise with the occasion. As our case is new, so we must think anew and act anew. --Abraham Lincoln, Dec. 1, 1862 http://showcase.netins.net/web/creative/lincoln/speeches/congress.htm 11/7/2008
In a stormy present, how do we Measure performance? Define excellent service? Account for what we do? Record history? Chart our course into the future? Show how we count? 11/7/2008
Educated guesses are not always correct. Sometimes the storm clouds of change (or emotion) hide our guiding stars. Metrics give us real measures by which to navigate. 11/7/2008
Definition: Metric A metric is a standard unit of measure or more generally, part of a system of parameters, or systems of measurement, or a set of ways of quantitatively/qualitatively and periodically measuring, assessing, controlling or selecting a person, process, event, or institution, along with the procedures to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments. http://en.wikipedia.org/wiki/Metrics 10/14/2008 11/7/2008
Definition: Quantity/Quality QuantitativeHow many? How fast? 25 bibliographic records per day 25 authority records per month QualitativeHow good? 25 copy bibliographic records per day 10 original records per day 25 Tables of Contents added 16 access points added per 100 records 2 errors per 100=2% error rate 11/7/2008
DefinitionExtent/Accuracy More on quality-- though it is often assumed and praised in the literature of bibliographic control, it doesnt seem to be well delineated(Graham, 214) 25 TOC added/30 access points added (extent: how much information is provided in the record) 2 errors per 100=2% error rate (accuracy: the correctness of what is provided) 11/7/2008
Individual Performance Paraprofessional Professional Tenure-track Annual reviews Special recognitions/awards Tenure documents 11/7/2008
Department Performance Acquisitions Cataloging Physical processing Reports to Dean/Manager 11/7/2008
Library Performance Technical services Reference Circulation etc. Reports to University President 11/7/2008
University Performance Learning Resources (Library) Biology Dept. Communications Dept. etc. Reports to Regents, Boards, etc. 11/7/2008
Individual Performance Training Performance evaluations/reviews Tenure 11/7/2008
Training New catalogers need: Specific, measurable, and achievable objectives (Charbonneau, 42) Evaluation based on real performance To be encouraged to set their own benchmarks and to participate in establishing unit benchmarks Standards that include both quantity and quality 11/7/2008
Definition: Benchmark noun 1. a standard of excellence, achievement, etc., against which similar things must be measured or judged 2. any standard of reference by which others can be measured or judged adjective of, pertaining to, or resulting in a benchmark: benchmark test, benchmark study www.dictionary.com, viewed 10/30/2008 11/7/2008
Benchmark An internal study of my own work/the units work, using quality and quantity measures, might help me/us establish a benchmark for myself/the unit against which I/we can measure performance/s. A less experienced employee might compare their performance against a benchmark of experience to know when they are up to speed Keeping stats 11/7/2008
11/7/2008
Time Analysis Allows for computation of cataloging speed (10 records per 2 hours) when job includes many duties Gives a clear picture of how much time a special project is taking from regular duties Useful in making decisions about reorganizations and staff levels Sometimes yields surprises! 11/7/2008
Tracking Time 11/7/2008
Ongoing Performance Evaluation Individual staff members Teams Praise and room for improvement comments based on real data, along with specific goals and objectives. In fact, all catalogers need what new catalogers need. The winds of change make this even more important. Metrics must change as the environment changes. 11/7/2008
Despite the negativity often leveled at production requirements, individual and departmental performance nearly always improves when the organization establishes specific expectations, tracks productivity, and rewards high volume, high quality throughput. Even professionals should be held accountable for the way they spend their time and for a measurable level of production. (Fischer, Lugg, and Boese, 3) 11/7/2008
Tenure Core performance factors: Professional communication; Expertise and technical knowledge; Creativity/innovation; Planning and organization; Accountability; Resource management The candidate seeking tenure or promotion will create a dossier documenting his or her effectiveness as a productive faculty member of the Library and the University. Excerpts from PSU Axe Library Faculty Promotion/Tenure Guidelines: http://library.pittstate.edu/office/FacultyTenure_PromotionGuidelines.pdf , viewed 11/5/2008 11/7/2008
My Tenure Portfolio Should demonstrate my understanding of my part of the institutional puzzle; my mission; department mission; the library mission; the university mission Will include metrics and time studies Must communicate my performance to non-library faculty 11/7/2008
Department Performance Cost analysis Workflow analysis Benchmarking Planning Decision making 11/7/2008
Cost Analysis How much does it cost to catalog a resource? How much would it cost to catalog at a higher/lower level of quality? (extent) How much does it cost to deliver material to the customer? (including purchase price and all processing) 11/7/2008
Definition: Input Inputs or input measures Easiest to quantify Typically grouped into five broad categories: budget, staff, collections, facilities, technology (Matthews, 20) Input: 100 new books, 2 staff members, 2 workstations, access to WorldCat 11/7/2008
Definition: Output Outputs or output measures Indicate the degree to which library and its services are utilized More often than not, are simply counts to indicate volume of activity (Matthews, 20) Output: 100 book records (10 original; 90 copy); 15 authority records; 25 books checked out in 5 weeks 11/7/2008
Workflow Analysis Process design/Flowcharting Time Identify bottlenecks Identify problems with space layout; movement between process areas Identify unnecessarily duplicated work Assign tasks most efficiently 11/7/2008
Benchmarking a Department/Library In the library literature, the term benchmarking is used most often in reference to comparing a librarys or units performance against that of a peer agency or unit, in order to improve performance and identify best practices. Library A outputs more cataloging records per month than Library B. Why? How does our performance compare to a tech services dept with similar acquisitions, staff, and processes? 11/7/2008
Planning/Decision Making Outsourcing, yes or no Request additional funds/positions Protect existing positions Special projects Eliminate backlogs without ceasing incoming work Reorganization of library departments Setting departmental standards 11/7/2008
Library Performance Assessment 11/7/2008
Assessment/Mission assessment of library performance should be defined and shaped by its connections and contributions to institutional goals and desired education outcomes. (Lindauer, 559) 11/7/2008
The Mission Statement provide materials and services that effectively enable members of the University community to attain their teaching and learning, discovery, and engagement goals creation and maintenance of collections efficient and effective services to all who use the Library strategic master planning links these goals, objectives, and strategies to the University budget Excerpts from Axe Library mission statement: http://library.pittstate.edu/office/ 11/7/2008
The Assessment Loop Reiterative, ongoing process; re-planning and continuous improvement Implement Evaluate Plan Use Results 11/7/2008
Definition: Culture of Assessment An organizational attitude that can be achieved by creating systems and structures that are based on continuous assessment and evaluation. A culture that is customer focused and uses assessment systematically (Dudden, 20) 11/7/2008
Balanced Scorecard (BSC) Looking at more than just the bottom line Kaplan, Robert, and David Norton. Balanced Scorecard: Translating Strategy Into Action. Boston: Harvard Business School Press, 1996. The system provides a method of aligning business activities to the vision and strategy of the organization, improving internal and external communications, and monitoring organization performance against strategic goals. Balanced Scorecard Institute www.balancedscorecard.org 11/7/2008
Balanced Scorecard at UVa User; Finance; Internal processes; and, Learning and the future Four to eight metrics are selected for each perspective, each metric with a specific target. At the end of the measurement period there should be no question as to whether the organization has successfully met the target. (Self, 57-58) 11/7/2008
Internal processes Acquire, create, organize, preserve and deliver information resources in a timely, efficient, and accurate manner MetricProcessing time for routine acquisitions Target 1Process 90% of in-print books from North America within one month http://www.lib.virginia.edu/bsc/metrics/all0708.html viewed 11/5/2008 11/7/2008
Definition: Dashboard More than a picture 90 00 1 80 5 44 87 2 70 52 3 60 4 50 49 5 40 6 124 6 7 30 6 70 8 20 9 99 10 10 0 1 2 3 4 5 6 7 11/7/2008
Dashboard Tied to real measurements Cataloging Reference Speed Delivery in real time Satisfaction 10 day Intranet/Internet 96% turnaround Visual picture of whats happening now New Reference Acquisitions Accuracy 100 95% 11/7/2008
晦庄恢河雨粥晦+ www.libqual.org a suite of services that libraries use to solicit, track, understand, and act upon users opinions of service quality Cost: $3,200.00 per institution (Website viewed 11/5/2008) 11/7/2008
晦庄恢河雨粥晦+ Survey instrument adapted from SERVQUAL (Parasuraman, Zeithaml, Berry) Library asks users to take the survey. Survey answers are sent to central database, analyzed and return to library in reports describing users experience of library service. Measures three dimensions of service quality: Information Control; Affect of Service; Library as Place. USER FOCUSED! 11/7/2008
University Performance Everything the library does should tie in with the University mission; the library is part of the performance of the entire University. 11/7/2008
OUTPUTS OUTCOMES For academic libraries: educational outcomes In universities and other school settings, librarians are being asked not just whether they provided learning resources but also if their service contributed to the success of the student (Dudden, 9) 11/7/2008
Those of us working in technical services and cataloging in particular need to find ways to assess the impact of our work on the population served, and to communicate this information (Show how we count!) Catalog use studies Catalog accuracy studies Something new? Make sure to assess service to both Brick and Click patrons! 11/7/2008
Questions to Ask About Metrics Do we have a culture of assessment that will enable ongoing measurement? How can we foster a culture of assessment? Are our plans based on our mission? What funding is available or can be made available for implementation of a long-term assessment project? 11/7/2008
More questions Are we measuring something that is actionable? Is it something that can be changed? Are we measuring something that is relevant (fits the mission)? Is our process iterative? Are we using the results in re- planning? Are we seeing continuous improvement? Is the process fostering the culture of assessment? 11/7/2008
Fair Winds! mmccune@pittstate.edu 11/7/2008
Works Cited Charbonneau, Mechael D. Production Benchmarks for Catalogers in Academic Libraries: Are We There Yet? Library Resources & Technical Services 49.1 (2005): 40-48. Dudden, Rosalind Farnam. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-To- Do-It-Manual With CD-ROM. New York: Neal-Schuman, 2007. Fischer, Ruth, Rick Lugg, and Kent C. Boese. Cataloging: How to Take a Business Approach. First published in The Bottom Line (2004). http://www.ebookmap.net/pages/Publications.php Graham, Peter S. Quality in Cataloging: Making Distinctions. Journal of Academic Librarianship 16.4 (1990): 213-218. 11/7/2008
Works Cited (cont.) Lindauer, Bonnie Gratch. Defining and Measuring the Librarys Impact on Campuswide Outcomes. College & Research Libraries 59.6 (1998): 546-570. Matthews, Joseph R. Evaluation and Measurement of Library Services. Westport, Conn.: Libraries Unlimited, 2007. Self, James. From Values to Metrics: Implementation of the Balanced Scorecard at a University Library. Performance Measurement and Metrics 4.2 (2003): 57-63. 11/7/2008

More Related Content

Measuring the Wind: Metrics in Technical Services

  • 1. Metrics in Technical Services Morgan O.H. McCune Pittsburg State University, Pittsburg, Kansas 8th Annual Brick and Click Libraries, Nov. 7, 2008 11/7/2008
  • 2. Beaufort Scale 00 0Calm 1Light Air 2Light Breeze 3Gentle Breeze 4Moderate Breeze 5Fresh Breeze 6Strong Breeze 7Near Gale 8Fresh Gale 9Strong Gale 10Whole Gale 11Violent Storm 11/7/2008 12Hurricane
  • 3. The dogmas of the quiet past are inadequate to the stormy present. The occasion is piled high with difficulty, and we must rise with the occasion. As our case is new, so we must think anew and act anew. --Abraham Lincoln, Dec. 1, 1862 http://showcase.netins.net/web/creative/lincoln/speeches/congress.htm 11/7/2008
  • 4. In a stormy present, how do we Measure performance? Define excellent service? Account for what we do? Record history? Chart our course into the future? Show how we count? 11/7/2008
  • 5. Educated guesses are not always correct. Sometimes the storm clouds of change (or emotion) hide our guiding stars. Metrics give us real measures by which to navigate. 11/7/2008
  • 6. Definition: Metric A metric is a standard unit of measure or more generally, part of a system of parameters, or systems of measurement, or a set of ways of quantitatively/qualitatively and periodically measuring, assessing, controlling or selecting a person, process, event, or institution, along with the procedures to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments. http://en.wikipedia.org/wiki/Metrics 10/14/2008 11/7/2008
  • 7. Definition: Quantity/Quality QuantitativeHow many? How fast? 25 bibliographic records per day 25 authority records per month QualitativeHow good? 25 copy bibliographic records per day 10 original records per day 25 Tables of Contents added 16 access points added per 100 records 2 errors per 100=2% error rate 11/7/2008
  • 8. DefinitionExtent/Accuracy More on quality-- though it is often assumed and praised in the literature of bibliographic control, it doesnt seem to be well delineated(Graham, 214) 25 TOC added/30 access points added (extent: how much information is provided in the record) 2 errors per 100=2% error rate (accuracy: the correctness of what is provided) 11/7/2008
  • 9. Individual Performance Paraprofessional Professional Tenure-track Annual reviews Special recognitions/awards Tenure documents 11/7/2008
  • 10. Department Performance Acquisitions Cataloging Physical processing Reports to Dean/Manager 11/7/2008
  • 11. Library Performance Technical services Reference Circulation etc. Reports to University President 11/7/2008
  • 12. University Performance Learning Resources (Library) Biology Dept. Communications Dept. etc. Reports to Regents, Boards, etc. 11/7/2008
  • 13. Individual Performance Training Performance evaluations/reviews Tenure 11/7/2008
  • 14. Training New catalogers need: Specific, measurable, and achievable objectives (Charbonneau, 42) Evaluation based on real performance To be encouraged to set their own benchmarks and to participate in establishing unit benchmarks Standards that include both quantity and quality 11/7/2008
  • 15. Definition: Benchmark noun 1. a standard of excellence, achievement, etc., against which similar things must be measured or judged 2. any standard of reference by which others can be measured or judged adjective of, pertaining to, or resulting in a benchmark: benchmark test, benchmark study www.dictionary.com, viewed 10/30/2008 11/7/2008
  • 16. Benchmark An internal study of my own work/the units work, using quality and quantity measures, might help me/us establish a benchmark for myself/the unit against which I/we can measure performance/s. A less experienced employee might compare their performance against a benchmark of experience to know when they are up to speed Keeping stats 11/7/2008
  • 18. Time Analysis Allows for computation of cataloging speed (10 records per 2 hours) when job includes many duties Gives a clear picture of how much time a special project is taking from regular duties Useful in making decisions about reorganizations and staff levels Sometimes yields surprises! 11/7/2008
  • 20. Ongoing Performance Evaluation Individual staff members Teams Praise and room for improvement comments based on real data, along with specific goals and objectives. In fact, all catalogers need what new catalogers need. The winds of change make this even more important. Metrics must change as the environment changes. 11/7/2008
  • 21. Despite the negativity often leveled at production requirements, individual and departmental performance nearly always improves when the organization establishes specific expectations, tracks productivity, and rewards high volume, high quality throughput. Even professionals should be held accountable for the way they spend their time and for a measurable level of production. (Fischer, Lugg, and Boese, 3) 11/7/2008
  • 22. Tenure Core performance factors: Professional communication; Expertise and technical knowledge; Creativity/innovation; Planning and organization; Accountability; Resource management The candidate seeking tenure or promotion will create a dossier documenting his or her effectiveness as a productive faculty member of the Library and the University. Excerpts from PSU Axe Library Faculty Promotion/Tenure Guidelines: http://library.pittstate.edu/office/FacultyTenure_PromotionGuidelines.pdf , viewed 11/5/2008 11/7/2008
  • 23. My Tenure Portfolio Should demonstrate my understanding of my part of the institutional puzzle; my mission; department mission; the library mission; the university mission Will include metrics and time studies Must communicate my performance to non-library faculty 11/7/2008
  • 24. Department Performance Cost analysis Workflow analysis Benchmarking Planning Decision making 11/7/2008
  • 25. Cost Analysis How much does it cost to catalog a resource? How much would it cost to catalog at a higher/lower level of quality? (extent) How much does it cost to deliver material to the customer? (including purchase price and all processing) 11/7/2008
  • 26. Definition: Input Inputs or input measures Easiest to quantify Typically grouped into five broad categories: budget, staff, collections, facilities, technology (Matthews, 20) Input: 100 new books, 2 staff members, 2 workstations, access to WorldCat 11/7/2008
  • 27. Definition: Output Outputs or output measures Indicate the degree to which library and its services are utilized More often than not, are simply counts to indicate volume of activity (Matthews, 20) Output: 100 book records (10 original; 90 copy); 15 authority records; 25 books checked out in 5 weeks 11/7/2008
  • 28. Workflow Analysis Process design/Flowcharting Time Identify bottlenecks Identify problems with space layout; movement between process areas Identify unnecessarily duplicated work Assign tasks most efficiently 11/7/2008
  • 29. Benchmarking a Department/Library In the library literature, the term benchmarking is used most often in reference to comparing a librarys or units performance against that of a peer agency or unit, in order to improve performance and identify best practices. Library A outputs more cataloging records per month than Library B. Why? How does our performance compare to a tech services dept with similar acquisitions, staff, and processes? 11/7/2008
  • 30. Planning/Decision Making Outsourcing, yes or no Request additional funds/positions Protect existing positions Special projects Eliminate backlogs without ceasing incoming work Reorganization of library departments Setting departmental standards 11/7/2008
  • 31. Library Performance Assessment 11/7/2008
  • 32. Assessment/Mission assessment of library performance should be defined and shaped by its connections and contributions to institutional goals and desired education outcomes. (Lindauer, 559) 11/7/2008
  • 33. The Mission Statement provide materials and services that effectively enable members of the University community to attain their teaching and learning, discovery, and engagement goals creation and maintenance of collections efficient and effective services to all who use the Library strategic master planning links these goals, objectives, and strategies to the University budget Excerpts from Axe Library mission statement: http://library.pittstate.edu/office/ 11/7/2008
  • 34. The Assessment Loop Reiterative, ongoing process; re-planning and continuous improvement Implement Evaluate Plan Use Results 11/7/2008
  • 35. Definition: Culture of Assessment An organizational attitude that can be achieved by creating systems and structures that are based on continuous assessment and evaluation. A culture that is customer focused and uses assessment systematically (Dudden, 20) 11/7/2008
  • 36. Balanced Scorecard (BSC) Looking at more than just the bottom line Kaplan, Robert, and David Norton. Balanced Scorecard: Translating Strategy Into Action. Boston: Harvard Business School Press, 1996. The system provides a method of aligning business activities to the vision and strategy of the organization, improving internal and external communications, and monitoring organization performance against strategic goals. Balanced Scorecard Institute www.balancedscorecard.org 11/7/2008
  • 37. Balanced Scorecard at UVa User; Finance; Internal processes; and, Learning and the future Four to eight metrics are selected for each perspective, each metric with a specific target. At the end of the measurement period there should be no question as to whether the organization has successfully met the target. (Self, 57-58) 11/7/2008
  • 38. Internal processes Acquire, create, organize, preserve and deliver information resources in a timely, efficient, and accurate manner MetricProcessing time for routine acquisitions Target 1Process 90% of in-print books from North America within one month http://www.lib.virginia.edu/bsc/metrics/all0708.html viewed 11/5/2008 11/7/2008
  • 39. Definition: Dashboard More than a picture 90 00 1 80 5 44 87 2 70 52 3 60 4 50 49 5 40 6 124 6 7 30 6 70 8 20 9 99 10 10 0 1 2 3 4 5 6 7 11/7/2008
  • 40. Dashboard Tied to real measurements Cataloging Reference Speed Delivery in real time Satisfaction 10 day Intranet/Internet 96% turnaround Visual picture of whats happening now New Reference Acquisitions Accuracy 100 95% 11/7/2008
  • 41. 晦庄恢河雨粥晦+ www.libqual.org a suite of services that libraries use to solicit, track, understand, and act upon users opinions of service quality Cost: $3,200.00 per institution (Website viewed 11/5/2008) 11/7/2008
  • 42. 晦庄恢河雨粥晦+ Survey instrument adapted from SERVQUAL (Parasuraman, Zeithaml, Berry) Library asks users to take the survey. Survey answers are sent to central database, analyzed and return to library in reports describing users experience of library service. Measures three dimensions of service quality: Information Control; Affect of Service; Library as Place. USER FOCUSED! 11/7/2008
  • 43. University Performance Everything the library does should tie in with the University mission; the library is part of the performance of the entire University. 11/7/2008
  • 44. OUTPUTS OUTCOMES For academic libraries: educational outcomes In universities and other school settings, librarians are being asked not just whether they provided learning resources but also if their service contributed to the success of the student (Dudden, 9) 11/7/2008
  • 45. Those of us working in technical services and cataloging in particular need to find ways to assess the impact of our work on the population served, and to communicate this information (Show how we count!) Catalog use studies Catalog accuracy studies Something new? Make sure to assess service to both Brick and Click patrons! 11/7/2008
  • 46. Questions to Ask About Metrics Do we have a culture of assessment that will enable ongoing measurement? How can we foster a culture of assessment? Are our plans based on our mission? What funding is available or can be made available for implementation of a long-term assessment project? 11/7/2008
  • 47. More questions Are we measuring something that is actionable? Is it something that can be changed? Are we measuring something that is relevant (fits the mission)? Is our process iterative? Are we using the results in re- planning? Are we seeing continuous improvement? Is the process fostering the culture of assessment? 11/7/2008
  • 48. Fair Winds! mmccune@pittstate.edu 11/7/2008
  • 49. Works Cited Charbonneau, Mechael D. Production Benchmarks for Catalogers in Academic Libraries: Are We There Yet? Library Resources & Technical Services 49.1 (2005): 40-48. Dudden, Rosalind Farnam. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-To- Do-It-Manual With CD-ROM. New York: Neal-Schuman, 2007. Fischer, Ruth, Rick Lugg, and Kent C. Boese. Cataloging: How to Take a Business Approach. First published in The Bottom Line (2004). http://www.ebookmap.net/pages/Publications.php Graham, Peter S. Quality in Cataloging: Making Distinctions. Journal of Academic Librarianship 16.4 (1990): 213-218. 11/7/2008
  • 50. Works Cited (cont.) Lindauer, Bonnie Gratch. Defining and Measuring the Librarys Impact on Campuswide Outcomes. College & Research Libraries 59.6 (1998): 546-570. Matthews, Joseph R. Evaluation and Measurement of Library Services. Westport, Conn.: Libraries Unlimited, 2007. Self, James. From Values to Metrics: Implementation of the Balanced Scorecard at a University Library. Performance Measurement and Metrics 4.2 (2003): 57-63. 11/7/2008