Forward in Reverse
Download as PPTX, PDF0 likes1,566 views
This document provides an overview of the architecture of the Forward system, which includes batch processing of library data, indexing the data using Solr, and the web application frontend. It discusses how raw data from the ILS and digital collections is extracted, merged, parsed and indexed to build the Solr search index. It also outlines the technologies used in the Ruby on Rails-based web application such as campus affiliation detection, the results display, and features like the BookReader for viewing full text books.
More Related Content
Viewers also liked (13)
Similar to Forward in Reverse (20)
Recently uploaded (20)
![[Webinar] Scaling Made Simple: Getting Started with No-Code Web Apps](https://cdn.slidesharecdn.com/ss_thumbnails/webinarscalingmadesimplegettingstartedwithno-codewebapps-mar52025-250305183437-f03c78a3-thumbnail.jpg?width=560&fit=bounds)
Forward in Reverse
- 1. Forward in ReverseA Gentle Overview Of Forward System ArchitectureEric, Mike & Steve WiLSWorld 2010
- 2. OutlineIntro to Forward with DemoBatch Processing (Backend)Web Application (Frontend)ChallengesQ&As throughout
- 3. Intro & Demo
- 6. We have gobs & gobs of data.
- 8. 1a) ILS Data
- 14. Antique Style KeyBy Stars*Go*Bluehttp://www.flickr.com/photos/artbydebora/1406682449/
- 15. Common Identifier = OCLC Number
- 16. Catalog Extract Processing Details14 Voyager Instances
- 17. 13M MARC bibliographic records extracted
- 19. Local C codeSorted, deduplicated and merged output: 8M records
- 20. 10GB Raw MARC dataWhy Merge?URLs
- 21. Formats
- 22. Holdings1b) Digital Collection Data
- 24. Fedora Extract Processing Details1 Fedora Repository
- 25. 13K First Class XML Objects extracted
- 27. Repository query languageXML output: METS XML package
- 28. Structural XML
- 30. 41MB XML data2) Index it
- 31. We take raw library data and process it with MARC/XML parsing tools and local parsing rulesin order to build a Solr search index.
- 32. Raw data (MARC & METS XML) Parsing libraries (Java code: marc4j, SAXParser)Local code that defines parsing rulesSolr index
- 33. 1. Raw data
- 34. LEADER 02000cam a22003734a 45 0001 6939454005 20051208125417.0008 051104s2004 enka $b 001 0 eng 010 $a 2003045349 035 $a (OCoLC)ocm52165958 040 $aDLC $c DLC $d XMA $d BAKER $d UKM 015 $a GBA430162 $2bnb016 7$a 012906573 $2Uk020 $a 0754605175 (alk. paper) 024 $a 99811375970 042 $apcc049 $a GZMA 050 00$a B3376.W564 $b W55355 2004 082 00 $a 111/.85/092 $2 21 245 00 $a Wittgenstein, aesthetics, and philosophy / $c edited by Peter B. Lewis. 260 $aAldershot, Hants, England ; $a Burlington, VT : $bAshgate, $c c2004. 300 $a xii, 255 p. : $b ill. ; $c 24 cm. 440 0 $aAshgateWittgensteinian studies 505 0$a Wittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. Hagberg504 $a Includes bibliographical references (p. 235-247) and index.
- 35. 02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enkab 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. HagbergaIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM
- 37. 2. MARC/XML parsing libraries
- 40. 02000cam a22003734a 45 001000800000005001700008008004100025010001700066035002300083040003000106015001900136016001800155020002800173024001600201042000800217049000900225050002800234082002000262245007400282260006800356300003400424440003600458505081100494504006401305600005001369700002501419938007101444945001901515946003001534946001301564947002101577948001601598994001201614693945420051208125417.0051104s2004 enkab 001 0 eng a 2003045349 a(OCoLC)ocm52165958 aDLCcDLCdXMAdBAKERdUKM aGBA4301622bnb7 a0129065732Uk a0754605175 (alk. paper) a99811375970 apcc aGZMA00aB3376.W564bW55355 200400a111/.85/09222100aWittgenstein, aesthetics, and philosophy /cedited by Peter B. Lewis. aAldershot, Hants, England ;aBurlington, VT :bAshgate,cc2004. axii, 255 p. :bill. ;c24 cm. 0aAshgate Wittgensteinian studies0 aWittgenstein and the aesthetic domain / Kjell S. Johannessen -- 2. Wittgenstein, anti-essentialism and the definition of art / Terry Diffey -- 3. Rules, creativity and pictures : Wittgenstein's Lectures on aesthetics / David Novitz -- 4. Criticism without theory / Mark W. Rove -- 5. On aesthetic reactions and changing one's mind / Lars Hertzberg -- 6. Wittgenstein and the arts : understanding and performing / Graham McFee -- 7. Wittgenstein's music / R.A. Sharpe -- 8. Wittgenstein on music and language / Oswald Hanfling -- 9. Ethics and aesthetics are one / Carolyn Wilde -- 10. Fiction and reality in the arts / IlhamDilman -- 11. Literature, human understanding and morality / Ben Tilghman -- 12. 'The self, thinking' : Wittgenstein, Augustine and the autobiographical situation / Garry L. HagbergaIncludes bibliographical references (p. 235-247) and index.10aWittgenstein, Ludwig,d1889-1951xAesthetics.1 aLewis, Peter,d1947- aBaker & TaylorbBKTYc99.95d99.95i0754605175n0004227086sactive c1d89087961587 a714694b2005-11-23c81.86 c99.95d1 aHEUR 4801bm,stk aSCNd348032 a92bGZM
- 44. 3. Local code
- 48. What is Solr?An XML API over a Lucene search index.
- 51. Access to Raw FormatsRaw MARC stored for Merged recordLive calls made to Fedoraweb services
- 52. Data RefreshBibliographic: weeklyCirculation status: nightly
- 54. For more information, seehttp://sdg.library.wisc.edu/blog/2010/03/03/solr-marc-indexing-based-on-diffs/
- 55. Web Application
- 56. Frontend?(X)HTMLJavaScriptCascading Style SheetsDesignInformation ArchitectureUser experienceChrome (images, icons, pretty)
- 57. Forward ColophonActiveRecordBaseWithoutTable (Rails plugin)ApacheBlacklight (Rails plugin)Blueprint CSSBookreader (jQuery)CapistranoCrontabEngines (Rails plugin)FedoraFreebase APIGeoIP (Ruby gem)Google Books APIHaml (Rails plugin)Happymapper (Ruby gem)HathiTrust APIjQueryKen (Ruby gem)LowPro (Prototype JS)MARC4JPassenger (modrails)Prototype JSPostgreSQLRaphaelRuby on RailsShibbolethSubversionSolr / LuceneSummon (Ruby gem)UW-Madison Libraries Staff Directory APIUWDC (Rails plugin)Voyager APITender love and attention
- 58. Campus AffiliationUsers localize to a school, allows us scope many features to their campus.GeoIPRubyGemMatch IP addresses with physical locations.Rapha谷lJavaScript Library Small JavaScript library that should simplify your work with vector graphics on the web.
- 59. Rapha谷lSVG elements, like the circles and squares in the Forward splash page, can be treated as XHTML elements allowing us to manipulate them with JavaScript and CSS.http://raphaeljs.com/
- 60. Campus HomepageForward application stack:Apache+Passenger (modrails)
- 61. Ruby on Rails
- 62. PostgreSQL
- 63. Apache SolrApache+PassengerPhusion Passenger is an Apache module, which makes deploying Ruby and Ruby on Rails applications on Apache a breeze.http://www.modrails.com/
- 64. Ruby on RailsRuby on Rails is an open-source web framework thats optimized for programmer happiness and sustainable productivity.http://rubyonrails.org/
- 65. PostgreSQLPostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness.http://www.postgresql.org/
- 66. Apache SolrSolr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling.http://lucene.apache.org/solr/
- 67. Results
- 68. Results Three columns
- 69. Results Data sources
- 70. Results Facets Solr
- 71. Results Solr + PostgreSQL + APIs
- 72. Results Context APIs
- 73. Results Three main columns
- 74. Results CSS grid
- 75. BlueprintBlueprint is a CSS framework, which aims to cut down on your development time. It gives you a solid foundation to build your project on top of, with an easy-to-use grid, sensible typography, useful plugins, and even a stylesheet for printing.http://blueprintcss.org/
- 77. Show Book
- 78. Show Image
- 79. Show Full Text Book
- 80. Show View Full Text Book
- 81. BookReaderThe Internet Archive BookReader is used to view books from the Internet Archive online and can also be used to view other books. http://github.com/openlibrary/bookreader
- 82. Challenges
- 83. ChallengesMerging MARC, METS extractsBatch processing time (Time/CPU constraints)Page level indexing (Bookviewer - memory/disk constraints)Voyager APIOrganization challengesbig project, small shopdealing with vendor silosmultiple cataloging standardsquality of services challenges
- 84. Thanks!