SHARE Notification Service Development
0 likes277 views
The document provides an overview of the Center for Open Science's Notification Service and how it gathers, normalizes, and notifies consumers of research output events from over 20 content providers. Key aspects of the Notification Service infrastructure are highlighted, including using RabbitMQ for reliable messaging, Elasticsearch for search and analytics, MongoDB/TokuMX for document storage, and Celery and Flask for asynchronous job processing and service APIs.
![GET https://frogworld.
com/api/frogs
Request Response
[
{
name: Kermit,
color: green,
type: felt
},
{
name: Travis,
color: green,
type: tree
}
]](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-10-320.jpg)
![POST https://osf.io/api/share
Request Response
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
Success](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-11-320.jpg)
![Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
origin: frogworld,
id: 10.100/frogworld.102,
description:
Exploring greenness.
}](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-15-320.jpg)
![Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
name: Easy Being Green?,
contributors: [
Frog Scientists Intl,
Amphibians United,
],
origin: frogworld,
doi: 10.100/frogworld.102
}](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-16-320.jpg)
![Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
title: Easy Being Green?,
source: frogworld,
id: 10.100/frogworld.102
}](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-17-320.jpg)
![Resource 1 Resource 2
{
title: Easy Being Green?,
contributors: [
Kermit,
Travis
],
source: frogworld,
id: 10.100/frogworld.102
}
{
title: Easy Being Green?,
contributors: [
No Contributors
],
source: frogworld,
id: 10.100/frogworld.102
}](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-18-320.jpg)
![{
"source": "stcloud",
"description": "A lengthy field study in Japan using interviews and other sources and focusing
on the nation's five leading national newspapers and Tokyo's major television stations finds
evidence of much overlap between industry and the news media, through interlocking
directorships and social club memberships,",
"contributors": {
"given": "Roya",
"family": "Akhavan-Majid",
"middle": "",
"email": ""
},
"title": "The Press as an Elite Power Group in Japan",
"dateCreated": "1970-01-01T00:00:00",
"dateUpdated": "2014-10-01T17:54:11+00:00",
"tags": [ "mass communication"],
"id": {
"url": "http://repository.stcloudstate.edu/comm_facpubs/3",
"serviceID": "oai:repository.stcloudstate.edu:comm_facpubs-1001",
"doi": ""
}
}](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-31-320.jpg)
![[ Intentionally Left Blank ]](https://image.slidesharecdn.com/txzq24bfqby3qyntot3t-signature-e9f6f3a1d8a3824d74159880334e257d9d9bb296070dff2728435f521ae29aa4-poli-150330134155-conversion-gate01/85/SHARE-Notification-Service-Development-51-320.jpg)
![Web Today, Good Tomorrow? Transactional archiving of web content [Long Version]](https://cdn.slidesharecdn.com/ss_thumbnails/burnhllpspfeb2017-v10-170207213047-thumbnail.jpg?width=560&fit=bounds)
More Related Content
Similar to SHARE Notification Service Development (20)
More from SHARE (8)
Recently uploaded (20)
SHARE Notification Service Development
- 1. SHARE NotificationServiceDevelopment Update from the Center for Open Science
- 2. Center for Open Science
- 6. COS-SHARE Development Team Brian Geiger Chris Seto Coral Sheldon-Hess Erin Braswell Fabian Von Feilitzsch
- 7. COS-SHARE Developer Alumni Casey Rollins Erica Baranski Faye Huynh Lauren Revere Ginny Huang Michelle Yao Peter Fan Saman Ehsan Xander Herrick
- 9. How are we getting data? Application Programming Interface
- 10. GET https://frogworld. com/api/frogs Request Response [ { name: Kermit, color: green, type: felt }, { name: Travis, color: green, type: tree } ]
- 11. POST https://osf.io/api/share Request Response { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } Success
- 15. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { title: Easy Being Green?, contributors: [ Kermit, Travis ], origin: frogworld, id: 10.100/frogworld.102, description: Exploring greenness. }
- 16. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { name: Easy Being Green?, contributors: [ Frog Scientists Intl, Amphibians United, ], origin: frogworld, doi: 10.100/frogworld.102 }
- 17. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { title: Easy Being Green?, source: frogworld, id: 10.100/frogworld.102 }
- 18. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { title: Easy Being Green?, contributors: [ No Contributors ], source: frogworld, id: 10.100/frogworld.102 }
- 21. Request Response GET http://repository.stcloudstate.edu/do/oai/? verb=ListRecords &metadataPrefix=oai_dc &from=2014-10-02T00:00:00Z <record> <date> </date> <title> </title> <contributors> <name>...</name> <name>...</name> </contributors> <description> </description> <id>...</id> </record>
- 30. <record> <header> <identifier>oai:repository.stcloudstate.edu:comm_facpubs-1001</identifier> <datestamp>2014-10-01T17:54:11Z</datestamp> </header> <metadata> <oai_dc:dc> <dc:title>The Press as an Elite Power Group in Japan</dc:title> <dc:creator>Akhavan-Majid, Roya</dc:creator> <dc:description> A lengthy field study in Japan using interviews and other sources and focusing on the nation's five leading national newspapers and Tokyo's major television stations finds evidence of much overlap between industry and the news media, through interlocking directorships and social club memberships </dc:description> <dc:date>1990-12-01T08:00:00Z</dc:date> <dc:identifier>http://repository.stcloudstate.edu/comm_facpubs/3</dc:identifier> <dc:source>Mass Communication Faculty Publications</dc:source> <dc:publisher>the Repository at St. Cloud State</dc:publisher> <dc:subject>Mass Communication</dc:subject> </oai_dc:dc> </metadata> </record>
- 31. { "source": "stcloud", "description": "A lengthy field study in Japan using interviews and other sources and focusing on the nation's five leading national newspapers and Tokyo's major television stations finds evidence of much overlap between industry and the news media, through interlocking directorships and social club memberships,", "contributors": { "given": "Roya", "family": "Akhavan-Majid", "middle": "", "email": "" }, "title": "The Press as an Elite Power Group in Japan", "dateCreated": "1970-01-01T00:00:00", "dateUpdated": "2014-10-01T17:54:11+00:00", "tags": [ "mass communication"], "id": { "url": "http://repository.stcloudstate.edu/comm_facpubs/3", "serviceID": "oai:repository.stcloudstate.edu:comm_facpubs-1001", "doi": "" } }
- 32. GET stcloud.edu/api/ Archive Response Archive Data Normalize Send to OSF Archive Request
- 34. Notify Consumers
- 35. OSF Metadata OSF Metadata and Search APIs
- 37. RSS
- 38. /rss/?q=open AND science AND source:scitech
- 39. PubSubHubbub
- 40. ResourceSync
- 41. /resync/resourcelist/?q=open AND science AND source:scitech
- 45. Inside the Notification Service 3:30 in this room
- 46. Recap of lessons learned
- 47. Today. Over 20 content providers Robust backend providing: archive of all research release events monitoring and extensive logging recovery from service interruptions Solid foundation to enable building of other services like curation and networking of research projects
- 48. More institutional repositories Tests with complementary types of content: Providers able to push content to SHARE NS Coming soon...
- 49. Today Early 2015 Fall 2015 Beta release Full release 20 + service providers Robust backend Output formats RSS ResourceSync Search More service providers Complementary data Refined metadata schema and push of content
- 50. Try it: http://share-dev.osf.io Follow on GitHub: https://github.com/ CenterForOpenScience/SHARE See more details: https://osf.io/29ypx/wiki/
- 51. [ Intentionally Left Blank ]
- 53. Elasticsearch RabbitMQ MongoDB/ TokuMX Celery Workers OSF App Servers ScrAPI Load Balancers Caching Proxies Raw UserProvider Provider Provider Fluentd Elasticsearch Kibana UserUser
- 54. Reliable Messages persist Redundant message storage Scalable easily clustered Supported Most used message broker RabbitMQ
- 55. Celery Asynchronous job execution Scalable Allows for any number of worker machines Fault resistant Large community Tight integration with RabbitMQ
- 56. MongoDB/TokuMX Fast Transaction-based Improve document storage Document-level locking
- 57. Elasticsearch Flexible Schema-less Built to be horizontally extensible Robust API
- 58. Fluentd Most robust log aggregation system Converts log messages to JSON for easy processing Emphasizes simplicity
- 59. Sentry No exceptions uncaught Real-time logging Easy triage
- 60. Kibana Nice visualizations Easy setup Simple to use Stats at a glance In depth progress monitoring
- 62. Harvesters Gathering metadata from providers
- 64. RSS Feed
- 68. SHAREintheworld
- 70. Ways to access SHARE
- 71. Search API
- 80. HTML