Model-Driven Analytics for Open Data APIs
Download as PPTX, PDF0 likes776 views
Work presented at the International Workshop on Engineering Open Data (WEOD), held in conjunction with 18th International Conference on Web Engineering (ICWE 2018) in C叩ceres, Spain on 5th June 2018.
![Lightweight Static Verification of [UML] Executable Models (An overview)](https://cdn.slidesharecdn.com/ss_thumbnails/eplanashsetslideshare-121005101705-phpapp01-thumbnail.jpg?width=560&fit=bounds)
More Related Content
Similar to Model-Driven Analytics for Open Data APIs (20)
More from Elena Planas (7)
Recently uploaded (20)
![Winner Analytix - Democratizing Data [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/winneranalytix-democratizingdataautosaved-250315074345-e10ea083-thumbnail.jpg?width=560&fit=bounds)
Model-Driven Analytics for Open Data APIs
- 1. Elena Planas eplanash@uoc.edu Open University of Catalonia Model-Driven Analytics for Open Data APIs David Baneres dbaneres@uoc.edu Open University of Catalonia International Workshop on Engineering Open Data (WEOD) Held in conjunction with 18th International Conference on Web Engineering (ICWE 2018) C叩ceres, Spain - 5th June 2018
- 3. goal of the open data movement: empower end-users to exploit and benefit from open-data
- 4. Citizens ask: How can we access and manipulate open data?
- 5. Citizens ask: How can we access and manipulate open data? Data providers ask: How our data is used by end-users?
- 6. Data providers ask: How our data is used by end-users? Citizens ask: How can we access and manipulate open data?
- 7. we provide a Model-Driven Analytical tool for Open Data APIs
- 8. we provide a Model-Driven Analytical tool for Open Data APIs our goal is to visualize how end-users interact with open data sources regarding several metrics
- 9. we provide a Model-Driven Analytical tool for Open Data APIs
- 10. we provide a Model-Driven Analytical tool for Open Data APIs Log gathering Log Transformation Computing & Visualizing metrics Step 1 Step 2 Step 3
- 11. Step 1: Log gathering
- 12. Record all the REQUESTS made by users to the different API end-points in a LOG FILE
- 13. Record all the REQUESTS made by users to the different API end-points in a LOG FILE
- 14. general REST request specific REST sub-request specific REST sub-request specific REST sub-request * Hamza Ed-Douibi, Javier Luis Canovas Izquierdo and Jordi Cabot. OpenAPItoUML: A tool to generate UML models from OpenAPI definitions ICWE 2018 Demo: 6th Wednesday. 09.00 - 10.30 API1 API2 API3
- 16. Record all the REQUESTS made by users to the different API end-points in a LOG FILE
- 17. LOG structure Opening: general request specific sub request 1 specific sub request n Closing: general request
- 18. Opening: general request specific sub request 1 specific sub request n Closing: general request #1 2017-11-10 11:02:20 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries #2 2017-11-10 11:02:23 - 200 http://restcountries.eu/rest/v2/all 2208 #3 2017-11-10 11:02:24 - 200 http://battuta.medunes.net/api/country/all?key=... 1076 #4 2017-11-10 11:02:24 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries yes Example LOG structure
- 19. Opening: general request specific sub request 1 specific sub request n Closing: general request #1 2017-11-10 11:02:20 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries #2 2017-11-10 11:02:23 - 200 http://restcountries.eu/rest/v2/all 2208 #3 2017-11-10 11:02:24 - 200 http://battuta.medunes.net/api/country/all?key=... 1076 #4 2017-11-10 11:02:24 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries yes Example LOG structure
- 20. Opening: general request specific sub request 1 specific sub request n Closing: general request #1 2017-11-10 11:02:20 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries #2 2017-11-10 11:02:23 - 200 http://restcountries.eu/rest/v2/all 2208 #3 2017-11-10 11:02:24 - 200 http://battuta.medunes.net/api/country/all?key=... 1076 #4 2017-11-10 11:02:24 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries yes Example LOG structure
- 21. #1 2017-11-10 11:02:20 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries #2 2017-11-10 11:02:23 - 200 http://restcountries.eu/rest/v2/all 2208 #3 2017-11-10 11:02:24 - 200 http://battuta.medunes.net/api/country/all?key=... 1076 #4 2017-11-10 11:02:24 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries yes LOG structure LOG information Example Time Requested APIs Server response Opening: general request specific sub request 1 specific sub request n Closing: general request
- 22. #1 2017-11-10 11:02:20 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries #2 2017-11-10 11:02:23 - 200 http://restcountries.eu/rest/v2/all 2208 #3 2017-11-10 11:02:24 - 200 http://battuta.medunes.net/api/country/all?key=... 1076 #4 2017-11-10 11:02:24 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries yes LOG structure LOG information Example Time Requested APIs Server response Opening: general request specific sub request 1 specific sub request n Closing: general request
- 23. #1 2017-11-10 11:02:20 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries #2 2017-11-10 11:02:23 - 200 http://restcountries.eu/rest/v2/all 2208 #3 2017-11-10 11:02:24 - 200 http://battuta.medunes.net/api/country/all?key=... 1076 #4 2017-11-10 11:02:24 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries yes LOG structure LOG information Example Time Requested APIs Server response Opening: general request specific sub request 1 specific sub request n Closing: general request
- 24. #1 2017-11-10 11:02:20 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries #2 2017-11-10 11:02:23 - 200 http://restcountries.eu/rest/v2/all 2208 #3 2017-11-10 11:02:24 - 200 http://battuta.medunes.net/api/country/all?key=... 1076 #4 2017-11-10 11:02:24 - http://localhost:8080/OpenDataForAll/ODataService.svc/Countries yes LOG structure LOG information Opening: general request specific sub request 1 specific sub request n Closing: general request Example Time Requested APIs Server response
- 25. Step 2: Log Transformation
- 26. TRANSFORM the initial LOG FILE to enrich its information
- 28. TRANSFORMED LOGINITIAL LOG Input timestamp general / for sub-query Output timestamp general / for sub-query Response time for each request and sub-request Total time for resolving a request Time
- 29. TRANSFORMED LOGINITIAL LOG Input timestamp general / for sub-query Output timestamp general / for sub-query Response time for each request and sub-request Total time for resolving a request Time General request Specific sub-requests Number of sub-requests for each general request Number of requested APIs in each general request Requested APIs
- 30. TRANSFORMED LOGINITIAL LOG Input timestamp general / for sub-query Output timestamp general / for sub-query Response time for each request and sub-request Total time for resolving a request Time General request Specific sub-requests Number of sub-requests for each general request Number of requested APIs in each general request Requested APIs Reliability of each request and sub-request Server response Reliability of each request and sub-request
- 31. Step 3: Computing & Visualizing metrics
- 35. the aim of the is to measure and report performance and volumes of manipulated APIs PERFORMANCE METRICS
- 36. API RELIABILITY Average RESPONSE TIME - by API - by request / sub-requests Average number of ACCESSED APIs for each request Average number of GENERATED SUB-REQUESTS for each request QUERY HISTORY PERFORMANCE METRICS
- 37. Response time by API: Response time by request / sub-request: % % % % % % API1 API2 API3 Request1 Sub-request1.1 Sub-request1.2 % API RELIABILITY Average RESPONSE TIME - by API - by request / sub-requests Average number of ACCESSED APIs for each request Average number of GENERATED SUB-REQUESTS for each request QUERY HISTORY PERFORMANCE METRICS
- 38. Accessed APIs for each request: % % Request1 Request2 Request3 % API RELIABILITY Average RESPONSE TIME - by API - by request / sub-requests Average number of ACCESSED APIs for each request Average number of GENERATED SUB-REQUESTS for each request QUERY HISTORY PERFORMANCE METRICS
- 39. Generated sub-requests for each request: % % Request1 Request2 Request3 % API RELIABILITY Average RESPONSE TIME - by API - by request / sub-requests Average number of ACCESSED APIs for each request Average number of GENERATED SUB-REQUESTS for each request QUERY HISTORY PERFORMANCE METRICS
- 40. Collecting response codes of each sub-request: Successfully served Error: Bad request Error: not found API RELIABILITY Average RESPONSE TIME - by API - by request / sub-requests Average number of ACCESSED APIs for each request Average number of GENERATED SUB-REQUESTS for each request QUERY HISTORY PERFORMANCE METRICS
- 41. Performance metrics can be filtered by several criteria: API RELIABILITY Average RESPONSE TIME - by API - by request / sub-requests Average number of ACCESSED APIs for each request Average number of GENERATED SUB-REQUESTS for each request QUERY HISTORY PERFORMANCE METRICS
- 42. the aim of the is to analyze the consumed data in the context of the UML model representing the requested APIs SEMANTIC METRICS
- 43. HEAT UML MODEL ENTITY/FIELD CONSUMPTION QUERY DIAGRAM SEMANTIC METRICS
- 44. Entities consumption is highlighted using different colours: High demand Low demand HEAT UML MODEL ENTITY/FIELD CONSUMPTION QUERY DIAGRAM SEMANTIC METRICS
- 45. Show the number of requests to an specific entity / field: HEAT UML MODEL ENTITY/FIELD CONSUMPTION QUERY DIAGRAM SEMANTIC METRICS
- 46. Show the navigability to resolve the performed query: HEAT UML MODEL ENTITY/FIELD CONSUMPTION QUERY DIAGRAM SEMANTIC METRICS
- 48. Search Processing Language (SPL) based on SQL index = TransformedLog | stats avg(timeForRequest) Example
- 50. dashboard (I) Average response time by API
- 51. dashboard (II) Average response time by request
- 54. dashboard (V) Heat Map & Consumption
- 55. Conclusions
- 56. we provide a Model-Driven Analytical tool for Open Data APIs the monitoring and visualization of the open data consumption report highly valuable information to data providers Improve data - Data precision - Avoiding overlapping - Removing non-accessed-data Infer new knowledge - New content to be published - Potential partnerships
- 57. Elena Planas eplanash@uoc.edu Open University of Catalonia Model-Driven Analytics for Open Data APIs David Baneres dbaneres@uoc.edu Open University of Catalonia Questions? * All the images of this presentation have been acquired from http://pixabay.com