2015 - Extract SF - Data Quality
0 likes261 views
This document discusses the importance of data quality and provides tips for ensuring high quality data. It notes that while data can be very useful, it is only valuable if it is clean and structured. When extracting large amounts of data, it recommends developing extractors, combining extractors, and automating the extraction process. For scaling operations, having processes to clean, validate, and maintain data quality is crucial. The document offers suggestions for writing effective XPaths and regex expressions to extract the right data. It also stresses the importance of measuring data quality through completeness, coverage, and detecting anomalies both during and after the extraction process.
![//div[@id="priceBlock"]/table/tbody/tr/td
[b/@class="priceLarge"]/b
better than
//*[@id="priceBlock"]/table/tbody/tr[2]/td[2]/b[1]](https://image.slidesharecdn.com/d2150aa4-f4b3-41e3-8867-f1f3f648d630-151101181511-lva1-app6891/85/2015-Extract-SF-Data-Quality-34-320.jpg)
2015 - Extract SF - Data Quality
- 1. It's Time to Start Caring About Data Quality Data Quality at Scale
- 3. Everyone is talking about how useful data is
- 4. data can save your business
- 5. data can save your life
- 7. but...
- 8. all that is only true if you have the right data
- 9. data tend to be dirty and unstructured
- 13. Ive created an extractor
- 15. Ive pass a bunch of queries (bulk)
- 18. and got a dataset
- 20. How can you QA this data?
- 21. eyeballing
- 24. eyeballing we can find anomalies without having domain expertise
- 25. Quick summary: - create extractors - combine extractors - schedule data extraction
- 26. What if we need to scale up?
- 27. if you have: - more than ~3 datasources - more than ~2 extractors per ds - big volume of queries - pre or post processing
- 28. you will need: - people to create and maintain extractors - process to clean and validate data
- 29. Data Quality think about it pre and post data extraction!
- 30. tips and tricks to increase data quality
- 31. XPaths
- 35. Regex
- 39. Required column
- 46. completeness
- 47. coverage
- 49. post extraction data quality improvements?
- 51. how we do it
- 54. Smart automation
- 57. normalization
- 58. confidence score
- 59. Human input
- 60. Transparency
- 61. summary
- 62. Data Quality is essential
- 63. think about it from the very beginning
- 64. develop a process to measure data quality before scaling up
- 65. if you dont want to reinvent the wheel - contact us!