Ibis + Substrait.pdf
0 likes46 views
This document discusses Ibis, an analytics library that generates SQL expressions and strings for execution on SQL backends. It introduces Substrait, a standardized relational algebra format that can serialize Ibis expressions. Substrait decouples the expression building in Ibis from the execution engines, allowing Ibis to potentially support hundreds of different backends. The document demonstrates compiling an Ibis expression to the Substrait format and shows how Ibis can consume any SQL dialect by compiling it to Ibis expressions via a Substrait producer.
Ibis + Substrait.pdf
- 1. Presented by: Ibis and Substrait: Standardized Analytics 06/23/2022 Hussain Sultan
- 2. 2 Ibis produces Expressions or Strings for execution on SQL backends t.groupby('region') .aggregate(topk=t.funding_rounds.count()) .sort_by(ibis.desc('topk')) .head() Produces SELECT region ,count(funding_rounds) AS topk FROM companies GROUP BY region ORDER BY topk SQL Strings generated for Execution Engine SQL Backends <sqlalchemy.sql.selectable.S elect object> SQLAlchemy Expression
- 3. 3 String Execution Backend Ibis API Operations Compile SQL String Backend Speci鍖c SQL Flavor and Support Type Check and Optimize Rewrite Build Ibis Expression
- 4. 4 Expression Generating Backends API Operation Compile SA Expression Backend Speci鍖c Expressions or SQLAlchemy Type Check and Optimize Rewrite Build Ibis Expression
- 5. 5 Ibis Backends - LOC SQLAlchemy-based Backends tend to be lower LOC
- 6. 6 What if Ibis supported 100s of backends? DB Engines lists 398 Databases in their popularity ranking https://db-engines.com/en/ranking
- 7. 7 Substrait: Serialized Relational Algebra substrait views ibis python r scala babel sql presto sql dplyr Producers Consumers ibis
- 8. 8 Substrait: Serialized Relational Algebra Plan Relations Types i8 i32 fp32 struct Expressions Functions add rank join agg filter read case literal Serialization protobuf text Types Expressions Functions Relations Plans Serialization formats
- 10. 10 Ibis-substrait producer is actively being developed SQL String Python Expression Under Development Ibis API Compile Substrait Producer
- 11. 11 Demo
- 12. 12 Takeaways Ibis as a substrait producer, decouples the execution engines and expression building Check out Gil Forsyths gist that explains the duckdb and ibis substrait integration: https://gist.github.com/gforsyth/496d680e1e29f0876df937e e5091e1b8
- 13. 13 Thank you!
- 14. 14 2. What if we want to consume any SQL DSL into Ibis expressions? API Ops DSL Execute
- 15. 15 Ibis support multiple execution paradigms SQL String Python Expression Direct Execution Focus of Todays Talk 1 Ibis API 2 Compile