Tracking: Ibis Support

[KeXiangWang]

ibis: the portable Python dataframe library.
ibis has supported postgresql.

We separate the Ibis suppport development into three stages:

• Support more functions to align RW with PG.
  • Tracking: System Information Functions #14003
• Enhance sqlalchemy-risingwave to satisfy all the requirement of ibis-postgres.
• Support a sqlglot-risingwave sqlglot.dialects.dialect.Dialect
• Contribute ibis to get official support.
  • feat(risingwave): init impl for Risingwave ibis-project/ibis#7954

Bugs found while supporting ibis.

• bug(frontend): CorrelatedInputRef has not been decorrelated in ArraySubquery #14423
• bug(expr): to_char consider unsupported format as plain text #14670
• bug(frontend): unnest cause panic #14734
• bug(frontend): parsing field name containing " #14731
• bug(optimizer): unexpected array unnesting #14735

Feature requests:

• func: levenshtein(character varying, character varying)
• func: textregexeq(character varying, character varying)
• func: make_date feat(expr): add support for make_date/time/timestamp #14827
• func: make_time feat(expr): add support for make_date/time/timestamp #14827
• func: make_timestamp feat(expr): add support for make_date/time/timestamp #14827
• func: make_interval
• func: random()
• func: log10(numeric, numeric)
• func: hstore(character varying[], character varying[])
• func: last(double precision)
• support limit null
• support data type like NUMERIC(38,9)
• support risingwave sqlglot.dialects.dialect.Dialect
• support information_schema.schemata
• fix get_columns in sqlalchemy-risingwave

[KeXiangWang]

Marked as Done temporarily. Will continue to support more features if requested by users.

[kszlim]

Curious if you have any fully featured examples of ibis-risingwave anywhere?

I think a really nice fully featured example could be tpch against an iceberg source with ibis-risingwave?

[KeXiangWang]

Curious if you have any fully featured examples of ibis-risingwave anywhere?

Currently no.

I think a really nice fully featured example could be tpch against an iceberg source with ibis-risingwave?

Thanks for the suggestion! We will consider that. Is there a specific reason you want to use ibis-risingwave? Does SQL API not meet your requirements? We also have psycopg2 support for python users. The doc is here.

[kszlim]

In general I find dataframe apis much easier for data engineering/feature generation. Also having access to Python UDFs for numpy/scipy/etc. is pretty useful/important.

[KeXiangWang]

You actually can use Python UDFs with the SQL API. Unfortunately the UDF functions haven't been supported in ibis-risingwave. cc @alicelyy PTAL