Use ibis DuckDB backend to access pandas dataframe instead of ibis pandas backend

[sundar-mudupalli-work]

Is your feature request related to a problem? Please describe.
A feature request - DVT currently uses a pandas backend that has been removed in Ibis 10.0 (we are currently on Ibis 5.1). You can use DuckDB to access data stored in Pandas dataframes. There are some performance benefits and some feature changes. Ibis explains why in this article. In version 9.0, ibis introduced the concept of a default backend and they chose DuckDB.
What version of DVT are you using?
8.3

[sundar-mudupalli-work]

Note the following was generated by Antigravity with some guidance from me.

Verification Report: Ibis DuckDB Backend Fix

I have investigated the fix replacing ibis.pandas.connect() with ibis.duckdb.connect().register() inside your current Python 3.10 workspace with Ibis 5.1.0 pinned.

🔴 Findings & Roadblocks

The fix fails to work on the current dependency stack due to several limitations in older Ibis releases:

1. Missing Secondary Dependencies

By default, triggering .table() lookups or view creations on the DuckDB backend with Ibis 5.1.0 triggers SQLAlchemy integration that fails without:

• duckdb-engine (For SQLAlchemy dialect mapping)
• sqlalchemy-views (For view triggers)

2. .register() API Limitation

Even with dependencies installed in a fresh Python 3.10 environment, continuous verification on standalone scripts yielded:
ValueError: Cannot infer appropriate read function for input

In Ibis 5.1.0, the DuckDB .register() method does not seamlessly ingest pandas.DataFrame variables without format inference failures.

3. Catalog Lookup Mismatch

Alternative methods like read_in_memory(df) do return table expressions, but do not register the string reference in the backend's table lookup catalog. This breaks any subsequent fully decoupled calls to client.table('table_name') that your application architecture leans on (get_ibis_table).

💡 Conclusion

The fix ibis.duckdb.connect().register() is likely designed for a newer version of the Ibis framework (where memory table ingestion behaves differently) and is incompatible with the 5.1.0 pin in setup.py.

To move forward safely with this fix, a bump to a much higher Ibis Version (e.g., 7+ or 9+) would be required, or reverting back to the supported pandas backend.