Support SQLite for custom data layer

[tjroamer]

Overview

• Adds custom SQLite data layer, which is very simple to set up, no need to create extra database!
• This datalayer inherits from ChainlitDataLayer, reusing its business logic and re-implements API functions of LiteralClient
• Inspired by @sandangel work on Support MongoDB for custom data layer #796 for MongoDB
• Inspired by @hayescode work on Support Postgres for custom data layer #825 for Postgres

How to configure

import chainlit.data as cl_data
from chainlit.data.sqlite import SQLiteDataLayer

# a default database file chainlit.db will be created in the working dir
cl_data._data_layer = SQLiteDataLayer()

# Or optionally provide own file (can be relative or absolute) as the init parameter
# this code will create C:/myfolder if it doesn't exist, and data is persited in mydata.db
cl_data._data_layer = SQLiteDataLayer(db_filepath="C:/myfolder/mydata.db")

Test

A simple sanity check is implemented in cypress/e2e/data_layer_sqlite, which is a simplified copy of cypress/e2e/data_layer.

Limitation

no file blob support, so chat history will not show attachments

[Rajatkhanna801]

Hi @tjroamer
I want to know If I want to create separate sqlite database for each user how I can do that.

[tjroamer]

Hi @tjroamer I want to know If I want to create separate sqlite database for each user how I can do that.

With the current chainlit data layer, user-specific sqlite database might not be straightforward to implement. One solution I can think of is that you create views for different users, and select only rows that relevant to that user. This would require some post-processing steps once you have the database as a single file.

[Rajatkhanna801]

@tjroamer
Actually as per my requirement I want separate sqlite database for each user and I think we can achieve that with sending custom database url.

[tjroamer]

Will these users use the same chatbot based on the same server? If this is the case, you cannot send custom database url for each user, since the database url is a server-side setting, and it is done only once when the server starts.

[Rajatkhanna801]

okay let me figure another way for that.

[Rajatkhanna801]

@tjroamer
I have configured Microsoft authentication and everytime new user logged in I am creating new database sqlite file.

But I am not sure how I can put data in that tables for specific user.

[intc-hharshtk]

I just see this when I load old thread How to fix it

[LOFWagner]

Any Idea on when this will be merged?

[qtangs]

I'm working on a new version of SQLAlchemyDataLayer so that it can support SQLite and other dialects. Will share it once I've done testing.

[dokterbob]

I just went quite deep into the existing SQLAlchemy layer as part of #1319, added tests and make it work with SQLite.

I then read the code in this PR and realised I wished it wasn't called SQLite; I don't think we need an SQLite-specific layer, we can suffice with using SQLAlchemy, but the code in this PR (and possibly what's lying in @qtangs's attic) has a lot of good stuff I'd love to merge.

All of this towards a much needed cleanup on the roadmap to moving all non-literalai (after all, it's the core product!) data layers into a separate community-maintained repo.

I'd invite you all to have a look at #1319 and am also inviting you to share ideas on a community repo and how we could best pull that off (I'm thinking to invite 2/3 community members and setup a monorepo, with data layers first - but only after having test coverage for data layers).

[dokterbob]

#1319 has just been merged, we now have test coverage for SQLAlchemy/SQLite.

Looking forward to integrate the goodies from here!

@tjroamer Feel like giving this a try? Note that you've shot this PR from your own main branch, might want to continue working in a feature branch (e.g. with a different name).

[qtangs]

I then read the code in this PR and realised I wished it wasn't called SQLite; I don't think we need an SQLite-specific layer, we can suffice with using SQLAlchemy, but the code in this PR (and possibly what's lying in @qtangs's attic) has a lot of good stuff I'd love to merge.

@dokterbob you're spot on. I've been building on top of the code here while integrating with SQLAlchemy. It has taken a while to rewrite everything and to test properly. Thanks for the reminder here.
I'm going to push the code as a draft PR for now while still not happy with it yet. But it's better to put it out there for the community to evaluate and correct. Hope this helps :)

[dokterbob]

Closing in favour of #1365.