RAG add-on package?

[colobas]

Hi @karthink , thanks for a great package.

I read somewhere in the Issues that you're planning to work on an addon package at some point to support RAG pipelines. I think that's awesome and am interested in helping.

If we have a way of producing context chunks from whatever RAG strategy, from the perspective of gptel, you already support having these chunks as context in your queries.

This two-part blog post (pt1 pt2) does a great job of breaking down the necessary high level components for a code-assistant RAG pipeline inspired in Cursor, and they provide a candidate implementation in Python (which maybe gptel can just leverage?)

Similar to the current context handling, gptel could maintain a list of vector databases and a user could select which one to include in their query?

Let me know if doing this via python inter-op is an option you are happy with and I'm happy to try to get to some MVP next weekend

[karthink]

I read somewhere in the Issues that you're planning to work on an addon package at some point to support RAG pipelines. I think that's awesome and am interested in helping.

@colobas That sounds great!

If we have a way of producing context chunks from whatever RAG strategy, from the perspective of gptel, you already support having these chunks as context in your queries.

I think there's more to it than that, but whatever needed on gptel's side shouldn't be too much work to add.

This two-part blog post (pt1 pt2) does a great job of breaking down the necessary high level components for a code-assistant RAG pipeline inspired in Cursor,

Thanks for the links. I've seen these before but haven't had the time yet to go through these, as my FOSS time has been taken up by gptel's UI issues.

and they provide a candidate implementation in Python (which maybe gptel can just leverage?)

Yes, I'd like to make it trivial to plug in RAG results into gptel's control flow. See next point.

Similar to the current context handling, gptel could maintain a list of vector databases and a user could select which one to include in their query?

I'm imagining something a little more generic than that, where a context source is a function call -- this source can do whatever is required (fetch, rerank etc) and return the results in a simple format that gptel can include with requests.

The UI doesn't need to be like this, but it might give you some idea of what I mean.

The reason I want gptel to farm out the retrieval to a source (and be agnostic) is that things are changing continuously in the RAG space, and the more assumptions I bake into gptel the harder it will be to maintain. For users who don't want to or can't set up a full-fledged RAG pipeline, I want it to be easy to use the same context source mechanism to plug in results from a source that does a simple text search, like by running ripgrep-all on your PDF documents, or feeding in recoll results.

Let me know if doing this via python inter-op is an option you are happy with and I'm happy to try to get to some MVP next weekend

In this vision Python inter-op would really be more like a shell pipe, so it should fit in just fine.

When you try it, let me know what's required on the gptel side. The next thing I need to work on is making gptel's control flow customizable so it can handle more general workflows.

[colobas]

Sounds great and completely agree with your comments.
I'll share updates next sunday/monday :)

[colobas]

hey @karthink in the meantime I was made aware of the Model Context Protocol, used by Zed. It seems to align with your notion of supporting generic sources

The reason I want gptel to farm out the retrieval to a source (and be agnostic) is that things are changing continuously in the RAG space, and the more assumptions I bake into gptel the harder it will be to maintain. For users who don't want to or can't set up a full-fledged RAG pipeline, I want it to be easy to use the same context source mechanism to plug in results from a source that does a simple text search, like by running ripgrep-all on your PDF documents, or feeding in recoll results.

so potentially the way to go is to implement MCP client functionality for gptel? and then, like you alluded to, RAG-like sources would just be a possible type of MCP source to be consumed by gptel. this would also let users benefit from a bunch of already implemented MCP services e.g.: https://github.com/modelcontextprotocol/servers

I'll give this a go over the weekend

[karthink]

I'm aware -- see #484 and this discussion. We'll try to support it in gptel eventually (~a few months)