Using context in gptel for chats

[munen]

Hi @karthink and @daedsidog

I saw that you've closed my PR for adding context in gptel. On the one hand, I'm a bit sad, because by now I have used my addition on a daily basis and have not yet found any bugs. On the contrary, it supercharged gptel and my general productivity. On the other hand, I'm looking forward to testing your (vastly more complex and therefore likely even more capable) PR. Thank you for your efforts 🙏

Before I can start using your addition, I have a basic workflow question. From the README change, I read:

You can include additional text regions, buffers or files with gptel's queries. This additional context is "live" and not a snapshot. Once added, the regions, buffers or files are scanned and included at the time of each query.

To me, that sounds like it could produce complicated bugs. Let me explain:

1. You add a file or region to the context.
2. You start a chat and ask a question.
3. You receive an answer, including a code block.
4. You update the code as you see fit.
5. For your next query, the context will automatically update.
6. In the chat from 2, you ask a followup question.

For step 6, the LLM will see the updated context for question 1 and 2. So the answer to question 1 will not necessarily make sense anymore. It will not see the original context, but the updated context plus it's original answer. Since the answer to 1 does not make sense, anything might happen when answering question 2.

For a chat history to make sense, I think it has to be immutable. Am I missing something?

Thanks, again, for your efforts! I'm looking forward to testing it as soon as this issue is cleared up (even if only in my head). My main workflow for using gptel is adding context and then asking the LLM questions until I'm happy. So, this question popped into my head immediately(;

P.S.: I can see the value in having an always up to date context outside of chats, for example when refactoring inline, of course.

[daedsidog]

For step 6, the LLM will see the updated context for question 1 and 2. So the answer to question 1 will not necessarily make sense anymore.

The PR also included ways for users to customize how the context appears to the model. From my experience of using even the default context message, I haven't encountered a lot of the model "confusion" that you describe, even though I understand entirely that, for a regular conversation, it should be confusing to it. Somehow, the OpenAI models I use seem to be capable with it. However, if this is a problem, you can customize gptel-context-wrap-function to make it clearer that the context is subject to dynamic patching, and then there is no issue whatsoever.

For a chat history to make sense, I think it has to be immutable. Am I missing something?

Anecdotally, what you describe (snapshot contexts) can also be achieved in how I previously used the context before I opened the PR, which was to mark the context, then use the gptel-context--string function to include the context directly into the history (you would have to disable context appending in the chat though). Alternatively, you can always start your query with informing the model of the dynamic context.

@karthink signaled that he is open to suggestions to expand the context functionality with the goal of making it flexible to include most possible use cases while keeping the default functionality simple & intuitive. In my opinion I think what you described is already covered, however.

[karthink]

@munen You're right about the conversation "history" changing as a result of "live" context.

Let's call the two ways of doing this "live context" and "snapshot context".

For a chat history to make sense, I think it has to be immutable. Am I missing something?

This is correct, but I suggest trying out the current behavior and seeing how LLMs respond. While you can drive it to a state where future responses don't make sense -- such as by deleting the context region half-way into the conversation -- in practice this does not occur. (And if required, you can indicate to the LLM that this is the current state of the context.)

With live context the current state of the conversation is introspectable from the chat + relevant buffers, and the history is not. But chat buffers are already fully modifiable, so what you see can be quite different from what was sent anyway. More generally, I think having hidden state is more confusing than having inconsistent history -- the most important thing is to be able to see what will be sent next.

If you snapshot context instead, the history is fixed but the current state is not apparent any more. This context feature creates new invisible state (i.e. not the visible state of regular Emacs buffers/files) and is confusing.

I think the idea of adding snapshot context makes sense. But for clarity, I think this kind of context should be included in the chat buffer itself. All you need then are utility commands to insert chunks from other buffers into the chat buffer (i.e. wrappers over append-to-buffer, insert-file etc).

---

This reasoning is independent of the other advantages of having live context, such as refactoring (as you point out), not having to remember to update/add to the snapshot being sent when working with constantly changing context, reducing costs and inference time etc.

Finally, I understand that most users will never use either kind of context, preferring to yank text into the chat buffer on the rare occasions they need it. For this reason, gptel-context, like most gptel features, is not loaded until it's required. (Even the transient menu and Curl handlers are loaded on demand, gptel.el does not load anything except itself.)