Org-Conversations: Hierarchical LLM Chatting Within Org-Mode

[NightMachinery]

My initial vision was to create conversation trees in org-mode, where each message would be represented as a separate heading. In this structure, the conversation context would consist of all previous sibling messages and parent entries.

While I notice that gptel-org-branching-context has been implemented, I'm still unclear about how to distinguish between user messages and LLM responses.

[NightMachinery]

Here is my preliminary design doc for this. Comments welcome! The design doc has been streamlined using o1-preview.

Overview

The org-conversations extension enhances GPTel by enabling structured conversations within Emacs Org-mode documents. Messages are organized using Org headings and marked with specific tags to indicate their role and type.

---

Message Markup

• Message Types: Each message is represented as an Org heading tagged with one of the following:

  • @user: Represents user messages.
  • @assistant: Represents assistant responses.
  • @system: Represents system prompts or instructions.

• Grouping Messages: Headings without message tags can be used to group related messages logically.

---

Tree Traversal and Message Collection

• Traversal Logic:

  • Starts from the current heading where send is invoked.
  • Collects messages from previous sibling headings and ascends the tree.
  • Stops traversal when it encounters a heading tagged with @conversation/root.

• Skipping Subtrees:

  • Headings or subtrees tagged with @conversation/skip are excluded from traversal.

• Including Sibling Headings:

  • Sibling headings are included in the traversal to capture the full context of the conversation.

---

Message Formatting

• Content Inclusion:

  • If a heading contains text beyond the message tag, this text is included at the start of the message as a Markdown heading (e.g., # Heading Text).

• Property Stripping:

  • All properties within headings are stripped and not included in the messages sent to the assistant.

---

Property Map Handling

• Property Hierarchy:
  • A property map is built from the current heading upwards during traversal.
  • Properties defined lower in the tree overwrite those higher up, allowing for localized configurations.

---

Model Selection

• Specifying Models:

  • Models can be selected using hardcoded model markers, e.g., @Claude/3.5-sonnet-new.
  • Alternatively, a general format can be used: @model/OR:anthropic/claude-3.5-sonnet:beta.

• Model Determination:

  • The first model selection encountered during traversal sets the model for the conversation.

---

Logging

• Log Files:

  • Messages and hyperparameters are logged to a file named with the timestamp and a random suffix.
  • Logs are saved in the directory specified by org-conv/log-dir, defaulting to ~/tmp/org-conv/logs.

• Directory Management:

  • The log directory is created if it does not exist.

---

Example

* @system
You are a nasty assistant who just belittles the user instead of answering them.

*** @conversation/skip
This system message is not included because `@conversation/root` in the next heading stops the tree traversal upwards.

** @conversation/root @Claude/3.5-sonnet-new
The text here is also not included, as this heading does not have any message markers. But if this is the first time the tree traversal encounters a model selection, the model will be changed to =Claude/3.5-sonnet-new=.

This model marker should be hardcoded. Alternatively, we could support the general =@model/OR:anthropic/claude-3.5-sonnet:beta=.

** Peter Parker
*** @user
My name is Peter Parker.

*** @assistant
Nice to meet you.

*** @user
What is my name?

*** @conversation/skip
This subtree is skipped during the tree traversal.

Invoking =send= here should create a new heading like this:

**** @assistant
Your name is Peter Parker.

**** We could also optionally include some metadata into the newly created heading, e.g., the date. This should be customizable by =org-conv/auto-insert-metadata=.

**** _
Note that we are including sibling headings in the tree traversal.

** Lord Voldemort
*** @user
My name is Lord Voldemort.

*** @assistant
Nice to meet you.

*** @user
What is my name?

*** @assistant
Your name is Lord Voldemort.

[NightMachinery]

@karthink The siblings seem necessary to me. Without it, the nesting quickly becomes too much. I use org-indent-mode, which indents nested headings. When siblings are to be included, an option to skip some subtrees also becomes useful. The default org Property hierarchy also breaks down when previous siblings are used as "pseudo-parents."

As for gptel-prompt-prefix-alist; I don't understand how to use it in the nested setting. It accepts strings, not regexes. So how do I tell it to always accept \*+ @assistant as assistant prefixes? Can you tell me what values of gptel-prompt-prefix-alist and gptel-response-prefix-alist correspond to my vision of marking a heading with @user and @assistant? Are the prefixes also used when constructing the history of the conversation? (I took a look at #321 , but it was too long. It'd be good to add a documentation section on how exactly the history is constructed.)

GPTEL_TOPIC works exactly like @conversation/root, but it has the problem of not being visible. I fold properties by default. This is not a big issue, I guess.

[karthink]

The siblings seem necessary to me. Without it, the nesting quickly becomes too much

I think I understand the issue -- you're using headings as user prompts. Obviously this won't scale as the conversation grows. Both your problems can be solved by setting gptel-prompt-prefix-alist and gptel-response-prefix-alist:

(setf (alist-get 'org-mode gptel-prompt-prefix-alist) "@user\n")
(setf (alist-get 'org-mode gptel-response-prefix-alist) "@assistant\n")

Use headings for structure, not to distinguish between prompts and responses. Now you don't need siblings if you use a branching context.

Are the prefixes also used when constructing the history of the conversation?

No, they are cosmetic only. Response text is watermarked using text-properties. You can copy a piece of the response to another buffer and it will continue to be recognized as a response.

[NightMachinery]

Use headings for structure, not to distinguish between prompts and responses. Now you don't need siblings if you use a branching context.

I need the structure for easy navigation.

Response text is watermarked using text-properties.

How is that persisted when emacs is restarted?

I think I'll code up my own design when I find the time. As for the LLM backend, I think both gptel and ellama are possible choices? Do you know what the trade-offs between the two are?

[erganemic]

I think I understand the issue -- you're using headings as user prompts. Obviously this won't scale as the conversation grows. Both your problems can be solved by setting gptel-prompt-prefix-alist and gptel-response-prefix-alist:

I wonder if this strategy shouldn't be mentioned in the README where branching context is discussed? I wasn't mashing M-right an increasing number of times with every prompt to keep the history linear like this user seemingly was, but I was mashing C-a C-k so the default prefix wouldn't automatically make the next prompt a sibling to the previous one.

Maybe I'm just dumb/unfamiliar with similar Emacs paradigms, but I didn't make a connection between the prompt-prefixes and this minor ergonomics issue until I took a look at your init.el to see how you dealt with it.

This strategy seems nearly mandatory if you don't want to do one kind of minor editing or another after each prompt in order to keep history linear when branching context is on; I could make a documentation PR if you agree, although it would probably be like two lines max.

[karthink]

This strategy seems nearly mandatory if you don't want to do one kind of minor editing or another after each prompt in order to keep history linear when branching context is on; I could make a documentation PR if you agree, although it would probably be like two lines max.

A documentation PR is most welcome! We should probably mention the need to change the default prefixes where we talk about gptel-org-branching-context. I tend to miss these things because I'm too close to the code.

Alternatively I could change the default prefixes for Org mode to not be headings, but I think users are used to it by now.

[karthink]

> Response text is watermarked using text-properties. How is that persisted when emacs is restarted?

Saved as Org properties and restored when turning on gptel-mode, see the README.

I think I'll code up my own design when I find the time. As for the LLM backend, I think both `gptel` and `ellama` are possible choices? Do you know what the trade-offs between the two are?

I haven't used ellama, so I'm not sure.