Media (and other) links.

[metachip]

Karthik, question/suggestion about links.

They are VERY convenient. Often more convenient than using context.

I understand that in chat buffers, one can include links to media files:

https://github.com/karthink/gptel?tab=readme-ov-file#including-media-images-documents-with-requests

1. Why is this limited to chat buffers only?

   I understand I can use the context to include media files in other buffer types but why not also allow links in buffer of other types, for example org or markdown?

2. Why is this limited to media files types?

   If a link is to a file that has a mime type that is not media known to the model, but it is text, why not include the contents of the file in the prompt at that point? That is, if it's a link to a txt, org, md or other text file type, just include it in the prompt.

3. Why do links need to be standalone?

   In org mode buffers and chat buffers, why not support links however they appear, not just when they are standalone. Likewise, if a link is internal, to another heading in the same or a different file, why not allow the body text in that heading be included in the prompt. It would for example, make it very easy to include "boilerplate" or other context by simply including links in the prompt text.

[2024-11-23 Sat 23:04]

[karthink]

Question 1: Because users can have lots of links in regular Org buffers, which are almost always intended for purposes other than sending data to the LLM.

Question 3: To signal intent more clearly, and to make the links whose data will be included stand out and easier to spot. Making it a "standalone" link is used as a proxy for the @file: syntax that some other LLM clients use.

I can make both of these customizable, but I think it's easy to forget that you've included something in a link that you didn't mean to send, especially in a non-chat buffer that's being used for something else primarily. If you have more thoughts on this I'd be curious to hear them.

---

Question 2: I like the idea of using links to set up per-buffer context lists, as a solution to #475. It's also more transparent than an internally maintained context list that is only accessible via running a command/opening up the context buffer.

The reason this is limited to media file types right now is just mental compartmentalization -- I added media support and @daedsidog added context support at two different times, and it didn't occur to me to combine the two. I'll work on adding this feature.

It's still ill-defined because there are many ways to link to buffers in Org, including user-defined link types. It's also difficult to link to regions of buffers robustly. On the other hand Markdown does not have any built-in way to link to buffers, only files. So we'll have to think about how to do this carefully. If you can provide a spec to help nail down the feature it'll be easier to implement.

One thing I can do but wish to avoid is to define a new gptel-specific Org link-type.

See also #325.

[daedsidog]

One thing to be wary of is cases when you do not wish links to be treated as actual file links. For example, I often use an Org document as a reference for prose and content material, and would not want to have links translated into media. I don't se a way around that other than having a gptel-specific link type.

...On second thought, if the link translation is handled only in the chat buffer, it's fine.

[karthink]

For example, I often use an Org document as a reference for prose and content material, and would not want to have links translated into media

Yes, this is why link translation is currently limited to chat buffers.

[metachip]

Question 1: Because users can have lots of links in regular Org buffers, which are almost always intended for purposes other than sending data to the LLM.

Understood, but as an org user this seems unnecessarily limiting.

How about any links that appear in a region of text that is inside a drawer called :CONTEXT: for example, be sent to the LLM and no other links. That way one can manually construct context explicitly.

I like the idea of using Org/markdown links to add to the context, as discussed in #481, as the context set automatically becomes buffer-local, transparent and easily editable. But that has other issues, including that you can't have "live" buffer regions in the context.

What if links that are within a CONTEXT drawer, which have a label name, which is suffixed with two line numbers, separated by colon, which is used to define a region within a referenced context file. More or less as you must be doing already within context buffers. So something like this:

:CONTEXT:
[[id:a18cbfc7-231d-414f-8f0f-4b64f7dff504]]
[[file:/some/other/path]]
[[file:/path/to/file][my source code 124:581]]
[[id:a18cbfc7-231d-414f-8f0f-4b64f7dff504][some context 4:10]]
# etc
:END:

1. Any link that targets a headline in an org file means to include the body of that headline.

2. Any link that targets a file means to include the contents of that file.

3. Any link that targets a file where there is a colon separated number pair in the link description means to target that region of lines within the target file.

4. Likewise, regions can be targeted from any source, for example a headline identified by ID.

I can think of some corner cases here. For example, if the referenced region changes because the referenced file is modified, the limits in the link would need to be updated. As I imagine, you do now with context buffers.

Question 3: To signal intent more clearly, and to make the links whose data will be included stand out and easier to spot. Making it a "standalone" link is used as a proxy for the @file: syntax that some other LLM clients use.

Fair enough. However, if links appear inside a context draw for org mode users, then this should not be a requirement.

I can make both of these customizable, but I think it's easy to forget that you've included something in a link that you didn't mean to send, especially in a non-chat buffer that's being used for something else primarily. If you have more thoughts on this I'd be curious to hear them.

I prefer DWIM as far as possible and avoiding customizations where possible. I don't know about others and their use of gptel, but I tend to assume org mode or markdown syntax for all the text that I'm using, even in files that are not explicitly of that syntax. Org mode in particular offers powerful syntactic sugar to enable gptel to be integrated in ways I've suggested here.

Question 2: I like the idea of using links to set up per-buffer context lists, as a solution to #475. It's also more transparent than an internally maintained context list that is only accessible via running a command/opening up the context buffer.

I'm 100% with you on this one.

The reason this is limited to media file types right now is just mental compartmentalization -- I added media support and @daedsidog added context support at two different times, and it didn't occur to me to combine the two. I'll work on adding this feature.

Excellent. I think this will be a major workflow gain, at least for me :)

It's still ill-defined because there are many ways to link to buffers in Org, including user-defined link types. It's also difficult to link to regions of buffers robustly. On the other hand Markdown does not have any built-in way to link to buffers, only files. So we'll have to think about how to do this carefully.

Agreed. One needs to consider this carefully in the context of files that are not org mode.

If you can provide a spec to help nail down the feature it'll be easier to implement.

I think the solution I suggested above will work in markdown as well as in org mode. That is, you co-opt the label part of the link to have syntax that you interpret for gptel purposes if it appears in a :CONTEXT: drawer. The equivalent of the idea of a context draw in Markdown might be to use front matter?

One thing I can do but wish to avoid is to define a new gptel-specific Org link-type.

Agree 100%. You do not want to change anything org mode does with links. Including customised link types and all the rest. However, the link label is not touched by org mode and is therefore "open slather" for you with gptel, So long as it appears within a context drawer!

[metachip]

More ideas from as an org mode user: further to the :CONTEXT: drawer suggestion to isolate LLM links, perhaps when working in org mode with gptel-org-set-topic set, or gptel-org-branching-context enabled, any link, not just those within a :CONTEXT: drawer should be resolved to the LLM. In this case, links are likely to be explicitly intended for dereferencing. The gptel-track-media variable would still apply so one can turn this off. If not already saved, the status of track media should be included in the gptel-org-set-properties data set.

[metachip]

The more I think about it, I think the link metaphor to embed context in gptel will be a great addition. It's simple, explicit and easily managed. Especially in org mode because of the rich features set apropos links and what one can do with them. For example MCP servers #484 presenting a RESTful API on a URL fits right in. If if not on a URL end-point, perhaps one can shell out using shell: or ssh: (tramp) links in some creative way too. Food for thought.