Add section about Jujutsu GPT

[nightscape]

I created a GPT to help myself in working with Jujutsu. I thought it might be useful to others as well.

Checklist

If applicable:

• I have updated CHANGELOG.md
• I have updated the documentation (README.md, docs/, demos/)
• I have updated the config schema (cli/src/config-schema.json)
• I have added tests to cover my changes

[martinvonz]

Thanks for your PR. I suspect I'm not in the target audience for this, and I don't have a GPT account to even try it. I'll let others comment if they find this useful.

[ilyagr]

The mkdocs-material plugin we use for docs also seems to have a way to add such a thing to the docs website. See the bottom-right corner of https://pdm-project.org/latest/ for an example. I actually tried using it at some point, and it was unexpectedly helpful, though I'm usually quite wary of such things.

[nightscape]

@ilyagr that looks cool indeed! The integration into the docs would make it much more discoverable.
Unfortunately, I don't seem to be able to ask the PDM bot anything.
Also LlamaIndex probably means one would have to self-host the inference engine somewhere.

[0xdeafbeef]

We can try https://sidekick.kagi.com/ as an alternative

[nightscape]

Cool! Just registered there 😃

[KingMob]

This is a cool idea, but iiuc, it's going to be limited by GPT's training set and cut-off date. For a fast-changing project like jj, it's almost certainly not going to have some of the latest changes.

Also, I have no data on this, but I suspect devs are more likely to have API keys for terminal and editor plugins, rather than the ChatGPT Plus subscriptions required for custom GPTs. I know I do, anyway. (Still don't know why OpenAI hasn't combined them for devs, but whatever...)

[nightscape]

Hey @KingMob, the Jujutsu GPT does not solely rely on the data it was trained on. I uploaded a PDF-ified version of the Jujutsu docs, which it uses as reference (btw, @martinvonz would it be possible to automatically CI-build a PDF containing the full documentation of JJ?).
Unfortunately, sometimes it still uses information from its training run instead of the provided documentation.
I'm still trying to figure this out.

Regarding ChatGPT Plus vs. API: I haven't checked if it is possible to build something like a GPT (especially the reference documentation part) using the API. There's Agents, but you have to host them on your own, and someone would have to provide his/her API token for the Agent, paying all the bills for people using the Jujutsu Agent.
With GPTs, that part is much simpler, because the usage is paid by the user and not the creator of the GPT.

[khionu]

I'm pretty against supporting LLMs for assisting in technical understanding. If ideas are communicated poorly, they can leave deeply rooted misunderstandings that make further learning harder. LLMs can't have conceptual understanding of their outputs. We have two venues (Discord and GitHub Discussions) for this community to help new members learn, and we have a fairly large number of people given the age of the project who are writing and maintaining their own styles of introductions. I'd rather not have the project officially support a tool that can provide bad information at any point without any oversight. Even humans have the oversight of their peers.

[necauqua]

While that's true, an LLM is a great search engine, esp. with google et al being defeated by seo spam. I think it's great at being the thing that points you to correct docs no matter the way you phrase your question.

Just regurgitating docs will lead to hallucinations like you mentioned, so yep nobody should immediately believe it's output, but I've been using a few of those "GPT"s (what a dumb name lol) to find pointers to docs to some thing and it was great.

Yet again this is me and you being cautious about their use, so I agree we shouldn't directly support this form of it.

P.S oh yep btw a huge issue with discord is that it's ungoogleable, like completely
Check out https://answeroverflow.com tho

[khionu]

so I agree we shouldn't directly support this form of it.

While I appreciate it, I still feel the need to counter a few of your points.

an LLM is a great search engine [...] the thing that points you to correct docs

We already have a search functionality in the docs. If that doesn't perform well enough, I'm sure we could look into how to improve it.

so yep nobody should immediately believe it's output

But people do. It's like a knife, you can teach a kid that the edge is sharp, but they're not going to fully process how that is a risk until they (or they see someone else) cut themselves. But unlike with knives, we're not making tools that can't be made fundamentally safer. We can choose to not adopt tools that people will cut themselves on.

[necauqua]

But people do [..]

well here we just agree, I implicitly meant just that when I wrote "this is me and you being cautious"

doesn't perform well enough

hmm, sure, I'm biased since the llm helped me to find stuff a bunch of times where google failed.

The major issue is a bunch of useful information is locked behind discord walls, but over there we're discussing just that right now

[khionu]

It seems the docs search is powered by https://lunrjs.com/ as a mkDocs plugin. It is tunable, but it would be a decent investment. It might be worth an issue to explore what we could do to make it easier for people to query.

[KingMob]

Discord is great for real-time interaction, but less suited for long-term documentation or searchability. It would be nice to move the best conversations/questions into the FAQ or other areas of the docs.

[khionu]

I've started work on a JJ Community org. While I'm against JJ taking an official stance supporting GPT, I'd be fine having this linked or homed under the Community Org. Whether or not that option is taken, this PR has gotten a bit old with no further comments, so I'd suggest we close it without merging.