Add memory streams

[levicki]

It would be very interesting if memory stream functionality from this academic paper was implemented for chat / adventure mode.

https://arxiv.org/abs/2304.03442

PDF

[levicki]

To add a little bit more context to this idea for people who are too busy to read the paper.

In their work they have implemented NPCs in an RPG game using LLM -- they are using some form of memory streaming to enable AI model to keep the context of the conversation, remember the events, locations, other characters, etc beyond the current token limit of 4096 tokens.

Needless to say, this enables much more human-like interaction because the LLM can keep track of things instead of starting to output nonsense after a while.

[one-some]

the core of this looks to be a cosine distance search with embeddings of short snippets of info, not suuuper suitable for big chunks of text. could probably use a summarizer model to break things down into chunks like this though

[levicki]

@one-some To be frank I don't understand the low level details — I just read an article describing their work, not the paper itself.

It sounded like something that could be used here. It is super interesting how they managed to get the NPCs to have an equivalent of human long term memory, remember and discuss events they attended together, remember what they talked about, what they need to do, etc.

I played a bit with KoboldAI in chat and adventure mode and I never got world info to do anything similar to that.

The point here being if I am not mistaken is that the world info in its current implementation is static (i.e. it is what you enter at the start and it doesn't change during conversation), and this memory stream thing they implemented is dynamic -- i.e. the LLM is "learning" stuff during conversation and is able to recall all of it later when proper context is encountered.

Maybe I am mistaken, but that's how I understand it. I am surprised nobody tried creating something like that before.

[levicki]

@henk717 What do you think of this? Is this something that could be useful?