[FEAT]: Remove the [CONTEXT 0][CONTEXT 2][CONTEXT 5] information

[morbificagent]

What would you like to see?

Hi together,
i am playing around with anythingllm connected to perplexity and its working good, but all responses are containing [CONTEXT 0][CONTEXT 2][CONTEXT 5] inside the text which makes it hard to read.

I have tried to filter it out by prompt but it seems to be impossible.

Would great if anyone has an idea how to filter it out in the response as it doesnt help anyone as nobody knows whats [CONTEXT 5]

[lewismacnow]

What would you like to see?

responses are containing [CONTEXT 0][CONTEXT 2][CONTEXT 5] inside the text which makes it hard to read.

I have tried to filter it out by prompt but it seems to be impossible.
[CONTEXT 5]

You can usually work around this with a tweaked System Prompt:

"Given the following conversation, relevant context, and a follow up question, reply with an answer to the current question the user is asking. Return only your response to the question given the above information following the users instructions as needed and do not mention or refer to the "context" specifically in your response. You respond in plain English only as if the provided context is knowledge you are aware of and do not respond with JSON or tool calls, respond like a human speaking to another human."

Another good method is to create a "new" model, with some existing conversation already baked into the model file:
Example from Ollama

MESSAGE user Context: [CONTEXT 0]: Grass is green [END CONTEXT 0] [CONTEXT 1]: The Sky is blue [END CONTEXT 1] Is the Sky green?
MESSAGE assistant Last time I checked, the sky was blue. Grass, on the other hand, is green!

[timothycarambat]

i am playing around with anythingllm connected to perplexity and its working good

How are you using AnythingLLM? The context is in the system prompt so that shouldn't be showing in an LLM response, and if it does that is just a model behavior. I haven't had it return context snippets in the response before

[morbificagent]

Hi together and thanks for the answers,

i have read a little bit and it looks like perplexity is delivering these [context x] "informations" inside the streamed text response.
And saying "dont deliver this..." in the prompt doesnt help.

Here are the tokens that are coming from perplexity:

[backend] info: Original Token: -Wissen
[backend] info: Original Token: darst
[backend] info: Original Token: ellt.[CONTEXT
[backend] info: Original Token: 1][CONTEXT
[backend] info: Original Token: 31][CONTEXT
[backend] info: Original Token: 54]

So i think there isnt a good way to filter them out.

[timothycarambat]

Ah, seems like a model behavior. I wonder if swapping that to XML would help. Lots of training data is XML/HTML so it may be better are stripping that out of the response. What perplexity model are you using so i can try to repro?

[morbificagent]

Its this one...

And i dont know why but sometimes single words in a sentence are in different languages when using this model...

[morbificagent]

any news about this? is it fixable or do we have to live with it because its a problem only on perplexity-side?