[Feature Request] Token Banning/Bias

[neCo2]

Not too familiar with what options the various backends make available, but if at all possible, a field to ban or bias certain tokens and phrases would be great to have.

[neCo2]

Read some stuff and basic bias/banning on a token basis seems simple enough to implement. llama.cpp, koboldcpp and the openAI compatible oogabooga endpoints just take a logit_bias in their request with a { key:value } dictionary where key is a token id#, and value is an integer between -100 and 100, with -100 being a ban.
It'd probably be best to take input as a string and then send that to the backend's tokenizer for ease of use.

Problem is, this only works for individual tokens, not phrases, and I've found no concrete answers on how this is implemented in the few places I know that have phrase bias like novelAI and AI21.
I found this which could be a solution? Essentially

1. Take first token of banned phrase and add it to stop tokens
2. If the token is generated, generation will stop
3. Then generate however many tokens the banned phrase is
4. If it generates the entire banned phrase, remove it from the prompt
5. Bias the first token of the phrase (or possibly all tokens of the phrase to achieve the total phrase bias that was initially set)
6. Generate number of tokens of the banned phrase again
7. Unbias tokens
8. Resume normal generation

What's your opinion on this? I feel like it'd be really inefficient to implement this way. Is there any way to mess with the parameters without stopping and restarting the generation?

I mostly use bias for punctuation and newlines which are all single tokens more often than not, so I might try to implement the single-token bias when I find some time, but phrase bias would be incredibly useful to have to stop some weird habits models tend to have.

[lmg-anon]

I think there's no need to include the token in the stop tokens given that we're streaming the tokens, which allows us to determine on the fly the point at which the banned sequence must be omitted from the prompt. But, I don't think there's any way around stopping the generation process to remove the sequence from the context and add the token bias.

[neCo2]

Ah right, we can just manually send a stop command.
I really wonder if that is the way it's implemented in NAI. I remember it sometimes getting stuck for a bit in places when a biased phrase was (probably) generated, so maybe?

[neCo2]

@lmg-anon I'm bad at finding API documentation, does KoboldCPP actually not have a detokenize endpoint?

[lmg-anon]

@neCo2 No, I don't think so.

[neCo2]

Maintainer said he'll consider adding one in future.

[culturedniichan]

This would be a very nice feature. This is what I found for the OpenAI API which I think ooba also implements now: https://platform.openai.com/docs/api-reference/chat/create#chat-create-logit_bias

But I have kinda little knowledge about the inner workings of these backends. For example, since the user would be typing words, like in NovelAI, I suppose it would need to be translated into the token ids by querying the API first? I think SillyTavern is doing it, I will take a look and see how it does it. But this would be great to be able to make certain sequences less likely

[lmg-anon]

@culturedniichan That's right. Also, in case you didn't notice, OP did a PR for this issue (but it's still not complete), you can take a look at it here: #25

[neCo2]

Closing this since regular token bias has been implemented now. Opened up a separate issue for phrase bias here: #58