API in chat mode

[Luftkrieg]

Is there any documentation or template for the API I can use for the chat mod?

[Xe]

Run the program in Chat mode and click on the API button at the bottom of the page. It'll tell you how the parameters differ.

[Xe]

Hmm, I'm not getting things to work though. I have been trying to reverse-engineer the WebSocket API to actually generate new text, but so far I'm not getting any of the actual data.

[oobabooga]

API in chat mode is not really supported at the moment. The API examples only work without --chat or --cai-chat.

This issue is related to this: #221

[Syeph0n]

Ah, that's a shame. I should have checked this before attempting to integrate into my discord bot 😅 Would obviously love to see that kind of functionality in the future though!

[burgerlawful]

Has there been any changes to this or is it still unsupported? Has anyone figured out any good way around it? I'd really like to use API with chat mode and characters

[mcmonkey4eva]

I think one of the big design improvements that's needed to text-gen-webui is decoupling the basic user interface format selection from the fundamental function of the program.

Which is to say, --chat shouldn't be a command line arg, it should just be a tab in the UI that can be clicked on with 0 second load time. The python backend shouldn't need to change to support it, just run generation through different codepaths depending on which tab the user is in.
Thus, similarly, API calls should 'just work' without having to worry about details like that.

[qbyss]

I also wish there was a way to use the API with chat mode, I have a voice recognition and speech synthesis that I want use that way.

[elakapmain]

Hopefully this functionality will be implemented in time. Seems like the API can work with stream, so it would be interesting to see it function with --chat or some other implementation.

[botchi09]

I can't get the API to work at all, even with the bogstandard JS example. I have a feeling it's just broken, full stop. The json decoder seems to have problems.

[Gigabyte5671]

Where did you find a JS example?
Using the same data structure as the python api-example, I'm attempting to access the API with JS's fetch method:

const response = await fetch(
  apiUrl + 'run/textgen',
  {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      data: [
        [prompt, params]
      ]
    })
  }
);

The server responds with:
Error 500: { error: null }

And raises the error:

...oobabooga-windows\installer_files\env\lib\json\decoder.py", line 355, in raw_decode

raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)

(Running both the Python server and JS client on Windows 11)

[DaveChini]

I was having the same issue.
Make sure you're not running the server with --chat, as running with that parameter skips API initialization. Next make sure you're mimicking the payload that's being passed in api-example.py, for vicuna llm mine looks like the following.
["### Human: what is gradio and how do i use it?\n### Assistant: ", {"max_new_tokens": 256, "do_sample": true, "temperature": 0.72, "top_p": 0.73, "typical_p": 1, "repetition_penalty": 1.1, "encoder_repetition_penalty": 1.0, "top_k": 0, "min_length": 0, "no_repeat_ngram_size": 0, "num_beams": 1, "penalty_alpha": 0, "length_penalty": 1, "early_stopping": false, "seed": -1, "add_bos_token": true, "custom_stopping_strings": ["### Human:", "### Assistant:"], "truncation_length": 2048, "ban_eos_token": false}]

oh i also had to modify the api-example.py's response to this, as windows doesnt recognize 0.0.0.0 as localhost
response = requests.post(f"http://localhost:7860/run/textgen", json={

[botchi09]

Where did you find a JS example?

It was in the API popout at the bottom of the page

[Gigabyte5671]

Thanks Dave and Botchi.

When running the server with...

python server.py --auto-devices --chat

...the web UI can be accessed from http://localhost:7860/ through my browser. At the bottom of the web UI I see a blank "api" section:

Clicking "Use via API 🚀" shows this overlay:

---

If I instead run the server with...

python server.py --model facebook_opt-350m --listen --no-stream

...the API can be accessed from http://localhost:7860/.

Running api-example.py works perfectly fine. However, the above JS code doesn't work (failing on the server side with the stated error). My payload looks like this:

[
  "What I would like to say is the following: ",
  {
    "max_new_tokens": 200,
    "do_sample": true,
    "temperature": 0.72,
    "top_p": 0.73,
    "typical_p": 1,
    "repetition_penalty": 1.1,
    "encoder_repetition_penalty": 1.0,
    "top_k": 0,
    "min_length": 0,
    "no_repeat_ngram_size": 0,
    "num_beams": 1,
    "penalty_alpha": 0,
    "length_penalty": 1,
    "early_stopping": false,
    "seed": -1,
    "add_bos_token": true,
    "custom_stopping_strings": [],
    "truncation_length": 2048,
    "ban_eos_token": false
  }
]

[sam2332]

any update on the api and chat mode?

[ruizcrp]

I am also wondering about that. At the moment, if you want to have api and chat at the same time, you would need the double of the resources - is that correct? As one would serve the api-calls and one the chat-queries? So no common queue is possible?

[sam2332]

No unfortunately I don't think it is possible to stream the API into the chat mode

[miaoyizi]

hello Whether the API interface supports concurrent requests？