help getting Q&A pairs back via API

[SilverSeed93]

Hi Everyone.

I'm attempting to write a python script to talk to Text Generation WebUI, provide it a file for context, and get Q&A pairs back from the LLM.

For the most part I don't know what I'm doing! I'm new at this and python both. I have managed to get this to work some of the time which is amazing in and of itself but the LLM will respond back with totally different raw responses and I'm having troubles parsing it into JSON.

Here is an example of when it works on file 1:
Assistant's raw response:
[{"question": "Who is the best tracker in Stargate SG-1?", "answer": "Master Bratac is the best tracker in the series Stargate SG-1, but if you are asking about intrateam skills than Tealc is the best tracker on the SG-1 team."},

Here is an example of when it breaks on the very next file:
Assistant's raw response:
Here are the questions based on the text: What are the two main types of hyperdrive systems? Assistant's raw response: Here are the questions based on the text: What are the two main types of hyperdrive systems?
Note: The second response provided 10 questions in non-json format and instead of answers just repeated the same 10 questions again not in json format.

Note: This is actually not about Stargate, just an example!

I've attempted many different prompts and provided an example of json I want attempting to force a structured response. I've tried find() to remove text that isn't json. I can't get the LLM to respond consistently. But... If the LLM doesn't respond properly I can't really parse properly!

Here is the bit I am not sure about:
`

headers = {
    "Content-Type": "application/json"
}

prompt = (
    "Please create distinct question and answer pairs from the following content. "
    "Ensure the response is a valid JSON array formatted like this: "
    "[{\"question\": \"[Question]\", \"answer\": \"[Answer]\"}].\n"
    f"{markdown_content}\n"
)

history = [{"role": "user", "content": prompt}]
data = {
    "mode": "chat",
    "messages": history,
    "max_tokens": 512,
}

`

I am using:
Model: Meta-Llama-3-8B-Instruct-Q8_0.gguf
Loader: llama.cpp
The rest of the settings are all default including assistant's.

Could anyone help me with this?

If I can manage to get this working I'd like to train an LLM to answer questions based on this data as, factually, as possible. However, I just read this old post:
#3038

Has anything changed since then? Can someone who has done this point me towards a few simple steps to look up so I can get started?

Thanks!

[Katehuuh]

With some luck you could try https://github.com/e-p-armstrong/augmentoolkit, if not try prompt by Trelis Research. If necessary you can force JSON output by using grammar file: Parameters>Generation>Filter by loader>Load grammar from file (.gbnf)>json.gbnf.

If I can manage to get this working I'd like to train an LLM to answer questions based on this data as, factually, as possible. However, I just read this old post: #3038

For training, I recommend https://github.com/hiyouga/LLaMA-Factory QLoRA to save VRAM.

[SilverSeed93]

With some luck you could try https://github.com/e-p-armstrong/augmentoolkit, if not try prompt by Trelis Research. If necessary you can force JSON output by using grammar file: Parameters>Generation>Filter by loader>Load grammar from file (.gbnf)>json.gbnf.

If I can manage to get this working I'd like to train an LLM to answer questions based on this data as, factually, as possible. However, I just read this old post: #3038

For training, I recommend https://github.com/hiyouga/LLaMA-Factory QLoRA to save VRAM.

Thank you! I will have a look into all of this.