Can gptel use the model loaded by GPT4All desktop by default?

[nickanderson]

In playing with the GPT4All capability, I am simply running the GPT4All desktop client with the API server enabled.

When GPT4All starts it loads the default model as configured in GPT4All.

~/gpt4all/bin$ ./chat 
[Warning] (Wed Nov 1 10:24:45 2023): Could not connect "org.freedesktop.IBus" to globalEngineChanged(QString)
[Debug] (Wed Nov 1 10:24:45 2023): deserializing chat "/~/.cache/gpt4all//gpt4all-35cfc704-21ec-48d5-8c2a-d8ed9a54ed83.chat"
[Debug] (Wed Nov 1 10:24:45 2023): deserializing chats took: 3 ms
llama_new_context_with_model: max tensor size =   102.55 MB
llama.cpp: using Vulkan on NVIDIA RTX A2000 8GB Laptop GPU

When I run gpt-send for the first time it tries to load the model configured in gptel, even if it's the same model that is loaded by default by the GPT4All desktop client. With only 8GB of memory, the model fails to load:

Error allocating memory ErrorOutOfDeviceMemory
error loading model: Error allocating vulkan memory.
llama_load_model_from_file: failed to load model
LLAMA ERROR: failed to load model from ~/.cache/gpt4all/mistral-7b-openorca.Q4_K_M.gguf

I often get something like ChatGPT response error: (((efb349aa1334d8d7e36483d9bf0c244f . 0)) Could not parse HTTP response.) Could not parse HTTP response. Though, sometimes I do actually get responses.

I guess that my issue is mainly that gptel fails to load the model and thus can't get responses. So, can gptel use the model loaded by GPT4All desktop by default if they are the same or should I be looking for a different solution?

[karthink]

Sorry, I don't understand your description. Do you want gptel to dynamically load the model that is currently active in GPT4All?

[nickanderson]

Sorry, it's not a good description.

Rather than having gptel load the model, I wonder if it can use the already loaded model.

When I launch GPT4All desktop, it loads whatever it's default model is and I don't have enough VRAM for gptel to load the same or a different model concurrently and I don't see how to make GPT4All Desktop avoid loading a model by default.

Does that make sense?

[karthink]

Rather than having gptel load the model, I wonder if it can use the already loaded model. When I launch GPT4All desktop, it loads whatever it's default model is and I don't have enough VRAM for gptel to load the same or a different model concurrently and I don't see how to make GPT4All Desktop avoid loading a model by default.

Ah, I see the confusion. gptel does not "load" anything -- it simply makes http requests using GPT4All's API. gptel is basically an http client/frontend for Curl. There is only one model loaded into memory, and it is handled by GPT4All.

[nickanderson]

Really?

Currently, GPT4All v2.5.2 is configured to use mistral-7b-instruct-v0.1.Q4_0.gguf (Minstral Instruct in the UI) by default.

When I launch GPT4All Desktop I see it's using my GPU:

~/gpt4all/bin$ ./chat
[Warning] (Wed Nov 1 15:10:37 2023): Could not connect "org.freedesktop.IBus" to globalEngineChanged(QString)
[Debug] (Wed Nov 1 15:10:38 2023): deserializing chat "/home/nickanderson/.cache/gpt4all//gpt4all-ccd56ba5-cb50-4968-ae0c-6b37821cd304.chat"
[Debug] (Wed Nov 1 15:10:38 2023): deserializing chat "/home/nickanderson/.cache/gpt4all//gpt4all-35cfc704-21ec-48d5-8c2a-d8ed9a54ed83.chat"
[Debug] (Wed Nov 1 15:10:38 2023): deserializing chats took: 4 ms
llama_new_context_with_model: max tensor size =   102.55 MB
llama.cpp: using Vulkan on NVIDIA RTX A2000 8GB Laptop GPU

In nvtop I can see that it's grabbed a bunch of memory.

I can immediately send something to it via it's UI and it responds, no other messages are logged.

With gptel configured to use the same model by default:

(gptel-make-gpt4all
 "GPT4All"                              ;Name of your choosing
 :protocol "http"
 :host "localhost:4891"                 ;Where it's running
 :models '( "mistral-7b-instruct-v0.1.Q4_0.gguf"
            "mistral-7b-openorca.Q4_K_M.gguf"
            )) ;Available models
;; (setq gptel-model "mistral-7b-instruct-v0.1.Q4_0.gguf")
;; (setq-default gptel-model "mistral-7b-openorca.Q4_K_M.gguf")
(setq-default gptel-model "mistral-7b-instruct-v0.1.Q4_0.gguf")
(setq-default gptel-max-tokens 800)
(setq gptel-directives   '((default . "You are a Nexus, Nick Anderson's helpful assistant. You have dry humor.")
                          (programming . "You are a large language model and a careful programmer. Provide code and only code as output without any additional text, prompt or note.")
                          (writing . "You are a large language model and a writing assistant. Respond concisely.")
                          (chat . "You are a large language model and a conversation partner. Respond concisely.")))

I evaluated the above elisp, then visited gptel-menu to check the settings, I needed to re-apply the default directive (plausibly I need to use setq-default there too). When I hit RET to send the prompt I got user-error: No ‘gptel-api-key’ found in the auth source.

Then I selected some text Finish this sentence: "I've got a lovely bunch of coconuts, " and ran gptel-send. Argh!, now I just keep getting the message about gptel-api-key which is really more #122 …. but I was previously seeing the error allocating memory.

[nickanderson]

There we go ...

In GPT4All desktop I switched the model to Minstral OpenOrca (mistral-7b-openorca.Q4_0.gguf). Then I used gptel-menu to switch to the same model and hit RET to send the prompt.

Finally I selected some text and used gptel-send at this point I got errors on the console where I launched GPT4All desktop.

Error allocating memory ErrorOutOfDeviceMemory
error loading model: Error allocating vulkan memory.
llama_load_model_from_file: failed to load model
LLAMA ERROR: failed to load model from /home/nickanderson/.cache/gpt4all/mistral-7b-openorca.Q4_0.gguf

While sending the same text in GPT4All Desktop UI has no issue.

[karthink]

Error allocating memory ErrorOutOfDeviceMemory

Using GPT4All's REST API takes more memory (according to the description when you hover over the button):

Can you try if this happens with a smaller model too?

[nickanderson]

Sure, I will try with Mini Orca.

[nickanderson]

I get the similar behavior.

Instead of out of memory I saw that it used more memory and it logged the following when I switched the model and sent the prompt:

llama_new_context_with_model: max tensor size =   103.76 MB
llama.cpp: using Vulkan on NVIDIA RTX A2000 8GB Laptop GPU

I don't get anything when I gpt-send, it says querying GPT4All... and nothing else.

But now I see "Server chat" in GPT4All in the GUI with the strings I sent from gptel. With that context selected I can't actually switch to a different model in the GPT4All UI. I have to go to a new context in order to make that model selector dropdown functional. And I can initiate a new context in GPT4All ui without running out of memory.

I guess 8GB just isn't enough to use it this way.