Replies: 4 comments 10 replies
-
You can post a request to load a different model.
I haven't yet found a way to ask the server which models are available. All learned from here: |
Beta Was this translation helpful? Give feedback.
-
See here for how listing models is used: Ex: def model_api(request):
response = requests.post(f'http://{HOST}/api/v1/model', json=request)
return response.json()
all_models = model_api({'action': 'list'})['result']
for m in all_models:
... |
Beta Was this translation helpful? Give feedback.
-
For those arriving here late to the game, and using API calls... In your CMD_FLAGS.txt ( or checked in Session => Boolean Command Line Flags )
Once running, these API calls allow you to check/list/modify models... To check the currently loaded model: To list available models: To load a different model: |
Beta Was this translation helpful? Give feedback.
-
@SalomonKisters anyone who comes across this thread. Here's an example from a small project you use for reference. Be sure to have the flags enabled that are mentioned above. import requests
class Oobabooga:
def __init__(self, ip='localhost', port='5000') -> None:
self.ip = ip
self.port = port
self.url = f'http://{self.ip}:{self.port}/v1/'
def get_models(self):
response = requests.get(self.url + 'internal/model/list')
return response.json()['model_names']
def load_model(self, args_json={}):
response = requests.post(self.url + 'internal/model/load/', json=args_json)
return response.json()
def unload_model(self, args_json={}):
response = requests.post(self.url + 'internal/model/unload/', json=args_json)
return response.json()
def get_model_info(self):
response = requests.get(self.url + 'internal/model/info')
return response.json()
def stop_generation(self):
response = requests.post(self.url + 'internal/stop-generation')
return response.json()
def chat_completion(self, args_json={}, stream=False):
if not stream:
response = requests.post(self.url + 'chat/completions', json=args_json)
return response.json()
else:
return requests.post(self.url + 'chat/completions',
headers={ "Content-Type": "application/json" },
json=args_json,
verify=False,
stream=True)
def completion(self, args_json={}, stream=False):
if not stream:
response = requests.post(self.url + 'completions', json=args_json)
return response.json()
else:
return requests.post(self.url + 'completions',
headers={ "Content-Type": "application/json" },
json=args_json,
verify=False,
stream=True) |
Beta Was this translation helpful? Give feedback.
-
Is it possible to change the model via the API? Also, is there an API docs page? I can't seem to find it, I don't have an API link at the bottom of my UI
Beta Was this translation helpful? Give feedback.
All reactions