LangChain compatibility?

[erenirmak]

Is it possible to integrate this model with the LangChain environment? Can it use LangChain tools?

[noshila]

Hey, any updates?

[khai-meetkai]

@noshila I think we can integrate this model with LangChain environment, as we can deploy functionary as an openai-compatible service. So any library, program, software that is using OpenAI can switch to using functionary as an alternative

[unfailingsalvage1448]

I tried it with vllm, using the langchain class ChatOpenAI, but got:

BadRequestError: Error code: 400 - {'object': 'error', 'message': "FunctionaryTokenizer.apply_chat_template() missing 1 required positional argument: 'tools'", 'type': 'BadRequestError', 'param': None, 'code': 400}

then I tried passing the tools as a model_kwargs "tools", but I get

BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted',

Not sure if you can fix this using the vanilla Langchain ChatOpenAI class. So you might need to develop a dedicated langchain class for this model.

Langchain is kind of esoteric when it comes to different model prompt templates, especially tool usage. I don't understand it at all. It would be cool if it worked the same as in the transformers library. But oc that is nothing you can fix.

[dwight-foster]

I have added functionary v3.1 small to ollama. You should be able to use the ollama integration in langchain to run the model and execute langchain functions. Here is the model link: https://ollama.com/dwightfoster03/functionary-small-v3.1. I have not tested it with langchain yet but it should work.

[jeffreymeetkai]

The template is wrong. We do not use this template in fine-tuning. We use the "User-defined Custom tool calling" format here with some adjustments on top of it.

We have looked at ollama integration but felt that it is not feasible currently as all our other models require converting json schema into Typescript schema. We will need to implement this rather complex conversion in the Go template syntax which may not be possible.

[dwight-foster]

I did not realize that thank you. I messed around with the prompt format this one just worked for me but I will add that to the readme. One question doesn’t 3.1 just use the same template as llama 3.1? That is why I chose it over 3.2 but I could be wrong.

[jeffreymeetkai]

Yes, it uses one of the templates in llama 3.1 but the documentation link shows that there are 2 prompt formats that can be used, either the tool definitions are in the user prompt or the system prompt (we chose the latter and made some more adjustments to it such as removing the built-in tools and date as llama3.1-instruct has been overfitted to calling those built-in tools).

All other Functionary models except for v3.1 uses our own prompt format which involves converting tool definitions from JSON schema to TS schema. I have managed to create working code for this in Jinja chat template as part of an effort to increase integration across a larger range of inference frameworks but ollama uses Go template syntax which I think is less powerful than Jinja. Thus, we are not sure if it is feasible to integrate all our models except v3.1 into ollama currently.