Plugins issue!

[soymh]

Hi dear developer[s] !

thanks for your great work ! I appreciate the hard work behind this useful project and your creative ideas!

Two things are in the project that I wanted to discuss:

1. The plugins do not work at all! something is wring in integrating them(using them) by the request sent to the API. I have no idea what is that.
2. That would be great if two things(yet now, maybe better ideas come after these!) to implement into this great project:

1st

The bot would be so much more functional if you implement a way the bot can comminucate with Together.ai or Openrouter.ai APIs .

2nd

If you provide an interface/way to integrate our own plugins through the bot! THIS WOULD BE A HUGE UPDATE!

BUT before this , the issue of the plugins have to be fixed , and I have no idea where does that come from!

Here is my .env structure;also I gotta admit that I have made some changes to the project to make it functional with Together.ai API:

# Your OpenAI API key
OPENAI_API_KEY=xxx

# Your Telegram bot token obtained using @BotFather
TELEGRAM_BOT_TOKEN=xxx

# Telegram user ID of admins, or - to assign no admin
ADMIN_USER_IDS=xxx

# Comma separated list of telegram user IDs, or * to allow all
ALLOWED_TELEGRAM_USER_IDS=xxx

# Optional configuration, refer to the README for more details
# BUDGET_PERIOD=monthly
# USER_BUDGETS=*
# GUEST_BUDGET=100.0
# TOKEN_PRICE=0.002
# IMAGE_PRICES=0.016,0.018,0.02
# TRANSCRIPTION_PRICE=0.006
# VISION_TOKEN_PRICE=0.01
# ENABLE_QUOTING=true
# ENABLE_IMAGE_GENERATION=true
# ENABLE_TTS_GENERATION=true
# ENABLE_TRANSCRIPTION=true
# ENABLE_VISION=true
# PROXY=http://localhost:8080
OPENAI_MODEL=meta-llama/Llama-3.3-70B-Instruct-Turbo
OPENAI_BASE_URL=https://api.together.xyz/v1
# OPENROUTER_FREE_ONLY=true
# ASSISTANT_PROMPT="You are a helpful assistant."
SHOW_USAGE=true
STREAM=false
MAX_TOKENS=1200
# VISION_MAX_TOKENS=300
# MAX_HISTORY_SIZE=15
# MAX_CONVERSATION_AGE_MINUTES=180
# VOICE_REPLY_WITH_TRANSCRIPT_ONLY=true
# VOICE_REPLY_PROMPTS="Hi bot;Hey bot;Hi chat;Hey chat"
# VISION_PROMPT="What is in this image"
# N_CHOICES=1
# TEMPERATURE=1.0
# PRESENCE_PENALTY=0.0
# FREQUENCY_PENALTY=0.0
# FLUX_TAI=true
IMAGE_MODEL=black-forest-labs/FLUX.1-dev
# IMAGE_QUALITY=hd
# IMAGE_STYLE=natural
# IMAGE_SIZE=1024x1024
# IMAGE_FORMAT=document
# VISION_DETAIL="low"
# GROUP_TRIGGER_KEYWORD=""
# IGNORE_GROUP_TRANSCRIPTIONS=true
# IGNORE_GROUP_VISION=true
# TTS_MODEL="tts-1"
# TTS_VOICE="alloy"
# TTS_PRICES=0.015,0.030
# BOT_LANGUAGE=en
# ENABLE_VISION_FOLLOW_UP_QUESTIONS="true"
VISION_MODEL="gpt-4o"
ENABLE_FUNCTIONS=true
PLUGINS=weather,wolfram,ddg_web_search,ddg_image_search,crypto,worldtimeapi,dice,youtube_audio_extractor,gtts_text_to_speech,whois,webshot,auto_tts
SHOW_PLUGINS_USED=true
WOLFRAM_APP_ID=xxx
WORLDTIME_DEFAULT_TIMEZONE=Asia/xxx
DUCKDUCKGO_SAFESEARCH=off

---

AND YES, I checked the plugins functionality before I make any changes to the project.before together.ai implementation the issue was still there!

Also I didn't make any changes in the way plugins are handled.

Thanks in advanced!

Happy New Year!

[n3d1117]

Hi @soymh and happy new year! I see you're using meta-llama/Llama-3.3-70B-Instruct-Turbo model. Plugins are based on OpenAI's function calling feature which is only available in OpenAI models. That's probably why they're not working in your setup

[soymh]

thank you so much!

I said that before making any changes for implementing together.ai , I tested the plugins and they weren't working.

Also I had to mention the model once then i used open-ai was gpt-4o-mini; sorry.

thanks again for helping

EDIT: since I haven't checked the plugins python files (yet), are there any specific line which tells the plugins are available for which model?
Even in this case I have tried gpt-4o-mini and plugins didn't work.

[n3d1117]

@soymh That's odd. I suggest you start with a super simple .env like this:

OPENAI_MODEL=gpt-4o-mini
PLUGINS=weather

and that should definitely work if you use the latest version. I just tested it with a "what's the weather tomorrow in Copenhagen?" question, and in the logs I can see the bot is using weather plugin correctly:

2024-12-31 12:01:10,070 - root - INFO - Calling function get_forecast_weather with arguments {"latitude":"55.6761","longitude":"12.5683","unit":"celsius","forecast_days":2}

are there any specific line which tells the plugins are available for which model?

All plugins are available for all models, as long as the model supports function calling

[soymh]

thanks, I tested that just now, exactly as you mentioned
nothing's changed
using a proxy for docker would be an issue ?
except that I don't know what/where is the problem...
sorry to bother, what version of the bot did you tested with? cause the issue maybe with latest changes

[n3d1117]

Do you see any error or is it just not calling any plugin?

Also make sure you're up to date with the latest version of this repo (main branch) as support for 4o models was only added a few days ago

[soymh]

It's not just calling the function:

chatgpt-telegram-bot-1  | 2024-12-31 14:53:58,783 - root - INFO - New message received from user @xxx (id: xxx)

and nothing else! no function calling!
Even I set the :

function_call = {
                        "name": "get_current_weather",
                        "arguments": json.dumps({
                            "latitude": "40.7128",   # Example latitude (New York City)
                            "longitude": "-74.0060", # Example longitude (New York City)
                            "unit": "celsius"        # Specify the temperature unit
                        })
                    }
                    common_args['function_call'] = function_call

in the openai_helper.py to force it using weather function ; BUT nothing happens!

Also make sure you're up to date with the latest version of this repo (main branch) as support for 4o models was only added a few days ago

Funny thing is I used this project about 6 months ago and it worked great with gpt-4o-mini !

(at that time there was no official gpt-4o support and you had to tweak it manually).

And now it doesn't work...

[n3d1117]

Can you check if your bot ever reaches this line? Like add a print statement in there or something. If it does, then I'm out of ideas 😅 but if it doesn't, we can go deeper to find the issue

[soymh]

I'm so sorry, Unfortunately it does 😆
I was able to get full common_args :

{'model': 'gpt-4o-mini', 'messages': [{'role': 'system', 'content': 'You are a helpful assistant.'}, {'role': 'user', 'content': "hi\nhow's the weather in Copenhagen?"}], 'temperature': 1.0, 'n': 1, 'max_tokens': 1200, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stream': False, 
'functions': [{'name': 'get_current_weather', 'description': 'Get the current weather for a location using Open Meteo APIs.', 
'parameters': {'type': 'object', 'properties': {'latitude': {'type': 'string', 'description': 'Latitude of the location'}, 'longitude': {'type': 'string', 'description': 'Longitude of the location'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit'],
 'description': 'The temperature unit to use. Infer this from the provided location.'}}, 'required': ['latitude', 'longitude', 'unit']}}, {'name': 'get_forecast_weather', 'description': 'Get daily weather forecast for a location using Open Meteo APIs.Today is Tuesday, December 31, 2024', 'parameters': {'type': 'object', 'properties': {'latitude': {'type': 'string', 'description': 'Latitude of the location'}, 'longitude': {'type': 'string', 
'description': 'Longitude of the location'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the provided location.'}, 'forecast_days': {'type': 'integer',
 'description': 'The number of days to forecast, including today. Default is 7. Max 14. Use 1 for today, 2 for today and tomorrow, and so on.'}}, 'required': ['latitude', 'longitude', 'unit', 'forecast_days']}}, 
{'name': 'answer_with_wolfram_alpha', 'description': 'Get an answer to a question using Wolfram Alpha. Input should the the query in English.', 'parameters': {'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'The search query, in english (translate if necessary)'}}, 'required': ['query']}}, 
{'name': 'web_search', 'description': 'Execute a web search for the given query and return a list of results', 'parameters': {'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'the user query'}, 'region': {'type': 'string', 'enum': ['xa-ar', 'xa-en', 'ar-es', 'au-en', 'at-de', 'be-fr', 'be-nl', 'br-pt', 'bg-bg', 'ca-en', 'ca-fr', 'ct-ca', 'cl-es', 'cn-zh', 'co-es', 'hr-hr', 'cz-cs', 'dk-da', 'ee-et', 'fi-fi', 'fr-fr', 'de-de', 'gr-el', 'hk-tzh', 'hu-hu', 'in-en', 'id-id', 'id-en', 'ie-en', 'il-he', 'it-it', 'jp-jp', 'kr-kr', 'lv-lv', 'lt-lt', 'xl-es', 'my-ms', 'my-en', 'mx-es', 'nl-nl', 'nz-en', 'no-no', 'pe-es', 'ph-en', 'ph-tl', 'pl-pl', 'pt-pt', 'ro-ro', 'ru-ru', 'sg-en', 'sk-sk', 'sl-sl', 'za-en', 'es-es', 'se-sv', 'ch-de', 'ch-fr', 'ch-it', 'tw-tzh', 'th-th', 'tr-tr', 'ua-uk', 'uk-en', 'us-en', 'ue-es', 've-es', 'vn-vi', 'wt-wt'], 'description': 'The region to use for the search. Infer this from the language used for thequery. Default to `wt-wt` if not specified'}}, 'required': ['query', 'region']}},
 {'name': 'search_images', 'description': 'Search image or GIFs for a given query', 'parameters': {'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'The query to search for'}, 'type': {'type': 'string', 'enum': ['photo', 'gif'], 'description': 'The type of image to search for. Default to `photo` if not specified'}, 'region': {'type': 'string', 'enum': ['xa-ar', 'xa-en', 'ar-es', 'au-en', 'at-de', 'be-fr', 'be-nl', 'br-pt', 'bg-bg', 'ca-en', 'ca-fr', 'ct-ca', 'cl-es', 'cn-zh', 'co-es', 'hr-hr', 'cz-cs', 'dk-da', 'ee-et', 'fi-fi', 'fr-fr', 'de-de', 'gr-el', 'hk-tzh', 'hu-hu', 'in-en', 'id-id', 'id-en', 'ie-en', 'il-he', 'it-it', 'jp-jp', 'kr-kr', 'lv-lv', 'lt-lt', 'xl-es', 'my-ms', 'my-en', 'mx-es', 'nl-nl', 'nz-en', 'no-no', 'pe-es', 'ph-en', 'ph-tl', 'pl-pl', 'pt-pt', 'ro-ro', 'ru-ru', 'sg-en', 'sk-sk', 'sl-sl', 'za-en', 'es-es', 'se-sv', 'ch-de', 'ch-fr', 'ch-it', 'tw-tzh', 'th-th', 'tr-tr', 'ua-uk', 'uk-en', 'us-en', 'ue-es', 've-es', 'vn-vi', 'wt-wt'], 'description': 'The region to use for the search. Infer this from the language used for thequery. Default to `wt-wt` if not specified'}}, 'required': ['query', 'type', 'region']}}, 
{'name': 'get_crypto_rate', 'description': 'Get the current rate of various crypto currencies', 'parameters': {'type': 'object', 'properties': {'asset': {'type': 'string', 'description': 'Asset of the crypto'}}, 'required': ['asset']}}, 
{'name': 'worldtimeapi', 'description': 'Get the current time from a given timezone', 'parameters': {'type': 'object', 'properties': {'timezone': {'type': 'string', 'description': 'The timezone identifier (e.g: `Europe/Rome`). Infer this from the location.Use Asia/Tehran if not specified.'}},'required': ['timezone']}}, 
{'name': 'send_dice', 'description': 'Send a dice in the chat, with a random number between 1 and 6', 'parameters': {'type': 'object', 'properties': {'emoji': {'type': 'string', 'enum': ['🎲', '🎯', '🏀', '⚽', '🎳', '🎰'], 'description': 'Emoji on which the dice throw animation is based.Dice can have values 1-6 for “🎲”, “🎯” and “🎳”, values 1-5 for “🏀” and “⚽”, and values 1-64 for “🎰”. Defaults to “🎲”.'}}}},
{'name': 'extract_youtube_audio', 'description': 'Extract audio from a YouTube video', 'parameters': {'type': 'object', 'properties': {'youtube_link': {'type': 'string', 'description': 'YouTube video link to extract audio from'}}, 'required': ['youtube_link']}},
{'name': 'google_translate_text_to_speech', 'description': "Translate text to speech using Google Translate's Text to Speech API", 'parameters': {'type': 'object', 'properties': {'text': {'type': 'string', 'description': 'The text to translate to speech'}, 'lang': {'type': 'string', 'description': 'The language of the text to translate to speech.Infer this from the language of the text.'}}, 'required': ['text', 'lang']}},
{'name': 'get_whois', 'description': 'Get whois registration and expiry information for a domain', 'parameters': {'type': 'object', 'properties': {'domain': {'type': 'string', 'description': 'Domain name'}}, 'required': ['domain']}},
{'name': 'screenshot_website', 'description': 'Show screenshot/image of a website from a given url or domain name.', 'parameters': {'type': 'object', 'properties': {'url': {'type': 'string', 'description': 'Website url or domain name. Correctly formatted url is required. Example: https://www.google.com'}}, 'required': ['url']}},
{'name': 'translate_text_to_speech', 'description': 'Translate text to speech using OpenAI API', 'parameters': {'type': 'object', 'properties': {'text': {'type': 'string', 'description': 'The text to translate to speech'}}, 'required': ['text']}}], 'function_call': {'name': 'get_current_weather', 'arguments': '{"latitude": "40.7128", "longitude": "-74.0060", "unit": "celsius"}'}}

Maybe the issue is elsewhere.
Thanks a lot for your help! I'll handle this from here!
Thanks for your great project!

Happy New Year!