Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check for supported file formats when displaying available inference engines #106

Open
dadmobile opened this issue Jun 10, 2024 · 1 comment

Comments

@dadmobile
Copy link
Member

dadmobile commented Jun 10, 2024

Specifically MLX only supports some weight file formats (safetensors and nfz I think?). We currently only check architecture which means you sometimes get a "No safetensors for..." error when trying to run a model with MLX.

There are several possible ways to address:

  • check weight file formats (currently there is a formats array in model gallery)
  • use allow_patterns to see if there's a supported file type (lots of reasons this might not work)
  • take advantage if we add MLX field to models that says minimum version required to support like transformers (separate issue)
  • create some other way for a plugin to take a model and return if it's supported (kind of ugly but more flexible)
@dadmobile
Copy link
Member Author

More users have hit: #187

We tried options that added a field to gallery models that sort of worked, but most users are importing models either from their drive or HuggingFace. So we will need to figure out how to detect this automatically in the code that generates the list of possible engines (and then possibly also needs to show somewhere why, for example, MLX isn't available if it is a standard Hugging Face modeal and you are on a Mac?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant