Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: --context-length max Option for Maximum Supported Context Length #118

Open
pkoloveas opened this issue Dec 4, 2024 · 0 comments

Comments

@pkoloveas
Copy link

pkoloveas commented Dec 4, 2024

Current Situation
When loading a model in LMS, users must manually specify the context length using the --context-length flag. If this flag is omitted, the context length defaults to the model's configuration value. However, there is no direct way to specify that the model should be loaded with its maximum supported context length.

This functionality is already available in the LMStudio GUI, where users can select the maximum supported token limit when loading a model. However, there is no equivalent option for command-line usage.

Proposed Solution
Introduce a --context-length max option, similar to the existing --gpu flag. This option would automatically set the context length to the maximum value supported by the model, based on its internal configuration.

Application and Benefits
This feature would be particularly useful in setups where models are loaded and unloaded iteratively for automated evaluations (e.g., via Python subprocess). Currently, users must manually determine and pass the maximum context length for each model, which is redundant (since LMS already has this information) and cumbersome (as it requires digging through model configs for every new model).

By implementing a max option, users could streamline workflows, reducing configuration overhead and enhancing usability. It would save time and simplify testing and evaluation pipelines involving multiple models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant