Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for input dependent grammar #14

Open
Saibo-creator opened this issue Feb 29, 2024 · 0 comments
Open

add support for input dependent grammar #14

Saibo-creator opened this issue Feb 29, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@Saibo-creator
Copy link
Collaborator

Saibo-creator commented Feb 29, 2024

The idea is to allow user to pass a list of grammar strings for a batch of inputs. And each grammar, input pair can be properly constrained, which is not supported in the current verison.

For the moment, regardless of batch size, the same batch can only be constrained with a single grammar object, which is equivalent to input-independent grammar.

In real world, function calls and json schema are great example where this is very useful. Now the two workarounds are:

  1. use batch size = 1, which will be very inefficient for GPU
  2. try to combine the grammars in a same batch to a more tolerent super set of each individual grammar, which is not already feasible and are relaxing the constraints.

To start this feature, a nice dataset is PennTree Bank where constraints can be input dependent.

@Saibo-creator Saibo-creator self-assigned this Feb 29, 2024
@Saibo-creator Saibo-creator added the enhancement New feature or request label Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant