You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The idea is to allow user to pass a list of grammar strings for a batch of inputs. And each grammar, input pair can be properly constrained, which is not supported in the current verison.
For the moment, regardless of batch size, the same batch can only be constrained with a single grammar object, which is equivalent to input-independent grammar.
In real world, function calls and json schema are great example where this is very useful. Now the two workarounds are:
use batch size = 1, which will be very inefficient for GPU
try to combine the grammars in a same batch to a more tolerent super set of each individual grammar, which is not already feasible and are relaxing the constraints.
To start this feature, a nice dataset is PennTree Bank where constraints can be input dependent.
The text was updated successfully, but these errors were encountered:
The idea is to allow user to pass a list of grammar strings for a batch of inputs. And each grammar, input pair can be properly constrained, which is not supported in the current verison.
For the moment, regardless of batch size, the same batch can only be constrained with a single grammar object, which is equivalent to input-independent grammar.
In real world, function calls and json schema are great example where this is very useful. Now the two workarounds are:
To start this feature, a nice dataset is PennTree Bank where constraints can be input dependent.
The text was updated successfully, but these errors were encountered: