Introduce outlines.models.transformers_multimodal #33
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lapp0
commented
Jun 19, 2024
| def __call__( # type: ignore | ||
| self, | ||
| prompts: Union[str, List[str]], | ||
| media: Union[str, Any], |
lapp0
force-pushed
the
multimodal-models
branch
2 times, most recently
from
11c0db1 to
a6c229e
Compare
|
Let’s call it |
lapp0
force-pushed
the
fix-mamba-integration
branch
from
f17913b to
48b6f8f
Compare
rlouf
force-pushed
the
fix-mamba-integration
branch
from
48b6f8f to
bf3694c
Compare
lapp0
force-pushed
the
fix-mamba-integration
branch
8 times, most recently
from
75dc370 to
acb0759
Compare
lapp0
force-pushed
the
multimodal-models
branch
4 times, most recently
from
bdf4097 to
653fe26
Compare
rlouf
reviewed
Jul 19, 2024
outlines/generate/api.py
Outdated
| return prompts, media | ||
|
|
||
| @classmethod | ||
| def _load_media(cls, media): |
There was a problem hiding this comment.
I'm not sure this should be part of the library?
There was a problem hiding this comment.
Probably not, it's a convenience, but unnecessary. Removing.
| from outlines.processors import OutlinesLogitsProcessor | ||
|
|
||
|
|
||
| class TransformersMultiModal(Transformers): |
There was a problem hiding this comment.
Suggested change
| class TransformersMultiModal(Transformers): | |
| class TransformersVision(Transformers): |
| yield self._decode_generation(output_group_ids) | ||
|
|
||
|
|
||
| def transformers_multimodal( |
There was a problem hiding this comment.
Suggested change
| def transformers_multimodal( | |
| def transformers_vision( |
lapp0
force-pushed
the
multimodal-models
branch
9 times, most recently
from
9cc0775 to
56b5918
Compare
lapp0
force-pushed
the
multimodal-models
branch
17 times, most recently
from
6adb73b to
9ae6e70
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Docs: https://github.com/lapp0/outlines/blob/multimodal-models/docs/reference/models/multimodal.md
Done:
Todo:
MultiModalSequenceGeneratorAdaptertest_generate.py, current model is too expensive to be part of test suite.llava-hf/llava-v1.6-mistral-7b-hfImprove docs:
<image>token in prompt