The Phi 3 Vision Model has support in the Rust, Python, and HTTP APIs. The Phi 3 Vision Model supports ISQ for increased performance.
The Python and HTTP APIs support sending images as:
- URL
- Path to a local image
- Base64 encoded string
The Rust API takes an image from the image crate.
Note: The Phi 3 Vision model works best with one image although it is supported to send multiple images.
Note: when sending multiple images, they will be resized to the minimum dimension by which all will fit without cropping. Aspect ratio is not preserved in that case.
You can find this example here.
We support an OpenAI compatible HTTP API for vision models. This example demonstrates sending a chat completion request with an image.
Note: The image_url may be either a path, URL, or a base64 encoded string.
Image:
Prompt:
<|image_1|>\nWhat is shown in this image? Write a detailed response analyzing the scene.
Output:
The image captures a breathtaking view of a snow-covered mountain peak under a cloudy sky. The mountain, blanketed in pristine white snow, stands majestically against the backdrop of the sky. A winding road snakes its way up the side of the mountain, disappearing into the distance and adding a sense of scale to the scene. The road is flanked by trees on both sides, their branches heavy with snow. The perspective of the image is from a low angle, looking up at the mountain and giving it an imposing presence. Despite its grandeur, there's a sense of tranquility that pervades the scene - a testament to nature's quiet beauty.
- Start the server
cargo run --release --features ... -- --port 1234 vision-plain -m microsoft/Phi-3-vision-128k-instruct -a phi3v
- Send a request
import openai
openai.api_key = "EMPTY"
openai.base_url = "http://localhost:1234/v1/"
completion = openai.chat.completions.create(
model="phi3v",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://www.nhmagazine.com/content/uploads/2019/05/mtwashingtonFranconia-2-19-18-108-Edit-Edit.jpg"
},
},
{
"type": "text",
"text": "<|image_1|>\nWhat is shown in this image? Write a detailed response analyzing the scene.",
},
],
},
],
max_tokens=256,
frequency_penalty=1.0,
top_p=0.1,
temperature=0,
)
resp = completion.choices[0].message.content
print(resp)- You can find an example of encoding the image via base64 here.
- You can find an example of loading an image locally here.
You can find this example here.
This is a minimal example of running the Phi 3 Vision model with a dummy image.
use either::Either;
use image::{ColorType, DynamicImage};
use indexmap::IndexMap;
use std::sync::Arc;
use tokio::sync::mpsc::channel;
use mistralrs::{
Constraint, DefaultSchedulerMethod, Device, DeviceMapMetadata, MistralRs, MistralRsBuilder,
ModelDType, NormalRequest, Request, RequestMessage, Response, Result, SamplingParams,
SchedulerConfig, TokenSource, VisionLoaderBuilder, VisionLoaderType, VisionSpecificConfig,
};
/// Gets the best device, cpu, cuda if compiled with CUDA
pub(crate) fn best_device() -> Result<Device> {
#[cfg(not(feature = "metal"))]
{
Device::cuda_if_available(0)
}
#[cfg(feature = "metal")]
{
Device::new_metal(0)
}
}
fn setup() -> anyhow::Result<Arc<MistralRs>> {
// Select a Mistral model
let loader = VisionLoaderBuilder::new(
VisionSpecificConfig {
use_flash_attn: false,
repeat_last_n: 64,
},
None,
None,
Some("microsoft/Phi-3-vision-128k-instruct".to_string()),
)
.build(VisionLoaderType::Phi3V);
// Load, into a Pipeline
let pipeline = loader.load_model_from_hf(
None,
TokenSource::CacheToken,
&ModelDType::Auto,
&best_device()?,
false,
DeviceMapMetadata::dummy(),
None,
None, // No PagedAttention.
)?;
// Create the MistralRs, which is a runner
Ok(MistralRsBuilder::new(
pipeline,
SchedulerConfig::DefaultScheduler {
method: DefaultSchedulerMethod::Fixed(5.try_into().unwrap()),
},
)
.build())
}
fn main() -> anyhow::Result<()> {
let mistralrs = setup()?;
let (tx, mut rx) = channel(10_000);
let request = Request::Normal(NormalRequest {
messages: RequestMessage::VisionChat {
images: vec![DynamicImage::new(1280, 720, ColorType::Rgb8)],
messages: vec![IndexMap::from([
("role".to_string(), Either::Left("user".to_string())),
(
"content".to_string(),
Either::Left("<|image_1|>\nWhat is shown in this image? Write a detailed response analyzing the scene.".to_string()),
),
])],
},
sampling_params: SamplingParams::default(),
response: tx,
return_logprobs: false,
is_streaming: false,
id: 0,
constraint: Constraint::None,
suffix: None,
adapters: None,
});
mistralrs.get_sender()?.blocking_send(request)?;
let response = rx.blocking_recv().unwrap();
match response {
Response::Done(c) => println!("Text: {}", c.choices[0].message.content),
_ => unreachable!(),
}
Ok(())
}You can find this example here.
This example demonstrates loading and sending a chat completion request with an image.
Note: the image_url may be either a path, URL, or a base64 encoded string.
from mistralrs import Runner, Which, ChatCompletionRequest, VisionArchitecture
runner = Runner(
which=Which.VisionPlain(
model_id="microsoft/Phi-3-vision-128k-instruct",
arch=VisionArchitecture.Phi3V,
),
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="phi3v",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://upload.wikimedia.org/wikipedia/commons/e/e7/ Everest_North_Face_toward_Base_Camp_Tibet_Luca_Galuzzi_2006.jpg"
},
},
{
"type": "text",
"text": "<|image_1|>\nWhat is shown in this image? Write a detailed response analyzing the scene.",
},
],
}
],
max_tokens=256,
presence_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res.choices[0].message.content)
print(res.usage)- You can find an example of encoding the image via base64 here.
- You can find an example of loading an image locally here.