Understanding the Pixtral-12b-2409 model

Model overview

Model name

mistral/pixtral-12b-2409:bf16

Compatible Instances

Model introduction

Pixtral is a vision language model introducing a novel architecture: 12B parameter multimodal decoder plus 400M parameter vision encoder. It can analyze images and offer insights from visual content alongside text. This multimodal functionality creates new opportunities for applications that need both visual and textual comprehension.

Pixtral is open-weight and distributed under the Apache 2.0 license.

Why is it useful?

• Pixtral allows you to process real world and high resolution images, unlocking capacities such as transcribing handwritten files or payment receipts, extracting information from graphs, captioning images, etc.
• It offers large context window of up to 128k tokens, particularly useful for RAG applications
• Pixtral supports variable image sizes and types: PNG (.png), JPEG (.jpeg and .jpg), WEBP (.webp), as well as non-animated GIF with only one frame (.gif)

How to use it

Sending Inference requests

To perform inference tasks with your Pixtral model deployed at Scaleway, use the following command:

curl -s \
-H "Authorization: Bearer <IAM API key>" \
-H "Content-Type: application/json" \
--request POST \
--url "https://<Deployment UUID>.ifr.fr-par.scw.cloud/v1/chat/completions" \
--data '{
       "model": "mistral/pixtral-12b-2409:bf16",
       "messages": [
        {
          "role": "user",
          "content": [
              {"type" : "text", "text": "Describe this image in detail please."},
              {"type": "image_url", "image_url": {"url": "https://picsum.photos/id/32/512/512"}},
              {"type" : "text", "text": "and this one as well."},
              {"type": "image_url", "image_url": {"url": "https://www.wolframcloud.com/obj/resourcesystem/images/a0e/a0ee3983-46c6-4c92-b85d-059044639928/6af8cfb971db031b.png"}}
          ]
        }
       ],
       "top_p": 1, 
       "temperature": 0.7, 
       "stream": false
}'

Make sure to replace <IAM API key> and <Deployment UUID> with your actual IAM API key and the Deployment UUID you are targeting.

Passing images to Pixtral

1. Image URLs If the image is available online, you can just include the image URL in your request as demonstrated above. This approach is simple and does not require any encoding.

2. Base64 encoded image Base64 encoding is a standard way to transform binary data, like images, into a text format, making it easier to transmit over the internet.

The following Python code sample shows you how to encode an image in base64 format and pass it to your request payload.

import base64
from io import BytesIO
from PIL import Image
def encode_image(img):
    buffered = BytesIO()
    img.save(buffered, format="JPEG")
    encoded_string = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return encoded_string
img = Image.open("path_to_your_image.jpg")
base64_img = encode_image(img)
payload = {
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"},
                },
            ],
        }
    ],
    ... # other parameters
}

Receiving Managed Inference responses

Upon sending the HTTP request to the public or private endpoints exposed by the server, you will receive inference responses from the managed Managed Inference server. Process the output data according to your application’s needs. The response will contain the output generated by the visual language model based on the input provided in the request.

Frequently Asked Questions

What types of images are supported by Pixtral?

• Bitmap (or raster) image formats, meaning storing images as grids of individual pixels, are supported: PNG, JPEG, WEBP, and non-animated GIFs in particular.
• Vector image formats (SVG, PSD) are not supported.

Are other files supported?

Only bitmaps can be analyzed by Pixtral, PDFs and videos are not supported.

Is there a limit to the size of each image?

The only limitation is in context window (1 token for each 16x16 pixel).

What is the maximum amount of images per conversation?

One conversation can handle up to 12 images (per request). The 13rd will return a 413 error.