Access to Vision Models

SambaNova Devs Hackathons
1

In the Pricing page in SambaNova cloud, I can see vision models are listed. But unable to access them in the playground. Do we get access for it in the APIs?

image
image1454×946 58.5 KB

Thanks

1 Like
2

Hi @notagodzilla

Well done on spotting this! We are in the process of rolling these out to use in the SambaNova Cloud & you should see this on the playground now!

Kind Regards

2 Likes
3

Hey @alex.penketh. Thanks for the clarification. Is the api available for use right now? I am using Vercel’s AI SDK in my project and when I integrated vision into my code base I am getting this error in the body.

'{"error":{"code":null,"message":"unexpected_error","param":null,"type":"unexpected_error"}}

Thanks!

1 Like
4

@notagodzilla Yes it should be working

this would be an example code

import os
import openai

client = openai.OpenAI(
    api_key=os.environ.get("SAMBANOVA_API_KEY"),
    base_url="https://api.sambanova.ai/v1",
)

response = client.chat.completions.create(
    model='Llama-3.2-90B-Vision-Instruct',
    messages=[{"role":"user","content":[{"type":"text","text":"."},{"type":"image_url","image_url":"data:image/webp;base64,<base64image>"}]},{"role":"system","content":[{"type":"text","text":"Hello there!\n\nHere is a concise summary of the image: A collection of gold coins is displayed on a wooden table, with some coins in plastic cases and others loose.\n\nIn the foreground, the gold coins are the main focus, with some placed in clear plastic cases and others scattered around them. The coins feature various designs and inscriptions, including the words \"LIBERTY\" and \"AMERICA.\" In the background, there are additional items on the table, including a small wooden box with a gold coin inside and a plastic case containing more coins. The overall atmosphere suggests a collector's display or a store selling gold coins."}]}],
    temperature =  0.1,
    top_p = 0.1
)
1 Like
5

Edit: Figured it out. Should be sending base64 images as image_url:{url: imageUrl } rather than directly pushing the base64 string.

I am using the openai npm package. I am still getting an unexpected_error from the api when attaching an image. Same model works fine if there is no image attached.

Error Message:

  error: {
    code: null,
    message: 'unexpected_error',
    param: null,
    type: 'unexpected_error'
  }
}

My Implementation:

    let {messages, model} : {messages: ChatCompletionMessageParam[], model: String, isVision: boolean} = await request.json();
    console.log(model)

    const openai = new OpenAI({
      baseURL: process.env.SAMBANOVA_BASE_URL,
      apiKey: process.env.SAMBANOVA_API_KEY,
    });
    const systemMessage: ChatCompletionMessageParam = {role: 'assistant', content: [{type: 'text', text: dedent(GENERATE_PROMPT)}]}
    messages = [systemMessage, ...messages]


    const completion = await openai.chat.completions.create({
      model: model as string,
      messages: messages,
    });

Checked this implementation with the OpenAI endpoint and it works fine. Are there any limitation to the image size and format?

Thanks

3 Likes
6

Hi @notagodzilla

I’m glad you found the solution to this, I am working with our internal teams to check the provided example to make sure this is helpful and correct moving forward!

In regards to the image limitations, as of writing this post, you are allowed the following file formats:
.jpg .jpeg .png & .webp up to 20MB in size.

If this changes in the future it will be announced in the release notes.

Kind Regards

2 Likes
7

More information regarding input format for the image payload is here sambanova_cloud_api_reference

Thanks & Regards

3 Likes