Open AI Compatibility

SambaNova Documentation SambaNova Cloud
1

Use the SambaNova Cloud API with OpenAI’s client libraries to easily upgrade your existing applications to use the fastest inference on the best open source models!

Use SambaNova APIs with OpenAI client libraries

Switching to SambaNova APIs with OpenAI’s client libraries is as simple as setting two values:

  1. Set the base_url to the “https://api.sambanova.ai/v1”.
  2. Set the api_key to the SambaNova Cloud API key.

    Don’t have an API key? Get yours from the SambaNova Cloud portal

Non-streaming example

The following code demonstrates using the OpenAI python client for non-streaming completions.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.sambanova.ai/v1", 
    api_key="YOUR SAMBANOVA API KEY"
)

response = client.chat.completions.create(
  model="Meta-Llama-3.1-8B-Instruct",
  messages=[
      {"role": "system", "content": "Answer the question in a couple sentences."},
      {"role": "user", "content": "Share a happy story with me"}
    ]
)

print(response.choices[0].message)

Streaming example

The following code demonstrates using the OpenAI python client for streaming completions.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.sambanova.ai/v1", 
    api_key="YOUR SAMBANOVA API KEY"
)

completion = client.chat.completions.create(
  model="Meta-Llama-3.1-8B-Instruct",
  messages = [
      {"role": "system", "content": "Answer the question in a couple sentences."},
      {"role": "user", "content": "Share a happy story with me"}
    ],
  stream= True
)

for chunk in completion:
  print(chunk.choices[0].delta)

Unsupported OpenAI features

The following features are not yet supported and will be ignored:

  • logprobs
  • top_logprobs
  • n
  • presence_penalty
  • frequency_penalty
  • logit_bias
  • tools
  • tool_choice
  • parallel_tool_calls
  • seed
  • stream_options: include_usage
  • response_format

Feature differences

temperature: The SambaNova Cloud API supports a value between 0 and 1.

Features unsupported by OpenAI clients

The SambaNova API supports the top_k parameter. This is not supported by the OpenAI client.

1 Like