Gradio compatibility

Knowledge Articles
1

Gradio is the fastest way easily create machine learning applications with a user-friendly web interface powered by SambaNova’s Inference API.

Installation

pip install sambanova-gradio==0.1.0

Basic Usage

Just like if you were to use the sambanova API, you should first save your sambanova API token to this environment variable:

export SAMBANOVA_API_KEY=<your token>

Then in a Python file, write:

import gradio as gr
import sambanova_gradio

gr.load(
    name='Meta-Llama-3.1-405B-Instruct',
    src=sambanova_gradio.registry,
).launch()

Run the Python file, and you should see a Gradio Interface connected to the model on sambanova!

image
image738×490 12.2 KB

Customization

Once you can create a Gradio UI from a sambanova endpoint, you can customize it by setting your own input and output components, or any other arguments to gr.Interface. For example, the screenshot below was generated with:

import gradio as gr
import sambanova_gradio

gr.load(
    name='Meta-Llama-3.1-405B-Instruct',
    src=sambanova_gradio.registry,
    title='Sambanova-Gradio Integration',
    description="Chat with Meta-Llama-3.1-405B-Instruct model.",
    examples=["Explain quantum gravity to a 5-year old.", "How many R are there in the word Strawberry?"]
).launch()

image
image756×498 12.4 KB

Composition

Or use your loaded Interface within larger Gradio Web UIs, e.g.

import gradio as gr
import sambanova_gradio

with gr.Blocks() as demo:
    with gr.Tab("405B"):
        gr.load('Meta-Llama-3.1-405B-Instruct', src=sambanova_gradio.registry)
    with gr.Tab("70B"):
        gr.load('Meta-Llama-3.1-70B-Instruct-8k', src=sambanova_gradio.registry)

demo.launch()
6 Likes