Support for JSON mode?

SambaNova Devs Discussion
1

Hello everyone,

I’m truly enthusiastic about your product; the inference speed and ability to host an extremely large model like Llama 405B is absolutely stellar!

I’m currently developing a platform that allows users to generate stories for children’s books. The platform features input and output validation using fast, lightweight LLMs. Right now, I’m working with Llama 3.1 70B on Groq, but I would like to test it with Llama 4.0 5B using SambaNova. However, it seems that SambaNova doesn’t support JSON mode at this time.

While I can prompt for JSON-structured output, the consistency of the output is not sufficient for application development. Sometimes the output starts as plain JSON, other times it includes leading text and special characters, or it may not adhere to the correct JSON structure throughout. For example, I’m requesting a JSON object structured as a list of 17 dictionaries, each containing 2 items like this:

[
  {
    "id": 0,
    "content": "this content part is usually around 3-4 sentences"
  },
  {
    "id": 1,
    "content": "this content part is usually around 3-4 sentences"
  }
]

I could implement custom parsing for the output to replicate the desired format, but this would require significant code refactoring. Ideally, I’d prefer to swap the models for testing without making those changes.

Do you have any suggestions for a solid workaround? I find the output structure too unstable for reliable custom parsing. Additionally, is there a roadmap or an intended support date for output structure or JSON mode?

Thank you!

2 Likes
3

@brnfnk I have submitted an ER for this and as soon as I have it verified for the roadmap I will communicate more. In the mean time have you reviewed the workaround in post :

-Coby

1 Like
4

Hi Coby, thanks for your reply.

I am using a similar workaround now where I filter the result between ‘[’ and ‘]’ before validating. It works around 90% of the times but a more robust solution like the output structure would be very welcome.

1 Like
5

I definitely understand the reason for your ask and will discuss with the engineering teams .

-Coby

6

@brnfnk Hope this suggestion helps, creating a list of 10 dictionaries containing 2 items in it.

from openai import OpenAI
import json

client = OpenAI(
    base_url="https://api.sambanova.ai/v1", 
    api_key="YOUR_API_KEY"
)

def get_response(prompt,query):
  response = client.chat.completions.create(
    model="Meta-Llama-3.1-405B-Instruct",
    messages=[
        {"role": "system", "content": prompt},
        {"role": "user", "content": query}])
  return response.choices[0].message.content

def post_process(resp):
  # split response based on comma
  items = resp.split(',')
  # Combine every two items together
  split_list = [', '.join(items[i:i+2]).strip() for i in range(0, length(items), 2)]
  # list of dict
  output = [json.loads(split_list[i]) for i in range(length(split_list))]
  return output

# Define the prompt for JSON output
prompt = '''Provide the information in below list format. Do not provde any extra text.
{"id":1,"Country":"ABC"},{"id":2,"Country":"XYZ"}
'''

query="give 10 country names starting with E in given format"
print("Query: ",query)

resp = get_response(prompt,query)
resp = resp.replace('\n','') # sometimes \n is present in response
print("Response: ",resp)

output = post_process(resp)

print('List of Dict:',type(output))
print(type(output[0]))

Output:

Query:  give 10 country names starting with E in given format
Response:  {"id":1,"Country":"Ecuador"},{"id":2,"Country":"Egypt"},{"id":3,"Country":"El Salvador"},{"id":4,"Country":"Equatorial Guinea"},{"id":5,"Country":"Eritrea"},{"id":6,"Country":"Estonia"},{"id":7,"Country":"Eswatini"},{"id":8,"Country":"Ethiopia"},{"id":9,"Country":"East Timor"},{"id":10,"Country":"England"}
List of Dict: <class 'list'>
<class 'dict'>
1 Like