Do you have a blacklist for your API endpoint? My dev environment is getting timed out all of a sudden, while my staging environment is not at all, working perfectly, same API key.
1 Like
There is no block list that would block one environment over the other . Which model are you accessing and what is the error returned form your API call?
-Coby
Looking at your calls I do see two calls that had 716ms TTFT and one call with a 20 second total latency ( for a larger return ).
-Coby
I am seeing this as well. I am using the DeepSeek-R1-Distill-Llama-70B model. The request just times out. It seems to reject my server’s TLS handshake (though I can get other https sites like google or openai without issue). On my local machine it’s just fine
Welcome to the community I am sorry you are having issues. Can you put in your call and the error /time you are getting please do not include your API key.
Also may I ask what region your call is coming from?
-Coby
client = httpx.AsyncClient()
url = “https://api.sambanova.ai/v1/chat/completions”
headers = {
“Content-Type”: “application/json”,
“Authorization”: f"Bearer {API Key}"
}
data = {
“model”: model_name,
“max_tokens”: 4000,
“messages”: [
{
“role”: “user”,
“content”: prompt
}
]
}
response = await client.post(url, headers=headers, json=data, timeout=None)
The model name is set to DeepSeek-R1-Distill-Llama-70B. I’m in the US-East region
Hey n.aelick
Can you take a peek at the following and see if this helps?
import asyncio
import httpx
async def main():
print("Welcome to the Sambanova AI Chat Completion API!")
print("Please enter your API key:")
api_key = input()
print("Please enter the model name:")
model_name = input()
print("Please enter your prompt:")
prompt = input()
client = httpx.AsyncClient()
url = "https://api.sambanova.ai/v1/chat/completions"
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}"
}
data = {
"model": model_name,
"max_tokens": 4000,
"messages": [
{
"role": "user",
"content": prompt
}
]
}
try:
response = await client.post(url, headers=headers, json=data, timeout=None)
response.raise_for_status()
print("Response:")
print(response.json())
except httpx.HTTPError as e:
print(f"Error: {e}")
asyncio.run(main())
python '/Users/sethk/http_async_test.py'
Welcome to the Sambanova AI Chat Completion API!
Please enter your API key:
{REDACTED}
Please enter the model name:
DeepSeek-R1-Distill-Llama-70B
Please enter your prompt:
How many R's in ARRRRRRRGH!!!
Response:
{'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '<think>\nTo determine the number of R\'s in "ARRRRRRRGH!!!", I will start by examining each character in the string.\n\nFirst, I\'ll identify the individual letters: A, R, R, R, R, R, R, R, G, H, and then the exclamation marks.\n\nNext, I\'ll count how many times the letter R appears in this sequence.\n\nAfter counting, I find that there are 7 R\'s in the given string.\n</think>\n\nTo determine the number of **R\'s** in the string **ARRRRRRRGH!!!**, follow these steps:\n\n1. **Write Down the String:**\n \n ```\n A R R R R R R R G H !!!\n ```\n\n2. **Identify and Count the R\'s:**\n \n - **A**\n - **R** (1)\n - **R** (2)\n - **R** (3)\n - **R** (4)\n - **R** (5)\n - **R** (6)\n - **R** (7)\n - **G**\n - **H**\n - **!!!**\n\n3. **Total Number of R\'s:**\n \n There are **7** R\'s in the string.\n\n\\[\n\\boxed{7}\n\\]', 'role': 'assistant'}}], 'created': 1740844825, 'id': 'd982cae8-ea0a-4193-a472-eab2424184ee', 'model': 'DeepSeek-R1-Distill-Llama-70B', 'object': 'chat.completion', 'system_fingerprint': 'fastcoe', 'usage': {'acceptance_rate': 4.25, 'completion_tokens': 273, 'completion_tokens_after_first_per_sec': 368.41804548184797, 'completion_tokens_after_first_per_sec_first_ten': 370.2598870056497, 'completion_tokens_per_sec': 345.7123665361163, 'end_time': 1740844825.5102913, 'is_last_response': True, 'prompt_tokens': 14, 'start_time': 1740844824.7196457, 'stop_reason': 'stop', 'time_to_first_token': 0.052353858947753906, 'total_latency': 0.7896738052368164, 'total_tokens': 287, 'total_tokens_per_sec': 363.44120584566076}}
Thanks!
Seth Kneeland
I get the same result as I get with my script. On my local desktop, the script runs fine. On my server, the same script waits forever (because of the timeout=None) until I end it. The issue seems to be with the TLS handshake
Can you curl to the endpoint from your server ? We have many people accessing from AWS vms.
If you can run a curl do it with -v for our endpoint and an endpoint you know works.
-coby
To sambanova:
curl -v https://api.sambanova.ai/v1/chat/completions
* Trying 34.46.226.125:443...
* connect to 34.46.226.125 port 443 failed: Connection timed out
* Failed to connect to api.sambanova.ai port 443 after 129592 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to api.sambanova.ai port 443 after 129592 ms: Couldn't connect to server
To openai:
curl -v https://api.openai.com/v1/models
* Trying 162.159.140.245:443...
* Connected to api.openai.com (162.159.140.245) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=api.openai.com
* start date: Jan 22 23:17:53 2025 GMT
* expire date: Apr 23 00:17:50 2025 GMT
* subjectAltName: host "api.openai.com" matched cert's "api.openai.com"
* issuer: C=US; O=Google Trust Services; CN=WE1
* SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /v1/models]
* h2h3 [:scheme: https]
* h2h3 [:authority: api.openai.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x563b1d03bf00)
> GET /v1/models HTTP/2
> Host: api.openai.com
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 401
< date: Sat, 01 Mar 2025 21:14:15 GMT
< content-type: application/json
< content-length: 151
< www-authenticate: Bearer realm="OpenAI API"
< openai-version: 2020-10-01
< x-request-id: 57beecfb92709ae9c692990b4ac3959f
< openai-processing-ms: 2
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< cf-cache-status: DYNAMIC
< set-cookie: __cf_bm=xIcK816WryXNGQ6uV0sv4ssB.bzPIP5D5r75paeGk5c-1740863655-1.0.1.1-fbSK4.y_Q.DTTXBh_MPqzvt60S668G61DR0PYYaWnOELKTWWWGLzP4aqvx9gaBHW_Obfhbvd0iUzKkqg_zuc5BEwh1EIALDvaBBMjYtfIbU; path=/; expires=Sat, 01-Mar-25 21:44:15 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
< x-content-type-options: nosniff
< set-cookie: _cfuvid=Q3yF3D_5ey1YYKrw4mIXWlLdiIJ2MsrS04g9A4aOoyg-1740863655333-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 919ba9b539550587-IAD
< alt-svc: h3=":443"; ma=86400
<
{
"error": {
"message": "Missing bearer authentication in header",
"type": "invalid_request_error",
"param": null,
"code": null
}
* Connection #0 to host api.openai.com left intact
I am engaging our engineering back end team to see if they see anything on their end .
1 Like
Sure, I will share with you in the chat to avoid posting them publicly