Hello everyone, I am using SambaNova Fast Api.
Does fast api provide the prompt caching?
If yes, could you let me know how I can implement?
Thank you.
1 Like
Technically, there’re many ways, most http clients provide option for caching request based on body. I feel, like ideally Sambanova is intended for projects, where you have to utilize fast but very dissimilar prompts.
You can also use proxy, like nginx or manually store responses into KV, memory or Vector storage
1 Like
@suslovwebhero and @gothesopre Prompt caching is indeed on the road map but we do not have an ETA as of yet.
-Coby
1 Like