Prerecord video demos of it, and have your site swap out the live demo with the pre-recorded videos after $x amount of money has been spent by visitors.
Not too bad, I had some gas in the tank from an old project sitting on my API account so it used up all of it. Expect in the few hundred dollars.
I'm considering switching to DeepSeek since it's way cheaper. I'll swap once I'm done testing out the API. You can use your own hosted LLM but it's not worth it at this moment.
Woah, a few hundred just to demo the thing. How could you keep this afloat?
That’s the tricky part about building LLM apps. I’d love to hear more from Indie devs because money is absolutely a bottle neck here.
For fun:
You don’t need an LLM for some of your calls I think. “Where is the Eiffel Tower”, Eiffel Tower is a NER that small NLP libraries can extract. Then it’s a simple long/lat lookup. You might be able to re-route 20% of your calls to a no-cost backend call.
never had any thought on monetizing this at all so maybe offer pro features down the line? idk
I'm used to just putting stuff out there for people to try out. Could be spent on worse things all things considered haha
CPU inference for LLMs takes forever (you'll get like 1tk/s on CPU) and limits you significantly in terms of model size/quality. You'll lock up all of your cores to provide service for a single user at a snail's pace.
I don't think it should even be considered as an option
llama 3.2 3b, qwen2.5 3B quantized to 4bit runs CPU inference quite fast. You can get a beefier VM and still save a ton of money. Depending on the context token length of this soluion, it's either fast or slow. If it's below 1024 tokens per request, you get around 10 sec delay, if you are at around 128 tokens I guess you would be somewhere at 1 sec for time to first token...
Comments
What are your api costs from publicly serving this if you don’t mind sharing (especially from HN traffic)?
I have some ideas I’d like to release, but LLM api pricing and sudden traffic from sites like this one seem scary.
Prerecord video demos of it, and have your site swap out the live demo with the pre-recorded videos after $x amount of money has been spent by visitors.
Not too bad, I had some gas in the tank from an old project sitting on my API account so it used up all of it. Expect in the few hundred dollars.
I'm considering switching to DeepSeek since it's way cheaper. I'll swap once I'm done testing out the API. You can use your own hosted LLM but it's not worth it at this moment.
Woah, a few hundred just to demo the thing. How could you keep this afloat?
That’s the tricky part about building LLM apps. I’d love to hear more from Indie devs because money is absolutely a bottle neck here.
For fun:
You don’t need an LLM for some of your calls I think. “Where is the Eiffel Tower”, Eiffel Tower is a NER that small NLP libraries can extract. Then it’s a simple long/lat lookup. You might be able to re-route 20% of your calls to a no-cost backend call.
never had any thought on monetizing this at all so maybe offer pro features down the line? idk I'm used to just putting stuff out there for people to try out. Could be spent on worse things all things considered haha
You could use CPU inference on a smaller local model (either always or after a demo budget is spent).
Don't forget to put up a "donate" button.
That sounds like a bad idea to me.
CPU inference for LLMs takes forever (you'll get like 1tk/s on CPU) and limits you significantly in terms of model size/quality. You'll lock up all of your cores to provide service for a single user at a snail's pace.
I don't think it should even be considered as an option
llama 3.2 3b, qwen2.5 3B quantized to 4bit runs CPU inference quite fast. You can get a beefier VM and still save a ton of money. Depending on the context token length of this soluion, it's either fast or slow. If it's below 1024 tokens per request, you get around 10 sec delay, if you are at around 128 tokens I guess you would be somewhere at 1 sec for time to first token...
My point still stands … 10 seconds is an eternity, and a 3B model isn’t that performant.
I’d rather not offer a demo at all than offer it with these parameters.