1. compiles your model code and dependencies without requiring Dockerfiles or K8s
2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)
3. it then spins up the runtime and gives you a production-ready URL endpoint
I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.
Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.
Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.
What happens when AWS interrupts a Spot instance? I don't see an ASG in the architecture — does the controller detect the terminated worker and provision a replacement automatically?
Great question. The controller provision a replacement automatically. The controller runs continuous health-check polls against the worker’s, if it doesn't respond, provisions a new one.
Why not use a AASG?
I chose to use a controller to make the solution cloud agnostic.
You got me interested, a rough table of model size to instance type to spot $/hr would help a lot. The 0.5B example is CPU only, so it doesn't say much about what a 7B or 70B actually costs.
You're right. A CPU only model is definitely is not a real prod workload where you'd need GPU machines.
Here is a rough table of model size to instance type to spot $/hr:
For up to 8B, you can use a c6i.xlearge which costs as spot about $0.4/hr.
For up to 70B, you can use a g5.12xlearge that costs you about $2/hr.
Besides the GPU machine be aware that you need a controller instance to route the requests and scale. That has a fixed cost of up to $0.08/hr. When not in use at all, just type veloxml down --all and it tears down the controller too for true $0/hr.
We're currently running larger LLM models benchmarks to add to the README this week.
Comments
Basically the CLI
1. compiles your model code and dependencies without requiring Dockerfiles or K8s
2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)
3. it then spins up the runtime and gives you a production-ready URL endpoint
I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.
Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.
Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.
What happens when AWS interrupts a Spot instance? I don't see an ASG in the architecture — does the controller detect the terminated worker and provision a replacement automatically?
Great question. The controller provision a replacement automatically. The controller runs continuous health-check polls against the worker’s, if it doesn't respond, provisions a new one.
Why not use a AASG?
I chose to use a controller to make the solution cloud agnostic.
You got me interested, a rough table of model size to instance type to spot $/hr would help a lot. The 0.5B example is CPU only, so it doesn't say much about what a 7B or 70B actually costs.
You're right. A CPU only model is definitely is not a real prod workload where you'd need GPU machines.
Here is a rough table of model size to instance type to spot $/hr:
For up to 8B, you can use a c6i.xlearge which costs as spot about $0.4/hr.
For up to 70B, you can use a g5.12xlearge that costs you about $2/hr.
Besides the GPU machine be aware that you need a controller instance to route the requests and scale. That has a fixed cost of up to $0.08/hr. When not in use at all, just type veloxml down --all and it tears down the controller too for true $0/hr.
We're currently running larger LLM models benchmarks to add to the README this week.