1. compiles your model code and dependencies without requiring Dockerfiles or K8s
2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)
3. it then spins up the runtime and gives you a production-ready URL endpoint
I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.
Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.
Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.
Comments
Basically the CLI
1. compiles your model code and dependencies without requiring Dockerfiles or K8s
2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)
3. it then spins up the runtime and gives you a production-ready URL endpoint
I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.
Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.
Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.