What happens when AWS interrupts a Spot instance? I don't see an ASG in the architecture — does the controller detect the terminated worker and provision a replacement automatically?
Great question. The controller provision a replacement automatically. The controller runs continuous health-check polls against the worker’s, if it doesn't respond, provisions a new one.
Why not use a AASG?
I chose to use a controller to make the solution cloud agnostic.
1. compiles your model code and dependencies without requiring Dockerfiles or K8s
2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)
3. it then spins up the runtime and gives you a production-ready URL endpoint
I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.
Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.
Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.
You got me interested, a rough table of model size to instance type to spot $/hr would help a lot. The 0.5B example is CPU only, so it doesn't say much about what a 7B or 70B actually costs.
You're right. A CPU only model is definitely is not a real prod workload where you'd need GPU machines.
Here is a rough table of model size to instance type to spot $/hr:
For up to 8B, you can use a c6i.xlearge which costs as spot about $0.4/hr.
For up to 70B, you can use a g5.12xlearge that costs you about $2/hr.
Besides the GPU machine be aware that you need a controller instance to route the requests and scale. That has a fixed cost of up to $0.08/hr. When not in use at all, just type veloxml down --all and it tears down the controller too for true $0/hr.
We're currently running larger LLM models benchmarks to add to the README this week.
What happens when AWS interrupts a Spot instance? I don't see an ASG in the architecture — does the controller detect the terminated worker and provision a replacement automatically?
Great question. The controller provision a replacement automatically. The controller runs continuous health-check polls against the worker’s, if it doesn't respond, provisions a new one.
Why not use a AASG?
I chose to use a controller to make the solution cloud agnostic.
Basically the CLI
1. compiles your model code and dependencies without requiring Dockerfiles or K8s
2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)
3. it then spins up the runtime and gives you a production-ready URL endpoint
I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.
Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.
Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.
You got me interested, a rough table of model size to instance type to spot $/hr would help a lot. The 0.5B example is CPU only, so it doesn't say much about what a 7B or 70B actually costs.
You're right. A CPU only model is definitely is not a real prod workload where you'd need GPU machines.
Here is a rough table of model size to instance type to spot $/hr:
For up to 8B, you can use a c6i.xlearge which costs as spot about $0.4/hr.
For up to 70B, you can use a g5.12xlearge that costs you about $2/hr.
Besides the GPU machine be aware that you need a controller instance to route the requests and scale. That has a fixed cost of up to $0.08/hr. When not in use at all, just type veloxml down --all and it tears down the controller too for true $0/hr.
We're currently running larger LLM models benchmarks to add to the README this week.