Open-Weight Models: Should You Run Them Yourself?

A question keeps coming up from solo builders and small teams: "Open models have gotten good — wouldn't running one ourselves be cheaper?" With Llama, Qwen, Mistral, Gemma, and gpt-oss all available as open weights, the option is real. But "we could" and "we should" are different questions. Here's how to decide, and how far to move.

Does running open weights actually pay off for your team?

For most small teams, a commercial API is still the right default. Self-hosting starts to win when at least two of the conditions below apply. Otherwise, GPU rental plus the hours your team spends babysitting the stack will exceed what you were paying per token.

• Data can't leave your perimeter — a client contract forbids external transfer, or you're in healthcare, legal, or public-sector work where sending data out triggers a review
• You run narrow, repetitive tasks at volume — classification, tagging, summarization, embeddings; calls where the correct answer space is small and the traffic is constant
• You need to pin the model version yourself — features where a vendor update that shifts tone would break something, like contract clause generation

"Self-hosting" has stages

Teams often jump straight to buying hardware. There are safer intermediate steps.

1. Call open models through a managed provider (Together AI, Fireworks AI, Groq, Amazon Bedrock, Azure AI Foundry)
2. Rent a dedicated GPU instance and serve the model yourself with an engine like vLLM
3. Run it on your own hardware and fine-tune on your data

Stage one alone buys you most of what matters: the ability to swap models at will, plus real leverage in pricing conversations.

Which work goes to open models, and which stays on frontier models?

Split by task difficulty, not by feature. Inside a single product, some calls are fine on a small open model and others will cause incidents if you downgrade them.

Take an online boutique. Routing an inquiry into shipping/exchange/restock, drafting product copy, and generating search embeddings can all run on a small open model. But interpreting the return policy to write the message that actually reaches the customer, or reading several order histories to propose compensation, belongs on a frontier model.

• Open model side: narrow answer space, short outputs, errors are obvious to a human reviewer, high call volume
• Frontier model side: long context, chained tool calls, output goes to the customer verbatim, money or legal liability is involved

What should you measure before switching?

Public benchmark scores are a starting hint, nothing more. The only trustworthy evidence is your own eval set, run head-to-head against your current model. Two weeks is enough to reach a decision.

1. Pull 100–300 real requests from recent logs and write a one-line pass criterion for each
2. Run your current model to establish a baseline score
3. Run two candidate open models through a managed API on the same set
4. If pass rates land near the baseline, route 10% of live traffic and watch latency and error rates
5. Compare a month of total cost — tokens plus GPU plus the hours you spent — then expand or roll back

What makes all of this cheap to do is architecture. If prompts and eval sets live outside your application code, swapping models becomes a config change rather than a deployment. Services built on Senaru follow the same rule. Even if you don't move to open weights today, being able to move is the durable win for a small team this year.