Every GenAI project starts with the same question: which model should we use? The honest answer is that it depends — and the teams that win treat model selection as an engineering decision, not a brand preference. Below is the exact framework we run with clients before writing a line of integration code.
Start with the workload, not the leaderboard
Public benchmarks rarely match your actual task. A model that tops a reasoning leaderboard may underperform on your domain-specific extraction job. Define 20–50 representative tasks from your real data first, then measure every candidate against them. That eval set becomes the single source of truth for the whole project.
Weigh the four real constraints
Capability is only one axis. Latency decides whether a model can sit in a live chat flow. Cost compounds fast at scale — a model that's 3× cheaper per token can be the difference between a viable unit economic and a money-losing feature. Data residency and privacy often rule options in or out before capability is even discussed, especially in fintech and healthcare.
Mix models per step
The best architectures rarely use one model everywhere. A cheap, fast model can handle routing and classification while a frontier model handles the hard reasoning step, and an open-source model fine-tuned on your data covers a narrow high-volume task. Designing the pipeline this way routinely cuts cost 40–60% with no measurable quality loss.
Plan for change
Models are released every few months. Abstract the model behind a thin interface so swapping providers is a config change, not a rewrite, and keep your eval set in CI so you can re-benchmark a new model against your tasks in an afternoon.
Key Takeaways
- Build a 20–50 task eval set from your real data before choosing a model.
- Score candidates on capability, latency, cost and data residency together.
- Use different models for different pipeline steps to optimize cost.
- Keep the model swappable and your eval set in CI.