Cost & latency tradeoffs
Every AI feature has two practical costs: money (roughly proportional to tokens in/out) and time (latency - how long a response takes). Bigger/smarter models are usually slower and pricier; smaller/faster models are cheaper but less capable. For example, running a heavy task through GPT-4o can cost 10x more per token than using Claude 3.5 Haiku for the exact same result. Product decisions constantly trade these off - you don't always need the smartest model for every task.
A common real pattern: use a small, cheap, fast model for simple tasks (classification, extraction) and reserve the expensive, slow, capable model only for the hard reasoning step - exactly mirroring the "orchestration" idea (E5). Latency also compounds in agent loops - five sequential model calls each taking 2 seconds is a 10-second wait, which matters hugely for user-facing products vs. background jobs.
Advisory exercise: for any AI feature proposal (yours or a vendor's), ask "which parts genuinely need the most capable model, and which are simple enough for something cheaper and faster?" - a question that immediately signals sophistication in a vendor conversation.
What's a common, smart pattern for managing AI cost and latency?