Fine-Tuning vs Prompting vs Decision Models: Picking Your Adaptation

"Should we fine-tune?" is the question engineering leaders ask when an AI feature underperforms, and it's usually the wrong first question, the way "should we rewrite in Rust?" is usually the wrong first question. This page is the decision tree, with the third branch most 2024-era comparisons don't have: question-designed decision models, which changed the arithmetic for a large class of tasks. Custody note: LLM vs traditional ML compares language models against trained classifiers; this page compares the ways of adapting language models themselves.

The three branches, honestly characterized

Prompting is behavior specification at inference time: instructions, examples, constraints in the request. Zero setup, instant iteration, fully inspectable, and paying a per-call token tax for its verbosity, with a ceiling on how much nuance instructions can carry.

Fine-tuning is behavior baked into weights: train on examples, get a model that is the behavior. Strong on style, format, and domain reflexes; expensive in the ways that don't show on the invoice: dataset curation, training loops, versioned model artifacts, re-tuning when requirements drift, and the loss of "just edit the sentence" iteration. It's a small ML program wearing a convenience costume.

Decision-model question design is the branch specific to closed-set work: instead of adapting a generalist, frame the task as operational questions for a model built to answer exactly those (the decision-model thesis). Adaptation lives in editable text like prompting, runs at verdict prices like nothing else, and applies only where the output is a label, score, or choice, which, per the reclassification audit, is more of your workload than you think.

The decision tree, in order

Is the task decision-shaped? Then question design first, cascade for the hard slice per routing, and fine-tuning only if extreme stable volume later justifies graduating to a trained artifact, using your logged verdicts as free training data.

Is it generative with quality gaps? Exhaust prompting with real evals before touching training: most "we need fine-tuning" cases are under-specified prompts, missing context, or absent guardrails, all fixable by Friday. Fine-tune when three conditions hold together: the behavior is stable (style guides, formats, domain voice, not this quarter's policy), you have thousands of genuinely good examples, and per-call token savings from shorter prompts times your volume beats the pipeline's carrying cost.

Is it knowledge the model lacks? Neither branch: that's retrieval's job, and RAG with measured faithfulness beats baking facts into weights that can't be updated or audited.

The sequencing that keeps winning across the directory's builds: prompt or question-design everything on day one, instrument with evals, and let the eval data nominate fine-tuning candidates rather than letting frustration nominate them. Teams that fine-tune first buy a training pipeline to discover what a boundary clause would have fixed; teams that never fine-tune leave token money on the table at scale. The tree exists so you do neither.

Frequently asked questions

When is fine-tuning actually worth it?

When behavior is stable, example data is plentiful and clean, and volume makes the shorter-prompt token savings exceed the pipeline's true cost, typically style/format/domain-voice tasks at scale, not policy or reasoning gaps.

Can prompting match fine-tuned quality?

On instruction-followable behavior with good examples and boundary clauses, usually closer than teams expect, and it iterates in minutes instead of training runs. Measure with a real eval suite before concluding either way.

Where do decision models fit in this comparison?

As the default for closed-set tasks: prompting-style editability at a fraction of either branch's cost, with fine-tuning reserved for the rare graduated workload. The small-model comparison covers the adjacent chat-tier tradeoffs.

Does fine-tuning fix hallucination?

No; missing knowledge is retrieval's problem and unsupported claims are a grounding-check problem. Fine-tuning shapes how a model answers, not what it knows.