AI news
AI model reviews
AI tools
How-to guides
Our staffAboutContact

How to optimize AI models in 7 easy steps (with examples)

How to optimize AI models in 7 steps: evals, prompts and data, hyperparameter tuning, LoRA, quantization, distillation and faster serving, with a worked example.

Close-up of a graphics card used for AI workloads
Photo: Rafael Pol on Unsplash

To optimize an AI model, you measure it first, then fix the cheapest problems before the expensive ones: prompt and data, then hyperparameters, then fine-tuning, then making the model smaller and faster with quantization, distillation and better serving. Most teams don't need all seven steps. I'd guess the first two solve half the problems people bring to fine-tuning.

This guide covers AI model optimization for two kinds of reader: people calling a hosted model through an API, and people running their own open models. The steps are the same; the tools differ. If you're fuzzy on the basics, our explainer on what an AI model is is a good warm-up.

Before you start

  • One clear job for the model, such as tagging support tickets or summarising contracts.
  • A test set of 50 to 200 real inputs with known correct answers.
  • A target for each thing you care about: accuracy, latency per request, cost per 1,000 requests and memory use.
  • Python with PyTorch if you host your own model, or API access if you don't.
  • A spreadsheet or eval tool to log every run, so you can compare versions side by side.

Step 1: Set a baseline with evals

You can't optimize what you haven't measured. OpenAI's model optimization guide starts the same way: write evals that set a baseline, then loop through prompting, fine-tuning and re-testing.

One practical wrinkle. OpenAI's hosted Evals dashboard and API go read-only on November 30, 2026, and the company points users to Promptfoo instead (it's on the deprecations page). A plain script that runs your test set and writes scores to a CSV works fine too.

Example eval spec:

Task: label each support ticket as billing, bug, account or other.

Metric 1: exact-match accuracy on 150 labelled tickets.

Metric 2: median latency per ticket, in milliseconds.

Metric 3: cost per 1,000 tickets.

Pass: accuracy of 92% or higher, latency under 300 ms, cost under $0.50.

Our guide on how to test AI models goes further on building test sets.

Step 2: Fix the prompt and the data first

Honestly, this is where I'd spend the first day. A clearer instruction or two good examples often moves accuracy more than any training run, and it costs nothing. OpenAI's guide lists three habits: include the context the model needs, give clear instructions and show example outputs (few-shot prompting).

If you train your own model, the same logic applies to data. Remove duplicates, fix wrong labels and check that every class has enough examples. Short, boring work, and in my view it's the step with the best return per hour, because a model trained on mislabelled examples will learn those mistakes faithfully no matter how clever the rest of your setup is.

Example prompt to audit your own prompt:

Here is my system prompt and 10 inputs where the model gave the wrong label. For each failure, say which part of the prompt was missing or unclear. Then suggest one revised prompt under 200 words. Don't add new labels.

For more on this, see our AI prompt engineering guide.

Step 3: Tune hyperparameters with a search tool

Hyperparameters are the settings you choose before training: learning rate, batch size, number of layers, dropout. Guessing them by hand is slow. Optuna is an open-source framework that searches them for you and can prune unpromising trials early, and it works with any machine learning framework.

Example Optuna setup:

import optuna

def objective(trial):

  lr = trial.suggest_float("lr", 1e-5, 1e-3, log=True)

  batch = trial.suggest_categorical("batch", [16, 32, 64])

  return train_and_score(lr, batch)  # your function, returns validation accuracy

study = optuna.create_study(direction="maximize")

study.optimize(objective, n_trials=50)

print(study.best_params)

Fifty trials is a reasonable start for a small model. For a large one, you'll probably want fewer, cheaper trials on a slice of the data, and only then a final full run with the winning settings so you aren't paying for fifty expensive training jobs.

Step 4: Fine-tune with LoRA when prompting isn't enough

Fine-tuning helps when you need a fixed output format, a specialist vocabulary or shorter prompts at scale. Full fine-tuning updates every weight, though, and that's expensive. LoRA trains small add-on matrices instead; the original LoRA paper reported cutting trainable parameters by 10,000 times and GPU memory by three times compared with fully fine-tuning GPT-3 175B.

Hugging Face's PEFT library is the usual way to do this on open models. We explain the idea in more depth in what LoRA is in AI.

A warning for API users: OpenAI is winding down its fine-tuning platform. It's closed to new users, and self-serve job creation becomes restricted on January 6, 2027, per its deprecations page. Existing fine-tuned models keep running until their base models retire. That surprised me, and I think it pushes more teams toward open models for this step.

Example LoRA config (PEFT):

from peft import LoraConfig, get_peft_model

config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05, target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")

model = get_peft_model(base_model, config)

model.print_trainable_parameters()

Step 5: Quantize the weights

Quantization stores weights in fewer bits, such as 8-bit or 4-bit integers instead of 32-bit floats. The model gets smaller and usually faster, with some accuracy loss you need to measure.

The numbers can be large. Google's LiteRT quantization guide lists dynamic range quantization as 4x smaller with a 2x to 3x speedup on CPU. PyTorch's torchao library says it quantized Llama-3-8B to int4 for 1.89x faster inference with 58% less memory. For ONNX models, ONNX Runtime has its own 8-bit tools.

Example (torchao, int4 weights):

pip install torchao

from torchao.quantization import Int4WeightOnlyConfig, quantize_

quantize_(model, Int4WeightOnlyConfig(group_size=32))

Rerun your Step 1 evals after this. Every time.

Step 6: Distill or prune to a smaller model

Distillation trains a small "student" model to copy a larger "teacher". The best-known case is DistilBERT, whose authors cut BERT's size by 40% while keeping 97% of its language understanding and running 60% faster. For many narrow jobs, a small distilled model is all you need (and it's far cheaper to run).

Pruning removes weights that contribute little. PyTorch ships torch.nn.utils.prune for this. In my view, pruning is the step most people can skip; quantization and distillation give clearer wins with less fiddling.

Example prompt to build a distillation dataset from a large model:

You are labelling support tickets for training data. For each ticket below, return JSON with two fields: "label" (billing, bug, account or other) and "reason" (one sentence). Return one JSON object per line and nothing else.

Step 7: Speed up serving with batching and caching

A fast model can still feel slow if it's served one request at a time on a server that recomputes the same long system prompt for every single user who sends a message. If you host open models, vLLM handles continuous batching, prefix caching, quantized weights and speculative decoding out of the box.

On hosted APIs, prompt caching is the easy win. Put the parts that never change (system prompt, examples, tool definitions) at the start of the request. OpenAI's prompt caching is on by default and discounts cached input by up to 95%. Anthropic's version uses a cache_control field with a five-minute default lifetime, and cache hits cost 5% or less of the normal input price across its listed models.

Example request layout for caching:

1. System prompt (fixed, about 1,500 tokens)

2. Ten labelled examples (fixed)

3. Today's ticket (changes every call)

Example: a support-ticket classifier from start to finish

Say a small team runs ticket tagging through a large hosted model. It's accurate but costs too much per month, and replies take over a second. Here's how the seven steps fit together.

  1. Build 150 labelled tickets and record baseline accuracy, latency and cost for the large model.
  2. Tighten the prompt: list the four labels with one-line definitions and add two examples per label. Rerun the evals.
  3. Use the large model to label 5,000 past tickets with the Step 6 prompt, then spot-check 200 by hand.
  4. Fine-tune a small open model on those labels with LoRA, and let Optuna pick the learning rate across 20 trials.
  5. Quantize the fine-tuned model to int4 with torchao and rerun the 150-ticket test.
  6. Serve it with vLLM, keeping the fixed instructions at the front so prefix caching kicks in.
  7. Compare the final row in your spreadsheet with the baseline. If accuracy dropped below 92%, go back to the step where it fell.

The order matters more than the tools. Each step gets judged against the same 150 tickets, so you always know which change helped and which didn't.

Common mistakes and how to fix them

Skipping the baseline

Without Step 1 numbers, every later change is a guess. Fix: freeze a test set before touching anything, keep it out of every training and tuning run, and write the baseline scores somewhere your whole team can see them before anyone starts experimenting.

Fine-tuning to fix a prompt problem

Missing label definitions? Write them into the prompt.

Quantizing without re-testing

Low-bit models can fail on rare inputs while looking fine on average. Fix: check accuracy per class, not just overall.

Tuning on the test set

If Optuna scores trials on the same data you report, your results will look better than they are. Keep a separate validation split for tuning.

Tips to get better results

  • Change one thing at a time, so you know what moved the score.
  • Track cost and latency next to accuracy; a 1% accuracy gain rarely justifies doubling cost.
  • Try a smaller base model before compressing a big one. Our list of open source AI models is a decent shortlist.
  • Count tokens, because shorter prompts mean lower bills and faster replies.
  • Re-run evals when a provider updates its model; outputs can shift between snapshots.

Next steps

Start with Step 1 this week, even if it's just 50 examples in a spreadsheet, because every other step in this guide depends on having a number to beat. If you're building a model from scratch rather than tuning one, read how to train an AI model next. And if this work interests you as a career, our guide to online generative AI engineer jobs shows where these skills get hired.

FAQ

What does AI model optimization mean?

It means making a model more accurate, faster, cheaper or smaller for one specific job. Common methods include better prompts, cleaner data, hyperparameter tuning, LoRA, quantization, distillation and smarter serving. The right mix depends on whether you call an API or host the model.

Does quantization hurt accuracy?

Usually a little, sometimes noticeably. It depends on the model, the bit width and the task, which is why you should rerun your evals afterwards and compare results class by class rather than only looking at the average. Google's LiteRT guide describes post-training quantization as giving "little degradation in model accuracy", and I'd expect 4-bit formats to need more care than 8-bit ones.

Should I fine-tune or improve my prompt?

Improve the prompt first, since it's cheap and easy to undo. Fine-tune when you need a strict format, a niche vocabulary or shorter prompts at high volume. OpenAI is restricting self-serve fine-tuning, so open models with LoRA may be the simpler route.

Can I optimize an AI model without a GPU?

Yes, for many steps. Prompt work, evals and prompt caching need no GPU at all. Quantized small models often run on CPU: LiteRT's integer quantization targets CPUs and microcontrollers. Fine-tuning and distillation are far slower without a GPU, so renting one by the hour is probably the practical choice.

Keep reading