Can ChatGPT or Claude actually improve a prompt?

Yes, for the mechanical part. A capable assistant can score a set of outputs against fixed criteria, spot the biggest recurring problem, and rewrite the prompt to fix it, round after round. What it won't do on its own is stay honest about its own scoring, which is why you keep the test cases and criteria fixed and spot-check its work.

How many test cases do I actually need?

Fewer real ones beat more made-up ones. 10 to 20 inputs pulled from real usage, covering the cases that actually go wrong, tell you more than 100 synthetic examples invented to look thorough. Realism matters more than volume.

How do I know the new prompt is actually better, not just different?

Score it against the exact same test cases and the exact same criteria as the original, and write both numbers down. If the total goes up on a fixed measurement, the improvement is real. If you can't point to that comparison, you don't actually know yet.

When does manual optimization stop being enough?

When you're running the same loop across many prompts, need it to happen on a schedule instead of an afternoon, or want to try more revisions per round than you can score by hand. That's when an Optimization Run in Baseline picks up the exact loop above and runs it unattended, at a scale a spreadsheet can't keep up with.

Optimize a prompt by hand, one honest round at a time

You can make a prompt meaningfully better without buying anything. Freeze a real test set, score the current prompt against a fixed set of criteria, then hand the revision work to an AI assistant you already use: change one thing, re-score, and keep the change only when the total goes up. Repeat a handful of times and most prompts improve noticeably in an afternoon. The real cost is your time, and knowing exactly when that cost stops being worth it.

What it is, and why it matters

Most people improve a prompt by eyeballing it: change a sentence, glance at a couple of outputs, decide it feels better, move on. The trouble is you can't actually tell. Without a fixed way to measure "better," every edit is a guess dressed up as a decision.

The fix doesn't take any special software. Freeze a real set of test cases so the ground never shifts under you, write down the criteria you're judging by, change exactly one thing at a time, and score the result against the same cases and the same criteria every round. That discipline, on its own, turns hand-tuning into something you can actually trust.

The part that's genuinely tedious is running that loop over and over: score, spot the pattern, revise, re-score, compare. That's mechanical work, and it's exactly what a chat assistant can take off your hands once you give it the right instructions. This guide walks the whole thing end to end, including the exact prompt to hand it.

See it in Baseline, step by step

Freeze a test set
Pick 10 to 20 real inputs, the kind your prompt actually has to handle, not invented edge cases. For each one, write down either the expected output or, when there's no single right answer, a plain description of what a good answer looks like. Keep this exact set unchanged for every round that follows; a moving target makes every score meaningless.
Score the current prompt against written criteria
Before changing anything, run the current prompt against every case and score each result against a fixed, written set of criteria, not a gut feeling. A spreadsheet with one row per case and one column per criterion genuinely works: write a score and a one-line reason in each cell. If you'd rather have a dedicated place to store and re-run these scores, platforms like LangSmith and Braintrust do the same job. Either way, write the scores down before you touch the prompt, so you have a real number to beat.

Hand the loop to an AI assistant

The repetitive part, running the same instructions round after round, is exactly what a chat assistant is good at. Claude, ChatGPT, Copilot, and Codex all handle this about the same; use whichever you already have open. Paste the block below into a fresh chat, then answer its questions about your prompt, your test cases, and your criteria.

You are helping me manually improve an AI prompt through structured rounds of testing and revision. Here is how to run this with me:

1. Ask me for three things if I haven't already given them: my current prompt, a set of test cases (each one an input plus either an expected output or a plain description of what a good answer looks like), and the criteria I'll judge answers by. If anything is missing, help me build it before we start.
2. Once you have the prompt, test cases, and criteria, proceed immediately to scoring. Do not ask for more test cases, do not request numeric weights, do not ask for concrete examples if abstract descriptions were provided. Work with exactly what you have been given. Treat abstract or pattern-style test cases as valid inputs; interpret them reasonably and score against them directly.
3. Run the current prompt against every test case. For each case, reason through what the prompt would likely produce and score it against the criteria. Record all scores in a table so we have a clear starting point.
4. Look across every scored case and name the single failure pattern that shows up most often: the one costing the most points across the whole set. Not every small issue, just the dominant one.
5. Make exactly one focused change to the prompt that targets that pattern. Do not rewrite the whole prompt. Do not fix multiple things at once.
6. Score the revised prompt against the exact same test cases using the exact same criteria. Record the new scores in a table.
7. Compare the new total to the previous round. If it improved, keep the revision and return to step 4. If it did not improve, revert to the previous best prompt and try a different angle on the same failure pattern.
8. Repeat steps 4 through 7. Stop after 5 rounds, or after 2 consecutive rounds with no improvement, whichever comes first.
9. When you stop, deliver all three of the following without waiting to be asked:
- The final prompt in full, ready to copy
- A table showing the score before and after for every test case across all rounds
- A short plain-language summary of what changed and why it helped

Two rules for the whole run: never change the test cases once we start, and judge every change by what it does to the whole set, never by whether it fixes one favorite case at the expense of others.

Let it iterate, and spot-check the work
The assistant will score, revise, re-score, and report back round by round. Read its before-and-after numbers rather than taking its word for an improvement, and skim a handful of individual answers yourself. One honest catch: when the same assistant both rewrites the prompt and scores the result, its own scoring tends to drift generous over time. Keep the criteria fixed and spot-check a few answers by hand every couple of rounds to catch that early.
Repeat until the wins stop, and know the real cost
Most prompts have a handful of genuine improvements left in them, then further rounds stop moving the score. That's the signal to stop, not a fixed count. Budget honestly for the time: one full round, scoring, a revision, re-scoring, and a spot-check, tends to run twenty minutes to an hour by hand, so five or six rounds is a real afternoon, not a quick fix.

How Baseline does it

Frozen Instances

Your 10 to 20 test cases become a set of Instances that stay locked for the whole run, the same discipline you were holding by hand, applied automatically every time.

Rubric-based judging

Your written criteria become a Rubric, so every attempt is scored the same way by the same standard, with no spreadsheet cell to fill in yourself.

An Optimization Run explores many at once

Instead of one focused revision per round, an Optimization Run tries many prompt versions in parallel against the same Rubric and keeps only the ones that measurably score higher.

An afternoon becomes minutes

The whole search, revise and score, runs unattended in the background while your team does something else, and reports back with the winning prompt and the proof.

See how Baseline automates this loop

What you get

Get a measurably better prompt this week, using tools you already have.
Turn a vague sense of "better" into a written score you can defend.
Learn exactly when to stop tuning by hand and let a search take over.
Walk into an automated run already fluent in the loop it's running for you.

Frequently asked questions

Can ChatGPT or Claude actually improve a prompt?: Yes, for the mechanical part. A capable assistant can score a set of outputs against fixed criteria, spot the biggest recurring problem, and rewrite the prompt to fix it, round after round. What it won't do on its own is stay honest about its own scoring, which is why you keep the test cases and criteria fixed and spot-check its work.
How many test cases do I actually need?: Fewer real ones beat more made-up ones. 10 to 20 inputs pulled from real usage, covering the cases that actually go wrong, tell you more than 100 synthetic examples invented to look thorough. Realism matters more than volume.
How do I know the new prompt is actually better, not just different?: Score it against the exact same test cases and the exact same criteria as the original, and write both numbers down. If the total goes up on a fixed measurement, the improvement is real. If you can't point to that comparison, you don't actually know yet.
When does manual optimization stop being enough?: When you're running the same loop across many prompts, need it to happen on a schedule instead of an afternoon, or want to try more revisions per round than you can score by hand. That's when an Optimization Run in Baseline picks up the exact loop above and runs it unattended, at a scale a spreadsheet can't keep up with.