I spent this term building an image captioning model. The classic setup, trained and evaluated on sampled COCO data. It landed at ROUGE-L 39 and BLEU 10.3, and I want to write down what those two numbers taught me, because it wasn't about captioning.
ROUGE-L of 39 suggested a system that was roughly doing its job, while BLEU of 10.3 suggested one that was barely functional. Both scores came from the same outputs and references. Reporting either number alone would have given a distorted picture of the model.
I read hundreds of captions beside their images and references. That inspection began after the score table was complete, and it explained more of the model's behavior than either aggregate metric.
What the numbers couldn't see
The model produced fluent sentences with an entity, action, and scene. BLEU penalized acceptable wording changes such as "a man riding a bike" against "person on a bicycle." Both metrics could also reward reference-like phrasing that described the image incorrectly, including captions that invented an object.
The metrics measured similarity to reference text. My evaluation question was whether the caption described the image correctly. Semantic substitutions and hallucinated objects exposed the difference between those two targets.
Some incorrect captions still scored well. A sentence describing a dog where none existed could match COCO phrasing closely enough to pass an n-gram comparison. Reading the caption beside the image exposed that failure immediately.
BLEU was designed for corpus-level machine translation rather than judging whether one caption described one image. CIDEr incorporates consensus across references, and SPICE evaluates semantic scene content. Those metrics address parts of the mismatch, but I still need output inspection to identify the failures hidden by an aggregate score.
Caption inspection
Matching words can point to the wrong object
Compare a faithful paraphrase with a fluent object error, against the same drawing and reference.
Scenario
An original, illustrative drawing and invented captions. Highlighted words show simple overlap, not a computed BLEU or ROUGE score.
Sequence
- Look at the imageCurrent
- Read the captionUpcoming
- Compare the wordsUpcoming
- Trace meaning to the sceneUpcoming
Look at the image
The drawing shows a person riding a bicycle: two wheels, a pedal frame, and a rider. This drawing is not a COCO sample or a historical model output.
Read the full explanation
Correct paraphrase
An original, illustrative drawing and invented captions. Highlighted words show simple overlap, not a computed BLEU or ROUGE score.
- Look at the image. The drawing shows a person riding a bicycle: two wheels, a pedal frame, and a rider. This drawing is not a COCO sample or a historical model output.
- Read the caption. “Someone pedals a bike” uses different words but preserves the visible object and action.
- Compare the words. The reference is “A person is riding a bicycle.” Marked tokens occur in that reference. This token illustration omits BLEU’s n-gram, clipping, brevity, and corpus aggregation machinery; it is not a metric implementation.
- Trace meaning to the scene. Bike points to the bicycle; pedals points to the rider’s action. Both claims have visual support despite different wording.
Wrong object
An original, illustrative drawing and invented captions. Highlighted words show simple overlap, not a computed BLEU or ROUGE score.
- Look at the image. The drawing shows a person riding a bicycle: two wheels, a pedal frame, and a rider. This drawing is not a COCO sample or a historical model output.
- Read the caption. “A person is riding a motorcycle” preserves most reference words but substitutes an object absent from the drawing.
- Compare the words. The reference is “A person is riding a bicycle.” Marked tokens occur in that reference. This token illustration omits BLEU’s n-gram, clipping, brevity, and corpus aggregation machinery; it is not a metric implementation.
- Trace meaning to the scene. Riding still describes the action, but motorcycle has no supporting object. More matching words do not repair that missing visual link.
Correct paraphrase
An original, illustrative drawing and invented captions. Highlighted words show simple overlap, not a computed BLEU or ROUGE score.
- Look at the image. The drawing shows a person riding a bicycle: two wheels, a pedal frame, and a rider. This drawing is not a COCO sample or a historical model output.
- Read the caption. “Someone pedals a bike” uses different words but preserves the visible object and action.
- Compare the words. The reference is “A person is riding a bicycle.” Marked tokens occur in that reference. This token illustration omits BLEU’s n-gram, clipping, brevity, and corpus aggregation machinery; it is not a metric implementation.
- Trace meaning to the scene. Bike points to the bicycle; pedals points to the rider’s action. Both claims have visual support despite different wording.
Wrong object
An original, illustrative drawing and invented captions. Highlighted words show simple overlap, not a computed BLEU or ROUGE score.
- Look at the image. The drawing shows a person riding a bicycle: two wheels, a pedal frame, and a rider. This drawing is not a COCO sample or a historical model output.
- Read the caption. “A person is riding a motorcycle” preserves most reference words but substitutes an object absent from the drawing.
- Compare the words. The reference is “A person is riding a bicycle.” Marked tokens occur in that reference. This token illustration omits BLEU’s n-gram, clipping, brevity, and corpus aggregation machinery; it is not a metric implementation.
- Trace meaning to the scene. Riding still describes the action, but motorcycle has no supporting object. More matching words do not repair that missing visual link.
Put your own caption beside the evidence
Change the wording, inspect the matching phrases, then check what the drawing actually supports.
A person is riding a bicycle.Grounding inspection
Bike and pedals describe the visible bicycle and action.
The reference is fixed. Text metrics compare against it; they never read the drawing.
Calculated toy text metrics: lowercase English words, punctuation removed, one reference. Clipping limits matches to reference counts; ordered overlap uses the longest common subsequence. These are not the article’s BLEU 10.3 or ROUGE-L 39, and neither score establishes visual correctness.
- BLEU: a Method for Automatic Evaluation of Machine TranslationThe original BLEU paper is useful context because BLEU was designed for cheap corpus-level MT evaluation, not image-understanding diagnosis.
- CIDEr: Consensus-Based Image Description EvaluationIntroduces a caption metric based on human consensus, which is closer to the captioning problem than raw n-gram precision alone.
- SPICE: Semantic Propositional Image Caption EvaluationSPICE shows why n-gram overlap is neither necessary nor sufficient for approximating human caption judgment.
What I changed after that
Written down so I don't let a clean table do too much work:
- A single number is a summary of a judgment, not the judgment itself. The metric was designed by someone, for some purpose, with known failure modes. BLEU's are documented in the literature going back years, and I still got fooled, because a number on a results table radiates more authority than it has earned.
- When two reasonable metrics disagree this much, the disagreement is information. It's the evaluation telling you it doesn't capture what you care about, and that you'd better go look with your own eyes.
- Reading the outputs is not the unscientific fallback. It's the ground truth the metrics are approximating. I found more real problems in two evenings of reading captions than in two weeks of watching scores.
Before trusting a metric, I inspect it beside the artifacts it summarizes: captions with images, extracted fields with source messages, and agent replies with the conversation that produced them.
I pinned a three-word note above my desk: look at the outputs. The models and metrics will change, but I expect to keep that part of the evaluation process.