Writing alt text at scale with AI vision — without the hallucinations

Alt text is the one metadata field with two audiences that both punish you for getting it wrong: screen-reader users who are misled by a bad description, and search engines that treat a fabricated one as a quality signal against you. AI vision makes writing it fast — the risk is that it also makes writing it confidently wrong.

Why models hallucinate on images

A caption model asked “what is this?” will always answer, even when the honest answer is “a person in a red jacket, brand unknown.” Left unconstrained it will guess a logo, a city, an emotion — and phrase the guess with the same certainty as a fact. In alt text, a guessed brand name is a trademark problem and a guessed location poisons a photographer’s whole catalogue.

Grounding is the whole game

The fix isn’t a better model, it’s a stricter prompt. Three rules do most of the work:

• Describe only what is visibly present; if something is unclear, leave it out rather than guess.
• Never invent brand names, place names, dates or the identity of people.
• Write one plain sentence for a screen reader — not a marketing headline.

Keep it distinct from the caption

Alt text and the visible caption serve different jobs. Alt text is a functional description of the image for someone who can’t see it; the caption adds context a sighted reader also gets. If they’re identical, one of them is wasted. Good pipelines generate them separately and check they differ.

Takeaway: AI is the right tool for alt text at scale — but only behind a grounding prompt and a “leave it blank rather than lie” rule. An empty field can be filled later; a confident fabrication propagates.