AI-assisted captions, alt text, and tags with Workers AI vision models

Writing alt text for every photograph is the boring half of running a photography site. A vision model can take the first pass.

This is the final post in the photography portfolio series . It covers the Workers AI suggestions that power the description, title, alt text, tag, and album-description fields on alex.edestudio.us and jamie.edestudio.us : how the model dispatcher picks an API shape, why the prompts read more like creative-writing briefs than instructions, and where Workers AI is genuinely useful versus where it confidently invents.

The “suggest” pipeline has two layers of dispatch happening at once. They are easy to confuse.

The model dispatcher picks the right Workers AI API shape for the model id. Workers AI hosts vision models with two completely different input schemas:

  • Legacy LLaVA-style models (@cf/llava-hf/llava-1.5-7b-hf, @cf/unum/uform-gen2-qwen-500m) take { image: number[], prompt, max_tokens }. The image is an array of byte values.
  • Modern models (@cf/google/gemma-4-26b-a4b-it, @cf/moonshotai/kimi-k2.6, llama-4-scout) take an OpenAI-compatible messages array with image_url content parts. The image is a base64 data URL.

I keep both branches alive because the trade-off is real: legacy LLaVA is fast and cheap, modern models write better captions. Some sites might never need the latter.

The prompt-variant dispatcher picks one of several creative-writing briefs for the field being suggested. A description can be “Caption”, “Minimal”, “Mood”, “Story”, or “Observational”. A title can be “Gallery”, “Minimal”, or “Place”. An album description can be “Curatorial”, “Narrative”, “Tagline”, or “Mood”. The photographer picks one in the UI; the dispatcher resolves it to the right prompt string before calling the model.

Together, the two layers mean any future model swap is a one-line config change, and any prompt experiment is a copy-edit in suggestVariants.ts.

// functions/lib/visionInference.ts (excerpt)
const LEGACY_VISION_MODELS = new Set([
  "@cf/unum/uform-gen2-qwen-500m",
  "@cf/llava-hf/llava-1.5-7b-hf",
]);

const THINKING_CAPABLE_MODELS = new Set([
  "@cf/google/gemma-4-26b-a4b-it",
  "@cf/moonshotai/kimi-k2.6",
]);

export async function runVisionInference(
  ai: Ai,
  model: string,
  imageBytes: Uint8Array,
  prompt: string,
  maxTokens: number,
): Promise<string> {
  if (LEGACY_VISION_MODELS.has(model)) {
    const result = await ai.run(model, {
      image: Array.from(imageBytes),
      prompt,
      max_tokens: maxTokens,
    });
    if (typeof result === "string") return result;
    return result?.description ?? result?.response ?? "";
  }

  // OpenAI-compatible: image as a base64 data URL inside messages[]
  const base64 = uint8ToBase64(imageBytes);
  const mime = detectImageMimeType(imageBytes);

  const params = {
    messages: [{
      role: "user",
      content: [
        { type: "image_url", image_url: { url: `data:${mime};base64,${base64}`, detail: "auto" } },
        { type: "text", text: prompt },
      ],
    }],
    max_completion_tokens: maxTokens,
  };

  if (THINKING_CAPABLE_MODELS.has(model)) {
    params.chat_template_kwargs = { enable_thinking: false };
  }

  const result = await ai.run(model, params);
  if (typeof result === "string") return result;
  return result?.choices?.[0]?.message?.content ?? result?.response ?? "";
}

Three things in there are worth pointing out.

enable_thinking: false on thinking-capable models. Gemma 4 and Kimi-K2.6 both ship with reasoning mode on by default. For “describe this photograph” tasks that is exactly the wrong setting. The model spends seconds thinking about whether the shadow is artistic before producing a caption that doesn’t benefit from the extra cycles. Disabling thinking via the documented chat_template_kwargs.enable_thinking flag cuts the time-to-first-token from “many seconds” to “subsecond”.

max_completion_tokens not max_tokens. Workers AI’s binding accepts both, but max_tokens is deprecated per the model’s input schema. New code should use max_completion_tokens and the older spelling can stay on the legacy path.

The uint8ToBase64 function is hand-written, not btoa(String.fromCharCode(...arr)). Spreading a large Uint8Array into String.fromCharCode blows the call stack on photos over a couple of MB. A boring for-loop is faster and survives any input size.

Prompt quality matters more than model choice. The same Gemma 4 model that produces a forgettable “A photo of a beach at sunset” with a generic prompt produces something usable with a well-shaped brief.

The shape I have converged on across all variants:

  1. Tell the model what kind of artefact this is. “A caption for a personal portfolio site” sets the register; “a gallery print title” sets a different one. Skip this and you get generic stock-photo copy.
  2. Anchor in something visible. “Open with a precise sensory detail” or “anchor in one concrete, visible element” makes the model commit to the image rather than describe it in the abstract.
  3. Forbid the obvious failure modes. Every prompt includes a “do not” list. The current bans cover “A photo of”, “This image depicts”, and adjective-as-substitute-for-thought words like beautiful, stunning, breathtaking, serene, captures, golden.
  4. Cap the length. Two sentences max for captions; 2 to 6 Title Case words for gallery titles; 4 to 8 words for a tagline. Without a cap the model will write a paragraph and bury the good line in the middle.
  5. Tell it to skip preamble and quoting. Otherwise half the outputs come back as "This image shows..." (with literal quote marks), and you spend the rest of the day stripping prefixes.

The five description variants live in suggestVariants.ts and the photographer picks one per photo. The “Caption” variant has a slightly different default that can be overridden via the SUGGEST_PROMPT env var in wrangler.jsonc, which is where I tune the canonical voice without redeploying code.

Tags are the variant that is hardest to get right. The bad failure mode is the model confidently inventing a tag like eiffel-tower because it saw a vaguely tower-shaped silhouette, or applying street-photography to a posed portrait because the background includes pavement.

The current tag prompt does two things to push back:

  • Hard-cap the count. Three to five tags. More than five and the model starts reaching.
  • Pass the existing tag vocabulary in. The prompt includes Existing portfolio tags: {EXISTING_TAGS} substituted from D1. The model is told to use a listed tag only when the photo obviously belongs with photos tagged that way, never just because it appears in the list, and to prefer a specific new tag over a loose vocabulary match.
const tagPrompt = env.SUGGEST_TAGS_PROMPT.replace(
  "{EXISTING_TAGS}",
  existingTags.join(", "),
);

The vocabulary-aware prompting reduces noisy duplicates (“street” vs “street-photography” vs “city-streets” all collapsing to whichever the model felt strongest about that day) without hard-coding the vocabulary into code. The existing tags come from D1 at request time.

Album-level suggestions are a different problem because the model gets four photos at once. The vision dispatcher above handles only one image at a time, so the album path makes the model do a different thing: instead of feeding four image URLs, the album prompt feeds the cover photo image plus a text list of “the album also contains photos titled X, Y, Z”.

const block = photoList
  ? `The album contains these photographs:\n${photoList}\n\n`
  : "";
return template.replace("{PHOTO_LIST}", block);

The result is good enough for a curatorial blurb without paying for four image inferences. If the model needs to “see” more than the cover, the album page is small enough that a few specific photo titles in text form gets it most of the way there.

This is the single most important rule of the whole feature. Nothing the model produces lands in D1 until the photographer hits Save. The UI shows the suggestion in the field, the photographer reads it, edits it, accepts it, or replaces it entirely. The Workers AI call is read-only as far as the database is concerned.

That means a model hallucination (the wrong landmark, a confident misidentification, an invented person) never reaches the public site. It reaches the photographer’s eyes, and the photographer either fixes it or ignores it.

For alt text in particular, this matters. AI-generated alt text that lands directly on a published page is an accessibility footgun: a model that confidently mislabels a dog as a cat just made every screen-reader user a passenger in the model’s confidence. A photographer in the loop is what turns a model’s first draft into reliable alt text.

Works well:

  • Atmosphere and mood descriptions. Models are good at the “what does this photograph feel like” question because it tolerates imprecision.
  • Generic tags from a fixed vocabulary. Light, subject category, dominant scene.
  • Short titles when the prompt is strict about length and bans cliches.

Does not work well:

  • Place names. The model will guess. If your photograph is “obviously” Big Sur, the model will say so even when it’s the Welsh coast.
  • Proper nouns of any kind. Brands, monuments, people. The confidence-to-correctness ratio is poor.
  • Anything that requires reading text inside the image (signs, captions, labels). OCR-adjacent tasks are not what these models are good at.

The UI quietly nudges the photographer toward the safer use cases by showing model suggestions as one of several variants, not as the only answer. If the “Place” title variant produces garbage on a photo where the model has no idea where it was taken, the photographer picks “Minimal” instead.

  • A model-recommendation pass. Right now the model is fixed per site via SUGGEST_MODEL in wrangler.jsonc . A small first-pass query that runs a cheap legacy LLaVA call and decides “this photo is busy enough to be worth a Gemma 4 caption” or “this is a simple subject, the cheap caption is fine” would save token-spend on photos that don’t warrant the big model.
  • Per-site tone presets. Both sites currently share the same prompt voice. A per-site tone override would be one env var per site away, letting each site develop a distinct voice over time.
  • Multi-image album captioning. The “four image URLs in one prompt” path is shipped in some Workers AI models but the binding does not pass it through cleanly yet. When it does, the album path can stop being a single-photo hack.
  1. Building a photography portfolio on Cloudflare’s full stack. The stack overview: Pages, D1, Images, Access, and Workers AI.
  2. Two sites, one codebase. The deploy-pages.mjs indirection, per-site D1, and Gitea CI.
  3. Cloudflare Images flexible delivery and retina srcSet. Variant setup, srcSet generation, and the original-download endpoint.
  4. Direct Creator Upload from the browser. Why a Pages Function should not proxy your image bytes, and what the three-request handshake looks like in DevTools.
  5. Stripping EXIF before upload and backfilling existing photos. What gets stripped, what stays, and the backfill script.
  6. AI-assisted captions, alt text, and tags. The Workers AI vision-model dispatcher and where it helps versus where it confidently invents. (this post)

The suggestions are an admin-only feature; the output is everywhere on both sites. Read any caption or any photo’s alt text and you are reading something the model drafted and the photographer accepted or rewrote.

That wraps the photography portfolio series. The overview post links back to all six posts. If you are building something similar and want to compare prompt notes or model choices, I am on LinkedIn .

×
Page views: …