Beyond images — palette, AI proxies, probes

Three things come out of an upload that are not a resized image. All three follow the same rule, and it is the question people ask first:

We produce the artifact. You decide what it means.

Nothing here runs a model on your behalf, ships a UI, or writes to your database. It hands you something cheap that would be expensive for you to compute, at the moment it is cheapest to compute — while the bytes are already open on our machine.

The colour palette

Extracted synchronously during processing — from the image itself, or from a video’s poster frame. Not a job, not a webhook you wait for: by the time the asset is ready, its palette is already on it.

A dominant colour plus up to six vibrant/muted swatches — the Spotify / Apple-Music shape.

There is no getter. The palette is a field on the asset DTO, already in the response the upload returned, so nothing fetches it:

import {
  bestTextContrast,
  iteratePaletteSwatches,
  pickAmbientBackground,
} from "@nitida/asset-client";

asset.palette;
// → { d: "#683600", v: "#C2AA48", m: "#8A7B4E", dv: "#683600", … } | null
//   compact keys on the wire: d v m · lv dv · lm dm

// The helpers read that shape so you never touch the keys yourself:
const ambient = pickAmbientBackground(asset.palette);
// → { hex: "#A7C2C6", textColor: "#000000" } | null

for (const { label, hex } of iteratePaletteSwatches(asset.palette)) {
  const { color, ratio, passesAA } = bestTextContrast(hex);
  // label is the long name — "dominant", "lightMuted", …
}

See the palette example for the full helper set on five real photographs, including the one where the ambient helper picks badly.

palette is null on any asset that has none. Branch on it; the type says AssetPalette | null for a reason. Measured across the platform, 2026-08-22:

kind	with a palette	without
image	24 199	788
video	1 522	585
audio · document · other	0	383

A video gets its palette from its poster frame, so it has one whenever a poster was generated. Audio, documents and other never do, and that is a rule, not a gap.

The video gap is not a backlog waiting to clear: those 585 have no poster at all, and a poster cannot be produced without decoding the video again. So the palette is recoverable only at the price of a fresh transcode — a cost decision, not a pending job. Six stragglers are a different story: their poster is registered but its object is missing from the CDN, a legacy row whose bytes were never migrated.

This paragraph used to be false

Until 2026-08-22 it claimed a video had a palette whenever it had a poster, while the table printed four lines below it showed 879 videos without one. Both were true at once because the Cloud Run job path generated the poster and never derived the palette from it — only the inline path did. Regenerating a poster therefore could not fix it, which is exactly how an outside reader disproved the sentence. The job path now derives it, and the assets that had already landed were backfilled.

What it is for: an ambient background behind a product image, a card tint, a placeholder colour while the image loads that is not grey. It ships on the DTO in a compact form deliberately — an earlier, verbose shape was 62% of the entire asset payload.

What it is not: a design system. It tells you what colours are in the image; whether that should become a gradient is your call.

How it behaves, since people ask: the image is downscaled to 100×100 and the swatches come from a small k-means plus HSL bucketing. It is deterministic — the same bytes always yield the same palette — and it costs no extra request: the palette ships on the DTO the upload already returns.

The AI proxy — and where that logic lives

This is the seam the question is really about.

        nitida                                     YOUR APP
        ──────                                     ────────
  upload a video
        │
        ├──▶ poster    ─────────────────────────▶  <video poster>
        ├──▶ video     ─────────────────────────▶  playback
        ├──▶ aiproxy   ──┐                          5 fps · 720p · CRF 28
        └──▶ probe[]   ──┤                          stills at known offsets
                         │
                         └──▶ webhook ──────────▶  ⚠️ YOU run the model here
                                                   (captioning, moderation,
                                                    scene detection, search)

aiproxy is an artifact, not a feature. It is a deliberately cheap re-encode — low frame rate, modest resolution, high CRF — because a model watching a video does not need 1080p60, and paying to send it one is how video AI budgets disappear.

We do not analyse anything. In our own product, a separate app (ai-media) receives the webhook and runs Gemini Vision for captioning. That app is a consumer of this platform, exactly like yours would be. The webhook carries the DTO plus aiProxyUrl and the ordered probe URLs; what happens next is product logic, and product logic does not belong in a media SDK.

So, concretely:

	where it lives
generating aiproxy / probe	nitida — ask for the preset
receiving “this asset is ready”	nitida → webhook → you
running captioning / moderation / embeddings	your app
storing the caption, searching it, showing it	your app

Skip aiproxy if nothing reads it. It is an extra encode on every video. Only ask for it when a model is actually on the other end.

Probes

Still JPEGs sampled at evenly spaced offsets, uploaded under indexed keys (-pr0.jpg, -pr1.jpg, …) so you can fetch them in order without extra metadata.

Useful for a quality check across a clip, a filmstrip scrubber, or giving a vision model stills instead of video — which is dramatically cheaper when you only need “what is in this?” rather than “what happens in this?”.

Webhooks, and why they can fail safely

Both webhooks are fire-and-forget on purpose. If your endpoint is down, the upload still succeeded and the bytes are still correct. You lose the notification, not the asset — reconcile with a retry trigger or a backfill pass.

The alternative — failing an upload because a downstream consumer was unavailable — trades a recoverable gap for an unrecoverable one.