# Beyond images — palette, AI proxies, probes

> The artifacts the platform produces that are not pictures, and the seam between what we generate and what you decide.

Three things come out of an upload that are not a resized image. All three
follow the same rule, and it is the question people ask first:

> **We produce the artifact. You decide what it means.**

Nothing here runs a model on your behalf, ships a UI, or writes to your
database. It hands you something cheap that would be expensive for you to
compute, at the moment it is cheapest to compute — while the bytes are already
open on our machine.

## The colour palette

Extracted **synchronously during processing** — from the image itself, or from
a video's poster frame. Not a job, not a webhook you wait for: by the time the
asset is `ready`, its palette is already on it.

A dominant colour plus up to six vibrant/muted swatches — the Spotify /
Apple-Music shape.

**There is no getter.** The palette is a field on the asset DTO, already in the
response the upload returned, so nothing fetches it:

```ts
import {
  bestTextContrast,
  iteratePaletteSwatches,
  pickAmbientBackground,
} from "@nitida/asset-client";

asset.palette;
// → { d: "#683600", v: "#C2AA48", m: "#8A7B4E", dv: "#683600", … } | null
//   compact keys on the wire: d v m · lv dv · lm dm

// The helpers read that shape so you never touch the keys yourself:
const ambient = pickAmbientBackground(asset.palette);
// → { hex: "#A7C2C6", textColor: "#000000" } | null

for (const { label, hex } of iteratePaletteSwatches(asset.palette)) {
  const { color, ratio, passesAA } = bestTextContrast(hex);
  // label is the long name — "dominant", "lightMuted", …
}
```

See [the palette example](/examples/palette/) for the full helper set on five
real photographs, including the one where the ambient helper picks badly.

`palette` is **`null` on any asset that has none.** Branch on it; the type says
`AssetPalette | null` for a reason. Measured across the platform, 2026-08-22:

| kind | with a palette | without |
|---|---|---|
| image | 24 199 | **788** |
| video | 1 522 | **585** |
| audio · document · other | 0 | **383** |

A video gets its palette from its **poster frame**, so it has one whenever a
poster was generated. Audio, documents and `other` never do, and that is a rule,
not a gap.

The video gap is **not** a backlog waiting to clear: those 585 have **no poster
at all**, and a poster cannot be produced without decoding the video again. So
the palette is recoverable only at the price of a fresh transcode — a cost
decision, not a pending job. Six stragglers are a different story: their poster
is *registered* but its object is missing from the CDN, a legacy row whose bytes
were never migrated.

:::caution[This paragraph used to be false]
Until 2026-08-22 it claimed a video had a palette whenever it had a poster,
while the table printed four lines below it showed 879 videos without one. Both
were true at once because the **Cloud Run job path** generated the poster and
never derived the palette from it — only the inline path did. Regenerating a
poster therefore could not fix it, which is exactly how an outside reader
disproved the sentence. The job path now derives it, and the assets that had
already landed were backfilled.
:::

**What it is for:** an ambient background behind a product image, a card tint, a
placeholder colour while the image loads that is not grey. It ships on the DTO
in a compact form deliberately — an earlier, verbose shape was **62% of the
entire asset payload**.

**What it is not:** a design system. It tells you what colours are *in* the
image; whether that should become a gradient is your call.

*How it behaves, since people ask:* the image is downscaled to 100×100 and the
swatches come from a small k-means plus HSL bucketing. It is **deterministic** —
the same bytes always yield the same palette — and it costs no extra request:
the palette ships on the DTO the upload already returns.

## The AI proxy — and where that logic lives

This is the seam the question is really about.

```
        nitida                                     YOUR APP
        ──────                                     ────────
  upload a video
        │
        ├──▶ poster    ─────────────────────────▶  <video poster>
        ├──▶ video     ─────────────────────────▶  playback
        ├──▶ aiproxy   ──┐                          5 fps · 720p · CRF 28
        └──▶ probe[]   ──┤                          stills at known offsets
                         │
                         └──▶ webhook ──────────▶  ⚠️ YOU run the model here
                                                   (captioning, moderation,
                                                    scene detection, search)
```

**`aiproxy` is an artifact, not a feature.** It is a deliberately cheap re-encode
— low frame rate, modest resolution, high CRF — because a model watching a video
does not need 1080p60, and paying to send it one is how video AI budgets
disappear.

**We do not analyse anything.** In our own product, a separate app (`ai-media`)
receives the webhook and runs Gemini Vision for captioning. That app is *a
consumer of this platform*, exactly like yours would be. The webhook carries the
DTO plus `aiProxyUrl` and the ordered `probe` URLs; what happens next is
product logic, and product logic does not belong in a media SDK.

So, concretely:

| | where it lives |
|---|---|
| generating `aiproxy` / `probe` | **nitida** — ask for the preset |
| receiving "this asset is ready" | **nitida** → webhook → you |
| running captioning / moderation / embeddings | **your app** |
| storing the caption, searching it, showing it | **your app** |

**Skip `aiproxy` if nothing reads it.** It is an extra encode on every video.
Only ask for it when a model is actually on the other end.

## Probes

Still JPEGs sampled at evenly spaced offsets, uploaded under indexed keys
(`-pr0.jpg`, `-pr1.jpg`, …) so you can fetch them in order without extra
metadata.

Useful for a quality check across a clip, a filmstrip scrubber, or giving a
vision model stills instead of video — which is dramatically cheaper when you
only need "what is in this?" rather than "what happens in this?".

## Webhooks, and why they can fail safely

Both webhooks are **fire-and-forget on purpose**. If your endpoint is down, the
upload still succeeded and the bytes are still correct. You lose the
notification, not the asset — reconcile with a retry trigger or a backfill pass.

The alternative — failing an upload because a downstream consumer was
unavailable — trades a recoverable gap for an unrecoverable one.