AppearanceEmbedder

class rapidtools.processing.reid.AppearanceEmbedder(model_id: str = 'openai/clip-vit-base-patch16', device: str = 'auto', batch_size: int = 16, backend: Callable[[Sequence[Image]], ndarray] | None = None)

Bases: object

L2-normalised image embeddings from a Hugging Face vision backbone.

The default checkpoint is the image tower of CLIP ViT-B/16, whose projected embedding separates “same parked vehicle, second pass” from “the car next to it” better than DINOv2 on street-level crops. A CLIP checkpoint loads its vision tower and projection only; any other checkpoint transformers.AutoModel can load is pooled through its pooler_output (the CLS token for DINOv2) or, failing that, the mean of its last hidden state. Weights are loaded on first use, so constructing the embedder is free and torch is only imported when an image is embedded.

Parameters:

• model_id – Hugging Face model identifier of a vision backbone: a CLIP checkpoint or one transformers.AutoModel can load.

• device – 'auto' picks CUDA when available and falls back to the CPU; any other value is passed to torch unchanged.

• batch_size – Number of images per forward pass.

• backend – Optional replacement for the model: a callable mapping a sequence of PIL images to an (n, d) array. Used in tests and for plugging in a different feature extractor.

Example

>>> from PIL import Image
>>> from rapidtools.processing.reid import AppearanceEmbedder
>>> embedder = AppearanceEmbedder('openai/clip-vit-base-patch16')
>>> vectors = embedder.embed(
...     [Image.open('car_a.jpg'), Image.open('car_b.jpg')]
... )
>>> vectors.shape, float(vectors[0] @ vectors[1])
((2, 512), 0.93)

Methods

embed	Embed images into unit-norm feature vectors.
pool_outputs	One vector per image from a backbone's output.

property is_loaded: bool

True once the backbone weights are in memory.

static pool_outputs(outputs: Any) → Any

One vector per image from a backbone’s output.

Prefers CLIP’s projected image_embeds, then a pooler_output (DINOv2’s CLS token), then the mean of last_hidden_state.

Parameters:

outputs – Model output object or mapping.

Returns:

Any – Tensor of shape (n, d).

Example

>>> import numpy as np
>>> from types import SimpleNamespace
>>> out = SimpleNamespace(pooler_output=np.ones((2, 3)), image_embeds=None)
>>> AppearanceEmbedder.pool_outputs(out).shape
(2, 3)

embed(images: Sequence[Image]) → ndarray

Embed images into unit-norm feature vectors.

Parameters:

images – PIL images; any mode, converted to RGB internally.

Returns:

np.ndarray – float32 array of shape (n, d) whose rows have unit L2 norm, so a dot product is a cosine similarity. An empty input yields an array of shape (0, 0).

Example

>>> import numpy as np
>>> from PIL import Image
>>> fake = lambda imgs: np.ones((len(imgs), 4))
>>> AppearanceEmbedder(backend=fake).embed([Image.new('RGB', (8, 8))])
array([[0.5, 0.5, 0.5, 0.5]], dtype=float32)