LlamaVisionInference

class rapidtools.models.LlamaVisionInference(model_id: str = 'meta-llama/Llama-3.2-11B-Vision-Instruct', device: str = 'auto', load_in_4bit: bool = False, temperature: float = 0.4, max_tokens: int = 2048)

Bases: BaseLocalInferenceModel

Local inference with Meta’s Llama 4 and Llama 3.2 Vision models.

Example

>>> from rapidtools.models import LlamaVisionInference
>>> LlamaVisionInference.list_available_models()[0]
'meta-llama/Llama-4-Scout-17B-16E-Instruct'
>>> model = LlamaVisionInference(
...     'meta-llama/Llama-3.2-11B-Vision-Instruct', load_in_4bit=True
... )
>>> out = model.run_inference(
...     ['front.jpg', 'side.jpg'], 'Return JSON with a damage_level key.',
...     json_mode=True,
... )
>>> out.text
'{"damage_level": "severe"}'

Methods

list_available_models	Return the curated Llama vision checkpoints.
run_batch	Process assets sequentially to avoid GPU out-of-memory crashes.
run_inference	Execute a forward pass and return the generated text.

classmethod list_available_models(auth_key: str | None = None) → list[str]

Return the curated Llama vision checkpoints.

Downloading them from the Hub requires accepting Meta’s licence.

Parameters:

auth_key – Unused; present for signature parity with API wrappers.

Returns:

list[str] – Repository IDs, newest first.

Example

>>> from rapidtools.models import LlamaVisionInference
>>> len(LlamaVisionInference.list_available_models())
6

run_inference(image_inputs: str | Path | list[str | Path], prompt: str | Path, json_mode: bool = False, temperature: float | None = None, max_tokens: int | None = None, config: GenerationConfig | None = None, **kwargs: Any) → ModelOutput | None

Execute a forward pass and return the generated text.

Parameters:

• image_inputs – One path/URL or a list of them.

• prompt – Prompt text or path to a prompt file.

• json_mode – Append a JSON-only instruction to the prompt (open models have no native JSON mode).

• temperature – Per-call temperature override.

• max_tokens – Per-call output-token limit override.

• config – A GenerationConfig; explicit keyword arguments take precedence over it.

• **kwargs – Ignored; accepted for interface compatibility.

Returns:

ModelOutput | None – Generated text plus raw token IDs, or None when no image loads or generation fails (e.g. out of memory).

Example

>>> from rapidtools.models import LlamaVisionInference
>>> model = LlamaVisionInference(load_in_4bit=True)
>>> model.run_inference('house.jpg', 'Is the roof intact?').text
'No, part of the roof is missing.'