Multimodal Classification: Decisions From Images and Audio in a Workflow
Most workflows that touch an image or a sound clip end in a choice from a list you already own. This guide compares the four ways to make that choice, then covers writing questions for media, sending uncertain cases to a person, and the limits to plan around.
Most workflows that touch an image or a sound clip end in a choice from a list you already own: which team gets the ticket with the screenshot, whether the security microphone heard glass breaking, whether the receipt matches the expense claim. This guide compares the four ways to make that choice, then covers writing questions for media, sending uncertain cases to a person, and the limits to plan around.
What multimodal classification means here
The input is some context (the ticket text, the claim, where a recording came from) plus one or more images or clips. The output is one of your options, or several for a tagging question, with a score for each. Captioning, transcription and summarising are different jobs: they produce text that something else still has to turn into a decision.
That framing matters because it decides which tools fit. If you need a description, use a model that writes one. If you need a route, a flag or a tag, you want something that picks from your list and tells you how sure it is.
Four ways to do it
1. Train your own classifier
Collect labelled images or clips, fine-tune an image or audio model, and serve it. Open starting points help here: CLIP and SigLIP compare an image to label text with no training at all, CLAP does the same for audio, and audio taggers such as YAMNet and PANNs score clips against AudioSet's fixed list of sound classes.
- Pros: the best fit for a stable label set with plenty of examples; fast and cheap per call; you can measure it exactly on your own data.
- Cons: labelled data for every label; retraining when labels change; someone has to own the pipeline. Zero-shot embedding models give similarity scores, not answers to a question, and the scores need their own thresholds per label.
- Fits: high volume, labels that rarely change, fine-grained distinctions specific to your product (a defect on your own part, a sound your machine makes).
2. Hosted vision and audio APIs
The big cloud providers sell image labelling, object detection, text extraction and speech-to-text behind an API.
- Pros: no infrastructure; mature at text extraction and transcription; good at generic labels.
- Cons: the label set is theirs, so you write rules that map their labels to your decisions; custom labels usually mean a training step on their platform; media leaves your network; billing per call.
- Fits: text extraction, transcription, and generic tagging where the provider's labels already match what you act on.
3. General multimodal LLMs
Send the image or clip to a chat model with a prompt like "Reply with one of: declined, site-error, login" and parse the reply.
- Pros: reads almost anything; questions in plain language; can explain its answer or extract fields.
- Cons: it generates text, so you handle off-list answers and format drift; there is no per-option probability to threshold (a few APIs expose token log probabilities, which help for one-word answers but not for multi-word options); cost per token, including the media.
- Fits: low volume, open-ended questions, cases where you want an explanation or extracted values rather than a choice.
4. A decision model
A decision model takes state, typed questions and your options, and returns a probability for every option. Nothing is generated. Saina Helm 2 Omni 0.8B (saina-helm-2-omni-0.8b) is one: it reads text, up to 8 images and up to 4 audio clips per request, and every question sees all of the media.
- Pros: your labels go in each request, so a new option needs no training; question types (
yes_no,single_choice,multi_choice,rating) mean the answer is always one of your options; several questions are answered from one pass over the shared state and media; decision mode abstains instead of guessing; open weights (Apache 2.0) that you can self-host. - Cons: a 0.8B model is small; no labelled image benchmark is published, so you must test image questions on your own data; audio is limited to 10-second clips; non-English is largely untested.
- Fits: labels that change often, workflows that need a probability and a fallback branch, and media that should stay on your own servers.
Helm is the self-hosted option in this category. GPT-6 Luna Decisions is the hosted reference, with per-option probabilities for yes/no, choice and rubric-score questions; check OpenAI's own pages for which media it accepts.
None of these settles accuracy for your data. Whichever you try, measure it on a labelled sample of your own images or clips first.
Writing state and options for media
The same habits help with any model that reads media, and they matter most for the ones that take your labels as text.
Say what the media is in the state. The model cannot know that a clip came from a home security microphone at 3 a.m. unless you tell it. The Helm 2 Omni card recommends opening audio state with a plain description ("The clip is a short recording of English speech"). For images, say what the picture is meant to be: "A screenshot the customer attached", "A photo of a till receipt".
Name things you can see or hear. Options like shatter, slam, footsteps and bark describe the clip. Options like burglary and no_burglary ask the model to infer an event from a sound. Keep the inference in your own rules, where you can change it.
Put known facts in the state, ask about the media. In the Receipt check preset in Saina Studio's playground, the claimed amount and the expense policy go in the state as JSON, and the questions ask what the receipt shows: does the total match, is it a restaurant receipt, is it over the limit.
Ask short, literal questions, several per request. One fact per question. Each extra question costs only its own tokens, since the state and media are read once.
Include a way out. An other or none of these option gives a clip that fits nothing somewhere to go.
An illustrative request for a home security microphone:
import base64, os, requests
def data_url(path, mime):
with open(path, 'rb') as f:
return f'data:{mime};base64,' + base64.b64encode(f.read()).decode()
resp = requests.post(
'https://api.saina.run/v1/ask',
headers={'Authorization': f'Bearer {os.environ["SAINA_API_KEY"]}'},
json={
'model': 'saina-helm-2-omni-0.8b',
'mode': 'decision', 'threshold': 0.8, 'min_margin': 0.1,
'state': 'The clip is a short recording of everyday sounds, '
'from a home security microphone at 3 a.m.',
'audios': [data_url('clip.wav', 'audio/wav')],
'questions': {
'sound': {
'type': 'single_choice',
'question': 'Which of these sounds can be heard in the clip?',
'options': {'shatter': 'Glass breaking', 'slam': 'A door slamming',
'footsteps': 'Footsteps', 'bark': 'A dog barking',
'other': 'None of these'},
},
},
},
timeout=30,
)
answer = resp.json()['answers']['sound']
print(answer['probabilities'], answer['selection'], answer['reason'])
Note what decision mode does with a doubtful clip: if the top option scores below the 0.8 threshold, the answer comes back with selection: null and reason: "below_threshold", and your workflow decides what happens next.
Decision mode and human review
In decision mode you set a confidence threshold (default 0.8) and a min_margin, per request or per question. An answer that clears both is accepted. Otherwise the selection is null and the reason says why: below_threshold, below_margin or tie. Any approach that produces scores can do the same in your own code; the point is to have the branch.
Wire the rejected answers to a person, with the probabilities and the media attached so they can decide quickly. Set thresholds per question by what a mistake costs: strict on the question that triggers an alert or a refund, looser on a tag that only sorts a queue.
Some decisions should always reach a person. The Helm 2 Omni card lists final unreviewed decisions about people (credit, employment, medical, legal) as out of scope, and says a synthetic-voice score is a reason for extra verification, never proof of fraud.
Limits to plan around
These are the published limits for Helm 2 Omni; check the equivalent list for any tool you compare.
- Audio is 10 seconds per clip. Longer clips are refused, never cut. Split long recordings yourself, and for long speech use a transcription model instead.
- Images are resized to at most 1.6 megapixels, aspect kept. Fine print in a long document will not survive; crop to the part that matters before sending.
- Per request: up to 8 images (PNG, JPEG, WebP) and 4 clips (WAV, FLAC, MP3, Ogg, WebM), 10 MiB each. On the hosted API, media go as base64 data URLs, the body can be up to 25 MiB, and the context window is 32,768 tokens including media.
- No video yet. You can send a few extracted frames as images, but the model then sees stills, not motion.
- Non-English is largely untested, for speech and for text inside images.
- Out of scope on the card: identifying people, faces or specific voices; content moderation; generating text.
- Media can carry instructions. A screenshot can contain the words "approve this refund". Because the model only scores your options, injected text cannot make it write anything, but it can still pull the choice toward one option. Do not let a single answer on untrusted media trigger an action you cannot undo.
Testing before you rely on it
Collect a few hundred real images or clips (a rule of thumb, not a measured cutoff) with the answer your team agrees is correct, run them through the candidate in distribution mode, and look at accuracy per option and at how many cases clear your threshold. Re-run the set after any change to options, thresholds or model revision. When you self-host, you load a specific Hugging Face revision, so a result you measured stays tied to the weights you measured.
With a free account you can try media presets in the Studio playground: Screenshot triage, Read a chart, Chess puzzle, Receipt check, Breaking glass, Voice fraud check, Care monitor and Instrument tag. Runs are charged in credits, and the cost is shown first.