Audio Classification API: Ask Your Own Questions About Short Clips

Most audio classification APIs return labels from a fixed list. This post covers the other way to do it: send a short clip with your own typed questions and options, and get a probability for each option. It explains how the audio is read, how to write the questions, what the published evaluation says, and where this approach does badly.

What you can ask about a clip

The useful questions are short and about what is audible. Some examples:

• A home security microphone at 3 a.m.: which of shatter, slam, footsteps or bark can be heard?
• An inbound bank call: is this speech synthesized or voice-converted by a computer?
• A bedside microphone in a care home: which of cough, speech, alarm or snoring are present? (several can apply)
• A clip uploaded to a music library: which instrument is playing?

Each of these is a decision with a fixed set of options that you define, which is what separates audio classification from transcription. The broader set of approaches for images and audio is in the multimodal classification guide.

The alternatives, and when they fit better

Pretrained audio taggers such as YAMNet and PANNs score a clip against AudioSet's fixed list of sound classes. They are free, small and run almost anywhere. If the classes you care about are already on that list, a tagger plus a few mapping rules may be all you need.

CLAP and similar audio-text models compare a clip to label text, so you can use your own labels without training. You get a similarity score per label, and you tune a cut-off for each label yourself.

Speech-to-text, then a text model. If the question is about what someone said (a complaint, a cancellation, an address), transcribe first with Whisper or a hosted speech API and classify the transcript. This is the right path for calls longer than a few seconds.

Training your own model fits one sound in one environment with plenty of labelled recordings, such as a specific alarm on a specific site.

General multimodal LLMs accept audio and answer in text. They are flexible, but you parse the reply and you do not get a probability per option to set a threshold on.

A decision model such as Saina Helm 2 Omni 0.8B takes the clip, your state and typed questions (yes_no, single_choice, multi_choice, rating), and returns a probability for every option. Nothing is generated. Its trade-offs: labels change per request with no training, several questions are answered from one pass, decision mode abstains instead of guessing, and the open weights (Apache 2.0) can run on your own servers. Against that, clips are capped at 10 seconds and it is a small model; the evaluation below shows where it is weak.

How Helm 2 Omni reads audio

Each clip is converted to a 16 kHz log-mel spectrogram, a picture of how energy at each frequency changes over time, and read by the same vision tower that reads images. There are no separate audio weights.

That design has consequences you should know before you send anything:

• A 16 kHz signal carries frequencies up to 8 kHz, so anything higher is not heard.
• Clips can be up to 10 seconds. Longer clips are refused, never cut, so split recordings yourself.
• Formats: WAV, FLAC, MP3, Ogg, WebM; up to 10 MiB each and up to 4 clips per request. On the hosted API, clips go as base64 data URLs.
• A 10-second clip counts as 124 input tokens.
• It recognises sound patterns. It does not transcribe.

Writing audio questions

The model card gives two rules: say what the clip is in the state, and name the sounds plainly in the options. Short, literal questions about what is audible work best.

Compare two ways of asking the security question. "Is there a break-in?" asks the model to infer an event. "Which of these sounds can be heard in the clip?" with shatter, slam, footsteps, bark asks what is audible, and leaves the inference to your own rule (alert on shatter, log the rest).

An illustrative request for a bank call:

import base64, os, requests

with open('call.wav', 'rb') as f:
    clip = 'data:audio/wav;base64,' + base64.b64encode(f.read()).decode()

resp = requests.post(
    'https://api.saina.run/v1/ask',
    headers={'Authorization': f'Bearer {os.environ["SAINA_API_KEY"]}'},
    json={
        'model': 'saina-helm-2-omni-0.8b',
        'mode': 'decision',
        'state': 'Inbound call to the bank: a short recording of English speech. '
                 'The caller asks to raise their daily transfer limit.',
        'audios': [clip],
        'questions': {
            'spoof': {
                'type': 'yes_no',
                'question': 'Is this speech synthesized or voice-converted by a computer (a spoof)?',
                'threshold': 0.9,   # a starting value to tune on your own calls
            },
        },
    },
    timeout=30,
)
spoof = resp.json()['answers']['spoof']
if spoof['selected'] is not False:      # yes, or not sure either way
    require_extra_verification()

Note the rule in the code: an answer that does not clear the threshold is treated like a yes, so uncertainty leads to more checking, not less. The card is explicit that a synthetic-voice score is a reason for extra verification, never proof of fraud.

What the published evaluation says

The Helm 2 Omni model card reports a held-out evaluation: 69.4% of 5,572 single-answer questions on 4,120 clips from 21 public sources. By source:

Read it with care. Questions have 2 to 8 options, so chance differs by source: 70% on a two-option question and 70% on an eight-option question are not the same result. These are public datasets recorded in their own conditions. It is the card's evaluation, not a guarantee on your microphones, rooms or users.

What it does badly

• Long speech. Transcribing long speech is out of scope, and 10 seconds is a hard cap. Use a speech recogniser.
• Words. Spoken command words at 67% says it is not a speech recogniser either.
• Languages. Non-English is largely untested, and identifying the spoken language was the weakest source above.
• People. Identifying specific voices or people is out of scope.
• High frequencies. Anything above 8 kHz is not in its input.
• Machines. General machine-fault detection is the 70% row above. For machine recordings there is a separate mechanical specialist; read its card for its evaluation and its limits before you rely on it, because it is trained on a small number of machines.

Test it on your own clips

Record or collect a sample of clips from the microphones you will actually use, label them, and run them in distribution mode with the options you plan to ship. Look at accuracy per option and at what share of clips clears your threshold. Then try one alternative from the list above on the same set. The result on your own recordings is the only number that should decide this.

With a free account you can run the Breaking glass, Voice fraud check, Care monitor and Instrument tag presets in the Studio playground. Runs are charged in credits, and the cost is shown first.

Related posts

• Multimodal classification (pillar)
• Image classification without training