Sound Event Detection for Alerts

A microphone in a shop at night hears the fridge, a car outside, the shutter rattling, and maybe once a year a window breaking. Sound event detection is the job of saying which of those happened. For an alert system, the hard part is staying quiet for everything else, so the one alert that matters still gets read.

This guide covers writing the sounds as options, keeping doubtful clips away from the pager, and feeding a stream into a model that takes clips of at most 10 seconds. The examples use Saina Helm 2 Omni, but most of it applies to any audio classifier.

What you are actually deciding

Research on sound event detection often means labelling sounds with start and end times in a long recording. Alerting usually needs less: is the sound I care about present in this short clip? That is clip-level tagging, and the time resolution is your window length.

Two shapes cover most alert rules:

• One sound, yes or no. "Can breaking glass be heard in this clip?"
• Several sounds at once. "Which of these can be heard?" as a multi_choice question over the target sounds and the usual confusions. Each sound gets its own score, so a clip can contain a cough and speech at once.

The second shape is usually better for alerts, for a reason covered next.

Write the sounds plainly, and include the confusions

The Omni model card's advice for audio: say in the state what the clip is, name the sounds plainly in the options, and ask short, literal questions about what is audible. In practice:

• State the source. "The clip is a short recording from a home security microphone in a kitchen at 3 a.m." tells the model what kind of audio to expect.
• Name sounds by what they sound like. "Glass shattering or breaking" rather than "intrusion event". The model hears sound, not intent.
• List the sounds that get mistaken for your target. A door slam, a dropped pan, cutlery in a drawer. If the only option on the list is glass, a loud bang has nowhere else to go. With a slam on the list, the probability for a slam can be high while the score for glass stays low, and your rule fires only on glass.

The Breaking glass preset in the Studio playground is built this way. Its state is a home security microphone at 3 a.m., and the options are shatter, slam, footsteps and bark, so a bang that is not glass has somewhere to go.

The care-home case has the same structure. Ask which of cough, speech, alarm and snoring can be heard in a clip from a bedside microphone at night, and write a rule that wakes the night nurse for coughing fits, not for every sound in the room. That is a prompt for a person to go and look: Omni's card lists final, unreviewed medical decisions as out of scope.

Thresholds: what goes to the pager and what goes to the log

Alert fatigue is a thresholding problem: if every clip that might contain glass pages someone, the pages stop being read.

With "mode": "decision" and a threshold, a multi_choice answer lists in selections only the sounds whose score reached the threshold, and comes back with reason: "below_threshold" when none did. A yes_no or single_choice answer that falls short comes back with no selection and a reason (below_threshold, below_margin or tie) instead of a guess. That gives you three outcomes to route:

1. Target sound accepted: alert.
2. Nothing accepted, or the target scored in a middle band you define: log the clip for review in the morning, or send it to a low-priority queue a person scans. No pager.
3. A known harmless sound accepted: log it or drop it.

Some rules that tend to help, all to be tuned on your own recordings:

• Per-question thresholds. A threshold can be set on the request or on each question. Use a strict one on the question that wakes people.
• Two windows before paging. Require the target in two overlapping windows, or in one window plus a second signal (a door contact, a motion sensor).
• Context in the state. Say whether the building should be empty; the questions can use it.

A request

Illustrative; the request shape follows the API docs, and the base64 string is cut short.

import base64, requests

def as_data_url(path, mime='audio/wav'):
    return f'data:{mime};base64,' + base64.b64encode(open(path, 'rb').read()).decode()

r = requests.post('https://api.saina.run/v1/ask',
    headers={'Authorization': f'Bearer {API_KEY}'},
    json={
        'model': 'saina-helm-2-omni-0.8b',
        'mode': 'decision',
        'threshold': 0.8,
        'state': 'The clip is a short recording from a home security microphone '
                 'in a kitchen at 3 a.m. Nobody should be in the building.',
        'audios': [as_data_url('window_0412.wav')],
        'questions': {
            'heard': {
                'type': 'multi_choice',
                'question': 'Which of these sounds can be heard in the clip?',
                'options': {'glass': 'Glass shattering or breaking',
                            'slam': 'A door slamming',
                            'drop': 'A heavy object dropped on the floor',
                            'alarm': 'An alarm or siren',
                            'voice': 'A person speaking or shouting'},
            },
        },
    })

heard = r.json()['answers']['heard']
if 'glass' in (heard.get('selections') or []):
    page_on_call(heard['memberships'])
elif heard['memberships']['glass'] > 0.3:          # middle band: an example to tune
    queue_for_review(heard['memberships'])

memberships holds a score for every option whether or not it was selected, so the middle band is your own rule on top of the model's threshold.

Windowing a stream into 10-second clips

Omni takes audio clips of up to 10 seconds, 10 MiB each, as base64 data URLs in the request body. A longer clip is refused, never cut, so a live stream has to be split before it is sent.

• Fixed windows with overlap. For example, 5-second windows starting every 2.5 seconds (illustrative values). Overlap means a sound that straddles a boundary appears whole in at least one window.
• One window per request. A request can carry up to 4 clips, but every question sees all of them and gets one answer. To get a separate decision per window, send each window in its own request.
• Gate before you send. A simple loudness gate on the device (skip windows that stay near the room's background level) means quiet windows are never sent. Glass breaking is not quiet.
• Know what the model hears. Each clip becomes a 16 kHz log-mel spectrogram, so sound above 8 kHz is not in the input. For most everyday sound events that is fine; for anything you know lives high in the spectrum, test first.
• Cost scales with clips. Media are billed as input tokens; a 10-second clip is 124 tokens.

What the published numbers say

The Omni card reports a held-out audio evaluation: 4,120 clips from 21 public sources, 5,572 single-answer questions, with 69.4% answered correctly overall. Two sources are close to alerting work:

Questions have 2 to 8 options, so chance differs by source, and these are clean public datasets, not your microphone in your room. Record a few dozen clips of the real target sounds and the real background where the microphone will hang, label them, and measure before you connect a pager.

When something else is the better choice

• A dedicated glass-break sensor. For intrusion alarms, purpose-built acoustic glass-break detectors are cheap, run without a network, and fit into alarm panels and monitoring contracts. If glass breaking is your only sound, start there.
• An audio tagging model on the device. Open models trained on AudioSet, such as YAMNet or PANNs, tag hundreds of fixed sound classes and run on small hardware. If your sounds are in their label set and you want everything on the device, they are a good fit.
• Training your own detector. If you have a few hundred labelled clips of your target and your background, a small classifier trained on them may do better than a general model on that one task.

A decision model fits when the sounds you care about change from site to site, when you want to describe them in words instead of retraining, or when the sound is one input among several (time of day, which door sensor fired, who should be in the building) in the same request.

Try it

With a free account, the Studio playground has Breaking glass and Care monitor presets. Runs are charged in credits, and the cost is shown first. The model card is at run-saina/saina-helm-2-omni-0.8b.

Related posts

• Multimodal classification (pillar)
• Audio classification API