v0.1.0 · benchmarked · Oct 2026 → Present
Mute Reality
Mute the leaf blower.
Keep the birds.
Choose which sounds exist: a CPU-only, real-time engine that mutes or keeps nine classes of sound through your headphones, with a benchmark that scores every claim against ground truth — including the case where it fails.
Highlights
The problem.
Noise cancelling is all or nothing: it removes the leaf blower and the birds, the traffic and the friend talking to you. University of Washington research on 'semantic hearing' showed you could choose what to keep, but it never shipped as something a person can run.
Real-time audio also has a harder floor than most software. Each 16 ms frame has to be processed in less than 16 ms, and a bug doesn't produce a wrong number — it produces a loud noise in someone's ears.
What I built.
Every 16 ms hop goes through a perfectly reconstructing STFT. A 48K-parameter streaming network (GTCRN) splits speech from everything else, and small per-class spectral dictionaries learned with KL-NMF give each other class a mask over the residual. The output remixes the pieces with per-class gains that slew between keep and mute, so a toggle never clicks; with everything kept, the engine is transparent.
Focus mode keeps one enrolled voice with a two-speed gate: a fast per-frame classifier trained at enrollment against a cohort of other voices, and a slower speaker embedding as a tie-breaker. The cohort is 're-recorded in your room', so target and cohort differ in voice, not in processing. Everything that reaches your ears goes through sanitising, a state reset on anything non-finite, and a hard −6 dBFS limiter.
Every test scene is built from separate stems, so every output has an exact reference. The benchmark scores three sets: the generators the dictionaries were learned from, held-out recipes for the same classes, and real ESC-50 recordings.
Screens.
What I considered.
Rejected
A gate that compares speaker embeddings on one-second windows
Rejected after measuring. It never opened: the embedding is noise on windows that short, and the target's own voice sat in the cohort. An oracle gate reached +13 to +14 dB, which showed the gate, not the separator, was the bottleneck.
Rejected
Make the conservative mute guard the default
It halves the damage on unfamiliar sounds but costs about 4–8 dB on the flagship street scene. It ships as --conservative, with both columns in the benchmark.
Chosen
Score every claim against ground-truth stems, held-out sounds included
It is how the in-distribution win and the held-out failure both became visible. The cost is a README whose headline table contains negative numbers.
Results.
- make demo, scored against ground-truth stems: +5.3 dB SI-SDR for muting the leaf blower on a street scene, +10.8 dB for keeping only the birds, and +4.5 dB for focusing one voice in a cafe against +0.4 dB for keeping all speech.
- Frame processing at 1.5 ms median (p95 1.9 ms) against a 16 ms budget; separation uses roughly 10–15% of one CPU core.
- Across sets, muting averages +3.7 dB in distribution and +3.1 dB on real ESC-50 recordings — but −4.5 dB on held-out sound recipes, where it makes things worse. That is in the README's first table.
- Hearing safety under test: NaN and infinite input, corrupted state, odd block sizes and a crashing model all leave the output finite and under the −6 dBFS ceiling. 61 tests.
Architecture
How it fits together.
- 01
Mic / file16 kHz mono
- 02
STFTsqrt-Hann · 16 ms hop
- 03
GTCRNspeech
KL-NMF9 class masks
- 04
Remixper-class gains · focus gate
- 05
Safetyreset · −6 dBFS · limiter → ears
In hindsight
Generalisation is the weak spot, and it is structural: the class dictionaries were learned from the same procedural generators that build the in-distribution scenes, so sounds unlike them get spread across the wrong classes and muting removes parts of what you kept. Every voice is synthetic, focus mode has never been measured on real people, and nobody has listened through physical headphones. The fix is a separator trained on real recordings. What I'd keep is the habit that exposed it — building every scene from stems, so a number can't flatter the method.
Tech specs.
- Type
- Project
- Where
- Personal
- When
- Oct 2026 → Present
- Duration
- 1 mo
- Status
- v0.1.0 · benchmarked
- Stack
- PythonONNX RuntimeNumPySciPysherpa-onnxPortAudioGitHub Actions
- Caveats
- Gains are measured against ground-truth stems. On held-out sound recipes, muting makes things worse on average (−4.5 dB); all voices are synthetic, and live audio has not been tested on physical hardware.