earshot fit: a sound file in, a Python recipe out, and no neural network in between

The post is two lines and two videos. Leon Lin (@LexnLin): "earshot v0.2 is out now! made a lot of changes and it can now turn a sound into code." The follow-up in the thread: "also new: 34 sound types, 183 sounds, all made with code. try it on your own sounds: earshot fit sound.wav -o recipe.py", with a 22-second video of real audio against code-made audio.

Update, 19 October: v0.3 adds a 201-sound bank, earshot sfx and earshot kit, and corrects one claim below about the seed banks. See the v0.3 section.

So I tried it on my own sounds. I cloned the repository at commit 9721326 (version 0.2.0 in pyproject.toml, MIT), installed it on a shared CPU, ran its tests, and fitted six targets. Two of them I synthesized myself, so I know the answer the fit should find.

What earshot is

earshot calls itself "ears for coding agents". Most of it is listening: describe, match, find, timeline, critique and sheet turn a sound file into words and numbers an agent can read. That half uses two open models, the CLAP text-audio embedder laion/larger_clap_general and the AudioSet tagger MIT/ast-finetuned-audioset-10-10-0.4593, about 1.4 GB on first use (reported). The figure below is earshot sheet on the project's launch film.

expand

The other half makes sound: earshot forge from code, earshot fit from a recording to code. Both are numpy and scipy.

What a "sound type" is

A sound type is a parameterized synthesizer. Each recipe module in earshot/forge/recipes/ defines a SPACE of named parameters with ranges, DEFAULTS, and a render(p, r) that builds the sound from those parameters and a random generator. The click recipe, for example, sums four layers and runs them through a finishing chain:

1. a 0.15 to 1.5 ms band-passed noise burst (the transient);
2. one to three damped partials at pitch (the "tink");
3. a short damped sine at 150 to 1200 Hz with a small pitch drop (the body);
4. a short band-passed noise tail (the grit), and an optional release click 25 to 90 ms later.

Then a high-pass, a dip at 2.5 to 4 kHz, light tanh saturation, a small space and an end gate. No recorded audio is used.

Twenty modules cover the 34 categories: 18 named after real-recording reference sets (whoosh_fast, click, impact, typing_key and so on) and 16 pro_* ones modelled on effects measured in professional launch films (reported, CHANGELOG.md). Each category is a family of four to six named characters, which are narrowed parameter ranges: click has soft, thock, tink, snap and double.

The "183 sounds" are not audio files in the repository, which holds no audio at all. They are parameter sets. Each recipe's seed bank in earshot/forge/seeds/*.json marks the members that were delivered, and those marks add up to exactly 183 across 19 files (measured). Because the parameters and the seed are committed, any of them re-renders: I rendered the pro_notify ding, a pro_pop chirp and the crisp fast whoosh from the bank and used them as fit targets below. (Corrected 19 October: the seed banks hold rounded parameters, so a re-render from them can differ from the delivered sound; v0.3 ships the exact ones. See the v0.3 section.)

Where do those parameters come from? From a search. earshot forge renders candidates from a recipe's SPACE, pre-screens them with model-free measurements (third-octave balance, envelope, artifacts), and sends the best to a critic that does use the models: CLAP's match to a text description against named alternatives, a realness score (logistic heads over CLAP embeddings and signal measurements, shipped as plain JSON), and closeness to licensed reference recordings that are not in the repository. An evolution strategy keeps the best and mutates them. That is where the machine learning in earshot lives: in judging sounds, not in making them.

How fit turns a recording into code

earshot fit is analysis by synthesis. It does not train anything. It measures the sound, fits small synthesis models whose parameters are things a sound designer would name, renders each one, and keeps the one whose render is closest to the recording. Four steps, from earshot/forge/fit/auto.py:

1. Name the class. character() measures attack, how long the sound takes to fall 40 dB, crest factor, tonal share (the energy in spectral peaks 12 dB over the local median), how persistent those peaks are from frame to frame, the glide of the strongest partial, and spectral flatness. From those it names a class: glide, ringing, struck, gesture or mixed. The class orders the candidate models.

2. Fit the candidates. There are three fitters, each a classic DSP method:

• modal.py for struck and ringing sounds: per detected onset, damped sinusoids found by ESPRIT on 15 sub-bands, amplitudes and phases by least squares, a fast "contact" made of very short modes, and band noise envelopes for whatever is left.
• sines.py for tonal sounds: McAulay-Quatieri partial tracking (STFT peaks joined frame to frame, glides kept), damped-sinusoid "atoms" at each strike, and the residual as noise in third-octave bands.
• spectral.py for noisy sounds: a time-varying 48-band noise gesture, stereo per band group, and texture statistics for crackle and flutter.

Hybrids stack them: the first layers keep only their deterministic part, and the next layer is fitted on what remains. Under 1 s all six candidates are fitted (the three fitters and three hybrids); at 1 s or longer, the class's first four.

3. Calibrate. Each candidate gets a post stage measured from its own renders: a smooth third-octave EQ onto the original's balance, the level, and a clamp that stops the render from standing above the original's envelope before the onset.

4. Score and keep the lowest. quality.score folds the metrics into one number:

Here mel is a multi-resolution log-mel distance in dB, tone is the RMS third-octave balance difference in dB, $r_\text{env}$ is the correlation of the 5 ms dB envelopes, onset is the octave error in onset sharpness, and excess and missing are dB of energy one sound has where the other is quiet. With --clap, $20\,(1 - \cos_\text{CLAP})$ is added. The weights were set on the same 12 recordings the project then measured on, which the documentation says itself (reported).

There is no gradient descent over synthesizer parameters and no neural network in the default path. The parameters come out of estimators (ESPRIT, least squares, peak tracking, band energies), and the only search is a choice among six candidates. I checked that bluntly: torch and transformers were not installed in my environment, and every fit ran.

The widget shows what the engine did on my six targets, read from the JSON each run wrote.

The class plan was overruled four times out of six. The ding and the pop were classed "struck", so modal went first; the twig was "mixed", also modal first; the whoosh was "gesture", spectral first. The partials model won all six, sometimes by a wide margin: 7.90 against modal's 25.65 on the ding (measured). The plan orders the work; the score decides.

What the emitted code looks like

This is the recipe fit wrote for my synthetic struck bar, abridged. The full file is 498 lines and 31.1 KB (measured). I cut the noise tables and replaced the absolute path in REFS with the file name.

"""synth_bar_fit: a forge recipe fitted from a real recording by `earshot fit` (sines model).
 
auto chose sines of sines (4.30), modal (5.33), hybrid:sines+spectral (5.36), spectral (12.13)
strike at 30 ms (1 strike(s)), 3 partials 523-2770 Hz, lowest strong 523 Hz, inharmonic (ratios 1.00, 2.70, 5.30), ...
 
Fit quality (original vs this recipe at DEFAULTS, seed 0):
  mel 1.74 dB, env r 1.000, tone 7.34 dB, onset err 0.03 oct (sharpness 47.9 vs 49.0 dB/ms), ...
"""
import numpy as np
 
from earshot.forge.fit import auto as _engine
 
TARGET = 'a bright ringing chime'
ALTS = []
REFS = ['synth_bar.wav']
KIND = 'sines'
 
SPACE = {
    'transpose': (-3.0, 3.0),   # semitones (partials and the residual spectrum together)
    'stretch': (0.85, 1.18, 'log'),   # time factor about the first strike
    'decay': (0.7, 1.43, 'log'),   # factor on every decay after its peak
    'brightness': (-3.0, 3.0),   # dB / octave about the partial centroid
    # ... vibrato, drift, residual, transient, vary, inharm
}
DEFAULTS = {'transpose': 0.0, 'stretch': 1.0, 'decay': 1.0, 'brightness': 0.0, 'vibrato': 0.0, 'drift': 0.0,
            'residual': 1.0, 'transient': 1.0, 'vary': 0.0, 'inharm': 0.0}   # the identity: re-renders the fit
 
MODEL = {
    'kind': 'sines',
    'sr': 48000,
    'duration': 1.0,
    'onsets': [0.03],
    # strongest first: birth t0 s, rise s, last breakpoint t1 s, release s, phase, [t, Hz], [t, dBFS]
    'partials': [
        {'t0': 0.03, 'rise': 0.00198, 't1': 1.00267, 'rel': 0.00533, 'ph': -0.005,
         'freq': [[0.03, 523.1], [0.976, 523.0], [1.00267, 521.9]],
         'amp': [[0.03, -6.0], [0.98133, -33.1], [0.992, -34.6], [0.99733, -37.3], [1.00267, -42.7]]},
        {'t0': 0.03, 'rise': 0.00198, 't1': 0.88533, 'rel': 0.00533, 'ph': 0.896,
         'freq': [[0.03, 1415.81], [0.048, 1414.01], [0.88533, 1413.66]],
         'amp': [[0.03, -12.1], [0.72533, -71.2], ...]},
        {'t0': 0.03, 'rise': 0.00198, 't1': 0.352, 'rel': 0.00533, 'ph': 2.041,
         'freq': [[0.03, 2770.28], [0.352, 2770.16]],
         'amp': [[0.03, -15.6], [0.32, -77.7], ...]},
    ],
    # per strike: damped sinusoid atoms [Hz, tau s, amplitude, phase]
    'transients': [{'t': 0.0295, 'atoms': [[15222.2, 3.47e-05, 7.6429916, -1.3443], ...]}],   # 31 atoms
    # noise in 1/3-octave bands (Hz centres): dB envelopes at `times`
    'residual': {'bands': [50.0, 63.0, ...], 'times': [...], 'db': [[...], ...]},   # 26 bands x 93 times
    'post': {'eq_db': [...], 'gain_db': 0.77, 'pre': {...}},
}
 
 
def render(p, r):
    """one take: p = knob values (missing keys take DEFAULTS), r = numpy Generator (noise / jitter)"""
    q = dict(DEFAULTS)
    q.update({k: v for k, v in p.items() if k in DEFAULTS})
    return _engine.render(MODEL, seed=int(r.integers(0, 2 ** 31 - 1)), **q)
 
 
def render_seed(seed=0, **knobs):
    """render with an explicit seed: render_seed(0) is the fit as measured"""
    q = dict(DEFAULTS)
    q.update(knobs)
    return _engine.render(MODEL, seed=seed, **q)

Three things to notice.

• The code is mostly data. The sound is the MODEL literal: three partials, each a few breakpoints of frequency and level. The two functions only hand it to the engine. Most of the 498 lines are the residual table, which here models the -70 dBFS noise floor I added to the target.
• It is not standalone. render() calls earshot.forge.fit.auto.render, so the module needs earshot installed. Each fitter also has a to_code() that writes a standalone numpy script, but the CLI writes the recipe module.
• The knobs are the point. transpose, decay and brightness are what make a recipe more useful than the WAV. earshot fit --forge recipe.py hands that SPACE to the forge's critic-driven search. The TARGET line is a guess for that critic, made from the fit; on the twig snap it guessed "an air whoosh".

The engine checks that the module's own render equals the model's render sample for sample. It did in all six runs (code_renders_fit: true, measured).

What I ran

A shared cloud VM: four cores of an Intel Xeon at 2.1 GHz, 15 GB of RAM, no GPU, other jobs running. A Python 3.12 venv with earshot installed from the clone and its DSP dependencies, but not torch or transformers: fit does not import them unless CLAP is asked for. Every command ran with two threads under nice -n 19.

The test suite (pytest, the fast set) gave 488 passed, 10 failed, 20 skipped in 237.68 s (measured). The 47 fit tests all passed. The ten failures are in test_interfaces.py: three need the mcp extra I did not install, and seven Gemini-client tests read their fake HTTP session's call log in a shape it did not have. None touch fit or forge.

The six targets:

1. Struck bar, synthesized by me: three damped modes, 523 Hz with a 0.30 s decay time constant, 1414 Hz with 0.10 s, 2771 Hz with 0.04 s, amplitudes 0.6, 0.3 and 0.2, a 1.5 ms noise contact at 30 ms and a -70 dBFS noise floor. 1.0 s.
2. Glide, synthesized by me: a tone rising from 600 to 1200 Hz over 0.25 s, a second harmonic at 0.3, a 5 ms rise and a 0.12 s decay. 0.5 s.
3. earshot's pro_notify ding, rendered from its seed bank. 0.26 s.
4. earshot's pro_pop chirp, from its seed bank. 0.162 s.
5. A twig snapping, a real recording: "Snapping twig (single)" by Stephan, CC0, from Wikimedia Commons, decoded to 48 kHz mono. 0.83 s.
6. earshot's fast whoosh (whoosh_fast, member crisp), from its seed bank, stereo. 1.06 s.

Each fit was one CLI call, earshot fit target.wav -o target_fit.py --json, with no --clap and no --report. Peak memory was 251 to 261 MB per call (measured).

Listen first

Every pair below shares one gain, so a level difference between target and fit is real. The strip under the players puts three distances on one scale, all ours and all log-mel L1: the target against the fit, two takes of the fit against each other (seed 0 against seed 1, the noise the model itself carries), and the target against white noise that follows the target's loudness contour, a model that knows the envelope and nothing about the spectrum.

All measured on 17 Oct 2026. Fit time is earshot's own; the whole CLI call, with Python start-up and file writing, took 6.64 to 38.22 s. The distance is 64 mel bands from 30 Hz to 20 kHz, an FFT of 2048 with a hop of 256, mean absolute dB over the cells within 60 dB of the louder sound's top. It is close to earshot's own mel_db but not the same metric; earshot's number for the same six pairs was 1.74 to 5.53 dB (measured).

How to read it (reasoned). On the tonal sounds the fit is far from its own seed floor, so the gap is the model missing something, not noise. On the whoosh the fit sits 0.27 dB above the distance between two takes of itself: for a noise sound that is about as close as any model with fresh noise can get. Every fit is closer to its target than the envelope-matched noise by 8 to 35 dB, so these are not just loudness contours.

expand

earshot also writes its own comparison next to every recipe. This is the one for the twig: the spectrograms, the 5 ms envelopes and the third-octave balance it uses in its score.

expand

Did it find the parameters?

The struck bar is the clean test, because I wrote down its parameters before fitting it.

The auto fit kept the partials model. Its three partials start at 523.1, 1415.8 and 2770.3 Hz, against 523, 1414 and 2771 (measured). The decay time constant is not stored directly, but each partial's level is a straight line in dB, so it falls out of two breakpoints: 27.1 dB over 0.951 s is a time constant of 0.305 s, 59.1 dB over 0.695 s is 0.102 s, and 62.1 dB over 0.29 s is 0.0406 s, against 0.30, 0.10 and 0.04 (reasoned from the measured breakpoints). The starting levels are -6.0, -12.1 and -15.6 dBFS, or 0, -6.1 and -9.6 dB relative, against 0, -6.0 and -9.5 for amplitudes of 0.6, 0.3 and 0.2 (reasoned).

Forcing --kind modal gives the exact answer, because modal is the model I synthesized the bar with: three modes at 523.0, 1414.0 and 2771.0 Hz, time constants 0.29998, 0.099976 and 0.040015 s, phases 0.0001, 1.0001 and 1.9982 rad against 0, 1 and 2 (measured). The auto engine scored that model 5.33 and the partials model 4.30, so it kept the partials. The modal fit's onset was twice as sharp as the target's, 103.2 against 47.9 dB/ms, and its recipe was 160 KB because it modelled my noise floor band by band (measured). The selection score cares about how the render sounds, not about which model is physically right, and here those disagree.

The glide is the weakest fit. The fundamental is tracked from 605.2 to 1200.6 Hz, against 600 to 1200 (measured). The second harmonic is not one partial: one track carries it from 1222 Hz to 0.273 s, a second is born at 0.137 s at 1667 Hz and carries it to the end, and a fourth, 40 dB down, wanders from 837 to 1190 Hz in between. That handover is the blip near 0.14 s in the spectrogram. The onset is blunted too: 61.5 dB/ms in the target, 16.7 in the fit, an error of 1.885 octaves, the worst of the six (measured). The documentation lists zips and long glides as untested, and a 0.25 s glide is where I would expect trouble to start.

On the real twig, envelope correlation was 0.894, the lowest of the six (measured), and the docstring's analysis says the 11 partials carry almost none of the energy ("residual -0.0 dB of the energy"): the snap is reproduced as third-octave noise envelopes with a few partials on top. That is a reasonable model of a snap. It is also why the generated TARGET reads "an air whoosh".

What surprised me

The partials model won everything. The documentation's own validation chose sines for 6 of 12 references and modal, hybrids or a sines-plus-spectral pair for the rest (reported). On my six it was sines every time, including the whoosh, where the class plan put the spectral noise model first and the whoosh's 11 partials sit between 109 and 442 Hz under a residual that holds most of the energy. Six sounds is a small sample, but here the tracked-partials model plus a band-noise residual was the general-purpose answer.

earshot's tone metric is not a quality score on sparse sounds. The struck bar fit reports tone_db 7.34, the second worst of the six, on the fit whose parameters are within 2.5% of the truth (measured). The bar has almost nothing between its three lines except a -70 dBFS floor, and a third-octave balance over those empty bands moves with the floor, not the sound (reasoned). The selection score weights it by 0.3, so it does not decide much.

Fitting is quick; long and dense sounds are not. The 0.162 s pop took 5.27 s; the README says 12 s for a 0.2 s pop on the author's machine (reported). The twig took 36.12 s, 14.27 of them in the three-layer hybrid alone (measured), and the documentation reports 250 to 340 s for 4.5 to 6.5 s impacts (reported). I did not fit the 4.8 s impact from the seed bank.

What I did not run, and the limits

• The quality claims. Median CLAP cosine 0.945 on 12 references, 10 of 12 with the same describe tag, and a 12-trial blind test in which one listener could not tell recording from render: all reported, on licensed recordings that are not public. I did not install CLAP, so I measured no cosine. One listener is a small test, and the project says so.
• Variations and morphs. I did not run --forge (it needs the CLAP critic and, for the forge's own categories, the reference sets), --morph, or the vary knob. The limits page says knob variations of a whoosh fitted as hundreds of modes "may sound metallic" (reported).
• The two videos in the thread. Real-against-code audio in a video is the author's comparison; I did not extract or measure it.
• Selection is in-sample. The score's weights were set on the same 12 references they were checked on, and the selection has no held-out test yet (reported).
• Not tested upstream: zips and long glides, risers, swipes, shimmers and stereo-heavy designed sounds (reported). My glide falls in that gap and was the weakest fit here.

Where it fits

earshot is aimed at the same agents that write launch videos in code; the one-shot launch videos piece looked at skills that ship 260 bundled sound effects to do that job, which is the problem forge and fit try to replace with parameters. For neural audio generation on the same kind of CPU, speech.cpp runs six speech model families in ggml, and Paradee distils a text-to-speech model to 8M parameters. earshot is the other end of audio: short effects, where a few partials and some band noise are enough and no network is needed to make the sound.

The honest summary of fit: it is careful, well-documented classic DSP, analysis by synthesis over three model families, with a hand-weighted score to choose between them. It recovered a known sound almost exactly, reproduced noise sounds about as closely as noise allows, and was weakest on a pitch glide. The code it writes is a readable parameter set plus a call into earshot, and its value is the knobs, not the bytes.

18 October: v0.3, kits and frame-timed effects

A day after this piece, Leon posted v0.3: "a library of 201 sounds, all made with code", earshot kit "can create a whole sound kit", earshot sfx "times a sound to the exact frame", and earshot now makes the sound for cinetic, his motion-design skill. I cloned it at 925ebe5 (version 0.3.0; 57 files and 5,656 added lines since 9721326), reinstalled it the same way on the same shared CPU, still without torch, and ran the new commands on the project's own examples.

The 201 sounds. Counted in the three bank files: 183 in forge_v2.json, 15 in fit.json, 3 in pluck.json (measured). The 183 are the v0.2 forge finals, now stored as complete render blocks at full precision; the build refuses to write one that does not re-render its delivered WAV (reported). That corrects a claim above. I rendered three of my targets from the seed banks, and the docs now say those hold rounded starting points: "re-rendered from them, 34 of 94 checked finals come out as different sounds" (reported). My fits are fits of real renders, but maybe not of the delivered sounds, and my published Opus files cannot settle it: Opus alone moves my distance as much (measured). The 15 new entries are CC0 recordings fitted as code (clicks, keys, pops, a domino, a bell, a short whoosh), each with its source. The 3 are tuned kalimba voices.

earshot sfx, a command 0.2 did not have, renders one bank sound aimed at the picture. The knobs edit the recipe's own parameters; nothing is time-stretched. For the whoosh family, the time to the loudest 5 ms (apex) is solved in closed form, corrected by up to two measured renders, then placed by trimming the quiet approach or padding silence; dur lands to the sample. The docs' example, whoosh_fast/crisp --dur 0.8 --apex 0.32, came out 0.800 s long with its loudest 5 ms at 0.3205 s, against 0.1305 s for the delivered take (measured, by earshot and again by my own window). At 60 fps a frame is 16.7 ms, so the 0.5 ms miss is 3% of a frame (reasoned). The call took 8.5 s, mostly start-up; the docs give 1.9 s warm. Every render also runs a model-free audit for clicks, clipping, DC, comb ring, cut tails, silence and rising pitch glides.

earshot kit. A kit is a JSON file naming one bank sound per film role (tick, hit, whoosh.short, key, bell, text and so on), with takes, knobs, a sync point and a mix level. kit init --preset calm sounded 17 roles and switched off 4 (riser, swell, suck, drop). kit render made 64 takes in 28.2 s, re-rolled one key take that failed the audit, and wrote a lock file with each take's measured onset, peak, end and loudness. kit check passed 64 of 64, and a fully cached re-render took 0.8 s (all measured). Six calm roles use fitted recordings, picked where earshot's CLAP match and realness readings preferred them (reported; I did not install CLAP to recheck).

cinetic is an MIT agent skill that directs short launch films in Remotion or HyperFrames. Its scripts/audio/kit.py finds an earshot interpreter (EARSHOT_PYTHON, or earshot on PATH) and shells out to earshot kit render and kit check; score.py puts the takes on the cue frames using the lock's sync points. Without earshot it plays takes already in the project, then synthesized effects with a warning. I read that code; I did not run cinetic or measure the launch video.

Did fit change? Only its takes. A render with vary above 0 now lets modes and partials ring out past the trimmed recording and high-passes at 8 Hz, and --takes scores candidates over seeds 0 to 2. I fitted one fresh struck bar with both versions: same model, same three partials, candidate scores within 0.03, earshot's mel distance 1.001 dB in 0.2 and 0.996 in 0.3, renders differing at -60 dB from the residual's new roll-off above 16 kHz (measured). The fit, bank, sfx and kit tests gave 133 passed and 4 skipped in 247 s, including 60 fit tests, 9 of them new for takes (measured).