# Learnable Spectral Activations: stop picking frequencies, learn how to mix them

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/learnable-spectral-activations
> date: 2026-10-23
> tags: 3d, representation-learning, computer-vision, audio, benchmarks, reproducibility

An implicit neural representation (INR) stores a signal as a function. Give a
small MLP a coordinate (a time stamp, a pixel position, a point and a view
direction) and it returns the value there (an audio sample, an RGB triple, a
density and a colour). There is no grid. You can query it between samples, take
its derivative analytically, and optimise through it. NeRF is the famous
instance; signed distance fields, like the one inside [OREN's octree
mapper](/articles/oren-sdf), are another.

The catch is old and well known. A plain ReLU MLP fed raw coordinates learns
smooth functions first and fine detail last, sometimes never. That is
**spectral bias**: gradient descent fits the low-frequency part of the target
much faster than the high-frequency part. A photo through a ReLU MLP comes out
blurry, and an audio clip comes out as mush.

*Learnable Spectral Activations* (LSA), by Tamir Shor, Or Litany and Alex
Bronstein (Technion, NVIDIA, ISTA), [arXiv 2610.07419](https://arxiv.org/abs/2610.07419),
NeurIPS 2026, comes with code. Litany's pitch on X: prior INR work tunes which
frequencies are available; LSA learns how to compose them. Below: what that
means for one neuron, the tables checked against the paper and code, and a
small CPU run.

<RepoCard repo="tamirshor7/Learnable-Spectral-Activations" />

## Every fix so far changes the frequency menu

The cleanest way to read INR design is the *structured dictionary* view of Yüce
et al. (2022), which the paper adopts. Feed a network a sinusoid of frequency
$\omega$, push it through a nonlinearity $\rho$, and you get harmonics:
$\rho(e^{j\omega x}) = \sum_k \alpha_k e^{jk\omega x}$. Stack layers and the
network's output is a weighted sum of sinusoidal **atoms** whose frequencies
come from a set $\mathcal{H}(\Omega)$ the architecture induces:

$$
f(x) = \sum_{\omega' \in \mathcal{H}(\Omega)} c_{\omega'} \sin(\langle \omega', x \rangle + \phi_{\omega'})
$$

Read the popular fixes through that lens and they all edit the menu:

- **Positional encoding / Fourier features** (NeRF; Tancik et al.) prepend
  $\gamma(x) = [\sin(2\pi Bx), \cos(2\pi Bx)]$. The matrix $B$ is the base
  frequency set. With random $B_{ij} \sim \mathcal{N}(0, \sigma^2)$, $\sigma$
  is a bandwidth knob: larger exposes higher frequencies and risks aliasing
  between samples.
- **SIREN** uses $\sin(\omega_0 u)$ as the activation, with $\omega_0 = 30$.
  $\omega_0$ sets the bandwidth the network can resolve.
- **FINER** uses $\sin(\omega_0(|u|+1)u)$: the local frequency grows with the
  magnitude of the pre-activation, so the bias initialisation picks which
  frequencies are reachable.
- **WIRE**, sinc activations and learned initialisations change the atoms'
  shape or where training starts.

The paper's argument is that all of these decide **which** atoms exist and
leave the neuron's own waveform prescribed. A SIREN unit's output is
$\sin(\omega_0 w^\top z)$: one sinusoid of the projection $w^\top z$. Its
harmonic content is fixed by the same weights $w$ that choose which feature the
unit looks at. To build an edge, which needs many harmonics in fixed amplitude
ratios, the network has to recruit several such units and line them up. The
waveform and the feature are welded together.

## The activation

LSA unwelds them. At hidden layer $\ell$ the activation is a residual,
truncated Fourier series of the scalar pre-activation $u$ (paper, Eq. 2):

$$
\phi^{(\ell)}(u) = u + \sum_{k=1}^{K} a^{(\ell)}_k \sin(2\pi k u)
$$

- $u = w^\top z + b$ is one neuron's pre-activation.
- $K$ is the number of harmonics: 16 for audio, 32 for images, 24 for NeRF and
  8 for the acoustic fields.
- $a^{(\ell)}_k$ are learnable and **shared by every neuron in the layer**, so a
  layer adds $K$ parameters, not $K \times$ width.
- The leading $u$ is a residual identity. The coefficients start near zero
  (standard deviation 0.01 for audio, $10^{-3}$ for images), so the network
  begins as an almost linear MLP and grows its nonlinearity during training.

Three properties fall out of the form.

**The waveform is decoupled from the feature.** $w$ still decides what the
unit looks at. The coefficients decide what it does with it: the same $u$ goes
through $K$ harmonics at once, and their relative amplitudes are learned. One
unit can emit a square-ish wave, a sawtooth, or something with a notch.

**It is affine in the coefficients.** For fixed pre-activations, $\phi$ is
linear in $a$. The gradient for $a_k$ is the residual's correlation with the
propagated harmonic $\sin(2\pi k u^{(\ell)})$. The paper writes this as an
exact finite-width identity for full-batch gradient flow (Eq. 4):

$$
-\dot{\mathcal{L}} = \eta_w \lVert J_w^\top r \rVert^2 + \eta_a \lVert \Psi^\top r \rVert^2
$$

where $r$ is the residual, $J_w$ the Jacobian with respect to ordinary weights
and $\Psi$ the Jacobian with respect to the harmonic coefficients. The second
term is extra descent the coefficients add whenever the residual has any
projection onto the harmonic directions. Equivalently, the empirical neural
tangent kernel gains a positive semidefinite term $\eta_a \Psi_0 \Psi_0^\top$
of rank at most (layers $\times K$). That is the formal content of "spectral
tuning becomes a more direct subproblem".

**It does not expand what the network can represent.** The paper is careful
here and I will be too: a wide enough periodic INR can already represent the
same signals. LSA changes the *factorisation*, not the function class. The
claim is about optimisation: the same target is easier to reach.

The widget below is one LSA activation with $K = 6$. Left: $\phi(u)$ against
the identity it starts from and against a single sine. Right: its spectrum,
which for this form is just the coefficient vector, one bar per harmonic. A
SIREN unit has one bar.

<HarmonicComposer />

The **fit edge** button makes the affine point concrete. The target is a soft
square wave, $0.45 \tanh(8 \sin 2\pi u)$, added on top of $u$. Because $\phi$
is linear in $a$, the best coefficients are a least-squares solve, not a search.
The best root-mean-square gap with only the first harmonic, which is all a
single sine can do, is 0.155 against a target RMS of 0.431. Adding harmonics
two and three brings it to 0.089, and five or six bring it to 0.057 (measured,
in the widget and in numpy). The even harmonics get zero weight: the target is
an odd square wave, and a square wave has no even harmonics. A SIREN or FINER
network gets there too, but by recruiting more units at different frequencies
and phases, which costs width and gradient steps.

The paper's own version of this is Figure 1a: a 1D target fitted with a budget
of one to four atoms.

<Figure src="https://media.thesatyajit.com/ai/articles/learnable-spectral-activations/fig1.png?v=a0e8771a1fbc2f99" alt="Three rows of four panels each, SIREN, FINER and Learned Harmonic, fitting the same wiggly 1D signal with one to four atoms. SIREN reaches 10.8, 14.8, 20.0 and 103.0 dB PSNR; FINER reaches 11.1, 16.0, 19.1 and 25.4 dB; the learned-harmonic row reaches 120.0 dB with a single atom." caption="A controlled 1D phantom with a limited atom budget. SIREN and FINER improve as atoms are added; the learned-harmonic unit fits with one. Read it as an illustration: the target looks like a short sum of harmonics of one base frequency, which a single learned-harmonic unit can represent exactly (LSA paper, Figure 1a)." />

Take the 120 dB with a grain of salt. The phantom appears to be built from a
few harmonics of one fundamental, which is the one case where a single
learned-harmonic unit is an exact fit (reasoned from the figure; the paper does
not give the phantom's formula). It shows the mechanism, not a margin.

## What the released code actually does

The repository ships the audio and 2D image experiments only. NeRF and the
neural acoustic fields are not in it (measured, at commit `35a040b`, 7 October
2026). It also has no licence file. The `THIRD_PARTY.md` lists the FINER, STAF
and SL2A sources, and the STAF and SL2A licences are kept, but the repo itself
grants nothing, so treat it as all rights reserved until that changes.

The image activation in `image_2d/lsa/models.py` is ten lines:

```python title="image_2d/lsa/models.py (released)"
class SpectralActivation(nn.Module):
    def __init__(self, harmonics=32, init_scale=1e-3, residual=True):
        super().__init__()
        self.harmonics = int(harmonics)
        self.residual = bool(residual)
        self.coeffs = nn.Parameter(init_scale * torch.randn(self.harmonics))

    def forward(self, x):
        y = 0.0
        for i in range(self.harmonics):
            y = y + self.coeffs[i] * torch.sin(2.0 * math.pi * (i + 1) * x)
        return x + y if self.residual else y
```

Two details matter that the equations do not show (measured from the code):

1. **The forward pass is a Python loop over $K$.** Every harmonic is a separate
   `sin`, a separate multiply-add and a separate saved tensor for autograd.
   That is the cost the paper's resource appendix measures, and the cost my CPU
   run hits.
2. **The first layer has no $\omega_0$.** Its weights and biases are uniform in
   $\pm 1/2$ (that is, $\pm 1/\text{fan-in}$ with two input coordinates), and the
   activation multiplies the pre-activation by $2\pi k$ for $k$ up to 32. So the
   first LSA layer is, in effect, a Fourier-feature bank along each neuron's
   direction $w$, with frequencies up to $2\pi \cdot 32 \cdot |w|$ and amplitudes
   learned per layer (reasoned). Hidden layers use SIREN's initialisation bound,
   $\sqrt{6/256}/30$. The harmonic basis is where the high frequencies come from,
   which sits a little awkwardly next to "LSA does not change which frequencies
   are available". It does not change the asymptotic menu; it does change what a
   finite network reaches from initialisation.

The training loop puts the coefficients in their own Adam group at a higher
learning rate: twice the weight rate for images, three times for audio
(`act_lr_mult` in `image_2d/configs/kodak_core.yaml` and `audio/run_core.py`).

## Checking the results

All numbers in this section are **reported**: the paper's, on an NVIDIA RTX
A6000, not re-run. Where I combine them, it is **reasoned**.

### Images: Kodak

Twenty-four 768×512 photos, raw coordinates, width 256, four nonlinear layers,
5,000 Adam updates, seed 0, best PSNR along the trajectory. Mean PSNR:

| Method | Mean PSNR (dB) | Setup |
|---|---|---|
| LSA ($K = 32$) | 38.110 | 199,043 params, 5,000 updates |
| FINER | 37.317 | 198,915 params, 5,000 updates |
| WIRE | 36.596 | up to 10,000 updates |
| STAF (global config) | 35.369 | matched budget |
| MFN GaborNet | 35.151 | up to 10,000 updates |
| SIREN | 34.990 | 198,915 params, 5,000 updates |

LSA beats FINER on 21 of 24 images and SIREN on all 24. The gap over FINER is
0.79 dB (reasoned). The paper selects LSA's configuration on six calibration
images (kodim02, 05, 07, 10, 13, 23); on the other 18, LSA reports 38.045 dB.

<Figure src="https://media.thesatyajit.com/ai/articles/learnable-spectral-activations/fig3.png?v=d05f79684114c034" alt="Bar chart of best PSNR for each of the 24 Kodak images, four bars per image for FINER, SIREN, STAF and LSA. Legend means: FINER 37.32, SIREN 34.99, STAF 35.37, LSA 38.11. LSA's green bar is highest on most images, FINER's blue bar close behind." caption="Per-image Kodak PSNR under fixed configurations and 5,000 updates. The error bars are not defined in the caption or text; each image is a single seed-0 fit (LSA paper, Figure 4)." />

Three things to hold the table to:

- **One seed.** Every image fit is seed 0. The paired statistics are across
  images, not across initialisations. With 21 of 24 wins the image-level result
  over FINER is not noise, but the 0.79 dB is a one-seed number.
- **The STAF and SL2A comparisons are protocol-sensitive.** Against full SL2A
  the holdout gap is 0.188 dB with ten wins and eight losses; the paper's own
  sign test gives $p = 0.815$. Against SL2A at matched parameters (199,171) the
  gap is 2.135 dB and LSA wins all 18. So "LSA beats SL2A" depends on which SL2A.
- **The ablations are informative.** $K = 4$ gives 30.45 dB on the six
  calibration images, $K = 32$ gives 39.50 dB and $K = 64$ gives slightly less.
  Freezing the coefficients at their near-zero initialisation drops held-out
  PSNR from 38.05 to 19.91 dB. That second number is not "LSA versus a fixed
  good activation". A frozen near-identity activation leaves an almost linear
  MLP, which cannot fit a photo at all (reasoned). It proves the coefficients
  are doing the work, which is the point of the ablation.

<Figure src="https://media.thesatyajit.com/ai/articles/learnable-spectral-activations/fig4.jpg?v=db45e41598a675e1" alt="Two Kodak images, a lighthouse and a mountain stream, each reconstructed by LSA, FINER and SIREN, with a row of spatial residual maps and a row of Fourier-domain residual maps underneath. Titles: lighthouse 38.67, 36.92, 34.52 dB; stream 31.91, 30.75, 27.77 dB. LSA's residual maps are darkest." caption="kodim21 and kodim13 with spatial and spectral residuals on a shared display scale. The paper notes these two were chosen for clarity; the dataset-level evidence is the table above (LSA paper, Figure 6)." />

### Audio: where the gap is largest, and needs Fourier features

One-second clips at 48 kHz, 10 from NSynth and 40 from LibriSpeech, width 256,
four layers, 5,000 full-batch updates. Each method gets the same 16 Gaussian
Fourier-feature bands at each bandwidth $\sigma$, and the paper sweeps
$\sigma$ from 0 (raw coordinates) to 100.

<Figure src="https://media.thesatyajit.com/ai/articles/learnable-spectral-activations/fig2.png?v=ae40343549f5f397" alt="Two line charts of mean PSNR against Fourier-feature bandwidth sigma from 0 to 100, for NSynth and LibriSpeech. On both, LSA starts below FINER at sigma 0, then climbs above it from sigma 20, reaching roughly 87 dB on NSynth and 91 dB on LibriSpeech at sigma 100. FINER stays in the 60s. SIREN is lowest and drops when Fourier features are added." caption="On-grid audio PSNR against Fourier-feature bandwidth. Shaded bands are standard deviations across clips (LSA paper, Figure 2)." />

At $\sigma = 100$ LSA reports 86.72 dB on NSynth and 90.78 dB on LibriSpeech.
The strongest STAF setting reaches 68.79 and 68.40 dB; FINER reaches 67.19 and
66.32 dB. Read three things off the figure before quoting those:

- **At $\sigma = 0$ FINER wins.** With raw coordinates LSA is below FINER on both
  datasets. The paper says so and explains it: LSA reshapes the frequencies it
  is given; it does not add a new input basis. The audio win is LSA plus
  Fourier features against FINER plus Fourier features.
- **The spread is large.** LSA's 86.72 dB on NSynth carries a standard deviation
  of 21.18 dB across ten clips; LibriSpeech's 90.78 carries 18.57.
- **The absolute numbers are past audibility.** 90 dB PSNR on a unit-peak
  waveform is an RMS error of about $3 \times 10^{-5}$, roughly one 16-bit
  quantisation step (reasoned). Both LibriSpeech and NSynth are distributed at
  16 kHz and are resampled to 48 kHz here, so the grid is oversampled three
  times over. These are
  memorisation benchmarks. The more useful audio result is the off-grid one:
  trained on 16,000 samples and evaluated at the midpoints, SIREN with large
  $\sigma$ collapses between samples and LSA does not (paper, Figure 3; the
  figure gives no table to check).

The released README notes that the core audio targets are zero-mean and
unit-peak, while the appendix says amplitudes are used "on the decoded
floating-point audio scale". It is applied identically to every method in the
core comparison, so it does not tilt the table, but the two descriptions do not
match (measured from both files).

### NeRF: a tie on PSNR, a small SSIM edge

Blender, eight scenes, FINER's reduced-view protocol (every fourth training
image, 200×200), an Instant-NGP-style backbone with width 182, 150,000 steps,
final checkpoint. LSA uses $K = 24$.

| Mean over 8 scenes | PSNR (dB) | SSIM |
|---|---|---|
| SIREN | 28.99 | 0.932 |
| FINER | 29.26 | 0.932 |
| LSA | 29.29 | 0.938 |

The means are reasoned from the paper's Table 1. LSA is best on PSNR in five
scenes and FINER in the other three; the mean PSNR gap to FINER is 0.03 dB,
with Lego going 0.65 dB the other way. SSIM is the more consistent signal:
LSA is best on seven of eight scenes. The paper calls these gains "moderate",
which is accurate. One run per scene, and LSA's learning rate is three times
the baselines' ("best stable settings"), so this is a parity result with a
mild structural-similarity edge, not a NeRF upgrade.

### Neural acoustic fields

Here LSA ($K = 8$) replaces the activation in the NAF decoder and nothing else.
Log-spectral L1 improves by 44.90% (macro over four scenes), MRSTFT log-L1 by
45.97% and the interaural time-delay error from 122.4 to 39.7 samples; waveform
PSNR improves by 0.43 dB. The largest relative effect in the paper, and the
one I can least check: the NAF code is not released, and the baseline
(waveform PSNR 11.417 dB) is far from the ground truth.

## What it costs

Here the paper is direct. The parameters are free: depth 4 with $K = 16$ adds
64 scalars to a 206,337-parameter audio MLP. The compute is not. On full-grid
48k audio fitting with 16 Fourier bands, an A6000 step takes (reported, Table
16):

| Depth 4, width 256 | Train step (ms) | Train peak mem (GB) | Eval (ms) |
|---|---|---|---|
| SIREN | 8.26 | 0.48 | 3.56 |
| FINER | 14.20 | 0.85 | 5.55 |
| LSA, $K = 16$ | 97.93 | 7.53 | 35.08 |

That is 6.9× FINER's step time and 8.9× its memory (reasoned; the paper rounds
to 7× and 9×). Both scale linearly in $K$: at $K = 64$ a step takes 341.09 ms
and 29.55 GB. The cause is the one in the code: the activation evaluates
batch × width × layers × $K$ sines and autograd keeps each one. An
"accumulate" implementation cuts inference memory, and the paper points at a
fused kernel as future work. The math is a natural fit for one: the
derivative of $\sum_k a_k \sin(2\pi k u)$ is $\sum_k 2\pi k\, a_k \cos(2\pi k u)$,
and both sums can come from one Chebyshev-style recurrence on $\sin$ and
$\cos$ of $2\pi u$ without storing $K$ tensors (reasoned; nobody has shipped it).

The paper leaves quality against wall clock to future work. That is the
experiment I could run.

## Running it: one image on a CPU

**Setup (measured).** The released `LSAImageMLP` and the released SIREN and
FINER classes (`image_2d/third_party/finer.py`), imported from the clone; my
own training loop mirroring `image_2d/lsa/train.py` (Adam, learning rate decayed
to 0.1× over the run, LSA coefficients at twice the weight rate, targets in
$[-1, 1]$). A fourth baseline is a ReLU MLP with NeRF-style positional encoding
($L = 10$ octaves, width trimmed to 121 to match parameters). The image is
scikit-image's `astronaut` (NASA, public domain) at 128×128, full batch.
Width 128, four nonlinear layers, $K = 32$. Every model has about 50,000
parameters. Hardware: a shared 4-core Intel Xeon at 2.10 GHz cloud VM, CPU only,
PyTorch 2.14.1, two threads, `nice 19`, with other jobs running.

Total compute for the run was about 28 minutes of wall time, over the
20-minute budget I set myself, almost all of it the LSA model.

| 500 steps, ~50k params | Final PSNR (dB) | ms per step | Training time (s) |
|---|---|---|---|
| FINER | 41.66 | 97 | 48.5 |
| SIREN | 37.80 | 82 | 41.0 |
| LSA, $K = 32$ | 35.12 | 2,767 | 1,383 |
| ReLU + positional encoding | 24.75 | 50 | 25.0 |
| SIREN, run for 2,000 steps | 48.61 | 92 | 184 |

<FitCurves />

What the run shows (all measured, one seed, one image):

- **At 500 steps LSA is behind, not ahead.** 35.12 dB against SIREN's 37.80
  and FINER's 41.66, and still climbing when the budget ends. That does not
  contradict the paper so much as sit at the start of its own curve: the
  paper reports LSA ahead of FINER on only 5 of 24 Kodak images at step 500,
  rising to 21 of 24 by step 2,000. I ran a smaller network on a smaller image
  for a tenth of the steps, with the paper's learning rates rather than tuned
  ones. It also says I could not reproduce "LSA separates from SIREN early" at
  this scale: the paper has LSA ahead of SIREN on 23 of 24 images at step 500.
- **Per step, the released activation is 26 to 34 times slower than SIREN on
  this CPU.** 1,614 ms against 62 ms in a quiet ten-step benchmark, 2,767 ms
  against 82 ms over the full run with the machine loaded. The paper's A6000
  ratio against SIREN is 11.9× (97.93 / 8.26 ms, reasoned); a CPU has no spare
  parallelism to hide 32 separate `sin` kernels and their saved tensors, and
  peak memory for the LSA process reached about 4.4 GB against a few hundred MB
  for the others.
- **At matched wall time it is not close.** SIREN given 2,000 steps reaches
  48.61 dB in 184 seconds, under a seventh of the 1,383 seconds LSA spent to
  reach 35.12 dB. On this hardware and at this size the honest summary is that
  LSA's advantage, if it arrives, arrives after a cost that a faster activation
  could spend on more steps.
- **The coefficients do move, and the last layer goes flat.** After 500 steps
  the first layer's 32 coefficients are spread between about −0.04 and +0.04
  with mixed signs; the last hidden layer's are all close to −0.042, a nearly
  uniform harmonic profile. The paper's Figure 19 describes the same pattern on
  Kodak: selective early layers, broad uniform support in the final hidden
  layer.
- **ReLU with $L = 10$ positional encoding is the weakest** at 24.75 dB. I did
  not tune $L$; ten octaves put the top frequencies past this 128-pixel grid's
  Nyquist limit, so treat it as a floor, not a fair baseline.

Treat this as a smoke test of the released code and a price check, not a
replication. Reproducing the Kodak table needs a GPU, 768×512 images, width
256 and 5,000 steps per fit, and the released `image_2d/run_core.py` does
exactly that.

## Where this leaves LSA

The idea is small and sound: learned, layer-shared harmonic amplitudes cut the
tie between a neuron's waveform and its feature for $K$ parameters per layer,
and give the optimiser clean new descent directions. On images that buys a consistent 0.8 dB over FINER and 3 dB over
SIREN at matched width and steps. On audio it buys a lot once Fourier features
are present, and nothing without them. On NeRF it buys parity.

What I would want before adopting it:

- **A wall-clock comparison.** At a fixed GPU-hour, is LSA still ahead of
  FINER? My CPU run never reached the point where LSA's per-step lead shows
  up, and the per-step price is large. A fused kernel would change the answer,
  and does not exist yet.
- **Seeds.** One seed per image, one run per NeRF scene.
- **The NeRF and NAF code.** The two field results are the most interesting and
  are not reproducible from the release.
- **A licence** on the repository.

Related reading: [RoPE](/architectures/rope) applies the same sinusoidal-feature
idea to token positions; the [Gaussian-splat appearance
benchmark](/articles/efficient-gaussian-appearance) is a field where the
frequency question moved out of the MLP into spherical harmonics; and
[functional gradient descent](/articles/functional-gradient-descent) is another
way to think about optimising a function rather than its weights.