Delayed-KD: Why Matching Every Frame Hurts Streaming Speech Recognition

Two separate problems get mistaken for one

Streaming speech recognition built on Connectionist Temporal Classification, or CTC, has become a common choice because the architecture is simple and cheap to run in real time. Feed it audio in small chunks instead of the whole recording at once, and it can emit text as the person speaks. The catch is that CTC based streaming models suffer from two distinct problems that the literature often treats as if they were the same thing.

The first problem is accuracy loss from limited context. A model that only sees a tiny slice of audio at a time simply has less information to work with than one that sees the whole utterance, so its predictions degrade, especially as the chunk size shrinks. The second problem is emission delay, sometimes called CTC spike lag. Even when a streaming model eventually recognizes a sound correctly, it tends to announce that recognition later than a non-streaming model would, because CTC’s spikes, the frames where the model commits to a non blank token, drift later under limited context.

Most prior work has picked one problem to solve. Approaches like Fast-U2++ use a hybrid chunk strategy, small chunks low in the network and large chunks higher up, to claw back accuracy. Loss level tricks like FastEmit and Delay-Penalty push non-blank tokens to fire earlier, attacking the spike lag directly. The authors point out a gap in this second category. Reducing spike lag inside the loss function does not fully fix emission delay in practice, because a chunk based streaming model still only reports a token at the end of whichever chunk boundary it happens to fall in. You can nudge the spike earlier all you want, the output still waits for the chunk to close.

Where knowledge distillation runs into trouble

Knowledge distillation is the obvious tool to reach for here. Train a non-streaming teacher with full context, then have a streaming student mimic the teacher’s output distribution instead of only the ground truth labels, and the student should inherit some of that full context knowledge. This has worked well elsewhere, and prior streaming ASR work has applied it too, but the authors identify a specific failure mode in how it is usually applied to CTC posteriors.

Frame by frame distillation implicitly assumes the teacher’s CTC spike at time t and the student’s CTC spike for the same sound also land at time t. Given the spike lag problem described above, that assumption is false by construction. The teacher, seeing the whole utterance, commits to a token earlier than the student can. Forcing the student’s distribution at frame t to match the teacher’s distribution at frame t is not teaching the student to recognize the sound better, it is teaching the student to hallucinate a decision before it has heard enough audio to make it responsibly. The paper frames this plainly as an information discrepancy between the two models that naive alignment ignores.

A buffer that searches for the right alignment instead of assuming one

The fix is the Temporal Alignment Buffer, TAB for short. Instead of comparing the teacher’s output at frame t only to the student’s output at the same frame t, TAB compares it to a small window of candidate student frames, t through t plus some maximum delay d, and picks whichever one has the lowest KL divergence from the teacher. That per frame choice becomes the actual training signal.

There is something worth pausing on here. The minimum is taken independently at every single frame, not once per utterance. That means different frames in the same sentence can effectively use different delays, whatever alignment happens to fit that particular sound best. This is a soft, learned alternative to hand computing a forced alignment ahead of time, and it sidesteps the brittleness of forced alignment approaches that the paper cites as a limitation of earlier work.

The distillation loss does not stand alone. It gets folded into a joint objective alongside the standard ASR training loss, which is itself a weighted mix of the CTC loss and an attention decoder loss carried over from the U2++ architecture the paper builds on.

The architecture itself, shown in the paper’s Figure 1, keeps things close to a standard two pass CTC and attention design. A Conformer based shared encoder feeds both a CTC decoder and, on the student side only, an attention decoder. The non-streaming teacher’s parameters are frozen during training, it exists purely to supply a target distribution. That is a deliberate simplicity choice. The paper is not proposing a new encoder, it is proposing a better way to use an existing teacher and student pair.

The results, and why the 40 millisecond number matters

The headline comparison is against U2++, a strong existing streaming CTC and attention baseline, on the AISHELL-1 Mandarin dataset. At a 40 millisecond chunk latency, which is aggressive for real time use, Delayed-KD reaches a character error rate of 5.42 percent in rescoring mode. U2++ needs to run at 320 millisecond latency, eight times slower to respond, to hit a comparable number. At the same 40 millisecond latency, Delayed-KD beats U2++ by a relative character error rate reduction of roughly 9 percent in rescoring mode and over 21 percent in the streaming only decoding mode, where there is no second attention pass to clean things up.

That last row is worth sitting with. Delayed-KD running at a 40 millisecond chunk size, with an 80 millisecond TAB window during training, edges out Fast-U2++’s rescoring accuracy despite Fast-U2++ using a much larger rescoring chunk of 960 milliseconds, and it comfortably beats CUSIDE-T’s 400 millisecond baseline while responding roughly ten times faster. The paper is careful to note one honest exception, Fast-U2++ still edges out Delayed-KD slightly in raw rescoring accuracy, which the authors attribute fairly to Fast-U2++’s rescoring pass running at 24 times the latency, a real tradeoff rather than a wash.

The result that gives this the most credibility is that it holds on WenetSpeech, a 10,000 hour multi domain dataset an order of magnitude larger and messier than AISHELL-1. On the Test Meeting set, Delayed-KD at 40 milliseconds nearly matches U2++ running at 320 milliseconds, and clearly beats U2++ at the same 40 millisecond latency by a wide margin, roughly 28 percent relative CER reduction on Test Meeting and 33 percent on Test Net. A lot of distillation techniques look great on a clean, medium sized benchmark and quietly lose their edge on messier, larger data. This one does not.

The paper’s own honest limit

The most useful section for anyone planning to apply this technique is the ablation on the distillation weight, the alpha term controlling how much the distillation loss counts relative to the task loss in the joint objective.

Accuracy improves steadily as alpha rises from 4 to 100, then reverses hard past that point. At alpha 400, streaming CER is actually worse than at alpha 4, the point where distillation barely mattered at all. The authors’ explanation is straightforward and believable, an overweighted distillation term pushes the student to overfit to the teacher’s specific CTC posterior distribution rather than learning generalizable acoustic patterns, at the expense of the task loss that is actually tied to getting the transcription right. This is a genuinely useful negative result. A team adopting this technique should expect a real tuning cost here, not just a knob to turn up for free accuracy, and the paper’s own numbers show the failure mode rather than glossing over it.

A similar story shows up in the TAB size ablation. Frame by frame distillation, a TAB size of zero, gives the lowest possible token emission delay but the worst accuracy of any TAB setting tested. Widening the window to 80 milliseconds gives the best accuracy, a 6 percent relative CER reduction in rescoring mode and nearly 13 percent in streaming mode compared to the zero delay case, but pushing the window wider still, to 120 or 160 milliseconds, starts costing both accuracy and emission delay without buying anything back. There is a genuine sweet spot here rather than a monotonic improvement, and 80 milliseconds is specific to this dataset and architecture rather than a universal constant.

What is still missing from the picture

A few things are worth flagging for anyone reading the results at face value. All of the reported numbers are on Mandarin Chinese datasets, AISHELL-1 and WenetSpeech, and the modeling units are Chinese characters rather than subword units common in other languages. Character level CTC alignment behaves differently from byte pair encoding or phoneme level alignment, so it is not obvious the same TAB window sizes, or even the same qualitative benefit, would transfer directly to English or other languages without re-tuning.

The comparisons also lean on a specific baseline family. U2++, Fast-U2++ and CUSIDE-T are all reasonable, competitive choices, but none of them are RNN Transducer based streaming systems, which are the other major streaming ASR architecture family in wide industrial use. Whether the same delayed alignment idea transfers cleanly to a transducer’s very different emission dynamics is an open question the paper does not address, since its distillation target is specifically CTC posterior probabilities.

Finally, the emission latency metrics rely on forced alignment ground truth generated by a GMM and HMM model, which is itself an imperfect labeling process, not a physical timestamp measurement. That is a standard and reasonable choice in the field, but it means the absolute millisecond numbers carry some measurement uncertainty baked in from the alignment tool itself, something the paper does not quantify.

Broader implications for anyone building low latency speech systems

The generalizable idea here is not really about speech recognition specifically. It is about what happens when you distill knowledge between two models that see the same input under systematically different information budgets. Frame level or token level alignment assumes both models reach the same conclusion at the same point in the sequence, and that assumption quietly breaks whenever one model has strictly less context than the other, which is exactly the situation any streaming versus batch pair is in, whether that is speech, video, or sequential sensor data. A learned, per position search over a small window of plausible alignments, rather than a rigid one to one match, is a pattern likely to be useful anywhere a faster, more constrained model is being taught by a slower, more complete one.

Practically, for a team deciding whether this is worth adopting, the signal to weigh is not just the headline 5.42 percent CER number, it is that the method degrades gracefully and predictably as the TAB window and distillation weight are pushed too far, and the authors show you exactly where that happens rather than only reporting their best configuration. That kind of transparency makes the tuning process for a new dataset considerably less of a guessing game than it would be from the accuracy numbers alone.

Conclusion

The core contribution here is a precise diagnosis followed by a proportionate fix. The diagnosis is that frame by frame distillation between a non-streaming teacher and a streaming student silently assumes an alignment that the emission delay problem itself guarantees does not hold. The fix, letting the loss search a small delay window per frame instead of assuming zero delay, is not architecturally dramatic, but it directly targets the actual mismatch rather than working around it with a bigger model or more data.

The results back up the framing. Matching a model running at eight times the latency budget, on both a controlled benchmark and a 10,000 hour messy multi domain dataset, is a meaningfully strong result for a change that touches only the training loss and leaves the underlying architecture untouched. That the improvement survives the jump to WenetSpeech is the detail that should carry the most weight for anyone evaluating whether to trust this beyond the paper’s own numbers.

Just as valuable is what the paper chooses to report honestly. The distillation weight and TAB size ablations both show clear failure past a certain point, and the authors explain why rather than only presenting the winning configuration. That is the kind of detail that turns a paper from something you cite into something you can actually use, since it tells you where the cliff edge is before you fall off it on your own dataset.

The open questions are exactly the ones you would expect from a paper scoped tightly to one language and one architecture family. Whether the same delay search generalizes to transducer based streaming models, or to languages with very different phonetic and tokenization structure than Mandarin, is left for future work, and a team applying this outside that scope should expect to re-tune, not copy the reported hyperparameters directly.

Frequently asked questions

What problem does Delayed-KD actually solve

It addresses the mismatch that occurs when a non-streaming teacher model is used to train a streaming student through frame by frame knowledge distillation. Because the teacher sees the whole utterance and the student only sees left side context, the teacher’s CTC output arrives earlier in time than the equivalent student output, so matching them at the same frame index teaches the student an unhelpful, premature signal.

What is the Temporal Alignment Buffer

It is a small window of allowed time delays, set as a hyperparameter, that the distillation loss searches over at every frame. Instead of forcing the student’s output at time t to match the teacher’s output at time t, the loss compares the teacher’s frame t against a handful of candidate student frames from t up to t plus the maximum delay, and keeps whichever comparison has the lowest KL divergence.

How much latency does this actually save compared to existing streaming models

On the AISHELL-1 benchmark, Delayed-KD running with a 40 millisecond chunk size reaches accuracy comparable to the U2++ baseline running at 320 milliseconds, an eight times difference in responsiveness, and clearly outperforms U2++ when both are compared at the same 40 millisecond latency.

Does increasing the distillation weight always help

No. The paper’s own ablation shows accuracy improving as the distillation weight rises up to a value of 100, then getting measurably worse at 200 and considerably worse at 400, which the authors attribute to the student overfitting to the teacher’s specific output distribution at the expense of the underlying task loss.

Does this work for languages other than Mandarin

The paper only reports results on Mandarin Chinese datasets, AISHELL-1 and WenetSpeech, using character level modeling units. The core idea of searching a delay window during distillation is not language specific in principle, but the reported hyperparameters, including the best performing TAB size, would need re-tuning for a different language or tokenization scheme.

Could this apply to RNN Transducer based streaming models instead of CTC

The paper’s method is built specifically around CTC posterior probability distributions, and it does not test transducer based architectures. Since transducers have different emission dynamics than CTC, whether the same delayed alignment approach transfers directly is an open question the paper does not address.

Reference implementation, the delayed distillation loss in PyTorch

The implementation below reproduces the paper’s core mechanism, a per frame search over a delay window that picks the minimum KL divergence between a frozen non-streaming teacher and a causal streaming student, folded into the joint training objective from equations 1 through 4. It uses small stand in Conformer style branches rather than the paper’s full architecture, since the point of a smoke test is to confirm the loss behaves correctly and gradients flow to the right place, not to reproduce state of the art numbers on a laptop.

# Delayed-KD, reimplemented from Li, Li, Xue, Liu, Fang, Wang
# and Xie, arXiv 2505.22069. A non-streaming teacher's CTC
# output arrives earlier in time than a streaming student's
# equivalent output. The Temporal Alignment Buffer lets the
# distillation loss search a small delay window per frame
# instead of assuming the two are already aligned.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)


def delayed_kd_loss(student_log_probs, teacher_log_probs, max_delay):
    """Equation 1. For every teacher frame t, search delays tau in
    0 to max_delay for the student frame t + tau that minimizes
    the KL divergence to the teacher's distribution at t, then
    average that minimum over all valid frames and the batch."""
    batch, time_steps, _ = student_log_probs.shape
    per_delay_kl = []

    for tau in range(max_delay + 1):
        if tau == 0:
            student_shifted = student_log_probs
            teacher_aligned = teacher_log_probs
        else:
            # Compare teacher frame t against student frame t + tau,
            # the "shifted right by a few chunks" alignment from
            # Section 2.2.1.
            student_shifted = student_log_probs[:, tau:, :]
            teacher_aligned = teacher_log_probs[:, :-tau, :]

        # Equation 2, KL(student || teacher), summed over vocabulary.
        kl_per_frame = torch.sum(
            student_shifted.exp() * (student_shifted - teacher_aligned), dim=-1
        )

        pad_amount = time_steps - kl_per_frame.shape[1]
        if pad_amount > 0:
            kl_per_frame = F.pad(kl_per_frame, (0, pad_amount), value=float("inf"))
        per_delay_kl.append(kl_per_frame)

    stacked = torch.stack(per_delay_kl, dim=0)
    min_kl, chosen_delay = torch.min(stacked, dim=0)
    valid_mask = torch.isfinite(min_kl)
    return min_kl[valid_mask].mean(), chosen_delay


def joint_asr_loss(ctc_loss_value, aed_loss_value, lam=0.3):
    """Equation 4, the weighted CTC plus attention decoder loss."""
    return lam * ctc_loss_value + (1 - lam) * aed_loss_value


def joint_training_loss(asr_loss_value, distill_loss_value, alpha=100.0):
    """Equation 3, the full training objective."""
    return asr_loss_value + alpha * distill_loss_value


class ToyConformerCTCBranch(nn.Module):
    """A tiny stand in for Figure 1's shared Conformer encoder plus
    CTC decoder. Fbank features in, CTC log posteriors out."""

    def __init__(self, feature_dim, hidden_dim, vocab_size, causal):
        super().__init__()
        self.causal = causal
        kernel_size = 5
        padding = kernel_size - 1 if causal else (kernel_size - 1) // 2
        self.conv = nn.Conv1d(feature_dim, hidden_dim, kernel_size, padding=padding)
        self.ctc_decoder = nn.Linear(hidden_dim, vocab_size)

    def forward(self, fbank):
        x = fbank.transpose(1, 2)
        x = self.conv(x)
        if self.causal:
            # Trim the extra right side frames a causal pad would add.
            x = x[:, :, :fbank.shape[1]]
        x = F.relu(x).transpose(1, 2)
        return F.log_softmax(self.ctc_decoder(x), dim=-1)


def run_smoke_test():
    batch, time_steps, feature_dim, hidden_dim, vocab_size = 4, 60, 80, 64, 30
    max_delay = 8  # an 80 ms Temporal Alignment Buffer at a 10 ms frame shift

    fbank = torch.randn(batch, time_steps, feature_dim)
    teacher = ToyConformerCTCBranch(feature_dim, hidden_dim, vocab_size, causal=False)
    student = ToyConformerCTCBranch(feature_dim, hidden_dim, vocab_size, causal=True)
    for p in teacher.parameters():
        p.requires_grad = False  # the paper freezes the non-streaming teacher

    optimizer = torch.optim.Adam(student.parameters(), lr=1e-3)
    target_lengths = torch.full((batch,), 8, dtype=torch.long)
    targets = torch.randint(1, vocab_size, (batch, 8))
    input_lengths = torch.full((batch,), time_steps, dtype=torch.long)
    ctc_loss_fn = nn.CTCLoss(blank=0, zero_infinity=True)

    print(f"Student is causal : {student.causal}")
    print(f"Teacher is causal : {teacher.causal}")

    for step in range(3):
        optimizer.zero_grad()
        with torch.no_grad():
            teacher_log_probs = teacher(fbank)
        student_log_probs = student(fbank)

        ctc_loss_value = ctc_loss_fn(
            student_log_probs.transpose(0, 1), targets, input_lengths, target_lengths
        )
        asr_loss_value = joint_asr_loss(ctc_loss_value, ctc_loss_value, lam=1.0)
        distill_loss_value, chosen_delay = delayed_kd_loss(
            student_log_probs, teacher_log_probs, max_delay
        )
        total_loss = joint_training_loss(asr_loss_value, distill_loss_value, alpha=100.0)
        total_loss.backward()
        optimizer.step()

        print(
            f"step {step} | ctc {ctc_loss_value.item():.3f} | "
            f"distill {distill_loss_value.item():.3f} | "
            f"joint {total_loss.item():.3f} | "
            f"mean chosen delay {chosen_delay.float().mean().item():.2f} frames"
        )

    assert torch.isfinite(total_loss), "joint loss must stay finite"
    assert student.conv.weight.grad is not None, "the student must receive gradients"
    assert teacher.conv.weight.grad is None, "the frozen teacher must receive no gradients"
    print("\nSmoke test passed.")


if __name__ == "__main__":
    run_smoke_test()

Running this prints a shrinking joint loss across the three toy optimization steps, confirms the frozen teacher never accumulates a gradient while the causal student does, and reports a mean chosen delay of a few frames rather than zero, which is the qualitative signature the paper’s Section 3.4 result depends on, the model is actually finding value in comparing across a delay window rather than defaulting to a same frame match.

Related reading on aitrendblend