A transcription company runs Whisper for call centres across the English speaking world. Over two years its engineers have trained a small LoRA adapter for every accent a client asked for, American, British, Indian, Spanish, Korean and more, each one a few megabytes of low rank matrices that bend the big model toward one way of speaking. Now a new client wants Portuguese accented English handled well by next week. The team has about seventeen hours of transcribed calls. Should they train another adapter from scratch, or can the adapters they already own do most of the work?
That is the question behind a paper in Information Fusion by Qing Hu, Yan Zhang, Zongyu Han, Xiaoqing Zhang and Zetong Cai of the School of Artificial Intelligence at Hebei University of Technology. Their method, VeLoRA, keeps every existing adapter frozen and learns only small scaling vectors that rescale the adapters’ matrices before averaging them. On ten English accents it comes within a tenth of a word error point of the best competing fusion method while training a few hundred times fewer parameters. The interesting parts are what those vectors can and cannot express, what the comparison quietly leaves out, and a small bug in the released code.
Key points
- VeLoRA fuses frozen LoRA experts by learning one scaling vector per expert on the rank side and one shared vector on the output side of every layer. On Whisper base it trains about 89,000 parameters, against 30.3 million for the strongest fusion method that fine tunes its experts.
- Averaged over ten accents of the AESRC corpus, VeLoRA reaches 5.0 percent word error on Whisper base and 3.2 percent on Whisper medium, against 4.9 and 3.1 for P-CAM, with overlapping bootstrap confidence intervals.
- Its advantage over plain LoRA rests on extra data. Each of the seven experts it fuses was trained on roughly 17 hours of another accent, about 119 hours in total, while the LoRA baseline sees only the target accent’s 17 hours. A baseline trained on the pooled data is missing.
- FasterVeLoRA folds the averaged experts into Whisper’s weights and learns a single output gain per layer. It trains 4.8 times faster on Whisper medium, but on dysarthric speech its 22.6 percent character error is worse than plain LoRA’s 21.2.
- In the released code, FasterVeLoRA scales a layer’s bias during training but not after its weights are merged for inference. The deployed model therefore differs slightly from the trained one, and the fix is one line.
- The dysarthric speech test covers 9 speakers with mild dysarthria out of 44 in the corpus. People with severe dysarthria, who need assistive recognition most, were left out because adaptation often failed to converge.
Note on health content. Part of this paper tests speech recognition for people with dysarthria, a motor speech disorder. This article explains published research on recognition software. It is not medical advice and does not evaluate any assistive or clinical product. Anyone choosing speech technology for a person with a speech disorder should involve a qualified speech and language professional.
A growing library of LoRA experts
Whisper, OpenAI’s speech recognition model trained on 680,000 hours of audio, is robust inside the conditions it was trained on and noticeably weaker outside them. The paper cites a 13.6 point increase in word error rate for Whisper large when it meets accented English in the VoxPopuli corpus. The standard fix is low rank adaptation, LoRA, which attaches two small matrices to each linear layer and trains only those. LoRA adapters are cheap to train and store, so organizations accumulate them, one per accent, speaker group, product line or acoustic condition.
A LoRA layer (Eq. 1)
$$ y = x\mathbf{W}_0 + x\mathbf{A}\mathbf{B} $$ $$ \mathbf{A} \in \mathbb{R}^{d \times r},\quad \mathbf{B} \in \mathbb{R}^{r \times k},\quad r \ll \min(d, k) $$\(\mathbf{W}_0\) is the frozen pretrained weight and \(\mathbf{A}\mathbf{B}\) the low rank update. Once trained, \(\mathbf{A}\mathbf{B}\) can be added into \(\mathbf{W}_0\), so a LoRA model runs as fast as the original.
Once a library exists, a new domain raises an obvious question. Can existing experts be combined instead of training from scratch? The paper sorts earlier answers into two families. Static expert fusion keeps every expert frozen and learns only how much weight each one gets, a handful of numbers per layer. Dynamic expert fusion also fine tunes the experts on the new domain. Static fusion is extremely cheap but limited by what the frozen experts already know. Dynamic fusion performs better but trains, and has to store, a full copy of every expert for every new domain.
Static and dynamic expert fusion share one form (Eq. 3)
$$ y = x\mathbf{W}_0 + x \sum_{i=1}^{N} \alpha_i(x)\, \mathbf{A}_i \mathbf{B}_i $$\(\alpha_i(x)\) is the weight of expert \(i\), fixed, learned or produced by a router network. In static fusion only \(\alpha\) is trained. In dynamic fusion \(\mathbf{A}_i\) and \(\mathbf{B}_i\) are trained as well.
Weight averaging of fine tuned models has a longer history. Mitchell Wortsman and colleagues showed with model soups that averaging fine tuned weights can improve accuracy without slowing inference, and LoraHub, from Chengsong Huang and colleagues, composed LoRA modules for new tasks. In speech, the strongest dynamic baselines in this paper, LanFusion and the CAM framework, come from the same Hebei group, which matters when reading the comparison.
From VeRA to VeLoRA
The starting point is VeRA, a method by Dawid Kopiczko, Tijmen Blankevoort and Yuki Asano that freezes a single pair of random low rank matrices shared by all layers and learns only two small scaling vectors per layer. VeRA is extremely parameter efficient in language tasks, but the authors found it trailing LoRA on accent adaptation, 6.4 percent word error against 5.6 on three AESRC accents with Whisper base.
VeRA, frozen random matrices with learned scaling vectors (Eq. 2)
$$ y = x\mathbf{W}_0 + x(\mathbf{A} * \mathbf{a})(\mathbf{B} * \mathbf{b}) $$ $$ \mathbf{a} \in \mathbb{R}^{r},\quad \mathbf{b} \in \mathbb{R}^{k} $$\(*\) is elementwise multiplication with broadcasting, so \(\mathbf{A} * \mathbf{a}\) rescales each column of \(\mathbf{A}\) and \(\mathbf{B} * \mathbf{b}\) each column of \(\mathbf{B}\).
Their diagnosis was that random matrices give the vectors too little to work with. A small experiment confirms it. Replacing the random matrices with a LoRA expert trained on American accented speech, and starting both vectors at 1 instead of VeRA’s 0.1 and 0, brings the error to 5.6, level with LoRA, while still training only the vectors. VeLoRA extends that idea from one expert to many.
VeLoRA layer (Eq. 4)
$$ y = x\mathbf{W}_0 + \frac{1}{N} \sum_{i=1}^{N} x(\mathbf{A}_i * \mathbf{a}_i)(\mathbf{B}_i * \mathbf{b}) $$\(\mathbf{a}_i \in \mathbb{R}^{r_i}\) is specific to expert \(i\), while \(\mathbf{b} \in \mathbb{R}^{k}\) is shared by all experts in the layer. Both start at 1. The fusion weight is a fixed \(1/N\), so the whole update can be merged into \(\mathbf{W}_0\) before inference.
Every linear layer in Whisper’s attention and feed forward modules, in both encoder and decoder, is replaced by a VeLoRA layer. Experts are chosen by a simple source data free rule. Each candidate expert is scored by its own word error rate on a small development set from the target domain, and the best N are kept, seven in the main experiments.
What the vectors can and cannot do
It helps to rewrite Eq. 4. Scaling the columns of \(\mathbf{A}_i\) by \(\mathbf{a}_i\) is the same as inserting a diagonal matrix between \(\mathbf{A}_i\) and \(\mathbf{B}_i\), and scaling the columns of \(\mathbf{B}_i\) by \(\mathbf{b}\) is the same as multiplying the whole update by a diagonal matrix on the output side.
The same update written as reweighted rank one pieces, our rewriting
$$ \Delta\mathbf{W} = \frac{1}{N} \Big( \sum_{j=1}^{Nr} a_j\, \mathbf{u}_j \mathbf{v}_j^{\top} \Big) \operatorname{diag}(\mathbf{b}) $$\(\mathbf{u}_j\) is a column of some expert’s \(\mathbf{A}_i\) and \(\mathbf{v}_j^{\top}\) the matching row of its \(\mathbf{B}_i\). With seven experts of rank 32, each layer offers 224 fixed rank one directions.
Seen this way, VeLoRA is a learned volume knob on each of the 224 rank one directions that the experts already contain, followed by a gain on each output channel. It can turn a direction up, down or off, and with \(a_j\) equal for every component of an expert it reduces exactly to a weighted sum of experts, the static fusion it was designed to beat. What it cannot do is create a direction that no expert contains. That is the source of both its efficiency and its ceiling. When the target domain resembles the experts, as accented English resembles other accented English, the right directions are probably already there. When it does not, no amount of rescaling will find them, which is the limitation the authors acknowledge in their conclusion.
The rewriting also explains why all ones is the right starting point. With \(\mathbf{a}\) and \(\mathbf{b}\) at 1, VeLoRA starts as plain uniform averaging of the experts. The paper’s ablation confirms it. With both vectors removed, Table 7 reports 6.6 percent on Whisper base and 3.9 on Whisper medium, exactly the scores of MAS-LoRA, the training free averaging baseline in Table 5. Everything VeLoRA gains beyond that, 1.6 points on base and 0.7 on medium, comes from learning the vectors.
The parameter count follows directly. Each adapted layer gets one entry of \(\mathbf{b}\) per output channel and one entry of \(\mathbf{a}\) per expert rank. For Whisper base, with 6 encoder and 6 decoder blocks, a model width of 512 and feed forward width of 2048, seven experts of rank 32 give 89,088 trainable parameters by our count, which matches the 0.090 million in the paper’s Table 4.
FasterVeLoRA, one gain per channel
VeLoRA still runs every expert’s matrices during training, which makes training slower than plain LoRA. FasterVeLoRA removes that cost. It averages the experts once, adds the average into Whisper’s weights, and then learns only a single scaling vector per layer.
FasterVeLoRA (Eqs. 5 and 6)
$$ \mathbf{W}_0′ = \mathbf{W}_0 + \frac{1}{N} \sum_{i=1}^{N} \mathbf{A}_i \mathbf{B}_i $$ $$ y = x\mathbf{W}_0′ * \mathbf{b} $$The vector \(\mathbf{b}\) rescales each output channel of the merged layer, including the part that came from the pretrained Whisper weights. Training cost no longer grows with the number of experts.
Notice what Eq. 6 does. It does not adjust the experts at all. It applies a learned gain to every output channel of every linear layer of the already merged model. That is a well known and very cheap form of adaptation, and here it rides on top of uniform expert averaging. On Whisper medium it cuts training time from 57.0 minutes for VeLoRA to 12.0, and the average error rises only from 3.2 to 3.3 percent. On Whisper base the cost is larger, 5.0 to 5.6 percent, a 12 percent relative increase. The abstract quotes the medium figure, 3.1 percent relative.
Key takeaway
VeLoRA learns a volume knob for each rank one direction the experts already contain, plus a gain per output channel. It starts as plain expert averaging and improves on it by learning those knobs. FasterVeLoRA keeps only the output gains, applied to a model that already has the averaged experts folded in.
Results on ten English accents
The main test uses AESRC, the corpus of the 2020 Accented English Speech Recognition Challenge, with ten accents, American, British, Chinese, Indian, Japanese, Korean, Portuguese, Russian, Spanish and Canadian, each with 20 hours of speech split 85, 5 and 10 percent into training, development and test sets. Each accent takes a turn as the target. The pool holds ten LoRA experts of rank 32, one per accent, and the target’s own expert is excluded, so the seven fused experts all come from other accents. All Whisper models were first fine tuned on the 100 hour LibriSpeech subset to curb repetitive output.
| Method | Family | Whisper base | Base params | Whisper medium | Medium params |
|---|---|---|---|---|---|
| Whisper, no adaptation | n/a | 11.3 [11.0, 11.5] | 0 | 6.3 [6.1, 6.4] | 0 |
| Fine tuning | No experts | 5.4 [5.3, 5.6] | 44.0 | 3.3 [3.2, 3.4] | 704.6 |
| LoRA | No experts | 5.5 [5.3, 5.6] | 4.3 | 3.4 [3.3, 3.5] | 34.6 |
| CAT | Static fusion | 6.1 [5.9, 6.2] | under 0.1 | 3.7 [3.5, 3.8] | under 0.7 |
| MAS-LoRA | Static fusion | 6.6 [6.4, 6.7] | under 0.1 | 3.9 [3.8, 4.0] | under 0.7 |
| LanFusion | Dynamic fusion | 4.9 [4.8, 5.1] | 30.3 | 3.2 [3.1, 3.3] | 242.2 |
| P-CAM | Dynamic fusion | 4.9 [4.7, 5.0] | 30.3 | 3.1 [3.0, 3.2] | 242.2 |
| VeLoRA | Vector fusion | 5.0 [4.9, 5.2] | 0.09 | 3.2 [3.1, 3.3] | 0.62 |
| FasterVeLoRA | Vector fusion | 5.6 [5.4, 5.7] | under 0.07 | 3.3 [3.2, 3.4] | under 0.52 |
The pattern is clear. Static fusion is cheap and clearly worse than training LoRA on the target. Dynamic fusion is best and expensive. VeLoRA sits within a tenth of a point of the dynamic methods with roughly 337 times fewer trainable parameters than P-CAM on Whisper base, by our arithmetic, and its confidence interval overlaps theirs. On Whisper large v2, with five experts of rank 16, it ties LanFusion at 2.7 percent with 0.94 million trainable parameters against 144.2 million.
The per accent numbers add texture. VeLoRA’s weakest case is Indian English, 5.4 percent on Whisper base against 4.7 for P-CAM, about 15 percent worse in relative terms, and 4.8 for LanFusion. Indian English may simply be the accent least represented by directions in the other nine experts, which is the limitation the rewriting above predicts. The expert count study also shows that fusion needs enough experts. With a single expert, VeLoRA reaches 5.7 percent, worse than plain LoRA’s 5.5. It needs three experts to pull ahead, and gains stop after seven.
“its performance on the target domain will be limited”
Hu and colleagues, on what happens when no suitable experts are available
Where the advantage comes from
VeLoRA beats plain LoRA, 5.0 against 5.5 on Whisper base, with a fiftieth of the parameters. It is tempting to read that as a better adaptation method. The more accurate reading is that VeLoRA uses more data. The LoRA baseline is trained only on the target accent’s training split, about 17 hours. VeLoRA fuses seven experts, each trained on about 17 hours of a different accent from the same corpus, roughly 119 hours of extra accented English with the same recording setup and transcription style. The authors say as much, writing that the effectiveness of the methods mainly stems from building on well trained experts.
That makes the missing control obvious. A LoRA or fine tuning baseline trained on the target data plus all the other accents’ data would show how much of the gain comes from fusion as a technique and how much from the extra speech. The paper does not include one. In fairness, the realistic scenario the authors describe is one where the experts exist but their training data do not travel with them, licensed adapters from the internet or a product line, or speaker adapters whose recordings cannot be shared. In that setting pooled training is not an option, and VeLoRA’s comparison against target only LoRA is the relevant one.
Two further details shape how far the results generalize. All experts in the pool were trained by the authors, on the same corpus, with the same backbone, rank and recipe, which is the friendliest possible version of the off the shelf scenario. Experts gathered from different teams would differ in rank, training data, backbone checkpoint and quality. And every number comes from a single training run with seed 0. The bootstrap intervals capture uncertainty from sampling the test set, not from training randomness, so small differences between VeLoRA and the dynamic methods, 0.1 points either way, should not be over read.
Key takeaway
VeLoRA’s win over plain LoRA is mostly a win of 119 extra hours of accented speech, accessed through frozen experts instead of raw data. That is valuable when only the experts can be shared. It is not evidence that VeLoRA adapts better than LoRA given the same data.
Efficiency, read carefully
The efficiency comparison in Table 6, measured on the American accent, deserves a close reading. Training time is the wall clock time until early stopping, so it mixes speed per step with how quickly each method converges, and it comes from a single run.
| Method | GPU memory | Training time | RTF |
|---|---|---|---|
| Fine tuning | 16.8 | 50.8 | 0.173 |
| LoRA | 10.4 | 44.4 | 0.171 |
| CAT | 11.3 | 137.8 | 0.175 |
| P-CAM | 21.0 | 136.0 | 0.600 |
| E-CAM | 10.4 | 37.7 | 0.167 |
| VeLoRA | 11.5 | 57.0 | 0.181 |
| FasterVeLoRA | 9.8 | 12.0 | 0.167 |
VeLoRA trains about 28 percent slower than plain LoRA on Whisper medium, 57.0 minutes against 44.4, because every step runs seven experts’ matrices. Its advantage is in what it stores, not in how fast it trains. FasterVeLoRA is the fast one, 3.7 times quicker than LoRA and 4.8 times quicker than VeLoRA. At inference, VeLoRA and FasterVeLoRA merge into ordinary weights and decode at essentially the speed of a fine tuned model, while methods that compute fusion weights per input, such as P-CAM, run three to four times slower.
The storage comparison needs one qualification. The paper reports 3.0 MB for VeLoRA against 1847.8 MB for dynamic fusion on Whisper medium with rank 64 experts, a reduction of more than 600 times. That compares what each method must save per new domain. VeLoRA also needs the frozen expert pool, which at rank 32 on Whisper medium is the same 242.2 million parameters, about 924 MiB, that dynamic fusion trains. The pool is shared across every target domain, so the per domain saving is real and grows with the number of domains. For a single new domain, the total footprint is not 600 times smaller.
A bug in the released FasterVeLoRA layer
The authors released their code as an ESPnet fork with a modified loralib, which is good practice and made this check possible. Reading the FasterVeLoRA layer, we found a mismatch between training and inference. In training mode the layer computes the merged linear layer’s output, bias included, and multiplies the whole output by \(\mathbf{b}\). When the model is switched to evaluation mode, the layer merges \(\mathbf{b}\) into the weight matrix by scaling its rows, but leaves the bias unchanged. The function used for decoding therefore differs from the trained one by \((\mathbf{b} – 1) * \text{bias}\) on every output channel of every layer that has a bias, which in Whisper includes the query, value, output and feed forward projections.
The size of the effect depends on how far \(\mathbf{b}\) moves from 1 and how large the biases are, and we could not measure it on Whisper. In our toy test below, the unscaled bias changed individual outputs by up to 1.7 logits and raised test error by about half a point. Since the vectors start at 1 and are trained for at most ten epochs, the real effect is probably small, but it works against FasterVeLoRA, not for it. The fix is to scale the bias by \(\mathbf{b}\) along with the weight at merge time and to divide it back when unmerging. A related detail is that unmerging divides by \(\mathbf{b}\) clamped at \(10^{-8}\), which would misbehave if any entry of \(\mathbf{b}\) ever became negative. The VeLoRA layer itself merges correctly.
Dysarthric speech, the harder test
The second experiment moves to speaker adaptation on CDSD, a Chinese dysarthric speech database. Dysarthria is a motor speech disorder caused by neurological conditions, and people living with it are among those who could benefit most from recognition that adapts to their voice. The corpus part used here has 44 speakers with one hour of recordings each. The authors selected 9 speakers with low severity dysarthria, explaining that adaptation for severely dysarthric speech often fails to converge. Each speaker took a turn as the target, with LoRA experts trained on the other eight fused for adaptation, and results are reported as character error rate.
| Method | Family | Average CER | Trainable params |
|---|---|---|---|
| Fine tuning | No experts | 21.2 | 44.0 |
| LoRA | No experts | 21.2 | 4.3 |
| CAT | Static fusion | 29.6 | under 0.1 |
| MAS-LoRA | Static fusion | 35.9 | under 0.1 |
| LanFusion | Dynamic fusion | 17.6 | 34.6 |
| P-CAM | Dynamic fusion | 17.7 | 34.3 |
| VeLoRA | Vector fusion | 18.0 | under 0.1 |
| FasterVeLoRA | Vector fusion | 22.6 | under 0.1 |
VeLoRA again lands close to the dynamic methods, 18.0 against 17.6 and 17.7, and is best on speaker 13 with 8.7 percent. FasterVeLoRA tells a different story. At 22.6 percent it is worse than both plain LoRA and fine tuning, at 21.2, and worse than VeLoRA on every one of the nine speakers. The authors attribute the larger gap to higher variability between speakers than between accents, which is plausible. When the speakers differ a lot, averaging their adapters into one set of weights blurs them, and a gain per channel cannot undo the blur. The paper’s statement that FasterVeLoRA outperforms static fusion by a large margin is accurate, but on this task it does not beat simply training LoRA on the target speaker. The fine tuning row also shows that adapting to one hour of dysarthric speech is hard. Character error stays above 20 percent even with full fine tuning.
The gap between this test and assistive use
The setting that motivates this experiment is a person with dysarthria using recognition in daily life, for messaging, dictation or controlling devices. Several distances separate the test from that setting. Only mild dysarthria was tested, and the authors removed the severe cases because adaptation failed, so nothing here speaks to the people who struggle most with standard recognition. All speakers come from one Chinese corpus and one recording protocol. Each target speaker contributes about an hour of speech, of which roughly 85 percent is used for training, an amount that may be burdensome to record for someone whose speech is effortful. Error rates near 18 percent mean roughly one character in five or six is wrong, which may still be too high for many everyday tasks, and the paper does not measure usability with the people concerned.
The privacy argument also needs care. The paper suggests that fusing experts is attractive because only the adapters, not the sensitive recordings, need to be shared. That is a real advantage over sharing audio, but an adapter trained on one person’s hour of speech is itself derived from their data. Models fine tuned on small datasets can retain information about the people in them, and whether these speaker adapters leak anything about their speakers was not examined. Sharing per speaker adapters should be treated as sharing a derivative of health related data until shown otherwise, and our overview of MITRE ATLAS and adversarial attacks on AI maps the kinds of attacks that target trained models directly. Our coverage of federated learning for surgical video discusses similar trade offs between sharing models and sharing data in medicine.
Our own toy check
To see the mechanisms without Whisper, we wrote a PyTorch reconstruction of every method and ran it on a synthetic problem. A small network is pretrained on clean inputs. Each “accent” distorts the inputs with its own mixture of shared low rank distortions, so experts trained on other accents carry partial knowledge of a new one. Nine source domains get a rank 4 LoRA expert each, the best five are selected for the target by error on a development set, and every method is trained on either 300 or 3000 target samples. Results are averaged over three seeds.
| Method | Error, 300 target samples | Error, 3000 target samples | Trainable |
|---|---|---|---|
| Backbone, no adaptation | 39.6 | 39.4 | 0 |
| LoRA on target data | 27.3 | 17.5 | 1136 |
| LoRA on target plus all source data | 34.2 | 26.1 | 1136 |
| Static fusion, plain averaging | 37.1 | 37.6 | 0 |
| Static fusion, learned expert weights | 33.3 | 31.8 | 15 |
| Dynamic fusion, experts fine tuned | 26.7 | 16.7 | 5680 |
| VeLoRA | 27.7 | 23.4 | 192 |
| FasterVeLoRA | 29.9 | 25.1 | 132 |
With 300 target samples the paper’s ordering appears in miniature. Static fusion barely helps, dynamic fusion is best, and VeLoRA lands within a point of it with 30 times fewer trainable parameters, essentially tied with LoRA trained on the target. FasterVeLoRA gives up a little more. With ten times more target data the picture changes. LoRA and dynamic fusion keep improving, to 17.5 and 16.7 percent, while VeLoRA stops at 23.4. That is the ceiling the rank one view predicts. When there is enough data to learn new directions, a method that can only reweight existing ones falls behind.
The pooled data control did not behave as we expected. Training LoRA on the target plus all nine source domains was worse than training on the target alone, because our synthetic domains distort the inputs in conflicting ways and one adapter cannot serve them all. Real accents of English share far more than our random distortions do, so pooling may well help there. The point is that the paper’s comparison cannot tell, because the control is missing.
The merge checks behaved as described above. VeLoRA’s merged layers reproduced the training outputs to within about \(10^{-5}\), and so did our corrected FasterVeLoRA merge. Merging FasterVeLoRA the way the released code does, with the bias left unscaled, changed outputs by up to 1.7 logits and raised test error by about half a point. In our toy the vectors drifted much further from 1 than they probably do in the paper’s short training runs, so the real effect is likely smaller.
These are toy numbers on a small synthetic network. They show how the methods behave relative to each other, not how they perform on Whisper.
Limitations of the study
- Unequal data. VeLoRA’s experts encode about 119 hours of extra accented speech that the LoRA and fine tuning baselines never see, and no pooled data baseline is reported.
- Friendly expert pools. All experts were trained in house on the same corpus, backbone and rank. Experts from different sources, the scenario used to motivate the method, were not tested.
- Single runs. All results use one training seed. The bootstrap intervals reflect test set sampling only, and many differences between methods are 0.1 or 0.2 points.
- Limited coverage of dysarthria. Nine speakers with mild dysarthria from one Chinese corpus, with severe cases excluded because adaptation did not converge.
- FasterVeLoRA on speakers. On dysarthric speech it underperforms plain LoRA and fine tuning, which the summary claims of effectiveness across domains do not reflect.
- Released code. The FasterVeLoRA layer leaves the bias unscaled when merging for inference, so its reported results come from a model slightly different from the trained one.
- Selection needs labels. Choosing experts requires a labeled development set in the target domain. The authors note selection becomes unreliable when that set is very small.
- A shared name. An unrelated NeurIPS 2024 method for memory efficient training is also called VeLoRA, so searches for code or follow up work should check which one they find.
What practitioners can take from this
If you already maintain a library of LoRA adapters for a speech or language model, VeLoRA is an easy experiment. It needs no change to the backbone, it merges into ordinary weights for inference, and the per domain artifact is a few megabytes at most. Start by measuring plain averaging of the best few experts, which costs nothing to train, then add the vectors. The gap between the two tells you how much the vectors earn on your data. Our article on ShareBERT and parameter sharing covers another way of buying efficiency by reusing weights the model already has.
Be careful about the regime. In our toy test, the vector only methods did well with little target data and fell well behind LoRA and dynamic fusion with ten times more, which is what the rank one view predicts. With 17 hours of target speech the paper still found VeLoRA competitive, so the ceiling depends on how close your experts are to your target. If you have plenty of target data and the experts are only loosely related, training LoRA on the target, or fine tuning the experts, may be the better use of compute.
If you deploy FasterVeLoRA from the released code, patch the merge so the bias is scaled too, and check that the merged model reproduces the training mode outputs on a few batches before trusting its error rates. That check takes seconds and would have caught the problem. For streaming or on device recognition, where every millisecond of latency counts, the merged form of either method is as fast as the base model, a point our analysis of delayed knowledge distillation for streaming speech recognition shows is often the binding constraint. More work on compressing and adapting large models sits in our Knowledge Distillation and Model Compression section, and the same parameter efficient ideas appear in vision in our analysis of MIFOMO for few shot hyperspectral classification.
Conclusion
VeLoRA’s core achievement is a clean answer to a practical question. Given a library of frozen LoRA adapters, a few scaling vectors per layer are enough to fuse them into an adapter for a new accent that performs within a tenth of a point of methods that fine tune every expert, while training and storing a few hundred times fewer parameters. On Whisper base that is about 89,000 numbers per new domain.
The conceptual shift is to adjust experts indirectly. Instead of choosing how much of each expert to use, or retraining the experts themselves, VeLoRA learns how much of each rank one direction inside the experts to keep, and how strongly to pass each output channel. Initialized at ones, it starts as plain averaging, and the ablation shows the learned vectors adding 1.6 points of word error reduction on Whisper base beyond that.
The idea travels beyond speech. Any setting with a growing library of adapters for one backbone, language models for different customers, vision models for different sensors, could use the same recipe, and the merged form makes it free at inference. The precondition is that the library already contains the directions the new domain needs.
The limits are as clear as the gains. The advantage over plain LoRA comes largely from the extra data inside the experts, with no pooled data control. The expert pools were built under ideal conditions, all results come from single runs, FasterVeLoRA falls behind plain LoRA on dysarthric speech, the dysarthria test excludes severe cases, and the released FasterVeLoRA code merges the bias incorrectly. None of these undermine the method, but each narrows what the numbers prove.
The useful next experiments are straightforward. Fuse experts from genuinely different sources, add a pooled data baseline, report several seeds, fix the merge, and test with people who have more severe dysarthria together with clinicians. If VeLoRA holds up there, a library of adapters will become something you can draw on without retraining it. For now it is a well engineered and honestly limited way to reuse what an organization has already trained.
A reference implementation in PyTorch
The listing below is our own editorial reconstruction, not the authors’ code, which is available in their ESPnet fork. It implements LoRA, static fusion with fixed or learned expert weights, dynamic fusion with trainable experts, VeLoRA as in Eq. 4 with expert specific \(\mathbf{a}_i\) and a shared \(\mathbf{b}\), and FasterVeLoRA as in Eqs. 5 and 6. Both merge into plain linear layers for inference, with the FasterVeLoRA merge scaling the bias correctly and a second function reproducing the released behaviour for comparison. It also includes the source data free expert selection of Algorithm 3, the idealized parameter counts of Table 3, an exact counter that reproduces the Whisper base figure, and the toy study above. It runs on a laptop CPU in about a minute.
"""
VeLoRA and FasterVeLoRA, vector based fusion of frozen LoRA experts.
Editorial PyTorch reconstruction of
Hu, Zhang, Han, Zhang and Cai,
"Vector-based LoRA expert fusion for efficient speech recognition",
Information Fusion 140 (2027) 104782.
The authors' own code lives in an ESPnet fork (branch LoraExpertTuning) and a loralib fork, both on GitHub.
This file is NOT that code. It restates Eqs. 1 to 6 and Algorithms 1 to 3 in plain PyTorch, adds the
baselines the paper compares against in simplified form, and runs a small synthetic test on a CPU.
Notation follows the paper. A linear layer is y = x W0 with W0 in R^{d x k}. A LoRA expert is A in R^{d x r}
and B in R^{r x k}. "*" is elementwise multiplication with broadcasting, so A * a scales the columns of A.
Contents
1. Layers LoRA, static fusion (SEF), dynamic fusion (DEF), VeLoRA (Eq. 4), FasterVeLoRA (Eqs. 5, 6)
2. Merging inference merges, including the bias, which matters for FasterVeLoRA
3. Wrapping swap every nn.Linear of a model for one of the layers above
4. Selection source data free selection of the top N experts (Algorithm 3)
5. Counting the idealised counts of Table 3 and an exact count for Whisper shaped models
6. Toy study synthetic "accents", a frozen backbone, a pool of experts and every method side by side
"""
import copy
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
# =====================================================================================
# 1. Layers. Each wraps a frozen nn.Linear. Weights follow nn.Linear (out x in), so x W0 = F.linear(x, weight).
# =====================================================================================
class LoRALinear(nn.Module):
"""Eq. 1, y = x W0 + x A B, A random, B zero, both trained. Scaling alpha / r = 1 as in the paper's setup."""
def __init__(self, base, r):
super().__init__()
self.base = base.requires_grad_(False)
self.A = nn.Parameter(torch.empty(base.in_features, r))
nn.init.kaiming_uniform_(self.A.T, a=math.sqrt(5))
self.B = nn.Parameter(torch.zeros(r, base.out_features))
def forward(self, x):
return self.base(x) + x @ self.A @ self.B
def expert(self):
return self.A.detach().clone(), self.B.detach().clone()
class _ExpertBank(nn.Module):
"""Concatenates N experts into one d x R and one R x k matrix, the efficient form of Section 5."""
def __init__(self, base, experts, trainable=False):
super().__init__()
self.base = base.requires_grad_(False)
self.N = len(experts)
self.ranks = [A.shape[1] for A, _ in experts]
A = torch.cat([A for A, _ in experts], 1)
B = torch.cat([B for _, B in experts], 0)
if trainable:
self.A, self.B = nn.Parameter(A.clone()), nn.Parameter(B.clone())
else:
self.register_buffer("A", A.clone())
self.register_buffer("B", B.clone())
class SEFLinear(_ExpertBank):
"""Static expert fusion, frozen experts. learn_coef=False is plain weight averaging (MAS-LoRA style).
learn_coef=True learns one coefficient per expert (CAT style)."""
def __init__(self, base, experts, learn_coef=True):
super().__init__(base, experts, trainable=False)
coef = torch.full((self.N,), 1.0 / self.N)
self.coef = nn.Parameter(coef) if learn_coef else nn.Parameter(coef, requires_grad=False)
def forward(self, x):
scale = torch.repeat_interleave(self.coef, torch.tensor(self.ranks))
return self.base(x) + ((x @ self.A) * scale) @ self.B
class DEFLinear(_ExpertBank):
"""Dynamic expert fusion with uniform weights, the experts themselves are fine tuned (LanFusion style)."""
def __init__(self, base, experts):
super().__init__(base, experts, trainable=True)
def forward(self, x):
return self.base(x) + (x @ self.A) @ self.B / self.N
class VeLoRALinear(_ExpertBank):
"""Eq. 4, y = x W0 + (1/N) sum_i x (A_i * a_i)(B_i * b). The a_i are expert specific, b is shared.
Both start at 1, so at step zero the layer equals plain uniform averaging of the experts."""
def __init__(self, base, experts):
super().__init__(base, experts, trainable=False)
self.a = nn.Parameter(torch.ones(self.A.shape[1]))
self.b = nn.Parameter(torch.ones(self.B.shape[1]))
def forward(self, x):
return self.base(x) + ((x @ self.A) * self.a) @ self.B * self.b / self.N
def merged(self):
"""Algorithm 1, line 6. Returns a plain nn.Linear, so inference costs nothing extra."""
lin = copy.deepcopy(self.base)
delta = (self.A * self.a) @ (self.B * self.b) / self.N # d x k
lin.weight.data += delta.T.detach()
return lin
class FasterVeLoRALinear(nn.Module):
"""Eqs. 5 and 6. Average the experts, fold them into W0 once, then learn one output scaling vector b.
The layer computes (x W0' + bias) * b, so b also scales the bias, which is how the released layer
behaves in training mode."""
def __init__(self, base, experts):
super().__init__()
lin = copy.deepcopy(base).requires_grad_(False)
delta = sum(A @ B for A, B in experts) / len(experts)
lin.weight.data += delta.T
self.lin = lin
self.b = nn.Parameter(torch.ones(base.out_features))
def forward(self, x):
return self.lin(x) * self.b
def merged(self):
"""Correct merge, scale every output row of the weight AND the bias by b."""
lin = copy.deepcopy(self.lin)
lin.weight.data *= self.b.detach().unsqueeze(1)
if lin.bias is not None:
lin.bias.data *= self.b.detach()
return lin
def merged_as_released(self):
"""What the released FasterVeLoRA layer does when switched to eval mode, as we read it.
The weight is scaled by b but the bias is not, so inference differs from the trained function
by (b - 1) * bias on every output channel."""
lin = copy.deepcopy(self.lin)
lin.weight.data *= self.b.detach().unsqueeze(1)
return lin
# =====================================================================================
# 3. Wrapping a model
# =====================================================================================
def linear_names(model):
return [n for n, m in model.named_modules() if isinstance(m, nn.Linear)]
def _set(model, name, new):
parent = model
*path, last = name.split(".")
for p in path:
parent = getattr(parent, p)
setattr(parent, last, new)
def _get(model, name):
m = model
for p in name.split("."):
m = getattr(m, p)
return m
def wrap(backbone, method, experts=None, r=4, learn_coef=True):
"""experts: list of dicts {linear name: (A, B)}, one dict per expert."""
model = copy.deepcopy(backbone)
for p in model.parameters():
p.requires_grad_(False)
for name in linear_names(model):
base = _get(model, name)
ex = [e[name] for e in experts] if experts else None
if method == "lora":
new = LoRALinear(base, r)
elif method == "sef":
new = SEFLinear(base, ex, learn_coef)
elif method == "def":
new = DEFLinear(base, ex)
elif method == "velora":
new = VeLoRALinear(base, ex)
elif method == "faster":
new = FasterVeLoRALinear(base, ex)
else:
raise ValueError(method)
_set(model, name, new)
return model
def merge_for_inference(model, as_released=False):
"""Replace every VeLoRA or FasterVeLoRA layer with a plain merged nn.Linear."""
model = copy.deepcopy(model)
for name, m in list(model.named_modules()):
if isinstance(m, VeLoRALinear):
_set(model, name, m.merged())
elif isinstance(m, FasterVeLoRALinear):
_set(model, name, m.merged_as_released() if as_released else m.merged())
return model
def n_trainable(model):
return sum(p.numel() for p in model.parameters() if p.requires_grad)
# =====================================================================================
# 4. Source data free selection, Algorithm 3. Score every expert on target development data, keep the best N.
# =====================================================================================
def select_experts(pool, evaluate, N):
"""pool: {expert name: expert dict}. evaluate(expert dict) -> error on target dev data, lower is better."""
scored = sorted(((evaluate(e), name) for name, e in pool.items()))
return [name for _, name in scored[:N]], scored
# =====================================================================================
# 5. Counting trainable parameters
# =====================================================================================
def idealized_count(method, L, d, N, r):
"""Table 3 of the paper."""
return {"FT": L * d * d, "LoRA": 2 * L * d * r, "SEF": L * N, "DEF": 2 * L * N * d * r + L * N,
"VeLoRA": L * (d + N * r), "FasterVeLoRA": L * d}[method]
def whisper_vector_count(d, ff, n_enc, n_dec, N, r, faster=False):
"""Exact VeLoRA (or FasterVeLoRA) count for a Whisper style encoder decoder in which every linear layer is
adapted. Encoder blocks hold q, k, v, out and two feed forward layers, decoder blocks add four cross attention
projections. Each layer gets b (its output size) and, for VeLoRA, a (N * r)."""
enc_out = [d, d, d, d, ff, d]
dec_out = [d] * 8 + [ff, d]
total = n_enc * sum(enc_out) + n_dec * sum(dec_out)
if not faster:
total += (n_enc * len(enc_out) + n_dec * len(dec_out)) * N * r
return total
# =====================================================================================
# 6. Toy study. Synthetic "accents": every domain distorts the same inputs with its own mix of shared
# rank one distortions, so other domains' experts carry useful knowledge about a new one.
# =====================================================================================
D_IN, HIDDEN, CLASSES, N_BASIS = 24, 64, 4, 8
def make_world(seed):
g = torch.Generator().manual_seed(seed)
teacher = nn.Sequential(nn.Linear(D_IN, 16), nn.Tanh(), nn.Linear(16, CLASSES))
with torch.no_grad():
teacher[0].weight.copy_(1.5 * torch.randn(16, D_IN, generator=g) / math.sqrt(D_IN))
teacher[2].weight.copy_(torch.randn(CLASSES, 16, generator=g))
teacher[0].bias.zero_(); teacher[2].bias.zero_()
basis = [0.5 * torch.randn(D_IN, 1, generator=g) @ torch.randn(1, D_IN, generator=g) / math.sqrt(D_IN)
for _ in range(N_BASIS)]
def domain(coefs):
return torch.eye(D_IN) + sum(c * P for c, P in zip(coefs, basis))
def sample(M, n, gen):
x = torch.randn(n, D_IN, generator=gen)
with torch.no_grad():
y = teacher(x).argmax(1)
return x @ M, y
def random_domain(gen):
c = torch.zeros(N_BASIS)
idx = torch.randperm(N_BASIS, generator=gen)[:3]
c[idx] = 1.2 * torch.randn(3, generator=gen)
return domain(c)
return sample, random_domain
def backbone_net():
return nn.Sequential(nn.Linear(D_IN, HIDDEN), nn.ReLU(), nn.Linear(HIDDEN, HIDDEN), nn.ReLU(),
nn.Linear(HIDDEN, CLASSES))
def train(model, x, y, steps, lr, bs=128, seed=0):
params = [p for p in model.parameters() if p.requires_grad]
if not params:
return model
opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
g = torch.Generator().manual_seed(seed)
for _ in range(steps):
idx = torch.randint(0, x.shape[0], (min(bs, x.shape[0]),), generator=g)
loss = F.cross_entropy(model(x[idx]), y[idx], label_smoothing=0.1) # the paper uses label smoothing too
opt.zero_grad(); loss.backward(); opt.step()
return model
@torch.no_grad()
def error(model, x, y):
return 100.0 * (model(x).argmax(1) != y).double().mean().item()
def lora_state(model):
return {name: m.expert() for name, m in model.named_modules() if isinstance(m, LoRALinear)}
def run_toy(seed=0, n_target=300, n_source=1500, n_pool=9, N=5, r=4, verbose=False):
torch.manual_seed(seed)
gen = torch.Generator().manual_seed(seed + 100)
sample, random_domain = make_world(seed)
# backbone, pretrained on undistorted speech, the stand in for Whisper
bb = backbone_net()
xs, ys = sample(torch.eye(D_IN), 8000, gen)
train(bb, xs, ys, steps=3000, lr=3e-3, seed=seed)
bb.requires_grad_(False)
# expert pool, one LoRA expert per source domain
src = [random_domain(gen) for _ in range(n_pool)]
pool, src_data = {}, []
for i, M in enumerate(src):
x, y = sample(M, n_source, gen)
src_data.append((x, y))
pool[f"domain{i}"] = lora_state(train(wrap(bb, "lora", r=r), x, y, steps=600, lr=3e-3, seed=seed + i))
# target domain with limited data, a small development set and a large test set
Mt = random_domain(gen)
xt, yt = sample(Mt, n_target, gen)
xd, yd = sample(Mt, 300, gen)
xe, ye = sample(Mt, 4000, gen)
names, _ = select_experts(pool, lambda e: error(wrap(bb, "sef", [e], learn_coef=False), xd, yd), N)
ex = [pool[n] for n in names]
out = {}
def report(name, model):
out[name] = (error(model, xe, ye), n_trainable(model))
report("backbone, no adaptation", bb)
report("LoRA on target data", train(wrap(bb, "lora", r=r), xt, yt, 600, 3e-3, seed=seed))
xp = torch.cat([xt] + [x for x, _ in src_data]); yp = torch.cat([yt] + [y for _, y in src_data])
report("LoRA on target plus all source data", train(wrap(bb, "lora", r=r), xp, yp, 1200, 3e-3, seed=seed))
report("SEF, plain averaging, no training", wrap(bb, "sef", ex, learn_coef=False))
report("SEF, learned expert weights", train(wrap(bb, "sef", ex), xt, yt, 600, 8e-3, seed=seed))
report("DEF, experts fine tuned", train(wrap(bb, "def", ex), xt, yt, 600, 3e-3, seed=seed))
velora = train(wrap(bb, "velora", ex), xt, yt, 600, 8e-3, seed=seed) # 8e-3, as in Table A.1
report("VeLoRA", velora)
faster = train(wrap(bb, "faster", ex), xt, yt, 600, 8e-3, seed=seed)
report("FasterVeLoRA", faster)
# merge checks
with torch.no_grad():
v_gap = (merge_for_inference(velora)(xe) - velora(xe)).abs().max().item()
f_ok = (merge_for_inference(faster)(xe) - faster(xe)).abs().max().item()
f_rel = (merge_for_inference(faster, as_released=True)(xe) - faster(xe)).abs().max().item()
b_dev = max((m.b - 1).abs().max().item() for m in faster.modules() if isinstance(m, FasterVeLoRALinear))
out["_merge"] = (v_gap, f_ok, f_rel, b_dev,
error(merge_for_inference(faster), xe, ye), error(merge_for_inference(faster, True), xe, ye))
out["_selected"] = names
return out
if __name__ == "__main__":
# ---- unit checks -------------------------------------------------------------------------------
torch.manual_seed(0)
base = nn.Linear(6, 5)
exps = [(torch.randn(6, 2), torch.randn(2, 5)) for _ in range(3)]
x = torch.randn(4, 6)
v = VeLoRALinear(base, exps)
s = SEFLinear(base, exps, learn_coef=False)
assert torch.allclose(v(x), s(x), atol=1e-6), "VeLoRA with a = b = 1 must equal uniform averaging"
with torch.no_grad():
v.a.uniform_(0.5, 1.5); v.b.uniform_(0.5, 1.5)
assert torch.allclose(v.merged()(x), v(x), atol=1e-5), "merge must not change the function"
f = FasterVeLoRALinear(base, exps)
with torch.no_grad():
f.b.uniform_(0.5, 1.5)
assert torch.allclose(f.merged()(x), f(x), atol=1e-5)
assert not torch.allclose(f.merged_as_released()(x), f(x), atol=1e-3), "unscaled bias changes the output"
assert whisper_vector_count(512, 2048, 6, 6, 7, 32) == 89088 # Table 4 reports 0.090M for Whisper base
assert whisper_vector_count(512, 2048, 6, 6, 7, 32, faster=True) == 67584
print("unit checks passed")
print("VeLoRA trainable vectors, Whisper base, 7 experts of rank 32:",
whisper_vector_count(512, 2048, 6, 6, 7, 32))
print("VeLoRA trainable vectors, Whisper medium:", whisper_vector_count(1024, 4096, 24, 24, 7, 32))
print("FasterVeLoRA, Whisper base:", whisper_vector_count(512, 2048, 6, 6, 7, 32, faster=True),
" Whisper medium:", whisper_vector_count(1024, 4096, 24, 24, 7, 32, faster=True))
# ---- toy study ------------------------------------------------------------------------------------
for n_target in (300, 3000):
runs = [run_toy(seed=s, n_target=n_target) for s in range(3)]
print(f"\ntarget training samples = {n_target}, mean test error over 3 seeds")
print(f"{'method':<38} {'error %':>8} {'trainable':>10}")
for k in runs[0]:
if k.startswith("_"):
continue
err = sum(r[k][0] for r in runs) / len(runs)
print(f"{k:<38} {err:8.2f} {runs[0][k][1]:>10}")
m = [r["_merge"] for r in runs]
print(f"merge check, max output change: VeLoRA {max(t[0] for t in m):.1e}, FasterVeLoRA correct "
f"{max(t[1] for t in m):.1e}, FasterVeLoRA with unscaled bias {max(t[2] for t in m):.3f} "
f"(largest |b - 1| = {max(t[3] for t in m):.3f})")
print(f"FasterVeLoRA test error after merging, correct {sum(t[4] for t in m) / 3:.2f}, "
f"unscaled bias {sum(t[5] for t in m) / 3:.2f}")
print("done")
Frequently asked questions
What is VeLoRA?
VeLoRA is a method from Hebei University of Technology, published in Information Fusion, for adapting a speech recognition model such as Whisper to a new domain by fusing existing LoRA adapters. The adapters stay frozen, and only small scaling vectors that rescale their matrices are trained on the new domain.
How is VeLoRA different from VeRA?
VeRA learns scaling vectors on top of frozen random matrices shared by all layers. VeLoRA replaces the random matrices with several pretrained LoRA experts, starts the vectors at 1 so training begins from plain averaging of the experts, and learns one vector per expert on the rank side plus one shared vector on the output side of each layer.
How many parameters does VeLoRA train?
With seven experts of rank 32 on Whisper base it trains about 89,000 parameters, reported as 0.090 million in the paper, against 4.3 million for LoRA and 30.3 million for the strongest fusion method that fine tunes its experts. On Whisper medium it trains about 0.62 million.
What is the difference between VeLoRA and FasterVeLoRA?
FasterVeLoRA averages the experts once, adds the average into Whisper’s weights and then learns only one gain per output channel of each layer. It trains 4.8 times faster than VeLoRA on Whisper medium, with average word error rising from 3.2 to 3.3 percent, but it loses more accuracy on Whisper base and on dysarthric speech.
Does VeLoRA work for dysarthric speech?
On nine Chinese speakers with mild dysarthria it reached an average character error rate of 18.0 percent, close to the best fusion methods at 17.6 and better than plain LoRA at 21.2. Speakers with severe dysarthria were not tested because adaptation often failed to converge, so the results say nothing about them.
Is the VeLoRA code available?
Yes. The authors released their code as a fork of the ESPnet toolkit on GitHub, on the LoraExpertTuning branch, together with a modified loralib. In our reading, the released FasterVeLoRA layer leaves the bias unscaled when merging for inference, which is easy to fix.
Read the paper and the code
The paper includes per accent results for every method, the ablations on vector sharing, initialization, rank and expert count, and the full dysarthric speaker table. The authors’ code is in a public ESPnet fork, on the LoraExpertTuning branch.
Hu, Q., Zhang, Y., Han, Z., Zhang, X., and Cai, Z. Vector based LoRA expert fusion for efficient speech recognition. Information Fusion 140 (2027) 104782. DOI 10.1016/j.inffus.2026.104782. Received 10 November 2025, revised 19 August 2026, accepted 12 September 2026, available online 16 September 2026. Paper title written here without its hyphen, following our style. Our reading of the released code refers to the LoraExpertTuning branch of github.com/WZLHQ/espnet and the loralib fork at github.com/WZLHQ/LoRA, as available on 20 November 2026.
This analysis is based on the published paper and an independent evaluation of its claims.
