Large language models — architectures, training, evaluation, and the tools built on them.
Omnimodal Large Language Models (OLLMs) like GPT-4o and Megrez have revolutionized how AI interacts with the world by seamlessly processing text, images, and audio. However, a critical performance gap persists: OLLMs perform significantly better with vision-text inputs than with vision-audio…
Enhancing Vision-Audio Capability in Omnimodal LLMs with Self-KD Read More »
Picture a company that ships a heavily guarded chatbot with an invisible watermark stitched into every reply, confident that any leaked or resold output can be traced back to its own servers. Now picture a small team, working with a…
Knowledge Distillation Can Forge and Erase LLM Watermarks Read More »
In the rapidly evolving world of artificial intelligence, Large Language Models (LLMs) are hitting performance ceilings—especially when it comes to complex reasoning tasks like math and logic. But what if the key to unlocking their next-level intelligence lies not in…
Machine Learning › Knowledge Distillation › Paper Analysis aitrendblend.com · Knowledge Distillation A seven billion parameter model writes a clean, helpful answer, then you try to ship it. It will not fit on the phone, the latency blows past your…
ToDi: Per Token KL Divergence Control for LLM Distillation Read More »
A well funded lab spends months and a serious pile of compute training a frontier language model. A smaller competitor never touches that budget. Instead it sends a few thousand prompts to the public API, saves the answers, and fine…
Doge: Stopping LLM Knowledge Theft With One Fine Tuned Layer Read More »
Most teams that want a smaller, cheaper version of a large language model end up stitching together the same handful of techniques by hand, supervised fine-tuning on a bigger model’s outputs, maybe a logit matching loss if they have access…
EasyDistill Packages LLM Distillation, But Its Top-k Shortcut Has A Cost Read More »
Most teams training a reasoning model today pick a side. Either they distill from a stronger teacher and accept that the student will never think a thought its teacher never had, or they run pure reinforcement learning and accept the…
KDRL Unifies Distillation And Reinforcement Learning Read More »
Shrinking a large language model down to a fraction of its size usually means teaching a small student model to imitate a much larger teacher. That imitation process is trickier than it sounds. Feed the student every training example at…
POCL: How A Progressive Overload Curriculum Improves LLM Distillation Read More »