Revolutionizing Keratitis Diagnosis: How Vision Transformers Are Transforming Eye Care

Analysis by the aitrendblend editorial team. Source paper, Won, Kim, Jeon, Cha and Lim, Computers in Biology and Medicine, 2025.

Why bacteria versus fungus is the question that matters most

Infectious keratitis is a genuine ocular emergency. Left untreated or treated with the wrong drug class, it can perforate the cornea, seed a deeper eye infection, and cause permanent vision loss. Bacterial and fungal organisms account for the overwhelming majority of cases, and the two demand different treatment paths entirely, with fungal keratitis in particular carrying worse visual outcomes and a much higher chance of needing surgery to resolve. Getting the call right, and getting it right quickly, changes what happens to the patient’s eye.

The trouble is that the two conditions can look alike, especially early on. Textbook fungal keratitis has irregular feathery borders, satellite lesions scattered around the main ulcer, and endothelial plaques, but the authors note plainly that many cases present atypically and are hard to tell apart from bacterial infection by eye, particularly before the disease has fully declared itself. That diagnostic ambiguity is exactly the gap this paper tries to close with imaging alone, without waiting on a culture result.

The diagnostic bottleneck a culture creates

Corneal scraping and culture remains the reference standard, but it is slow and it is not available everywhere. The authors point out directly that this creates real barriers in developing countries and in regions where access to specialized eye care is limited, precisely the settings where a fast, camera based screening tool would matter most. Earlier work from this same group had already shown a convolutional network, built on ResNet-50, could classify bacterial and fungal keratitis from anterior segment photographs with an accuracy of 87.8 percent. This paper asks whether a newer architecture, and a smarter way of combining multiple photograph types, can do better.

Three ways to let three photographs talk to each other

Anterior segment photography is not one image type, it is several. Broad-beam images give a wide, evenly lit view of the eye. Slit-beam images use a narrow light source to show depth and layered structure within the cornea. Blue-light images, typically taken after fluorescein staining, highlight the corneal surface defect itself. Each view carries different diagnostic information, and the paper’s central methodological question is how a deep learning model should combine them.

The researchers tested three fusion strategies, deliberately ordered from simplest to most structured.

Simply add

The most naive approach pools every image type into one dataset and trains a single Vision Transformer on the mixture, letting the network sort out any relationship between image types on its own, with no explicit mechanism to do so. This was also the only strategy compatible with a single image type, which let the authors use it as a like for like comparison against a plain single modality model.

Vector add

Here each image type gets its own dedicated Vision Transformer. Every transformer produces a classification token, the single summary vector a Vision Transformer uses to make its final prediction, and the vectors from each separately trained model are simply summed together before the final classification layer. This gives each modality its own feature extractor, but the combination step itself is still a blunt instrument, addition with no weighting or interaction.

Self attention

The most structured approach again trains a separate Vision Transformer per image type, but instead of just adding the classification tokens together, it lets them interact. Each classification token becomes a query, and the tokens from the other image types serve as the keys and values in a standard self attention calculation, meaning the model computes how much each image type’s summary should attend to information carried by the other image types before the tokens are concatenated for final classification. This is a meaningfully different operation from vector add, since it lets one image type’s diagnostic signal get modulated by what another image type is showing, rather than treating them as independent inputs added at the end.

Because a given patient can contribute several photographs of each type, the vector add and self attention strategies do not simply pair one broad-beam image with one slit-beam image. They compute every possible pairing across a patient’s full set of images, following the formula above where \(B_i\), \(S_i\) and \(L_i\) are the counts of broad-beam, slit-beam and blue-light images available for the ith patient. This multiplies the effective size of the training set considerably beyond the raw photograph count, which matters a great deal for a study built on only 79 patients.

What the numbers show

The dataset behind all of this comprised 2,089 images from 79 patients, 1,235 associated with bacterial keratitis and 854 with fungal keratitis, split into 498 broad-beam, 995 slit-beam and 596 blue-light photographs. Patients, not images, were split into training and validation sets at a ratio of 62 to 17, and the whole process was repeated across three non overlapping cross validation folds.

Two things about this table are worth sitting with. First, self attention fusion beat vector add fusion on every single image combination tested, and both beat the naive simply add strategy in the multi image conditions. Second, the single best AUROC score in the entire study, 0.93, came from combining only two of the three image types, broad-beam and slit-beam, not all three together. Adding blue-light images on top actually pulled AUROC down slightly to 0.91, even though it pushed AUPRC up to its own best score of 0.93. More image types is not automatically better, which image types you combine, and how, matters more than how many you throw at the model.

The naive approach produced a genuinely instructive failure. When broad-beam and slit-beam images were simply pooled into one Vision Transformer with no fusion mechanism at all, AUROC dropped to 0.67, worse than either single image type Vision Transformer on its own. Stacking more visual information into an unstructured model actively hurt performance here, which is a useful caution against assuming that giving a model more data automatically helps if the model has no way to relate that data internally.

Our results indicate that a single ViT model cannot efficiently utilize different types of images. In cases involving two or more types of images, the performance of a single model decreased compared with that of cases with only one type of image. Won, Kim, Jeon, Cha and Lim, Computers in Biology and Medicine, 2025

How this compares against a standard convolutional network

The team ran a ResNet-50 baseline through the identical set of image combinations, chosen specifically because it has a similar parameter count to the base Vision Transformer, which keeps the comparison reasonably fair rather than pitting a small model against a large one. ResNet-50 topped out at an AUROC of 0.77 using blue-light images alone, and its combined image conditions actually performed worse than several of its single image conditions, peaking at 0.72 for all three image types together. Every Vision Transformer configuration in the study, including the weakest single image ViT at 0.71, either matched or beat ResNet-50’s best result, and the strongest ViT configurations cleared it by a wide margin.

Set against earlier published work, the gap looks even larger. The authors cite a DenseNet161 model reaching an AUROC of 0.85 and a ResNet-50 model reaching 0.82 in prior literature, and a VGG19 based DeepKeratitis model reaching an AUPRC of 0.86. Their own self attention Vision Transformer’s AUROC of 0.93 and AUPRC of 0.93 sit above both of those benchmarks, though it is worth noting these are different patient populations and different institutions, so the comparison is suggestive rather than a controlled head to head.

Two surprising results from the ablation study

Two findings in the ablation section cut against what most people would assume going in, and the authors do not smooth them over.

The first concerns transfer learning. The team pretrained their Vision Transformers using DINO, a self supervised method, and compared pretrained against randomly initialized models. For the single image type model and the naive simply add model, pretraining helped, pushing AUROC from 0.93 to 0.94 and from 0.89 to 0.94 respectively. But for the two multi model fusion strategies, vector add and self attention, pretraining actually hurt slightly, with self attention dropping from 0.97 without pretraining to 0.95 with it. The authors’ explanation is that the DINO checkpoint was learned on single, standalone images, so it gives each separately trained Vision Transformer a strong but individually biased starting point, one that does not necessarily help when the whole point of the fusion mechanism is to find features shared or complementary across image types rather than idiosyncratic to any one of them.

The second concerns image resolution. Conventional wisdom in computer vision says higher resolution should help, or at worst not hurt. Here, for every fusion strategy except the single image type model, 128 by 128 pixel images outperformed 224 by 224 pixel images, sometimes by a wide margin, the naive simply add strategy dropped from an AUROC of 0.94 at low resolution to just 0.89 at higher resolution. Only the single image type model followed the expected pattern, doing slightly better at the higher resolution. The authors’ reading is that when a model is trying to extract features common across several image types simultaneously, a lower resolution may make those shared, coarser patterns easier to find, while a higher resolution may just add more image specific detail that a fusion mechanism has to work around rather than benefit from.

The clinical translation gap

It is worth being direct about how far a 0.93 AUROC in this study sits from a validated clinical tool. The entire dataset comes from a single hospital, Samsung Medical Center, and a single healthcare system, which means the model has never been tested against a different population, a different camera setup, or a different local mix of infecting organisms, all of which can vary by region and climate as the authors themselves note in their introduction. Fungal keratitis is more common in tropical regions and developing countries, precisely the settings where this kind of tool would be most valuable, and precisely the settings this dataset does not include.

The study also does not have a true held out test set. With only 79 patients, the authors made the explicit and reasonable choice to evaluate on the validation set rather than carve out a separate test set that would have shrunk an already small dataset further, and they mitigated this with three way cross validation using non overlapping validation folds. That is a defensible way to work with a genuinely scarce clinical dataset, but a reader should understand it is a different, somewhat more optimistic evaluation setup than a model tested once on data it never touched during any part of development.

Honest limitations

The authors lay out their own limitations clearly rather than leaving readers to infer them. The patient count, 79 individuals, is small enough that they call out the need for external validation at other institutions before the model’s generalizability can be considered established. The study also only separates bacterial from fungal keratitis, leaving out viral keratitis, Acanthamoeba keratitis, and noninfectious immune mediated keratitis entirely, categories that a real emergency room encounter would need to rule in or out alongside the bacteria versus fungus question.

The ablation study itself, for computational reasons, was run using only broad-beam and slit-beam images for the multi model conditions rather than the full three way combination, and used only slit-beam images for the single model condition, which means the specific ablation numbers should be read as directionally informative rather than as a complete factorial test of every configuration. The authors also note that because their model classifies in a single end to end step rather than through a dedicated lesion detection stage, the attention heatmaps are not always sharp, and they suggest a separate lesion detection module as a concrete way to improve interpretability in future work.

Conclusion

The clearest lesson in this paper has nothing to do with keratitis specifically. It is that combining multiple imaging modalities is not free, and the mechanism used to combine them can matter more than which modalities you choose or how many you add. A naive pooling of image types actively degraded performance below what a single image type achieved alone, while giving separate models per modality a structured way to exchange information through self attention turned the same three photograph types into a materially stronger diagnostic signal.

The two counterintuitive ablation results, transfer learning helping single models but hurting fused ones, and lower resolution outperforming higher resolution specifically in fusion settings, are the kind of findings that only show up when a team runs the uncomfortable comparison rather than assuming the answer. Both point toward the same underlying idea, that a fusion architecture has different needs than a single modality architecture, and design choices imported wholesale from single image classification do not automatically transfer.

Clinically, the appeal of this approach is specific and real. Corneal scraping and culture takes time an actively progressing infection may not have, and the imaging used here, broad-beam, slit-beam and blue-light photography, is already standard equipment in an ophthalmology clinic rather than something exotic that would need new hardware to deploy. A tool that could triage bacterial against fungal keratitis from photographs already being taken, even as a second opinion rather than a replacement for culture, addresses a genuine bottleneck in regions where cultures are slow or unavailable.

The honest limitations here are not small print. A single center, 79 patient dataset with no external validation is an early stage result, not a deployment ready product, and the authors are explicit that generalizability across regions with different climates, different dominant organisms and different camera equipment remains unproven. What the paper does establish convincingly is the architectural principle, that structured fusion beats naive fusion, and that principle is likely to outlast this particular dataset even as the diagnostic numbers themselves get revisited with larger, multi center data.

Read against the broader push to bring AI assisted diagnosis to under resourced eye care settings, this study is best understood as a proof of concept for a specific fusion mechanism, backed by a real if modest clinical dataset, rather than a finished screening tool. The next test that actually matters is whether the same self attention fusion strategy holds up on photographs from a hospital, a population and a set of cameras this team has never seen.

A working PyTorch implementation

The block below is a runnable, simplified implementation of the three fusion strategies compared in the paper, simply add, vector add and self attention token fusion, built around a small Vision Transformer with a classification token, along with a training loop using cross entropy loss and an evaluation function for AUROC and AUPRC. It is written to make the mechanics concrete rather than to reproduce the paper’s exact 12 layer, 6 head, DINO pretrained architecture or its clinical dataset.

# vit_keratitis_fusion_reference.py
# A compact, runnable reference implementation of the three fusion strategies
# from Won, Kim, Jeon, Cha and Lim, Computers in Biology and Medicine 2025,
# simply add, vector add and self attention token fusion across image types.
# This is an educational reference, not a reproduction of the paper's exact ViT.

import torch
import torch.nn as nn
import torch.nn.functional as F


class PatchEmbedding(nn.Module):
    def __init__(self, image_size=128, patch_size=8, in_channels=3, embed_dim=384):
        super().__init__()
        self.num_patches = (image_size // patch_size) ** 2
        self.proj = nn.Conv2d(in_channels, embed_dim, kernel_size=patch_size, stride=patch_size)
        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.pos_embed = nn.Parameter(torch.zeros(1, self.num_patches + 1, embed_dim))
        nn.init.trunc_normal_(self.pos_embed, std=0.02)
        nn.init.trunc_normal_(self.cls_token, std=0.02)

    def forward(self, x):
        b = x.shape[0]
        tokens = self.proj(x).flatten(2).transpose(1, 2)
        cls = self.cls_token.expand(b, -1, -1)
        tokens = torch.cat([cls, tokens], dim=1)
        tokens = tokens + self.pos_embed
        return tokens


class TransformerEncoderLayer(nn.Module):
    def __init__(self, embed_dim=384, heads=6, mlp_ratio=4.0):
        super().__init__()
        self.norm1 = nn.LayerNorm(embed_dim)
        self.attn = nn.MultiheadAttention(embed_dim, heads, batch_first=True)
        self.norm2 = nn.LayerNorm(embed_dim)
        hidden = int(embed_dim * mlp_ratio)
        self.mlp = nn.Sequential(
            nn.Linear(embed_dim, hidden),
            nn.GELU(),
            nn.Linear(hidden, embed_dim),
        )

    def forward(self, x):
        normed = self.norm1(x)
        attn_out, _ = self.attn(normed, normed, normed)
        x = x + attn_out
        x = x + self.mlp(self.norm2(x))
        return x


class VisionTransformer(nn.Module):
    """A small ViT that returns its classification token as a summary vector."""

    def __init__(self, image_size=128, patch_size=8, in_channels=3, embed_dim=384, depth=6, heads=6):
        super().__init__()
        self.patch_embed = PatchEmbedding(image_size, patch_size, in_channels, embed_dim)
        self.layers = nn.ModuleList(
            [TransformerEncoderLayer(embed_dim, heads) for _ in range(depth)]
        )
        self.norm = nn.LayerNorm(embed_dim)

    def forward(self, x):
        tokens = self.patch_embed(x)
        for layer in self.layers:
            tokens = layer(tokens)
        tokens = self.norm(tokens)
        cls_token = tokens[:, 0]
        return cls_token, tokens


class SimplyAddClassifier(nn.Module):
    """Strategy one, pool image types into one dataset and use a single ViT."""

    def __init__(self, embed_dim=384, num_classes=2, **vit_kwargs):
        super().__init__()
        self.vit = VisionTransformer(embed_dim=embed_dim, **vit_kwargs)
        self.head = nn.Linear(embed_dim, num_classes)

    def forward(self, image):
        cls_token, _ = self.vit(image)
        return self.head(cls_token)


class VectorAddClassifier(nn.Module):
    """Strategy two, one ViT per image type, classification tokens summed."""

    def __init__(self, num_modalities, embed_dim=384, num_classes=2, **vit_kwargs):
        super().__init__()
        self.vits = nn.ModuleList(
            [VisionTransformer(embed_dim=embed_dim, **vit_kwargs) for _ in range(num_modalities)]
        )
        self.head = nn.Linear(embed_dim, num_classes)

    def forward(self, images_by_modality):
        # images_by_modality: list of tensors, one per image type, same batch size
        cls_tokens = [vit(img)[0] for vit, img in zip(self.vits, images_by_modality)]
        summed = torch.stack(cls_tokens, dim=0).sum(dim=0)
        return self.head(summed)


class SelfAttentionFusionClassifier(nn.Module):
    """
    Strategy three, one ViT per image type, classification tokens exchanged and
    refined with self attention across modalities before concatenation, following
    the fusion mechanism described in Section 2.3 of the paper.
    """

    def __init__(self, num_modalities, embed_dim=384, heads=6, num_classes=2, **vit_kwargs):
        super().__init__()
        self.vits = nn.ModuleList(
            [VisionTransformer(embed_dim=embed_dim, **vit_kwargs) for _ in range(num_modalities)]
        )
        self.cross_attn = nn.MultiheadAttention(embed_dim, heads, batch_first=True)
        self.head = nn.Linear(embed_dim * num_modalities, num_classes)

    def forward(self, images_by_modality):
        cls_tokens = []
        all_tokens = []
        for vit, img in zip(self.vits, images_by_modality):
            cls_token, tokens = vit(img)
            cls_tokens.append(cls_token)
            all_tokens.append(tokens)

        fused_cls_tokens = []
        for i, query_cls in enumerate(cls_tokens):
            # the query is this modality's CLS token, keys and values come from every
            # other modality's full token sequence, matching the paper's description
            # of CLS tokens being exchanged and attended against remaining tokens
            other_tokens = torch.cat(
                [all_tokens[j] for j in range(len(all_tokens)) if j != i], dim=1
            )
            query = query_cls.unsqueeze(1)
            attended, _ = self.cross_attn(query, other_tokens, other_tokens)
            fused_cls_tokens.append(attended.squeeze(1))

        concatenated = torch.cat(fused_cls_tokens, dim=1)
        return self.head(concatenated)


def classification_loss(logits, labels):
    # Matches the CrossEntropy loss described in Section 3.2 of the paper.
    return F.cross_entropy(logits, labels)


@torch.no_grad()
def compute_auroc_auprc(scores, labels):
    """A minimal, dependency free AUROC and AUPRC computed by sorting scores."""
    order = torch.argsort(scores, descending=True)
    sorted_labels = labels[order]

    positives = sorted_labels.sum().item()
    negatives = len(sorted_labels) - positives
    if positives == 0 or negatives == 0:
        return float("nan"), float("nan")

    tp, fp = 0.0, 0.0
    tpr_prev, fpr_prev = 0.0, 0.0
    auroc = 0.0
    precisions, recalls = [], []

    for label in sorted_labels.tolist():
        if label == 1:
            tp += 1
        else:
            fp += 1
        tpr = tp / positives
        fpr = fp / negatives
        auroc += (fpr - fpr_prev) * (tpr + tpr_prev) / 2
        tpr_prev, fpr_prev = tpr, fpr
        precisions.append(tp / (tp + fp))
        recalls.append(tpr)

    auprc = 0.0
    for i in range(1, len(recalls)):
        auprc += (recalls[i] - recalls[i - 1]) * precisions[i]

    return auroc, auprc


def train_one_step(model, optimizer, images_by_modality, labels, fusion="self_attention"):
    model.train()
    optimizer.zero_grad()
    if fusion == "simply_add":
        logits = model(images_by_modality[0])
    else:
        logits = model(images_by_modality)
    loss = classification_loss(logits, labels)
    loss.backward()
    optimizer.step()
    return loss.item()


def smoke_test():
    """Runs one training step and one evaluation step on random dummy data."""
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    torch.manual_seed(0)

    batch_size, channels, image_size = 6, 3, 128
    num_modalities = 2

    images_by_modality = [
        torch.rand(batch_size, channels, image_size, image_size, device=device)
        for _ in range(num_modalities)
    ]
    labels = torch.randint(0, 2, (batch_size,), device=device)

    model = SelfAttentionFusionClassifier(
        num_modalities=num_modalities, embed_dim=384, heads=6, num_classes=2,
        image_size=image_size, patch_size=8, depth=4,
    ).to(device)
    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.9)

    loss_value = train_one_step(model, optimizer, images_by_modality, labels)

    model.eval()
    with torch.no_grad():
        logits = model(images_by_modality)
        scores = F.softmax(logits, dim=1)[:, 1]
    auroc, auprc = compute_auroc_auprc(scores.cpu(), labels.cpu())

    print(f"Smoke test training loss {loss_value:.4f}")
    print(f"Smoke test AUROC on random batch {auroc}")
    print(f"Smoke test AUPRC on random batch {auprc}")
    assert torch.isfinite(torch.tensor(loss_value))
    print("Smoke test passed")


if __name__ == "__main__":
    smoke_test()

Frequently asked questions

What is the difference between bacterial and fungal keratitis

Both are corneal infections, but they are caused by different organisms and typically need different treatments. Fungal keratitis tends to have irregular, feathery borders and satellite lesions, is more common in tropical and developing regions, and is more often associated with poor visual outcomes and the need for surgery, though the paper notes many cases present atypically and are hard to distinguish from bacterial keratitis by appearance alone.

Can this Vision Transformer replace a corneal culture

No. It is a research classification model evaluated on a retrospective, single center dataset of 79 patients with no external validation. Corneal scraping and culture remains the reference standard, and any clinical use of an imaging based classifier would need prospective validation, regulatory clearance and oversight by qualified ophthalmologists.

Why did adding more image types sometimes make results worse

When broad-beam and slit-beam images were simply pooled into one Vision Transformer with no fusion mechanism, AUROC dropped to 0.67, below either single image type model alone. The model had no structured way to relate the two image types, and the paper’s self attention fusion strategy was built specifically to fix that by letting each image type’s classification token attend to information from the other image types.

What were the best performing image combinations

Combining broad-beam and slit-beam images with self attention fusion produced the study’s best AUROC of 0.93. Combining all three image types with self attention fusion produced the best AUPRC of 0.93, though its AUROC was slightly lower at 0.91.

Did transfer learning help

It depended on the fusion strategy. Pretraining with DINO helped the single image type and simply add models, but slightly hurt the vector add and self attention fusion models, likely because the pretraining was learned on single standalone images rather than on the shared features a fusion model needs to find across modalities.

How many patients were in the study

79 patients contributed 2,089 images total, 1,235 associated with bacterial keratitis and 854 with fungal keratitis, split 62 to 17 between training and validation with three way non overlapping cross validation.

Related reading