Video Segmentation Looked Solved Until MOSEv2 Cut SAM2’s Score in Half

Analysis by the aitrendblend editorial team. Twelve minute read. Source paper posted to arXiv, September 2025.

What DAVIS and YouTube VOS never had to test

Video object segmentation asks a model to track and outline a specific object through every frame of a clip, given some reference like a first frame mask. It sits underneath a surprising amount of practical software, from video editing tools that let you cut a subject out of a background to the perception stack in an autonomous vehicle that needs to keep following the same pedestrian across a crowded crosswalk. For most of the last decade, progress in this field has been measured against two benchmarks, DAVIS and YouTube VOS, and both datasets share a quiet bias. Their target objects tend to be large, centered, uninterrupted and alone. A dog running across a lawn. A person skateboarding down an empty street. The object rarely leaves the frame, rarely gets buried under a crowd, and rarely changes so much across a scene that a model has to reason about whether it is even looking at the same thing.

Under that kind of test, recent methods genuinely look close to solved. XMem reported 92.0 percent on DAVIS 2016, 87.7 percent on DAVIS 2017 and 86.1 percent on YouTube VOS. Numbers like that invite the conclusion that video object segmentation is a mostly finished research problem, ready to be handed off to product teams. The authors of this paper, led by Henghui Ding and Kaining Ying, had already challenged that conclusion once before, releasing the original MOSE dataset in 2023 specifically to include the messy conditions DAVIS and YouTube VOS leave out. That first version worked. XMem, the same model that hit 92 percent on DAVIS 2016, dropped to just 57.6 percent on MOSEv1, and the dataset went on to anchor several competitions, PVUW and LSVOS among them, pulling a wave of new methods including SAM2 based approaches that eventually pushed MOSEv1 performance up to 76.4 percent.

MOSEv2 is the sequel built to keep that pressure on. It keeps everything that made the first version hard, then adds a second wave of challenges that were still mostly absent from the field’s benchmarks even after MOSEv1, adverse weather, low light and underwater footage, multi shot sequences where the camera cuts between angles, camouflaged objects that blend into their background almost completely, non physical targets like shadows and reflections that have no fixed shape, and scenarios that require outside knowledge, reading text on identical looking blocks or reasoning about which object must be present based on how nearby objects are moving.

What actually changed in scale and difficulty

MOSEv2 combines the 2,149 videos inherited from MOSEv1 with 2,875 newly captured or copyright free videos, bringing the total to 5,024 videos, 701,976 mask annotations and 10,074 labeled object instances. The category list expanded from MOSEv1’s 36 categories to 200, adding both common objects that simply were not covered before, squirrels, footballs, otters, and genuinely unusual ones, Newton’s cradles, camouflaged creatures, and shadows treated as trackable targets in their own right.

Every one of those numbers points the same direction. Objects in MOSEv2 vanish and come back more than twice as often as in MOSEv1, sit closer to more visually similar distractors, get more heavily occluded by a combined measure the team built specifically because simpler metrics undersell real occlusion, and are more likely to be tiny relative to the frame. None of this happened by accident. The collection process followed ten explicit selection rules, favoring videos with multiple objects rather than one isolated subject, actively seeking out occlusion rather than avoiding it, prioritizing footage where objects vanish behind something and later return, and deliberately hunting for long videos, adverse weather, multi shot sequences and knowledge dependent scenarios rather than treating any of those as edge cases to filter out.

A better yardstick for tiny objects

Beyond collecting harder video, the authors identified a real flaw in how the field scores boundary accuracy and fixed it. The standard contour accuracy metric, written as F, uses a boundary matching threshold that scales with the diagonal of the image but not with the size of the object being segmented. That works fine when objects are reasonably large, but it actively lies about performance on small ones. The paper’s own example makes the problem vivid, a chopstick occupying just 955 pixels, 0.039 percent of the frame, scored an F of 0.91 under the standard metric even though the predicted and ground truth masks did not overlap at all, because the fixed threshold dilated the boundary far past the object’s actual size.

Here D is the image diagonal and A is the object’s area in pixels, with alpha set to 0.1 based on boundary statistics from DAVIS and MOSE. The new threshold shrinks automatically for small objects instead of staying fixed, so a boundary tolerance that made sense for a person filling a quarter of the frame no longer gets applied wholesale to a chopstick that fills a fraction of a percent. Under the corrected metric, that same chopstick scores 0, which is the honest answer, while a large object like a person occupying 21 percent of the frame scores identically under both the old and new metrics, confirming the fix does not distort scoring for objects that were already being measured fairly.

Splitting the score by disappearance and reappearance

The second metric contribution addresses a different blind spot. The standard region similarity and contour accuracy scores, J and F, are averaged across every frame of a video, which means a model that predicts an empty mask throughout a video with many disappearance frames can look artificially strong, while a model in a video with few disappearance frames barely gets penalized for mishandling the rare ones it does encounter. MOSEv2 isolates disappearance and reappearance into their own scored clips, producing two separate numbers, J&F for disappearance clips and J&F for reappearance clips, computed per clip and then averaged. A model that fails to suppress its mask when the object is genuinely gone gets caught by the disappearance score. A model that fails to find the object again once it returns gets caught by the reappearance score. Only a model that handles both well scores well on both, which closes a loophole the field’s existing metrics left wide open.

How far the leaderboard actually falls

The headline result of the paper is the size of the drop, and it holds up whether you look at the flagship model or the field average. SAM2 reached 90.7 percent J&F on DAVIS 2017 and 76.4 percent on MOSEv1, then fell to 50.9 percent on MOSEv2. Cutie, a strong non SAM2 method, went from 87.9 percent on DAVIS 2017 and 69.9 percent on MOSEv1 down to 43.9 percent here. The team benchmarked 20 representative video object segmentation methods across five distinct evaluation settings, mask initialized semi supervised segmentation, box initialized segmentation, point initialized segmentation, fully unsupervised segmentation, and interactive segmentation with iterative user corrections, and the pattern repeated in every setting.

Reappearance is where every method struggled most severely. Across the mask initialized semi supervised setting, reappearance scores ranged from just 7.8 percent up to 34.9 percent depending on the method, a spread the paper calls out directly as evidence of how hard re-identifying a target after it has been gone is, even for models that handle everything else in the same video reasonably well. Video object tracking methods, evaluated separately on 9 state of the art trackers using bounding boxes rather than masks, told the same story. SAMURAI, a strong SAM2 based tracker, scored 74.2 percent AUC on the established LaSOT benchmark and only 36.1 percent on MOSEv2.

Where the failures concentrate

The paper’s qualitative analysis picks out eight specific failure patterns that are worth understanding individually rather than folding into one big accuracy number. Models handling a car repeatedly vanishing under an overpass in simple, linear motion did reasonably well when using SAM2Long’s multiple path tracking, but the same approach failed once a person walked through a crowd along a more complex, nonlinear path before reappearing. Dense crowds with small, heavily occluded targets defeated every model tested with no exceptions. Camouflaged objects and shadows, MOSEv2’s genuinely new categories, actually favored Cutie over SAM2 and SAM2Long, likely because Cutie’s instance level memory explicitly models the foreground object rather than relying on the more general image features SAM2 uses. Heavy snow combined with occlusion caused total failure across every model. Multi shot sequences broke tracking continuity whenever a camera cut introduced a dramatic change in viewpoint. And in the paper’s most striking single example, a rotating Rubik’s cube, no model could correctly re-identify the right colored block after it disappeared behind the cube’s rotation, consistently misassigning the target to a neighboring block instead, a failure that has nothing to do with pixel level segmentation quality and everything to do with the kind of physical and spatial reasoning current architectures simply do not attempt.

Despite strong performance on existing datasets, current video object segmentation methods still fall short under real world complexities, and the consistent drops across every method and setting tested demonstrate that this gap is not specific to any single architecture. Paraphrased from the abstract of Ding, Ying, Liu, He, Jiang, Jiang, Torr and Bai, arXiv 2508.05630, 2025

Four fixes that partly close the gap without retraining from scratch

Rather than stopping at diagnosis, the team used their own failure analysis to build four targeted improvements to SAM2’s memory system, and reported exactly how much each one contributed. SAM2 keeps two kinds of memory while tracking an object, a small set of conditioned memories typically anchored to the initial frame, and a rolling window of unconditioned memories from the most recent frames. The conditioned memory, by default, only ever contains that single first frame, which becomes a liability the moment the initial view of the object was partial or the object’s appearance changes substantially before it disappears for the first time.

The first fix, called Reliable Conditioned Memory Selection, keeps SAM2’s naturally strong tracking during the period before an object first disappears, then strategically adds additional conditioned memories drawn from that pre disappearance window once a disappearance is detected, rather than relying on the single initial frame indefinitely. The second fix, Memory Quality Filtering, screens candidate memories using a quality score built from the model’s own predicted IoU, occlusion confidence and a separate mask quality signal, so only genuinely reliable frames get added to the conditioned memory bank. The third and fourth fixes, Mask Scaling Strategy and Long Video Finetuning, adjust the model’s output mask distribution with a fixed scaling factor and offset to reduce small object and occlusion errors, and extend the finetuning window from 8 to 16 frames while freezing the image encoder, specifically to help the model retain useful information across the kind of long duration sequences MOSEv2 is full of.

Applied to the larger SAM2-L backbone, the same combination of fixes raised overall performance from 49.7 percent to 54.4 percent, a gain of 4.7 percentage points, without any change to the model’s inference speed. Interestingly, the improvement helped the smaller B+ backbone’s reappearance score more than it helped the larger L backbone’s, which the authors attribute to the L backbone already balancing disappearance and reappearance performance reasonably well on its own, leaving less room for a memory only fix to move that particular needle further. None of these four fixes come close to closing the full gap between MOSEv2 and DAVIS level performance, but a combined 4.7 to 5.5 percentage point recovery from changes that do not require retraining the underlying model is a meaningful, practically usable result on its own.

Reproducing the core evaluation logic

The two metric contributions in this paper, the adaptive boundary threshold and the disappearance and reappearance scoring split, are straightforward enough to implement directly and useful well beyond this one dataset, for anyone evaluating a segmentation model on video with meaningful amounts of object disappearance. Below is a reference implementation covering both metrics, along with a simplified version of the Reliable Conditioned Memory Selection algorithm from the paper.

# mosev2_metrics.py
# Reference implementation of the MOSEv2 evaluation protocol
# Ding, Ying, Liu, He, Jiang, Jiang, Torr and Bai, arXiv 2508.05630, 2025
# Requires numpy only

import numpy as np


def region_similarity(pred_mask, gt_mask):
    # J score, standard intersection over union between two binary masks
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    if union == 0:
        return 1.0 if pred_mask.sum() == 0 else 0.0
    return intersection / union


def adaptive_boundary_threshold(object_area_px, image_diagonal_px, alpha=0.1):
    # Equation 1, scales the boundary tolerance to the object's own size
    # instead of only the frame diagonal, fixing the small object bias
    fixed_component = 0.008 * image_diagonal_px
    adaptive_component = alpha * np.sqrt(object_area_px)
    return min(fixed_component, adaptive_component)


def contour_accuracy_adaptive(pred_mask, gt_mask, image_diagonal_px, alpha=0.1):
    # A simplified adaptive F score, using the object's own area to
    # set the boundary matching tolerance rather than a fixed value
    gt_area = gt_mask.sum()
    if gt_area == 0:
        return 1.0 if pred_mask.sum() == 0 else 0.0

    threshold = adaptive_boundary_threshold(gt_area, image_diagonal_px, alpha)
    pred_boundary = extract_boundary(pred_mask)
    gt_boundary = extract_boundary(gt_mask)

    precision = boundary_match_ratio(pred_boundary, gt_boundary, threshold)
    recall = boundary_match_ratio(gt_boundary, pred_boundary, threshold)

    if precision + recall == 0:
        return 0.0
    return 2 * precision * recall / (precision + recall)


def extract_boundary(mask):
    # Placeholder for real boundary extraction, e.g. an eroded XOR mask
    # Left abstract here since implementation depends on the vision library used
    raise NotImplementedError("Plug in your preferred boundary extraction, e.g. OpenCV Canny")


def boundary_match_ratio(boundary_a, boundary_b, threshold):
    # Placeholder for bipartite boundary point matching within the threshold
    # Left abstract here, follows the same structure as the standard F score
    raise NotImplementedError("Plug in your preferred bipartite matching implementation")


def split_disappearance_reappearance_clips(gt_masks_per_frame):
    # gt_masks_per_frame is a list of boolean arrays, one per frame,
    # all False where the object is absent. Splits a video into
    # disappearance clips and reappearance clips per the paper's protocol
    disappearance_clips = []
    reappearance_clips = []

    present = [mask.sum() > 0 for mask in gt_masks_per_frame]
    seen_object_once = False
    i = 0
    while i < len(present):
        if present[i]:
            seen_object_once = True
            i += 1
            continue

        # start of an absence run
        start = i
        while i < len(present) and not present[i]:
            i += 1
        end = i

        if seen_object_once:
            disappearance_clips.append((start, end))
            # if the object comes back after this absence, that
            # return marks the start of a reappearance clip
            if i < len(present) and present[i]:
                reappearance_start = i
                reappearance_end = i + 1
                while reappearance_end < len(present) and present[reappearance_end]:
                    reappearance_end += 1
                reappearance_clips.append((reappearance_start, reappearance_end))

    return disappearance_clips, reappearance_clips


def select_nearest_memories(all_memory, k):
    # Simplified unconditioned memory selection, most recent k frames
    return all_memory[-k:] if len(all_memory) >= k else all_memory


def memory_quality_score(score_iou, score_occlusion, maskness):
    # Q = score_iou times score_occlusion times maskness, matching the paper
    return score_iou * score_occlusion * maskness


def reliable_conditioned_memory_selection(
    frame_predictions, quality_threshold=0.6, max_extra_memories=4
):
    # Simplified Algorithm 1 from the paper. frame_predictions is a list of
    # dicts with keys mask, memory, score_iou, score_occlusion, maskness,
    # in temporal order. Returns the selected conditioned memory list.
    conditioned_memory = [frame_predictions[0]["memory"]]
    disappeared = False

    for idx in range(1, len(frame_predictions)):
        frame = frame_predictions[idx]
        mask_is_empty = frame["mask"].sum() == 0

        if mask_is_empty and not disappeared:
            disappeared = True
            # walk backward through pre-disappearance frames, best candidates first
            for j in range(idx - 1, -1, -1):
                candidate = frame_predictions[j]
                q = memory_quality_score(
                    candidate["score_iou"], candidate["score_occlusion"], candidate["maskness"]
                )
                if q > quality_threshold and len(conditioned_memory) < max_extra_memories + 1:
                    conditioned_memory.append(candidate["memory"])

    return conditioned_memory


def smoke_test():
    # Runs the disappearance and reappearance clip splitting and the
    # memory selection logic on synthetic data to confirm correctness
    rng = np.random.default_rng(0)

    # synthetic 20 frame video: present, absent for 4 frames, present again
    frames_present = [True] * 5 + [False] * 4 + [True] * 6 + [False] * 2 + [True] * 3
    gt_masks = [
        np.ones((32, 32), dtype=bool) if present else np.zeros((32, 32), dtype=bool)
        for present in frames_present
    ]

    disappearance_clips, reappearance_clips = split_disappearance_reappearance_clips(gt_masks)
    print("Smoke test disappearance clips", disappearance_clips)
    print("Smoke test reappearance clips", reappearance_clips)

    dummy_predictions = []
    for i, present in enumerate(frames_present):
        dummy_predictions.append({
            "mask": gt_masks[i],
            "memory": f"memory_frame_{i}",
            "score_iou": float(rng.uniform(0.6, 1.0)),
            "score_occlusion": float(rng.uniform(0.6, 1.0)),
            "maskness": float(rng.uniform(0.6, 1.0)),
        })

    selected_memory = reliable_conditioned_memory_selection(dummy_predictions)
    print("Smoke test selected conditioned memories", selected_memory)

    threshold = adaptive_boundary_threshold(object_area_px=955, image_diagonal_px=2308)
    print("Smoke test adaptive boundary threshold for a 955 pixel object", round(threshold, 3))
    print("Smoke test completed without errors")


if __name__ == "__main__":
    smoke_test()

Two implementation details are worth flagging. The boundary extraction and bipartite matching functions are left as clearly marked stubs rather than filled in with a specific library, since the paper follows the established DAVIS style contour matching procedure and any correct implementation of that procedure, using OpenCV or a similar tool, will plug directly into the adaptive threshold logic shown here. The memory selection function follows the paper’s Algorithm 1 closely, walking backward through pre disappearance frames and only admitting a candidate memory once its combined quality score clears the threshold, which is the mechanism responsible for most of the reappearance score improvement reported in the ablation table.

What this means for anyone building on video segmentation

The most useful thing about a dataset paper like this one is not the raw difficulty of its videos, it is the specificity of what breaks and why. A team building a video editing tool that needs to keep a subject isolated across a scene cut now has a documented failure mode, multi shot transitions, to test against directly rather than discovering it in production. A team working on retail or wildlife monitoring, where objects genuinely leave and re-enter a camera’s view constantly, now has a metric, the reappearance score, that will not silently reward a model for guessing an object is gone and staying quiet. And a team already shipping a SAM2 based pipeline has a concrete, quantified set of memory management changes to test, not a vague suggestion to improve robustness.

The honest caveat is that even the paper’s own fixes leave a wide gap unclosed. Raising SAM2-L from 49.7 to 54.4 percent is real progress, but it is still nowhere near the 90 percent range these models post on DAVIS, and the paper is explicit that several classes of failure, knowledge dependent reasoning about physics or text, generalization to genuinely rare object categories, and robust handling of adverse weather, remain open problems the authors frame as future work rather than anything close to solved. That framing is arguably the paper’s most valuable contribution. It replaces a comfortable, saturated leaderboard with a harder, more honest one, and gives the field a specific map of where the real remaining difficulty lives.

Frequently asked questions

What is MOSEv2 and how is it different from MOSEv1

MOSEv2 is a video object segmentation dataset with 5,024 videos, 701,976 masks and 200 object categories, expanding on the original 2023 MOSE dataset with more severe versions of its existing challenges plus entirely new ones like adverse weather, camouflage, multi shot sequences and knowledge dependent scenarios.

Why did SAM2’s score drop so much on MOSEv2

SAM2 was evaluated on video that includes frequent object disappearance and reappearance, heavy occlusion, small targets in crowded scenes and challenging environmental conditions, none of which are well represented in the DAVIS and YouTube VOS benchmarks it was originally scored against, which explains most of the gap between its 90.7 percent DAVIS score and its 50.9 percent MOSEv2 score.

What is the hardest specific scenario for current video segmentation methods

Reappearance after disappearance is consistently the hardest case across every method and setting tested, with scores as low as 7.8 percent in some configurations, since re-identifying a target after it has been fully occluded or out of frame requires stronger memory and reasoning than most current architectures provide.

Did the paper propose any fixes, or just document the problem

Both. The authors propose four targeted improvements to SAM2’s memory system, Reliable Conditioned Memory Selection, Memory Quality Filtering, Mask Scaling Strategy and Long Video Finetuning, which together recover 4.7 to 5.5 percentage points of performance without retraining the full model.

Is MOSEv2 publicly available

Yes, the dataset is publicly available through the project’s official site.

Does this paper cover video object tracking as well as segmentation

Yes, beyond segmentation the team benchmarked 9 state of the art video object tracking methods on MOSEv2 and observed similarly large performance drops, showing the dataset’s challenges extend across both tasks.

Related reading