Why GPT-4 Rewritten Prompts Only Sometimes Improve HUMAN Motion Simulation

The Problem With Evaluating Simulated Motion Against Itself

Human motion simulation supports a genuinely practical set of jobs, ergonomic risk assessment, safety training, and digital human modeling for manufacturing lines where getting a worker’s posture wrong on paper is a lot cheaper than getting it wrong on the floor. Traditional methods, physics based modeling, motion capture, and video tracking, do this well but at a real cost. They need calibrated hardware, trained operators, and enough time per task that testing eight different work scenarios is a serious undertaking.

Generative AI offers a faster alternative. Text to motion models built on GANs, Variational Autoencoders, or diffusion architectures can turn a sentence into a full body animation. MotionGPT is a widely used example, a system that pairs a Vector Quantized Variational Autoencoder motion tokenizer with a language model built on T5, trained on datasets such as HumanML3D so that a phrase like “a person walks and sits on a chair” becomes a smooth, temporally consistent motion sequence. That is a genuinely useful capability, and it is also where the paper’s two central concerns start.

The first concern is about how these systems get evaluated. Most reported fidelity numbers for AI generated motion are embedding based, meaning they measure how internally consistent a model’s outputs are with each other, not how closely those outputs resemble an actual human body moving through space. For a system meant to support safety decisions, that is a meaningful gap. A model can score well against its own embedding space while producing a motion no real worker would ever perform.

The second concern is upstream of the model entirely. MotionGPT, like most text to motion systems, was trained on a specific style of instruction drawn from its training corpus. Feed it a task description phrased differently, more casually, more abstractly, or with vocabulary the training data rarely used, and the mismatch between what you said and what the model learned to expect can quietly degrade the motion it produces. Two people describing the same task in their own words could get two different quality levels of simulation back, for reasons that have nothing to do with the task itself.

How G-AI-HMS Works

G-AI-HMS runs as a two stage pipeline. In the first stage, GPT-4 rewrites a natural language task description into a structured prompt that matches MotionGPT’s training vocabulary. In the second stage, MotionGPT turns that rewritten prompt into a motion sequence, exactly as it would for any other input. What makes the system worth studying is not the second stage, which is just running MotionGPT normally, but the first stage and the evaluation framework built to check whether it actually helped. You can read the full method and every task result in the original paper, published in Knowledge-Based Systems.

A Structured Rewriting Step, Not A Paraphrase

The authors are explicit that this translation stage is meant to do more than smooth out a sentence’s wording. It is a controlled semantic adaptation. MotionGPT implicitly expects text that resembles its training corpus, HumanML3D, in structure and vocabulary. An unconstrained, casually phrased instruction can introduce exactly the kind of out of distribution phrasing the model was never trained to interpret well.

GPT-4 is used to close that gap by acting as what the authors call a semantic normalizer. It restructures vague or goal oriented instructions into concrete, physically grounded action sequences built from frequent action keywords pulled directly from the HumanML3D dataset. The paper gives a clear example of what this looks like in practice. An abstract instruction such as “prepare for exercise” gets decomposed into something MotionGPT can actually execute, stand upright, stretch both arms upward, bend slightly forward, then return to a neutral standing pose. The system prompt driving this rewrite is fixed and deterministic under controlled settings, so the same instruction produces the same rewritten prompt every time, which matters if you want the pipeline to be something other than a black box creative tool.

Motion Synthesis With MotionGPT

Once the prompt is rewritten, MotionGPT does the actual generation, unmodified and without any fine tuning on the authors’ own data. A motion sequence of frames is passed through an encoder that compresses it into a shorter sequence of continuous latent vectors.

Each continuous latent vector is then snapped to its nearest entry in a learned codebook of 512 embeddings, the vector quantization step that gives the model its discrete, token based vocabulary of motion.

A decoder then reconstructs a motion sequence from that quantized token sequence, and the whole tokenizer is trained to minimize a combination of reconstruction, embedding, and commitment losses.

For an actual prompt, the language model side of MotionGPT predicts a sequence of these discrete motion tokens conditioned on the input text, and the decoder turns that token sequence into the final output video. Every part of this stage is the published MotionGPT system running exactly as released. The only thing G-AI-HMS changes is what text goes in.

Turning Video Into Ground Truth With MediaPipe

The evaluation side of the pipeline needed a way to compare AI generated motion against something real, so the authors filmed a human subject performing each task from three camera angles and extracted joint positions using MediaPipe, Google’s open pose estimation framework. MediaPipe returns 33 landmarks per frame, which get reduced to the 22 joints that match MotionGPT’s own HumanML3D based skeleton, using a fixed mapping between MediaPipe landmarks and HumanML3D joint indices.

Before any comparison could happen, every motion source, the real human video, plain MotionGPT’s output, and G-AI-HMS’s output, had to be put through the same normalization. Each joint position is first centered relative to the head joint at every frame, removing any difference in where the subject happened to stand in the frame.

Then a scale factor derived from the distance between the head and left foot removes differences in body size or camera distance, so a taller subject and a shorter one, or a closer camera and a farther one, land on the same scale.

Because MediaPipe measures from a top left origin while the motion generation side uses a bottom left convention, the vertical axis gets flipped for consistency, and a two stage filter, a median filter followed by a zero phase Butterworth low pass filter, removes sensor jitter and generative noise without introducing any lag. Finally, since the three sources naturally produce different numbers of frames, every sequence is resampled by linear interpolation down to the shortest common frame count before any metric gets computed.

Three Ways To Measure Fidelity

With every sequence on the same spatial scale and the same timeline, three metrics do the actual comparison. Mean Per Joint Position Error averages the raw Euclidean distance between generated and reference joint positions across every joint and every frame.

Procrustes Aligned MPJPE asks a slightly different question. It first finds the best possible rotation to align the generated pose with the reference pose, using singular value decomposition on the cross covariance matrix between the two, then measures position error after that optimal alignment.

That distinction matters more than it might first appear. Ordinary MPJPE penalizes a motion for being in the wrong place. Procrustes aligned MPJPE forgives a rigid rotation or offset and asks whether the pose’s actual shape, its posture, is right. A motion sequence can score badly on one and reasonably on the other, and the difference tells you whether the problem is positioning or genuine posture inaccuracy. Dynamic Time Warping rounds out the set by measuring temporal alignment cost, how well the timing and pacing of one joint’s trajectory matches the reference, independent of exact frame by frame correspondence.

The Experiment

Eight tasks were chosen to span a range of biomechanical demands, from simple locomotion to fine manual manipulation. A single human subject performed each one under IRB approval, recorded from three angles for full body coverage.

For every task, the same spoken instruction produced three videos, the real human performing the task, MotionGPT’s output when fed the plain human phrased description, and G-AI-HMS’s output when fed the GPT-4 rewritten version of that same description. All three then went through the identical normalization and metric pipeline described above.

What The Results Show

Averaged across all eight tasks, G-AI-HMS produced lower MPJPE in six tasks, lower PA-MPJPE in four tasks, and lower DTW in seven tasks compared to plain MotionGPT. A paired statistical comparison across every joint and task found a significant improvement in two of the three metrics.

Read together, this is a specific and useful result rather than a blanket win. GPT-4 rewritten prompts produced motion that landed closer to the real joint positions and stayed better aligned in time. They did not produce meaningfully better posture once rotation and translation were accounted for. Whatever the rewriting step improves, it is spatial placement and pacing, not the underlying shape of the pose itself.

That reversal is worth taking seriously rather than treating as noise. In Task 2, throwing a large object, MotionGPT with the plain human prompt beat G-AI-HMS in 14 of 22 joints, most sharply at the right wrist and right hand. The authors’ own explanation is that natural, experience grounded human phrasing can implicitly encode biomechanical detail, the exact way a hand releases an object, the compensatory adjustment of a wrist during a throw, that a structured, vocabulary constrained rewrite ends up stripping out. GPT-4’s job in the pipeline is to make phrasing predictable and aligned with training vocabulary, and that same predictability can flatten exactly the idiosyncratic, hard to verbalize detail that makes a throw or a seated seated turn look genuinely human rather than generically plausible.

Rewriting a prompt for vocabulary alignment makes a motion model’s job easier, and that same simplification can quietly erase the biomechanical nuance a human describer already knew to include without saying it directly. Reading of the task level findings in G-AI-HMS, Arizona State University

One more detail deserves a direct explanation, because it is easy to misread from the results tables alone. The head joint shows an MPJPE of almost exactly zero for every method, in every task, which on first glance looks like remarkable accuracy. It is not. Recall that every motion sequence gets root-centered relative to the head joint before any comparison happens, which means the head’s position is defined as the coordinate origin for all three sources by construction. Its MPJPE cannot be anything other than close to zero, regardless of how good or bad the underlying simulation is. The real signal for that joint’s fidelity shows up in PA-MPJPE and DTW instead, both of which still show meaningful error for the head and neck region, because those metrics capture rotation and timing, which the normalization step does not remove.

Where This Falls Short

The paper is candid about several limitations that matter for anyone considering this approach for a real deployment. The entire human reference dataset comes from a single subject performing each of the eight tasks once. That is enough to demonstrate the pipeline and run a meaningful statistical comparison across many joints and frames, but it is not enough to say how G-AI-HMS would perform against the natural variation across different bodies, movement styles, or skill levels. MediaPipe itself, while validated elsewhere with strong reliability for several joint movements, can still struggle under occlusion, rapid movement, or difficult depth estimation, meaning the ground truth it produces is not perfectly clean either. The evaluation also assumes a strict one to one mapping between a single static prompt and a single reference motion, with no mechanism for combining multiple prompts, adjusting a description mid motion, or incorporating feedback. And the prompt vocabulary itself does not yet include modifiers like speed, caution, or specific body constraints, the kind of detail that might matter most for exactly the tool use and hand dominant tasks where the current approach underperformed.

A Working Python Implementation

The GPT-4 rewriting step and MotionGPT synthesis are not something a local script can reproduce, since both depend on proprietary or heavyweight pretrained systems. What is fully reproducible, and arguably the more broadly useful piece for anyone building a similar evaluation, is the fidelity scoring pipeline itself. The code below implements spatial normalization, temporal resampling, and all three metrics, MPJPE, PA-MPJPE with Procrustes alignment, and DTW, and runs them on a synthetic pair of joint sequences to confirm the math lines up correctly.

# motion_fidelity.py
# Reference style implementation of the G-AI-HMS evaluation pipeline
# Iyer, Macwan, Hude, Jeong, and Guo, Knowledge-Based Systems 349 (2026) 116451

import numpy as np
from scipy.signal import medfilt, butter, filtfilt
from scipy.interpolate import interp1d


def root_center_and_scale(motion, head_idx=15, foot_idx=11):
    # motion: (T, J, 3) array. Center on the head joint, scale by
    # the head-to-foot distance, following Eqs. (5)-(7) in the paper.
    head = motion[:, head_idx:head_idx + 1, :]
    centered = motion - head
    scale = np.linalg.norm(motion[:, foot_idx, :] - motion[:, head_idx, :], axis=-1)
    scale = np.clip(scale, 1e-6, None)
    return centered / scale[:, None, None]


def filter_trajectory(signal, median_kernel=11, cutoff=0.05, order=4):
    # Median filter for impulsive spikes, then a zero-phase
    # Butterworth low-pass filter for residual jitter (Eq. 9).
    smoothed = medfilt(signal, kernel_size=median_kernel)
    b, a = butter(order, cutoff, btype="low")
    return filtfilt(b, a, smoothed)


def resample_to_length(motion, target_len):
    # Linear interpolation to a common frame count (Eqs. 10-12).
    T = motion.shape[0]
    old_t = np.linspace(0, 1, T)
    new_t = np.linspace(0, 1, target_len)
    resampled = np.zeros((target_len,) + motion.shape[1:])
    for j in range(motion.shape[1]):
        for d in range(motion.shape[2]):
            f = interp1d(old_t, motion[:, j, d], kind="linear")
            resampled[:, j, d] = f(new_t)
    return resampled


def mpjpe(pred, gt):
    # Eq. (13): mean Euclidean distance across all joints and frames.
    return np.mean(np.linalg.norm(pred - gt, axis=-1))


def pa_mpjpe(pred, gt):
    # Eqs. (14)-(16): Procrustes alignment via SVD, then MPJPE
    # on the rotated prediction.
    T, J, _ = pred.shape
    pred_flat = pred.reshape(T * J, 3)
    gt_flat = gt.reshape(T * J, 3)

    pred_c = pred_flat - pred_flat.mean(axis=0)
    gt_c = gt_flat - gt_flat.mean(axis=0)

    H = pred_c.T @ gt_c
    U, _, Vt = np.linalg.svd(H)
    R = U @ Vt

    aligned = (pred_c @ R).reshape(T, J, 3)
    gt_reshaped = gt_c.reshape(T, J, 3)
    return np.mean(np.linalg.norm(aligned - gt_reshaped, axis=-1))


def dtw_distance(series_a, series_b):
    # Simple O(n*m) dynamic time warping on a single joint's
    # trajectory, following the classic DP formulation.
    n, m = len(series_a), len(series_b)
    cost = np.full((n + 1, m + 1), np.inf)
    cost[0, 0] = 0
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            d = np.linalg.norm(series_a[i - 1] - series_b[j - 1])
            cost[i, j] = d + min(cost[i - 1, j], cost[i, j - 1], cost[i - 1, j - 1])
    return cost[n, m]


def dtw_score(pred, gt):
    # Eq. (18): mean per-joint DTW cost across all joints.
    n_joints = pred.shape[1]
    scores = [dtw_distance(pred[:, j, :], gt[:, j, :]) for j in range(n_joints)]
    return float(np.mean(scores))


def evaluate_fidelity(sim_motion, gt_motion):
    # Full pipeline: normalize both sequences, resample to a
    # common length, then compute all three metrics.
    sim_norm = root_center_and_scale(sim_motion)
    gt_norm = root_center_and_scale(gt_motion)

    target_len = min(sim_norm.shape[0], gt_norm.shape[0])
    sim_aligned = resample_to_length(sim_norm, target_len)
    gt_aligned = resample_to_length(gt_norm, target_len)

    return {
        "MPJPE": mpjpe(sim_aligned, gt_aligned),
        "PA_MPJPE": pa_mpjpe(sim_aligned, gt_aligned),
        "DTW": dtw_score(sim_aligned, gt_aligned),
    }


if __name__ == "__main__":
    np.random.seed(0)

    # Build two synthetic motion sequences: a "ground truth" walking
    # cycle and a "simulated" version with mild noise, a slight
    # phase shift, and a small scale mismatch, to exercise the
    # full normalization and scoring pipeline end to end.
    n_frames, n_joints = 60, 22
    t = np.linspace(0, 4 * np.pi, n_frames)

    gt_motion = np.zeros((n_frames, n_joints, 3))
    for j in range(n_joints):
        gt_motion[:, j, 0] = np.sin(t + j * 0.1) * (1 + j * 0.02)
        gt_motion[:, j, 1] = j * 0.1 + np.cos(t + j * 0.1) * 0.2
        gt_motion[:, j, 2] = np.sin(t * 0.5 + j * 0.1) * 0.1

    phase_shift = 0.15
    sim_motion = np.zeros_like(gt_motion)
    for j in range(n_joints):
        sim_motion[:, j, 0] = np.sin(t + j * 0.1 + phase_shift) * (1 + j * 0.02) * 1.05
        sim_motion[:, j, 1] = j * 0.1 + np.cos(t + j * 0.1 + phase_shift) * 0.2
        sim_motion[:, j, 2] = np.sin(t * 0.5 + j * 0.1) * 0.1
    sim_motion += np.random.normal(0, 0.01, sim_motion.shape)

    print("Running motion fidelity smoke test on synthetic sequences")
    results = evaluate_fidelity(sim_motion, gt_motion)
    for name, value in results.items():
        print(f"{name}: {value:.4f}")

    assert results["MPJPE"] < 1.0, "MPJPE unexpectedly large for a mild synthetic offset"
    print("Smoke test passed. All three metrics computed without errors.")

Where G-AI-HMS Ends Up

G-AI-HMS demonstrates something narrower and more useful than a general claim that language models improve motion generation. It shows that aligning a task description’s vocabulary with a text to motion model’s training distribution measurably improves spatial and temporal fidelity, verified against real human video rather than against another model’s output, for the kinds of tasks where the improvement should intuitively show up, cyclic, repetitive, low complexity movement. Walking, painting a flat surface, carrying a load, these are exactly the motions where a clean, structured prompt gives a model less room to go wrong.

The more interesting result is where that improvement stops. Tool use, hand dominant actions, and multi phase transitions like sitting down are where G-AI-HMS’s rewriting step lost ground to plain human phrasing, and the paper’s own explanation, that natural language from someone who has actually thrown an object or turned in a swivel chair carries implicit biomechanical detail that a vocabulary aligned rewrite can flatten out, is a genuinely useful finding for anyone building similar pipelines. Standardizing a prompt buys you predictability. It does not automatically buy you the specific, hard to verbalize detail a domain expert’s own words already contained.

That distinction generalizes well past motion simulation. Any pipeline that inserts a language model as a normalization layer in front of a specialized downstream system, a code generator, a robotics planner, a diagram tool, is making the same trade. Standardized input reduces the downstream model’s chance of misinterpreting an unfamiliar phrasing, and it can simultaneously discard exactly the situational nuance that made the original phrasing useful in the first place. Knowing which of those two effects dominates for a given task category, rather than assuming the rewriting step is a universal improvement, is the practical lesson worth carrying forward.

The limitations are real and worth restating plainly. This is a single subject study across eight tasks, MediaPipe’s own pose estimation has known weak points under occlusion and rapid movement, and the evaluation framework currently handles one static prompt mapped to one motion with no room for mid sequence adjustment or multi prompt combination. None of that undermines the core finding, but it does mean the numbers in this paper describe a controlled demonstration rather than a validated production benchmark.

Strip away the acronym, and G-AI-HMS is making a fairly grounded argument. A language model can make a downstream generative system easier to use correctly, but easier to use correctly is not the same as universally better, and the honest way to find out which tasks benefit is to check the result against a real human doing the same thing, not against another model’s idea of what that motion should look like.

Frequently Asked Questions

Related Reading On aitrendblend.com