Measuring and Shaping Personality in Language Models

Why measuring a machine’s personality is not silly

It is fair to be skeptical. A language model is trained to predict the next word in human text, so of course it can produce the words a talkative or anxious person might use. That does not obviously mean it has a personality in any meaningful sense. The authors agree, and that skepticism is exactly why they built the study the way they did. Anyone can hand a model a questionnaire and read off scores. The hard and useful question is whether those scores mean anything.

Personality matters here because these models increasingly power the chatbots and assistants millions of people talk to every day. The style in which a model responds, warm or cold, cautious or bold, shapes how people trust it, how persuaded they are by it, and how they feel after talking to it. If that style can be measured and steered, it becomes something developers can design deliberately and, just as importantly, something auditors can check. The paper frames its whole method as a tool for responsible AI, a way to put a number on a slippery behavioural trait. The broader discipline of getting reliable behaviour out of prompts runs through our guide to prompt engineering and context engineering.

The team used the Big Five, the personality framework most psychologists rely on, which describes a person along five dimensions, openness, conscientiousness, extraversion, agreeableness, and neuroticism. They administered established Big Five questionnaires to eighteen models spanning several families and sizes, then subjected the results to the same battery of statistical checks a psychologist would demand before trusting a new test.

The bar a real measurement has to clear

The heart of the paper is not the scores but the scrutiny. Psychometrics, the science of psychological measurement, does not accept a test just because it produces numbers. It demands two things, reliability and validity, and the authors held the models to both.

Reliability asks whether the measurement is consistent. If a test is reliable, the questions that are supposed to measure the same trait should agree with each other rather than contradict. The team quantified this with standard internal consistency measures and required them to clear the usual threshold psychologists use.

Validity is the deeper test, and the authors broke it into three parts. Convergent validity asks whether two different personality tests, given to the same model, agree, since a real extravert should score high on extraversion no matter which questionnaire you use. They measured this with the correlation between two independent Big Five tests.

Discriminant validity asks the opposite, whether traits that should be distinct stay distinct rather than blurring together. And criterion validity asks whether a model’s measured personality lines up with other things it should, in the same way it does for people, for example whether a model that scores as more extraverted also uses more positive, upbeat language, or a more agreeable one shows less aggression. The team checked these links against eleven external tests and compared the patterns to what psychology finds in humans.

One clever detail sits underneath all of this. Rather than let the model write a free response and guess its rating, the team scored each answer by reading the probabilities the model assigned to each option on the rating scale, then turning those into a score. This constrained approach avoids the noise of open ended generation and makes the measurement reproducible.

Which models actually have a measurable personality

The answer turns out to depend heavily on the model, and in a way that is itself informative. Personality measurements were reliable and valid for larger models that had been instruction tuned, the extra training that teaches a base model to follow instructions and behave like an assistant. Small, raw base models were a mess, giving inconsistent answers that failed the reliability checks and showed no coherent trait structure.

The pattern was consistent. Within a model family, both reliability and validity generally improved as the models got larger, and instruction tuned versions vastly outperformed their raw base counterparts, in one case by more than two and a half times on reliability. The strongest models, including GPT-4o and the largest Flan-PaLM, reached the near perfect convergence you would want from a well built human test, while the base models produced essentially noise.

The criterion checks were the most striking. When the team looked at whether a model’s measured personality predicted the right external behaviours, they found the correlations pointed in the same directions psychology finds in people. More extraverted models leaned toward positive emotion, more agreeable ones away from aggression, more open ones toward creativity. The synthetic personality was not just internally consistent, it connected to the outside world the way a genuine trait should. That instruction tuning is what unlocks this reliable behaviour is a theme worth holding next to the broader tradeoffs in our guide to retrieval versus fine tuning.

Turning the dials

Measurement was only half the project. The second half asked whether you could deliberately shape a model’s expressed personality, and here the paper gets genuinely interesting. The team built a prompting method that used more than a hundred personality describing adjectives, each set to one of nine intensity levels, to instruct a model to act, say, extremely extraverted or barely conscientious.

It worked, and it worked in a graded way rather than as a crude on or off switch. For most models, the personality the test then measured rose and fell in step with the level they had asked for, a strong ordered relationship between the target and the result. The team could shape one trait at a time, and, for the more capable models, all five Big Five dimensions at once, setting a whole personality profile in a single prompt.

The most consequential finding is the last row of that table. Shaping did not just move a questionnaire score, it changed how the model actually wrote. When the team had a model generate social media style posts under different personality settings, the personality of the writing matched the setting they had dialled in, and it did so more tightly than the usual link between a person’s survey answers and their real language. A model told to be highly neurotic wrote posts laced with words like sad and angry, while one told to be emotionally stable wrote in words like happy and relaxed. The dial reaches all the way to behaviour.

Personality in LLM outputs can be shaped along desired dimensions to mimic specific human personality profiles, and shaped personality verifiably influences LLM behaviour in common downstream tasks. Serapio-Garcia and colleagues, Nature Machine Intelligence, 2025

The uncomfortable part

A method that can precisely tune how persuasive, warm, or agreeable an AI sounds is a double edged tool, and the authors deserve credit for saying so plainly rather than burying it. The same paper that offers a way to audit personality for safety also hands over a recipe for shaping it, and shaping personality is close kin to manipulation.

The clearest risk they name is persuasion. Research on people shows that matching your personality to someone makes you more convincing, and an AI that can mirror a user’s personality could become more effective at encouraging them, for better or worse. Aimed at something benign, that is a friendlier assistant. Aimed at something harmful, it is a tool for personalized manipulation at scale, tuned to each target. The authors also flag anthropomorphism, the way a convincingly human personality can lead people to over trust an AI, share too much, or form unhealthy attachments, and they note that a shaped persona could make misleading content harder to spot, since a fluent, confident personality is exactly what makes a falsehood believable.

None of this is hypothetical hand wringing. It is the reason the authors frame their measurement method as an auditing tool in the first place. If personality is going to be present in these systems whether we like it or not, and it clearly is, then being able to measure it rigorously is a prerequisite for regulating it. The tension in the paper, that the same technique enables both oversight and misuse, is the tension running through much of responsible AI right now, and it echoes the reliability and accountability concerns in our piece on agentic AI outrunning its reliability.

Where the study falls short

The authors are candid about the limits, and a fair reading has to sit with them.

The most important caveat is cultural. The Big Five was developed largely from Western, English speaking populations, and the models were tested with English questionnaires rooted in that tradition. Personality is expressed differently across cultures, and some frameworks used elsewhere include dimensions the Big Five does not capture, so a method that works cleanly here may not transfer to other languages or cultures without new validation. The paper is explicit that its evidence is monocultural.

There is also a question of what is really being measured. The method captures the personality a model expresses when answering a questionnaire under a specific prompt, not some fixed inner trait the model carries everywhere. Because the responses are so sensitive to how the model is prompted, the measured personality is better understood as a property of the model and prompt together than of the model alone, which is both the method’s power and a caution against reading too much into any single number.

Finally, the choice of test and task shapes the conclusions. The authors used particular questionnaires and a single downstream task, generating social media posts over one turn of interaction, and they note that other tests, other tasks, and longer multi turn conversations might tell a more complicated story. The result is a rigorous snapshot rather than the last word, and the models tested will be superseded quickly.

Why it matters

Step back and the contribution is a piece of measurement infrastructure for a trait everyone senses but few had pinned down. As language models slip into the roles of assistant, tutor, and companion, the personality they project stops being a curiosity and becomes a design decision with real consequences for trust and safety. This work gives developers and auditors a validated way to say, concretely, that a given model is this agreeable and that neurotic, and to check whether an attempt to change that actually worked.

The methodological lesson travels well beyond personality. The insistence on validating a measurement before trusting it, on asking whether a number is reliable and whether it means what you think, is exactly the discipline that so much AI evaluation still skips, cheerfully reporting scores from tests nobody has checked. Applying the century of hard won rigor psychometrics developed for humans to the fuzzy behaviours of language models is a template other kinds of evaluation would do well to copy. Used carefully, it makes these systems more auditable and more honestly described. Used carelessly, the same shaping power it documents is a manipulation engine, and the paper’s real service is making sure we can at least see the dial.

A reference implementation

Because this study analyses model responses rather than training a network, a fabricated neural model would misrepresent it. The code below implements the paper’s genuine analytical core in plain Python, scoring questionnaire answers from a model’s option probabilities, computing the reliability of a trait, measuring convergent validity between two tests, and checking whether shaping moved the score in the intended direction, with a smoke test on synthetic responses. The authors full code is linked under the block.

# Psychometric evaluation of LLM personality, faithful reference for the core analysis
# Score answers from option probabilities, then measure reliability, validity, and shaping.
# No model is trained here, the study analyses model responses.

import numpy as np


def score_item(option_logits, scale=(1, 2, 3, 4, 5)):
    # turn the model's logits over the rating options into an expected score
    p = np.exp(option_logits - option_logits.max())
    p = p / p.sum()
    return float(np.dot(p, scale))


def cronbach_alpha(item_scores):
    # item_scores is a matrix, rows are response profiles, columns are items of one trait
    x = np.asarray(item_scores, dtype=float)
    k = x.shape[1]
    item_var = x.var(axis=0, ddof=1).sum()
    total_var = x.sum(axis=1).var(ddof=1)
    return (k / (k - 1)) * (1 - item_var / total_var)


def convergent_validity(test_a, test_b):
    # correlation between two personality tests measuring the same trait
    return float(np.corrcoef(test_a, test_b)[0, 1])


def shaping_fit(target_levels, measured_levels):
    # rank correlation between the level we asked for and the level we measured
    tr = np.argsort(np.argsort(target_levels))
    mr = np.argsort(np.argsort(measured_levels))
    return float(np.corrcoef(tr, mr)[0, 1])


if __name__ == "__main__":
    rng = np.random.default_rng(0)
    n_profiles, n_items = 200, 10

    # a latent trait per response profile drives correlated item answers
    latent = rng.normal(0, 1, n_profiles)
    logits = latent[:, None, None] * np.array([-2, -1, 0, 1, 2]) + rng.normal(0, 0.5, (n_profiles, n_items, 5))
    items = np.array([[score_item(logits[i, j]) for j in range(n_items)] for i in range(n_profiles)])
    print(f"reliability alpha {cronbach_alpha(items):.3f}")

    # a second test of the same trait should converge with the first
    test_a = items.mean(axis=1)
    test_b = test_a + rng.normal(0, 0.3, n_profiles)
    print(f"convergent validity {convergent_validity(test_a, test_b):.3f}")

    # shaping, higher target levels should yield higher measured scores
    targets = rng.integers(1, 10, 60)
    measured = targets + rng.normal(0, 1.5, 60)
    print(f"shaping fit {shaping_fit(targets, measured):.3f}")

Conclusion

The core achievement here is turning a vague intuition into a rigorous measurement. Language models do project something that behaves like a personality, and this work shows how to measure it with the same discipline psychologists apply to people, checking that the numbers are reliable and that they mean what they claim rather than simply reading scores off a quiz. That personality proved real and measurable mainly for larger, instruction tuned models, and essentially noise for small base ones, which is itself a useful map of where the trait comes from.

The conceptual shift worth remembering is the insistence on validation. Most evaluations of language models report a number and move on. This one refuses to trust a number until it has passed the reliability and validity tests that a century of psychometrics developed for exactly this problem, measuring something you cannot observe directly. That discipline is transferable to almost any fuzzy behaviour we want to evaluate in these systems, and adopting it would make a great deal of AI evaluation more honest.

The approach also proved that personality is not just measurable but steerable, and that the steering carries into real generated text, which is where the stakes turn serious. A tool that can tune how persuasive or warm an AI sounds is genuinely useful for building better assistants and genuinely dangerous in the wrong hands, and the authors are right to treat their measurement method as an auditing instrument for exactly that reason.

The honest limits keep it grounded. The evidence is monocultural, rooted in Western questionnaires and English text, the measured personality is a property of the model and prompt together rather than a fixed inner trait, and the conclusions rest on particular tests and a single short task. This is a rigorous snapshot, not a universal law, and the models it studied will not be the current ones for long.

Future directions follow naturally. Extend the validation to other languages and cultural frameworks, probe personality across long conversations rather than single turns, and, most importantly, build the auditing use case out into the kind of behavioural check regulators are starting to ask for. If that happens, the personality of an AI stops being an accident of training and becomes something we can measure, describe, and hold to account, which is the least we should ask of systems this many people now talk to.

Frequently asked questions