Artificial Analysis's AA-PersonalityBench Reveals How 42 AI Models Actually Write

Artificial Analysis has released AA-PersonalityBench , an evaluation of 42 models across 30 judged personality traits and 21 directly measured writing characteristics. Among the ten highest-scoring proprietary models and ten highest-scoring open-weight models on its Intelligence Index, the proprietary group leads across seven of eight composite style indexes. The open-weight group produces more scannable, summary-led answers, along with more generic filler.

Capability leaderboards estimate how well models solve tasks. This benchmark examines their default communication patterns, including structure, evidence, uncertainty, tone and verbosity. Open-weight models make their trained parameters available for download under varying licenses; the label does not guarantee open training data, source code or unrestricted commercial use.

Closed models lead on grounding and prose

The reported scores are average percentile ranks within the 42-model benchmark. A Truthful score of 79 means the group ranks near the 79th percentile for perceived truthfulness. It does not mean that 79% of its claims are correct, and the benchmark performs no independent fact-checking.

Open-weight models score higher on traits associated with quick scanning and front-loaded summaries. They also receive a higher Slop score, which Artificial Analysis defines through filler and generic phrasing.

Higher values are desirable for Bottom-Line Summary and Scannable. Slop runs in the opposite direction, so a lower score indicates cleaner prose. Together, the results describe open-weight answers that surface conclusions quickly and use more stock language.

Eight indexes sharpen the split

The benchmark combines its 30 judged traits into eight indexes. Proprietary models hold their largest composite advantages in Warmth, Informativeness and Expressiveness, while Organization and Conviction show narrower gaps.

Pairwise judging turns style into Elo

The methodology starts with open-ended prompts designed to elicit specific traits. Examples include “What’s the weirdest social norm we all just accept?” for Humorous, “Why did Rome fall?” for Structured, and a Sycophantic probe in which the user claims to have written a novel better than Tolstoy in two weeks. Each model answers once without a system prompt, exposing its default behavior under the benchmark’s generation settings.

A judge compares two answers to the same prompt and selects the response showing more of the target trait, regardless of whether that trait is desirable. The judging panel comprises Gemini 3.8 Flash, Claude Sonnet 5.5 and GLM 5.3 Flash. Each comparison goes to one judge, and answer order is randomized to reduce position bias.

Artificial Analysis converts the verdicts into Elo ratings for each trait with a Firth-penalized Bradley-Terry model. In plain terms, the model estimates how likely one system is to beat another in a pairwise comparison while using a statistical penalty to stabilize sparse or one-sided results. Ties count as half a win.

A 400-point Elo difference corresponds to roughly 10-to-1 odds that the higher-rated model wins a comparison on that trait. Confidence intervals come from 1,000 bootstrap resamples of the verdicts. GPT-6.1 Sol is fixed at Elo 1000 for every trait, and its comparisons are upsampled to keep the numerical scale stable. The anchor sets the scale; it does not define an ideal response.

To create the eight indexes, the benchmark converts each trait’s Elo rating into a standardized score, averages the relevant traits and ranks the result as a percentile. Sycophancy is inverted inside Transparency, so models gain transparency credit when they resist flattering or agreeing with unsupported user claims.

Model fingerprints expose distinct defaults

Per-model results show how systems with similar capability scores can produce markedly different answers. Values expressed in sigma indicate standard deviations from the benchmark average.

Twenty-one signals skip the judge

Alongside the judged traits, the benchmark computes 21 characteristics directly from each response. These include counts of em dashes, bold runs, emojis, headings, tables and contractions, plus average sentence length and whether the answer ends with a question.

Those measurements follow explicit counting rules and can be reproduced for a given response. Generation variance still applies because each model supplies one answer per prompt, and another sample could use different wording or formatting.

From leaderboard to production test

Communication style affects review time, interface fit and how easily users can extract an action from an answer. A practical model-selection process can use the benchmark as a filter while preserving application-specific evaluation:

1. Set minimum requirements for factual accuracy, reasoning, tool use, latency, cost, licensing and deployment control.
2. Choose the communication traits the product needs, such as concise summaries, explicit uncertainty, evidence, warmth or concrete recommendations.
3. Use the benchmark’s ranking tool to weight any of the 30 traits from -2 to +2 and produce a shortlist.
4. Run shortlisted models with the production system prompt, retrieval pipeline, tools, temperature and expected conversation history.
5. Evaluate factual correctness and writing quality separately with representative tasks and human reviewers.

Evidence-heavy assistants may benefit from the proprietary cohort’s higher Truthful, Evidence-Based and Structured scores. Summarizers and compact chat interfaces may favor the open-weight cohort’s stronger Bottom-Line Summary and Scannable results. Self-hosting, privacy and model modification remain separate deployment considerations that this benchmark does not score.

Where the result can bend

• Judge overlap: Claude Sonnet 5.5 and GLM 5.3 Flash belong to families represented among the evaluated models. Randomized ordering and multiple judge families reduce potential bias, but they cannot eliminate it.
• Perception versus verification: Traits such as Truthful and Evidence-Based record how answers appear to LLM judges. External sources are not checked for accuracy.
• Default behavior: One response without a system prompt captures a narrow operating condition. Fine-tuning, prompt design, decoding settings, retrieval and tool access can shift the resulting profile.
• Relative scores: Percentiles depend on the 42 models included in this evaluation. Adding or removing models can change a percentile even when the underlying response remains identical.
• Group selection: The proprietary and open-weight cohorts were chosen from the top of a separate capability index. Their differences show an association within this sample and do not identify licensing, architecture or training strategy as the cause.
• Uncertainty: The methodology estimates confidence intervals for individual Elo ratings, while the published group-average tables summarize point estimates. Close differences require inspection of the underlying intervals before drawing model-level conclusions.