Orthogonal Ensembles and Tested Explanations for Performer-Independent Body-Motion Emotion Recognition

What the contest asked for

Name which emotion a person is acting, from the movement of their body alone, for people the system has never seen before.

Entrants in the MMAC Challenge are given motion capture recordings of people acting emotions, each one labelled with the emotion the performer was asked to act, and have to build a system that names the emotion in a recording it was not trained on. The organisers keep the labels of the test recordings to themselves, so no entrant can score their own system on the test set, and a leaderboard the organisers run decides the contest.

A machine that reads a body rather than a face is useful wherever a face is not there to be read: at a distance, in the dark, turned away from the camera, or in a setting where filming faces would not be acceptable.

The recordings come from the Diverse Intercultural E-Motion Database of Asian Performers, DIEM-A, published by Cheng, Tseng, Fujiwara, Schneider and Kitamura at ACII 2025. One recording is one performer acting one emotion while a motion capture system follows reflective markers worn on the body: a 57-marker Vicon rig for most of the corpus, and a 41-marker OptiTrack rig for the five earliest Japanese performers, none of whom appear in the training split used here. What reaches the model is the skeleton that tracking produces: 24 body joints moving in three dimensions, plus one node carrying where the body is in the room, so 25 points in total. There is no video of the face, no sound and no view of the scene. Recordings run at 120 frames per second and a typical one lasts about 845 frames; every model on this page reads a fixed window of 64 of them, about half a second of movement (0.53 s), the window the organisers' own reference system reads.

There are twelve emotions to choose between: anger, contempt, disgust, fear, joy, sadness, surprise, jealousy, shame, guilt, gratitude and pride. Naming one of twelve by chance is right 8.3% of the time. Two things make it hard to do much better. Emotions close in meaning move alike, so a system takes jealousy for contempt and guilt for sadness or shame. And every person moves in their own way, so the gap between two people acting the same emotion can be wider than the gap between one person acting two emotions. The contest puts that second problem at the centre by splitting the performers 74 for training and 18 for test with nobody in both, which stops a system from succeeding by learning how one particular person moves.

Entries are scored by Macro-F1, which scores each of the twelve emotions on its own and averages the twelve results, so a system cannot do well by getting the common emotions right and ignoring the rare ones. It runs from 0 to 100%. The training split holds 7,992 recordings from 40 Japanese and 34 Taiwanese performers, 666 for each emotion. The test split holds 1,944 recordings from 18 performers, 9 Japanese and 9 Taiwanese.

Only one number on this page is scored against the organisers' test labels, since entrants do not have them: the final leaderboard result, 37.23% Macro-F1 and 37.50% accuracy over the 18 test performers, which received the Best Performance Award. Every other number here is measured on a stand-in built from the training performers alone: those 74 are divided into ten groups, and the whole procedure runs ten times, each time training on nine groups and testing on the group left out. One such run is a fold, and the paper calls the arrangement leave-performers-out, since nobody in a run's test recordings appears in its training recordings. It reproduces what makes the contest hard, on data whose labels we hold, and the leaderboard score sits inside the range that stand-in predicted.

What this work found

Combining models that make different mistakes did better than any single model we could build, and the account of what the model reads was tested rather than drawn.

Retrained on that split, the reference model the organisers supply, STGCN++, reaches 25.73 ± 4.03% Macro-F1. It treats the skeleton as a graph, with joints for nodes and bones for edges. The system submitted here reaches 36.80 ± 4.00%, which is 11.07 percentage points higher, written +11.07 pp.

That gain comes from disagreement between models rather than from a better model. Eleven models each score the twelve emotions, and the system averages those scores, with nothing in the averaging learned or tuned. The eleven are built in four ways, which is the point, because models built differently make different mistakes: graph convolution, attention, hybrid MLP, and models trained beforehand on other motion data and left unchanged here. Making one model bigger did not do the same work: the best single model we trained reaches 30.05 ± 3.48%. How far apart their mistakes fall is measured below.

The other half of the work tests its explanations instead of drawing them. Six tests were run and all six are reported, five positive and one negative. The main positive result concerns Laban Movement Analysis, a vocabulary movement professionals use to describe how a body moves, in terms such as how far the head bows or how far the trunk leans, written by people rather than learned from data. Rank the four body regions, head, arms, legs and torso, by how much the model used them, rank them again by those written attributes, and the two orders agree at Spearman ρ = +0.500, where +1 would be the same order, 0 no relation and −1 the reverse. A control built from ordinary motion measurements reaches only +0.033, about fifteen times weaker. For sadness the parts the model used are the ones an analyst would name: a bowed head, a lower head height, and a forward lean of the trunk.

The work claims less than those numbers might suggest on their own. The headline is the ten-run stand-in on the labelled training performers rather than the leaderboard score; that score, 37.23% Macro-F1 on the hidden test set, is a single number with no spread; and a Macro-F1 near 37% is low, which we do not defend as good. The limits section states each of these with the check that bounds it.

Results

Every row below is scored on the same ten runs over held-out performers, so the numbers compare directly.

The eleven models hold 12.98 M learned numbers between them. Answering one recording means running all eleven over the same 64-frame window and averaging their scores, with no fitted weights. Averaging the raw scores rather than the probabilities is worth +1.15 pp, and not because one model produces larger numbers than another: rescaling every model's scores to a common spread first still leaves +0.99 pp, and keeping only the order in which each model ranks the twelve emotions leaves +0.69 pp.

On the organisers' hidden test set, whose labels entrants never see, the submitted system scored 37.23% Macro-F1 and 37.50% accuracy on the final leaderboard and received the challenge's Best Performance Award. That is one score over 18 performers with no spread to report, so it stays out of Table 1; it sits inside the 36.80 ± 4.00% range from the ten runs.

Why combining the models works

The eleven models, which the paper calls members, rarely fail on the same recording, and no single one of them carries the result.

Taken pair by pair, the correlation between which recordings two of them get wrong falls between 0.15 and 0.43, and the lowest values belong to the models trained beforehand on other data, which differ most from the rest. This is what the title of the work calls orthogonal errors: the members go wrong in different directions, so the average cancels their mistakes rather than piling them up. Moderate correlation does not mean redundancy: the pair whose errors agree most still supply the largest single contribution. Removing the whole pretrained block at once costs −2.91 pp, so the gain is spread across the four kinds of model rather than carried by one.

The errors that survive fall where the task itself is ambiguous, between emotions close in meaning: jealousy into contempt, guilt into sadness or shame, which matches a low maximum confidence of about 62%. Macro-F1 varies by roughly a factor of four from one performer to another, about half of it how hard the performer is for any model and the other half which model is looking. The averaging cancels that second half, and the explanations below look at the same disagreement.

Explanations that were tested

Every explanation here comes with a test that could have failed, and one of the six did.

A picture of where a model looked cannot be wrong, since a decorative one looks exactly like a real one. So each claim here is paired with a check that could fail: hide a body part, add noise to it, or edit how it moves, and see whether the answer changes. The checks run after training on a strong member of the submitted system, cost 0 pp of Macro-F1 and need no retraining, and the two that carry the most weight were run again on the eleven-model system itself.

The checks group the body differently. Hiding works over six body parts, and again over all 25 tracked points; the Laban comparison works over four regions, which are head, arms, legs and torso. The noise in the second check has σ = 0.02.

On the submitted eleven-model system the Laban agreement is ρ = +0.517 (95% interval [+0.333, +0.700], p = 0.001) and the part-hiding gap stays positive in all ten runs, so the finding describes the system we submitted and not only one member of it. The motion edits corroborate that the model relies on that motion without proving causation, and the agreement between hiding and editing is itself only suggestive, at ρ = 0.49 over 6 body parts, p = 0.33, so the claim rests on the 0.500 against 0.033 contrast instead.

What this work does not claim

Each limit below comes with the check that bounds it.

• The headline is a stand-in for the test, not the leaderboard result. 36.80 ± 4.00% comes from the ten runs over the 74 labelled training performers. The leaderboard result on the hidden test set is 37.23% Macro-F1 and 37.50% accuracy, a single score with no spread, and it sits inside the stand-in's range. Every comparison here uses the same split, so the +11.07 pp gain is measured under matched conditions.
• We did not re-run the official baseline. The organisers report 25.21 ± 4.49% over all 92 performers, which we quote as an outside reference point. Our own reproduction of 25.73 ± 4.03% sits within one standard deviation of it, and every improvement is measured against that reproduction.
• The absolute score is low. A Macro-F1 near 37% across twelve emotions is not good in isolation, and we never argue that it is. It is about 4.4 times the 8.3% chance level, the gain is +43% over the matched reproduction, and all twelve emotions improve.
• The Laban agreement is about meaning, not cause. It shows that the parts the model used are the parts a movement analyst would name. Whether the model depends on them is a separate question, tested by hiding them (+0.124 ± 0.031) and by checking the ranking under noise (ρ = +0.983 ± 0.039).
• Motion edits corroborate rather than prove. The edits move answers in the same order of parts that hiding produces, which is two methods agreeing rather than a causal account. That agreement rests on 6 body parts (ρ = 0.49, p = 0.33), so the claim leans on the 0.500 against 0.033 contrast.
• The Japan and Taiwan difference is a split in the data. It is entangled with when the recordings were made and with how individuals move, and we do not read it as a cultural finding. A classifier can tell a performer's country from the motion 69.8% of the time, so country is tangled with the movement itself. Dropping the seven earliest Japanese training performers moves the score on the pooled held-out answers by −0.44 pp, 36.50% against 36.94%.
• The explanation results describe two specific systems. They cover a strong member of the submitted system and the submitted eleven-model system itself, not skeleton emotion models in general.

Terms

The table and the charts keep the paper's wording. These are the words they use that the text above does not.

STGCN++

The model the organisers supply as a reference point. It treats the skeleton as a graph, with joints as nodes and bones as edges.

LPO, leave-performers-out

The arrangement described at the top of this page, in which nobody appears in both the training and the test recordings.

Logit-mean

How the eleven models are combined: average the raw score each of them gives each emotion, before those scores are turned into probabilities. Nothing in the average is learned or tuned.

Pooled out-of-fold

Figure 2 labels its axis this way. Pooling gathers the held-out answers from all ten runs and scores them once, which gives 36.94%. The headline 36.80 ± 4.00% scores each run and averages the ten instead. The two are conventions on one system and should not be mixed.

Kinematics control

What Figure 3 calls the control built from ordinary motion measurements: each joint's speed, acceleration, range of motion and energy.

Trainable parameters

The count of numbers a model learns during training. The eleven here come to 12.98 M between them.

Citation

We will add pages and a DOI once the proceedings are published.

@inproceedings{nishida2026orthogonal,
  title     = {Orthogonal Ensembles and Tested Explanations for
               Performer-Independent Body-Motion Emotion Recognition},
  author    = {Nishida, Naoto and Ishiguro, Yoshio},
  booktitle = {2026 International Conference on Affective Computing and
               Intelligent Interaction Workshops and Demos (ACIIW)},
  year      = {2026},
  note      = {MMAC Challenge: Cross-Cultural Emotion Recognition from
               Body Movements}
}

The paper appears in the ACII 2026 Workshops proceedings for the MMAC Challenge. A preprint is on arXiv: arxiv.org/abs/2609.02510. The code is at github.com/nawta/diema-challenge.