The code and prose in this chapter were written by Claude Code (an AI coding agent) under human direction. They have not been line-by-line verified by a human and must not be published or cited as-is.
These are exploratory analyses of preliminary, in-progress LEVANTE data. Results have not been reviewed or validated and require checking by the LEVANTE Data Coordinating Center before any use.
This appendix is the June 2026 forensic account, kept as a record. The consolidated Stories chapter, on data rebuilt from raw with the PR #27 corrections, revises three of its conclusions (see that chapter’s “June conclusions → now” table):
Our June ToM analyses (stories_tom, tom_dif_multigroup; now retired to old/ and superseded by the consolidated Stories chapter) found large cross-site DIF concentrated on reality-check (control) items — and excluding them bought scalar invariance. But that never made theoretical sense: comprehension checks should be the most translatable items. Hypothesis: the DIF is a data-flow bug (item mismapping in early deployments), not real item functioning.
To test it we built a shadow processing pipeline: raw (unprocessed) Bogotá + Leipzig data, re-processed with the levantemodels internals but keeping the columns production drops — item_uid_source, the raw item string (deployment version codes), answer/distractors (English stimulus keys — language-independent ground truth), timestamps, and run-level task_version/corpus. Heavy downloads are cached in data/tom_raw/; this notebook re-derives results from those caches.
Verdict up front: the hypothesis is substantially confirmed — but the mechanism is richer than one mismap. We found provable, site×epoch-specific defects (answer-key inversions, hostile-attribution content mislabeled as ToM, mapping shifts), split across BOTH sites. Repairing just the provable defects removes ~70% of the scalar non-invariance (χ² 702 → 212).
source(here::here("common.R"))
suppressMessages(library(mirt))
loop <- read_rds(here("data/tom_raw/dif_loop_results.rds"))
prepG <- read_rds(here("data/tom_raw/prepG_results.rds"))
prepH <- read_rds(here("data/tom_raw/prepH_results.rds"))
both <- loop$bothTracing levantemodels’s process-trials.R and recode_tom:
item_uid, a curated trial_items map (trial_id → uid, assigned retroactively), and a content-keyed mapping_items join. The production assertion that these never conflict is commented out.tom_story4_…) is constructed at recode time from the raw item string’s leading digits + corpus-metadata entry.item string is a deployment version code (1c, 4d, 4c_new3) and answer/distractors are English stimulus keys (rug, shelf, medium_cup) — both usable as independent ground truth.both |> count(site, item_uid_source) |>
pivot_wider(names_from = site, values_from = n, values_fill = 0L) |>
knitr::kable(caption = "Which source supplies each trial's item identity")| item_uid_source | CO | DE |
|---|---|---|
| item_uid_trial | 6033 | 8027 |
| item_uid | 0 | 1457 |
Key asymmetry: CO has no deployed uids at all — its ToM identity rests entirely on the retroactive trial-map (all CO ToM data is from the May–June 2024 beta deployments). DE mixes map-based (early) and deployed (late) ids — with no overlap, so the sources can never cross-validate each other. Additionally, ~4,200 valid CO trials carry no identity information at all and are silently dropped by production processing.
ha <- both |> filter(str_detect(answer, "hostile-attribution") |
str_detect(distractors, "hostile-attribution"))
ha |> mutate(ym = format(as.Date(server_timestamp), "%Y-%m")) |>
count(site, ym, name = "n_trials") |>
knitr::kable(caption = glue("{nrow(ha)} trials in {n_distinct(ha$run_id)} runs carry hostile-attribution answer keys but ToM item ids"))| site | ym | n_trials |
|---|---|---|
| DE | 2024-09 | 1708 |
| DE | 2024-10 | 1372 |
ha |> count(item_uid, sort = TRUE) |> head(8) |>
knitr::kable(caption = "ToM uids polluted by ha-keyed trials (top 8)")| item_uid | n |
|---|---|
| tom_deception_emotion_reasoning_1 | 110 |
| tom_deception_false_belief_1 | 110 |
| tom_deception_false_belief_2 | 110 |
| tom_deception_false_belief_3 | 110 |
| tom_deception_reality_check_1 | 110 |
| tom_deception_reality_check_2 | 110 |
| tom_deception_reality_check_3 | 110 |
| tom_interpretation_emotion_reasoning | 110 |
110 Leipzig runs (Sept–Oct 2024) logged hostile-attribution answer keys under ToM uids — spread across deception, interpretation, and moral-reasoning items, including the hand-flagged deception_reality_check_2/_3 and moral_reasoning_reality_check_1. Zero such trials in CO. (Curiously these trials’ accuracy patterns match CO’s clean values — the labels are corrupt but the scoring of that epoch behaved like the canonical items; the late-DE epoch is where the keys go wrong, below.)
A comprehension question cannot score below chance unless its key is wrong. Binomial screen of every (item × site) cell vs. its chance level (ha-epoch excluded):
prepG$broken |>
mutate(across(c(acc, chance), \(x) round(x, 2))) |>
select(site, item_uid, n, accuracy = acc, chance) |>
knitr::kable(caption = "Cells significantly BELOW chance (p < .01): provably broken keys")| site | item_uid | n | accuracy | chance |
|---|---|---|---|---|
| DE | tom_deception_reality_check_2 | 193 | 0.29 | 0.50 |
| DE | tom_deception_reality_check_3 | 193 | 0.17 | 0.50 |
| CO | tom_moral_reasoning_reality_check_1 | 238 | 0.08 | 0.50 |
| CO | tom_reference_reference | 226 | 0.19 | 0.33 |
Four provable defects, split across sites: CO’s moral_reasoning_reality_check_1 (8% on a 2AFC! → inverted key: flipping gives 92% ≈ DE’s 98%) and reference_reference; DE’s (late-epoch) deception_reality_check_2/_3 (29%/17% → inverted; the flips land near CO’s values). All four are on the paper’s hand-flagged list.
Using the version-code blocks and English answer keys (modal per site × block × position):
shelf, rc_2↔︎coats in both; story-5 reference↔︎medium_cup, rc↔︎large_cup in both) — the wholesale “IDs reversed between versions” scenario is ruled out.tree/fountain keys in CO but behind-tree/behind-fountain in DE — different stimulus variants pooled under the same item.2f block a 47-run sub-epoch maps two consecutive positions to the same uid (false_belief_2 twice).6d trials to false_belief_1/_2 (canonically 6c items) — with option-count mismatches proving the content differs — explaining the hand-flagged second_order_false_belief_2–5.Fast rest-score logistic DIF (correct ~ rest_z + site per item, BH-corrected) across data preparations:
summ <- function(res, lab) res |>
mutate(qtype = if_else(str_detect(item, "reality_check"), "control", "target")) |>
group_by(qtype) |>
summarise(n_items = n(), n_flag = sum(p_adj < .05),
mean_abs_beta = round(mean(abs(beta_site)), 2), .groups = "drop") |>
mutate(prep = lab)
bind_rows(
summ(loop$resA, "A: production identity (baseline)"),
summ(loop$resC, "C: identity rebuilt from version-code + position"),
summ(prepG$resG, "G: ha-epoch + broken cells removed"),
summ(prepH$resH, "H: repaired (keys flipped, unrecoverables dropped)")
) |> select(prep, qtype, n_items, n_flag, mean_abs_beta) |>
knitr::kable(caption = "DIF by prep: controls vs targets (mean |site beta|, logits)")| prep | qtype | n_items | n_flag | mean_abs_beta |
|---|---|---|---|---|
| A: production identity (baseline) | control | 10 | 6 | 1.47 |
| A: production identity (baseline) | target | 18 | 9 | 0.61 |
| C: identity rebuilt from version-code + position | target | 23 | 9 | 0.70 |
| G: ha-epoch + broken cells removed | control | 7 | 4 | 3.88 |
| G: ha-epoch + broken cells removed | target | 16 | 7 | 0.76 |
| H: repaired (keys flipped, unrecoverables dropped) | control | 10 | 5 | 2.98 |
| H: repaired (keys flipped, unrecoverables dropped) | target | 16 | 8 | 0.75 |
Reading the ladder honestly: position-rebuilt identity (C) halves control DIF — but is itself invalid because question order differs by site. The removal/repair preps (G/H) eliminate the proven defects but concentrate the remaining contrast on residual items (the mean over surviving controls rises even as the proven offenders are fixed). The decisive test is the multigroup model on the full item set:
prepH$mg |> knitr::kable(caption = "Multigroup Rasch, scalar vs configural (controls INCLUDED)")| prep | n_items | n_persons | X2 | df | p | dBIC |
|---|---|---|---|---|---|---|
| unrepaired (ha-epoch dropped) | 26 | 427 | 701.5 | 25 | 0 | 550.0 |
| repaired (flips + drops) | 25 | 427 | 212.2 | 24 | 0 | 66.8 |
Repairing only the provable defects (3 key-flips + 1 cell drop + ha-epoch exclusion) cuts the scalar-vs-configural misfit by ~70% (χ² 702 → 212; ΔBIC 550 → 67). Full scalar invariance is still rejected — see residuals.
prepH$resH |> arrange(p_adj) |> head(6) |>
mutate(beta_site = round(beta_site, 2)) |>
select(item, beta_site, n, p_adj) |>
knitr::kable(caption = "Largest residual DIF after repair")| item | beta_site | n | p_adj |
|---|---|---|---|
| tom_story1_reality_known_false_belief | 1.48 | 427 | 0.0000009 |
| tom_story2_moral_reasoning_reality_check_2 | 3.99 | 373 | 0.0000009 |
| tom_story3_interpretation_false_belief | 1.66 | 415 | 0.0000480 |
| tom_story4_deception_false_belief_2 | -1.51 | 325 | 0.0000730 |
| tom_story3_interpretation_reality_check_1 | 3.22 | 381 | 0.0000771 |
| tom_story5_reference_reality_check | 1.33 | 412 | 0.0004021 |
The residual profile is suspicious in the same direction: late-DE accuracy of 98–99% on several controls (moral_rc_2, interpretation_rc_1) vs CO at ~65% — near-ceiling one-sided performance on supposedly informative items. This smells like further platform/keying differences (e.g., late-DE displaying or scoring these as trivially correct), but logs alone can’t prove it. These need task-code / corpus archaeology (compare the deployed theory-of-mind task builds for the CO May-2024 vs DE late-2024 epochs).
add_item_ids; add a below-chance validator rule (any item-site cell significantly below chance fails ingest); investigate the 4,200 unmappable CO ToM trials; document that ToM item identity is retro-mapped for all pre-July-2024 data.