The code and prose in this chapter were written by Claude Code (an AI coding agent) under human direction. They have not been line-by-line verified by a human and must not be published or cited as-is.
WarningPreliminary & unvetted analysis
These are exploratory analyses of preliminary, in-progress LEVANTE data. Results have not been reviewed or validated and require checking by the LEVANTE Data Coordinating Center before any use.
Scope
The Memory task (memory-game, forward + backward Corsi block span) is one of the longest tasks in the battery, and — because it is an untimed staircase — its length grows with ability. The DCC question: can we cut its length, and how? The task is not CAT; the levers are the staircase rules. Candidates: get to backward faster, require fewer correct repetitions per span length, skip staircase steps, tighten the error budget. Scores are IRT (multigroup Rasch, backward items simply calibrate as harder items on the same dimension), so shortened runs can be rescored on the same scale — the question is what we lose in precision, validity, and test–retest reliability.
Data: levante_data_latest:e9pf:v1_2 (2026-08 refresh; the four calibration datasets), model multigroup_dataset scalar Rasch from metadata v2_3. Heavy computation cached by _prep_memory_length.R.
Reverse-engineered from 1,644 runs of trial data (transition probabilities, episode structure, block terminations):
Forward block starts at length 2; +1 length after 3 consecutive correct; an error keeps you at the same length (there is no step-down).
A block ends after 3 cumulative errors (modern deployments; ~15% of runs, mostly early task versions, show larger budgets).
Backward restarts at length 2 and runs the same rule. (386 further runs stop after forward — attrition across all ability levels, not a rule; they are excluded above.)
Items calibrate as mg_{direction}_{grid}_len{N}-{attempt} with difficulties tied across attempts, so each (direction, grid, length) is one difficulty — a clean ~1-logit-per-rung ladder:
Code
res$dlook |>mutate(difficulty =-d) |>select(dir, grid, len, difficulty) |>pivot_wider(names_from = len, values_from = difficulty) |>arrange(dir, grid) |> knitr::kable(digits =1, caption ="Item difficulty (logits) by direction × grid × span length. Grid (2×2 vs 3×3) is a deployment property of the run, not a within-run phase.")
Item difficulty (logits) by direction × grid × span length. Grid (2×2 vs 3×3) is a deployment property of the run, not a within-run phase.
dir
grid
2
3
4
5
6
7
8
backward
2grid
-1.9
-1.4
-0.9
0.4
0.9
1.8
NA
backward
3grid
-1.8
-1.1
-0.3
0.7
1.9
2.6
NA
forward
2grid
-3.5
-2.9
-1.7
-0.4
0.6
1.6
NA
forward
3grid
-3.1
-2.3
-1.3
-0.1
1.3
2.1
2.4
The design consequence: every rung costs at least 3 correct trials, no matter how far below the child’s span it is, and strong children pay the ladder twice (forward, then backward from scratch).
Run duration (first-to-last-trial span) and its correlation with ability
dataset_label
n
median_min
q90_min
r_duration_theta
Colombia — Bogotá
448
3.36
5.74
0.75
Colombia — Caquetá/Boyacá
364
3.97
6.31
0.76
Germany — Leipzig
543
4.46
7.59
0.48
Canada — Ontario
288
4.56
6.92
0.73
Code
dur |>group_by(q =ntile(score, 4)) |>summarise(n =n(), median_min =median(mins), q90_min =quantile(mins, .9),median_trials =median(n_tr), .groups ="drop") |> knitr::kable(digits =1, caption ="Pooled duration by ability quartile: the top quartile takes more than twice as long as the bottom")
Pooled duration by ability quartile: the top quartile takes more than twice as long as the bottom
q
n
median_min
q90_min
median_trials
1
411
2.7
4.1
11
2
411
3.9
5.5
18
3
411
4.5
6.4
23
4
410
6.1
8.0
29
Code
dur |>ggplot(aes(x = score, y = mins, color = dataset_label)) +geom_point(alpha =0.3, size =0.8) +geom_smooth(method ="lm", se =FALSE) +scale_color_manual(values = dataset_pal, name =NULL) +labs(x ="Memory ability (θ)", y ="Run duration (min)",title ="The Memory task is a progressive tax on ability") +theme(legend.position ="bottom")
The claim checks out: median run ≈ 4.5 min, but duration correlates r ≈ .5–.8 with θ within dataset; the top ability quartile has a median of ~6 min and a 90th percentile of ~8 min — and trial timestamps stop at the last trial, so with instructions/practice the experienced length for those children is comfortably in the reported 8–10 min range.
Where do the minutes go? Per-trial time is mechanical in span length (sequence presentation is ~1 s per element plus reproduction):
Code
res$dt_med |>ggplot(aes(x = len, y = med)) +geom_line() +geom_point() +labs(x ="Span length", y ="Median seconds per trial",title =sprintf("Seconds per trial ≈ %.1f + %.1f × length", res$dt_coef[1], res$dt_coef[2]))
Code
easy <- mg |>inner_join(res$full |>select(run_id, est), by ="run_id") |>left_join(res$dlook, by =c("dir", "grid", "len")) |>filter(!is.na(d)) |>mutate(p_hat =plogis(est + d),secs = res$dt_coef[1] + res$dt_coef[2] * len)easy |>group_by(q =ntile(est, 4)) |>summarise(pct_time_easy =sum(secs[p_hat > .9]) /sum(secs),pct_time_backward_easy =sum(secs[p_hat > .9& dir =="backward"]) /sum(secs), .groups ="drop") |> knitr::kable(digits =2, caption ="Share of testing time spent on trials the child passes with >90% probability ('easy-trial tax'), by ability quartile")
Share of testing time spent on trials the child passes with >90% probability (‘easy-trial tax’), by ability quartile
q
pct_time_easy
pct_time_backward_easy
1
0.03
0.00
2
0.16
0.00
3
0.20
0.01
4
0.29
0.08
3. Rescoring machinery (validation)
All shortened variants are rescored with the calibrated Rasch difficulties and each dataset’s EAP prior via direct quadrature. Full-length rescoring reproduces the published pipeline scores:
Code
knitr::kable(res$validation, digits =3,caption ="Quadrature EAP on full runs vs published scores")
Quadrature EAP on full runs vs published scores
dataset
n
r_vs_published
pilot_mpieva_de_main
544
0.986
pilot_uniandes_co_bogota
448
0.990
pilot_uniandes_co_rural
364
0.999
pilot_western_ca_main
288
0.995
And the Rasch model is well-calibrated for simulating responses — predicted vs observed accuracy by ability quintile × length:
A smaller error budget is a pure truncation of the observed data (the path up to the Nth error is identical under either rule), so these need no simulation — we rescore what children actually did.
Error-budget truncations, rescored from observed data. median_min = predicted time on the retained trials (per-trial time model); DE retest ≈ 3.6 months; Bogotá stability ≈ 16 months.
policy
median_min
mean_se
r_with_full
retest_de
n_de
stability_bogota
n_bog
budget1
1.43
0.71
0.88
0.42
206
0.32
54
budget2
2.29
0.59
0.96
0.48
206
0.32
54
budget3 (current)
3.12
0.51
1.00
0.58
206
0.46
54
5. Simulation: restructuring the staircase
Path-changing policies are simulated from each run’s full-data θ̂ using the calibrated difficulties (the anchor policy S0 replicates the current rules — compare it to the observed row to calibrate trust; policy differences within the simulation are the inference target).
sim_summary <- res$sims |>group_by(policy) |>summarise(median_min =median(mins), q90_min =quantile(mins, .9),mean_se =mean(se), .groups ="drop") |>left_join( res$sims |>group_by(policy, rep) |>summarise(r =cor(est, theta), .groups ="drop_last") |>summarise(r_with_full =mean(r), .groups ="drop"),by ="policy")# simulated retest: per rep, pair each retest user's two wavessim_pairs <- res$sims |>inner_join(sc |>select(run_id, age), by ="run_id") |>group_by(policy, rep, dataset, user_id) |>filter(n() >=2, !is.na(age)) |>arrange(age) |>summarise(gap_mo = (age[2] - age[1]) *12, t1 = est[1], t2 = est[2],.groups ="drop") |>filter(gap_mo >=2)sim_retest <- sim_pairs |>filter(dataset %in%c("pilot_mpieva_de_main", "pilot_uniandes_co_bogota")) |>group_by(policy, rep, dataset) |>summarise(r =cor(t1, t2), n =n(), .groups ="drop") |>group_by(policy, dataset) |>summarise(r =mean(r), .groups ="drop") |>pivot_wider(names_from = dataset, values_from = r) |>rename(sim_retest_de = pilot_mpieva_de_main,sim_stability_bog = pilot_uniandes_co_bogota)sim_table <- sim_summary |>left_join(sim_retest, by ="policy")obs_row <-tibble(policy ="OBSERVED (current)",median_min =median(res$full$mins),q90_min =quantile(res$full$mins, .9),mean_se =mean(res$full$se),r_with_full =1,sim_retest_de = replay_summary$retest_de[ replay_summary$policy =="budget3 (current)"],sim_stability_bog = replay_summary$stability_bogota[ replay_summary$policy =="budget3 (current)"])knitr::kable(bind_rows(obs_row, sim_table), digits =2,caption ="Simulated staircase policies. Minutes are trial time only (no instructions). The OBSERVED row uses real data; S0 is the simulation anchor — trust differences between S-rows, calibrated by S0 vs OBSERVED.")
Simulated staircase policies. Minutes are trial time only (no instructions). The OBSERVED row uses real data; S0 is the simulation anchor — trust differences between S-rows, calibrated by S0 vs OBSERVED.
policy
median_min
q90_min
mean_se
r_with_full
sim_retest_de
sim_stability_bog
OBSERVED (current)
3.12
5.72
0.51
1.00
0.58
0.46
S0_current
3.54
5.80
0.48
0.90
0.41
0.37
S1_promote2
3.09
4.76
0.50
0.89
0.38
0.34
S2_bwd_start3
3.41
5.52
0.49
0.89
0.43
0.32
S3_promote2_bwd3
2.97
4.58
0.52
0.88
0.40
0.31
S4_promote2_starts3
2.76
4.33
0.53
0.88
0.37
0.38
S5_promote2_budget2_bwd3
2.14
3.59
0.59
0.85
0.38
0.28
Code
res$sims |>group_by(policy, q =ntile(theta, 4)) |>summarise(median_min =median(mins), .groups ="drop") |>pivot_wider(names_from = q, values_from = median_min,names_prefix ="θ Q") |> knitr::kable(digits =1, caption ="Median simulated minutes by ability quartile: where each policy saves its time")
Median simulated minutes by ability quartile: where each policy saves its time
policy
θ Q1
θ Q2
θ Q3
θ Q4
S0_current
1.9
3.1
4.0
5.4
S1_promote2
1.8
2.7
3.4
4.5
S2_bwd_start3
1.9
3.0
3.9
5.1
S3_promote2_bwd3
1.8
2.7
3.3
4.3
S4_promote2_starts3
1.7
2.5
3.1
4.0
S5_promote2_budget2_bwd3
1.2
1.9
2.5
3.3
6. Round 2: the team proposal, a CAT topline, and a fast-track
S6 ramp-1 + verify (team proposal): advance after a single correct (“even if a learner gets lucky, they will stall at the next”); fail-out = 3 errors at the current length; then 2 verification trials at length n−1; both blocks start at 2.
S7 CAT (topline, deliberately infeasible UX): each trial presents the maximum-information length (either direction) given the running EAP, at fixed test lengths of 8/12/16 trials — the information-per-minute bound that any staircase approximates.
S8 fast-track: the S4 rules, plus “if you’re good at forward, get to backward sooner” — clearing forward length 5 exits the forward block immediately, and backward then starts at length 4.
Median simulated minutes by ability quartile, round 2
policy
θ Q1
θ Q2
θ Q3
θ Q4
S0_current
1.9
3.1
4.0
5.4
S4_promote2_starts3
1.7
2.5
3.1
4.0
S6_ramp1_verify
2.9
4.1
4.7
5.5
S7_cat12
2.0
2.3
2.6
2.8
S7_cat16
2.6
3.0
3.5
3.8
S7_cat8
1.4
1.6
1.7
1.8
S8_fasttrack
1.7
2.4
2.8
3.1
7. Round 3: the deployed rules, exactly — and an age-safe package
Dev feedback (2026-09) clarified the deployed implementation: blocks end at min(a hardcoded trial cap of 21, 3 cumulative errors), and trials come from a fixed corpus. Both claims check out in the data:
Code
blocks <- mg |>group_by(dataset, run_id, dir) |>summarise(n_tr =n(), n_err =sum(!correct), .groups ="drop")blocks |>mutate(ending =case_when(n_tr ==21~"hit 21-trial cap", n_err >=3~"3+ errors",TRUE~"incomplete/other")) |>count(dir, ending) |>group_by(dir) |>mutate(pct =round(n /sum(n), 3)) |> knitr::kable(caption ="How blocks end. No block exceeds 21 trials; cap-bound blocks with >3 errors are older task versions with different error budgets.")
How blocks end. No block exceeds 21 trials; cap-bound blocks with >3 errors are older task versions with different error budgets.
dir
ending
n
pct
backward
3+ errors
1227
0.972
backward
hit 21-trial cap
20
0.016
backward
incomplete/other
15
0.012
forward
3+ errors
1533
0.932
forward
hit 21-trial cap
93
0.057
forward
incomplete/other
18
0.011
Notes for implementation:
The 3-cumulative-errors rule is confirmed (it is what rounds 1–2 reverse-engineered), as is promotion after 3 consecutive correct: across 4,087 post-error transitions, length never advances after an error, so the deployed corpus pointer is behaviorally a staircase.
The cap binds for the able kids: 5.2% of forward blocks (1.4% backward) end at 21 trials with <3 errors in modern deployments. Because rounds 1–2 simulated without the cap, they overstated time for exactly those children — hence this cap-aware rerun.
Corpus-trim vs rule-based is psychometrically a wash: the published trials carry no attempt index (mg_forward_3grid_len4); attempt numbering happens at scoring prep, and difficulties are tied across attempts. Any implementation producing the same length schedule scores identically — the choice is engineering, not measurement.
Should forward still start at 2? The youngest kids say yes
Code
sc1 <- sc |>distinct(run_id, .keep_all =TRUE)mga <- mg |>inner_join(sc1 |>select(run_id, age), by ="run_id") |>filter(!is.na(age)) |>mutate(age_band =cut(age, c(4, 6, 7, 8, 10, 13),labels =c("5-6", "6-7", "7-8", "8-10", "10+")))first_enc <- mga |>filter(dir =="forward") |>group_by(run_id, age_band, len) |>slice_min(trial_number, n =1) |>ungroup() |>filter(len <=4) |>group_by(age_band, len) |>summarise(n =n(), accuracy =mean(correct), .groups ="drop") |>pivot_wider(names_from = len, values_from =c(n, accuracy))stuck <- mga |>filter(dir =="forward") |>group_by(run_id, age_band) |>summarise(mx =max(len), .groups ="drop") |>group_by(age_band) |>summarise(n =n(), pct_never_pass_len2 =mean(mx ==2), .groups ="drop")knitr::kable(first_enc, digits =2,caption ="Forward first-encounter accuracy by age band × length. Length-3 rows are survivors of length 2 — unconditional accuracy at 3 would be lower.")
Forward first-encounter accuracy by age band × length. Length-3 rows are survivors of length 2 — unconditional accuracy at 3 would be lower.
age_band
n_2
n_3
n_4
accuracy_2
accuracy_3
accuracy_4
5-6
130
89
53
0.71
0.70
0.57
6-7
182
150
108
0.84
0.82
0.69
7-8
209
171
137
0.82
0.83
0.75
8-10
448
416
355
0.93
0.90
0.79
10+
569
540
485
0.95
0.91
0.88
NA
25
23
22
0.92
0.91
0.82
Code
knitr::kable(stuck, digits =2,caption ="Share of runs that never pass forward length 2, by age band")
Share of runs that never pass forward length 2, by age band
age_band
n
pct_never_pass_len2
5-6
130
0.32
6-7
182
0.18
7-8
209
0.18
8-10
448
0.07
10+
569
0.05
NA
25
0.08
For 5–6-year-olds, length 2 is not a warm-up: first-encounter accuracy is .72 and 31% of their runs never pass length 2 at all (vs 5–7% at ages 8+). Starting forward at 3 would hand a third of the youngest children nothing but failures. The age-safe move: keep forward starting at 2 (with promote-on-2 the cost to able kids is a single extra ~16 s of easy trials), and start backward at 3 only for children who cleared forward length 3.
Median simulated minutes by ability quartile, cap-aware
policy
θ Q1
θ Q2
θ Q3
θ Q4
C0_current
1.9
3.1
4.0
5.4
C1_devcaps
1.9
3.1
3.8
4.5
C2_promote2
1.8
2.7
3.5
4.5
C3_safe
1.8
2.7
3.3
4.3
C4_S4caps
1.7
2.5
3.1
4.1
Code
res3$sims |>filter(theta <-1.5) |>group_by(policy) |>summarise(n_runs =n_distinct(run_id),mean_fwd_successes =mean(n_succ_fwd),median_min =median(mins), .groups ="drop") |> knitr::kable(digits =1,caption ="Experience of low-ability children (θ < −1.5): forward successes per run under each policy")
Experience of low-ability children (θ < −1.5): forward successes per run under each policy
policy
n_runs
mean_fwd_successes
median_min
C0_current
344
5.3
1.8
C1_devcaps
344
5.3
1.8
C2_promote2
344
4.1
1.7
C3_safe
344
4.1
1.7
C4_S4caps
344
2.6
1.6
Verdict
1. The length complaint is verified and is a tax on ability. Duration correlates r ≈ .5–.8 with θ; the top ability quartile has a median of ~6 min and a 90th percentile of ~8 min of pure trial time — with instructions and practice, that is the reported 8–10 minute experience. Top-quartile children spend 29% of their testing time on trials they pass with >90% probability, mostly re-climbing rungs far below their span.
2. Don’t touch the error budget — it is where the measurement lives. The exact-replay analysis (real data, truncated and rescored) shows that cutting the per-block error budget from 3 to 2 saves ~27% of time but costs real reliability: DE 3.6-month retest drops .58 → .48 and Bogotá 16-month stability .46 → .32. Errors are the informative trials — they bracket the child’s ceiling where Rasch information peaks — and the three errors per block are most of the task’s signal.
3. The fat is the promotion rule and the restarts. In the calibrated simulation (anchor S0 reproduces the current rules; sim retest values are attenuated relative to observed — compare S-rows to each other, not to the observed row):
Promote after 2 consecutive correct (S1): −13% median time, −18% at the 90th percentile; SE .50 vs .48; retest within simulation noise of the anchor.
Start backward at length 3 (S2): −4% overall (concentrated in able kids); no reliability cost.
Both, plus forward also starts at 3 (S4): −22% median, −25% at q90 (top-quartile median 5.4 → 4.0 min of trial time); SE .53 vs .48; retest indistinguishable from the anchor (DE .37 vs .41, Bogotá .38 vs .37 — differences within rep noise).
The aggressive add-on (S5: also budget 2) reaches −40% but inherits the replay-demonstrated reliability cost of the smaller error budget.
Recommendation to the DCC (round 1, later refined): adopt promote-2 with higher starts (S4). Round 3 — after the dev team clarified the deployed 21-trial cap and the youngest children’s data were examined — revises this to the C3 package below; see point 5.
4. Round-2 verdicts.
The ramp-on-1 proposal (S6) is measurement-neutral but time-negative. Its precision and reliability exactly match the current task (SE .48, r .90, simulated retest .41/.39 — the best of any variant), but it takes longer than the current rules (median 4.33 vs 3.54 min; q90 6.16 vs 5.80). The intuition — “a lucky pass stalls at the next length” — is exactly right about validity, but the stall is where the cost lives: under promote-on-1, the 1–2 errors a child accumulates before each lucky pass count toward no budget and reset on promotion, so every rung becomes a small grind, and the final fail-out (3-at-length, plus 2 verification trials) happens at the child’s longest, slowest trial lengths. Time shifts from cheap short trials to expensive long ones. The verification idea itself is fine — it is the promote-on-1 error-reset that backfires.
The CAT topline says the staircase is already near the frontier. A max-information CAT matched to current precision (16 trials: SE .47, r .90, retest .41/.35) saves only ~8% of time over the current task — because informative trials for able children are long trials, CAT cannot escape the seconds-per-length physics. CAT-12 lands on essentially the S4 tradeoff (SE .53, retest .37/.34, 2.44 vs 2.76 min). So there is no large adaptive headroom left on the table: S4 captures most of what even an infeasible CAT could deliver.
The forward fast-track (S8) is a worthwhile optional add-on to S4. Exiting forward after clearing length 5 (backward then starts at 4) specifically attacks the longest sessions: q90 falls 4.33 → 3.39 min (top-quartile median 4.0 → 3.1), at a small further precision cost (SE .55 vs .53) and unchanged reliability (.35/.35 vs .37/.38 — within rep noise). If the DCC wants the worst-case session under ~5 minutes including instructions, S4 + fast-track is the package.
5. Round 3 (final): the deployed rules are cap + errors, and the levers are complementary. With the deployed 21-trial cap in the simulation (C0 anchor):
The dev’s cap-only reduction (14/12) fixes the top and nothing else: q90 falls 5.80 → 4.63 min but the median barely moves (3.53 → 3.44) — the error-bound majority is untouched. Conversely promote-2 alone (C2) fixes the middle (median −13%) but leaves the cap-bound top long (q90 4.77). The two levers address different children.
The youngest children argue against starting forward at 3: 31% of 5–6-year-olds never pass forward length 2, their first-encounter accuracy at length 2 is .72, and under start-at-3 (C4) low-ability children’s forward successes per run drop from 4.1 to 2.6 — a meaningfully harsher experience for ~0.2 min of extra saving.
Final recommendation — C3, the age-safe package: promote after 2 consecutive correct; forward starts at 2; backward starts at 3 only if the child cleared forward length 3 (else 2); caps 14 forward / 12 backward; 3-error budget unchanged. Effect vs deployed rules: median −15% (3.53 → 3.00 min of trial time), 90th percentile −21% (5.80 → 4.57), SE .48 → .51, simulated retest within rep noise (DE .38 vs .42; Bogotá .34 vs .36), and the youngest children’s success experience preserved.
Implementation note: under the tied-difficulty scoring convention, trimming the fixed corpus vs moving to rule-based generation are measurement-equivalent — whichever is easier to build. The corpus just needs enough sequences per length to cover repeats within a block (current bank depth: 6–15 presentations per direction × grid × length were observed and calibrated).
Two caveats. First, the simulation trusts the Rasch model’s person homogeneity (no sequence-specific effects beyond length; the calibration plot in §3 supports this). Second, changing the staircase changes which attempt-numbered items exist, so the scoring bank needs the new (direction × grid × length) items mapped — trivial under the current tied-difficulty convention, but it should be done deliberately, and a recalibration pass once ~500 runs of the new form exist would confirm the difficulty ladder transfers.
