The code and prose in this chapter were written by Claude Code (an AI coding agent) under human direction. They have not been line-by-line verified by a human and must not be published or cited as-is.
WarningPreliminary & unvetted analysis
These are exploratory analyses of preliminary, in-progress LEVANTE data. Results have not been reviewed or validated and require checking by the LEVANTE Data Coordinating Center before any use.
Scope
Sentence Reading Efficiency (SRE) is a speeded literacy measure adapted from ROAR (intro-sre; Yeatman et al., 2024). Children read simple sentences one at a time and judge each true or false (2AFC), reading as many as they can in 180 seconds. ROAR’s official score is correct − incorrect within the time window — the subtraction is a guessing correction (a random guesser nets ≈ 0). LEVANTE z-scores this count within dataset so it sits on a roughly comparable scale to the IRT tasks (score_type = "guessing_adjusted_number_correct_scaled").
The scoring bug.levantemodels::score_sre() computes filter(trial_number <= 180) then (sum(correct) - sum(!correct)) / 180 — i.e. it truncates at the first 180 trials, not 180 seconds. (The /180 divisor is cosmetic: it cancels in the z-score.) This notebook asks:
Timing forensics: do runs actually last 180 seconds? (Spoiler: no — there is real structure here, including double-length runs.)
Corrected scoring: rate-normalize to a per-180s net-correct rate. How much do scores change? Do age / ROAR-Phoneme (PA) / ROAR-Word (SWR) correlations change, within and across sites?
Efficiency scoring: discussion of correctness+RT models (drift-diffusion IRT and alternatives) plus an EZ-diffusion proof of concept on the trial data.
Data: levante_data_pilots_hackathon:fpx0:v1_1 via the new levante package, cached by tasks/_pull_sre_hackathon.R.
sre |>group_by(dataset_label) |>summarise(runs =n_distinct(run_id), children =n_distinct(user_id),trials =n(), .groups ="drop") |> knitr::kable(caption ="SRE trial data by dataset (hackathon v1_1)")
SRE trial data by dataset (hackathon v1_1)
dataset_label
runs
children
trials
Colombia — Bogotá
806
624
30832
Colombia — Caquetá/Boyacá
576
569
22474
Germany — Leipzig
151
151
7002
Canada — Ontario
91
86
7053
1. Timing forensics: is 180 seconds always 180 seconds?
Two per-run duration measures: time-on-sentences (cumulative RT — the task records per-sentence RT in ms) and wall-clock span (last minus first trial server timestamp). Neither is exactly the timer, but they bracket it: cumulative RT omits inter-trial overhead; timestamp span omits the first trial’s duration and can inflate under server lag or pauses.
Per-run duration (seconds): cumulative RT and wall-clock span
dataset_label
cum_rt_q05
cum_rt_median
cum_rt_q95
cum_rt_max
span_q05
span_median
span_q95
span_max
Colombia — Bogotá
108
160
170
174
157
180
204
369
Colombia — Caquetá/Boyacá
87
161
171
175
135
178
200
373
Germany — Leipzig
150
164
171
174
169
182
197
561
Canada — Ontario
156
166
326
342
165
176
365
404
Code
run_time |>ggplot(aes(x = cum_rt, fill = dataset_label)) +facet_wrap(vars(dataset_label), scales ="free_y") +geom_histogram(binwidth =10, boundary =0, show.legend =FALSE) +geom_vline(xintercept =180, lty =2) +scale_fill_manual(values = dataset_pal) +labs(x ="Cumulative RT per run (s)", y ="Runs",title ="Most runs sum to just under 180 s of reading time",subtitle ="Dashed line = nominal 180 s deadline. Note the Canadian runs near 330 s.")
Three regularities:
The 180 s deadline is real but not exact. Median time-on-sentences is ~160–165 s (the remainder is inter-trial overhead: wall-clock span medians ~180 s). So the timer fires on schedule for most runs.
A left tail of short runs exists everywhere — Colombia especially (rural 5th percentile ≈ 87 s of reading time). Under the current rule these short runs are scored as if the child had the full window.
Canada (Western) has genuinely double-length runs.
Block structure: some Canadian runs contain two 180-s blocks
The item bank prefix identifies sentence corpora. Colombia interleaves three corpora (spanish_lab, spanish_ai, spanish_ai-human) within one timed block; Germany uses one corpus (uzh). But 20 Canadian runs contain two sequential blocks (lab sentences, then ai sentences), each with its own ~180-s window — roughly 360 s of testing scored as if it were one window.
Code
run_blocks <- sre |>group_by(dataset_label, run_id, corpus) |>summarise(block_rt =sum(rt_s), n =n(), .groups ="drop_last") |>summarise(blocks =paste(sort(corpus), collapse =" + "),two_windows =sum(block_rt >120) >=2,.groups ="drop")run_blocks |>count(dataset_label, blocks, two_windows) |>arrange(dataset_label, desc(n)) |> knitr::kable(caption ="Corpus composition of runs. two_windows = ≥2 corpora each with >120 s of reading time (sequential blocks, not interleaving)")
Corpus composition of runs. two_windows = ≥2 corpora each with >120 s of reading time (sequential blocks, not interleaving)
dataset_label
blocks
two_windows
n
Colombia — Bogotá
spanish_ai + spanish_ai-human + spanish_lab
FALSE
785
Colombia — Bogotá
spanish_ai + spanish_lab
FALSE
20
Colombia — Bogotá
spanish_ai + spanish_ai-human
FALSE
1
Colombia — Caquetá/Boyacá
spanish_ai + spanish_ai-human + spanish_lab
FALSE
563
Colombia — Caquetá/Boyacá
spanish_ai + spanish_lab
FALSE
13
Germany — Leipzig
uzh
FALSE
151
Canada — Ontario
aiV1P1
FALSE
27
Canada — Ontario
lab
FALSE
25
Canada — Ontario
ai + lab
TRUE
20
Canada — Ontario
aiV1P2
FALSE
19
Code
run_time |>ggplot(aes(x = n_trials, fill = dataset_label)) +facet_wrap(vars(dataset_label), scales ="free_y") +geom_histogram(binwidth =10, boundary =0, show.legend =FALSE) +geom_vline(xintercept =180, lty =2) +scale_fill_manual(values = dataset_pal) +labs(x ="Trials per run", y ="Runs",title ="Only Canada ever reaches the 180-trial truncation point",subtitle ="Dashed line = the current rule's truncation at 180 trials")
Side-quest verdict: the first-180-trials filter is a no-op for Germany and Colombia (no run exceeds 180 trials — max 140). It bites only in Canada (5 runs exceed 180 trials, max 238), and there it truncates arbitrarily across the two-block structure. Meanwhile the actual problem is the reverse: effective testing time varies from ~1 to ~4 minutes across runs, and a raw count treats them all as equal.
2. Corrected scoring: rate normalization
The corrected score is a rate: net correct per 180 s of reading time,
then z-scored within dataset (preserving the pipeline’s scaling convention). We use time-on-sentences rather than wall-clock span as the denominator because it is robust to server-timestamp lag and pauses; the two agree closely (sensitivity check below). Runs with under 30 s of reading time are excluded from rate scoring (extrapolation too aggressive).
First, verify we can reproduce the published (buggy) scores exactly:
Code
old_score <- sre |>group_by(dataset, run_id) |>filter(trial_number <=180) |>summarise(net =sum(correct) -sum(!correct), .groups ="drop") |>group_by(dataset) |>mutate(score_old =scale(net)[, 1]) |>ungroup()old_score |>inner_join(sre_scores_pub, by =c("dataset", "run_id")) |>group_by(dataset_label) |>summarise(n =n(), r_with_published =cor(score_old, score), .groups ="drop") |> knitr::kable(digits =3,caption ="Local reimplementation of the current rule vs published scores (r ≈ 1; residual differences come from a few scored runs with no rows in the trials table, which shift the z-scale slightly)")
Local reimplementation of the current rule vs published scores (r ≈ 1; residual differences come from a few scored runs with no rows in the trials table, which shift the z-scale slightly)
SRE validity correlations within dataset, current vs corrected scoring
dataset_label
criterion
n
r_current
r_corrected
delta
Colombia — Bogotá
age
533
0.60
0.60
0.00
Colombia — Bogotá
pa
365
0.44
0.45
0.00
Colombia — Bogotá
swr
198
0.55
0.57
0.02
Colombia — Caquetá/Boyacá
age
567
0.44
0.42
-0.03
Colombia — Caquetá/Boyacá
pa
557
0.59
0.57
-0.02
Germany — Leipzig
age
151
0.54
0.54
0.01
Germany — Leipzig
swr
150
0.59
0.58
-0.01
Canada — Ontario
age
86
0.32
0.39
0.08
Canada — Ontario
pa
69
0.45
0.42
-0.03
Code
pooled <- sre_first |>pivot_longer(c(age, pa, swr), names_to ="criterion", values_to ="crit") |>filter(!is.na(crit)) |>group_by(criterion) |>summarise(n =n(),r_current =cor(score_old, crit),r_corrected =cor(score_new, crit), .groups ="drop") |>mutate(delta = r_corrected - r_current)knitr::kable(pooled, digits =2,caption ="Pooled across datasets (all scores z-scaled within dataset; PA/SWR θ are by-language, so pooled r mixes calibrations — interpret within-site values first)")
Pooled across datasets (all scores z-scaled within dataset; PA/SWR θ are by-language, so pooled r mixes calibrations — interpret within-site values first)
criterion
n
r_current
r_corrected
delta
age
1337
0.50
0.49
-0.01
pa
991
0.44
0.45
0.00
swr
370
0.54
0.53
-0.01
Longitudinal stability (Bogotá, ~16-month gap)
Bogotá has ~180 children with SRE at two waves. Rank-order stability across 16 months is a demanding criterion: a better score should preserve more true-score variance.
3. Fast guessing: the bridge to model-based scoring
The C − I correction neutralizes guessing in expectation (a guesser nets ≈ 0), but guessing still costs signal: guessed trials burn testing time and add noise. And guessing is far from uniform across sites:
Code
sre |>mutate(rt_bin =cut(rt_s, c(0, .5, 1, 2, 4, 8, 16, Inf))) |>group_by(dataset_label, rt_bin) |>summarise(accuracy =mean(correct), pct =n(), .groups ="drop_last") |>mutate(pct = pct /sum(pct)) |>ungroup() |>ggplot(aes(x = rt_bin, y = accuracy, color = dataset_label,group = dataset_label)) +geom_hline(yintercept =0.5, lty =3) +geom_line() +geom_point(aes(size = pct), alpha =0.7) +scale_color_manual(values = dataset_pal, name =NULL) +scale_size_area(max_size =6, guide ="none") +labs(x ="RT (s)", y ="Accuracy",title ="Sub-second responses are at chance — and their prevalence varies hugely by site",subtitle ="Point size ∝ share of site's trials in that RT bin; dotted line = 2AFC chance") +theme(legend.position ="bottom")
This is a strong argument for scoring models that use the joint (correct, RT) signal per trial, rather than counts alone.
4. Toward an efficiency score from correctness + RT
The modeling landscape
SRE’s design — self-paced 2AFC trials under a global deadline, with per-trial RT and accuracy — admits several principled treatments:
Rate normalization (implemented above). Minimal fix: net correct per unit time. Already a speed–accuracy composite; requires no model. Its weakness: it treats a slow-and-accurate child and a fast-and-sloppy child with equal net rates as identical, and fast guessing adds noise it can only average over.
Count models (Rasch Poisson counts / offset-Poisson GLMM). The classical psychometrics of speed tests (Rasch, 1960): model the count of correct responses with a log link and offset(log(time)), giving each child a latent log-rate; corpus/site enter as fixed effects, children as random effects. This is rate normalization done as a measurement model — it handles varying time windows natively and yields standard errors. It still ignores per-trial RT structure.
Diffusion IRT (van der Maas et al., 2011; Molenaar et al., 2015; diffIRT, or GLMM approximations). Each trial’s (choice, RT) is a Wiener first-passage process; person drift rate \(v\)is reading efficiency, boundary separation \(a\) is caution, \(T_{er}\) is encoding/motor time. Items get drift/boundary loadings. This cleanly separates ability from speed–accuracy tradeoff and soaks up fast guesses (they are low-drift/low-boundary behavior, not ability). Cost: heavy estimation; item-level calibration needs coverage (fine in Colombia at ~60–80 runs per item; thin in Canada at ~9).
Hierarchical speed–accuracy models (van der Linden, 2007). Separate IRT model for accuracy and lognormal model for RT, with correlated person ability θ and speed τ. Less mechanistic than the diffusion model; scales well; efficiency is a composite of θ and τ.
Signed residual time scoring (Maris & van der Maas, 2012). Score \((2x - 1)(d - t)\) per item — correct-and-fast is best, wrong-and-fast is worst. Elegant and provably optimal under a specific IRT model, but it assumes a per-item deadline \(d\); SRE has only a global deadline, so \(d\) would be an arbitrary pseudo-deadline. Not a natural fit.
Zero-filled IRT (ROAR-team suggestion)
The ROAR team has begun exploring a simple item-based approach: fill the response matrix with zeros for unreached items, then fit a standard IRT model. Speed enters the score through the block of zeros — a fast accurate reader clears more of the bank. We fit one dense Rasch model per item bank (Spanish: Bogotá + rural jointly, 1,382 runs × 376 items; German: 151 × 200; English: 91 × 530 after dropping 18 never-reached items; _fit_sre_zerofill_irt.R).
Bogotá 16-month stability: zero-filled Rasch vs corrected rate
n_pairs
rate
zerofill
181
0.72
0.72
Code
guess_share <- sre |>group_by(dataset, run_id) |>summarise(pct_fast =mean(rt_s <1), .groups ="drop")zf_first |>inner_join(guess_share, by =c("dataset", "run_id")) |>group_by(dataset_label) |>summarise(n =n(),r_rate_fastguess =cor(score_new, pct_fast),r_zerofill_fastguess =cor(zf_z, pct_fast), .groups ="drop") |> knitr::kable(digits =2,caption ="Correlation of each score with a run's fast-guess share (RT < 1 s). Positive = the score REWARDS guessing")
Correlation of each score with a run’s fast-guess share (RT < 1 s). Positive = the score REWARDS guessing
dataset_label
n
r_rate_fastguess
r_zerofill_fastguess
Colombia — Bogotá
622
-0.36
0.19
Colombia — Caquetá/Boyacá
567
-0.42
0.46
Germany — Leipzig
151
-0.10
0.10
Canada — Ontario
86
0.04
0.24
Where behavior is clean, zero-fill is essentially the rate score by another route: r = .94 with the corrected rate in Germany, near-identical validity there, and Bogotá 16-month stability a statistical tie (.72 vs .72). But it has no guessing correction: a fast guesser converts wasted trials into lucky 1s among the zeros, so the score correlates positively with a run’s fast-guess share (+.46 in rural Colombia) where the rate score correlates negatively (−.42), and validity collapses exactly there (rural PA r = .27 vs .57; rural age .19 vs .42).
The obvious repair — a fixed 2AFC guessing floor (Rasch with g = .5, as in the LEVANTE multiple-choice models) — backfires: unreached items are structural zeros, and whole-bank accuracy (~10% for a typical run) sits far below any g = .5 floor, so the model cannot represent the data (EM fails to converge; validity collapses everywhere, e.g. rural age r = .08). The floor is right for “attempted and guessed,” wrong for “never reached.” A repaired variant would need a different treatment of the two zero types — at which point the model is no longer the simple suggestion, and the rate score already does the job.
Proof of concept: EZ-diffusion per run
The EZ-diffusion model (Wagenmakers, van der Maas, & Grasman, 2007) inverts the simple diffusion model from three per-person summary statistics (accuracy, mean and variance of correct RTs) — no item structure, no iterative fitting. It is the cheapest possible test of whether diffusion-style decomposition adds validity beyond the rate score.
Code
ez_diffusion <-function(pc, vrt, mrt, s =0.1) { s2 <- s^2 L <-qlogis(pc) x <- L * (L * pc^2- L * pc + pc -0.5) / vrt v <-sign(pc -0.5) * s * x^(1/4) a <- s2 * L / v y <--v * a / s2 mdt <- (a / (2* v)) * (1-exp(y)) / (1+exp(y))tibble(drift = v, boundary = a, ter = mrt - mdt)}ez_run <- sre |>filter(rt_s >=0.25, rt_s <=30) |>group_by(dataset, dataset_label, run_id, user_id) |>filter(n() >=15, sum(correct) >=10) |>summarise(n =n(), pc_raw =mean(correct),mrt =mean(rt_s[correct]), vrt =var(rt_s[correct]),.groups ="drop") |># edge-correct accuracy away from 0.5 and 1mutate(pc =pmin(pmax(pc_raw, 0.5+1/ (2* n)), 1-1/ (2* n)))ez_run <-bind_cols(ez_run, ez_diffusion(ez_run$pc, ez_run$vrt, ez_run$mrt))ez_run |>group_by(dataset_label) |>summarise(n_runs =n(), across(c(drift, boundary, ter), median),.groups ="drop") |> knitr::kable(digits =2, caption ="Median EZ-diffusion parameters by dataset")
Bogotá 16-month stability: SA-model scores vs corrected rate
n_pairs
rate
theta
tau
composite
181
0.72
0.31
0.46
0.68
The decomposition is diagnostic gold but scoring lead. The ability–speed correlation flips sign across sites (≈ 0 in DE/Canada, −.23 Bogotá, −.52 rural CO): where fast guessing is common, the “speed” dimension is contaminated by disengagement, so neither τ nor any fixed θ+τ composite travels across sites. The composite matches the rate score where behavior is clean (r ≈ .9 in DE/Canada) and underperforms it where it isn’t (rural age validity .28 vs .42; Bogotá stability .68 vs .72).
Diffusion IRT (pooled brms Wiener model)
The full model: every trial’s (choice, RT) as a Wiener process, with dataset fixed effects plus run and item random effects on drift (the IRT part), run effects on boundary separation and non-decision time, bias fixed at .5 (~64k trials; item banks are disjoint across sites, so sites are linked by priors, not anchors). Prototyped locally with brms (_fit_sre_ddm.R), then exported to raw Stan (_export_sre_ddm_stan.R) and fit on Sherlock (sherlock_sre_ddm/; 24 cores, 9.5 h, 4 × 500/500). Drift-side parameters converged acceptably (R̂ ≤ 1.06); the run-level non-decision-time hierarchy mixed poorly (R̂ ≈ 1.6) — a known fragility of ndt under a hard global deadline — which cautions against interpreting the ndt estimates but leaves the drift comparison intact.
Bogotá 16-month stability: diffusion-IRT drift vs corrected rate
n_pairs
rate
ddm_drift
181
0.72
0.56
Code
ddm_first |>ggplot(aes(x = score_new, y = drift_z, color = dataset_label)) +facet_wrap(vars(dataset_label)) +geom_point(alpha =0.4, size =0.8, show.legend =FALSE) +scale_color_manual(values = dataset_pal) +labs(x ="Corrected rate score (z)", y ="DDM-IRT drift (z)",title ="Diffusion-IRT drift vs the corrected rate score")
The item side of the model behaves exactly as an IRT model should — item drift effects track empirical item accuracy and (negatively) item RT:
Code
item_emp <- sre |>group_by(item_uid) |>summarise(accuracy =mean(correct), median_rt =median(rt_s), n =n(),.groups ="drop") |>filter(n >=20)ddm$item |>inner_join(item_emp, by ="item_uid") |>ggplot(aes(x = drift_re, y = accuracy)) +geom_point(alpha =0.3, size =0.8) +geom_smooth(method ="lm", color ="black") +labs(x ="Item drift effect (logit-ish)", y ="Empirical item accuracy",title ="The IRT part works: item drift effects track item difficulty")
So the item-level structure that EZ lacked is genuinely recovered here — and person-level drift still underperforms the rate score where they differ. That localizes the problem: it is not missing item information, it is that the diffusion decomposition discards stable, valid between-child variance in time allocation (caution, pacing) that the rate score keeps.
Verdict
1. The timer is mostly honest, but the trial-count rule was aimed at the wrong problem. The first-180-trials filter is a no-op in Germany and Colombia (max 140 trials/run) and bites only in Canada. The real heterogeneity is in time: 20 Canadian runs contain two sequential ~180-s blocks (~360 s of testing scored as one window), and there is a left tail of short runs everywhere (rural Colombia 5th percentile ≈ 87 s of reading time) that the current rule scores as if they had the full window.
2. Rate normalization changes little in aggregate, a lot for specific kids. Corrected-vs-current r ≈ .997 (DE), .975–.978 (CO), but .82 in Canada (mean |Δ| = 0.45 z there; the double-block runs drop, short runs rise). Validity correlations with age/PA/SWR move by |Δr| ≤ .03 everywhere except Canadian age validity, which improves (.32 → .39); Bogotá 16-month stability ticks up (.70 → .72). So the correction is about fairness of individual scores (especially any longitudinal use of the Canadian data), not about rescuing aggregate structure — which was already fine.
3. Efficiency modeling: a four-rung ladder, and the rate score wins every rung. Every model-based alternative is less valid and/or less stable than the corrected rate score where they differ. The scoreboard below compares all methods on every available metric.
Scoreboard: every method × every metric
Validity columns are n-weighted means of within-dataset correlations (criteria from the hackathon scores: PA/SWR/Vocab/TROG θ, first run per child). Reliability uses the best available estimator per method: odd/even-trial split-half with Spearman–Brown for the trial-computable scores (rate, EZ), model-based empirical reliability for the latent-score methods (zero-fill: mirt EAP rxx; SA/DDM: var(θ̂)/(var(θ̂)+mean(SE²))). Germany test–retest uses levante_data_latest v1_2 (the hackathon subset has no DE repeats): n = 23 pairs at ~3.6 months, rescored per method (DDM would need a full refit — left blank).
method_labels <-c(rate ="Corrected rate (C − I per 180 s)",zerofill ="Zero-filled Rasch (ROAR)",sa_composite ="Hierarchical SA composite",ddm_drift ="Diffusion-IRT drift",ez_drift ="EZ-diffusion drift")scoreboard <- validity |>left_join(reliability, by ="method") |>left_join(retest_de, by ="method") |>left_join(stability, by ="method") |>mutate(method =factor(method, levels =names(method_labels),labels = method_labels)) |>arrange(method)knitr::kable(scoreboard, digits =2,col.names =c("Method", "Age", "PA", "SWR", "Vocab", "TROG","Reliability", "(estimator)","DE retest (3.6 mo, n=23)","Bogotá stability (16 mo, n=181)"),caption ="Scoring-method scoreboard. Validity = n-weighted mean within-dataset r; all scores z within dataset.")
Scoring-method scoreboard. Validity = n-weighted mean within-dataset r; all scores z within dataset.
Method
Age
PA
SWR
Vocab
TROG
Reliability
(estimator)
DE retest (3.6 mo, n=23)
Bogotá stability (16 mo, n=181)
Corrected rate (C − I per 180 s)
0.50
0.48
0.57
0.40
0.44
0.83
split-half (SB)
0.83
0.72
Zero-filled Rasch (ROAR)
0.36
0.31
0.52
0.32
0.20
0.88
EAP rxx
0.78
0.72
Hierarchical SA composite
0.42
0.39
0.55
0.46
0.33
0.67
model rxx
0.69
0.68
Diffusion-IRT drift
0.47
0.48
0.49
0.44
0.50
0.87
model rxx
NA
0.56
EZ-diffusion drift
0.42
0.45
0.46
0.40
0.47
0.84
split-half (SB)
0.23
0.44
Reading the scoreboard:
The corrected rate wins everything that matters for a primary score. Best or tied on age (.50), PA (.48), SWR (.57), best on both repeat-administration metrics (DE 3.6-month retest .83; Bogotá 16-month stability .72), with an honest split-half reliability of .83.
One real nuance: speed-purged scores track untimed comprehension slightly better. Diffusion-IRT drift has the best TROG correlation (.50 vs .44 for rate) and drift/SA edge the rate score on Vocab — while the rate score wins on the reading-efficiency-adjacent criteria (SWR, PA, age). That is a sensible division of labor: drift isolates processing efficiency (closer to language knowledge), while the rate score’s pacing component is signal for reading fluency specifically. It validates drift as a research variable without changing the scoring recommendation.
Zero-fill’s high nominal reliability (.88) is partly an artifact: the deterministic block of zeros is “reliable” in the rxx sense without being valid — it posts the worst validity column of any method on TROG (.20) and PA (.31). Reliability estimators also aren’t directly comparable across rows (split-half vs model-based).
EZ-diffusion’s DE retest of .23 (vs .83 for the rate score on the same 23 children) is the cleanest single demonstration that inverting three summary statistics per run does not produce a usable score.
Item structure is not the missing ingredient: the full diffusion-IRT fit recovers sensible item parameters (item drift effects correlate .52 with item accuracy, −.69 with item median RT) and item information lifts drift stability from .44 (EZ) to .56 — but still far short of .72. Validity tells the same story (e.g. Bogotá age r: rate .60, DDM drift .53; DE age: .54 vs .44), with only scattered small wins for model-based scores (DE SWR: DDM .62 vs rate .58). The diagnosis: SRE’s net rate is the behaviorally meaningful quantity — it bundles decision efficiency and stable individual differences in pacing/caution, and every decomposition that strips the latter out throws away true-score variance. The site-dependent ability–speed correlation (≈ 0 in DE/Canada, −.52 in rural CO, where fast = guessing) additionally means no fixed recombination of decomposed parts travels across sites.
Recommendations for the DCC:
Adopt rate normalization now (patch below). It fixes the Canadian double-block and short-run distortions at zero modeling cost and is what the ROAR “net correct in 3 minutes” score means.
If a measurement model is wanted, the next step is an offset-Poisson / Rasch-Poisson counts GLMM (net-correct count with offset(log(time)), corpus fixed effects, child random effects) — it is rate normalization with standard errors and corpus adjustment, not a new scoring philosophy.
Diffusion IRT is now tested, not just argued against: don’t use it for the primary SRE score. The full hierarchical Wiener fit (9.5 h on Sherlock) yields drift scores that are less stable (.56 vs .72) and mostly less valid than the rate score. The decomposition (drift vs boundary vs ndt) remains available in data/sre_ddm_estimates.rds for substantive questions about speed–accuracy tradeoff and its development — boundary separation declines with age in every dataset — but that is a research variable, not a score.
Use RT for data quality regardless: sub-second responses sit at chance and range from 3% of trials (DE) to 31% (rural CO). A fast-guess flag (e.g. proportion of RT < 1 s) is a cheap, useful run-level QC covariate.
