Pattern Matching & Sentence Understanding — training vs development?

ImportantAI-generated — not for publication

The code and prose in this chapter were written by Claude Code (an AI coding agent) under human direction. They have not been line-by-line verified by a human and must not be published or cited as-is.

WarningPreliminary & unvetted analysis

These are exploratory analyses of preliminary, in-progress LEVANTE data. Results have not been reviewed or validated and require checking by the LEVANTE Data Coordinating Center before any use.

Question

The v1.0 release notes flagged Pattern Matching (matrix reasoning) and Sentence Understanding (TROG) as showing systematic T2 > T1 within both modes — possibly a retest/practice (training) effect rather than development. We revisit this on corrected (v1.2) data using three discriminators:

  1. Observed gain vs. expected development = within-child T2−T1 vs. (cross-sectional age slope × retest interval). Excess beyond development ⇒ practice.
  2. Retest interval (Leipzig has ~3-month and ~7-month groups): a fixed practice bump is interval-independent; development scales with interval.
  3. Repeated items: if accuracy jumps more on items seen at both waves than on new items, that’s item-specific practice/memory.

Two confounds to watch: most longitudinal pairs cross the CAT/non-CAT mode boundary, and test length can differ across waves (shorter ⇒ more EAP shrinkage).

Code 
source(here::here("common.R"))
sc <- read_rds(here("data/scores_all_sites.rds"))
tasks <- c("matrix-reasoning","trog")
w <- sc |> filter(task_id %in% tasks,
                  dataset %in% c("pilot_mpieva_de_main","pilot_uniandes_co_bogota"),
                  !is.na(score), !is.na(age)) |>
  group_by(dataset, task_id, user_id) |> arrange(age) |>
  mutate(wave = row_number()) |> ungroup() |> filter(wave <= 2)
pairs <- w |> group_by(dataset_label, task_label, user_id) |>
  filter(n_distinct(wave) == 2) |>
  summarise(t1 = score[wave==1], t2 = score[wave==2],
            gap = age[wave==2]-age[wave==1],
            a1 = adaptive[wave==1], a2 = adaptive[wave==2],
            na1 = num_attempted[wave==1], na2 = num_attempted[wave==2],
            .groups = "drop") |>
  mutate(gain = t2 - t1, mode = if_else(a1==a2, "mode-stable", "NA→CAT"))

1. Within-child gain vs expected development

Code 
xsec <- w |> filter(wave==1) |> group_by(dataset_label, task_label) |>
  summarise(xsec_slope = coef(lm(score ~ age))[2], .groups="drop")
pairs |> group_by(dataset_label, task_label) |>
  summarise(n=n(), mean_gain=mean(gain), mean_gap=mean(gap), .groups="drop") |>
  left_join(xsec, by=c("dataset_label","task_label")) |>
  mutate(expected_dev = xsec_slope*mean_gap, excess = mean_gain - expected_dev) |>
  select(dataset_label, task_label, n, mean_gain, xsec_slope, mean_gap,
         expected_dev, excess) |>
  knitr::kable(digits = 2,
               caption = "Observed gain vs cross-sectional expectation (excess > 0 ⇒ practice-like)")
Observed gain vs cross-sectional expectation (excess > 0 ⇒ practice-like)
dataset_label task_label n mean_gain xsec_slope mean_gap expected_dev excess
Colombia — Bogotá Pattern Matching 52 0.33 0.42 1.33 0.55 -0.23
Colombia — Bogotá Sentence Understanding 80 0.83 0.32 1.33 0.43 0.40
Germany — Leipzig Pattern Matching 191 -0.37 0.42 0.36 0.15 -0.52
Germany — Leipzig Sentence Understanding 139 -0.11 0.14 0.34 0.05 -0.16

The dramatic T2 > T1 from the release notes does not replicate on corrected data. Leipzig now shows small declines (Pattern Matching −0.37, Sentence Understanding −0.11); Bogotá shows gains (PM +0.33, below its developmental expectation +0.55; SU +0.83, above its expectation +0.43). So the only “excess” gain is Bogotá Sentence Understanding (+0.40), and only there.

2. The confounds: mode and test length

Code 
pairs |> group_by(dataset_label, task_label) |>
  summarise(n=n(),
            pct_mode_stable = round(mean(mode=="mode-stable")*100),
            med_items_t1 = median(na1, na.rm=TRUE),
            med_items_t2 = median(na2, na.rm=TRUE),
            .groups="drop") |>
  knitr::kable(caption = "Mode transitions and test length per wave")
Mode transitions and test length per wave
dataset_label task_label n pct_mode_stable med_items_t1 med_items_t2
Colombia — Bogotá Pattern Matching 52 0 NA 33.5
Colombia — Bogotá Sentence Understanding 80 0 NA 41.0
Germany — Leipzig Pattern Matching 191 100 27 28.0
Germany — Leipzig Sentence Understanding 139 28 90 41.0
  • Bogotá PM & SU are 100% NA→CAT — every pair crosses the mode boundary, so their changes mix development with the mode transition.
  • Leipzig Sentence Understanding is mostly NA→CAT and halves in length at T2 (≈ 86 → 41 items) ⇒ heavier EAP shrinkage; its small change is not interpretable as practice.
  • Leipzig Pattern Matching is the one clean cell: mostly mode-stable (155 NA→NA + 36 A→A) with equal test length (27 vs 28 items) — so its decline is a real signal, not an artifact.

3. Clean test — Leipzig Pattern Matching (mode-stable, equal length)

Code 
lpm <- pairs |> filter(str_detect(dataset_label,"Leipzig"),
                       task_label=="Pattern Matching", mode=="mode-stable") |>
  mutate(interval = if_else(gap < 0.42, "~3 months", "~7 months"))
lpm |> group_by(interval) |>
  summarise(n=n(), mean_gap=round(mean(gap),2), mean_gain=round(mean(gain),2),
            se=round(sd(gain)/sqrt(n()),2), .groups="drop") |>
  knitr::kable(caption = "Leipzig PM mode-stable gain by retest interval")
Leipzig PM mode-stable gain by retest interval
interval n mean_gap mean_gain se
~3 months 143 0.27 -0.28 0.13
~7 months 48 0.61 -0.64 0.25

There is no practice bump — the change is slightly negative, and (within noise) does not grow into a positive gain at either interval. A genuine retest/practice effect would show a positive jump even over a short interval; we see the opposite. The small decline is most consistent with regression to the mean plus the modest test-retest reliability of matrix reasoning, not training.

4. Repeated-item check

If practice were memory for specific items, accuracy should rise more on items seen at both waves. Feasible only where items repeat.

Code 
tr <- load_levante_trials(task_ids = tasks) |>
  filter(dataset %in% c("pilot_mpieva_de_main","pilot_uniandes_co_bogota")) |>
  mutate(correct = as.numeric(correct))
rw <- w |> select(run_id, wave)
itw <- tr |> inner_join(rw, by="run_id") |>
  group_by(task_id, user_id, item_uid) |>
  filter(n_distinct(wave) >= 1) |>
  summarise(seen_both = n_distinct(wave)==2,
            acc_t1 = mean(correct[wave==1], na.rm=TRUE),
            acc_t2 = mean(correct[wave==2], na.rm=TRUE), .groups="drop")
# overlap rates
tr |> inner_join(rw, by="run_id") |>
  distinct(task_id, user_id, wave, item_uid) |>
  group_by(task_id, user_id, item_uid) |> summarise(nw=n_distinct(wave), .groups="drop") |>
  group_by(task_id) |> summarise(pct_items_repeated = round(mean(nw==2)*100,1), .groups="drop") |>
  knitr::kable(caption = "Share of items seen at both waves")
Share of items seen at both waves
task_id pct_items_repeated
matrix-reasoning 4.3
trog 18.7
Code 
# repeated-item accuracy change (trog only — matrix overlap ~4%)
itw |> filter(task_id=="trog", seen_both, !is.nan(acc_t1), !is.nan(acc_t2)) |>
  summarise(n_repeated_item_obs = n(),
            acc_t1 = round(mean(acc_t1),3), acc_t2 = round(mean(acc_t2),3),
            change = round(mean(acc_t2-acc_t1),3)) |>
  knitr::kable(caption = "TROG: accuracy on items seen at BOTH waves")
TROG: accuracy on items seen at BOTH waves
n_repeated_item_obs acc_t1 acc_t2 change
7431 0.873 0.887 0.014

Pattern Matching repeats only ~4% of items (Mars-IB uses parallel retest forms), so memorization is structurally unlikely. TROG repeats ~19%; on those repeated items accuracy does rise modestly — but a few-point bump on a fifth of items can’t explain a large overall effect, and the overall TROG change is confounded by mode/length anyway.

5. Real development is present (cross-sectional)

Code 
sc |> filter(task_id %in% tasks,
             dataset %in% c("pilot_mpieva_de_main","pilot_uniandes_co_bogota"),
             !is.na(score)) |>
  group_by(dataset, task_id, user_id) |> arrange(age) |> slice(1) |> ungroup() |>
  ggplot(aes(x = age, y = score, color = dataset_label)) +
  geom_point(alpha = 0.2, size = 0.7) +
  geom_smooth(method = "lm", se = TRUE) +
  facet_wrap(~task_label) +
  scale_color_brewer(palette = "Set1", name = NULL) +
  labs(x = "Age (years)", y = "Ability (θ)",
       title = "Both tasks show clear cross-sectional development",
       subtitle = "Older children score higher — the age signal is real") +
  theme(legend.position = "bottom")

Verdict

  • The release-notes “T2 > T1 training effect” was largely the scoring bug. On corrected data it does not replicate: Pattern Matching is flat-to-slightly declining, and only Bogotá Sentence Understanding shows gain beyond cross-sectional expectation.
  • The one clean longitudinal cell — Leipzig Pattern Matching (mode-stable, equal length) — shows no practice gain, arguing against a strong retest/training effect for matrix reasoning.
  • It can’t be fully resolved for these tasks with current data: most pairs cross the CAT/non-CAT boundary, test length changes across waves (EAP shrinkage), and item overlap is low. The honest read is “no robust training effect on corrected data; remaining signal is confounded.”
  • To answer it cleanly: a fixed-form, fixed-length retest (same items, same number, no mode change) would isolate practice from development. Worth flagging to the DCC.
Code 
write_rds(pairs, here::here("data", "pm_su_pairs.rds"))