Adaptive Placement Testing and Computerized Adaptive Testing (CAT): IRT, Cold-Start Calibration, and Knowledge Spaces

Findings

1. IRT basics — three nested models, one realistic for cold start

p(θ) = c + (1-c)·σ(a(θ-d)) gives correctness probability from ability θ and item parameters difficulty d, discrimination a, guessing c [1][4]. Fixing c=0, a=1 gives 1PL/Rasch — difficulty is the only free item parameter, and its sufficient-statistic property lets item difficulty be estimated by conditional MLE without knowing ability first [2]. 2PL frees a: higher discrimination means an item separates just-below- from just-above-difficulty test-takers more cleanly [1]. 3PL adds guessing c, appropriate for multiple-choice items (≈0.25 floor on four options) [1]. Both need substantially more per-item data than Rasch — the literature default is “hundreds” of responses [1][4].

2. CAT mechanics — selection, estimation, stopping, exposure

The loop: estimate θ from responses so far; pick the unused item maximizing Fisher information at that θ; administer; update; repeat [3]. MLE is asymptotically unbiased but undefined for perfect/zero response patterns; Bayesian EAP/MAP resolves this via a prior at the cost of slight bias [3]. Stopping rules: standard-error threshold (variable length), fixed item count, or classification-confidence threshold for pass/fail decisions [3]. Sympson-Hetter draws a random number per candidate item against an item-specific exposure parameter to probabilistically block even the best item [3][12]. Randomesque selects uniformly from the top 5-10 most informative items [3]. Comparative studies (Ozturk & Dogan 2015; Leroux et al. 2013, 2016) test these against newer “progressive-restricted standard error” methods on 3PL/GPC models, generally finding precision/exposure trade-offs rather than a clear winner [12][13][14]. Duolingo’s 2026 S2A3 paper replaces blocking with Thompson Sampling, treating exposure limits as stochastic constraints inside item selection [5]. Content balancing is the orthogonal constraint that selection must also hit a target content mix, not just maximize information [3].

3. The cold-start problem, named and solved by a live production system

Duolingo’s AutoIRT paper names exactly Math Challenge’s situation: cold-start (new item, no responses, calibrated from content features only), jump-start (small pilot sample blended with the operational bank), warm-start (recalibration as population/UI/prep materials shift) [4]. Their solution: an AutoML ensemble (random forests, LightGBM, XGBoost, CatBoost) trained on item content features, projected onto interpretable IRT parameters via Monte Carlo EM, avoiding the need for hundreds of live responses before an item is usable [4]. This establishes that provisional, feature/expert-derived difficulty is a professional starting point with a real upgrade path, not a shortcut.

The alternative compatible with a small team is Elo-based online calibration: both AutoIRT and the wider literature note Elo as a long-standing online procedure for the Rasch model, updating person and item ratings after every response with no offline fitting step [4][9]. A 2019 EDM paper (Abdi, Khosravi, Sadiq & Gasevic) extends single-concept Elo to a multivariate form for multi-tagged items, reporting improved predictive accuracy [10]. Topic 13 of this project’s own research already recommends Elo/Math Garden’s HSHS rule as the concrete precedent: O(1) updates (good fit for Durable Objects), validated in children’s arithmetic [9]. A third, later-stage path: BOBCAT (Ghosh & Lan, 2021) frames item selection itself as bilevel optimization rather than pure Fisher information — a plausible v2/v3 upgrade once enough logged data exists to train it [11].

4. ALEKS and Knowledge Space Theory — a graph, not a scale

ALEKS originated at UC Irvine in 1994 (NSF-funded), acquired by McGraw-Hill in 2013 [8]. Knowledge Space Theory (Doignon & Falmagne) models a domain as concept set Q; a learner’s state is one feasible subset, constrained by prerequisites, not every subset of Q [7]. This defines a partial order over feasible states. The outer fringe is what the learner is ready to learn next (prerequisites satisfied); the inner fringe is what was just acquired [7]. ALEKS’s assessment narrows down which feasible state a learner occupies, then recommends the outer fringe — a personalized path through a prerequisite graph, not a percentile score [7][8]. This is the reference design for a skill-tree UI, with IRT/Elo handling within-node difficulty ordering.

5. Duolingo English Test — the most concretely documented CAT in ed-tech

In the DET version studied (V8), each session runs 18 Yes/No-Vocabulary items (5 seconds each, judging real vs. algorithmically-generated fake words) and 9 Vocab-in-Context items (20 seconds each, fill-in-the-blank) — 27 items for two of 14 total task types, drawn from pools of 3,290 and 585 items respectively [4]. θ sits on a continuous Normal(0,1)-prior scale, with the reported score as the posterior mean using the full posterior during calibration, not a point estimate [4]. Duolingo has also published specifically on Responsible AI for this test, framing quality/equity as an ongoing validity-argument chain (domain definition → evaluation → generalization → explanation → extrapolation → utilization) — a useful checklist shape even outside high-stakes admissions testing [6].

6. NWEA, i-Ready, Khan Academy, IXL — thinner public detail

NWEA’s MAP Growth reports on a proprietary RIT scale (“Rasch unIT”), grade-independent and continuous; NWEA’s research center publishes item-parameter-drift and validation studies presupposing an IRT/Rasch foundation, but the exact item-selection algorithm was not found on public pages reachable in this research [15]. Khan Academy’s own materials describe mastery learning as the pedagogical model, but its “Course challenge” placement mechanics are not detailed in openly accessible pages [16]. IXL’s product page states plainly that “the difficulty of the questions adapts automatically,” confirming CAT-like behavior, without publishing scale, item count, or selection rule [17]. i-Ready’s technical manual was not retrievable in this session; no specific claim is made about it beyond its existence as an adaptive diagnostic. This gap is itself a finding: the Duolingo and ALEKS/academic literatures are the usable blueprints for v1; the K-12 diagnostic vendors treat internals as trade secrets.

7. How many items, and per-topic vs. global

No universal minimum exists, but two data points bound the range. Fixed-length high-stakes CATs commonly run to tens of items for a tight standard-error target [3]. A 2024 study chaining short CATs for the Force Concept Inventory found repeated 5-item adaptive administrations (9 times across a semester) reached efficiency “comparable to that of the pre-post method” for tracking change — context-specific (repeated formative measurement, not one-shot placement), but evidence that short-and-repeated can substitute for long-and-single [18]. This favors per-topic placement (10-15 items per branch, matching the project brief) over one long global test: a child can place at “3rd-grade arithmetic” and “kindergarten geometry” simultaneously, which a single global score cannot represent but a per-topic CAT and ALEKS’s knowledge-space model both can [7].

8. Not feeling like a test to a 6-year-old

No source reviewed here addresses child UX directly, but two structural facts translate into constraints. Because CAT targets each item near the learner’s actual ability by construction, a well-implemented adaptive placement naturally produces a mixed-success experience rather than a wall of failure — the mechanism itself protects the feeling, provided the UI doesn’t editorialize on top of it (no visible countdowns or wrong-answer buzzers) [3]. Because short, per-topic placement (§7) is both defensible and better suited to a child’s attention span, it can be delivered as a sequence of short themed mini-games rather than one continuous exam — building on the already-child-validated Math Garden precedent from topic 13 rather than re-deriving tone here [9].

Design implications

1. Use Rasch-style (1PL) provisional difficulty for v1, not 2PL/3PL. With zero response history, only a single difficulty parameter per item is realistic; discrimination/guessing need data v1 won’t have [1][2][4].
2. Algorithm: expert-tagged difficulty + closest-difficulty selection. Tag every item 1-100 by hand. At each step, select the unused item whose tag is closest to the current ability estimate — a parameter-free stand-in for maximum-Fisher-information selection [3][4].
3. Ability update: Elo, not MLE/EAP. ability += K * (outcome - expected), expected a logistic function of (ability − item difficulty), matching the Rasch response form; O(1) per response, fits a Durable Object or a D1 write per turn, consistent with the topic-13 precedent [1][9].
4. Decreasing K within one session. Large K for the first 3-4 items (fast convergence from the age-seeded guess), smaller K after (stability) — the within-session analogue of chess Elo’s provisional-rating period [9].
5. Stopping rule: hard cap of 15 items, early-stop from item 8 on stability. Stop early if the last 4 responses have alternated around the same tier (±1) with no net drift — a proxy for “SE is small enough” without a calibrated model to compute SE from [3].
6. Per-topic placement, not one global score, matching ALEKS’s many-states model and the project’s own skill tree [7][8].
7. Age seeds item 1 only; ability estimate governs everything after. The age input is a prior, not a ceiling or floor — 2-3 answers should be able to move the estimate a full tier.
8. Log every response (item id, tagged difficulty, outcome) from day one. This is exactly the “jump-start” data Duolingo’s team requires before any recalibration; without it from launch, implication 9 starts late [4].
9. Upgrade path: batch-refit tagged difficulties into an actual Rasch model at ~200-400 responses/item, blending the expert prior with the empirical estimate rather than discarding it, since per-item volume will be uneven [4].
10. 2PL only after Rasch is stable and volume is high (“hundreds” of responses per the literature); skip 3PL guessing parameters entirely unless the format is fixed-option multiple choice, since guessing isn’t identifiable otherwise [1][4].
11. Exposure control only once traffic outpaces the item pool. In v1, just avoid repeating an item within one session; add Sympson-Hetter-style blocking or randomesque top-N only when telemetry shows a handful of items dominating selection [3][12].
12. UX per age band: ages ~4-6 — single character-guided mini-game, no visible score/timer/“test” language, celebratory feedback regardless of correctness, ending in a narrative transition, not a results screen. Ages ~7-11 — a “warm-up quest” with an item-count progress bar (never a correctness bar), light narrative, still no numeric score shown. Ages ~12-17 and adult/expert — transparent framing (“so we can start you at the right level”) is fine and often preferred, but still avoid “assessment of your ability” language; a well-targeted adaptive test is genuinely closer to guided practice than exam-taking once it’s converging correctly, which supports this framing at every age [3].
13. Age is a theme and a seed, never a hard placement limit — the entire point of an adaptive test per the brief is that ability, not age, sets the level.

Open questions for the project owner

1. Mandatory placement before any practice, or optional with an age-default fallback and a later “recalibrate” action?
2. What K schedule for the decreasing-K Elo update — fixed (e.g., 1.0 / 0.5 / 0.25) or tuned empirically post-launch from logged data?
3. Single-author item-difficulty tagging, or a lightweight multi-rater process (2-3 people, reconcile disagreements) before any items ship?
4. Should every topic share a 15-item cap, or should broad branches (e.g., all-of-arithmetic) get a longer cap than narrow ones (e.g., long division)?
5. When placement disagrees sharply with stated age (a 5-year-old placing at 3rd grade), show it plainly, soften it, or ask the parent to confirm first?
6. Is periodic re-placement (every N weeks, or after M wrong answers at the current tier) a v1 feature, or a later addition once the core loop is validated?
7. Given how thin NWEA/i-Ready/IXL/Khan Academy’s public documentation is, is there appetite to pursue their technical manuals under research-data agreements, or is the Duolingo/ALEKS/academic basis here sufficient for now?