Behavioral/Process Data in Learning Systems: What Hesitation and Timing Mean, and Where Child Privacy Law Draws the Line

Findings — Part 1: What the signals mean (learning science)

1. Answer-changing helps far more than folk belief predicts. Kruger, Wirtz & Miller’s “first instinct fallacy” study (JPSP, 2005) reviewed 70+ years of prior research and found that in not one of 33 studies were test-takers hurt, on average, by changing answers — despite 55% of surveyed instructors advising otherwise. Their own eraser-mark study of 1,561 exams found 51% of changes went wrong-to-right, 25% right-to-wrong, 23% wrong-to-wrong; of 1,231 students who changed at least one answer, 54% were net helped, 19% net hurt, 27% neither [1]. The mismatched belief traces to counterfactual regret: abandoning a correct answer produces a uniquely painful “if only” memory, making rare bad outcomes feel common [1]. An answer-change or erasure is not, by itself, evidence of poor understanding.

2. Rapid, near-zero-latency answers are the real red flag — for disengagement, not gaming. Wise & Kong’s “response time effort” (RTE) measures the proportion of items answered above a solution-behavior time threshold. A cited meta-analysis found RTE strongly associated with performance, more so than self-reported effort. Applied to PISA 2015, filtering rapid guessers raised country mean scores but barely moved rankings [4]. Thresholds range from a fixed cutoff (e.g., 5 seconds) to item-specific norms (e.g., 15% of typical solution time) [4]. The correct use is a floor (“too fast to reflect engagement”), not a graded penalty across the whole latency range.

3. “Gaming the system” is detectable, not monolithic, and detectors get gamed back. Baker, Corbett, Koedinger & Roll define gaming as exploiting system regularities rather than thinking about the material — systematic guessing, hammering help until the system answers [2][3]. Their model distinguishes GAMED-HURT students from GAMED-NOT-HURT students who game without learning loss. Best-separating features: several quick actions in a row, high error rates on multiple-choice steps, quick numeric answers, brief help requests or errors in rapid succession [3]. The detector transferred across student cohorts (A’=0.76–0.92) but poorly across tutor-lesson interfaces without retraining — gaming signatures are interface-specific [3]. The authors warn: a static, guessable detector invites students to “learn how to game the gaming detector” [3]. A direct warning against a fixed “erratic behavior = cheating” scoring rule.

4. “Wheel-spinning” describes a student stuck, not misbehaving. Beck & Gong’s AIED 2013 paper names a mastery-learning-tutor phenomenon: a student working on a skill far longer than the model expects mastery to take, without reaching it — burning time and motivation for no gain. Follow-on ACM work pursued early detection so a system could intervene before the student gives up. Sustained low performance despite many attempts is a different signal from momentary hesitation, calling for a change of strategy, not a lower score.

5. Confusion, boredom and frustration have characteristic transition patterns. D’Mello & Graesser’s widely-cited affect-dynamics work found boredom is the “stickiest” state — hard to escape once entered — while confusion is a pivot that resolves productively (toward re-engagement) or unproductively (toward frustration, then boredom) depending on whether the environment scaffolds it. These states were inferred from interaction patterns (latency, errors, help-seeking), not sensors. The actionable target is interrupting the boredom/frustration spiral early via short aggregate windows of struggle, not single-event flags.

6. PISA and NAEP now treat process data as a research asset, with real limits. NAEP’s process-data program publishes time-stamped interaction records — time on task, navigation/revisit counts, tool use, action sequences — and invites researchers to derive their own behavioral variables rather than asserting fixed interpretations [5], itself a sign of how unsettled raw-process-data interpretation still is. PISA’s rapid-guessing/RTE program (finding 2) is the most mature applied result of this line of work.

7. Fluency and automaticity show up as distribution shape, not single-trial speed. Cognitive-arithmetic research treats automaticity as a shift in the whole response-time distribution: fast and low-variance (memory retrieval) versus slow and high-variance (counting/derived strategies), with an occasional long right tail even in fluent responders. Fluency should never be scored from one trial: only a stable pattern across many trials distinguishes “not yet automatic” from “one slow day.”

8. Keystroke dynamics is mature — built for authentication, not classroom scoring. The field standardizes dwell time (key-hold duration) and flight time (release-to-next-press interval), plus digraph timing and error-correction patterns, used for continuous authentication with confidence thresholds rather than pass/fail [10]. The domain exists to identify who is typing; Math Challenge’s interest is closer to how an already-known child types — lower-risk, but it borrows the exact feature vocabulary a regulator will use to ask “is this biometric data?”

Findings — Part 2: Where privacy law draws the line

9. Under GDPR, biometric data is only reinforced (“special category,” Art. 9) when used for unique identification. Art. 4(14) defines biometric data as personal data from technical processing of physical, physiological or behavioural characteristics that allow unique identification; the Art. 9 trigger requires that identification purpose specifically [7][11]. Keystroke dynamics fits the behavioral-characteristics half of the definition. The determining fact is purpose: logging latency/erasure counts to adapt difficulty is not, by that use alone, special-category processing — but fingerprinting which child is typing from rhythm would cross the line immediately. The EU AI Act’s Art. 3(34) drops the identification requirement, so its broader risk-tiered rules can apply regardless of intent [11] — a second, independent reason to avoid any per-child typing-rhythm profile.

10. No confirmed BIPA case addresses keystroke dynamics directly. Illinois’ BIPA names fingerprints, retina/iris scans, voiceprints, and hand/face geometry as its core examples; its heavy class-action litigation history has centered on those modalities. This is an open legal question, not settled law — flag to counsel rather than assume either way.

11. The DSA’s Article 28 bans profiling-based ad targeting of minors, and separately rejects “collect more to comply.” Art. 28(2) prohibits ads based on profiling of users known to be minors; Art. 28(1) requires “appropriate and proportionate measures” for privacy/safety; Art. 28(3) states there is no obligation to process additional data just to assess whether a user is a minor [6]. For a product already declared for children from age 4, ad-targeting profiling is a non-issue, but the “no extra collection to prove age” principle generalizes: don’t collect one data type merely to justify another.

12. The UK’s AADC sets the most concrete standard: profiling off by default, strict minimization, no dark patterns. It instructs services to “prevent profiling… unless strictly necessary, and switch it off by default,” “collect and keep only what’s absolutely needed,” and avoid “dark patterns… that encourage children to share more than they need to” [8]. ICO guidance separately discourages profiling children “wherever possible,” since children “may be more easily influenced than adults” [9]. Any adaptive/scoring use of behavioral signals should be a narrow, disclosed exception to a profiling-off default.

13. Data minimization for telemetry has an established shape: aggregate on-device, store derived features, discard raw streams. Compute a small set of derived, non-reversible features on-device — time to first keystroke, edits before submit, answer-changed (bool), response-time z-score vs. the child’s own baseline — and transmit only those. NAEP’s own process-data program publishes derived action-sequence variables, not raw input-device signal — a precedent for responsible process data at the assessment-industry state of the art [5].

Signal catalogue table

Signal	What it indicates	How to compute it	Privacy risk	Raw or derived only
Total response latency	Fluency, only in aggregate (finding 7)	Render vs. submit timestamp	Low	Derived: z-scored vs. child’s own baseline
Time-to-first-keystroke	Retrieval latency vs. “still reading”	Render vs. first input timestamp	Low	Derived duration only
Rapid-guess flag	Disengagement, per PISA RTE (finding 2)	Latency vs. item-calibrated floor [4]	Low	Derived boolean
Answer changed before submit (count)	Usually self-correction, net-positive (finding 1)	Count of final-answer edits	Low	Derived count, not edit content
Erasure/backspace count	Weak alone; conflates revision with anxiety	Count of delete events	Medium if paired with per-key timing	Derived count only — no per-key timing
Full keystroke timing (dwell/flight per key)	The keystroke-dynamics feature set (finding 8)	Per-key press/release timestamps	High — behavioral-biometric definition applies [11]	Never store raw; compute aggregates, discard
Help/hint requests in quick succession	Possible help-abuse gaming (finding 3), aggregate only	Count + interval per session/week	Low	Derived aggregate rate
Repeated attempts without mastery	Possible wheel-spinning (finding 4) — intervention signal	Attempts vs. model expectation	Low	Derived count
RT distribution shape, many trials same fact type	True fluency signal (finding 7), never single-trial	Mean + variance per fact family	Low	Derived distribution summary
Quick low-effort sequences across items	Aggregate gaming pattern (finding 3)	Rolling window over last N items	Low–medium if retained long	Derived rolling score, short retention
Navigation/revisit pattern	Engagement signal, per NAEP (finding 6)	Navigation event count	Low	Derived count

Design implications

1. Never score latency or erasure count directly as a penalty. Findings 1 and 7 show slow, corrected, or edited responses are frequently the better cognitive behavior — subtracting points for “took a long time” or “changed the answer” is pedagogically backwards and contradicted by 70+ years of testing research.
2. The only defensible use of a hard latency floor is a disengagement filter, not a graded penalty. Following PISA’s RTE model (finding 2), flag “too fast to have engaged” as a binary floor calibrated per problem type — not a sliding scale that keeps penalizing as speed decreases toward normal.
3. Compute fluency from many trials of the same fact family, never a single trial. Automaticity is (low mean AND low variance) across repeated exposures (finding 7); a single slow-but-correct answer should never lower a fluency score, only withhold it pending more data.
4. Treat “answer changed before submit” as neutral-to-positive, never a red flag. Log it as a count only (finding 1); if it enters scoring at all, it should credit self-monitoring, not dock points.
5. Reserve gaming-style detection for cross-session aggregates, never single-event flags, and never expose exact thresholds in the UI. Baker et al.’s own warning (finding 3) is that a visible, static detector gets learned around; if built, it should trigger a hint-mode intervention, not a score penalty, and its thresholds should be revisited periodically.
6. Keep “stuck” (wheel-spinning), “disengaged” (gaming pattern), and “mastered” (fluent-and-fast) as three separate states with three separate responses, not one composite behavior score: stuck → change strategy/escalate; disengaged → adjust incentives, don’t just penalize; fluent → advance difficulty.
7. Log at the level of derived features computed on-device, never raw interaction streams. Per finding 13, dwell/flight-time-per-keystroke should never leave the device or persist beyond the single aggregate computation; synced data is bounded to counts, booleans, and distributional summaries.
8. Never build a model that identifies which child is at the keyboard from typing rhythm, even internally. Per finding 9, this is the bright line that turns an ambiguous behavioral-biometric question into an unambiguous GDPR Art. 9 one — and it is not a feature the product needs, since the child is already authenticated by account.
9. Default all behavioral-derived scoring/profiling to off, with narrow, disclosed exceptions, mirroring the AADC’s profiling-off-by-default norm (finding 12) — meet the strictest applicable regime globally rather than geofence protections.
10. Set a short retention period for raw-ish session timing logs, separate from the long-lived mastery/fluency model — e.g., 30–90 days to compute rolling statistics, then keep only the rolled-up per-skill score. This range is a proposal for the owner to confirm, not a cited legal requirement.
11. Don’t collect additional data (stricter age checks, device sensors) merely to justify collecting behavioral telemetry, per finding 11’s rejection of “collect more to comply” reasoning.
12. Any adaptive use of behavioral signals should be explainable to a parent in plain language — “we adjust problem pace based on how quickly and confidently you answer, not by watching how you type” — since AADC transparency (finding 12) covers how the system is explained, not only what it collects.
13. Publish the exact list of derived features collected in the parent-facing privacy notice, and treat any addition as requiring a fresh privacy review — minimization as a gate, not a one-time choice.

Open questions for the project owner

1. Should Math Challenge commit to never persisting raw per-keystroke timing (dwell/flight time), even transiently on a server, computing all aggregates on-device only — foreclosing the keystroke-dynamics/biometric question entirely?
2. What retention window is acceptable for session-level interaction logs before reduction to per-skill aggregates? The 30–90 day range in Implication 10 is a proposal, not a sourced legal requirement.
3. Will Math Challenge have UK/EU users in the near term, making AADC/GDPR/DSA directly binding rather than best practice to emulate globally?
4. Should gaming-pattern detection (Implications 5–6) ship in v1 at all, given that even Baker et al.’s validated detectors don’t transfer across different problem-interface types without retraining?
5. Should “checks own work” framing of answer-changing (Implication 4) be visible to the child as encouragement, or purely internal to scoring — this changes both UX and the AADC transparency analysis?