Feedback and Formative Assessment in Mathematics — Evidence for an AI Tutor

Findings

1. Hattie & Timperley’s feedback model (2007)

Effective feedback answers three questions: “Where am I going?” (feed up), “How am I going?” (feed back), “Where to next?” (feed forward) [1]. It operates at four levels: task, process (strategy/method), self-regulation, and self (personal praise, “you’re so smart”). Task/process feedback aimed at self-regulation is powerful; self-level praise is the weakest of the four and can dilute the others when combined in one message (e.g., “Great job, you’re brilliant!” tacked onto a correctness notice) [1].

2. Kluger & DeNisi (1996): feedback can hurt

A meta-analysis of 607 effect sizes / 23,663 observations found a positive average effect (d = .41), but over a third of feedback interventions decreased performance [2]. Feedback Intervention Theory explains the split: feedback that redirects attention to the self (ego-involving, comparative, praise/blame) draws resources away from the task and can suppress performance after failure; feedback that keeps attention on the task and gap-closing strategy tends to help. This is the evidentiary basis for treating “always give feedback” as false.

3. Black & Wiliam and the formative-assessment evidence base

Reviewing 250+ studies, Black & Wiliam (1998) found formative assessment raises test outcomes with effect sizes of 0.4–0.7 — larger than most educational interventions — with the biggest gains for low-attaining students [3]. Conditions: information must be used to adjust teaching in near-real time, feedback must say how to close the gap (not just how far off), and students need ownership (self-/peer-assessment). This argues for a continuous formative loop (attempt → explain → adjust next problem) rather than a one-shot end-of-session report.

4. Timing: immediate vs. delayed feedback

A recent meta-analysis (51 studies, 1988–2024, 160 effect sizes) found no significant average difference between immediate and delayed feedback, but math tasks showed larger effects than other subjects, and immediate feedback raised learner confidence in computer-based math practice even without changing accuracy gains [4]. Elaboration (what feedback says) mattered more than timing (when it arrives) [4][5]. Net: timing is secondary, content is primary — but immediacy still helps confidence and prevents a wrong procedure from being rehearsed further.

5. Feedback content taxonomy (Shute, 2008)

Shute distinguishes Knowledge of Results (KR) (correct/incorrect only), Knowledge of Correct Response (KCR) (states the answer), Elaborated Feedback (EF) (explains why, with cues/examples/strategies), and Answer-Until-Correct. EF generally outperforms KR/KCR, but excessive elaboration can be detrimental, overloading working memory [5]. This argues for elaborated-but-short feedback, not exhaustive re-teaching of what a student already got right.

6. Praise for effort vs. ability, and inflated praise

Mueller & Dweck (1998, six studies): children praised for intelligence showed, after a subsequent failure, less persistence, less enjoyment, more low-ability self-attributions, and worse performance than children praised for effort/strategy; 92% of effort-praised children chose harder follow-up puzzles vs. 33% of intelligence-praised children [6]. Brummelman et al. (2014, 2017) found inflated praise predicts lower self-esteem over time, and higher narcissism in children who already have high self-esteem; non-inflated, accurate praise produced neither effect [7]. Together: praise process/strategy, keep it proportionate, never praise fixed traits.

7. ITS/CAI feedback meta-analyses

VanLehn (2011): intelligent tutoring systems reach d ≈ 0.58 vs. no tutoring, nearly matching human tutoring. Step-based tutoring (feedback at each solution step) reached d ≈ 0.76 — almost as good as a human tutor — while answer-based systems (final-answer-only feedback) reached only d ≈ 0.40 [8]. Strong signal: comment on steps/work, not just the final answer, wherever the format captures intermediate work.

8. LLM-generated math tutoring feedback (2023–2026)

MathDial (EMNLP 2023) built 3,000 tutoring dialogues because raw LLMs “fail at tutoring” — they generate incorrect feedback or reveal solutions too early (“telling@k”) [9]. SocraticLM and PEARL train models to withhold answers and scaffold with questions instead [10][12]. MathTutorBench (EMNLP 2025): solving ability does not transfer to good tutoring, pedagogy and competence trade off, and quality degrades over longer dialogues [10]. LLMs also produce fluent-but-wrong chain-of-thought, distinct from answer-revealing [13]. The one field RCT, Tutor CoPilot (2024, 783 tutors, ~350k messages), found AI suggestions increased probing questions and decreased generic praise, a 4pp mastery gain (p<0.01), concentrated among lower-rated tutors [11]. Khanmigo evaluations report it beats raw GPT-4o at catching errors, and structured performance signals improved next-item correctness ~6% — but regular usage stays low (~15%) [14].

9. Age-appropriate phrasing

Early-childhood guidance (NAEYC, Wisconsin DCF) recommends descriptive, specific feedback over generic praise (“you counted the beans again and got the same number” vs. “good job”), since specificity lets a child connect feedback to a repeatable action [15]. The age gradient runs from concrete/sensory language for young children toward abstract meta-cognitive language (strategy, why, transfer) for older students.

Design implications for Math Challenge

1. Structure every tutor message as feed-up / feed-back / feed-forward: (a) restate the goal, (b) say what happened relative to it, (c) give one concrete next step. Never stop at (b) — that leaves the highest-value part of Hattie & Timperley’s model unused [1].

2. Never combine task feedback with self/trait-level praise in the same sentence. Ban “Correct! You’re so smart at math” — split correctness from encouragement, and keep encouragement about effort/strategy, never ability. Follows from Kluger & DeNisi’s finding that self-level attention capture is the likely mechanism behind feedback backfiring [2][6].

3. Comment on the student’s work/steps, not only the final answer, wherever the format captures intermediate steps. The single highest-leverage architectural choice per VanLehn’s ITS meta-analysis (step-based d≈0.76 vs. answer-based d≈0.40) [8].

4. Keep elaborated feedback short — 3 to 6 sentences, one worked example maximum. Shute’s excessive-elaboration backfire means the prompt needs an explicit length cap, not “explain everything you can” [5].

5. Do not let the tutor give away the next problem’s answer or method prematurely mid-attempt (e.g., in a hint flow before submission) — the MathDial/“telling@k” failure mode. Constrain the tutor to Socratic/step-scaffolded hints during an active attempt, and reserve full worked explanations for the post-submission review [9][10][12].

6. Guard against confidently-wrong chain-of-thought. Validate any generated step-by-step explanation against a deterministically computed correct solution before showing it — the LLM should narrate a known-correct derivation, not freely re-derive the math, given documented fluent-but-wrong reasoning chains [13].

7. Immediate feedback for correctness/completion signals (right/wrong, points earned); a short delay (sub-second to a few seconds) is fine for the deeper “why” explanation, but not end-of-session — immediate feedback helps confidence and prevents a wrong procedure from being rehearsed further [4].

8. Reserve pattern-level feedback for an end-of-session summary, distinct from per-problem feedback: e.g., “fastest on multiplication facts, slowest on multi-step word problems; next session adds more scaffolded word problems.” This maps to Black & Wiliam’s formative loop — using aggregated evidence to adjust the next instructional unit, not just the next sentence [3].

9. Age-tiered FEEDBACK TEMPLATES for the tutor prompt:

   • Ages ~4–6: 1–2 short sentences, concrete/sensory, no abstract strategy talk. Template: [concrete observation] → [simple correct step] → [effort praise tied to the specific action]. Example: “You counted the apples one by one — there are 7, you said 6; let’s count together: 1, 2, 3… You’re getting really good at counting carefully.”
   • Ages ~7–10: 3–4 sentences naming the specific step where things diverged, one named strategy, effort/strategy praise. Template: [what you got right] + [the exact step that went off track] + [why the correct step works] + [strategy-based encouragement].
   • Ages ~11–14: 4–5 sentences introducing the why behind the rule, inviting comparison to the correct approach, using subject vocabulary. Template: [feed up: what the problem tested] + [feed back: where reasoning matched/diverged] + [correct rule with a mini worked step] + [feed forward: a related problem type to watch for].
   • Ages 15+ / adult: Terse, technical, peer-level; skip encouragement boilerplate, focus on precision (“correct but not minimal; here’s a faster path”), offer depth on request.

   All bands: never fixed-trait framing (“you’re not a math person”); always name the specific action, never a global judgment.

10. Feedback to avoid, because evidence says it backfires: generic trait/ability praise [6]; inflated/superlative praise for routine correctness [7]; correctness-only feedback with no path forward when wrong [5]; revealing the full solution before the attempt is over [9][10]; long re-teaching of material already mastered [5]; comparative/normative feedback (“behind other kids your age”) — the exact ego-shift mechanism behind feedback-induced performance drops [2].

11. Tie gamification feedback to effort/process signals (persistence, strategy use, improvement over own baseline), not only speed or streaks, so scoring doesn’t reintroduce ability-framed feedback via leaderboards or fixed-talent badges.

12. Require the tutor prompt to self-check a short rubric before emitting a message: separates task from praise; names a concrete next step; within the age-band’s length cap; avoids revealing next-attempt answers; any worked step is validated against a computed ground truth. This operationalizes the rules above as a gate, not a hope.

Open questions for the project owner

1. Should immediate per-problem feedback and the fuller AI-tutor explanation always show together, or should ages 4–6 get a simplified inline reaction immediately and the fuller explanation only in a parent/session review?
2. Do we currently capture intermediate work/steps on multi-step problems, not just the final answer? If not, is that worth prioritizing given the step-based-vs-answer-based ITS gap (d≈0.76 vs 0.40)?
3. Should end-of-session summaries go to the child, the parent, or both, with different phrasing (child-facing encouragement vs. parent-facing diagnostic detail)?
4. How should the tutor validate its worked-solution narration against ground truth — a separate deterministic solver, or a secondary verification LLM pass?
5. Do we want a “novice teacher” fallback (plain correct-answer reveal) when a full Socratic/elaborated explanation would be too slow or costly, and at what latency/cost threshold?