MaestroMaestro Overview — contents
The Concept
Mastery & the Knowledge Check
How a node decides mastery. Every node ends in an Evidence Check — a diagnostic instrument, not a graded test and not a gate. It reads how a learner thinks and turns that into a diagnosis that updates one evolving picture of the learner.
Maestro deliberately rejects a threshold/score model. The Evidence Check captures four signals and emits a structured diagnosis (five outputs, no score). A rules layer updates a mastery belief per Knowledge Component. "Mastered" is a state that combines belief, depth, no active misconception, and — for milestones — readiness conditions. The learner never sees a score; they see coaching.
Why not a threshold model
The instinct is to treat mastery as a score crossing a cutoff. Maestro does not, for a precise reason: a single measurement collapses cases that must be handled differently.
- A right answer with flawed reasoning looks identical to a right answer with sound reasoning.
- A wrong answer from a slip looks identical to a wrong answer from a genuine wrong model.
Any threshold-based system adapts wrongly on both. Maestro treats mastery as a structured diagnosis that separates them, and as a belief that updates over time rather than a state set once.
The four signals
To diagnose what the error was and why, the Evidence Check gathers up to four signals:
| Signal | What it is |
|---|---|
| Response | What the learner produced — the answer, the elements present. |
| Reasoning | Why they answered as they did, in their own words. This turns a wrong answer into a diagnosable one — and it is the primary diagnostic input, captured verbatim, not a supplementary field. |
| Confidence | How certain they are (a simple self-rating). This separates a misconception from a simple gap. |
| Process signals | Light traces (attempts, time) used only to strengthen a diagnosis — supportive, never a verdict. |
The structured diagnosis
The Evidence Check converts the signals into a structured diagnosis of five things — and it emits no score and no pass/fail:
- Error type — one of: secure mastery, slip, fragile, knowledge gap, misconception.
- SOLO depth — one of: surface, multi-element, relational, extended-abstract.
- Specific misconception — if detected, matched against the node's governed bindings (never invented).
- Learner confidence — self-reported at submission.
- Match confidence — the diagnostic layer's own confidence in its diagnosis.
The base diagnosis crosses correctness with confidence, then reasoning refines it.
| Confident | Unsure | |
|---|---|---|
| Correct | Secure mastery | Fragile / lucky |
| Incorrect | Misconception | Knowledge gap |
Reasoning then refines: a wrong answer with sound reasoning is likely a slip, not a fault; a right answer with flawed reasoning is fragile even if confident; a wrong answer matching a known wrong model is tagged as that specific misconception.
The AI evaluation layer matches a response against a fixed, governed set of options for the node and reports its own match_confidence (low | medium | high). It never freely diagnoses; misconception_id is always one of the node's provided ids or "none". When a response is wrong but matches nothing, it sets unexplained_error = true. The most important property is calibration — the system must know when it doesn't know — so uncertain cases can be routed somewhere safe rather than acted on.
Error types, readiness & SOLO
| Error type | Plain meaning | Learner-facing readiness state |
|---|---|---|
| Secure mastery | Understands it, and knows they do. | Ready (Advanced if extended-abstract) |
| Slip | Knows it; careless mistake. | Ready |
| Fragile | Right answer, weak ground. | Partially Ready |
| Knowledge gap | Concept not yet formed. | Not Ready |
| Misconception | Confident, specific wrong model. | Not Ready (targeted) |
Readiness states (Not Ready / Partially Ready / Ready / Advanced) are a learner-facing simplification derived from the error types — an output label, not the diagnosis itself. Depth of correct understanding is captured separately on the SOLO scale, which climbs: surface (one idea in isolation) → multi-element (several ideas as a list) → relational (ideas connected and integrated) → extended-abstract (generalises beyond the example, transfers to new situations — this is what earns the optional challenge).
Mastery belief & what "mastered" means
The rules layer takes the diagnosis and updates the mastery belief for the node's Knowledge Component — a probability that climbs and dips with evidence, not a flag set once:
| Diagnosis | Effect on belief |
|---|---|
| Secure at relational depth (first attempt) | Raises belief substantially. |
| Fragile | Raises belief partially; marks the KC for re-check. |
| Slip | Little effect; noted for pattern detection. |
| Knowledge gap | Lowers belief; routes to remediation or prerequisite. |
| Misconception | Lowers belief; sets the misconception flag. |
Not a threshold crossing. A state in the learner model combining four conditions: (1) mastery belief above the level needed for that KC's role in the course — higher for critical and bridge nodes; (2) SOLO depth at or above what the node's mastery statement requires; (3) no active confirmed misconception on the KC; (4) for milestones only, the specific readiness conditions the milestone contract requires. The bar is authored per node, not global — different KCs have different bars because they play different roles.
The confirming probe
The system gathers evidence until its confidence is high enough to act, and the bar rises with the cost of the action. A clear secure-mastery or clean gap can be acted on from a single rich item. But a flagged misconception triggers a disruptive, costly reroute — so it earns one confirming probe first: the same idea in a slightly different surface form. If the error recurs, the misconception is stable; if it vanishes, it was a slip. Evidence also accumulates across the pathway, since concepts recur — so a single wrong diagnosis is recoverable. This restraint keeps the learner's pathway stable rather than thrashing on noise.
The learner model
The learner model is the spine — the running, evolving picture that turns separate node judgments into genuine adaptivity. It holds one record per learner, with a row per Knowledge Component encountered:
| It holds | Meaning |
|---|---|
| Mastery belief | The probability the learner has mastered this KC — climbs and dips with evidence. |
| SOLO depth | The depth band most recently evidenced. |
| Misconception flags | Any wrong model currently active for this learner on this KC. |
| History | Last seen, number of encounters, whether the belief is holding or fading. |
| Confidence calibration | How well the learner's confidence tracks their actual correctness, accumulated over time. |
Every node reads the relevant rows before it judges; every Evidence Check writes an updated belief and any flag back. Because a KC recurs across nodes, the model self-corrects. It also carries real duties: privacy, the learner's right to see their own profile, and treating a low belief as a transient state to support, never a fixed label.
The AI produces a diagnosis; deterministic rules own the belief update, the misconception-flag lifecycle, and status derivation. This is what makes mastery decisions reproducible and auditable. Build in anti-thrashing safeguards (bounds on how much one check moves a belief) and optional decay/fading, both tunable. The evidence record and learner-model shapes are given in the Published Course Package and Architecture & Trust Layer.
The misconception library
This is the single most important reliability decision in the platform. Open-ended diagnosis ("invent the explanation for this error") is something LLMs do poorly — they hallucinate plausible-but-wrong misconceptions. So Maestro never asks for open diagnosis. Instead, the AI matches a response against a governed, finite list of named misconceptions — a closed-recognition task it is reliable at. The library is what converts the hard problem into the tractable one.
It is AI-generated and human-governed, through a lifecycle with two human gates:
Each entry carries an ID, a name, the attached KC, the wrong model stated plainly, its origin, a detection signal (answer + reasoning pattern), a trap that surfaces it, a confirming probe, targeted remediation, and optional prevalence/severity. The human never authors — they accept, reject, or refine. As learners hit the node, responses the AI cannot confidently match are logged as unexplained errors; recurring clusters become candidates for promotion.
Formative vs summative assessment
Two categories of assessment are in scope, running on different logic. They must be built as separate systems, not one merged "assessment engine."
| Formative — Evidence Checks | Summative — milestone assessments | |
|---|---|---|
| Where | Inside every node. | Four milestones in MDLD602 (A1–A4). |
| What it is | Captures response, reasoning, confidence; produces a structured diagnosis. | A redesigned-for-contribution artifact against a rubric. |
| Judged by | AI-diagnosed — produces mastery signals, not scores. | SME-graded against the rubric with AI assistance — produces the milestone grade. |
| Reachable | Continuously through the journey. | Only once the readiness gate unlocks. |
How a grade relates to mastery
The framing runs the other way in Maestro: grades do not translate into mastery decisions — mastery decisions produce readiness, readiness unlocks submission, and submission produces a grade.
The grade reflects the quality of the artifact, not a re-derivation of whether the KCs were mastered — that decision was already made at the Evidence Check level and remains authoritative. A submitted artifact contributes additional evidence to the learner model, but it never retroactively re-decides mastery. What each diagnosis does next is the subject of Adaptivity & Routing.