"Our assessors can recite the level definitions from memory. Why do their assessments still fall apart the first time a decision relies on one?"
That complaint comes in the second year of most readiness training programmes, and it does not mean the course failed to teach the scale. The course taught a set of terms and ended there. Readiness assessment is a discipline of evidence, and the set of terms used is only its surface. Once a programme is rebuilt around that distinction, the practice changes in the order in which topics are taught, the criteria for a pass, and who is permitted to sign a request.
What the level definitions are actually made of
The published scale is not a ladder of labels. Every level lists knowledge and includes an exit criterion that sets out the documentation needed before the next level applies. NASA's Earth Science and Technology Office sets out criteria of that kind: documented test performance demonstrating agreement with analytical predictions, a documented definition of the relevant environment, documented definition of scaling requirements. Those clauses make the level value no longer the content. The content is who observed what, where, and against which prediction.
DefProc's practitioner account of the same scale makes the point operationally. At the proof-of-concept stage, the company tested a sensor with air to keep the work cheap and low risk; at the following stage, the same component went to a third-party laboratory for testing with the real gas. The step between those levels is a change of environment and a change of observer. No new form was filed to make it happen.
MITRE's AiDA reference makes it clear that experts must add their professional opinion to the levels, and its supporting-information column seeks who performed the tests, when, and how the test environment differed from the operational one. That column is the curriculum. A programme that does not make assessors able to answer those questions from evidence has taught the incorrect subject well.
The reason the field spreads with more difficulty than the terms used for it
The scale crosses sectors without difficulty. The discipline does not, because what counts as evidence is local. The authors of the TRL-IS study found that very few published studies in health and social science implementation research had used the technology readiness scale to evaluate readiness at all, and that adapting it required removing laboratory testing, limiting how freely the operational environment is invoked, and drawing a sharper line between a pilot in a relevant environment and a demonstration in the real world. Those are not presentational edits. They change which observations are significant.
While our primary objective is to advance technology from a Technology Readiness Level (TRL) of 3 to 6, we acknowledge that TRL definitions can be subjective and the scale imprecise. Consequently, we use TRL definitions as a guideline rather than an absolute metric when evaluating projects.
The Aerospace Technology Institute writes that in its own programme guidance, and it funds engineering. A training programme misrepresents the tool it teaches by presenting the scale as an exact measure when an organisation with that much technical depth uses the scale as a guideline. The honest syllabus teaches the level text as a shared language for an assessment, and spends its remaining time on the assessment.
Six changes that appear once the course is rebuilt
- The limit assertion comes first, and the learner records it before touching any evidence regulation. No level is assigned if the component being assessed is not defined tightly enough to exclude the surrounding system.
- Evidence rules are written before evidence is collected. The learner chooses in advance which observations are valid, which observers are acceptable, and what a disagreement between two observations signifies.
- Evidence is collected from live work. A signed test report shows what a technician recorded; the operator, the maintenance log and the failure queue show what happened.
- Raters are calibrated against each other and against the definitions. Two assessors reviewing the same evidence pack and reaching different levels is the finding that matters most, and a course that never produces that disagreement has tested no gaps in understanding.
- The proposition has its configuration and its time. A level linked to a detail without a revision, a build version or a deployment day is not a fact about an actual entity.
- The output is written so that it can be falsified. A declaration that a system is ready is not verifiable, but a declaration that a set group performed a set task in a set environment, with the supporting observations listed, is.
None of these is a reporting convention. Each one changes what the assessor does on the day.
A trained assessor should be able to complete the log
If the training has landed, the learner can fill a structured evidence log from live evidence, unaided. The shape below is the test. Ask for it at the end of the course and read what comes back.
claim:
thing: pressure sensor, revision C
boundary: sensing element and housing only
owner: named engineer accountable for the claim
environment: unheated domestic utility cupboard, winter cycle
evidence:
- observer: third-party laboratory
when: 2026-02-14
test: hydrogen flow at ambient temperature
- observer: field trial operator
when: 2026-05-02
test: same unit, installed, occupied property
prediction: agreed for both runs before testing
result: met on the first run, drifted outside tolerance on the second
unverified: behaviour below freezing, long-duration exposure
expires: on any change to the boundary, the environment or the owner
A blank field is a finding. An assessor who is unable to say what remains unverified has not assessed readiness; they have repeated a level.
Delivery formats compared on what the learner can do afterwards
| Training format | What the learner can do afterwards | Where it fails when the claim is examined |
|---|---|---|
| Reciting the published definitions | Name the level and describe the general stage | Cannot say what was tested, by whom, or under what conditions |
| Group scoring of a worked case study | Apply the definitions consistently to a supplied scenario | The scenario supplies the evidence, so the learner never has to find it |
| Shadowing a live assessment | Recognise the shape of an evidence pack | Has never carried responsibility for a boundary statement |
| Running an assessment under supervision, with the draft claim reviewed | Defend a claim against a reviewer's questions | Depends on the supervisor's standard, which may not be repeatable |
| Blind re-scoring of archived cases against a held-back answer | Converge with other raters on the same body of evidence | Measures agreement about the past, not judgement about a moved environment |
| Re-scoring a case after the environment, configuration or owner has changed | Detect when a previous level no longer holds | Rarely offered, because it needs two versions of the same case |
The figures that set the calibration burden
Figures in the table below come from the TRL-IS study, NASA's Earth Science and Technology Office and the Aerospace Technology Institute.
| Figure | What it measures | Source |
|---|---|---|
| 9 | Levels in the generic scale, each carrying its own exit criteria | NASA Earth Science and Technology Office |
| 3 to 6 | The part of the scale the institute's programme is primarily aimed at | Aerospace Technology Institute |
| 11 | Published health and social science implementation studies that had used the scale to evaluate readiness | TRL-IS scoping review |
| 0.90 | Inter-rater reliability achieved by the adapted health and social science checklist across ten raters | TRL-IS validation |
That reliability figure is the one to keep in view when designing a course. It was reached with an adapted checklist, a defined panel process and repeated scoring of real cases. It is a ceiling produced by deliberate calibration.
What to measure once the course has finished
Generic guidance on technical skills training is confident about evaluation. Articulate's guidance recommends tracking outcomes with frameworks such as the Kirkpatrick method, mapping skills gaps before designing content, and refining continuously. That advice is sound for software rollouts and equipment operation, and it does not transfer intact, because the outcome here is neither task performance nor learner satisfaction. Six things are worth measuring instead:
- Whether two assessors working from the same evidence pack reach the same conclusion, and where they part company.
- One can check how many draft assertions survive one adversarial question about the observer, the time or the environment.
- A request comes with a configuration and a time attached.
- The total of missing limit declarations in the submission.
- Whether the assessor can name what remains unverified without prompting.
- Whether a stated fact is withdrawn or downgraded when the environment changes, and how quickly.
Those measurements survive a change of project, which satisfaction scores do not.
What changes in practice, then, is narrow and consequential. The course ends testing whether people can identify a level and starts testing whether they can defend a position. An assessor trained that way gives the same answer when the decision meeting begins, because the answer is already in the file.
Sources
- Adaptation of the technology readiness levels for impact assessment in implementation sciences: The TRL-IS checklist
- Technology Readiness Levels - NASA Earth Science and Technology Office
- Technology Readiness Level (TRL) 1 to 9 Explained
- Technology Readiness Levels (TRL) | Aerospace Technology Institute
- Technology Readiness Levels (TRLs) | AiDA
- What Is Technical Skills Training? | Articulate
While our primary objective is to advance technology from a Technology Readiness Level (TRL) of 3 to 6, we acknowledge that TRL definitions can be subjective and the scale imprecise.
