A food hygiene rating in a shop window is a single score, and the inspection behind it logged which practices were observed, when, by which officer, and what the kitchen was doing while the officer watched. The value summarises one visit under one set of conditions. A sticker stays in the window and provides less information about the internal operation as the menu, the shift pattern or the volume of covers change.
A technology readiness level earns its place in the way that sticker does, by standing for tests that have already been run. Its value turns on how closely the conditions of those tests match the conditions of the job now being decided. The question an assessor is usually handed concerns a decision. This question queries whether this system, in the configuration now proposed, can do the work it is being funded to do, in the place where that work happens, and who is accountable for saying so. A level is one input to that decision, and on its own it is a thin one.
A technology readiness level compresses a testing history into a single digit. The scale in widest use has nine levels, and each level is defined by two things: the form the technology has reached and the environment it has met. NASA publishes and maintains the scale as a measurement system to assess the maturity level of a particular technology, from TRL 1 at the base to TRL 9 at the top, and attaches an observable fact to every point on it: paper studies at the base, analytical work above them, laboratory integration of components, testing with realistic supporting elements in a simulated environment, a prototype demonstrated in a relevant environment, a prototype demonstrated in an operational environment, qualification through test, and a system proven under mission conditions. Near the top of the scale NASA uses the language of airworthiness, describing a technology first as flight qualified and then as flight proven.
A link that a level means to convey is a relation, and the relation has four parts: A called part, a stated configuration, a time, and a task performed in the environment where that task will actually operate. Evidence is located underneath the relation. A level and a stated fact can part company on the same day. Two systems can occupy the same rung of the scale and support opposite conclusions, because one has been exercised in the conditions that matter and the other has only been described as though it had been.
Confusion between the digit and the value is easy to understand, because a level takes a phrase to state and a position takes a file to defend. Three publishers have put the same limit in their own words. The Aerospace Technology Institute says that the definitions can be subjective and the scale imprecise, and uses them as a guideline when it weighs projects. EURAXESS points out that the scale is self-declared and can differ from sector to sector, and the European Union has used it in funded projects since 2012. MITRE's AiDA guidance calls the levels a helpful knowledge-based standard and shorthand, and adds that they must be supplemented with expert professional opinion. An assessment starts at this level, and the evidence determines where it ends.
A level has some data and lacks other data
A level does four things honestly. It gives projects a shared set of terms, so a funder and a research team can talk without inventing a private scale. It filters applications coarsely, which is why funding calls ask for a band and leave the precise score to the assessment. It points to the evidence that should exist next, because each level description identifies the work that produces it. It tracks progress over time when the same assessor applies the same definitions to the same item.
A level has four limits. The system is unable to show how vital the part is to the whole. The TRL does not show how old the item being graded is. It is unable to split the distinct parts of readiness that it merges into a single digit. A decision maker is unable to know if the setting in the assessment resembles the setting where the job will operate.
Place the four strengths beside the four limits and a working guideline emerges. A level is safe to use as a filter, as a shared label, and as a pointer to the next piece of work. It is unsafe as the only basis for releasing money, because release is a decision about the future and the value reflects a past test.
A Single Value Loses Information
James Smith's report for the Software Engineering Institute sets out four problems that arise when a level is treated as a complete definition, and builds the argument on non-developmental software: commercial off-the-shelf, government off-the-shelf, and open source. The four problems generalise well beyond off-the-shelf products, which is why they are worth comparing with any level definition.
| Problem | What it means | The question it forces on any level claim |
|---|---|---|
| Blurring | Several aspects of technology and product readiness collapse into a single number | Which aspect is this number reporting? |
| Criticality | The scale does not account for how critical a product or technology is to the system as a whole | What happens elsewhere if this element fails? |
| Ageing | The scale does not account for software technology and product ageing | How old is the artefact, and what has changed since it was tested? |
| Life cycle variation | The scale gives no means to handle how the relative contributions of the various aspects of readiness vary across the life cycle of a system | Has the weighting between aspects changed since the last assessment? |
The four problems above come from SEI report CMU/SEI-2004-TR-013, An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software.
The Scales in Circulation and the Question Each Was Built to Answer
| Scale or instrument | Who publishes it | What it grades | What it accepts as evidence | Where it stops earning its place |
|---|---|---|---|---|
| NASA TRL | NASA | A technology project's maturity across nine levels, from basic research to mission proven | Analytical and laboratory studies, component integration, prototype demonstration in relevant and operational environments, qualification, operational success | When the thing at stake is a product already in the market and the development history belongs to a vendor |
| ATI TRL | Aerospace Technology Institute | Aerospace technology maturity, with the funding programme aimed at the middle band of the scale | Ground test, high-fidelity ground-based test, flight demonstration where required, qualification against customer requirements, operational experience | When the scale is treated as an absolute metric; the institute itself treats it as guidance |
| EU TRL as EURAXESS presents it | European Union funded programmes since 2012 | Maturity with sector translations; the drug discovery mapping runs from basic research through clinical trials to a marketed product | Sector convention plus the applicant's own declaration | When a level earned in one sector is read as though it meant the same work in another |
| Machine Learning TRL | Lavin and colleagues, in Nature Communications | Machine learning and artificial intelligence systems, with explicit distinctions from traditional software engineering | Systems engineering process evidence applied to model behaviour | When a standard software checklist is applied and model behaviour goes unexamined |
| NDI software alternative | Software Engineering Institute, Carnegie Mellon | Separate dimensions of readiness, reported one by one | The separate aspects, reported as separate dimensions | When a comparable shorthand is what the reader needs from the number |
Sources for the table above: You can find information from NASA, the Aerospace Technology Institute, EURAXESS, the University of Oxford's research archive of the paper by Lavin and colleagues, and the SEI report by James Smith.
Six steps turn a level into a justification
- Name the task, the environment and the group who will perform it. Put the job in the same phrase as the technology, and explain the surroundings in operational terms: the working medium, the temperature range, the duty cycle, the load, the network conditions, whatever the setting fixes.
- Fix the configuration and the time. List which build, which firmware, which part ID and which supporting elements were present during the test. A level report without a setup is unable to be remade, and what is unable to be remade is unable to be studied.
- The evidence guideline is established before testing starts. Determine which metrics are valid: * Third-party supervision, calibrated tools, a stated sample size, a comparison against a baseline, and a signed test report. An evidence guideline created after the results appear transforms the assessment into a negotiation about the result.
- Test in the cheapest conditions that can settle the question, then climb in order. Cost rises steeply as the environment approaches the real one, and each step should close the question the previous step left open.
- Keep the failures and the design changes in the file. A level earned through a change of design conveys more to a reader than a level earned in a single clean run, because it shows which failure mode has been closed.
- An external party to the programme should try to reproduce the result. Give them the file and the configuration, and inquire which proposition the evidence supports and which proposition it does not.
What Each Change of Conditions Buys
| Change of conditions | Claim it supports | Effort and cost it adds | What it leaves open |
|---|---|---|---|
| Substitute the real working medium for a safe stand-in | That the principle survives contact with the substance the system will meet in service | Handling, safety supervision, a test partner | Long-run behaviour, and behaviour outside the tested envelope |
| Move supervision to a third party | That the reading does not depend on the team that built the device | A laboratory booking, a documented method, a second signature | Whether the sampled units represent the production build |
| Replace the bench with a rig built to reproduce service conditions | That the device tolerates temperature, humidity and concentration changes together | Rig design, repeated runs, elapsed time | Installer behaviour, network conditions, user handling |
| Move into an external laboratory or an industrial site | That the device works where the work is done, with the equipment that is actually there | Site access, planning, a safety case | Whether that site represents every site in the rollout |
| Install in homes or on a live network | That the technology survives ordinary use and produces usable data | Support, maintenance, data handling, consent | Performance at production volume |
| Run the production line | That the design can be made at the rate and tolerance the market requires | Tooling, regulatory testing, a technical file | Little about readiness; this is the point where the work has been demonstrated |
The hydrogen sensing log is for use in order.
DefProc's published account of its hydrogen sensing work shows the ladder in operation on one product family, and it shows how much of each advance came from changing the surroundings while the hardware stayed largely as it was.
| Level | What was tested | Conditions of the test | Who supervised |
|---|---|---|---|
| TRL 3 | A proof of concept for a smart gas pressure sensor, with candidate parts chosen from the results | Air, in the company's own workshop, to keep the phase low in cost and risk | DefProc's own team |
| TRL 4 | The same proof of concept | Hydrogen flow, in a laboratory | DNV |
| TRL 5 | A prototype redesigned to resemble a smoke detector for the H2Go project | A rig built to reproduce service conditions, with temperature, humidity and hydrogen concentration varied | DefProc's own team |
| TRL 6 | A fully functional prototype | External laboratory testing, with more precise instrumentation | Health and Safety Executive Laboratories |
| TRL 7 | Devices from that prototype line | Semi-detached homes fitted with hydrogen equipment | NGN's Low Thornley hydrogen homes test site |
| TRL 8 | A separate product, the Push to Talk device, prepared for production | Regulatory and electromagnetic compatibility testing, tooling for injection moulding, UKCA marking | DefProc with an industrial design partner |
The sequence above follows DefProc's account of its own projects.
Read down the table and the pattern is plain. The engineering question at each step was narrow, and the change that answered it was mostly a change of place, of medium, or of who was watching. Read across the last two columns and the cost is plain too: supervision by an outside body and testing at a site in use are the two steps that pull in a partner, a schedule and a safety case.
Use a previous level to make a decision
Identify which choice the level supports, and seek the identity of the owner of that choice and when it is due. A level attached to no decision has nowhere to earn its place.
Ask for four things in writing: The test setup, the time of the test, the environment the test reproduced, and the group that supervised it. An answer missing any one of the four is a description of an intention.
Enquire what would change the level. If no information that could be learned in the coming months would change it, the TRL is decorative, and a steering group is being requested to fund on the strength of a label.
Use the questions in the table above to find which of the four failures the digit hides: criticality, ageing, blurring, and the way the weight of each aspect shifts across a life cycle. Collect the answers next to the level, so the next reader inherits the caveats along with the digit.
A technology readiness level earns its place when it is linked to a task, a place, a time and a signature, and when the user knows which questions it has answered. Where those attachments are missing, the level still does one useful job, because it says where to begin asking.