Technical ReadinessTechnical Readiness
What Technical Readiness Model Actually Does
Technical Readiness

What Technical Readiness Model Actually Does

Robert FosterWritten by Robert Foster

Two programmes can run the same technical readiness model over the same technology and reach opposite conclusions, and both can follow the published level definitions exactly. The first course uses the model as a labelling device. A user reviews the definitions, selects the description that best matches the current state of the work, puts the digit on a slide, and allows that digit to influence a funding decision it was never designed to support. That course takes an afternoon, is free, and produces a value with no identified owner, no limit, no evidence requirement and no time assigned to it. The second course uses the model as a tool. An area is set for the item being assessed and a user is assigned to it. The criteria for what will be accepted as evidence are established before any data is collected. Evidence comes from work that has been run. The output is a timed declaration that limits what the next meeting can approve, and the declaration expires.

Neither course is dishonest. They answer different questions, and the gap between them is where readiness figures are inaccurate.

What a readiness model converts

The function of any readiness model, whatever the scale is called, is a conversion. It converts observations about completed work into a prediction about work not yet attempted. The observations are the cheap part. NASA's description of the Technology Readiness Levels sets the pattern: the levels are a measurement system used to assess the maturity of a particular technology, and each project is evaluated against the parameters for each level before a rating is assigned. Parameters. ITONICS's guide to the readiness frameworks expresses the same principle in a single line: a maturity level is determined through formal evaluation, based not on potential, but on proven performance.

The scope of a model's utility and its limitations both stem from that conversion. The Technical Readiness Model is unable to report on work that has not happened, on conditions that have not been entered, or on a configuration that has not been built. What it can do is make the distance between what has been shown and what the decision requires visible, in terms two parties can argue about without re-opening the whole programme.

What a level reports about form and environment

Read the nine levels as a sequence and one pattern is hard to miss: each step changes either the form of the article being tested or the conditions it is tested in, and often both. At level four, multiple component pieces are tested with one another. At level five the article is a breadboard and the testing is more rigorous than at four. At level six it is a fully functional prototype or a representational model, demonstrated in a relevant environment. At level seven the working model is demonstrated in a space environment. At level eight it is flight qualified, and at nine it has been flight proven during a successful mission. That progression does not specify the proportion of the final product that is complete. It says what was put before the test and under what conditions it survived.

The conditions are not a detail. The Aerospace Technology Institute uses TRL definitions as a guideline because they can be subjective and the scale imprecise. The same page records a working assessment that full-scale flight demonstration is closer to level seven than level six, depending on the technology, the system it belongs to and the objectives of the demonstration, and that introducing a fundamental system change such as a new primary energy source can make flight demonstration necessary before level six can be reached at all. Two assessors can therefore disagree about one value without either being careless. They are disagreeing about whether the test conditions were representative, which is the part of the premise the digit represents.

What makes an assessment credible, objective, reliable and useful

GAO-20-48G views a technology readiness assessment as a way to evaluate if a technology is developed enough to be incorporated into a system without too much risk, and it records the cost of an incorrect evaluation: technologies that are not adequately mature have been the source of programme delays and cost increases. Its more useful contribution is the test it hands to oversight bodies, who are told to ask whether an assessment demonstrates credibility, objectivity, reliability and usefulness. Those four traits apply to a readiness model as readily as to a readiness report, and a rubric that is unable to produce them is a questionnaire.

The Manufacturing Technology Centre built its Technology Readiness Measurement Toolkit with researchers at Heriot-Watt University for the case where a single scale will not do. HVM Catapult's announcement of the toolkit says it improves on the established technology and manufacturing scales by clarifying and integrating them and by incorporating wider issues such as the readiness of supply chains to deliver necessary components. It also puts an effort figure on the work: three days in total, including a half day on site. That figure is the most useful item on the page for anyone commissioning an assessment, because it marks the limit between a tool and a document review. An assessment that never visits the site and never looks at the supply chain is an analysis of papers with a scale attached.

What each scale grades

When more than one scale is in use, the useful question ceases to be which is best and becomes which object each one is permitted to grade, and what it will accept as proof.

Scale titles, level totals and the questions each framework answers are set out in ITONICS's guide to the fourteen readiness frameworks; the manufacturing scale range is given by Linden Innovation.

Instrument The object it grades What it accepts as evidence The question it answers
Technology readiness level One technology or component A test or demonstration in a stated environment Does it work?
System readiness level A system assembled from subsystems Subsystem levels plus integration and interface assessment Does it work together?
Innovation readiness level A project, across customer, technology, business model, intellectual property, team and funding Validation performed with real customers, suppliers and backers Is every dimension ready at once?
Manufacturing readiness level Production Process capability, tooling, supply chain and quality under expected production conditions Can it be made at the required scale?
Operational readiness level Operations Processes, infrastructure, training and support in place Can it be run once deployed?
Commercial readiness level Commercial fit Demand, pricing, regulatory position, customer traction Will anyone pay for it?

The overlap between these scales is smaller than their shared terms suggest. A programme can have a high value on the first row and a low value on the fourth, and the fourth determines the time. Linden Innovation records the common version of that gap: A product is at level nine on the technology scale while production is at four or five, with about eighteen months of scale-up work before a customer receives it.

Where a level is read back into a funded decision

Funding gates are where a level ceases to be a summary and becomes a condition of entry. Work that is in one band can be funded by one scheme and refused by another, and the bands are published.

Bands below are as listed by Linden Innovation for Dutch and European programmes, and by the Aerospace Technology Institute for its own programme.

Programme or gate TRL band it accepts The stage of work that band covers
ATI programme 3 to 6 Proof of concept through demonstration in representative conditions
WBSO 1 to 5 Basic research through validation in a relevant environment
Vroegefasefinanciering 2 to 3 Concept formulation through proof of concept
Innovatiekrediet 3 to 9 Proof of concept through proven operation
DEI+ 6 to 8 Demonstration through qualification
Horizon Europe EIC Accelerator 6 to 9 Demonstration through market deployment

Two facts follow from the band column. A level is a commercial fact as well as a technical one, because it determines which money can be applied for. The second is that the same body of work can be described at different levels to different funders truthfully, provided the band definitions are viewed as ranges. The Aerospace Technology Institute uses its own definitions as a guideline and allows support below its stated band where a proposal is industry-led, aligned with the programme and able to reach higher levels within a reasonable timeframe.

What the lowest grade reveals about progress

Every scale in the table above grades one dimension, and a programme is not one dimension. Linden Innovation makes the operative point: the multi-dimension approach identifies which dimension is the weakest link, and that link is almost always the binding constraint on overall progress. Readiness is therefore not the highest value a programme can justify in a meeting. It is the lowest value it must avoid, and the report must say which one that is.

How far a programme has to travel sharpens the difference between the two courses. IDC is a product design consultancy that works across the whole scale, and it explains two developments it took through its process. A wearable breast pump entered at level five with the general technology proven but not in the configuration the user needed, and left at level nine as an approved commercial product in production. A cell therapy bioreactor entered as an idea at level one and reached level five as a working machine and sterile consumable whose performance had been independently verified, which is what allowed the company to raise venture capital. IDC summarises the bands as ideas and research at one to three, engineering, prototyping and refinement at four to six, and demonstration, production scale-up and launch at seven to nine. The model said where each programme stood on the day. It did not say how long the next band would take.

No published scale prints an expiry time either. Umit Bititci, professor of business performance at Heriot-Watt University, put the problem in a form worth keeping:

TRLs and MRLs have traditionally been used as a snapshot of progress achieved so far, rather than as a tool to estimate the future time and investment required to get the product to market.

A snapshot has a time or it is empty. The same applies to the scope it was included in and the label attached to the fact. Updating the configuration, the environment, the accountable owner or the decision makes the value obsolete even if the technology remains the same.

Where the model grades a management system instead of a device

The word readiness covers objects with little in common. An ISO 27001 readiness assessment, as CyberOne explains it for UK organisations, is a diagnostic performed before the audit to evaluate the information security management system. Its evidence is documents, records and answers. It produces a list of non-conformities with owners and a log of applicability that shows which of the standard's ninety-three controls apply and justifies every exclusion. The work takes five to ten days for most small and medium sized organisations and extends to several weeks where operations and supply chains are complex. The Technical Readiness Model precedes the two audit stages, the first on documentation and the second on operational effectiveness.

The item being graded determines what the model can output and who is competent to use it. A scale applied to a device reports what survived a test. A scale applied to a management system reports whether the organisation can produce evidence on demand. Both are useful. They are not substitutes, and a programme that buys one while needing the other discovers the difference at the worst possible time.

Test a model against the choice you face

Log the choice first, with its owner and its time, because every other check depends on it. Compare the scope of the model against the scope of the decision: A component-level scale is unable to inform a choice about a service, and a production scale is unable to determine whether an operator can manage the system on a Monday morning. Then check that the model fixes its evidence criteria before collection begins and specifies who is entitled to produce evidence, because criteria created after the results are in get bent to fit them. Check that the model has a place to log the weakest element, the configuration assessed and the time, as a report that is unable to include those three items will be quoted without them. Ask whether the assessment can be performed on live work and check the effort against the three days the MTC toolkit takes, site visit included. Finally, before accepting the report, run the four properties from GAO-20-48G across it, asking whether it is credible, objective, reliable and useful. Each of those is a question about the assessment.

This section determines what is chosen

Using a model as a label is acceptable. It answers the question of where the programme stands, quickly and cheaply, and there are meetings where that is all anyone wants. A model used as a tool answers the harder question of what can be approved next, and it does so with a limit, an owner, an evidence requirement and an expiry. The test for which one you have bought is blunt. If a readiness score has never stopped a project from being approved, no one bought the tool. They bought a label, and the slide should say so.

Sources