What is the output of a technical readiness framework?

The output is a framework document. It includes the decision being served, the accountable owner, evidence rules, and a rating sheet linked to actual demonstrations.

How should evidence be collected for the framework?

Evidence must be collected from live work such as test records, logs, and operational data. You should collect artefacts produced during work rather than relying on documents of intent.

How do you score a technical readiness assessment?

Rate based on the lowest piece of critical evidence rather than an average. Record missing evidence as unknown and keep reviewer dissent visible in the record.

Technical Readiness Framework: What Good Looks Like

The completed framework document allows an unfamiliar user to understand and apply the process. This identifies the decision the framework exists to serve, the individual accountable for the fact, and the deadline by which that fact has to land. It defines the scope of the assessment. It specifies, before any evidence is gathered, which evidence is valid. It has a score sheet where every line points to a demonstration that actually happened, in a specific environment, at a known time. It ends with a re-check trigger and an owner for the next round. No part of the framework depends on your recollection of why a choice was made.

That document is the output. What follows is the order of work that produces it.

1. Name the decision the framework serves, and its owner

A framework with no decision attached becomes a scoring hobby. Log the choice in a single phrase that includes a verb of choosing, the identity of the user who will choose, and the day by which they must choose. Two examples of the shape: The finance director and the board approve the migration of the billing service as the owner on a specific day. The investment committee provides the time to fund the second production line for the owner and operations lead.

Verify your progress before proceeding: Read that text to a colleague and have them repeat back what will be decided and by whom. If they are unable to, the framework lacks a purpose for readiness.

The GAO's Technology Readiness Assessment Guide regards this as the purpose of the entire process. Assessments establish whether technology is developed enough to be incorporated into a system without too much risk, and the guide identifies credibility, objectivity, reliability and usefulness as the marks of a high-quality assessment. Which of those four matters most depends entirely on the decision you have just written down.

2. Define the scope of the system being assessed.

The unit is not the technology in the abstract. It is the technology in the configuration now proposed, in the environment where it will actually run. List the scope as a short list in three parts: The scope defines the system limits and the included interfaces. Identify the assumptions the scope relies on.

Check: Give the scope description to an engineer who has not attended the project meetings. They should be able to say what has been excluded and why that matters.

DefProc's walkthrough of the nine technology readiness levels makes the limit visible in practice. The distance between a prototype validated in a laboratory and the same prototype demonstrated where it will be used is the distance the rating turns on. Change the environment and you change the score without changing a single component.

3. Establish the evidence criteria before collecting evidence.

Determine beforehand which environments produce valid evidence, what a demonstration must leave as a trail, who confirms a piece of evidence is genuine, and how conflicting evidence is resolved. Create a guideline for each. These rules describe evidence you have not yet seen, which is exactly the point of writing them now.

Check: Use the guidelines on a sample object. Can two people reach the same verdict on it without talking to each other first?

Salvador-Carulla and colleagues, writing in the TRL-IS adaptation published through PMC, rewrote this layer when they moved the readiness levels into implementation science. Laboratory testing was removed, the use of the operational environment was narrowed, and the distinction between a pilot in a relevant environment and a demonstration in the real world was made more sharply. Their adapted checklist then produced strong agreement between raters with different levels of experience. Evidence rules usually cause assessor disagreement.

4. Collect evidence from live work, not from documents

A document records intent. A demonstration records capability. Ask for what the work produced: Test records, logs, incident reports, release documentation, configuration files, operational data, and the outcomes users actually experienced provide evidence of readiness. Trace each item to a time and to the individual who produced it.

The scope of a technical readiness report set out by WireApps covers architecture, infrastructure, security posture, test coverage and operational readiness. Every one of those areas generates artefacts as a by-product of doing the work, whether or not anybody is assessing it. Collect those first and document second.

Check: you can point to the evidence that supports every point in the draft and to the demonstration the evidence came from.

5. Score only what the evidence shows

Rate the lowest piece of critical evidence. Log an unknown when evidence is missing. Keep the dissent in the log beside the rating if a reviewer disagrees, instead of resolving it in private.

The Aerospace Technology Institute, in its guidance on technology readiness levels, sets out the nine levels and views them as guidance because the definitions can be subjective and the scale imprecise. A framework that hides that imprecision is claiming a certainty it is unable to deliver.

Check: the rating sheet has one line per requirement, each line lists its supporting document, and items lacking a document are visibly blank.

6. Run the framework against the decision before the decision meeting

Dry run it. Give the finished assessment to the decision owner and inquire what they would do differently now. The framework is decoration if there is no answer.

Check: at least one risk has moved onto the decision agenda, and at least one assumption has been retired. This is the usefulness test from the GAO guide in practice, and it is the one most frameworks fail.

7. Decide, then keep the framework alive

Ratings expire. List the events that require a review: a change of environment, a change of configuration, a change in volume or load, a change of accountable owner, a change to the platform underneath. Each trigger has a title and a time assigned to it.

This matters because the scale has an edge that is easy to miss. QualityInspection.org says in its comparison of the levels with product development that a technology at the top of the readiness scale is not thereby ready for mass production, because pre-production preparation and manufacturing validation are beyond the scale altogether. The framework must explicitly state when volume affects a decision.

Which tool fits which job

The catalogue below follows the survey of readiness frameworks published by ITONICS, which counts fourteen in current use. Each entry states what it rates.

Technology readiness level assesses a single technology across nine steps, from basic principles observed to successful operation, and does not address cost, manufacturing or demand.
System readiness level rates a system assembled from subsystems, including how they integrate and whether they meet mission objectives together.
Integration readiness level rates the interfaces, data flows and technical compatibility between components, which matters most when parts arrive from different suppliers.
Manufacturing readiness level rates process capability, tooling, supply chain and repeatability at production volume.
Design readiness level rates the engineering design against production feasibility, materials, tolerances and assembly constraints.
Commercial readiness level rates demand fit, competitive position, pricing, regulatory status and customer validation.
The operational readiness level evaluates the processes, infrastructure, training and support needed to run the system at scale.
Market and technology readiness level combines technical maturity and commercial viability into one score, so novelty alone does not support a funding decision.
Data readiness rates whether data is available, reliable, labelled and fit for the decisions it will feed.
Service maturity rates a service under real users and real resource constraints, which is why health and public services reach for it.
Policy readiness rates a policy proposal against how implementable it is.
KTH innovation maturity assesses technology, market, team, intellectual assets and funding at once, and does not assume a straight line between them.
Dual-use readiness separates customer, funding and technology tracks for technologies with both commercial and defence applications.

IDC, a product design consultancy, reports that readiness levels are common in academia and venture capital while being little used and little understood across many areas of industry. That is the gap a working framework closes. Its value is not the score it produces. This common language lets engineering, management and finance argue about the same milestone and know they are arguing about the same objective.

The two tests a finished framework passes

First, a stranger can rerun it and reach a rating close enough to the original that the difference does not change the decision. The TRL-IS result shows this is achievable once the evidence rules are written properly.

Second, it changes a decision. If a framework has never moved a risk onto an agenda or retired an assumption, it is a filing exercise, and the next one should be built differently.

Sources

Assessments exist to establish whether technology is developed enough to be incorporated into a system without too much risk

GAO Technology Readiness Assessment Guide