What is explainable AI (XAI)?

Explainable AI (XAI) is the set of methods and design choices that let people understand why an AI system produced a particular output. NIST describes four principles: the system gives reasons, the reasons make sense to the intended user, they accurately reflect how the output was produced, and the system stays within its limits.

Why it matters for your business

When an AI system helps decide who gets a loan, an interview or a refund, someone will eventually ask why. Customers, staff, auditors and regulators all do. Without an explanation you cannot check a decision for errors, correct it or defend it.

There is a legal side too. Article 86 of the EU AI Act lets a person affected by a decision that an organisation takes on the basis of the output of certain high-risk AI systems listed in Annex III (excluding those in point 2 of that Annex) ask that organisation for a clear and meaningful explanation of the system's role in the decision and the main elements of the decision. The decision must have legal effects or similarly significantly affect the person, and the person must consider that it adversely affects their health, safety or fundamental rights. The right applies only where Union law does not already provide it, and Union or national law can set exceptions.

How it works

Some models are simple enough to read directly, such as a short scoring rule. For complex models, a second technique estimates which inputs mattered most, or the system shows the evidence it relied on, such as the documents it retrieved. NIST's four principles give a practical test: does the system give reasons, are they understandable to the person receiving them, do they accurately reflect how the output was produced, and does the system operate only under the conditions it was designed for.

Example

Picture a 120-person equipment-finance company that uses a model to flag late-payment risk. Instead of a bare score, the tool shows the three factors that pushed the score up (illustrative: payment history, order size, sector) and the records behind them. A credit controller can spot a wrong input, fix it and explain the outcome to the customer.

Costs, limits and when not to use it

Explanations take build and review time, and simpler, more readable models can be less accurate than complex ones. An explanation can also be convincing but wrong, which is why NIST treats explanation accuracy as a separate principle: test that the explanation reflects what the system actually did. For low-stakes drafting, checking the output is usually enough.

Explanations support human oversight but do not replace it: an explanation helps a reviewer judge an output, while the power to override is a separate control.

As of November 2026, the European Commission says the rules for Annex III high-risk systems have been extended to 2 December 2027 following the AI Omnibus. Check which obligations apply to your use. This is general information and not legal advice.