Degree to which a model's mechanics can be understood; distinct from (and contested against) explainability.
In routine clinical use of decision-support tools, interpretability is judged at the bedside: a risk score or alert counts as interpretable when the treating clinician can see which patient variables drove it, check those drivers against the chart and the patient in front of them, and judge whether the pattern is clinically plausible. What matters is not access to model internals but whether the output can be integrated into clinical reasoning and safely overridden; a score that cannot be reconciled with clinical judgment is treated as uninterpretable regardless of vendor documentation.
Identify the patient variables driving a decision-support output, test them against clinical findings, and document the reasoning when following or overriding the tool.
External validation of the widely deployed Epic Sepsis Model (Wong et al. 2021) found poor discrimination and high alert burden; clinicians who could not see what drove alerts had little basis for calibrated overrides.
Clinicians divide over whether feature-level insight is required for safe use, or whether documented external validation and calibration evidence justify reliance on scores whose drivers they cannot inspect.
OmniGloss seed synthesis, 2026 (machine-drafted, pending expert validation)
NIST AI 100-3, 'The Language of Trustworthy AI' glossary, interpretability entry (attributing NSCAI: the ability to understand the value and accuracy of system output)
For patients and patient advocates, interpretability is measured at the point where an AI-informed decision reaches the person it affects: a model is interpretable only if its contribution to a diagnosis, triage ranking, or coverage decision can be conveyed in terms a patient can question during consent and shared decision-making. Understanding confined to data scientists or hospital committees does not count; the operative test is whether the patient can ask why this recommendation was made about them and receive an answer that supports agreeing, refusing, or seeking a second opinion.
Demand and use plain-language accounts of what drove an AI-informed recommendation when consenting to, refusing, or challenging a proposed course of care.
e.g., a patient told that a transplant-listing score weighed her prior admissions can contest an error in her record; a patient told only that 'the algorithm decided' cannot.
Anchors the affected-person-intelligibility position in the documented dissensus over whose understanding interpretability must serve, against the view that expert-level understanding by clinicians and reviewers is sufficient.
EU High-Level Expert Group on AI, Ethics Guidelines for Trustworthy AI (2019)
For hospital leadership deciding whether to adopt a clinical AI tool, interpretability is a procurement and deployment criterion: the committee asks whether the tool's logic can be understood well enough by the clinicians who will use it to support training, safe-use protocols, and incident review. In practice this is assessed through vendor documentation of input variables and model behavior, demonstrations of typical and edge cases, and whether failure modes can be anticipated and assigned to a monitoring plan. The reference audience is the professional user; patient-facing explanation is handled as a consent and communication matter, not an adoption test. A tool whose behavior cannot be characterized for its intended users is rejected or confined to low-stakes workflows.
Weigh vendor evidence about a tool's understandability against intended clinical use, and set deployment scope, training, and monitoring requirements accordingly before authorizing adoption.
e.g., a hospital AI committee approves an interpretable regression-based deterioration score for general wards but restricts a black-box imaging triage tool to use with mandatory radiologist review.
Anchors the expert/professional-audience position in the documented dissensus over whose understanding interpretability must serve: the committee's test is clinician comprehension, with patient-facing intelligibility treated as a separate consent duty.
NIST AI Risk Management Framework 1.0 (2023)
Among developers of clinical prediction models, interpretability is a property of the model class chosen before training: models whose parameters can be read directly against clinical knowledge, such as sparse scoring systems, logistic regression with clinically curated variables, or generalized additive models whose risk curves can be plotted and reviewed. On this operationalization, a post-hoc explanation attached to a black box is not interpretability but an approximation of it; for high-stakes clinical decisions the build requirement is a model a clinician-reviewer can inspect variable by variable, with any accuracy cost measured and justified rather than assumed.
Select and fit model classes whose learned parameters clinicians can inspect directly, and quantify any performance difference against black-box alternatives before claiming an accuracy justification for opacity.
Rudin (2019) argues that for high-stakes decisions inherently interpretable models should be built instead of explained black boxes, reporting that accuracy losses are often negligible on structured clinical and justice data.
Anchors the inherent-interpretability position in the documented dissensus over whether interpretability is a property of transparent model classes or achievable post hoc on black boxes.
Interpretable-ML literature (inherent vs. post-hoc debate; e.g., Rudin 2019; Lipton 2018)
inherent vs. post-hoc interpretability distinction
C. Rudin (2018). Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence
In medical-device regulatory practice, interpretability is handled as a labeling and risk-management obligation for AI-enabled software: reviewers ask whether the sponsor has characterized how the device output is generated, including intended inputs, the logic or model basis, and known failure modes, clearly enough for the intended user to interpret the output within the clinical workflow, and whether that account is supported by verification and validation evidence. Interpretability claims that cannot be tied to documented device behavior are treated as unsupported labeling; opacity that prevents users from recognizing malfunction becomes a safety finding.
Assess whether device documentation lets the intended clinical user understand what generated an output and recognize when it should not be relied on, and require remediation where it does not.
The FDA/Health Canada/MHRA Good Machine Learning Practice guiding principles (2021) call for users to be provided clear, essential information about the model's basis, inputs, and limitations.
FDA AI/ML-based Software as a Medical Device materials, incl. Good Machine Learning Practice: Guiding Principles (2021)
GMLP guiding principles (transparency to users)
For credit officers and underwriters working with scoring models, interpretability is operational when the score decomposes into reason codes and factor contributions the officer can act on: explain an adverse decision to a customer in the required form, judge whether a borderline case warrants a documented override, and spot when the score is reacting to stale or erroneous bureau data. The officer does not need the estimation method; the working test is whether each score comes with ranked drivers consistent with the file, so the human step in the credit decision adds control rather than rubber-stamping.
Translate score drivers into accurate adverse-action reasons and justified overrides, and escalate cases where the stated drivers contradict the customer file.
e.g., an underwriter seeing 'high revolving utilization' as the top driver can verify it against the bureau report before issuing the adverse-action notice, and escalate if utilization was recently paid down.
For model risk committees and senior model owners in banking, interpretability means the model can be effectively challenged by qualified internal parties: its conceptual soundness can be assessed, its sensitivities to key inputs demonstrated, and its limitations understood well enough for the committee to accept model risk and set use restrictions. The reference audience is the second line and the board, not the retail customer; a model is interpretable enough to approve when independent, technically competent reviewers can understand and criticize its design and behavior, whatever the customer-facing explanation later says.
Evaluate whether independent qualified reviewers can understand and challenge a model's design and behavior, and approve, restrict, or reject its use on that basis.
e.g., a risk committee approves a gradient-boosted default model for capital planning because validation demonstrated its sensitivities, but caps its use in pricing where behavior was less well characterized.
Anchors the expert-oversight-sufficiency position in the documented dissensus over whose understanding interpretability must serve; customer-facing intelligibility is treated as a separate communication duty.
Federal Reserve SR 11-7, Supervisory Guidance on Model Risk Management (2011)
effective challenge; model development, implementation, and use
Among credit- and fraud-model developers, interpretability is achieved through engineered constraints and attribution tooling layered on performant learners: monotonicity constraints that force sensible directionality, feature-importance and Shapley-value decompositions that yield per-decision reason codes, and partial-dependence analyses filed with model documentation. On this operationalization a gradient-boosted model is interpretable when its behavior is characterized, meaning attributions are stable across resamples, respond correctly to controlled input perturbations, and reconcile with domain expectations; restricting the model class is viewed as an unnecessary accuracy sacrifice.
Implement monotonicity constraints and per-decision attribution methods, verify attribution stability under perturbation and resampling, and package the evidence for validation and reason-code generation.
e.g., a fraud team ships an XGBoost model with SHAP-based reason codes and a monotonicity test suite, treating those artifacts as the model's interpretability evidence.
Anchors the post-hoc-characterization position in the documented dissensus over whether interpretability requires inherently transparent model classes or can be delivered by evidenced characterization of a black box.
Interpretable-ML literature (post-hoc attribution methods; e.g., Lipton 2018; Doshi-Velez & Kim 2017)
post-hoc attribution and model-agnostic explanation methods
In model validation practice, interpretability is an evidentiary chain the validator can traverse: complete development documentation, replicable code and data lineage, benchmark and sensitivity results, and documented limitations, sufficient for an independent reviewer to reconstruct why the model behaves as it does and where it fails. Validators operationalize it through the artifacts the second line can test, not the developer's intuition, and record an interpretability weakness as a validation finding that limits model tier or use. What a black-box explanation tool proves is judged by whether its outputs can themselves be replicated and stress-tested.
Reconstruct model behavior independently from documentation, code, and data lineage; test claimed explanations for replicability; and record unexplainable behavior as a finding with use restrictions.
e.g., a validator reruns the developer's attribution analysis on a holdout vintage; when top drivers reorder materially, the model is approved only with tightened monitoring triggers.
Validation teams differ on whether post-hoc attribution reports constitute acceptable evidence of conceptual soundness or merely descriptive supplements; treatment of SHAP-style artifacts varies across institutions and reviewers.
model validation: conceptual soundness, outcomes analysis
For conduct supervisors and data-protection officers overseeing consumer-facing credit decisions, interpretability is defined by the legal sufficiency of the account given to the affected customer: automated decisions must be traceable to meaningful information about the logic involved, concrete enough to support the customer's rights to contest the decision and obtain human intervention under GDPR Article 22. Internal expert understanding does not discharge the duty; the operative test is whether the firm can produce, for each decision, specific and intelligible reasons that a customer, a supervisor, or a court could act upon.
Verify that each automated consumer decision can be accompanied by meaningful, decision-specific information about the logic involved, and treat inability to produce it as a compliance failure.
The CJEU's SCHUFA judgment (C-634/21, 2023) treated credit scoring as an automated decision under Article 22, sharpening the demand that firms explain score logic to affected individuals.
Live variance over how specific 'meaningful information about the logic involved' must be, ranging from general model logic to decision-specific reasons, with data-protection authorities, courts, and firms taking different lines.
In newsroom practice, an AI tool is interpretable when the journalist using it can account for its output well enough to stand behind the story: knowing broadly what sources or data the tool draws on, why it surfaced or generated a particular claim, and where it is likely to fail, so every machine-produced element can be independently verified before publication. The test is editorial rather than technical: if a reporter cannot reconstruct how a lead, summary, or transcript came about, the output is treated as an unverified tip, not as reporting.
Trace how an AI tool produced a claim or artifact, verify it against independent sources, and treat outputs whose origins cannot be reconstructed as unusable for publication.
e.g., a data journalist using a document-classification model reports only findings she can re-verify in the underlying documents, treating unexplained classifications as leads to check rather than facts.
For editors-in-chief and creative directors deciding where generative tools may enter production, interpretability is what makes responsibility assignable: the organization must understand a tool's behavior, including how outputs depend on prompts and training data, how often it fabricates, and whether it reproduces protected material, well enough to decide which uses are compatible with authorship claims, AI-content disclosure duties, and the outlet's liability for what it publishes. A tool whose behavior cannot be characterized at this level is confined to uses where a human fully reworks and answers for the output.
Authorize generative-tool uses only where the tool's behavior is understood well enough to assign authorship and liability, and set disclosure and human-review requirements for each permitted use.
e.g., a magazine permits AI-assisted headline brainstorming but bars unreviewed AI drafting of factual copy after finding it cannot predict when the tool invents quotes.
Art. 50 (transparency obligations for certain AI systems)
For engineers building recommendation and generative pipelines in media, interpretability is pursued through instrumentation of systems whose internals resist direct reading: offline feature attribution on ranking models, ablation and counterfactual prompt tests on generative components, and telemetry linking output changes to model or data updates. The working standard is behavioral: a system is interpretable to the degree the team can predict how ranking or generation will shift under a defined change and can localize regressions to a component. Full mechanistic understanding of foundation-model internals is treated as a research aspiration, not a shipping requirement.
Instrument ranking and generative components so output changes can be attributed to inputs, prompts, or model versions, and maintain tests that predict behavior under defined changes.
e.g., a streaming service's ranking team maintains counterfactual test suites showing how recommendations shift when watch-history features are ablated, using them to explain ranking changes to editorial stakeholders.
Anchors the behavioral-characterization position in the documented dissensus over interpretability's boundary: predictive control over input-output behavior is claimed as interpretability without transparent internals, a usage mechanistic-interpretability proponents and inherent-model advocates reject.
NIST AI 100-3, 'The Language of Trustworthy AI' glossary, interpretability entry (attributing NSCAI: predicting what happens given a change in input or parameters)
For standards editors and ombudspeople reviewing algorithmically shaped media, interpretability is the capacity to give the public an honest account of why the system did what it did: why a story was promoted, a creator demonetized, an image generated as it was. It is assessed against audience trust rather than engineering access; an account must identify the operative factors in terms an affected creator or reader can contest, and must expose whose interests the system's objectives encode. Opacity here is a power problem: what cannot be accounted for publicly cannot be legitimately imposed on audiences.
Assess whether the organization can publicly account for algorithmic choices affecting audiences and creators, and press for changes where the operative factors cannot be honestly stated.
e.g., an ombudsman investigating complaints that a platform buries certain outlets asks the operator to state the ranking factors involved; inability to answer is itself the finding.
In case handling, an algorithmic score or flag is interpretable when the caseworker can turn it into reasons that belong in the case file: which recorded circumstances of this applicant triggered the flag, what would change the outcome, and whether those grounds are legally relevant to the decision at hand. Administrative practice demands decision-specific grounds, because the caseworker rather than the system signs the decision and must defend it in objection proceedings. An output that cannot be converted into stateable reasons cannot lawfully carry weight in the file.
Convert algorithmic outputs into decision-specific, legally relevant reasons recorded in the case file, and set aside outputs whose grounds cannot be stated and defended.
e.g., a benefits caseworker who can see a fraud-risk flag rests on rapid address changes can weigh and record that ground; a bare risk number is set aside.
For officials who answer for an agency's decisions, interpretability is the preserved capacity of the administration to give reasons: a system may support decisions only if the agency can explain each affected outcome to the citizen, the objection board, and ultimately a court, in terms of the legal criteria the decision must rest on. Understanding held solely by the vendor or the data-science unit does not satisfy this; the duty to state reasons attaches to the decision as received by the citizen. Systems that cannot sustain citizen-facing reason-giving are not authorized for determinative use.
Authorize algorithmic support only where each affected decision can be explained to citizens and courts in legally relevant terms, and withdraw systems that erode the duty to give reasons.
The Dutch SyRI welfare-fraud system was struck down by The Hague District Court in 2020 partly because its risk indications could not be meaningfully examined or contested by those affected.
Anchors the affected-person-intelligibility position in the documented dissensus over whose understanding interpretability must serve: reason-giving to the citizen, not expert comprehension, is the operative test.
Art. 86 (right to explanation of individual decision-making)
Among data scientists building models for public bodies, interpretability is commonly enforced as a design constraint: prefer model classes whose decision logic can be printed into the administrative record, such as rule lists, scorecards, and penalized regressions, because every automated influence on a citizen's case may have to survive objection proceedings, freedom-of-information requests, and judicial review. Choosing a black box and attaching explanations afterwards imports litigation and legitimacy risk; the marginal accuracy of an opaque model rarely outweighs a decision the agency cannot fully defend, so inherent transparency is the default and deviation requires explicit justification.
Default to model classes whose full decision logic can be disclosed in administrative and judicial proceedings, and document explicit justification whenever an opaque model is proposed.
e.g., a municipality replaces a black-box risk model for benefit-fraud triage with a published scorecard so that objection boards can review the exact criteria applied.
Interpretable-ML literature (inherent vs. post-hoc debate; e.g., Rudin 2019)
In official statistics, interpretability travels through methodological documentation rather than model inspection alone: when statistical offices adopt machine-learning methods for imputation, coding, or nowcasting, the method counts as interpretable when its assumptions, inputs, and behavior are documented to professional standards, its influence on published figures can be quantified, and the methodology can be explained and defended publicly under the sound-methodology and accessibility commitments of the European Statistics Code of Practice. The operative artifacts are methodological reports and quality declarations that let users and peer reviewers understand how estimates were produced.
Document model assumptions, inputs, and effects on published statistics to code-of-practice standards, and publish methodology so users and peer reviewers can understand and challenge estimates.
e.g., a statistical office introducing ML-based occupation coding publishes the method, its error profile against manual coding, and its estimated effect on published employment series.
European Statistics Code of Practice (2017 revision)
Principle 7 (sound methodology); Principle 15 (accessibility and clarity)
For public-sector algorithm auditors, interpretability is an access property of the whole deployment, not just the model: whether the auditor can obtain the documentation, code, training-data lineage, and versioned decision logs needed to reconstruct how the system produced its outputs during the audited period, and whether responsible officials can themselves explain the system they operate. Audit practice operationalizes it as a checklist of inspectable artifacts; a system whose behavior cannot be reconstructed from retained records is reported as unauditable, a governance finding against the agency independent of the model's technical merits.
Verify that documentation, versioned models, and decision logs permit reconstruction of past outputs, and report systems that cannot be reconstructed as unauditable governance failures.
The Netherlands Court of Audit's 2022 review of algorithms used by central government found several systems whose operation the responsible ministries could not adequately explain or document.
For working creators whose income depends on platform recommendation systems, interpretability is the practical ability to form a reliable working theory of the ranking system that distributes their work: what the system rewards, what triggers suppression, and whether observed reach changes reflect their own choices or opaque platform updates. Because platforms disclose little, creators operationalize interpretability collectively, comparing analytics, running informal experiments, and trading folk knowledge; the gap between these folk theories and the system's actual mechanics is itself the grievance, since livelihoods are governed by mechanics no one outside the platform can inspect.
Track how content changes correlate with reach, distinguish evidenced patterns from folklore, and identify decisions that require explanation from the platform rather than speculation.
e.g., video creators systematically A/B-testing thumbnails and upload times to reverse-engineer recommendation behavior, while acknowledging that an unannounced algorithm change can invalidate every inferred rule.
Creators, platforms, and researchers dispute whether analytics dashboards plus actionable heuristics amount to interpretability, or whether only disclosure of the actual ranking factors would; the validity of creator folk theories is itself debated.
Among medical-imaging AI developers, interpretability is commonly operationalized as post-hoc visual evidence: saliency maps, Grad-CAM heat maps, and prototype or attention displays intended to show clinicians where in the image the model 'looked'. The build practice treats an overlay that highlights clinically sensible regions as interpretability delivered at the point of care. This operationalization is under active challenge from within the field, as studies show saliency methods can be unstable, only loosely tied to the model's computation, and persuasive to clinicians even when wrong, so teams increasingly pair overlays with quantitative faithfulness checks.
Validate that visual explanation outputs faithfully track model computation before exposing them to clinicians, and measure their effect on user reliance, not only their plausibility.
Ghassemi, Oakden-Rayner and Beam (2021) argue that current explainability methods in healthcare offer 'false hope', showing saliency-style outputs can look similar for correct and incorrect predictions.
Imaging developers split on whether saliency overlays constitute clinical interpretability or misleading reassurance; published evaluations argue current post-hoc explanations in healthcare cannot bear the safety weight placed on them.
Interpretable-ML literature (saliency-method faithfulness critiques)
post-hoc saliency methods and faithfulness evaluation
Julius Adebayo et al. (2018). Sanity Checks for Saliency Maps. Unknown Journal
In agronomic modelling, interpretability is the degree to which model mechanics map onto biophysical meaning: vegetation indices tied to canopy state, phenological parameters corresponding to green-up and harvest, process parameters with units an agronomist can sanity-check against crop physiology. A deep network on raw time series may score higher yet offer no such handles; interpretable structure is what lets domain experts detect when a model is right for the wrong reasons — fitting an irrigation artefact rather than crop growth.
Prefer and demand model structures whose parameters carry biophysical meaning, and use that structure to check whether predictions rest on agronomic signal rather than artefacts.
e.g., an NDVI-threshold phenology model whose parameters correspond to observable green-up and mowing dates is retained over a marginally more accurate deep network that offers no agronomic handle for error analysis.
NIST AI 100-3, The Language of Trustworthy AI: An In-Depth Glossary of Terms
interpretability entry: degree to which a model's workings can be presented in terms understandable to a human
For test, evaluation, and accreditation staff, interpretability is the degree to which a system's input-output behavior can be understood and predicted well enough to certify it: given a change in input, sensor picture, or parameters, can the evaluator anticipate what the system will do? It is assessed at the mechanism level, through inspectable architectures, sensitivity analyses, and behavioral probing across the operational envelope, and is distinguished from explainability, which concerns run-time reasons given to an operator. Low interpretability directly narrows the conditions under which an autonomous function can be accredited and the authorities delegated to it.
Probe and document how a system's outputs respond to input and parameter changes across the operational envelope, and scale accredited autonomy to the level of understanding achieved.
E.g., an accreditation team maps an onboard classifier's behavior across depression angles, obscuration levels, and decoy densities before certifying it for cueing, but not for autonomous engagement.
entry: interpretability (NSCAI definition, predicting effect of input changes)
In learning analytics, interpretability is a design constraint on the models themselves: because teachers and advisers, not data scientists, act on the outputs, the field shows a deliberate preference for models whose mechanics can be inspected — small feature sets, monotonic relationships, rule lists — over marginally more accurate black boxes. A model is operationally interpretable when an educator can see which learner behaviors drive a prediction, check that those drivers are pedagogically plausible rather than proxies for demographics, and anticipate how the prediction would change if the student's behavior changed. Post-hoc explanation layers are treated as a weaker substitute for this property.
Prefer inspectable learner models where stakes are high, verify that predictive drivers are pedagogically plausible rather than demographic proxies, and reject models whose mechanics educators cannot examine.
e.g., an advising team that replaces a gradient-boosted dropout model with a five-feature rule list, accepting slightly lower accuracy so advisers can see and challenge why each student was flagged.
Interpretability literature (argument for inherently interpretable models in high-stakes decisions)
preference for inherently interpretable models over post-hoc explanation in high-stakes settings
Process engineers prefer models whose parameters mean something in engineering units: a regression coefficient in microns per degree, a control chart whose limits come from process capability, a transfer function with poles you can relate to the mechanism. Interpretability, in this practice, is the degree to which a model's internal structure maps onto process physics — so that when the model surprises you, the surprise is a hypothesis about the machine rather than a curiosity about the network. Where black-box models win on accuracy, plants often deploy them for detection while keeping an interpretable model for diagnosis and control.
Choose model classes whose parameters map to process variables when the model informs control decisions, and document what each significant parameter means physically.
e.g., a plant keeps a linear thermal-compensation model on its machining centers — its two coefficients are physically meaningful and checkable — while a neural network only screens parts for later CMM measurement.
Interpretability literature (Doshi-Velez and Kim 2017; Lipton 2018)
interpretability as the degree to which a model's mechanics can be understood by a human
For legal-tech and expert-witness practice, interpretability is a property of the method itself: the path from input to output can be reconstructed and stated — coefficients, search terms, decision rules — without post-hoc rationalization. It matters because litigation reverse-engineers decisions: a scrutable method lets counsel show exactly why a document was ranked responsive or a figure produced, while an opaque one leaves only after-the-fact explanation that opposing experts will characterize as speculation. Where outputs must be defended under oath, practice therefore prefers inherently interpretable methods over marginally more accurate opaque ones, a preference the interpretable-ML literature argues is rational for high-stakes decisions.
When selecting methods whose outputs may be litigated, weigh interpretability as an admissibility and cross-examination property, and record why any opaque method was chosen over a scrutable alternative.
e.g., for a class-certification damages analysis, an expert adopts an explicit scoring formula rather than a neural network, accepting a small accuracy loss so every step can be defended on the stand.
Interpretable-ML literature (Rudin 2019; Doshi-Velez & Kim 2017)
argument for inherently interpretable models in high-stakes decisions
For the engineers who run routing and ETA systems, interpretability is being able to trace mechanically how an output came about: which constraints and cost terms drove the solver to a tour, which features moved an ETA, and what will change if a parameter such as stop time, speed profile, or cutoff is adjusted. Inspectable components, gradient-boosted ETA models with legible features or solvers with explicit constraints, are preferred over opaque ones precisely because dispatch disputes require replaying a decision and what-if analysis is a daily planning tool.
Choose model and solver formulations whose decisions can be replayed and perturbed, document constraint and feature effects, and support the what-if queries planners actually run.
e.g., replaying yesterday's solver run with the closed dock removed to show a customer that the missed slot came from their own late gate closure, not from the plan.
interpretability (NSCAI): extent to which cause and effect can be observed and effects of input or parameter changes predicted
Among gig workers comparing notes in forums and the platform data teams who tune the dispatcher, interpretability is the degree to which the allocation machinery can be worked out as usable cause and effect: does declining three orders in a row demonstrably cut your offer rate, does going offline at surge hurt tomorrow's dispatch, which inputs actually move the ranking? Where the scoring is a black box, workers substitute folk theories traded in group chats and car parks; an interpretable system is one where the predicted consequence of a worker's action matches what the dispatcher then does.
Probe how specific inputs — acceptance rate, ratings, hours online — change a system's allocation outputs, and distinguish evidenced mechanisms from folk theory before advising workers or management.
e.g., couriers test the belief that rejecting cash orders lowers dispatch priority by logging offers across paired accounts, finding no measurable effect — folk theory corrected by probing.
In marketing measurement, interpretability is the degree to which a budget-moving model's mechanics can be read and argued with by the people who own the budget: marketing-mix coefficients an econometrician can defend to a CFO, uplift-tree splits a CRM lead can sanity-check against category knowledge, monotonic constraints that keep a price-elasticity curve believable. It matters because reallocating media spend is a boardroom negotiation — a channel-attribution model whose parameters cannot be inspected loses to a simpler regression whose story survives cross-examination, whatever the holdout metrics say.
Choose model forms whose parameters the budget owner can inspect and challenge, verify learned effects against known commercial mechanics, and refuse spend reallocations justified only by uninspectable scores.
e.g., a marketing-mix model shows a negative TV coefficient; because the regression is inspectable, analysts trace it to a collinear sponsorship variable rather than concluding that TV destroys sales.
Doshi-Velez and Kim (2017), Towards a Rigorous Science of Interpretable Machine Learning
intrinsically interpretable models versus post-hoc explanation
In methodological work, interpretability is a property of the model class rather than of an added explanation layer: the fitted object is small enough, and its parameters carry enough meaning, that a researcher can read the mapping from inputs to outputs directly, as with a sparse regression reported with coefficients and intervals, a shallow decision list, or a generalized additive model with plotted shape functions. It is operationalized through constraints imposed before fitting, such as sparsity, monotonicity, or additivity, and reported as a modelling choice with its accuracy cost, so readers can judge the trade between inspecting a model and merely probing it.
Decide before fitting whether the claim requires an inspectable model, impose the constraints that deliver it, and report the accuracy cost of that choice explicitly.
e.g., a risk score published as a scoring table with integer weights that reviewers can check against domain knowledge, in place of a gradient-boosted ensemble carrying post-hoc attributions.
One camp holds that high-stakes settings demand inherently interpretable models because post-hoc explanations of opaque models are unreliable; another treats interpretability and post-hoc explainability as interchangeable routes to understanding. The two disagree over whether an explained black box counts as interpretable at all.
Interpretable-ML / XAI research literature (registry anchor)
inherently interpretable models for high-stakes decisions; critiques of post-hoc explanation fidelity
W. James Murdoch et al. (2019). Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences
Among ML researchers and engineers, interpretability names what you can verify about a model's internal mechanics: which features, circuits, or components drive which behaviors, established through ablations, probing, and mechanistic analysis rather than post-hoc stories. Builders operationalize it as a property purchased at design time - simpler model classes, monotonic constraints, inspectable intermediate representations - and distinguish it from explainability, which packages accounts of behavior for an audience. The distinction is enforced unevenly, even inside the profession.
State whether a use case needs inspectable mechanics or only audience-facing explanations, choose the model class accordingly, and verify claimed understanding with ablation or probing experiments.
e.g., a team replaces a gradient-boosted risk model with a monotonic, constrained variant so reviewers can verify that raising income never lowers the score - interpretability purchased at a measured accuracy cost.
What varies is the boundary with explainability: mechanistic-interpretability researchers reserve the term for causal understanding of model internals, while much applied practice uses it interchangeably with post-hoc explanation tooling.
NIST AI 100-3, The Language of Trustworthy AI glossary
interpretability entries (NSCAI; Google ML glossary)