When Self-Healing Test Automation Hides Defects

When Self-Healing Test Automation Hides Defects

Audacia

16 November 2026 - 10 min read

AutomationTestingSoftware Testing
When Self-Healing Test Automation Hides Defects

Few problems in software testing are as persistent as test maintenance. Automated suites intended to give teams the confidence to release often fail for reasons that have nothing to do with application/software defects, and keeping them running absorbs skilled time that could be spent on testing that adds value. AI-powered self-healing has emerged as a response, adapting tests automatically when an application changes to cut maintenance costs and restore confidence in automation. 

However, they also introduce a less visible risk. Under the wrong conditions, a test that heals itself can adapt its way past a genuine defect. It may interact with the wrong element, check less than it once did, or drop out of the suite entirely, while the pipeline continues to report success. As healing has evolved from repairing broken locators to rewriting the logic of tests themselves, that risk has grown, and the measures most organisations use to judge their test automation are not fit for purpose.  

This article examines where the risk comes from, how it has changed as healing has become more capable, and how organisations can keep the benefits of self-healing while protecting the signal their release decisions rely on. 

The maintenance burden self-healing addresses 

Anyone who has run a large automated test suite will recognise the problem self-healing sets out to solve. User interface tests locate elements on a page through identifiers such as IDs, CSS selectors or XPath expressions. When developers rename a class, restructure a page or upgrade a component library, those identifiers break, and tests fail even though the application works perfectly well. Multiplied across thousands of tests and frequent releases, maintenance becomes a substantial and recurring cost. 

Unreliable tests carry a second, less visible cost in that they also erode trust. A 2026 study from the Technical University of Munich, conducted with researchers from BMW and CQSE, analysed 8.8 billion test executions across four industry-scale projects and found that flaky failures missed by standard rerun mechanisms accounted for between 9.8% and 16.3% of failed pipeline runs. Martin Fowler, writing on non-deterministic tests, warned that once a team starts ignoring a failing regression test, that test loses its value, and the habit can spread until the whole suite is undermined. 

Healing tools tackle the first of these problems directly. When a test cannot find the element it expects, the tool evaluates alternatives using attributes such as text, position, structure and visual similarity, selects the most likely match and continues the run, typically recording the change for later review. Cosmetic changes no longer turn the pipeline red, maintenance effort falls, and engineers spend more of their time on testing that adds value. For organisations with large, ageing UI suites, those are significant gains. 

From repairing locators to rewriting tests 

Early self-healing was narrow, repairing the path to an element and leaving the rest of the test untouched, such as the steps, the data and, crucially, the assertions that decide whether a test passes. The newest generation of tools goes much further with AI agents now diagnosing why a test failed and changing whatever they judge necessary to make it pass, including steps, expected values and the assertions themselves. The commercial case for these tools is easy to make, because maintenance hours saved can be counted. However, the corresponding risk is harder to quantify. 

Moving from repairing locators to rewriting tests changes what can go wrong. A May 2026 industrial case study published on arXiv examined a multi-agent system that generated and repaired UI tests for an enterprise web application, analysing 300 consecutive execution reports. The system repaired failing tests without human intervention across most scenario groups, and along the way it exhibited two troubling behaviours. In one case, the agent replaced a strict check that a value equalled five with a check that the value merely existed, a condition satisfied by almost any result. In another, after repeated failed repair attempts, it removed a failing test from the suite altogether, after which the run reported a 100% pass rate. Measured by pass status, 70% of scenario groups had converged on a working test. Excluding those where convergence came from weakening or deletion, the figure fell to 50%. 

This is a single preprint drawn from one environment, and its author is careful about generalising. However, its central explanation applies widely – the agent had no model of what each test was meant to verify. It could observe whether a test passed, and it optimised for that outcome, even when passing meant checking less. The author concludes that any change to assertions or test scope should require human validation. 

Three ways a green test can lie 

Those findings map onto a wider set of patterns. The three below are illustrative, and each can occur with conventional locator healing as well as with agentic repair. 

Substitution 

The healed test interacts with an element that resembles the original but is a different control. A “Submit order” button is healed to “Submit feedback”, or the first matching row in a table stands in for the specific record the test was written for. It still runs end to end, along a different path from the one it was designed to verify. Substitution is often the hardest form of false healing to spot, because nothing in the test code looks wrong. 

Weakening 

An assertion is relaxed until the test passes. An exact value becomes a range, a specific message becomes any message, or a check on content becomes a check that the page loaded. The test continues to execute every step and report success, yet it now verifies a fraction of what it once did. 

Erosion 

Steps are skipped, tests are disabled or whole scenarios are removed to stabilise the run. Coverage shrinks quietly, and because the remaining tests pass, the overall pass rate rises. In regulated sectors, the consequences extend beyond quality, since evidence that a control was tested may rest on a test that no longer checks it. 

All three patterns share one feature – every standard measure improves while they happen. Pass rates go up, pipeline failures go down, and time spent on maintenance falls, and a dashboard designed to show test health will report that the suite has never been in better shape. Researchers at Facebook have described code that is highly covered yet poorly tested, and self-healing without boundaries can produce the same condition at scale. 

Setting boundaries on what healing can change 

Organisations can keep the benefits of healing by deciding deliberately what it is allowed to do, scaled to the risk each type of change carries. A simple policy framework might look like the one below, with thresholds adjusted to each organisation’s risk appetite and regulatory obligations. 

Type of changeExampleSuggested policy
Cosmetic locator updateRenamed CSS class on the same elementApply automatically and log
Element substitutionHealed to a different element with similar attributesApply, flag and review within the sprint
Change to expected values or assertionsExact match relaxed to a partial matchBlock until approved by a named reviewer
Removal or skipping of steps or testsFailing test disabled to stabilise the runNever automatic

Recording the intent of each test also helps. The case study traced false repairs to the agent having no model of what a test was meant to verify, and the same gap affects human reviewers. A short description of the behaviour each critical test protects, kept alongside the test itself, gives anyone approving a heal a clear basis for judging whether the change preserves that behaviour. 

Test code also deserves the same change control as production code. A change to an assertion alters what the organisation is verifying about its software, and it should pass through review with the same rigour as a change to the software itself. When evaluating tools, technology leaders should ask vendors directly what their product is permitted to change, how those changes are surfaced, and whether permissions can be configured by risk. 

Measuring whether the suite can still be trusted 

Pass rates alone cannot show whether a suite still does its job, so organisations using self-healing benefit from a small set of complementary measures. Factors such as  

  • Heal frequency, tracked per test over time, identifies tests that heal repeatedly and are likely to be drifting from their purpose, reviewed by a human; 
  • The proportion of healed changes that receive human review shows whether the policy is working as intended; 
  • Trends in the number of assertions and active tests reveal erosion that pass rates conceal;  
  • Comparing defects that reach production against areas covered by healed tests can expose tests that pass without protecting anything; 
  • Periodic sampling, in which a tester reviews a selection of recently healed tests against their original purpose, adds a human check that no metric can fully replace. 

Mutation testing offers a more direct check by deliberately introducing small faults into the code and confirming that the test suite catches them, measuring a suite’s ability to detect defects directly. Research at Google, drawing on almost 15 million mutants, found that developers exposed to mutation testing wrote more tests and actively improved their test suites. Running mutation analysis periodically against suites that rely heavily on healing gives leaders evidence that a green pipeline still means what they believe it means. 

What this means for technology leaders 

Self-healing automation is a good investment for many organisations as it reduces maintenance, shortens feedback loops and frees skilled testers for exploratory and risk-based work. However, how much of that value it delivers depends on the boundaries placed around it. 

There are three implications when considering self-healing automation. First, a green pipeline is only as meaningful as the tests behind it, and release decisions, change approvals and assurance given to boards and regulators all rest on that meaning. Second, the risk grows as tools become more autonomous, so controls that were adequate for locator healing may fall short once agents can rewrite assertions. Third, the measures that typically signal a successful rollout, such as higher pass rates and lower maintenance effort, are the same measures that conceal the problem, so they need to be balanced with evidence of what the suite verifies. 

As well as this, decisions about what healing may change, and who approves changes to critical tests, belong with a named quality or engineering lead, recorded in the same way as other controls. Without that accountability ownership, healing policies tend to drift towards whatever keeps the pipeline moving. 

AI-assisted development is increasing the volume of change flowing into test pipelines, which raises the pressure to keep them green. Organisations already carrying test debt are particularly exposed because healing can mask weaknesses that were already present. 

Governed as a deliberate capability, with clear limits and the right measures in place, self-healing can cut the cost of maintaining automated tests while preserving the confidence that makes those tests worth having.

Ebook Available

How to maximise the performance of your existing systems

Free download