AI in software testing means using language models and agents to write, run and analyze tests – from unit tests, through E2E scenarios in Playwright, to automated code review – as well as testing the AI features of an application itself with evaluations and security tests. AI noticeably speeds up test automation, but it doesn't relieve the team of deciding what to test and why. Used badly, it can produce hundreds of tests that pass yet check nothing that matters.
In this article we discuss how AI helps with particular types of tests, how to test applications that themselves use language models, and which parts of the testing process are not worth handing over to a machine.
Software testing and AI – two different topics
The phrase "testing with AI" covers two things that are worth separating:
- AI as a tool for testers and developers – generating unit and E2E tests, analyzing failures, code review, preparing test data. This applies to any application, including one with no AI features at all.
- Testing AI features in an application – checking whether a chatbot, assistant or agent answers correctly, doesn't make things up, doesn't leak data and can't be manipulated. This requires different methods from classic tests, because model output is not fully deterministic.
We'll cover the first area first, then the second.
AI-generated unit tests
Unit tests are the area where AI delivers results fastest. An agent such as Claude Code can read a function or class, understand how it works, generate a set of tests, run them and fix the ones that fail for technical reasons. This is especially useful in legacy code that has no tests but needs to be changed.
What to watch out for
- Tests that lock in bugs. AI writes tests based on how the code works, not how it should work. If a function calculates a discount incorrectly, the test will often "confirm" the wrong result. That is why it is worth giving the agent requirements or examples of expected results, not just the code.
- Tautological tests. A test that checks whether a mocked function returns what the mock returns increases code coverage but not safety.
- Too many mocks. Replacing every dependency with a stub means the test no longer checks real behavior.
- Coverage as a goal. A high coverage percentage generated by AI is easy to achieve and easy to overrate. A better measure is whether the tests detect deliberately introduced bugs – a technique known as mutation testing.
Good practice: first a human describes the cases to check (including edge cases and invalid data), then AI writes the tests, and finally a developer reviews them as carefully as production code.
E2E tests in Playwright with the help of AI
Playwright is an open-source framework for automating end-to-end tests of web applications, developed by Microsoft. It lets you drive browsers based on the Chromium, Firefox and WebKit engines, run tests in parallel and in CI, and, when something fails, analyze the test run in the Trace Viewer (screenshots, network requests, individual steps). It has built-in auto-waiting for elements, which reduces the problem of flaky tests, and a test generator that records user actions.
How AI helps with Playwright
- Scenarios from a description. You describe a user journey ("register an account, confirm the email, add a product to the cart, pay with a test card") and the agent writes a Playwright test, runs it and fixes it.
- An agent in a real browser. With the Playwright MCP server, an AI agent can open the application itself, click, fill in forms and check the result – and then save the run as a permanent test. We write about the protocol in our article What is MCP (Model Context Protocol).
- Failure analysis. The agent reads the report and the test trace, determines whether the test or the application failed, and proposes a fix.
- Test maintenance. After an interface change, AI updates selectors and steps across many tests at once.
- Better locators. The agent can rewrite brittle CSS selectors into locators based on roles and labels (e.g. a button with a given name), which are more resilient to visual changes and also support accessibility.
Pitfalls of AI-written E2E tests
An agent tasked with "making the test pass" may weaken an assertion, add a hard-coded wait or skip a failing step. So every change to tests should go through review with the question: does the test still check what it was supposed to check? E2E tests are also more expensive to maintain than unit tests – it is worth limiting them to key business paths and covering the rest with lower-level tests.
AI code review – automated code review
AI code review tools analyze the changes in a pull request and leave comments: potential bugs, missing exception handling, inconsistencies, readability issues and sometimes security gaps. This feature is offered by GitHub Copilot, among others, and agents such as Claude Code or Codex can be run in the CI pipeline to review changes according to agreed rules.
Where AI code review works well:
- a quick first round of comments before the change reaches a human,
- catching typical mistakes and oversights (null, missing validation, typos in names),
- enforcing team conventions described in the instructions,
- summarizing a large pull request for the reviewer.
Where it falls short:
- assessing whether a change meets the business requirements,
- architectural decisions and their long-term consequences,
- bugs that stem from context outside the changed files,
- too many comments – a flood of minor remarks dulls reviewers' attention.
AI code review is an addition, not a replacement. At 4B Systems, every change – including one prepared by an AI agent – is reviewed by a developer.
Test automation with AI – where else it helps
| Area | What AI does well | What to watch out for |
|---|---|---|
| Unit tests | Fast generation, tests for legacy code | Locking in bugs, tautological tests |
| Integration tests | API scenarios, error cases | Tests that depend on order and database state |
| E2E (Playwright) | Scenarios from a description, selector updates, trace analysis | Weakened assertions, flakiness, maintenance cost |
| Test data | Realistic, varied synthetic data | Don't use real personal data |
| Failure analysis | Reading logs and stack traces, reproducing the issue | Confident but wrong diagnoses |
| Code review | First round of comments, conventions, summaries | No business context, too many comments |
| Test documentation | Scenario descriptions, test plans from requirements | Completeness verified by a human |
Testing applications with AI features
When an application itself uses a language model – as a chatbot, assistant, knowledge search engine or agent – classic tests are not enough. The same prompt can produce different answers, and the "correct" answer is often not one specific text. Three additional types of tests are needed.
Evaluations (evals)
An evaluation is a set of test cases – questions, documents, tasks – together with criteria for assessing the answers. The criteria can be simple (does the answer contain the right amount, does it have the required JSON format) or descriptive (is the answer consistent with the returns policy, is the tone appropriate). Some assessments are done by code, some by another model grading according to agreed rules (so-called LLM-as-a-judge), and some – on a sample – by a human. Evaluations are run with every change of prompt, model, data or RAG configuration to catch quality regressions. It is worth continually adding cases where the application failed for real users.
Hallucinations
A hallucination is an answer that sounds plausible but is untrue or not supported by the sources. In RAG applications, tests check, among other things:
- whether the answer is based solely on the documents provided,
- whether the cited sources actually contain the stated information,
- whether the model admits it doesn't know when the documents contain no answer,
- how it behaves with trick questions and out-of-scope questions.
Prompt injection and security
Prompt injection is an attempt to force the model to ignore its instructions – directly ("forget your previous instructions and…") or indirectly, through content the model reads: a document, an email, a web page with hidden instructions. It is one of the main threats listed in the OWASP Top 10 for LLM Applications. Tests should check, among other things, whether the application:
- does not reveal system instructions or other users' data,
- respects the user's permissions even if the model is manipulated,
- does not take actions (e.g. sending an email, changing data) without confirmation when the process requires it,
- handles model-generated content safely, e.g. does not insert it into a page without escaping.
The most important rule: safeguards cannot rely solely on instructions for the model. Permissions, validation and confirmations must work in the application code. We write more about designing such applications in our article How to build an AI app.
What not to hand over to AI in testing
AI can do a large part of the work in test automation, but a few elements should stay with people:
- Deciding what to test. Choosing critical paths, business risks and priorities requires knowledge of the product and the customers.
- Acceptance criteria. A human defines how the system should behave. AI can help write them down, but it shouldn't invent them.
- Test review. Tests written by AI should be read as carefully as production code – especially changes that "fix" a failing test.
- Security testing. AI helps find vulnerabilities, but the final assessment and penetration testing require specialists.
- Exploratory and UX testing. Sensing that something is confusing or annoying for users is still a human domain.
- Production data. The agent should not have access to production or real personal data – tests run on test environments and synthetic data.
- The release decision. A green pipeline is a necessary condition, but the decision to deploy is made by the team.
How we test applications at 4B Systems
In web and mobile app projects, we combine classic practices with AI tools: agents help write unit tests and E2E scenarios in Playwright, analyze failures and prepare the first round of comments in code review, while developers decide on the scope of testing and review every change. In applications with AI features, we build evaluation sets and prompt injection resistance tests before deployment. Read more about our approach on our page about AI apps and AI-assisted development.
Summary
AI genuinely speeds up software testing: it generates unit tests, writes and maintains E2E scenarios in Playwright, analyzes failures and supports code review. It does, however, require oversight – AI-written tests can lock in bugs or check nothing at all. Applications with AI features additionally need to be tested with evaluations, for hallucinations and for prompt injection. Decisions about what to test, acceptance criteria, security and releases remain with people.
Need an application that is not only built quickly but also properly tested? 4B Systems builds applications with the help of AI, with automated tests in Playwright, evaluations of AI features and a developer review of every change. See our AI-assisted development and automated testing services and describe your project – we will prepare a quote within 48 hours.
Frequently asked questions
Can AI write automated tests instead of a developer?
AI can write a large share of tests, especially unit tests and E2E scenarios, and run and fix them. A developer should still define what to test and review the generated tests, because AI can lock in incorrect code behavior or write tests that check nothing meaningful.
What is Playwright?
Playwright is an open-source end-to-end testing framework for web applications, developed by Microsoft. It drives Chromium-, Firefox- and WebKit-based browsers, automatically waits for page elements and lets you analyze test runs in the Trace Viewer.
Does AI code review replace human code review?
No. AI works well as a first round of comments and for catching typical mistakes, but it doesn't assess compliance with business requirements or architectural decisions. A change should be approved by a developer.
How do you test a chatbot or an AI assistant?
Besides classic tests, you need evaluations – a set of questions with criteria for assessing the answers, run with every change of prompt, model or data. On top of that come tests for hallucinations, behavior on out-of-scope questions and resistance to prompt injection.
What is prompt injection?
It is an attempt to force the model to ignore its instructions, either directly in a user message or indirectly through the content of a document, email or web page the model reads. Protection requires safeguards in the application code – permissions, validation and confirmations – not just instructions for the model.
Can production data be used when testing with AI?
It shouldn't be. Tests should run on test environments and synthetic data, and AI agents should not have access to production or real personal data, also for GDPR reasons.