Test oracle
A test oracle is the source used to determine the expected result of a test, and so to decide whether an observed result is right: a specification, a formula, a previous version, a reference implementation, or a human judgement. Every test has one, stated or not, because a result cannot be marked passed or failed without something to compare it against, and the oracle is that something.
The longer version
Running the software produces an output; deciding whether the output is correct requires a second source of truth, and that source is the oracle. For a sum on an invoice the oracle is arithmetic. For a rendered page it is a design file or the previous release's screenshot. For a search ranking it is often a person saying whether the results look reasonable, because nobody can write the correct order down in advance. The oracle determines what the expected result in a test case says, and a test is only as trustworthy as the oracle its expected result came from.
The oracle and the expected result are related and not the same thing. The expected result is one value, written into one test case, for one input. The oracle is the rule or reference that value was derived from, and it answers for every input. Where the expected result was copied from the oracle correctly, the difference does not matter. Where the oracle is itself wrong, a stale specification or an earlier build that had the same bug, every expected result derived from it is wrong together, and the tests pass while the software is broken.
The oracle problem is the name for the cases where no cheap oracle exists: machine learning outputs, simulations, anything whose correct answer would cost as much to compute as the software costs to run. Teams answer it with partial oracles: properties that must hold whatever the exact value (the total never goes negative, sorting twice changes nothing), comparison against a trusted previous version, or a person reviewing a sample. Naming the oracle in the test plan is what stops 'the tester thought it looked fine' from being mistaken for a specification.
Common questions
What is the difference between a test oracle and an expected result?
The expected result is a single value written into a single test case for a single input. The test oracle is the source that value came from, a specification, a formula, a reference build or a person, and it can answer for any input. A wrong oracle makes every expected result derived from it wrong at once.
What is the oracle problem in software testing?
The oracle problem is the situation where no practical source of correct answers exists, so a tester can run the software and still not know whether the output is right. Ranking, recommendation, simulation and machine learning outputs all have it. Teams work around it with properties that must hold, comparison against a previous version, or human review of a sample.
Can a person be a test oracle?
Yes, and for some outputs a person is the only oracle available: whether a page looks right, whether search results are reasonable, whether a generated text reads well. The cost is consistency, since two reviewers disagree and one reviewer drifts, so human oracles are used on samples and written down as judgements rather than treated as a specification.
What are the types of test oracle?
A specified oracle, where a requirement or a formula gives the answer; a derived oracle, where a previous version, a reference implementation or a second calculation gives it; an implicit oracle, where a crash, a timeout or a negative total is wrong whatever the specification says; and a human oracle, where a person judges. Most test suites use several of them without naming any.
What happens when the test oracle is wrong?
Every expected result derived from the oracle is wrong together, and the tests pass while the software is broken. A stale specification or an earlier build carrying the same defect are the usual cases: the software reproduces the oracle's mistake, the comparison matches, and nobody is alerted. The defence is to name the oracle in the test plan so a suspicious pass can be traced to its source.
Bug reports that arrive ready to triage.
Collect replay, console, network and browser context with the Free plan.