Had a thought the other day and wanted to run it past you.
Tests increase in complexity starting from unit tests all the way up to system tests.
As the tests get more complex, they can become slower (because one test is doing a bunch of stuff per test case), or harder to maintain.
Unit tests have a purpose, and they’re very quick. We can also mock inputs to simulate real-world data, to see what a function would do with it. However, they are only as smart as the tests and the data–they won’t “accidentally” find any problems that aren’t related to the unit or the data being used.
Integration tests also have a purpose, where you see how units interact with each other. They’re a little more complex but still usually small enough to be manageable.
However, as you move to integration tests all the way up to full system tests, there is a great chance to find bugs that couldn’t be found at the unit level.
I’m wondering: if there can be a test created at the unit level, that would test for the bug caught at the integration or the system level.
So say you have a system of small programs, A B C, linked in a chain like A -> B -> C. Let’s also say you have the following tests:
- Unit tests for A, B and C
- Integration tests for A -> B and B -> C
- System level tests for A -> … -> C
Then if your unit tests all pass, but the integration test failed, that means (at least in my experience) there’s a particular bug in some part of the code, at the unit level. Same for a failure at the system level.
There may then be some arrangement of data that can be fed to that unit, to detect that same bug later. And since unit tests run much faster, it would make for a very good regression test that can be run as soon as some code is checked in.
The whole reason to run something other than a unit test is to expose some kind of bug at that level that we wouldn’t likely find at the unit level. At the unit level, we “don’t know what we don’t know”, so we may not be aware of what data combinations would be reasonable at the unit level, unless we run it at a higher complexity. But once a bug is found, turn that specific bug fix into a unit test.
What do you think, am I off base on this?