Towards better test runners/frameworks
Test runners/frameworks caused me a lot of grief over the years. It's part of the reason why really appreciate how standard #Rust test runner works. As usual with Rust, core developers, and the community at very least avoided the most common misfeatures and made the right thing to be the easiest and most natural thing to do.
I've taken some time and collected a list of feature request, misfeature rants, that I just want to share.
Quoting my reddit comment:
In almost every codebase I had to work with professionally, someone always had introduced another lame testing framework with its own DSL for something as basic as an assertion. Each language, each test framework, I now have to guess/look-up the crappy DSL of the day.
My take on DSLs: you have to be introducing some very substantial benefits to outweigh the mental burden of introducing yet another DSL.
And all this assertion DSLs are usually really clunky, trying to mimic the natural language. There's a reason why we generally don't make function calls look like
url.download.all.resources(). So why do people keep reintroducing this in testing/assertion frameworks?
Just use the most standard and natural syntax/library for the given language. In the case of Rust
assert_eq! macro etc.
Test tagging as opposed to naming
Naming tests is a difficult and not particularly useful task. A much more versatile approach would be tagging your tests. It also should be possible to run a subset of all tests using filtering:
--filter "user AND (account OR notification)"
I actually implemented this in a small test runner that I use for some things internally, and it works beautifully so far. It's probably the only feature that I'd like added (maybe with a library) to Rust's standard test runner.
Randomized execution order
Too many times I've seen (sometimes accidental) unit tests that are interdependent. Test runners like Mocha with sequential and deterministic order execution make it much too easy to do the wrong thing and not notice or get away with it.
By randomizing the execution order, we just make it very hard or almost impossible.
No setup functions/hooks
beforeEach hooks or JUnit-like
setUp methods are a bad idea.
Unit tests should be as self-contained, self-describing as possible. If you have some common initialization logic, just put it in
init_test() function, and call explicitly at the beginning of tests that need it. And frankly, if your setup requires so much work to use utility functions, maybe you should reconsider your APIs, what and how you're testing etc.
setUp-like functionality tend to accumulate cruft with time, leading to over-initialization and unintentional coupling of tests. Relying on developers not to use/abuse them is wishful thinking. IMO: tools drive users behavior. If your tool (testing framework) makes abuse easy, developers will just abuse it.