Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.
Big fan of nextest and this is my first time seeing this site. I'll be real I feel a bit ridiculous commenting this but you might want to consider rephrasing this:
>Treat tests as cattle, not pets. Detect and terminate slow tests.
Not sure saying, "hey, treat your tests as an animal you can kill at will" paints the right image.
This is from the Kubernetes saying of "treat servers like cattle, not pets". Of course, some people like me keep cattle as pets, but then again I also name my servers, even the virtual or containerised ones.
I mean, I like animals too, but in context it does make sense. The context was to treat them as "obtainable yet ultimately killable entities you keep as a group, not individuals", which cattle pretty much is. Unless you consider keeping cattle as draft animals, but I think that stopped being the main purpose a long time ago.
It got the point across, at a time where most people basically acquired servers, kept them until they died, and he was trying to push a development workflow where you constantly close("kill")/bring up new servers.
IMO, "make your servers fungible" is a better way to express the intent: slightly shorter, no metaphors, although "fungible" is a less common word. Maybe that's just me. (Edited to add: "make your tests fungible" has I think the wrong connotation; I think the original wording on the blog is about test executions -- and "make your test executions fungible" does seem like a good goal, similar to ACID guarantees for database transactions.)
Before that, I used to call them "ephemeral", and of course half the people asked what "ephemeral" means, probably "fungible" would be met with similar question, unless the crowd is cryptocurrency-adjacent, that term seems understood there.
Fungible/non fungible is a good alternative, and maybe the technically correct word. But I think in that case it doesn't apply and the change the author did is better.
Thank you very much for developing nextest. It is what allows our projects like rama [1] to run thousands and thousands of tests in a blink of an eye! Keep it up!
Can you run tests serially in the (horrible) case when tests need to build on one another?
How are you querying for the tests? Is that just built into rust's test stuff?
Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.
> How are you querying for the tests? Is that just built into rust's test stuff?
Just running --list against test binaries.
> Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.
This is possible in principle, but nextest doesn't really inject itself into tests like that (injection can cause reliability issues in practice, and a big focus of nextest is reliability). Forking is also not possible in multithreaded programs.
Hey, I love nextest. But, perhaps because of the one-process per test approach, endpoint security solutions like CrowdStrike Falcon or Palo Alto Cortex tend to make computers hang whenever tests kick-in. I would love if you were able to introduce a workaround, because none of those companies will fix their stuff. I am guessing a possible mitigation would be to have stagger the first invocation of any large test binary, but I haven't had a chance to dig deep into this issue.
Nextest does run tests in lexicographic (binary, test name) order by default, so first executions tend to be staggered. Unfortunately this is a known problem that is much larger than nextest, and it is why Windows introduced Dev Drive with a mode to disable AV/EDR on those drives entirely.
I wish I had a better answer here — it is frustrating how poorly behaved the EDR stuff tends to be. Maybe I should try and network a bit to get in touch with the people working on this stuff. (Planning to be at DEF CON this year, so come find me if you work on this stuff and will be there!)
Super happy user here! It's an excellent piece of engineering.
We're running a fork that supports a "sidecar" server for running multiple integration tests against. So if any tests that need the server are included, it spawns the server, runs the integration tests, and then shuts it down. By re-using the same server we speed up our runs tremendously.
Thanks! BTW you might enjoy setting CARGO_TERM_COLOR=always in your environment :) dtolnay/rust-toolchain does this automatically but it looks like you aren't using that action.
I think in practice you're right that GitHub Actions' caching slowness can end up dominating performance. I don't have any great tips other than to measure.
Yes we use it for rama [1]. You can check its justfile and CI workflow file how we use it. Those run thousands and thousands of tests thx to nextest and what feels like instantly (once compiled).
Large projects build with rama use it as well. But those are proprietary from partners so sadly cannot share those.
the execution model of cargo-nextest makes my test suite extremely slow to run - Most of my tests rely on postgres. Since cargo-nextest uses a separate process per test, there is no efficient way to share the connection pool.
seems like `cargo nextest run` just runs `--lib` tests by default? however, `cargo test` is not so slow if you do `cargo test --lib`. how do I get nextest to execute the doc tests, too?
The benchmarks are against non-doctest cargo test runs. (But you should measure against your own project, of course!) Unfortunately Cargo doesn't provide enough information for nextest to run doctests reliably with.
Thanks for posting about this! I'm the main author of nextest, and it represents my best foot forward for how Rust testing should be done. Happy to answer questions though I might be a bit intermittent.
Big fan of nextest and this is my first time seeing this site. I'll be real I feel a bit ridiculous commenting this but you might want to consider rephrasing this:
>Treat tests as cattle, not pets. Detect and terminate slow tests.
Not sure saying, "hey, treat your tests as an animal you can kill at will" paints the right image.
This is from the Kubernetes saying of "treat servers like cattle, not pets". Of course, some people like me keep cattle as pets, but then again I also name my servers, even the virtual or containerised ones.
Yeah that was indeed the inspiration (though I'm pretty sure it predates Kubernetes!) but the juxtaposition with "terminate" is unfortunate.
I liked the way it was phrased. You can't make everybody happy. :)
It’s a horrible saying in that context also.
I mean, I like animals too, but in context it does make sense. The context was to treat them as "obtainable yet ultimately killable entities you keep as a group, not individuals", which cattle pretty much is. Unless you consider keeping cattle as draft animals, but I think that stopped being the main purpose a long time ago.
It got the point across, at a time where most people basically acquired servers, kept them until they died, and he was trying to push a development workflow where you constantly close("kill")/bring up new servers.
IMO, "make your servers fungible" is a better way to express the intent: slightly shorter, no metaphors, although "fungible" is a less common word. Maybe that's just me. (Edited to add: "make your tests fungible" has I think the wrong connotation; I think the original wording on the blog is about test executions -- and "make your test executions fungible" does seem like a good goal, similar to ACID guarantees for database transactions.)
Before that, I used to call them "ephemeral", and of course half the people asked what "ephemeral" means, probably "fungible" would be met with similar question, unless the crowd is cryptocurrency-adjacent, that term seems understood there.
Who here is old enough to know about PetSets, since we're talking this topic and k8s...
That (unpleasant) saying predates Kubernetes by at least half a decade.
That's fair! I'll find a way to rephrase it.
edit: Updated to "Detect and handle slow tests". Thanks again!
It resonated with me!
Fungible/non fungible is a good alternative, and maybe the technically correct word. But I think in that case it doesn't apply and the change the author did is better.
Jesus Christ do we seriously need to limit our speech and get rid of perfectly good analogies for every special little snowflake?
Like all things in life, there is a balance :) I didn't remove the cattle vs pets analogy but I did replace "terminate" with "handle".
"Treat tests as annuals not perennials"
"Treat tests as workers not management"
Thank you very much for developing nextest. It is what allows our projects like rama [1] to run thousands and thousands of tests in a blink of an eye! Keep it up!
[1] https://ramaproxy.org
First, I like how all this works, it's great.
Can you run tests serially in the (horrible) case when tests need to build on one another?
How are you querying for the tests? Is that just built into rust's test stuff?
Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.
> Can you run tests serially in the (horrible) case when tests need to build on one another?
Yes: https://nexte.st/docs/configuration/test-groups/ (edit: though tests that build on each other is a bit harder — test groups are meant for when tests need access to a shared resource)
> How are you querying for the tests? Is that just built into rust's test stuff?
Just running --list against test binaries.
> Would it be possible to fork the test process? It'd be pretty interesting if you could spawn a test process, and then fork it for each test to save both on memory and any static state stored within the test.
This is possible in principle, but nextest doesn't really inject itself into tests like that (injection can cause reliability issues in practice, and a big focus of nextest is reliability). Forking is also not possible in multithreaded programs.
Hey, I love nextest. But, perhaps because of the one-process per test approach, endpoint security solutions like CrowdStrike Falcon or Palo Alto Cortex tend to make computers hang whenever tests kick-in. I would love if you were able to introduce a workaround, because none of those companies will fix their stuff. I am guessing a possible mitigation would be to have stagger the first invocation of any large test binary, but I haven't had a chance to dig deep into this issue.
Nextest does run tests in lexicographic (binary, test name) order by default, so first executions tend to be staggered. Unfortunately this is a known problem that is much larger than nextest, and it is why Windows introduced Dev Drive with a mode to disable AV/EDR on those drives entirely.
I wish I had a better answer here — it is frustrating how poorly behaved the EDR stuff tends to be. Maybe I should try and network a bit to get in touch with the people working on this stuff. (Planning to be at DEF CON this year, so come find me if you work on this stuff and will be there!)
Wouldn't running the tests in a container solve that issue? Or is that another thing that gets flagged?
Have there been any discussions about upstreaming this into cargo proper? Are there any significant downsides to nextest compared to its predecessor?
The How it works [1] and Why process-per-test? [2] pages should answer your questions.
[1] https://nexte.st/docs/design/how-it-works/
[2] https://nexte.st/docs/design/why-process-per-test/
Ah, I see. You're aiming to become the hashbrown of testing.
Oh gosh, were we to be so lucky :) just aiming to solve problems my coworkers and users see, and doing it with care, is all.
Thank you! It's great.
The "execution model" page[1] is documentation at its best!
It answered 90% of the questions I had at the monent. Thank you!
[1] https://nexte.st/docs/design/how-it-works/
Super happy user here! It's an excellent piece of engineering.
We're running a fork that supports a "sidecar" server for running multiple integration tests against. So if any tests that need the server are included, it spawns the server, runs the integration tests, and then shuts it down. By re-using the same server we speed up our runs tremendously.
Discussion thread on gh: https://github.com/nextest-rs/nextest/discussions/3330
I love nextest, it's been great. This along with bacon catches a lot of issues.
I love nextest. without it my CI could take hours
https://github.com/tsz-org/tsz/actions/runs/29002057457/job/...
watch it running 32.5k unit tests without breaking a sweat!
Thanks! BTW you might enjoy setting CARGO_TERM_COLOR=always in your environment :) dtolnay/rust-toolchain does this automatically but it looks like you aren't using that action.
Thanks! Any pro tips for sharding? I landed on single job because couldn't get cache to work properly for shards to be fast enough to worth it
I think in practice you're right that GitHub Actions' caching slowness can end up dominating performance. I don't have any great tips other than to measure.
I've basically abandoned GA's caching for most things because measurements ended up showing things were slower with caching than without :woozy:
Anybody using this in production?
edit: Thanks, will try!
Yes, for a long while – I believe it's fairly widely used (and it's absolutely excellent!)
Yes we use it for rama [1]. You can check its justfile and CI workflow file how we use it. Those run thousands and thousands of tests thx to nextest and what feels like instantly (once compiled).
Large projects build with rama use it as well. But those are proprietary from partners so sadly cannot share those.
[1]: https://github.com/plabayo/rama
Yep. Tokio uses it for their tests in CI as well last I checked.
Been using it for years, for opensource stuff and at work. No issues whatsoever.
Yes, been using it for 18’ish months. Works great.
Happy user here !
I somehow tried to make sense of the name as a superlative form of "next". Perhaps next-test would have been fine?
That is indeed the pun =)
Oh, sorry then, thanks for the clarification :)
Coming from the DC area, this comment reminds me of how the metro payment system "Smartrip" was recently renamed to "Smart Trip".
the execution model of cargo-nextest makes my test suite extremely slow to run - Most of my tests rely on postgres. Since cargo-nextest uses a separate process per test, there is no efficient way to share the connection pool.
seems like `cargo nextest run` just runs `--lib` tests by default? however, `cargo test` is not so slow if you do `cargo test --lib`. how do I get nextest to execute the doc tests, too?
The benchmarks are against non-doctest cargo test runs. (But you should measure against your own project, of course!) Unfortunately Cargo doesn't provide enough information for nextest to run doctests reliably with.
It looks like doctests aren't supported: https://github.com/nextest-rs/nextest/issues/16