This same statement has been said countless times, replacing “AI” with whatever trend is big at the time. It has also been wrong in every case where that thing is said to replace QA.
just from first principles its really not. given its propensity to just make shit up and cheat, ai is really useful when there is some kind of formal or exhaustive checking as a wall for it to throw crap at. so if you rigorously define success and spend a bunch of tokens, you could easily save time and money. but if you ask the ai 'is this correct' and it says 'yes!', or even 'no!', then you've really learned nothing. this isn't a can you can kick arbitrarily far down the road.
I think writing tests is a great use of ai, but only if the tests themselves are throughly reviewed or are themselves validated by statements in a formal system.
I think you could use AI for this if you do it properly as a sort of adversarial coding project. Have it build tests to break the code. Don't give it the job of making a test suite that passes.
I think people higher up are warning against a naive mistake, which is asking one agent to write both the product and the test suite. Here, making things up and cheating becomes a problem of quality theater...
Totally agree. My point is simply that this aligns with existing roles.
The QA role (regardless of whether it's performed by an AI or a human) typically involves breaking tests, not writing them. (This applies more to unit tests, integration tests blur this line.)
For me QA is tasked with making sure that the product functions as intended. that means coming up with clever test suites to explore the spaces that the developers didn't think to, making sure the coverage for operational problems is adequate, and also long term tracking of performance and memory utilization.
this doesn't sound like your model, and I'm unclear what it means to break a test. maybe test automation, in which case, sure that seems fair game for AI, but that not where the real meat is.
> For me QA is tasked with making sure that the product functions as intended. that means coming up with clever test suites to explore the spaces that the developers didn't think to, making sure the coverage for operational problems is adequate, and also long term tracking of performance and memory utilization.
This pretty much. My last job didn't really have QA so I did the next best thing which is writing a bunch of integration tests for the use cases that matters for the product. They were not an indication for correctness, but more like a canary to warn me if I break something while developing. Bugs reported by consumers usually warn me of area not well covered.
QA would play the same whole. They shouldn't need to check for code correctness, their most useful task is to surface bugs that breaks the product requirements (performance, security, business logic,...). And for that, having knowledge of the implementation is unnecessary. In the above example of writing integration tests, I took care of only using the public interface of the modules.
That largely aligns with my view. I think in a large enough codebase (roughly the size QA becomes necessary/relevant) it becomes impossible/infeasible to guarantee correctness. I think the line "clever test suites" highlights the distinction.
In my mind, tests are everyone's responsibility, but the goal differs. A regular developer adds features and should be writing tests to prove their code functions as intended. QA does not add features, so their goal is finding holes in code added by others.
It's the adversarial relationship mentioned by a parent:
> Have [QA] build tests to break the code. Don't give [QA] the job of making a test suite that passes.
The article makes a strong point that the slice that needs humans is dwindling quickly. The article seems to suggest our ability to write prose for other humans is our main differentiator.
Even that is not a differentiator. LLMs are perfectly capable of following writing style guides, it's just that their default writing style has tells and clichés, and most people don't care enough to instruct it to deviate from that style with any specificity.
MITM service, Internet gatekeeper and robber baron.