16f6c0aecf
Inline adoc files when they are included exactly once. Also fix language tags because this inlining gives us better information on what language the code is written in.
56 lines
1.9 KiB
Plaintext
56 lines
1.9 KiB
Plaintext
== Why is this an issue?
|
|
|
|
Unstable / flaky tests are tests which sometimes pass and sometimes fail, without any code change. Obviously, they slow down developments when engineers have to rerun failed tests. However the real problem is that you can't completely trust these tests, they might fail for many different reasons and you don't know if any of them will happen in production.
|
|
|
|
|
|
Some tools, such as TestNG, enable developers to automatically retry flaky tests. This might be ok as a temporary solution, but it should definitely be fixed. The more flaky tests you add, the more chances there are for a bug to arrive in production.
|
|
|
|
|
|
This rule raises an issue when the annotation ``++org.testng.annotations.Test++`` is given a ``++successPercentage++`` argument with a value lower than ``++100++``.
|
|
|
|
|
|
=== Noncompliant code example
|
|
|
|
[source,java]
|
|
----
|
|
import org.testng.annotations.Test;
|
|
|
|
public class PercentageTest {
|
|
@Test(successPercentage = 80, invocationCount = 10) // Noncompliant. The test is allowed to fail 2 times.
|
|
public void flakyTest() {
|
|
}
|
|
}
|
|
----
|
|
|
|
|
|
== Resources
|
|
|
|
* https://testng.org/doc/documentation-main.html#annotations[TestNG documentation - Annotations]
|
|
* https://engineering.atspotify.com/2019/11/18/test-flakiness-methods-for-identifying-and-dealing-with-flaky-tests/[Test Flakiness - Methods for identifying and dealing with flaky tests]
|
|
|
|
|
|
ifdef::env-github,rspecator-view[]
|
|
|
|
'''
|
|
== Implementation Specification
|
|
(visible only on this page)
|
|
|
|
=== Message
|
|
|
|
* Make this test stable and remove this "successPercentage" argument.
|
|
|
|
|
|
=== Highlighting
|
|
|
|
The "successPercentage" argument
|
|
|
|
|
|
'''
|
|
== Comments And Links
|
|
(visible only on this page)
|
|
|
|
=== on 3 Sep 2020, 11:57:10 Nicolas Harraudeau wrote:
|
|
Multiple solutions exist for JUnit but none of them seems to be used that much. https://github.com/junit-team/junit5/issues/1558[A ticket] asks for an equivalent to "successPercentage "since 2018.
|
|
|
|
endif::env-github,rspecator-view[]
|