55 lines
1.6 KiB
Plaintext
55 lines
1.6 KiB
Plaintext
== Why is this an issue?
|
|
|
|
include::../common/why.adoc[]
|
|
|
|
Some tools, such as TestNG, enable developers to automatically retry flaky tests. This might be acceptable as a temporary solution, but it should eventually be fixed. The more flaky tests you add, the more chances there are for a bug to arrive in production.
|
|
|
|
|
|
This rule raises an issue when the annotation ``++org.testng.annotations.Test++`` is given a ``++successPercentage++`` argument with a value lower than ``++100++``.
|
|
|
|
|
|
=== Noncompliant code example
|
|
|
|
[source,java]
|
|
----
|
|
import org.testng.annotations.Test;
|
|
|
|
public class PercentageTest {
|
|
@Test(successPercentage = 80, invocationCount = 10) // Noncompliant. The test is allowed to fail 2 times.
|
|
public void flakyTest() {
|
|
}
|
|
}
|
|
----
|
|
|
|
|
|
== Resources
|
|
|
|
* TestNG documentation - https://testng.org/#_annotations[Annotations]
|
|
* Spotify Engineering - https://engineering.atspotify.com/2019/11/18/test-flakiness-methods-for-identifying-and-dealing-with-flaky-tests/[Test Flakiness - Methods for identifying and dealing with flaky tests]
|
|
|
|
|
|
ifdef::env-github,rspecator-view[]
|
|
|
|
'''
|
|
== Implementation Specification
|
|
(visible only on this page)
|
|
|
|
=== Message
|
|
|
|
* Make this test stable and remove this "successPercentage" argument.
|
|
|
|
|
|
=== Highlighting
|
|
|
|
The "successPercentage" argument
|
|
|
|
|
|
'''
|
|
== Comments And Links
|
|
(visible only on this page)
|
|
|
|
=== on 3 Sep 2020, 11:57:10 Nicolas Harraudeau wrote:
|
|
Multiple solutions exist for JUnit but none of them seems to be used that much. https://github.com/junit-team/junit5/issues/1558[A ticket] asks for an equivalent to "successPercentage "since 2018.
|
|
|
|
endif::env-github,rspecator-view[]
|