diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 455e197dc9b..2e062b6401d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,6 +31,11 @@ Alternatively, you can create a Launcher: Go to Run - Debug Configurations... Se Until you are familiar with the CDT code base, using a Debug Launcher and experimenting with breakpoints is probably the best way to get a warm fuzzy feeling if you have doubts about modifications you are making. +### Testing + +The Eclipse CDT project has extensive JUnit Plug-in tests covering many parts of the CDT code base. +These tests are run automatically by the CI system (currently Jenkins and GitHub actions) using Maven + Tycho Surefire. +See [CDT Testing notes](TESTING.md) for more information. ### Creating Pull Requests diff --git a/TESTING.md b/TESTING.md index d0559773598..e6beeaf4b64 100644 --- a/TESTING.md +++ b/TESTING.md @@ -6,37 +6,83 @@ This document is a collection of various notes on testing and writing JUnit test Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for contributing information, including setting up a development environment. -### How do I run CDT JUnit test suite? +## How do I run CDT JUnit test suite? There are special Java packages to test CDT itself. -Their names generally end with suffix ".tests". +Their names generally end with suffix `.tests`. They are normally run during the build of CDT for example you can see results for Jenkins builds [here](https://ci.eclipse.org/cdt/job/cdt/job/main/lastCompletedBuild/testReport). -You can run JUnit tests in UI this way: +You can run JUnit Plug-in tests in UI this way: -1. Select test CDT package (ends with ".tests", for example org.eclipse.cdt.core.tests). -2. Right-click on the class, suite, package, source folder or project you want to run in Package Explorer and select Run As-\>JUnit Plugin Test. - -The build machine generally use the standard [Tycho Surefire](https://www.eclipse.org/tycho/sitedocs/tycho-surefire-plugin/plugin-info.html) class name patterns for [includes](https://www.eclipse.org/tycho/sitedocs/tycho-surefire-plugin/integration-test-mojo.html#includes) and [excludes](https://www.eclipse.org/tycho/sitedocs/tycho-surefire-plugin/integration-test-mojo.html#excludes) to identify tests during automated builds. +1. Select test CDT package (ends with `.tests`, for example org.eclipse.cdt.core.tests). +2. Right-click on the class, suite, package, source folder or project you want to run in Package Explorer and select _Run As_ -\> _JUnit Plugin Test_. ![run_junit_tests.png](images/run_junit_tests.png "screenshot of how to run junit plug-in tests") +### Using limited sets of plug-ins in tests -Some sets of tests are grouped in "suites", these can be used to run subsets of tests that may exist across numerous packages. -An example is the CDT core's [AutomatedIntegrationSuite](https://github.com/eclipse-cdt/cdt/blob/main/core/org.eclipse.cdt.core.tests/suite/org/eclipse/cdt/core/suite/AutomatedIntegrationSuite.java). +You may want to restrict the plug-ins included in the launch configuration. +This has benefits: + +1. Start-up time of the tests can be much faster, leading to faster development cycles. +2. Other plug-ins can cause unrelated errors in the error log. +3. More closely matches what happens at the command line with Maven + Tycho Surefire + +For most of the test plug-ins including only the `org.eclipse.cdt` and `org.eclipse.platform` feature + the test bundle is the recommended flow. +For example to run tests in the `org.eclipse.cdt.core.tests` project, configure the _Plug-ins_ tab like this: + +![junit_use_features.png](images/junit_use_features.png "screenshot of how to configure to use features") + +Refer to the `tycho-surefire-plugin` section in the `pom.xml` to see what features are included when testing with Maven + Tycho Surefire. + +### Controlling which test runner to use + +When running JUnit plug-in tests there are two critical settings to control how the tests run: + +- Which thread the JUnit test code runs in. +Some tests, e.g. SWT Bot tests, expect the JUnit code to be run outside the `main` thread. +Other tests rely running the JUnit code on the `main` thread. + - In the Eclipse IDE this is controlled with the _Run in UI thread_ setting in the _Test_ tab of the launch configuration. + - With Maven + Tycho Surefire this corresponds to the [`useUIThread`](https://tycho.eclipseprojects.io/doc/latest/tycho-surefire-plugin/test-mojo.html#useUIThread) setting. + - Refer to the `pom.xml` to see what it the correct setting for any individual project. +- Whether or not to start a UI. +Headless tests, those that don't test the UI like `org.eclipse.cdt.core.tests`, don't require a UI to be running. + - For these tests launching with the `[No Application] - Headless Mode` application is recommended in the _Main_ tab of the launch configuration. + - For tests that need the UI the `org.eclipse.ui.ide.workbench` application is recommended in the _Main_ tab of the launch configuration. + - With Maven + Tycho Surefire this corresponds to the [`useUIHarness`](https://tycho.eclipseprojects.io/doc/latest/tycho-surefire-plugin/test-mojo.html#useUIHarness) setting. + - Refer to the `pom.xml` to see what it the correct setting for any individual project. + +Generally speaking: + +- core tests don't use UI thread and use the headless harness. +- ui tests use the UI thread and the UI harness +- swtbot tests don't use the UI thread but do use the UI harness + +As an example, this is how the `org.eclipse.cdt.core.tests` project is configured: + +![junit_ui_settings.png](images/junit_ui_settings.png "screenshot of how to configure UI settings") + +## File name patterns + +The build machine generally use the standard [Tycho Surefire](https://www.eclipse.org/tycho/sitedocs/tycho-surefire-plugin/plugin-info.html) class name patterns for [includes](https://www.eclipse.org/tycho/sitedocs/tycho-surefire-plugin/integration-test-mojo.html#includes) and [excludes](https://www.eclipse.org/tycho/sitedocs/tycho-surefire-plugin/integration-test-mojo.html#excludes) to identify tests during automated builds. +See [BUILDING](BUILDING.md) for more information. + +## Marking tests as Slow or Flaky Note that there could be intermittent failures in random tests, if you are getting those, try to rerun the tests. -Tests can be marked as flaky, see the subsequent section for more details. -### Marking tests as Slow or Flaky +Tests in CDT can be marked as Slow or Flaky to prevent them running as part of the standard test suites. +See excludedGroups section in [BUILDING](BUILDING.md) to skip slow or flaky tests sections above. -Tests in CDT can be marked as Slow or Flaky to prevent them running as part of the standard test suites. See excludedGroups to skip slow or flaky tests sections above. +To exclude (or run) slow or flaky tests in the GUI, configure the `Include and exclude tags` in the JUnit launch configuration. + +![junit5_excludes.png](images/junit5_excludes.png "screenshot of how to configure include and exclude tags") The proper way to mark a test as slow or flaky is to add a JUnit5 @Tag on the test with `flakyTest` or `slowTest`. The canonical values for these are in the JUnit5 base test `org.eclipse.cdt.core.testplugin.util.BaseTestCase5`. These tags can only be applied to JUnit5 (aka Jupiter) tests. If a test needs converting, do that in a separate commit before adding the tags so that the test refactoring can be verified before excluding the test from normal runs. -### Converting tests to JUnit5 +## Converting tests to JUnit5 To take advantage of new features, such as excluding flaky and slow tests, individual tests need to JUnit5 (aka Jupiter). If a test is currently written in JUnit4 or JUnit3 style it needs to be converted to JUnit5 first. Those tests that currently derive from `org.eclipse.cdt.core.testplugin.util.BaseTestCase` can change to `org.eclipse.cdt.core.testplugin.util.BaseTestCase5` and make further adjustments. Common adjustments are: - refactoring `setUp`/`tearDown` methods to use `@BeforeEach` and `@AfterEach` annotations diff --git a/images/junit5_excludes.png b/images/junit5_excludes.png new file mode 100644 index 00000000000..110f5040d98 Binary files /dev/null and b/images/junit5_excludes.png differ diff --git a/images/junit_ui_settings.png b/images/junit_ui_settings.png new file mode 100644 index 00000000000..6710f985409 Binary files /dev/null and b/images/junit_ui_settings.png differ diff --git a/images/junit_use_features.png b/images/junit_use_features.png new file mode 100644 index 00000000000..347c799b072 Binary files /dev/null and b/images/junit_use_features.png differ