## Pull Request
### ci: Run tests in parallel (part 1)
- Tests can now be arbitrarily grouped into sub-directories.
- Some tests are now run in parallel.
- CI will now spawn 4 jobs to run the whole test suite in parallel.
## Individual Commits
### tests(chore): Rename test files to serial and parallel types
- `test_helper.bats` needs more work than this PR provides to be compatible with parallel tests, so must remain as a serial test for now.
- `spam_bounced.bats` had failures as a serial test, but works well converted to a parallel test in a future commit.
### tests(CI): Adjust Makefile & GHA workflow to support new test layout
These updates support running tests that have been relocated into `serial` and `parallel/set*` directories.
- `make tests` now calls the two make targets beneath it. The only difference is that `serial` continues the "1 test at a time" approach used prior to this PR, while the `parallel` target increases the `--jobs` arg to run multiple tests concurrently (_configured by `PARALLEL_JOBS`_).
- The `test/%` target leverages Bash syntax magic to ease running single tests without providing the exact path.
- This syntax also supports providing multiple test names (eg: `make test/clamav,template`) to run.
- `**` (globstar) allows for future improvements that can group multiple test files into sub-directories by their scope (eg: anti-spam, ssl, etc).
---
chore: Add `shopt -s globstar` to other targets
I realized that other targets should have this as well in case it is not set.
It is better to be more explicit here than to have weird errors due to `**` not expanding properly.
---
fix(Makefile): Add back `.PHONY` targets
I encountered `make` telling me the target was already up-to-date, which of course is nonsense.
I therefore added back the `.PHONY` targets to ensure tests are always run.
---
docs: Added instructions for running a single test
See https://github.com/docker-mailserver/docker-mailserver/pull/2857/files#r1008582760
### tests(chore): Use `REPOSITORY_ROOT` export var from Makefile
Allows for using `load` with an absolute path instead of a relative one, which makes it possible to group tests into different directories.
Parallel tests differ slightly, loading the newer `helper/common.bash` and `helper/setup.bash` files instead of the older `test_helper/common.bash` which serial tests continue to use.
### tests(refactor): `common.bash` helper split into two files
The current `test/test_helper/common.bash` was getting large. Setup logic has been extracted out into a new file.
`common.bash` resides in a directory named `test_helper/`, the `test_` prefix is redundant.
As an interim solution this provides a new approach for the updated tests, while the "old" tests can use the "old" `common.bash`. Eventually all tests should migrate to the new approach in `helper/` instead of the older `test_helper/`.
The new helper files are located under `test/helper/` (_which drops the `test_` prefix_). The new and updated helpers apply the new naming convention for ENV variables (_such as `CONTAINER_NAME` or `IMAGE_NAME`_).
---
Some refactoring occurred, including new methods like `_run_in_container()` and `_default_teardown()`.
---
I encountered a situation before in which the updated tests would fail because there were collisions of ENV names in the tests (_for example with `CONTAINER_NAME`_).
### tests(refactor): Conversion to parallel tests and use revised helpers
- Introduced `CONTAINER_NAME` and `TEST_NAME_PREFIX` as new vars for better managing test consistency (DRY).
- `CONTAINER_NAME` replaces any repeated container name with the variable. The value will differ slightly as the prior prefix (`mail_`) has been changed to `dms-test-`.
- `TEST_NAME_PREFIX` provides a prefix value for each `@test` description string.
---
chore: Add a reference template for tests
- Introduced `CONTAINER_NAME` and `TEST_NAME_PREFIX` as new vars for better managing test consistency (DRY).
- `CONTAINER_NAME` replaces any repeated container name with the variable. The value will differ slightly as the prior prefix (`mail_`) has been changed to `dms-test-`.
- `TEST_NAME_PREFIX` provides a prefix value for each `@test` description string.
---
chore: Add a reference template for tests
The current `test/test_helper/common.bash` was getting large. Setup logic has been extracted out into a new file.
`common.bash` resides in a directory named `test_helper/`, the `test_` prefix is redundant.
As an interim solution this provides a new approach for the updated tests, while the "old" tests can use the "old" `common.bash`. Eventually all tests should migrate to the new approach in `helper/` instead of the older `test_helper/`.
The new helper files are located under `test/helper/` (_which drops the `test_` prefix_). The new and updated helpers apply the new naming convention for ENV variables (_such as `CONTAINER_NAME` or `IMAGE_NAME`_).
---
Some refactoring occurred, including new methods like `_run_in_container()` and `_default_teardown()`.
---
I encountered a situation before in which the updated tests would fail because there were collisions of ENV names in the tests (_for example with `CONTAINER_NAME`_).
Allows for using `load` with an absolute path instead of a relative one, which makes it possible to group tests into different directories.
Parallel tests differ slightly, loading the newer `helper/common.bash` and `helper/setup.bash` files instead of the older `test_helper/common.bash` which serial tests continue to use.
These updates support running tests that have been relocated into `serial` and `parallel/set*` directories.
- `make tests` now calls the two make targets beneath it. The only difference is that `serial` continues the "1 test at a time" approach used prior to this PR, while the `parallel` target increases the `--jobs` arg to run multiple tests concurrently (_configured by `PARALLEL_JOBS`_).
- The `test/%` target leverages Bash syntax magic to ease running single tests without providing the exact path.
- This syntax also supports providing multiple test names (eg: `make test/clamav,template`) to run.
- `**` (globstar) allows for future improvements that can group multiple test files into sub-directories by their scope (eg: anti-spam, ssl, etc).
---
chore: Add `shopt -s globstar` to other targets
I realized that other targets should have this as well in case it is not set.
It is better to be more explicit here than to have weird errors due to `**` not expanding properly.
---
fix(Makefile): Add back `.PHONY` targets
I encountered `make` telling me the target was already up-to-date, which of course is nonsense.
I therefore added back the `.PHONY` targets to ensure tests are always run.
---
docs: Added instructions for running a single test
See https://github.com/docker-mailserver/docker-mailserver/pull/2857/files#r1008582760
- `test_helper.bats` needs more work than this PR provides to be compatible with parallel tests, so must remain as a serial test for now.
- `spam_bounced.bats` had failures as a serial test, but works well converted to a parallel test in a future commit.
Currently a change detection would be triggered and during processing, a CRLF is converted to LF, which updates the `postfix-accounts.cf` file and triggers another change event.
No need for the first approach to add an account, and it is the culprit for causing the CRLF to appear.
This new script is a clean way of handling the installation of packages.
I think the huge `RUN` command in `Dockerfile` was hard to read and
maintain.
Using a script is a non-issue, as the image is rebuilt whenever the
script is touched.
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
dependabot[bot]
github-actions[bot]
Brennan Kinney
Georg Lauterbach
Casper
Leon George