Custom parameters must be referenced to be retained when `postconf -n` is run. If those parameters are referenced by `postfix-master.cf` this needs to update `master.cf` before updating `main.cf`.
* chore: Detect missing final newline in configs read
These lines will be not be processed by `read`, emit a warning to raise awareness.
* fix: Ensure parsed config has final newline appended (when possible)
This functionality was handled in `accounts.sh` via a similar sed command (that the linked references also offer).
`printf` is better for this, no shellcheck comment required either.
We additionally don't attempt to modify files that are read-only.
* fix: Ensure parsed configs have CRLF to LF corrected (where possible)
Likewise, this runtime fix was only covering two config files. It now applies to all callers of this method.
* fix: Sanitize `postfix-master.cf` via helper
This feature should have been using the helper to avoid user error from their config updates accidentally introducing subtle breakage implicitly (due to CRLF or missing final newline).
* tests: Add test cases for new helpers
* tests: `rm` is redundant when using `BATS_TEST_TMPDIR`
This temporary directory is created and removed implicitly. Even after a test failure.
* chore: Remove old `postfix-virtual.cf` migration logic
This was introduced in 2018, there should be no one needing to rely on this anymore?
* tests: Remove comment on sed failure concern
* chore: Add entry to `CHANGELOG.md`
* Apply suggestions from code review
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* move log/filter functions into own file
* add ShellCheck global directives
* use new function for tracking logs
The new function, called `_send_email_with_mid`, aligns with suggestions
from @polarethene and is heavily simplified compared to its predecessor
`_send_email_and_get_id`. New helpers will be introduced to filter logs
according to the MID constructed in this function.
* new filters for searching logs with MID
* use new filters (and sending) functions
* add new helper for asserting non-existence of log message
* use new filters in tests
* Apply suggestions from code review
- `_mid` / `MID` => `_msgid` / `MSG_ID`
- Revised documentation / tooltip comments
* Apply suggestions from code review
* fix tests
* use more distinct names for MSG_ID headers
* update `_filter_service_log` to not use `-i -E`
Moreover, I added a function to print the whole mail log. Appropriate
comments were added to this function to indicate that one should only
use this function when necessary.
* adjust helpers to new helper filter
* follow-up of previous commit
* add CHANGELOG entry
* Apply suggestions from code review
* chore: Update OAuth2 to use new log helper
* Apply suggestions from code review
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* added explicit `_regexp` filters for logs
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* fix: Dovecot PassDB should restrict allowed auth mechanisms
This prevents PassDBs incompatible with certain auth mechanisms from logging failures which accidentally triggers Fail2Ban.
Instead only allow the PassDB to be authenticated against when it's compatible with the auth mechanism used.
* tests: Use `curl` for OAuth2 login test-cases instead of netcat
`curl` provides this capability for both IMAP and SMTP authentication with a bearer token. It supports both `XOAUTH2` and `OAUTHBEARER` mechanisms, as these updated test-cases demonstrate.
* chore: Add entry to `CHANGELOG.md`
* tests: OAuth2 - Replace Python `/userinfo` endpoint with Caddy
Better documented, easier flow and separation of concerns via Caddy.
The python code had additional noise related to setting up a basic API which is abstracted away via `Caddyfile` config that's dedicated to this task.
* tests: OAuth2 - Minimize noise + Improve test assertion
Caddyfile can use an Access Token instead of a JWT. Much smaller and correct for this OAuth2 configuration. This new value has been documented inline.
Likewise the `sub` field returned is not important to this test. `email_verified` is kept as it may be helpful for further coverage testing.
The actual test-case has better assertions for success and failure by checking for Dovecot logs we expect instead of netcat response.
`oauth2` to `auth` for the Caddy container hostname is not necessary, just a more generic subdomain choice.
* tests: OAuth2 - Caddyfile `imap/xoauth2` route dynamic via query string
This way is more flexible and doesn't require modifying the `Caddyfile` directly, while still easy to use.
Additionally simplifies understanding the Caddyfile to maintainers by removing the `route` directive that was required to ensure a deterministic order of vars.
* tests: OAuth2 - `/imap/xoauth2` respond with IMAP commands for netcat
Since this is the only intended usage, might as well have it respond with the full file content.
* tests: OAuth2 - Implement coverage for `OAUTHBEARER`
Caddyfile route for `/imap/` now accepts any subpath to support handling both `xoauth2` and `oauthbearer` subpaths.
Both SASL mechanisms represent the same information, with `XOAUTH2` being a common mechanism to encounter defined by Google, whilst `OAUTHBEARER` is the newer variant standardized by RFC 7628 but not yet as widely adopted.
The request to `/userinfo` endpoint will be the same, only the `credentials` value to be encoded differs.
Instead of repeating the block for a similar route, this difference is handled via the Caddyfile `map` directive.
We match the path context (_`/xoauth2` or `/oauthbearer`, the `/imap` prefix was stripped by `handle_path` earlier_), when there is a valid match, `sasl_mechanism` and `credentials` map vars are created and assigned to be referenced by the later `respond` directive.
---
Repeat the same test-case logic, DRY with log asserts extracted to a common function call. This should be fine as the auth method will be sufficient to match against or a common failure caught.
* tests: OAuth2 - Minor revisions
Separate test cases and additional comment on creating the same base64 encoded credentials via CLI as an alternative to running Caddy.
Added a simple `compose.yaml` for troubleshooting or running the container for the `/imap/xoauth2` / `/imap/oauthbearer` endpoints.
* tests: OAuth2 - Route endpoints in Caddyfile with snippets instead
`reverse_proxy` was a bit more convenient, but the additional internal ports weren't really relevant. It also added noise to logging when troubleshooting.
The `import` directive with Snippet blocks instead is a bit cleaner, but when used in a single file snippets must be defined prior to referencing them with the `import` directive.
---
`compose.yaml` inlines the examples, with slight modification to `localhost:80`, since the Caddyfile examples `auth.example.test` is more relevant to the tests which can use it, and not applicable to troubleshooting locally outside of tests.
* chore: Add entry to `CHANGELOG.md`
* chore: Additional context on access token
* feat: add support for MTA-STS for outgoing mails
* Hook-up mta-sts-daemon into basic process handling test
* fix: Call python script directly
The python3 shebang will run it, which will now meet the expectations of the process testing via pgrep. fail2ban has the same approach.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Disable `smtputf8` support in config
This was always configured disabled at runtime, better to just set explicitly in `main.cf` unless config diverges when Dovecot is enabled to opt-out of this feature.
* fix: `supervisor-app.conf` - Correct `postgrey` log location
Looks like this should have been like every other service and reference a log file(s) based on program name in the supervisor log directory.
* tests: Adjust log location for `postgrey_enabled.bats`
* chore: Extract out Dovecot Quota test cases into new test file
Test cases are just cut + paste, no logic changed there yet.
* chore: Rename test case descriptions
* chore: Use `setup ...` methods instead of direct calls
* chore: Adjust `_run_in_container_bash` to `_run_in_container`
Plus some additional bug fixes in the disabled test case
* tests(refactor): Revise ENV test cases for max mailbox and message sizes
* tests(refactor): Revise ENV test cases for mailbox and message limits v2
Removes the extra variables and filtering in favour of explicit values instead of matching for comparison.
- Easier at a glance to know what is actually expected.
- Additionally reworks the quota limit checks in other test cases. Using a different formatter for `doveadm` is easier to match the desired value (`Limit`).
* chore: Sync improvement from `tests.bats` master
---
NOTE: This PR has been merged to avoid additional maintenance burden without losing the improvements. It was not considered complete, but remaining tasks were not documented in the PR.
* outsource Rspamd ENVs into explicit helper
This will allow us to uniformly source the helper and get the values
from everywhere consistently. This is more than desirable since we will
be using these values not only for the Rspamd setup, but also for DKIM
management and during change-detection.
* integrate Rspamd into changedetection
We outsource one more function to reside in the helper script for Rspamd
so that we can call this function from the Rspamd setup and from the
changedetection functionality too.
* realize deprecation of old commands file for Rspamd
THIS IS A BREAKING CHANGE!
This change realizes the log message: "Using old file location now
(deprecated) - this will prevent startup in v13.0.0" Startup will now
fail.
* added '--force' option to Rspamd DKIM script
* use new helper to get ENVs for Rspamd in DKIM script
* remove the need for linking directories
This was unnecessary, as explained in
https://github.com/docker-mailserver/docker-mailserver/pull/3597#discussion_r1369413599
* Apply suggestions from code review
review by @polarathene
* apply more review feedback from @polarathene
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370885519>
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370904201>
* update documentation
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Instead of using `etc/rspamd/override.d/dkim_signing.conf`, we will now
be using `/tmp/docker-mailserver/rspamd/override.d/dkim_signing.conf`.
The new location is persisted (and linked again during startup) and
hence better suited.
* simplify `_setup_logrotate`
* adjust Rspamd's log file and improve it's management
* add information to docs about Rspamd log
* update log query helper to allow another file location
* bail in case `LOGROTATE_INTERVAL` is invalid
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Some deployment scenarios are not compatible with `5000:5000` static vmail user with `/var/mail`. This feature allows adjusting the defaults to a UID / GID that is compatible.
Signed-off-by: vincent <vincent@ducamps.win>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Instead of exit status of `124` (_signifies timeout_), it should fail with `1` (failure) like the others. Handled via using `_run_in_container_bash()` (_`timeout` failure `124` does not propagate and is treated as `1` instead_).
In this case we are waiting on the status of the mail being sent, the pattern provided to `grep` is too specific and results in a timeout. Instead since we only expect the one log entry, match any status and assert the expected pattern afterwards.
This provides a more helpful failure output that informs us that mail was at least processed by Postfix, but the sent status is not what we expected.
### Before
```
✗ [ENV] (POSTFIX_DAGENT) delivers mail to existing account [60327]
(from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
in test file test/tests/parallel/set3/mta/lmtp_ip.bats, line 47)
`assert_success' failed
-- command failed --
status : 124
output :
--
```
### After
```
✗ [ENV] (POSTFIX_DAGENT) delivers mail to existing account [1425]
(from function `assert_output' in file test/test_helper/bats-assert/src/assert_output.bash, line 178,
in test file test/tests/parallel/set3/mta/lmtp_ip.bats, line 48)
`assert_output --regexp "${MATCH_LOG_LINE}=sent .* Saved)"' failed
-- regular expression does not match output --
regexp : postfix/lmtp.* status=sent .* Saved)
output : Sep 28 04:12:52 mail postfix/lmtp[721]: 23701B575: to=<user1@localhost.localdomain>, relay=127.0.0.1[127.0.0.1]:24, delay=0.08, delays=0.07/0/0.01/0, dsn=4.2.0, status=deferred (host 127.0.0.1[127.0.0.1] said: 451 4.2.0 <user1@localhost.localdomain> Internal error occurred. Refer to server log for more information. [2023-09-28 04:12:52] (in reply to end of DATA command))
--
```
The expected pattern is logged as `assert_success` confirms a valid match for the log line of interest was found, and we have the mismatched value to debug the failure against.
This was missed during original review.
On a linux host, processes running within a container have been visible via commands like `pgrep`. This is does not appear to be the case with WSL2 + Docker Desktop (Windows), resulting in test failure.
The command should have been run from within the container regardless.
* chore: Use `.yml` extension
Both of these files support the `.yml` extension. Normalize on that.
* fix: Add `.gitattributes` to ensure `LF` line-endings are committed
Avoids accidentally committing files with `CRLF` when they're created on Windows. Or worse, if some editors don't detect `LF` and would introduce mixed line-endings with `CRLF`.
Shouldn't be a problem in practice as we already have a linting check to catch this via CI during PRs. This file is complimentary, in that it should automate that concern away.
* chore: Drop management of `SASLAUTHD_*` ENV
- `variables-stack.sh` does not need to manage all these extra ENV or store them. They're not used anywhere else.
- `saslauthd.sh` is the only consumer of these ENV which are effectively direct key/value mappings, with some defaults provided / inherited.
Instead of trying to conditionally support key/value pairs when ENV is set, we could instead use `sed` to delete lines with empty values.
* chore: Drop fallbacks + update configs to match docs
- Drop deprecated support:
- `DOVECOT_HOSTS` is an ENV deprecated since v10.
- Fallback for missing URI scheme introduced for Dovecot and SASLAuthd in v10.
- Adding error log message when no LDAP URI scheme is detected for the supported ENV (when set).
- Docs updated for ENV to reflect the mandatory requirement. `mailserver.env` partially synced equivalent sections.
- Provided base LDAP configs (for overriding) likewise updated from `domain.com` to `example.com`.
- LDAP test updated for required `ldap://` URI scheme. Common ENV shared across LDAP configs hoisted out of the Postfix group.
* chore: Remove unset lines in generated `saslauthd.conf`
* chore: Use white-space in query filters to improve readability
* tests: LDAP ENV query filters documented
- These filters remain roughly the same as they were before. The conditions are the same, but restructured to make the complimentary constraints more separated from the actual target attribtues.
- The DOMAIN query additionally includes the `mailAlias` from `PostfixBookMailAccount` class as well.
- Heavy inline documentation breaking down how the filters work for any maintainer to reference. This will likely be migrated after revision into our user docs for LDAP. Some quirks have also been noted with advice for future changes.
* tests: LDAP - Support test-case specific containers
A bit more complicated than other test files due to the larger ENV config array that most containers may need to share.
Example introduced with the test-case checking custom config file support.
* tests: Adjust LDAP test configs
- Paths for `.ldif` files used with volumes shortened
- Postfix LDAP `.cf` files adjusted to conventions used in LDAP tests.
- The `uniqueIdentifier` attribute is not appropriate and was relying on `objectClass: extensibleObject` as a workaround to allow it. A more appropriate attribute to use instead is `userID` (_short name: `uid`_).
- Removing `extensibleObject` now requires switching the user accounts to use `inetOrgPerson` class (_which inherits from `organizationalPerson`_). which allows the attributes `givenName`, `userID` and `mail` (_also provided via the `PostfixBookMailAccount` class_).
- The LDAP root object now uses `dc` attributes for `example.test` instead of `localhost.localdomain`. This has nothing to do with DMS or LDAP containers networking config, nor the users mail addresses.
- Users are now grouped under the organizational unit of `users` instead of `people`. Purely a naming change out of preference, no functional difference.
The LDAP test ENV has been updated to accommodate the above changes. An additional ENV override was required for SASLAuthd to switch an attribute set for `ldap_filter` in `/etc/saslauthd.conf` from the implicit default of `uniqueIdentifier` (_that we set during startup as an ENV default for fallback_) to the `userID` attribute.
* chore: Adjust default DKIM size (`open-dkim`) from 4096-bit to 2048-bit
4096-bit is excessive in size for DKIM key. 2048-bit is plenty.
* chore: Additional revisions to `open-dkim` command help output
- The examples use `keysize 2048`, but as that's the new default it makes sense to change that.
- Other help text was also revised.
- Last example for domains did not need to demonstrate the other options. Changed example domains to more appropriate values.
* docs: Revise DKIM docs
Primarily for the change in default key size, but does revise some text to better communicate to the user.
- While the referenced RFC advises 512-bit to 2048-bit key size, we now explicitly discourage `512-bit` as it's not secure. `1024-bit` is still likely safe for most, but `2048-bit` is a good default for those not rotating their keys.
- Adjusted the domains example to match the new `setup config dkim domain` domains example.
- Tip for changing default key size changed to "info" with added clarity of lowering security or increasing it (excessively).
- Rspamd section is minor formatting changes, with the exception of clarifying the "main domain" for the mail accounts is assumed as the DMS FQDN with any subdomain (like `mail.`) stripped away. This is not great, but a legacy issue that needs to be addressed in future.
- `docs-rspamd-override-d` ref removed, and usage replaced with equivalent ref `docs-rspamd-config-dropin`, while `docs-rspamd-config-declarative` ref was not in use and also removed.
- Revised the `<selector>.txt` DNS formatting info section to better communicate with the reader. Additionally it had mixed usage of default `mail` and custom `dkim-rsa` selectors (_file content and output_).
* docs: Sync DKIM commands help messages and update DKIM docs for LDAP
- Adopt the help options format style from the `rspamd-dkim` into `open-dkim` command. And convert `./setup.sh` to `setup`. `selector` option has been implemented. for a while now.
- Update `rspamd-dkim` examples help output to align with `open-dkim` command examples.
- Give both DKIM command tools a consistent description. The two tools differ in support for the `domain` option (_implicit domain sourcing for default account provisioner, and support for multiple domains as input_).
- DKIM docs for LDAP domain support revised to better communicate when explicit domain config is necessary.
* tests: Adjust test-cases for `setup config dkim` change
`rspamd_dkim.bats`:
- Update assert for command help output.
- Don't bother creating a DKIM key at 512-bit size.
`setup_cli.bats`:
- Update assert for command help output of the `setup config dkim` (OpenDKIM) command.
* docs: Update DKIM section for large keys to newer RFC
The linked discussion from 2021 does mention this updated RFC over the original. That removes outdated advice about `512-bit` key length support.
The discussion link is still kept to reference a comment for the reader to better understand the security strength of 2048-bit RSA keys and why larger keys are not worthwhile, especially for DKIM.
* docs: Extract out common DKIM generation command from content tabs
Should be fine to be DRY here, not specific to `open-dkim` or `rspamd` generation/support. Previously rspamd lacked support of an equivalent command in DMS.
* docs: DKIM refactoring
- Shifted out the info admonition on key size advice out of the content tabs as it's now generic information.
- Indented the 4096-bit warning into this, which is less of a concern as the default for our DKIM generation tools is consistently 2048-bit now.
- Reworked the LDAP and Rspamd multi-domain advice. To avoid causing a bad diff, these sections haven't been moved/merged yet.
* docs: Revise DKIM docs
Advice for managing domains individually with LDAP and Rspamd extracted out of the content tabs. Default domain behaviour explained with extra info about OpenDKIM + FILE provisioner sourcing extra domains implicitly.
**TL;DR:**
- New image is actively maintained vs existing one that is over 5 years old.
- Slight improvement to LDAP tree config via `.ldif` files.
- No more `Dockerfile` required to build, we can just rely on `docker run`.
`osixia/openldap` has not seen any activity since Feb 2021, while our `Dockerfile` was fixed to v1.1.6` (Feb 2018).
Startup time for this new image is around 5 seconds? (_The LDAP test uses a standard 20 second timeout check to wait until the server is ready before continuing with starting the DMS image_).
This commit migrates to `bitnami/openldap` which required modifying the `01_mail-tree.ldif` to also include adding the root object to start successfully. This image is actively maintained and one of the most popular OpenLDAP images on DockerHub.
The user account `.ldif` files have minimal changes:
- Lines moved around for better organization
- Additional comments for context
- Removal of inherited `objectClass` attributes (`person`, `top`) from the `orgnizationalPerson` class. Attribute `sn` changed to long form `surname` and values corrected with `givenName`. `changetype: add` was also not necessary.
Additionally the image does not support the `.schema` format, they must be converted to `.ldif` which has been done for `postfix-book.schema`.
See PR for more details.
* tests: Switch to setup helper and conventions
* tests: Adapt run command to new conventions
- We have two helper methods with implicit `CONTAINER_NAME` reference, which is a bit more DRY and improves readability.
- `wc -l` + `assert_output 1` converted to use helper `_should_output_number_of_lines 1`
- `DOMAIN` var changed from `my-domain.com` to local testing domain `example.test`.
* tests: Refactor `setup_file()`
- Test wide ENV defined at the top
- OpenLDAP build and run logic grouped together. Added notes, network alias and tty not required.
- Extracted out special LDAP Postfix/Dovecot ENV into separate array. LDAP specific provisioning / auth ENV also included, with comments + linebreak to better group related ENV.
- Likewise additional ENV to support test cases has been extracted to a separate array with additional comments for context.
- Those two arrays are expanded back into the main `CUSTOM_SETUP_ARGUMENTS` that configure hostname and network for the DMS container.
* tests: Refactor the LDAP account table query testcase
- Covers 3 accounts to test from LDAP.
- 2 are the same query against users/aliases/groups tables in Postfix, only differing by account queried (and expected as returned result).
- 1 separate test to ensure a difference in config is supported correctly.
- Extracted repeated test logic into a helper method.
- Added additional context in comments about the creation source of these LDAP accounts and their related Postfix config / interaction. Direct reference to special case PR (since `git blame` will be less useful).
* tests: Use iteration for `grep` setting checks
More DRY approach. With a bit more helpful failure context via `assert_output` (_and only grepping the key_). Simpler to grok what's being covered.
* tests: DRY test email delivery
A bit more verbose with the new helper method. `test-email.txt` template is only used by the LDAP test, as is the `sendmail` command.
Helper will take two args to support the testcases, but at a later date should be refactored to be consistent with the `_send_email()` helper (_which presently uses `nc` that is required for plain-text mail/auth, otherwise we'd have used `openssl`, bigger refactor required_).
* tests: Slight revisions and relocating testcases
- Dovecot quota plugin testcase revised to check files exist instead of rely on `ls` failure.
- Moved Postfix quota plugin testcase into prior dovecot testcase for quota plugin check. Better error output by only querying the `smtpd_recipient_restrictions` setting (_which should be the only one configured for the service_).
- Moved the saslauthd and pflogsumm testcases (_no changes beyond revised comments_) above the `ATTENTION` comment, and one testcase below the comment that belonged to that group.
* tests: Simplify openldap `docker build` command
- `--no-cache` was creating a new image on the Docker host each time the test is run. Shouldn't be any need to build without cache.
- No need to use `pushd` + `popd`, can just provide the path context directly, and the `./Dockerfile` is an implicit default thus `-f` not required either.
Additionally removed the old `checking` prefix from testcase names.
* tests: Move LDAP specific config into `test/config/ldap/`
- No changes to any of these config files, just better isolation as not relevant to any other tests.
- Section heading in `setup_file()` added to distinguish the remainder of the function is dedicated to the DMS container setup.
- Comment providing some context about the `mv` to maintainers, this should be done after defaults are initialized but before starting up the container.
* chore: Appease the lint gods
* Apply suggestions from code review
Brennan Kinney
Ikko Eltociear Ashimine
Kirill Kirilenko
Georg Lauterbach
Casper
Joerg Sonnenberger
Keval Kapdee
allddd
Vincent Ducamps
H4R0