Commit Graph

322 Commits

Author SHA1 Message Date
Georg Lauterbach 03772f612a
scripts: get all `policyd-spf` setup in one place (#3263) 2023-04-15 00:40:42 +02:00
Georg Lauterbach 9a284150b2
Rspamd: replace `reject_unknown_client_hostname` with Rspamd `HFILTER_HOSTNAME_UNKNOWN` and make it configurable (#3248) 2023-04-11 18:51:23 +02:00
Georg Lauterbach 806d3efef9
Rspamd: add greylisting option & code refactoring (#3206) 2023-04-11 09:16:57 +02:00
Georg Lauterbach 9ee33a81b7
scripts: make `policyd-spf` configurable (#3246) 2023-04-11 08:52:43 +02:00
Casper f4fe5bf527
Update SA_KILL values; follow up to #3058 (#3204) 2023-03-27 01:59:43 +02:00
Casper 6fa06f4986
Fix: only chmod when there are files (#3203) 2023-03-26 20:30:34 +08:00
Georg Lauterbach e58dd1b95b
Rspamd: more features (#3159) 2023-03-18 23:32:48 +08:00
Casper 2da3e1b022
fix: SRS setup (#3158) 2023-03-12 12:10:45 +01:00
Jack Pearson 0baf82f7d0
fix: TLS setup (self-signed) error message should include `SS_CA_CERT` (#3168) 2023-03-10 18:25:46 +13:00
Georg Lauterbach dab70709d9
scripts: improve panic helpers (#3155) 2023-03-06 10:06:50 +01:00
Georg Lauterbach b5fc40eb7a
fix regression introduced in #3153 (#3157) 2023-03-05 22:23:25 +01:00
Casper a8f6fa6181
DRY: Replace path with variable in mail_state.sh (#3153) 2023-03-05 16:59:01 +01:00
Georg Lauterbach 4e9ffbf224
fix(Postfix): special bits for maildrop and public directory (#3149)
* fixed special bits for maildrop and public dir

After changing the group, special bits are lost, but they should be set for the directories `/var/spool/postfix/{maildrop,public}`, otherwise you see the following error:

```
postfix/postdrop[17400]: warning: mail_queue_enter: create file maildrop/729504.17400: Permission denied
```

* fix: Match octal permissions originally provided

Officially Postfix source seems to imply:
- `730` for `maildrop/` (_but has mentioned a sticky bit in the past, set-gid bit only for the postdrop binary involved_)
- `710` for `public/`

Both folders are assigned the same group that `postdrop` belongs to which has the SGID permission for it's executable. SGID special bit on`public/` doesn't seem necessary, but left as-is to match the default from Debian.

---------

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-03-05 20:57:40 +13:00
Georg Lauterbach 8ec5dbec74
scripts: touchups for v12.0.0 (#3144) 2023-03-04 10:57:42 +01:00
Georg Lauterbach 5e9849d94f
rspamd: rename `ENABLE_REDIS` & add persistence for Redis (#3143) 2023-03-04 10:45:43 +01:00
Georg Lauterbach f0edcc28d9
config: ensure SASL socket file is not inside a volume mount (#3131) 2023-03-03 23:42:55 +01:00
Brennan Kinney 9a9380150e
fix: Avoid creating an unnecessary syslog socket for Postfix (#3134)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2023-03-03 23:08:51 +01:00
Georg Lauterbach 5ec6845c96
config/ENV: improve Postfix config for spoof protection (#3127) 2023-03-03 15:55:32 +01:00
Georg Lauterbach 9e2f96448a
scripts: remove PostSRSD wrapper (#3128)
* remove PostSRSD wrapper

The setup is now completely done during _actual_ setup phase. The
wrapper did not even catch signals (SIGINT, etc.), which I think is
strange.

I also added all the ENVs the wrapper relied on (which previously could
have been unset) to the variables script.

* forgot adjusting the `Dockerfile`
2023-03-02 17:44:01 +01:00
Georg Lauterbach f3edcf9cd5
issue warning in case of improper restart (#3129) 2023-03-01 16:33:54 +01:00
Georg Lauterbach 0949f16344
scripts/ENV: make disabling Redis possible (#3132)
* make disabling Redis possible

* add documentation

* Apply suggestions from code review

* Update docs/content/config/environment.md

---------

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-03-01 16:24:59 +01:00
Georg Lauterbach dfc2b39308
scripts: housekeeping & cleanup setup (2/2) (#3123) 2023-02-28 10:25:23 +01:00
Georg Lauterbach 9ead9a54ac
scripts: follow up of #3115 (feedback) (#3124) 2023-02-27 23:37:35 +01:00
Georg Lauterbach 4b04c3e31c
scripts: housekeeping & cleanup setup (1/2) (#3121) 2023-02-27 20:21:45 +01:00
Georg Lauterbach f35b60042f
scripts: split `setup-stack.sh` (#3115) 2023-02-26 11:42:14 +01:00
Georg Lauterbach bee9e3627d
rspamd: add feature for adjusting options with a file parsed by DMS (#3059)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-02-19 12:36:43 +01:00
Georg Lauterbach 40e10d755d
setup: improve Amavis setup routine (#3079)
* improve Amavis setup routine

see https://github.com/docker-mailserver/docker-mailserver/pull/3046#issuecomment-1423718811
2023-02-18 19:22:20 +01:00
Georg Lauterbach 632012aead
add information so `Envelope From` is properly set (#3081) 2023-02-18 16:21:51 +01:00
Brennan Kinney 1c8a160621
chore: Remove delay starting the change detection service (#3064)
* chore: Only replace `CHKSUM_FILE` when a change has been processed

* chore: Change Detection service should be the last daemon started

* chore: Remove 10 second startup delay for change detector

There should be no concern with conflicts as any writes should have already been done by the time this daemon service is started.

* tests(fix): `smtp_delivery.bats` must wait for Amavis

The change event for adding a user can be processed much sooner now, which means Amavis may not yet be ready.

Added extra condition to wait on at least the Amavis port being reachable, and some failure asserts with the mail queue to better catch / debug when this problem occurs.

* chore: Add some minor delay to avoid Amavis failing to connect
2023-02-18 15:51:28 +01:00
Casper 26861ddf92
fix order (#3082) 2023-02-13 00:05:52 +01:00
Georg Lauterbach ac1df91181
chore(Postfix): disable DNSBLs (#3069) 2023-02-11 00:24:09 +01:00
Casper 021a4a3c09
Fix: Make logrotate state persistant (#3077)
Fixes https://github.com/docker-mailserver/docker-mailserver/issues/3075
2023-02-09 22:34:26 +00:00
Brennan Kinney 88767f7cc8
tests(refactor): `open_dkim.bats` (#3060)
* tests(refactor): Make test cases for opendkim keysizes DRY

- These all do roughly the same logic that can be split into two separate methods.
- `_should_generate_dkim_key()` covers a bit more logic as it can be leveraged to handle other test cases that also perform the same logic.
- The `config/opendkim/` doesn't seem necessary for tests. Only the first few test cases here are testing against it, so we can conditionally make that available. `process_check_restart.bats` also depended on it to run OpenDKIM successfully, but this was due to the `setup-stack.sh` config defaults failing to find an "empty" file forcing `supervisord` to constantly restart the process..
- With this, there we inverse the default opendkim config, so we don't have to mount unique / empty subfolders for each test case, followed by copying over the two extra configs.

* tests(refactor): DRY up more test cases

All the remaining test cases but the last one were refactored here for a clean commit diff. The last test case will be refactored in the following commit.

Plenty of repeated logic spread across these test cases, now condensed into shared methods.

* tests(refactor): Make final test case DRY

* chore: Migrate to new testing helpers

* chore: Revise test case descriptions

* tests(refactor): Improve and simplify assertions

* tests(refactor): Use common container setup instead of `docker run`

- As the majority of test cases are only running `open-dkim` helper, we don't actually have to wait for a full container setup. So an alternative container start is called.
- Also improves assertions a bit more instead of just counting lines.
- Some test cases don't bind mount all of `/tmp/docker-mailserver` contents, thus don't raise permission errors on subsequent test runs.
- Instead of `rm -f` on some config files, have opted to mount them read-only instead, or alternatively mount an anonymous empty volume instead.
- Collapsed the first three test cases into one, thus no `setup_file()` necessary.
- Shift the `_wait_for_finished_setup_in_container()` method into `_common_container_setup()` instead since nothing else is using `_common_container_start()` yet, this allows for avoiding the wait.

* tests(refactor): Collapse dkim key size test cases into single test case

This makes these tests a bit more DRY, and enhances the raised quality issue with these tests. Now not only is the domain checked in the generated DNS dkim record, but we also verify the key size is corrected in the public and private keys via openssl.

* chore: Revise container names

* chore: Swap order of test case 1 and 2

* tests(refactor): Assert generated log output

- `__should_have_tables_trustedhosts_for_domain` shifted in each test case to just after generating the domains keys.
- Asserts `open-dkim` logs instead of just counting them.
- Added checks for domains that should not be present in a test case.
- Additional coverage and notes about the alias from vhost `@localdomain.com`
- Single assert statement with switch statement as all are using common args.

* chore: Minor changes

* tests(refactor):  Share `find` logic in helpers and tests

* tests(fix): Listing file content does not need to match line order

The order printed from local system vs CI differed causing the CI to fail. The order of lines is irrelevant so `--index` is not required.

Additionally correct the prefix of the called method to be only one `_` now that it's a `common.bash` helper method.

* chore: Collapse custom DKIM selector test into custom DKIM domain test

These cover the same test logic for the most part, the first domain could also be testing the custom selector.

`special_use_folders.bats` + `mailbox_format_dbox` can assert lines instead, removing the need for `--partial`.

* Apply suggestions from code review

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>

* chore: Split switch statement method into wrapper methods

---------

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2023-02-10 00:18:06 +13:00
Georg Lauterbach 00b1d88ed7
removal: configomat (submodule) (#3045) 2023-02-05 12:39:05 +01:00
Georg Lauterbach 774a340d76
chore(Amavis): only add configuration to Postfix when enabled (#3046)
* only add Amavis configuration to Postfix when enabled

Since I am running Rspamd nowadays, I noticed there still are ports open
that belong to Amavis. This is because the Amavis configuration is a
fixed part of Postfix's `master.cf`. I changed that. Now, the Amavis
section is added when Amavis really is enabled.

I took the chance and added proper indentation to `master.cf`; hence the
diff is a bit fuzzy. **But**, only the Amavis part was adjusted, the
rest is just styling.
2023-02-03 17:32:07 +01:00
Georg Lauterbach 1a0c2a351a
rspamd: follow-up of #3016 (#3039) 2023-01-30 08:23:58 +01:00
Brennan Kinney 24d0c358a1
chore: Remove the wrapper script for Postfix (#3033) 2023-01-29 15:53:18 +01:00
Brennan Kinney 74da0197e5
chore: Remove package `gamin` (#3030)
This appears to have been added to replace the `fam` package in an early version of DMS with Courier for IMAP instead of Dovecot on an Ubuntu 14.04 base image.

It does not appear to serve a purpose anymore.
2023-01-29 12:38:27 +00:00
Brennan Kinney 3d8cfc5b5a
chore: Remove wrapper script for fail2ban service (#3032)
* chore: Remove the wrapper script for `fail2ban`

- This does not appear necessary. The server can be run with foreground mode.
- `daemons-stack.sh` removal of the socket can be handled by the fail2ban server when using the `-x` option.

* chore: Remove `touch /var/log/auth.log`

These were both added as supposed fixes in 2016 for the then Ubuntu 2014 base image.

Removing them causes no failures in tests.

* fix: Install optional python packages for `fail2ban`

These have barely any overhead in layer weight. The DNS package may provide some QoL improvements, while the `pyinotify` is a better alternative than polling logs to check for updates.

We have `gamin` package installed but `fail2ban` would complain in the log that it was not able to initialize the module for it. There only appears to be a `python-gamin` dependent on EOL python 2, no longer available from Debian Bullseye.
2023-01-29 12:35:57 +00:00
Georg Lauterbach 555fbb78c4
feature: provide better rspamd suppport (#3016)
* added options to toggle OpenDKIM & OpenDMARC

rspamd can provide DKIM signing and DMARC checking itself, so users
should be able to disable OpenDKIM & OpenDMARC. The default is left at
1, so users have to to opt-in when the want to disable the features.

* misc small enhancements

* adjusted start of rspamd

The order of starting redis + rspamd was reversed (now correct) and
rspamd now starts with the correct user.

* adjusted rspamd core configuration

The main configuration was revised. This includes AV configuration as
well as worker/proxy/controller configuration used to control the main
rspamd processes.

The configuration is not tested extensively, but well enough that I am
confident to go forward with it until we declare rspamd support as
stable.

* update & improve the documentation

* add tests

These are some initial tests which test the most basic functionality.

* tests(refactor): Improve consistency and documentation for test helpers (#3012)

* added `ALWAYS_RUN` target `Makefile` recipies (#3013)

This ensures the recipies are always run.

Co-authored-by: georglauterbach <44545919+georglauterbach@users.noreply.github.com>

* adjusted rspamd test to refactored test helper functions

* improve documentation

* apply suggestions from code review (no. 1 by @polarthene)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>

* streamline heredoc (EOM -> EOF)

* adjust rspamd test (remove unnecessary run arguments)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-01-25 10:28:59 +01:00
Brennan Kinney cb8e336d25
fix: Ensure state persisted to `/var/mail-state` retains correct group (#3011)
* fix: RSPAM ENV should only add to array if ENV enabled

* fix: Correctly match ownership for Postfix content

- `/var/lib/postfix` dir and content is `postfix:postfix`, not `postfix:root`.
- `/var/spool/postfix` is `root:root` not `postfix:root` like it's content.
- Add additional comments, including ownership changes by Postfix to `/var/spool/postfix` when process starts / restarts.

* fix: Ensure correct `chown -R` user and groups applied

These were all fine except for clamav not using the correct clamav group. `fetchmail` group is `nogroup` as per the group set by the debian package.

Additionally formatted the `-eq 1 ]]` content to align on the same columns, and added additional comment about the purpose of this `chown -R` usage so that it's clear what bug / breakage it's attempting to prevent / fix.

* refactor: `misc-stack.sh` conditional handling

The last condition doesn't get triggered at all AFAIK.  Nor does it make sense to make a folder path with `mkdir -p` to symlink to when the container does not have anything to copy over?

- If that was for files, the `mkdir -p` approach seems invalid?
- If it was for a directory that could come up later, it should instead be created in advance? None of the current values for `FILES` seem to hit this path.

Removing as it doesn't seem relevant to current support.

Symlinking was done for each case, I've opted to just perform that after the conditional instead.

Additional inline docs added for additional context.

* chore: Move amavis `chown -R` fix into `misc-stack.sh`

This was handled separately for some reason. It belongs with the other services handling this fix in `misc-stack.sh`.

The `-h` option isn't relevant, when paired with `-R` it has no effect.

* fix: Dockerfile should preserve `clamav` ownership with `COPY --link`

The UID and GID were copied over but would not match `clamav` user and group due to numeric ID mismatch between containers. `--chown=clamav` fixes that.

* chore: Workaround `buildx` bug with separate `chown -R`

Avoids increasing the image weight from this change by leveraging `COPY` in the final stage.

* chore: `COPY --link` from a separate stage instead of relying on scratch

The `scratch` approach wasn't great. A single layer invalidation in the previous stage would result in a new 600MB layer to store.

`make build` with this change seems to barely be affected by such if a change came before copying over the linked stage, although with `buildx` and the `docker-container` driver with `--load` it would take much longer to import and seemed to keep adding storage. Possibly because I was testing with a minimal `buildx` command, that wasn't leveraging proper cache options?

* lint: Appease the linting gods

* chore: Align `misc-stack.sh` paths for `chown -R` operations

Review feedback

Co-authored-by: Casper <casperklein@users.noreply.github.com>

* fix: Reduce one extra cache layer copy

No apparent advantage of a `COPY --link` initially in separate stage.

Just `COPY --chown` in the separate stage and `COPY --link` the stage content. 230MB less in build cache used.

* fix: Remove separate ClamAV stage by adding `clamav` user explicitly

Creating the user before the package is installed allows to ensure a fixed numeric ID that we can provide to `--chown` that is compatible with `--link`.

This keeps the build cache minimal for CI, without being anymore complex as a workaround than the separate stage was for the most part.

* chore: Add reference link regarding users to `misc-stack.sh`
2023-01-25 12:53:47 +13:00
Brennan Kinney 8d80c6317f
tests(refactor): Adjust `mail_changedetector` + change detection helpers (#2997)
* tests(refactor): `mail_changedetector.bats` - Leverage DRY methods

`supervisorctl tail` is not the most reliably way to get logs for the latest change detection and has been known to be fragile in the past.

We can instead read the full log for the service directly with `tac` and `sed` to extract all log content since the last change detection.

Common asserts have also been extracted out into separate methods.

* tests(chore): Remove sleep and redundant change event

Container 1 is still blocked at this point from an existing lock and change event.

Make the lock stale immediately and no extra sleep is required when paired with the helper method to wait until the event is processed (which should remove the stale lock).

* tests(refactor): Add more DRY methods

- Simplify the test case so it's easier to grok.
- 2nd test case (blocking) extracts out initial setup into a separate method and merges the later service restart logic which is redundant.
- Additional comments for improved context of what is going on / expected.

* tests(chore): Revise the change detection helper method

- Add explicit counting arg to change detection support.
- Extract revised logic into it's own generic helper method.
- Utilize this for a separate method that monitors for a change event having started, but not waiting for completion.

This allows dropping the 40 sec of remaining `sleep` in `mail_changedetector` test. It was also required due to potentially missing the timing of a change event completing concurrently in a 2nd container that needed to be waited on and then checked.

* tests(chore): Migrate to current test conventions

- Switch to common container setup helpers
- Update container name and change usage to variables instead.
- Adopt the new convention of prefix variable for test cases (revised test case descriptions).

* tests(chore): Remove legacy change detection

This has since been replaced with the new helper watches the `changedetector` service logs directly instead of only detecting a change has occurred via checksum comparison.

No tests use this method anymore, it was originally for `tests.bats`. Thus the tests in `test_helper.bats` are being dropped too. The new helper has test coverage in `changedetector` tests.

* chore: Lock removal should not incur `sleep 5` afterwards

- A new lock should be created by this script after removal. The sleep doesn't help avoid a race condition with lock file creation after removal.
- Reduces test time as a bonus.
- Added some additional comments to test.

* tests(chore): `tls_letsencrypt.bats` leverage improved change detection

- No need to wait on the change detection service anymore during container startup.
- No need to count change events processed either as waiting a fixed duration is no longer relied on.
- This makes the reload count method redundant, dropped.

* tests(chore): Convert `setup-cli.bats` to new test conventions

This test file was already adapted to the original common setup helpers.

- `TEST_NAME` replaced with `CONTAINER_NAME`.
- Prefix var added, test case descriptions drop explicit prefix.
- No other changes.

* tests(chore): Extract out helpers related to change-detection

- New helper file for sharing these helpers to tests.
- Includes the helpful log method from changedetector tests.
- No longer need to maintain duplicate copies of these methods during the test migration. All tests that use them are now importing the separate helper file.
- `tls_letsencrypt.bats` has switched to using the log helper.
- Generic log count helper is removed from `test_helper/common.bash` as any test that needs it in future can adapt to `helper/common.bash`.

* tests(refactor): `tls_letsencrypt.bats` remove `_get_service_logs()`

This helper does not seem useful as moving away from `supervisorctl tail` and no other tests had a need for it.

* tests(chore): Remove common setup methods from `test_helper/common.bash`

No other tests depend on this. Future tests will adopt the revised versions from `helper/setup.bash`.

Additionally updates `helper/setup.bash` comments that are no longer applicable to `TEST_TMP_CONFIG` and `CONTAINER_NAME`.

* chore: Use `|| true` to simplify setting `EXPECTED_COUNT` correctly
2023-01-16 20:39:46 +13:00
Brennan Kinney 133eb9bc2e
tests(refactor): `mail_lmtp_ip.bats` (#3004)
* chore: Drop ENV `ENABLE_POSTFIX_VIRTUAL_TRANSPORT`

* tests(chore): Remove redundant `dovecot-lmtp` config

None of this is needed. Only relevant change is changing the LMTP service listener for Dovecot and that can be delegated to `user-patches.sh`.

* tests(refactor): Use `user-patches.sh` instead of replacing config file

The only relevant changes in `test/config/dovecot-lmtp` regarding LMTP was:
- `/etc/dovecot/dovecot.conf` (`protocols = imap lmtp`) and `/etc/dovecot/protocols.d/` (`protocols = $protocols lmtp`).
- `conf.d/10-master.conf` only changed the LMTP service listener from a unix socket to TCP on port 24 (_this was the only change required for the test to pass_).

None of those configs are required as:
- `protocols = imap pop3 lmtp` [is the upstream default](https://doc.dovecot.org/settings/core/#core_setting-protocols), no need to add `lmtp`.
- The LMTP service listener is now configured for the test with `user-patches.sh`.

* tests(refactor): `mail_lmtp_ip.bats`

- Converted to new testing conventions and common container helpers.
- `ENABLE_POSTFIX_VIRTUAL_TRANSPORT` was not relevant, dropped.
- Revised test cases, logic remains the same.
- Large custom config used was not documented and doesn't appear to serve any purpose. Simplified by replacing with a single modification with `user-patches.sh`.
- Added some additional comments for context of test and improvements that could be made.

* tests(chore): Adjust comments

The comment from `mail_hostname` provides no valid context, it was likely copied over from `tests.bats` in Oct 2020 by accident.

The email sent is just for testing, nothing relevant to LMTP.

---

Added additional comment for test to reference extra information from.

* tests(chore): Update similar log line matching

Extracts out the match pattern and formatting commands into separate vars (reduces horizontal scrolling), and includes extra docs about what the matched line should be expected to look like.
2023-01-15 18:33:31 +13:00
Brennan Kinney a7e6439a39
fix: Workaround `postconf` write settling logic (#2998)
* fix: Workaround `postconf` write settle logic

After updating `main.cf`, to avoid an enforced delay from reading the config by postfix tools, we can ensure the modified time is at least 2 seconds in the past as a workaround. This should be ok with our usage AFAIK.

Shaves off 2+ seconds roughly off each container startup, reduces roughly 2+ minutes off tests.

* chore: Only modify `mtime` if less than 2 seconds ago

- Slight improvement by avoiding unnecessary writes with a conditional check on the util method.
- Can more comfortably call this during `postfix reload` in the change detection cycle now.
- Identified other tests that'd benefit from this, created a helper method to call instead of copy/paste.
- The `setup email restrict` command also did a modification and reload. Added util method here too.

* tests(fix): `mail_smtponly.bats` should wait for Postfix

- `postfix reload` fails if the service is not ready yet.
- `service postfix reload` and `/etc/init.d/postfix reload` presumably wait until it is ready? (as these work regardless)

* chore: Review feedback - Move reload method into utilities
2023-01-13 10:10:58 +13:00
Casper 6ac59ef871
Fix several typos (#2990) 2023-01-10 14:13:50 +01:00
Casper eeb6b72b3e
Add tools (ping & dig) to the image (#2989)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2023-01-09 13:13:36 +01:00
Georg Lauterbach 26ac48f34a
feature: provide initial Rspamd support (#2902) 2023-01-05 08:39:00 +01:00
Gabriel Euzet a00cdcdee9
fix regex in quota activation code (#2958) 2023-01-04 18:37:00 +01:00
Georg Lauterbach 3a8f6b74ad
update: bump Fail2Ban version to v1.0.2 (#2959) 2023-01-04 17:57:08 +01:00
René Plötz 55f6260bad
fix: Ensure relay host properly handles credentials check (#2965)
A recent change broke the conditional check. Reverting.
2022-12-28 08:50:03 +13:00
Brennan Kinney 4dda5f8b1f
chore: Drop support for deprecated TLS versions (#2945)
* chore: Set `TLS_INTERMEDIATE_SUITE` to only use TLS 1.2 ciphersuites

Removes support of the following cipher suites that are only valid for TLS 1.0 + 1.1:

- `ECDHE-ECDSA-AES128-SHA`
- `ECDHE-RSA-AES128-SHA`
- `ECDHE-ECDSA-AES256-SHA`
- `ECDHE-RSA-AES256-SHA`
- `DHE-RSA-AES128-SHA`
- `DHE-RSA-AES256-SHA`

* chore: Update TLS version min and ignore settings

These are now the same as modern settings.

* fix: Remove min TLS support workaround

No longer required now that outdated TLS versions have been dropped.

* tests: Remove support for TLS 1.0 and 1.1 ciphersuites

* tests: Remove support for TLS 1.0 and 1.1 ciphersuites (Port 25)

The removed SHA1 cipher suites are still supported in TLS 1.2, thus they've been excluded for port 25 via the `SHA1` exclusion pattern in `main.cf`.
2022-12-24 02:30:43 +13:00
Brennan Kinney b58165762a
fix(changedetector): Use service `reload` commands instead of `supervisorctl restart <service>` (#2947)
With `reload` a change detection event during local testing can be processed in less than a second according to logs. Previously this was 5+ seconds (_plus additional downtime for Postfix/Dovecot to become available again_).

In the past it was apparently an issue to use `<service> reload` due to a concern with the PID for wrapper scripts that `supervisorctl` managed, thus `supervisorctl <service> restart` had been used. Past discussions with maintainers suggest this is not likely an issue anymore, and `reload` should be fine to switch to now 👍 

---

**NOTE:** It may not be an issue in the CI, but on _**local systems running tests may risk failure in `setup-cli.bats` from a false positive**_ due to 1 second polling window of the test helper method, and a change event being possible to occur entirely between the two checks undetected by the current approach.

If this is a problem, we may need to think of a better way to catch the change. The `letsencrypt` test counts how many change events are expected to have been processed, and this could technically be leveraged by the test helper too.

---

**NOTE:** These two lines (_with regex pattern for postfix_) are output in the terminal when using the services respective `reload` commands:

```
postfix/master.*: reload -- version .*, configuration /etc/postfix
dovecot: master: Warning: SIGHUP received - reloading configuration
```

I wasn't sure how to match them as they did not appear in the `changedetector` log (_**EDIT:** they appear in the main log output, eg `docker logs <container name>`_).

Instead I've just monitored the `changedetector` log messages, which should be ok for logic that previously needed to ensure Dovecot / Postfix was back up after the `restart` was issued.

---

Commit history:

* chore: Change events `reload` Dovecot and Postfix instead of `restart`

Reloading is faster than restarting the processes.

Restarting is a bit heavy handed here and may no longer be necessary for general usage?

* tests: Adapt tests to support service `reload` instead of `restart`

* chore: Additional logging for debugging change event logs

* fix: Wait on change detection, then verify directory created

Change detection is too fast now (0-1 seconds vs 5+).

Directory being waited on here was created near the end of a change event, reducing that time to detect a change by the utility method further.

We can instead check that the directory exists after the change detection event is completed.

* chore: Keep using the maildir polling check

We don't presently use remote storage in tests, but it might be relevant in future when testing NFS.

This at least avoids any confusing failure happening when that scenario is tested.
2022-12-24 01:57:24 +13:00
Brennan Kinney fe21fe78e2
chore: Remove legacy ENV `SASL_PASSWD` (#2946)
As per deprecation notice from v11.3 release notes, and a related prior PR; this ENV is to be removed.

It's no longer considered useful, and none of the tests that configured it were actually using it for relaying anything.
2022-12-23 15:30:40 +13:00
Casper b75fc448ea
fix: `dovecot-fts-xapian` dependency when using dovecot community repository (#2937) 2022-12-21 11:26:03 +01:00
georglauterbach 6108b28c1d Revert "update: update to latest version of Fail2Ban (#2903)"
This reverts commit 295d7f0a05.
2022-12-19 23:02:16 +01:00
Georg Lauterbach 295d7f0a05
update: update to latest version of Fail2Ban (#2903) 2022-12-19 08:40:49 +01:00
Georg Lauterbach 0ef8d5ae2b
fix: adjust DNSBL return code interpretation (#2890) 2022-11-07 09:28:47 +01:00
Casper 897d84f0f9
scripts: Improve error handling, when parameters are missing (#2854) 2022-10-29 11:04:35 +02:00
Casper 474eb503ae
Fix: Make fetchmail data persistant (#2851) 2022-10-23 03:12:27 +02:00
Casper a930aeb18a
Remove unusual space from shebang line (#2834) 2022-10-17 10:40:09 +02:00
Casper ac795a5976
fix: run Amavis cron job only when Amavis is enabled (#2831) 2022-10-16 11:52:53 +02:00
Casper 032ed00a17
fail2ban: enable network bans (#2818) 2022-10-15 12:01:59 +02:00
Casper 6d016ba592
Run user-patches.sh right before starting daemons (#2817) 2022-10-14 23:11:29 +00:00
Georg Lauterbach 4ab23061a1
style: minor adjustments (#2786)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-10-02 13:00:15 +02:00
Casper 157fde23af
Add ability to build with Dovecot community repository (#2797) 2022-09-29 23:26:45 +02:00
Casper a09d46a677
#2755 follow up; bring back fail2ban package installation (#2796) 2022-09-29 09:02:56 +02:00
Georg Lauterbach 32c508aa2a
ci: enhance build process (#2755)
This new script is a clean way of handling the installation of packages.
I think the huge `RUN` command in `Dockerfile` was hard to read and
maintain.

Using a script is a non-issue, as the image is rebuilt whenever the
script is touched.

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-09-21 09:31:28 +02:00
Casper 1f137f3eca
Only calculate checksums, when there are files to monitor. (#2776) 2022-09-19 10:45:40 +02:00
Georg Lauterbach 39774df85d
scripts: improve `helpers/log.sh` (#2754)
This PR prepares for other PRs that use the newly introduced helper
functions. The `_log` function itself was adjusted a bit to be shorter
and more concise.
2022-09-03 20:59:56 +00:00
Casper efed7d9e44
Remove unnecessary postconf switch '-e' and use single quotes where possible (#2746) 2022-08-29 13:26:44 +02:00
Georg Lauterbach ab55343d8e
scripts: rework environment variables setup (#2716)
* outsourcing env variable setup

This commit contains major parts of the work of refactoring the setup
and usage of environment variables. It outsources the setup into its own
script and provides dedicated functions to be executed at a later point in time.

A **new** env variable was added: `USER_PROVISIONG` which provides a
better way of defining which method / protocol to use when it comes to
setting up users. This way, the `ENABLE_LDAP` variable is deprecated,
but all of this is backwards compatible due to a "compatibility layer", a function provided by the new variables script.

This is not a breaking change. It mostly refators internal scripts. The
only change facing the user-side is the deprecation of `ENABLE_LDAP`. We
can prolong the period of deprecation for this variable as long as we
want, because the new function that ensures backwards compatibility
provides a clean interface for the future.

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-08-22 08:31:32 +02:00
Brennan Kinney 26053c22bd
fix: `_create_accounts()` should run after waiting (#2731)
When waiting on an account to be added to `postfix-accounts.cf`, Dovecot account creation during the startup process had already run.

Startup continued without properly creating the mail account for Dovecot. Methods like `setup email list` (with `ENABLE_QUOTAS=1`) would fail. `changedetector` service was required to be triggered to re-create Dovecot users.

- Wrapped the logic for wait + shutdown into a function call.
- Moved `_create_accounts()` to bottom of the setup function.
2022-08-22 10:44:23 +12:00
Brennan Kinney 57aeb6db2a
refactor: CLI commands for database management (#2654)
See the associated PR for more detailed commentary on specific changes.

### Commands refactored:
- User (**All:** add / list / update / del + _dovecot-master variants_)
- Quota (**All:** set / del)
- Virtual Alias (**All:** add / list /del)
- Relay (**All:** add-relayhost / add-sasl / exclude-domain)

### Overall changes involve:
- **Fairly common structure:**
  - `_main` method at the top provides an overview of logical steps:
    - After all methods are declared beneath it (_and imported from the new `helpers/database/db.sh`_), the `_main` is called at the bottom of the file.
    - `delmailuser` additionally processes option support for `-y` prior to calling `_main`.
  - `__usage` is now consistent with each of these commands, along with the `help` command.
  - Most logic delegated to new helper scripts. Some duplicate content remains on the basis that it's low-risk to maintenance and avoids less hassle to jump between files to check a single line, usually this is arg validation.
  - Error handling should be more consistent, along with var names (_no more `USER`/`EMAIL`/`FULL_EMAIL` to refer to the same expected value_).
- **Three new management scripts** (in `helpers/database/manage/`) using a common structure for managing changes to their respective "Database" config file.
  - `postfix-accounts.sh` unified not only add and update commands, but also all the dovecot-master versions, a single password call for all 4 of them, with a 5th consumer of the password prompt from the relay command `addsaslpassword`.
  - These scripts delegate actual writes to `helpers/database/db.sh` which provides a common API to support the changes made.
     - This is more verbose/complex vs the current inline operations each command currently has, as it provides generic support instead of slightly different variations being maintained, along with handling some edge cases that existed and would lead to bugs (notably substring matches).
     - Centralizing changes here seems wiser than scattered about. I've tried to make it easy to grok, hopefully it's not worse than the current situation.
     - List operations were kept in their respective commands, `db.sh` is only really managing writes. I didn't see a nice way for removing the code duplication for list commands as the duplication was fairly minimal, especially for `listalias` and `listdovecotmasteruser` which were quite simple in their differences in the loop body.
     - `listmailuser` and `delmailuser` also retain methods exclusive to respective commands, I wasn't sure if there was any benefit to move those, but they were refactored.
2022-07-29 12:10:23 +12:00
Casper 428477a878
Set configomat output to loglevel debug (#2701) 2022-07-28 20:55:21 +02:00
Georg Lauterbach 684903f8d2
docs: Revise Kubernetes documentation (#2696) 2022-07-28 10:26:53 +12:00
Andreas Perhab 68477e9047
fix: typo in changedetector ready message (#2663) 2022-06-28 11:02:43 +02:00
Thomas VIAL a84b8a130c
fix: Dovecot Quota dummy accounts for aliases should check for existing users with an exact user key lookup (#2640)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-06-15 12:31:58 +02:00
Brennan Kinney 7fe2f21df4
fix: Amavis should reload config for `/etc/postfix/vhost` updates (#2616)
* fix: Reload `amavisd-new` when vhost config is updated

Amavis was not aware of new domains in `/etc/postfix/vhost` as it did not refresh it's sources upon change detection.
2022-06-15 19:11:10 +12:00
Brennan Kinney 851ec8cbcd
refactor: Revise `check-for-changes.sh` (#2615)
- Inline docs for `check-for-changes.sh` have been shuffled around and revised a bit.
- Change processing extracted from the main change detection loop method to their own methods:
   - `_get_changed_files()` - Clarifies what is going on (and how) without having to look it up. To reduce noise in the main logic loop, extracted to a separate method.
   - `_postfix_dovecot_changes()` - The bulk of change processing was moved to this method. I've added conditionals to only run relevant logic.
   - `_ssl_changes()` - Just shifted, no logic changed. `REGEX_NEVER_MATCH` and `ACME_CERT_DIR` vars scope set to `local`.
2022-06-12 11:36:37 +12:00
Brennan Kinney e3cc627e18
refactor: Share a common helper (vhost builder) for sourcing domains (#2620)
* chore: Split vhost helper method and use filepath vars

- Helpers `accounts.sh` and `aliases.sh` can move their vhost code into this helper.
- They share duplicate code with `bin/open-dkim` which will also leverage this vhost helper going forward.

* chore: Sync vhost generation logic into helper

- Chunky commit, but mostly copy/paste of logic into a common method.
- `bin/open-dkim` additionally wrapped relevant logic in a function call and revised inline docs.

* chore: Include LDAP vhost support

- Revises notes for LDAP vhost support.
- This now ensures LDAP users get vhost rebuilt to match the startup script for when change detection support is enabled.
- `bin/open-dkim` will additionally be able to support the default `DOMAINNAME` var (set via `helpers/dns.sh`) for LDAP users instead of requiring them to provide one explicitly.

* chore(`bin/open-dkim`): Ensure `DOMAINNAME` is properly set

- This will ensure LDAP users insert the same `DOMAINNAME` value as used during container startup.
- The container itself should panic at startup (during `helpers/dns.sh`) if this isn't configured correctly already, thus it should not introduce any breaking change to users of this utility?

* chore: Set the 2nd value as blank `_`

Line is split by a delimiter such as white-space (or via IFS: `|`), the blank `_` var is to indicate we're not interested in that value, but still leverage how `read -r` works, instead of splitting the var ourselves first thing.

* chore: Remove shellcheck disable lines

No longer applicable with the switch to `_`
2022-06-10 10:57:10 +12:00
Brennan Kinney c314c9c471
chore(`check-for-changes.sh`): Drop redundant guards (#2623)
* chore: Remove requirement for `postfix-accounts.cf`

This is an old requirement from when the change detector service was first introduced. It's no longer relevant.

* chore: Do not needlessly create `postfix-aliases.cf`

The config was created regardless to workaround early change detection support. No longer necessary to require the file to exist.

* chore: Drop guards requiring `/tmp/docker-mailserver` to exist

Legacy guards when this was the only location change detection location supported.

There does not appear to be any need for changing into this directory at the start of `check-for-changes.sh` as we use absolute filepaths (originally monitored files were checked with relative paths to this config dir).

* chore: Revise inline docs

* chore: Add change detection monitoring for extra configs

These are also handled at run-time in the current change detection support, so it makes sense to allows these config updates to also trigger change events.
2022-06-09 19:48:07 +12:00
Brennan Kinney 0a722276a8
chore: Extract out `/var/mail` ownership workaround (#2628)
Keep it in sync between the two locations via shared helper method.
2022-06-08 10:09:19 +12:00
Brennan Kinney 54904aa02c
chore(housekeeping): Normalize how config files filter out unwanted lines (#2619)
* chore(`aliases.sh`): Filepath to local var `DATABASE_VIRTUAL`

* chore(`accounts.sh`): Filepath to local var `DATABASE_ACCOUNTS`

* chore(`accounts.sh`): Filepath to local var `DATABASE_VIRTUAL`

* chore(`accounts.sh`): Filepath to local var `DATABASE_DOVECOT_MASTERS`

* chore(`bin/open-dkim`): Filepaths to local vars (accounts,virtual,vhost)

* chore(`relay.sh`): Filepath to local var `DATABASE_SASL_PASSWD`

* chore: Rename method

Prior PR feedback suggested a better helper method name.

* chore: Normalize filtering config lines as input for iterating

* chore: Remove `_is_comment` helper method

No longer serving a purpose with more appropriate filter method for pre-processing the entire config file.
2022-06-07 01:07:30 +12:00
Georg Lauterbach 0010786d18
scripts: improve custom user-supplied Postfix configuration (#2598)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-06-06 09:30:10 +02:00
Brennan Kinney 1f3e672018
fix(listmailuser): Don't parse comments when reading config (#2617)
* fix(listmailuser): Don't parse comments

Avoids passing comments to `dovecot_quota_to_hr()` which fails to handle it and would throws errors.

* chore: Move config filter method to `helpers/utils.sh`
2022-06-06 11:02:52 +12:00
Brennan Kinney 40e2d88482
chore: Merge `helpers/sasl.sh` into `helpers/relay.sh` (#2605)
This helper was to support an earlier ENV for SASL auth support. When extracting logic into individual helpers, it was assumed this was separate from relay support, which it appears was not the case.

---

The `SASL_PASSWD` ENV is specified in tests but no longer used. There is no `external-domain.com` relay configured or tested against anywhere in the project.

The ENV was likely used in tests prior to improved relay support that allowed for adding more than a single set of relay credentials.

---

It likewise has no real relevance anywhere else outside of `relay.sh` as it's the only portion of code to operate with it.

It's only relevant for SASL auth as an SMTP client, not the SMTP server (`smtpd`) SASL support that is delegated to Dovecot. Functionality has been completely migrated into `relay.sh` as a result.

Documentation is poor for this ENV, it is unlikely in wide use? Should consider for removal.

---

The ENV has been dependent upon `RELAY_HOST` to actually enable postfix to use `/etc/postfix/sasl_passwd`, thus not likely relevant in existing setups?

---

Migrate `/etc/postfix/sasl_passwd` check from `tests.bats` as it belongs to relay tests.
2022-06-06 10:59:42 +12:00
Casper ffe20b8ef0
Follow up for #2562 (#2618)
* add related LDAP ENVs

* remove useless line

* sort lines
2022-06-05 21:33:21 +02:00
Brennan Kinney 3d6e7a7bb8
service(postfix): Better handling of the `compatibility_level` setting (#2597)
* chore: Fix typo

* chore: Apply explicit chroot default for `sender-cleanup`

The implicit default is set to `y` as a compatibility fallback, but otherwise it is [advised to set to `n` going forward](http://www.postfix.org/COMPATIBILITY_README.html#chroot).

Test was changed to catch any backwards-compatibility logs, not just those for `chroot=y`. `using` added as a prefix to avoid catching log message whenever a setting is changed that the default compatibility level is active.

* chore: Set `compatibility_level` in `main.cf`

We retain the level`2` value previously set via scripts. This avoids log noise that isn't helpful.

Applied review feedback to give maintainers some context with this setting and why we have it presently set to `2`.
2022-06-05 12:10:20 +12:00
Brennan Kinney d04aceccae
fix: Only add state of enabled services for volume mount (#2608)
* fix: Conditionally add service state

These services will no longer copy over state unless they are enabled.

The biggest offender here was ClamAV as it's database that is baked into the docker image is over 200MB and would copy over to every container instance with a volume mounted state directory.

* chore: Add Dovecot to conditional support
2022-06-05 12:02:25 +12:00
Brennan Kinney c862e1451d
chore(housekeeping): Create `helpers/change-detection.sh` (#2610)
* chore: Extract change-detection method to it's own helper

This doesn't really belong in `helpers/ssl.sh`. Moving to it's own helper script.

* chore: Co-locate related change-detection method from container startup

It seems relevant to migrate the related support during startup for the change detection feature into this helper.

I opted to move the call from `start-mailserver.sh` into the `_setup` call at the end for a more explicit/visible location.

* chore: Move `CHKSUM_FILE` into `helpers/change-detection.sh`

It belongs there, not in `helpers/index.sh`.

* chore: Revise inline documentation

* tests(fix): Ensure correct functionality

Presently `test/test_helper.bats` is using it's own  `CHKSUM_FILE` instead of sourcing the var for the filepath.

`test_helper/common.bash` was calling a method to check for changes, but this helper may not correctly detect letsencrypt related changes as these are not ENV rely on, but global vars handled by `helpers/dns.sh`, so that should be run first like it is for `check-for-changes.sh`.

* tests(chore): Use `CHKSUM_FILE` var from helper

* chore: `addmailuser` should use `CHKSUM_FILE` var

* chore: Update `check-for-changes.sh` log message with correct path
2022-06-05 11:59:54 +12:00
Brennan Kinney 598aee166c
refactor: Revised `relay.sh` helper (#2604)
* chore: Make `_populate_relayhost_map` easier to grok

Changes to `sed` handling that made it quicker to grok, and thus easier for maintainers like myself:
- Switched regex to [extended regex](https://www.gnu.org/software/sed/manual/html_node/Extended-regexps.html).
- Extracted `sed` patterns to be self-descriptive local vars.
- Used a function to reduce noise from intent of loop input (each line as `DOMAIN_PART`).

Input for the loop is filtered through `sort -u` to drop duplicates, reducing iterations.
 
`DOMAIN` loop var renamed to less vague `DOMAIN_PART`. Additional comment in the containing method clarifies what the domain part refers to.

---

`|` regexp syntax needed to be escaped due to switch. Not documented in the earlier link. `-r`/`-E` (ERE) aka extended regexp syntax is [detailed here](https://learnbyexample.github.io/learn_gnused/breere-regular-expressions.html#cheatsheet-and-summary).

* chore: Drop unnecessary postfix parameters

`smtp_tls_note_starttls_offer = yes` - Only adds a log entry to let you know when an unencrypted connection was made, but STARTTLS was offered:
https://www.postfix.org/postconf.5.html#smtp_tls_note_starttls_offer

`smtp_tls_CAfile` is unnecessary. This was added before `smtp_tls_CApath = /etc/ssl/certs` was several months later via a separate PR.

* chore: Move `smtp_` parameters to relevant sections

These have been shifted to relevant logic for now.

---

NOTE: `SASL_PASSWD` previously needed to define `RELAY_HOST` to set `smtp_sasl_password_maps` to enable the `/etc/postfix/sasl_passwd` table. This change now additionally blocks early on in `_relayhost_sasl`. Not likely important due to `RELAY_HOST` logic, user should be using the `RELAY_USER` + `RELAY_PASSWORD` ENV or `postfix-sasl-password.cf` instead.

Especially the sender dependent parameters which are only relevant with user provided configs really.

`SASL_PASSWD` is the oldest ENV for relay support before any other relay feature arrived. It is poorly documented and should not be used.

Potential breakage risk considered acceptable.

* chore: Revise inline docs

Further clarifying current processing logic and adding some additional notes for future work.

* chore: Use a common ENV relay-host getter

The mapping should be in sync between the two configs.

I also wanted to raise awareness of current state of support, which will likely need some refactoring.

This also removes the need for the `RELAY_PORT` fallback method.

The log message was adjusted as configuration is potentially for more than one relay host beyond the currently required ENV config to enable support.

---

NOTE: The ENV `DEFAULT_RELAY_HOST` skips modifying the default transport for an authenticated relay (locked behind `RELAY_HOST` to activate). It presently will only relay mail through a relay host on port 25 instead of delivering directly to the destination. A separate use-case.

* chore: Revise config examples

More verbose example configs with expanded documentation.

Additional doc references for SASL support and cautioning maintainers that may reference popular relay service providers docs. May later be migrated to a "maintainers" section in official docs and link to that.

Brief overview description of what `_populate_relayhost_map` is doing.

* chore: Add notes pertaining to future work

`_populate_relayhost_map` will get some refactoring in future and likely introduce some breaking changes for a future major release.

* chore: Better document relay support inline

This helper now includes a description of it's purpose, links to relevant user docs and supported `setup.sh` commands.

Intent is to keep a maintainer of the feature aware of anything relevant to this feature.
2022-06-05 11:27:15 +12:00
Casper 9a739113f4
chore: Migrate `SASLAUTHD_*` vars into `start-mailserver.sh` (#2562) 2022-06-04 19:28:13 +00:00
Georg Lauterbach 05e45c349a
scripts: fix error message caused by `grep` on first starts (#2591)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-05-28 20:59:31 +02:00
Georg Lauterbach b7e5d42e09
scripts: more consistency in `start-mailserver.sh` (#2587)
* consistently name functions (starting with `_`) in `start-mailserver.sh`

Most of the functions that execute the different stacks during startup
were not prefixed with `_`, but all our other functions are. This has
now been fixed.

* cleanup in `start-mailserver.sh`

I adjusted the comments for all sections in the start script so they are
properly displayed again.
2022-05-15 23:37:21 +02:00
Maximilian Hippler 777d4a8cb7
scripts: loglevel warning message fix (#2588)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2022-05-13 10:50:45 +02:00
Casper 372e3fe35e
Use color variables instead of escape codes (#2565) 2022-05-10 17:50:33 +02:00
Andrew Cornford 2f3cbfc144
feat: Support for Dovecot master accounts (#2535)
Dovecot master accounts can now be configured in DMS via `setup.sh`.
A master account is useful for administration purposes, or to perform mailbox backups of every user account over IMAP.
Upstream Docs: https://doc.dovecot.org/configuration_manual/authentication/master_users/

Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-05-08 00:28:32 +02:00
Casper 2977cb6962
Streamline 'printf' usage and consequently use a format string (#2564) 2022-05-05 17:48:21 +02:00
Casper 628e902233
Remove unnecessary quotes from command substitutions (#2561) 2022-05-05 10:28:38 +02:00
Casper 32fba30bbc
Don't issue warning when DKIM is not used (#2563) 2022-04-29 19:03:16 +02:00
Casper cbcc3823d3
Fix changedetector restart loop (#2548)
* only restart changedetector, if exit is unexpected.

* prevent supervisord from restarting changedetector on error --> endless loop

* add quotes
2022-04-19 21:09:25 +02:00
Casper de61d42e68
Add ban feature to fail2ban script (#2538) 2022-04-19 08:44:51 +00:00
Brennan Kinney 9aaf15b38f
chore: (setup-stack.sh) Fix a small typo (#2552) 2022-04-19 12:53:45 +12:00
Brennan Kinney 1b1877f025
refactor: letsencrypt implicit location discovery (#2525)
* chore: Extract letsencrypt logic into methods

This allows other scripts to share the functionality to discover the correct letsencrypt folder from the 3 possible locations (where specific order is important).

As these methods should now return a string value, the `return 1` after a panic is now dropped.

* chore: Update comments

The todo is resolved with this PR, `_setup_ssl` will be called by both cert conditional statements with purpose for each better documented to maintainers at the start of the logic block.

* refactor: Defer most logic to helper/ssl.sh

The loop is no longer required, extraction is delegated to `_setup_ssl` now.

For the change event prevention, we retrieve the relevant FQDN via the new helper method, beyond that it's just indentation diff.

`check-for-changes.sh` adjusted to allow locally scoped var declarations by wrapping a function. Presently no loop control flow is needed so this seems fine. Made it clear that `CHANGED` is local and `CHKSUM_FILE` is not.

Panic scope doesn't require `SSL_TYPE` for context, it's clearly`letsencrypt`.

* fix: Correctly match wildcard results

Now that the service configs are properly updated, when the services restart they will return a cert with the SAN `DNS:*.example.test`,  which is valid for `mail.example.test`, however the test function did not properly account for this in the regexp query.

Resolved by truncating the left-most DNS label from FQDN and adding a third check to match a returned wildcard DNS result.

Extracted out the common logic to create the regexp query and renamed the methods to communicate more clearly that they check the FQDN is supported, not necessarily explicitly listed by the cert.

* tests(letsencrypt): Enable remaining tests

These will now pass. Adjusted comments accordingly.

Added an additional test on a fake FQDN that should still be valid to a wildcard cert (SNI validation in a proper setup would reject the connection afterwards).

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2022-04-18 22:52:50 +12:00
Georg Lauterbach 412f675bfe
setup: added grace period for account creation (#2531) 2022-04-18 10:48:48 +02:00
Georg Lauterbach a1726dc45a
scripts: added `TZ` environment variable to set timezone (#2530) 2022-04-06 16:48:41 +02:00
Georg Lauterbach b1594a8b1c
log/scripts: introduce proper log level fallback and env getter function (#2506)
This PR does two small things:

1. The log level, in case it is unset, will now be "calculated" from
   `/etc/dms-settings` and not always default to `info`. This way, we
   can ensure that more often than not, the log level the user chose
   when starting DMS is used everywhere.
2. I noticed that the way I obtained the log level could be used to
   obtain any env variable's log level. I therefore added a function to
   `utils.sh` in case we use it in the future.
2022-04-05 17:10:01 +02:00
Georg Lauterbach c7b16a599c
log: adjust level and message(s) slightly for four messages (#2532)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-04-05 16:21:07 +02:00
Georg Lauterbach a9305a073f
firewall: replace `iptables` with `nftables` (#2505)
* first adjustments to use Fail2Ban with nftables

* replace `iptables` -> `nftables` and adjust tests

nftables lists IPs a bit differently , so the order was adjusted for the
tests to be more flexible.

* line correction in mailserver.env

* change from `.conf` -> `.local` and remove redundant config

* revert HEREDOC to `echo`

Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-04-05 15:13:59 +02:00
Georg Lauterbach 21c218ac68
scripts: remove `DMS_DEBUG` (#2523)
* remove DMS_DEBUG from tests
* remove DMS_DEBUG from doc and scripts
* updated issue template
* re-add description about removal of DMS_DEBUG
2022-04-03 13:29:10 +02:00
Georg Lauterbach 35fb744ffb
scripts: refactored `check-for-changes.sh` (#2498)
* refactored `check-for-changes.sh`

I refactored `check-for-changes.sh` and used the new log. `_notify` can
therefore be deleted as it is used no more.
I opted to source `/etc/dms-settings` as a whole to
future-proof the script. When the DNS adjustments PRs (that do not exist
by now but will exit in the future) are done, we can then remove
`_obtain_hostname_and_domainname` because we're already writing the
variables to `/etc/dms-settings`. I left instructions in the script in
the form of TODO comments.

Because we now log the date for all messages of the changedetector, we
need to `tail` a bit more log than before.
2022-04-02 19:39:15 +02:00
Ikko Ashimine 05f680b472
fix: typo in setup-stack.sh (#2521) 2022-04-02 10:41:46 +02:00
Georg Lauterbach a54d774587
scripts: refactored `daemon-stack.sh` (#2496)
* refactored `daemon-stack.sh`

A new method was introduced to uniformaly start daemons and log output
accordingly. The methods for daemon start were renamed (plural ->
singular), therefore the adjustments in `start-mailserver.sh`.

* cleaned Fetchmail setup from `daemon-stack.sh`

Not sure why, but the Fetchmail setup was somehow happening in
`daemon-stack.sh` - this is not supposed to be the case. I relocated the
setup into `setup-stack.sh`, where it belong.

* delete old, unnecessary script in `target/bin/`

These are unused leftovers from the last commit, that relocated the
setup of Fetchmail into `setup.stack.sh`.

* corrected changedetector function name

* Apply suggestions from code review

* adjusted `debug-fetchmail` script

It is absolutely fine to source `setup-stack.sh` because sourcing the
script does not execute a single function (by desing of the script).
This way, we retain functionality.

* praise be ShellCheck

* added `log.sh` to `debug-fetchmail` as a dependency

* final cleanup

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-03-27 09:43:39 +02:00
Georg Lauterbach 7721a48b9b
scripts: refactoring & miscellaneous small changes (#2499)
* `update-check.sh` now uses the new log

* refactored `setup-stack.sh`

The changes are:

1. Replaced `""` wiht `''` where possible (reasoning: Bash is very
   implicit and I'd like to use `''` where possible to indicate no
   variables are expanded here)
2. `> /file` -> `>/file` according to our style guide
3. Some log adjustments for messages where I deemed it appropriate
4. Then, an error message from a Dovecot setup was also prevented (by
   adding a check whether the directory is present before a `: >...`
   command would create a file in this directory).

These are all small, miscellaneous changes that I wanted to combine into
one commit and ultimately one PR because I see no point in opening a PR
for every small change here. I hope this is fine.

* added a small `sleep` to the `_shutdown` function

This ensure the last log message is actually logged before Supervisor
logs the message that it received a SIGTERM. This makes reading the log
easier because now the causal relationship is shown (we are terminating
Supervisor, and not someone else and we're just logging it).

I forgot to replace `""` with `''` in `update-check.sh`, so I included
it here because this is the last commit before PR review.

* re-add exit on successful update (only)

* re-added date information to update-check log messages

* added `_log_with_date` function

The new function will log a message with a proper timestamp. This is all
handled in `log.sh`, we therefore not need to source other files too.

This will be used in the future by `check-for-changes.sh` as well :)

Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-03-26 10:17:08 +01:00
Georg Lauterbach b9dbec3276
scripts: refactored scripts located under `target/bin/` (#2500)
* refactored scripts located under `target/bin/`

The scripts under `target/bin/` now use the new log and I replaced some
`""` with `''` on the way. The functionality stays the same, this mostly
style and log.

* corrected fail2ban (script and tests)

* corrected OpenDKIM log output in tests

* reverted (some) changes to `sedfile`

Moreover, a few messages for BATS were streamlined and a regression in
the linting script reverted.

* apple PR feedback

* improve log output from `fail2ban` script

The new output has a single, clear message with the '[  ERROR  ]  '
prefix, and then output that explains the error afterwards. This is
coherent with the logging style which should be used while providing
more information than just a single line about IPTables not functioning.

* simplified `setquota` script

* consistently named the `__usage` function

Before, scripts located under `target/bin/` were using `usage` or
`__usage`. Now, they're using `__usage` as they should.

* improved `sedfile`

With `sedfile`, we cannot use the helper functions in a nice way because
it is used early in the Dockerfile at a stage where the helper scripts
are not yet copied. The script has been adjusted to be canonical with
all the other scripts under `target/bin/`.

* fixed tests

* removed `__usage` from places where it does not belong

`__usage` is to be used on wrong user input, not on other failures as
well. This was fixed in `delquota` and `setquota`.

* apply PR review feedback
2022-03-26 09:30:09 +01:00
Georg Lauterbach b7bf40ac21
renamed function `_errex` -> `_exit_with_error` (#2497) 2022-03-21 15:01:07 +01:00
Casper 37c8e44566
Restart supervisord early (#2494) 2022-03-21 13:42:12 +01:00
Georg Lauterbach 24031ae365
scripts: new log (#2493)
* added new `_log` function

With `_log`, the `_notify` method wa rendered obsolete. `_notify` was
not completely removed due to test failures in `check-for-changes.sh`.

The new `_log` function properly uses log levels such as `trace`,
`debug`, `info`, `warn` and `error`. It provides a cleaner solution
and renders `DMS_DEBUG` obsolete too (as only `_notify` depends on it).

* converted all helper script to new `_log` function

* converted all startup stacks to new `log` function

* `start-mailserver.sh` now uses new `_log` function

* final test and misc small script adjustments

* updated documentation
2022-03-21 07:07:52 +01:00
Casper 1bfc6d944a
Use FQDN as REPORT_SENDER default value. (#2487) 2022-03-18 19:41:02 +01:00
Georg Lauterbach 321ae744fa
scripts: small refactorings (#2485)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-03-17 11:24:30 +01:00
NorseGaud 1ab8ea96b2
acme_extract -> acme_extract.py + F_BIN to avoid .py 2022-03-06 22:17:29 -05:00
Georg Lauterbach d101d0aa0a
fix: remove SA reload for KAM (#2456)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-03-04 23:38:29 +01:00
Casper b9fdb1e7b5
Introduce CLAMAV_MESSAGE_SIZE_LIMIT env (#2453) 2022-03-03 16:17:01 +01:00
Casper a8a8c859dc
Rename config examples directory (#2438) 2022-03-02 22:54:14 +00:00
Georg Lauterbach e6af5a118f
Adjust envrionment variables - more sensible defaults (#2428)
The new setup will now set env variables on one place and on one place
only. The old setup used two separate places wich is not DRY and
confusing.

Some default values changed:

1. PFLOGSUMM_TRIGGER: logrotate => none
2. REPORT_SENDER: mailserver-report@HOSTNAME => mailserver-report@DOMAIN
3. REPORT_RECIPIENT: "0" => POSTMASTER_ADDRESS

One env variable was renamed: REPORT_INTERVAL => LOGROTATE_INTERVAL

I believe these defaults to be more sensible, especially the REPORT_RECIPIENT
address. The PFLOGSUMM_TRIGGER value was changed to `none` because otherwise
people would start getting daily Postfix log summary reports automatically.
Now, this is opt-in, and reports are sent only when enabled properly.

Some of the variables changed were marked as deprecated. I removed the note,
as the variables now bear some (sane) defaults again for other variables
(i.e.) REPORT_RECIPIENT is now default for other recipient addresses.

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-03-03 10:22:17 +13:00
Casper 57c52d7b5b
PERMIT_DOCKER=none as new default value (#2424)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-03-02 16:03:40 +01:00
Georg Lauterbach 399284c092
fix: do not add accounts that already exists to account files (#2419)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-02-21 13:31:03 +01:00
Georg Lauterbach b61dfe1e24
refactoring: split helper functions into smaller scripts (#2420) 2022-02-21 11:56:57 +01:00
Georg Lauterbach 2927cc47c7
Add SpamAssassin KAM (#2418) 2022-02-21 10:48:28 +01:00
Georg Lauterbach ec8b99335e
Add changedetector functionality for `${SSL_TYPE} == manual` (#2404)
Now, setups that use `SSL_TYPE=manual` will profit from the changedetector as well. Certificate changes are picked up and properly propagated.
2022-02-18 11:29:51 +01:00
Christian Raue 908e95fa74
consistently make 1 the default value for `SPAMASSASSIN_SPAM_TO_INBOX` (#2361)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-02-17 09:31:45 +01:00
Georg Lauterbach ce9e86c462
fix: double occurence of `/etc/postfix/regexp` (#2397) 2022-02-12 09:20:45 +13:00
Philipp Fruck 4c3af32692
Rootless Podman security update (#2393)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-02-09 10:25:09 +01:00
Georg Lauterbach ede2b2394a
improvement: get rid of subshell + `exec` in `helper-functions.sh` (#2401)
* get rid of subshell + exec

The new way of executing `sha512sum` should work as well as the old way
but without the clutter and possible problems the usage of subshells +
exec incurs.

Moreover, there was a misconception about array expansion. Using `""`
around an expanding array (`${ARRAY[@]}`) is quite fine (and actually
the preffered way), not because it makes the expansion _one_ string
(this would be `${ARRAY[*]}`), but it makes sure when elements are
expanded, each element has `""` around them so to speak, i.e. there is
no re-splitting of these elements.

* removed old concerns in comments

* increase test and check for changes sleep duration
2022-02-09 11:21:45 +13:00
Christian Raue da17e8bf0b
add env var `LOGWATCH_SENDER` (#2362)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2022-01-10 12:29:11 +01:00
Casper 29c2d975ec
Fix harmless startup errors (#2357) 2022-01-08 00:25:09 +01:00
Casper 0c31f71358
Introduce DOVECOT_INET_PROTOCOLS env (#2358) 2022-01-06 00:53:18 +01:00
Casper 9d5a9a16a0
Introduce ENABLE_DNSBL env (#2342)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2022-01-03 22:03:46 +01:00
Casper f7465a5088
Prevent race condition on supervisorctl reload (#2343)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2021-12-29 17:36:29 +01:00
Casper d46e094280
Remove quotes to meet style guidelines (#2330)
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2021-12-21 17:01:40 +01:00
Georg Lauterbach 99cc9fec2a
Updated ShellCheck to `0.8.0` and Hadolint to `2.8.0` (#2329)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2021-12-19 11:56:22 +01:00
Brennan Kinney 6d06149581
fix: Restore detection of letsencrypt certificate file changes (#2326)
The `DYNAMIC_FILES` var was quote wrapped, treating all filepaths to create checksums for as a single string that would be ignored instead of processed individually.

Removed the quotes, and changed the for loop to an array which accomplishes the same goal.


* fix: Prevent unnecessary change detection event

`acme.json` change would extract new cert files, which would then be hashed after restarting services and considered a change event, running through the logic again and restarting services once more when that was not required.

The checksum entries for those cert files are now replaced with new entries containing updated checksum hashes, after `acme.json` extraction.
2021-12-19 11:25:15 +13:00
Georg Lauterbach a4095a7d48
Disable `service stats` for Dovecot explicitly (#2292)
Co-authored-by: Casper <casperklein@users.noreply.github.com>

Co-authored-by: Casper <casperklein@users.noreply.github.com>
2021-12-12 12:08:34 +01:00
Casper a8a39a0a1a
Rename function and adjust comment (#2306) 2021-11-28 17:02:07 +01:00
Brennan Kinney 5254f7c658
fix: `check-for-changes.sh` should not fall out of sync with shared logic (#2260)
Removes duplicate logic from `check-for-changes.sh` that is used/maintained elsewhere to avoid risk of problems, as this code is already starting to diverge / rot.

---

Previously the change detection support has had code added for rebuilding config upon change detection which is the same as code run during startup scripts. Unfortunately over time this has fallen out of sync. Mostly the startup scripts would get maintenance and the contributor and reviewers may not have been aware of the duplicate code handled by `check-for-changes.sh`.

That code was starting to diverge in addition to some changes in structure (_eg: relay host logic seems interleaved here vs separated out in startup scripts_). I wanted to address this before it risks becoming a much bigger headache.

Rather than bloat `helper-functions.sh` further, I've added a `helpers/` folder extracting relevant common logic between startup scripts and `changedetector`. If you want to follow that process I've kept scoped commits to make those diffs easier. Some minor changes/improvements were added but nothing significant.

---

- chore: Extract relay host logic to new `relay.sh` helper
- chore: Extract `/etc/postfix/sasl_passwd` logic to new `sasl.sh` helper
- chore: Extract `postfix-accounts.cf` logic to new `accounts.sh` helper
- chore: Extract `/etc/aliases` logic to new `aliases.sh` helper
- chore: Extract `/etc/postfix/vhost` logic to new `postfix.sh` helper

- chore: Add inline docs for Postfix configs
> These are possibly more verbose than needed and can be reduced at a later stage.
> They are helpful during this refactor process while investigating that everything is handled correctly.

`accounts.sh`: 
- Add note regarding potential bug for bare domain setups with `/etc/postfix/vhost` and `mydestination` sharing same domain value.

`relay.sh`: 
- Remove the tabs for a single space delimiter, revised associated comment.
- Add PR reference for original `_populate_relayhost_map` implementation which has some useful details.


Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2021-11-21 09:33:49 +13:00
Georg Lauterbach ae70142d8f
Housekeeping (#2302)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2021-11-20 10:38:40 +01:00
Brennan Kinney 7ca056852f
tests(feat): Complete rewrite of letsencrypt tests (#2286)
* chore: Normalize container setup

Easier to grok what is different between configurations.

- Container name usage replaced with variable
- Volumes defined earlier and redeclared when relevant (only real difference is `VOLUME_LETSENCRYPT`)
- Contextual comment about the `acme.json` copy.
- Quoting `SSL_TYPE`, `SSL_DOMAIN` and `-h` values for syntax highlighting.
- Moved `-t` and `${NAME}` to separate line.
- Consistent indentation.

* chore: DRY test logic

Extracts out repeated test logic into methods

* chore: Scope configs to individual test cases (1/3)

- Preparation step for shifting out the container configs to their own scoped test cases. Split into multiple commits to ease reviewing by diffs for this change.
- Re-arrange the hostname and domain configs to match the expected order of the new test cases.
- Shuffle the hostname and domainname grouped tests into tests per container config scope.
- Collapse the `acme.json` test cases into single test case.

* chore: Scope configs to individual test cases (2/3)

- Shifts the hostname and domainname container configs into their respective scoped test cases.
- Moving the `acme.json` container config produces a less favorable diff, so is deferred to a follow-up commit.
- Test cases updated to refer to their `${CONTAINER_NAME}` var instead of the hard-coded string name.

* chore: Scope configs to individual test cases (3/3)

Final commit to shift out the container configs.

- Common vars are exported in `setup_file()` for the test cases to use without needing to repeat the declaration in each test case.
- `teardown_file()` shifts container removal at end of scoped test case.

* chore: Adapt to `common_container_setup` template

- `CONTAINER_NAME` becomes `TEST_NAME` (`common.bash` helper via `init_with_defaults`).
- `docker run ...` and related configuration is now outsourced to the `common.bash` helper, only extra args that the default template does not cover are defined in the test case.
- `TARGET_DOMAIN`establishes the domain folder name for `/etc/letsencrypt/live`.
- `_should*` methods no longer manage a `CONTAINER_NAME` arg, instead using the `TEST_NAME` global that should be valid as test is run as a sequence of test cases.
- `PRIVATE_CONFIG` and the `private_config_path ...` are now using the global `TEST_TMP_CONFIG` initialized at the start of each test case, slightly different as not locally defined/scoped like `PRIVATE_CONFIG` would be within the test case, hence the explicit choice of a different name for context.

* chore: Minor tweaks

- Test case comment descriptions.
- DRY: `docker rm -f` lines moved to `teardown()`
- Use `wait_for_service` helper instead of checking the `changedetector` script itself is running.
- There is a startup delay before the `changedetector` begins monitoring, wait until it ready event is logged.
- Added a helper to query logs for a service (useful later).
- `/bin/sh` commands reduced to `sh`.
- Change the config check to match and compare output, not number of lines returned. Provides better failure output by bats to debug against.

* chore: Add more test functions for `acme.json`

This just extracts out existing logic from the test case to functions to make the test case itself more readable/terse.

* chore: Housekeeping

No changes, just moving logic around and grouping into inline functions, with some added comments.

* chore: Switch to `example.test` certs

This also required copying the source files to match the expected letsencrypt file structure expected in the test/container usage.

* chore: Delete `test/config/letsencrypt/`

No longer necessary, using the `example.test/` certs instead.

These letsencrypt certs weren't for the domains they were used for, and of course long expired.

* chore: Housekeeping

Add more maintainer comments, rename some functions.

* tests: Expand `acme.json` extraction coverage

Finally able to add more test coverage! :)

- Two new methods to validate expected success/failure of extraction for a given FQDN.
- Added an RSA test prior to the wildcard to test a renewal simulation (just with different cert type).
- Added extra method to make sure we're detecting multiple successful change events, not just a previous logged success (false positive).

* tests: Refactor the negotiate_tls functionality

Covers all ports (except POP) and correctly tests against expected verification status with new `example.test` certs.

The `FQDN` var will be put to use in a follow-up commit.

* tests: Verify the certs contain the expected FQDNs

* chore: Extract TLS test methods into a separate helper script

Can be useful for other TLS tests to utilize.

* chore: Housekeeping

* chore: Fix test typo

There was a mismatch between the output and expected output between these two files "find key for" and "find key & cert for". Changed to "find key and/or cert for" to make the warning more clear that it's issued for either or both failure conditions.

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2021-11-16 16:00:16 +01:00
Brennan Kinney 584577787a
refactor: Internal HOSTNAME and DOMAINNAME configuration (#2280)
Better logical flow, handling and inline documentation.

Despite the verbosity, it's better to make this visible here for maintenance and debugging purposes than trying to dig through issue/PR or commit history for it.

* fix: Panic when HOSTNAME is misconfigured
* chore: Add more comment docs for maintainers
* tests(fix): Use `--domainname` not ENV `DOMAINNAME`

Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2021-11-16 10:50:45 +13:00
Casper cac0647250
improve: sort environment variables (#2298) 2021-11-14 09:11:44 +13:00
Georg Lauterbach 6c1c669619
make Bash start user-patches.sh (#2295) 2021-11-13 10:19:56 +01:00