docs: Remove ENV `ONE_DIR` (#3840)

* docs: Better document DMS volumes

* docs: Remove any mention of `ONE_DIR` ENV

* chore: Remove `ONE_DIR` ENV from scripts

Only `ONE_DIR=0` has any effect. As the actual feature is now dependent upon the `/var/mail-state` location existing.

It is advised not mounting anything there instead if wanting to avoid runtime state consolidation.

* docs: Adjust link ref convention

This is more search friendly / organized to find references to all DMS volumes.

* lint: Ensure final newline is present

VSCode by default excludes this if the last line rendered is removed (rendered as a separate blank line).

A separate setting can enforce adding the final newline upon save regardless.
This commit is contained in:
Brennan Kinney 2024-01-29 10:35:19 +13:00 committed by GitHub
parent 11c508cd11
commit 3b11a8305e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
15 changed files with 102 additions and 67 deletions

View File

@ -40,6 +40,9 @@ The most noteworthy change of this release is the update of the container's base
### Updates
- **Environment Variables:**
- `ONE_DIR` has been removed (legacy ENV) ([#3840](https://github.com/docker-mailserver/docker-mailserver/pull/3840))
- It's only functionality remaining was to opt-out of run-time state consolidation with `ONE_DIR=0` (_when a volume was already mounted to `/var/mail-state`_).
- **Tests:**
- Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786))

View File

@ -40,7 +40,6 @@ data:
POSTMASTER_ADDRESS: postmaster@example.com
UPDATE_CHECK_INTERVAL: 10d
POSTFIX_INET_PROTOCOLS: ipv4
ONE_DIR: '1'
ENABLE_CLAMAV: '1'
ENABLE_POSTGREY: '0'
ENABLE_FAIL2BAN: '1'

View File

@ -4,9 +4,64 @@ hide:
- toc # Hide Table of Contents for this page
---
This is a list of all configuration files and directories which are optional or automatically generated in your [`docker-data/dms/config/`][docs-dms-config-volume] directory.
## Volumes
## Directories
DMS has several locations in the container which may be worth persisting externally via [Docker Volumes][docker-docs::volumes].
- Often you will want to prefer [bind mount volumes][docker-docs::volumes::bind-mount] for easy access to files at a local location on your filesystem.
- As a convention for our docs and example configs, the local location has the common prefix `docker-data/dms/` for grouping these related volumes.
!!! info "Reference - Volmes for DMS"
Our docs may refer to these DMS specific volumes only by name, or the host/container path for brevity.
- [Config](#volumes-config): `docker-data/dms/config/` => `/tmp/docker-mailserver/`
- [Mail Storage](#volumes-mail): `docker-data/dms/mail-data/` => `/var/mail/`
- [State](#volumes-state): `docker-data/dms/mail-state/` => `/var/mail-state/`
- [Logs](#volumes-logs): `docker-data/dms/mail-logs/` => `/var/log/mail/`
[docker-docs::volumes]: https://docs.docker.com/storage/volumes/
[docker-docs::volumes::bind-mount]: https://docs.docker.com/storage/bind-mounts/
### Mail Storage Volume { #volumes-mail }
This is the location where mail is delivered to your mailboxes.
### State Volume { #volumes-state }
Run-time specific state lives here, but so does some data you may want to keep if a failure event occurs (_crash, power loss_).
!!! example "Examples of relevant data"
- The Postfix queue (eg: mail pending delivery attempt)
- Fail2Ban blocks.
- ClamAV signature updates.
- Redis storage for Rspamd.
!!! info "When a volume is mounted to `/var/mail-state/`"
- Service run-time data is [consolidated into the `/var/mail-state/` directory][mail-state-folders]. Otherwise the original locations vary and would need to be mounted individually.
- The original locations are updated with symlinks to redirect to their new path in `/var/mail-state/` (_eg: `/var/lib/redis` => `/var/mail-state/lib-redis/`_).
Supported services: Postfix, Dovecot, Fail2Ban, Amavis, PostGrey, ClamAV, SpamAssassin, Rspamd & Redis, Fetchmail, Getmail, LogRotate, PostSRSd, MTA-STS.
!!! tip
Sometimes it is helpful to disable this volume when troubleshooting to verify if the data stored here is in a bad state (_eg: caused by a failure event_).
[mail-state-folders]: https://github.com/docker-mailserver/docker-mailserver/blob/v13.3.1/target/scripts/startup/setup.d/mail_state.sh#L13-L33
### Logs Volume { #volumes-log }
This can be a useful volume to persist for troubleshooting needs for the full set of log files.
### Config Volume { #volumes-config }
Most configuration files for Postfix, Dovecot, etc. are persisted here.
This is a list of all configuration files and directories which are optional, automatically generated / updated by our `setup` CLI, or other internal scripts.
#### Directories
- **sieve-filter:** directory for sieve filter scripts. (Docs: [Sieve][docs-sieve])
- **sieve-pipe:** directory for sieve pipe scripts. (Docs: [Sieve][docs-sieve])
@ -14,7 +69,7 @@ This is a list of all configuration files and directories which are optional or
- **ssl:** SSL Certificate directory if `SSL_TYPE` is set to `self-signed` or `custom`. (Docs: [SSL][docs-ssl])
- **rspamd:** Override directory for custom settings when using Rspamd (Docs: [Rspamd][docs-rspamd-override-d])
## Files
#### Files
- **{user_email_address}.dovecot.sieve:** User specific Sieve filter file. (Docs: [Sieve][docs-sieve])
- **before.dovecot.sieve:** Global Sieve filter file, applied prior to the `${login}.dovecot.sieve` filter. (Docs: [Sieve][docs-sieve])
@ -42,7 +97,6 @@ This is a list of all configuration files and directories which are optional or
- **user-patches.sh:** this file will be run after all configuration files are set up, but before the postfix, amavis and other daemons are started. (Docs: [FAQ - How to adjust settings with the `user-patches.sh` script][docs-faq-userpatches])
- **rspamd/custom-commands.conf:** list of simple commands to adjust Rspamd modules in an easy way (Docs: [Rspamd][docs-rspamd-commands])
[docs-dms-config-volume]: ../../faq.md#what-about-the-docker-datadmsconfig-directory
[docs-accounts-quota]: ../../config/user-management.md#quotas
[docs-aliases-regex]: ../../config/user-management.md#configuring-regexp-aliases
[docs-dkim]: ../../config/best-practices/dkim_dmarc_spf.md#dkim

View File

@ -345,7 +345,7 @@ volumes:
```
[docs-accounts-add]: ../user-management.md#adding-a-new-account
[docs-volumes-config]: ../advanced/optional-config.md
[docs-volumes-config]: ../advanced/optional-config.md#volumes-config
[docs-env-opendkim]: ../environment.md#enable_opendkim
[docs-env-rspamd]: ../environment.md#enable_rspamd
[docs-rspamd-config-dropin]: ../security/rspamd.md#manually

View File

@ -45,11 +45,6 @@ Default: 5000
The Group ID assigned to the static vmail group for `/var/mail` (_Mail storage managed by Dovecot_).
##### ONE_DIR
- 0 => state in default directories.
- **1** => consolidate all states into a single directory (`/var/mail-state`) to allow persistence using docker volumes. See the [related FAQ entry][docs-faq-onedir] for more information.
##### ACCOUNT_PROVISIONER
Configures the provisioning source of user accounts (including aliases) for user queries and authentication by services managed by DMS (_Postfix and Dovecot_).
@ -648,10 +643,10 @@ Controls the spam score threshold for triggering an action on mail that has a hi
- [It will be quarantined][amavis-docs::quarantine] regardless of the `SA_KILL` action to perform.
- With `D_PASS` the delivered mail also appends an `X-Quarantine-ID` mail header. The ID value of this header is part of the quarantined file name.
If emails are quarantined, they are compressed and stored at a location dependent on the [`ONE_DIR`](#one_dir) setting:
If emails are quarantined, they are compressed and stored at a location:
- `ONE_DIR=1` (default): `/var/mail-state/lib-amavis/virusmails/`
- `ONE_DIR=0`: `/var/lib/amavis/virusmails/`
- Default: `/var/lib/amavis/virusmails/`
- When the [`/var/mail-state/` volume][docs::dms-volumes-state] is present: `/var/mail-state/lib-amavis/virusmails/`
!!! tip
@ -1056,9 +1051,9 @@ you to replace both instead of just the envelope sender.
- password for default relay user
[docs-rspamd]: ./security/rspamd.md
[docs-faq-onedir]: ../faq.md#what-about-docker-datadmsmail-state-folder-varmail-state-internally
[docs-tls]: ./security/ssl.md
[docs-tls-letsencrypt]: ./security/ssl.md#lets-encrypt-recommended
[docs-tls-manual]: ./security/ssl.md#bring-your-own-certificates
[docs-tls-selfsigned]: ./security/ssl.md#self-signed-certificates
[docs-accounts-quota]: ./user-management.md#quotas
[docs::dms-volumes-state]: ./advanced/optional-config.md#volumes-state

View File

@ -33,7 +33,7 @@ DMS will automatically ban IP addresses of hosts that have generated 6 failed at
### Custom Files
!!! question "What is [`docker-data/dms/config/`][docs-dms-config-volume]?"
!!! question "What is [`docker-data/dms/config/`][docs::dms-volumes-config]?"
This following configuration files inside the `docker-data/dms/config/` volume will be copied inside the container during startup
@ -44,7 +44,7 @@ This following configuration files inside the `docker-data/dms/config/` volume w
- with this file, you can adjust F2B behavior in general
- there is an example provided [in our repository on GitHub][github-file-f2bconfig]
[docs-dms-config-volume]: ../../faq.md#what-about-the-docker-datadmsconfig-directory
[docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config
[github-file-f2bjail]: https://github.com/docker-mailserver/docker-mailserver/blob/master/config-examples/fail2ban-jail.cf
[github-file-f2bconfig]: https://github.com/docker-mailserver/docker-mailserver/blob/master/config-examples/fail2ban-fail2ban.cf

View File

@ -83,9 +83,15 @@ DMS does not set a default password for the controller worker. You may want to d
### Persistence with Redis
When Rspamd is enabled, we implicitly also start an instance of Redis in the container. Redis is configured to persist its data via RDB snapshots to disk in the directory `/var/lib/redis` (_which is a symbolic link to `/var/mail-state/lib-redis/` when [`ONE_DIR=1`](../environment.md#one_dir) and a volume is mounted to `/var/mail-state/`_). With the volume mount the snapshot will restore the Redis data across container restarts, and provide a way to keep backup.
When Rspamd is enabled, we implicitly also start an instance of Redis in the container:
Redis uses `/etc/redis/redis.conf` for configuration. We adjust this file when enabling the internal Redis service. If you have an external instance of Redis to use, the internal Redis service can be opt-out via setting the ENV [`ENABLE_RSPAMD_REDIS=0`](../environment.md#enable_rspamd_redis) (_link also details required changes to the DMS Rspamd config_).
- Redis is configured to persist its data via RDB snapshots to disk in the directory `/var/lib/redis` (_or the [`/var/mail-state/`][docs::dms-volumes-state] volume when present_).
- With the volume mount the snapshot will restore the Redis data across container restarts, and provide a way to keep backup.
Redis uses `/etc/redis/redis.conf` for configuration:
- We adjust this file when enabling the internal Redis service.
- If you have an external instance of Redis to use, the internal Redis service can be opt-out via setting the ENV [`ENABLE_RSPAMD_REDIS=0`](../environment.md#enable_rspamd_redis) (_link also details required changes to the DMS Rspamd config_).
### Web Interface
@ -145,7 +151,7 @@ DMS brings sane default settings for Rspamd. They are located at `/etc/rspamd/lo
### Manually
!!! question "What is [`docker-data/dms/config/`][docs-dms-config-volume]?"
!!! question "What is [`docker-data/dms/config/`][docs::dms-volumes-config]?"
If you want to overwrite the default settings and / or provide your own settings, you can place files at `docker-data/dms/config/rspamd/override.d/`. Files from this directory are copied to `/etc/rspamd/override.d/` during startup. These files [forcibly override][rspamd-docs-override-dir] Rspamd and DMS default settings.
@ -156,7 +162,6 @@ If you want to overwrite the default settings and / or provide your own settings
Note that when also [using the `custom-commands.conf` file](#with-the-help-of-a-custom-file), files in `override.d` may be overwritten in case you adjust them manually and with the help of the file.
[rspamd-docs-override-dir]: https://www.rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories
[docs-dms-config-volume]: ../../faq.md#what-about-the-docker-datadmsconfig-directory
[rspamd-docs-config-directories]: https://rspamd.com/doc/faq.html#what-are-the-locald-and-overrided-directories
### With the Help of a Custom File
@ -197,7 +202,7 @@ You can also have comments (the line starts with `#`) and blank lines in `custom
You want to start using Rspamd? Rspamd is disabled by default, so you need to set the following environment variables:
```cf
```env
ENABLE_RSPAMD=1
ENABLE_OPENDKIM=0
ENABLE_OPENDMARC=0
@ -252,3 +257,6 @@ While _Abusix_ can be integrated into Postfix, Postscreen and a multitude of oth
[Abusix]: https://abusix.com/
[abusix-rspamd-integration]: https://docs.abusix.com/abusix-mail-intelligence/gbG8EcJ3x3fSUv8cMZLiwA/getting-started/dmw9dcwSGSNQiLTssFAnBW#rspamd
[docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config
[docs::dms-volumes-state]: ../advanced/optional-config.md#volumes-state

View File

@ -634,7 +634,7 @@ This setup only comes with one caveat: The domain has to be configured on anothe
Use self-signed certificates only for testing purposes!
This feature requires you to provide the following files into your [`docker-data/dms/config/ssl/` directory][docs-optional-config] (_internal location: `/tmp/docker-mailserver/ssl/`_):
This feature requires you to provide the following files into your [`docker-data/dms/config/ssl/` directory][docs::dms-volumes-config] (_internal location: `/tmp/docker-mailserver/ssl/`_):
- `<FQDN>-key.pem`
- `<FQDN>-cert.pem`
@ -876,7 +876,7 @@ By default DMS uses [`ffdhe4096`][ffdhe4096-src] from [IETF RFC 7919][ietf::rfc:
Despite this, if you must use non-standard DH parameters or you would like to swap `ffdhe4096` for a different group (eg `ffdhe2048`); Add your own PEM encoded DH params file via a volume to `/tmp/docker-mailserver/dhparams.pem`. This will replace DH params for both Dovecot and Postfix services during container startup.
[docs-env::ssl-type]: ../environment.md#ssl_type
[docs-optional-config]: ../advanced/optional-config.md
[docs::dms-volumes-config]: ../advanced/optional-config.md#volumes-config
[docs-faq-baredomain]: ../../faq.md#can-i-use-a-nakedbare-domain-ie-no-hostname
[github-file-compose]: https://github.com/docker-mailserver/docker-mailserver/blob/master/compose.yaml

View File

@ -125,5 +125,3 @@ service imap-login {
!!! note
Port `10993` is used here to avoid conflicts with internal systems like `postscreen` and `amavis` as they will exchange messages on the default port and obviously have a different origin then compared to the proxy.
[docs-optionalconfig]: ../../config/advanced/optional-config.md

View File

@ -360,20 +360,6 @@ DMS does not manage those concerns, verify they are not causing your delivery pr
- [mail-tester](https://www.mail-tester.com/) can test your deliverability.
- [helloinbox](https://www.helloinbox.email/) provides a checklist of things to improve your deliverability.
### Special Directories
#### What About the `docker-data/dms/config/` Directory?
This documentation and all example configuration files in the GitHub repository use `docker-data/dms/config/` to refer to the directory in the host that is mounted (e.g. via a bind mount) to `/tmp/docker-mailserver/` inside the container.
Most configuration files for Postfix, Dovecot, etc. are persisted here. [Optional configuration][docs-optional-configuration] is stored here as well.
#### What About the `docker-data/dms/mail-state/` Directory?
This documentation and all example configuration files in the GitHub repository use `docker-data/dms/mail-state/` to refer to the directory in the host that is mounted (e.g. via a bind mount) to `/var/mail-state/` inside the container.
When you run DMS with the ENV variable `ONE_DIR=1` (default), this directory will provide support to persist Fail2Ban blocks, ClamAV signature updates, and the like when the container is restarted or recreated. Service data is [relocated to the `mail-state` folder][mail-state-folders] for the following services: Postfix, Dovecot, Fail2Ban, Amavis, PostGrey, ClamAV, SpamAssassin, Rspamd & Redis.
### SpamAssasin
#### How can I manage my custom SpamAssassin rules?
@ -390,14 +376,15 @@ The default setup `@local_domains_acl = ( ".$mydomain" );` does not match subdom
Put received spams in `.Junk/` imap folder using `SPAMASSASSIN_SPAM_TO_INBOX=1` and `MOVE_SPAM_TO_JUNK=1` and add a _user_ cron like the following:
```conf
# This assumes you're having `environment: ONE_DIR=1` in the `mailserver.env`,
# with a consolidated config in `/var/mail-state`
#
# m h dom mon dow command
# Everyday 2:00AM, learn spam from a specific user
0 2 * * * docker exec mailserver sa-learn --spam /var/mail/example.com/username/.Junk --dbpath /var/mail-state/lib-amavis/.spamassassin
```
!!! example
**NOTE:** This example assumes you have a [`/var/mail-state` volume][docs::dms-volumes-state] mounted.
```conf
# m h dom mon dow command
# Everyday 2:00AM, learn spam from a specific user
0 2 * * * docker exec mailserver sa-learn --spam /var/mail/example.com/username/.Junk --dbpath /var/mail-state/lib-amavis/.spamassassin
```
With `docker-compose` you can more easily use the internal instance of `cron` within DMS. This is less problematic than the simple solution shown above, because it decouples the learning from the host on which DMS is running, and avoids errors if the mail server is not running.
@ -405,6 +392,8 @@ The following configuration works nicely:
??? example
**NOTE:** This example assumes you have a [`/var/mail-state` volume][docs::dms-volumes-state] mounted.
Create a _system_ cron file:
```sh
@ -418,9 +407,6 @@ The following configuration works nicely:
Edit the system cron file `nano ./docker-data/dms/cron/sa-learn`, and set an appropriate configuration:
```conf
# This assumes you're having `environment: ONE_DIR=1` in the env-mailserver,
# with a consolidated config in `/var/mail-state`
#
# '> /dev/null' to send error notifications from 'stderr' to 'postmaster@example.com'
#
# m h dom mon dow user command
@ -495,10 +481,10 @@ $spam_quarantine_to = "quarantine\@example.com";
```
[fail2ban-customize]: ./config/security/fail2ban.md
[docs::dms-volumes-state]: ./config/advanced/optional-config.md#volumes-state
[docs-maintenance]: ./config/advanced/maintenance/update-and-cleanup.md
[docs-override-postfix]: ./config/advanced/override-defaults/postfix.md
[docs-userpatches]: ./config/advanced/override-defaults/user-patches.md
[docs-optional-configuration]: ./config/advanced/optional-config.md
[docs::env::sa_env]: ./config/environment.md#spamassassin
[docs::env::sa_kill]: ./config/environment.md#sa_kill
[github-comment-baredomain]: https://github.com/docker-mailserver/docker-mailserver/issues/3048#issuecomment-1432358353
@ -510,4 +496,3 @@ $spam_quarantine_to = "quarantine\@example.com";
[github-issue-1639]: https://github.com/docker-mailserver/docker-mailserver/issues/1639
[github-issue-1792]: https://github.com/docker-mailserver/docker-mailserver/pull/1792
[hanscees-userpatches]: https://github.com/hanscees/dockerscripts/blob/master/scripts/tomav-user-patches.sh
[mail-state-folders]: https://github.com/docker-mailserver/docker-mailserver/blob/c7e498194546416fb7231cb03254e77e085d18df/target/scripts/startup/misc-stack.sh#L24-L33

View File

@ -2,9 +2,9 @@
title: Usage
---
This pages explains how to get started with DMS. The guide uses Docker Compose as a reference. In our examples, a volume mounts the host location [`docker-data/dms/config/`][docs-dms-config-volume] to `/tmp/docker-mailserver/` inside the container.
This pages explains how to get started with DMS. The guide uses Docker Compose as a reference. In our examples, a volume mounts the host location [`docker-data/dms/config/`][docs::dms-volumes-config] to `/tmp/docker-mailserver/` inside the container.
[docs-dms-config-volume]: ./faq.md#what-about-the-docker-datadmsconfig-directory
[docs::dms-volumes-config]: ./config/advanced/optional-config.md#volumes-config
## Preliminary Steps

View File

@ -30,10 +30,6 @@ LOG_LEVEL=info
# debug => Also show debug messages
SUPERVISOR_LOGLEVEL=
# 0 => mail state in default directories
# 1 => consolidate all states into a single directory (`/var/mail-state`) to allow persistence using docker volumes
ONE_DIR=1
# Support for deployment where these defaults are not compatible (eg: some NAS appliances):
# /var/mail vmail User ID (default: 5000)
DMS_VMAIL_UID=

View File

@ -7,7 +7,7 @@ function _setup_save_states() {
STATEDIR='/var/mail-state'
if [[ ${ONE_DIR} -eq 1 ]] && [[ -d ${STATEDIR} ]]; then
if [[ -d ${STATEDIR} ]]; then
_log 'debug' "Consolidating all state onto ${STATEDIR}"
# Always enabled features:
@ -111,9 +111,7 @@ function _setup_save_states() {
# Ref: https://github.com/docker-mailserver/docker-mailserver/pull/3625
chmod 730 "${STATEDIR}/spool-postfix/maildrop"
chmod 710 "${STATEDIR}/spool-postfix/public"
elif [[ ${ONE_DIR} -eq 1 ]]; then
_log 'warn' "'ONE_DIR=1' but no volume was mounted to '${STATEDIR}'"
else
_log 'debug' 'Not consolidating state (because it has been disabled)'
_log 'debug' "'${STATEDIR}' is not present; Not consolidating state"
fi
}

View File

@ -127,9 +127,9 @@ expand_keys = true;
EOF
# Here we adjust the Redis default configuration that we supply to Redis
# when starting it. Note that `/var/lib/redis/` is linked to
# `/var/mail-state/redis/` (for persisting it) if `ONE_DIR=1`.
# Here we adjust the Redis default configuration that we supply to Redis when starting it.
# NOTE: `/var/lib/redis/` is symlinked to `/var/mail-state/redis/` when DMS is started
# with a volume mounted to `/var/mail-state/` for data persistence.
sedfile -i -E \
-e 's|^(bind).*|\1 127.0.0.1|g' \
-e 's|^(daemonize).*|\1 no|g' \

View File

@ -141,7 +141,6 @@ function __environment_variables_general_setup() {
VARS[LOGWATCH_INTERVAL]="${LOGWATCH_INTERVAL:=none}"
VARS[LOGWATCH_RECIPIENT]="${LOGWATCH_RECIPIENT:=${REPORT_RECIPIENT}}"
VARS[LOGWATCH_SENDER]="${LOGWATCH_SENDER:=${REPORT_SENDER}}"
VARS[ONE_DIR]="${ONE_DIR:=1}"
VARS[PERMIT_DOCKER]="${PERMIT_DOCKER:=none}"
VARS[PFLOGSUMM_RECIPIENT]="${PFLOGSUMM_RECIPIENT:=${REPORT_RECIPIENT}}"
VARS[PFLOGSUMM_SENDER]="${PFLOGSUMM_SENDER:=${REPORT_SENDER}}"