From b5356568d5370e03a7f0c2fbdf93296cd6cc7ab9 Mon Sep 17 00:00:00 2001 From: Mitchel Humpherys Date: Sun, 29 Sep 2019 12:06:07 -0700 Subject: [PATCH 01/24] docker_run.sh: Add env var to specify storage directory Currently `docker_run.sh` bind-mounts some directories rooted at the working directory when the script is invoked. Add an environment variable so that this storage location can be specified at invocation time without having to change to a different directory. Also creates PIHOLE_BASE if it doesn't exist already. --- README.md | 2 ++ docker_run.sh | 7 +++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1b1b8a4..afdb934 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,8 @@ If you're using a Red Hat based distrubution with an SELinux Enforcing policy ad Volumes are recommended for persisting data across container re-creations for updating images. The IP lookup variables may not work for everyone, please review their values and hard code IP and IPv6 if necessary. +You can customize where to store persistent data by setting the `PIHOLE_BASE` environment variable when invoking `docker_run.sh` (e.g. `PIHOLE_BASE=/opt/pihole-storage ./docker_run.sh`). If `PIHOLE_BASE` is not set, files are stored in your current directory when you invoke the script. + Port 443 is to provide a sinkhole for ads that use SSL. If only port 80 is used, then blocked HTTPS queries will fail to connect to port 443 and may cause long loading times. Rejecting 443 on your firewall can also serve this same purpose. Ubuntu firewall example: `sudo ufw reject https` **Automatic Ad List Updates** - since the 3.0+ release, `cron` is baked into the container and will grab the newest versions of your lists and flush your logs. **Set your TZ** environment variable to make sure the midnight log rotation syncs up with your timezone's midnight. diff --git a/docker_run.sh b/docker_run.sh index 071f83d..0197f34 100755 --- a/docker_run.sh +++ b/docker_run.sh @@ -2,14 +2,17 @@ # https://github.com/pi-hole/docker-pi-hole/blob/master/README.md +PIHOLE_BASE=${PIHOLE_BASE:-$(pwd)} +[[ -d "$PIHOLE_BASE" ]] || mkdir -p "$PIHOLE_BASE" || { echo "Couldn't create storage directory: $PIHOLE_BASE"; exit 1; } + docker run -d \ --name pihole \ -p 53:53/tcp -p 53:53/udp \ -p 80:80 \ -p 443:443 \ -e TZ="America/Chicago" \ - -v "$(pwd)/etc-pihole/:/etc/pihole/" \ - -v "$(pwd)/etc-dnsmasq.d/:/etc/dnsmasq.d/" \ + -v "${PIHOLE_BASE}/etc-pihole/:/etc/pihole/" \ + -v "${PIHOLE_BASE}/etc-dnsmasq.d/:/etc/dnsmasq.d/" \ --dns=127.0.0.1 --dns=1.1.1.1 \ --restart=unless-stopped \ pihole/pihole:latest From 2228115a97a6baaac5e1596b8a5d407e6dba1c8f Mon Sep 17 00:00:00 2001 From: "Andrew J. Huffman" Date: Mon, 4 May 2020 17:15:35 -0400 Subject: [PATCH 02/24] Fixes #557 * Refactored the prepare_configs function in bash_functions.sh * Now able to set settings via the UI or ENV Variables - ENV Variables override when restarted * Added ENV Variables for ADMIN_EMAIL, TEMPERATUREUNIT, and WEBUIBOXEDLAYOUT - Created setup_temp_unit function in bash_functions.sh - Created setup_ui_layout function in bash_functions.sh - Created setup_admin_email function in bash_functions.sh * Updated README.md - Added docs on new ENV Variables - Updated True/False settings docs so they were quoted, otherwise misleading to k8s users and not "true" booleans Signed-off-by: Andrew J. Huffman --- README.md | 13 ++++++---- bash_functions.sh | 62 ++++++++++++++++++++++++++++------------------- start.sh | 17 ++++++++++++- 3 files changed, 61 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 15d2cbb..0db226d 100644 --- a/README.md +++ b/README.md @@ -98,25 +98,28 @@ There are other environment variables if you want to customize various things in | Docker Environment Var. | Description | | ----------------------- | ----------- | +| `ADMIN_EMAIL: `
*Optional Default: ''* | Set an administrative contact address for the Block Page | `TZ: `
**Recommended** *Default: UTC* | Set your [timezone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) to make sure logs rotate at local midnight instead of at UTC midnight. | `WEBPASSWORD: `
**Recommended** *Default: random* | http://pi.hole/admin password. Run `docker logs pihole \| grep random` to find your random pass. | `DNS1: `
*Optional* *Default: 8.8.8.8* | Primary upstream DNS provider, default is google DNS | `DNS2: `
*Optional* *Default: 8.8.4.4* | Secondary upstream DNS provider, default is google DNS, `no` if only one DNS should used -| `DNSSEC: `
*Optional* *Default: false* | Enable DNSSEC support -| `DNS_BOGUS_PRIV: `
*Optional* *Default: true* | Enable forwarding of reverse lookups for private ranges -| `DNS_FQDN_REQUIRED: `
*Optional* *Default: true* | Never forward non-FQDNs -| `CONDITIONAL_FORWARDING: `
*Optional* *Default: False* | Enable DNS conditional forwarding for device name resolution +| `DNSSEC: <"true"\|"false">`
*Optional* *Default: "false"* | Enable DNSSEC support +| `DNS_BOGUS_PRIV: <"true"\|"false">`
*Optional* *Default: "true"* | Enable forwarding of reverse lookups for private ranges +| `DNS_FQDN_REQUIRED: <"true"\|"false">`
*Optional* *Default: true* | Never forward non-FQDNs +| `CONDITIONAL_FORWARDING: <"true"\|"false">`
*Optional* *Default: "false"* | Enable DNS conditional forwarding for device name resolution | `CONDITIONAL_FORWARDING_IP: `
*Optional* | If conditional forwarding is enabled, set the IP of the local network router | `CONDITIONAL_FORWARDING_DOMAIN: `
*Optional* | If conditional forwarding is enabled, set the domain of the local network router | `CONDITIONAL_FORWARDING_REVERSE: `
*Optional* | If conditional forwarding is enabled, set the reverse DNS of the local network router (e.g. `0.168.192.in-addr.arpa`) | `ServerIP: `
**Recommended** | **--net=host mode requires** Set to your server's LAN IP, used by web block modes and lighttpd bind address | `ServerIPv6: `
*Required if using IPv6* | **If you have a v6 network** set to your server's LAN IPv6 to block IPv6 ads fully | `VIRTUAL_HOST: `
*Optional* *Default: $ServerIP* | What your web server 'virtual host' is, accessing admin through this Hostname/IP allows you to make changes to the whitelist / blacklists in addition to the default 'http://pi.hole/admin/' address -| `IPv6: `
*Optional* *Default: True* | For unraid compatibility, strips out all the IPv6 configuration from DNS/Web services when false. +| `IPv6: <"true"\|"false">`
*Optional* *Default: "true"* | For unraid compatibility, strips out all the IPv6 configuration from DNS/Web services when false. | `INTERFACE: `
*Advanced/Optional* | The default works fine with our basic example docker run commands. If you're trying to use DHCP with `--net host` mode then you may have to customize this or DNSMASQ_LISTENING. | `DNSMASQ_LISTENING: `
*Advanced/Optional* | `local` listens on all local subnets, `all` permits listening on internet origin subnets in addition to local. | `WEB_PORT: `
*Advanced/Optional* | **This will break the 'webpage blocked' functionality of Pi-hole** however it may help advanced setups like those running synology or `--net=host` docker argument. This guide explains how to restore webpage blocked functionality using a linux router DNAT rule: [Alternative Synology installation method](https://discourse.pi-hole.net/t/alternative-synology-installation-method/5454?u=diginc) | `DNSMASQ_USER: `
*Experimental Default: root* | Allows running FTLDNS as non-root. +| `TEMPERATUREUNIT`:
*Optional Default: c* | Set preferred temperature unit to `c`: Celsius, `k`: Kelvin, or `f` Fahrenheit units. +| `WEBUIBOXEDLAYOUT: `
*Optional Default: boxed* | Use boxed layout (helpful when working on large screens) To use these env vars in docker run format style them like: `-e DNS1=1.1.1.1` diff --git a/bash_functions.sh b/bash_functions.sh index 89de84a..e90ead4 100644 --- a/bash_functions.sh +++ b/bash_functions.sh @@ -48,7 +48,7 @@ prepare_configs() { # Also similar to preflights for FTL https://github.com/pi-hole/pi-hole/blob/master/advanced/Templates/pihole-FTL.service chown pihole:root /etc/lighttpd chown pihole:pihole "${PI_HOLE_CONFIG_DIR}/pihole-FTL.conf" "/var/log/pihole" "${regexFile}" - chmod 644 "${PI_HOLE_CONFIG_DIR}/pihole-FTL.conf" + chmod 644 "${PI_HOLE_CONFIG_DIR}/pihole-FTL.conf" # not sure why pihole:pihole user/group write perms are not enough for web to write...dirty fix: chmod 777 "${regexFile}" touch /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port /var/log/pihole.log @@ -60,27 +60,11 @@ prepare_configs() { # Update version numbers pihole updatechecker # Re-write all of the setupVars to ensure required ones are present (like QUERY_LOGGING) - + # If the setup variable file exists, if [[ -e "${setupVars}" ]]; then - # update the variables in the file - local USERWEBPASSWORD="${WEBPASSWORD}" - . "${setupVars}" - # Stash and pop the user password to avoid setting the password to the hashed setupVar variable - WEBPASSWORD="${USERWEBPASSWORD}" - # Clean up old before re-writing the required setupVars - sed -i.update.bak '/PIHOLE_INTERFACE/d;/IPV4_ADDRESS/d;/IPV6_ADDRESS/d;/QUERY_LOGGING/d;/INSTALL_WEB_SERVER/d;/INSTALL_WEB_INTERFACE/d;/LIGHTTPD_ENABLED/d;' "${setupVars}" + cp -f "${setupVars}" "${setupVars}.update.bak" fi - # echo the information to the user - { - echo "PIHOLE_INTERFACE=${PIHOLE_INTERFACE}" - echo "IPV4_ADDRESS=${IPV4_ADDRESS}" - echo "IPV6_ADDRESS=${IPV6_ADDRESS}" - echo "QUERY_LOGGING=${QUERY_LOGGING}" - echo "INSTALL_WEB_SERVER=${INSTALL_WEB_SERVER}" - echo "INSTALL_WEB_INTERFACE=${INSTALL_WEB_INTERFACE}" - echo "LIGHTTPD_ENABLED=${LIGHTTPD_ENABLED}" - }>> "${setupVars}" } validate_env() { @@ -123,7 +107,7 @@ setup_dnsmasq_dns() { setupDNS1="${setupDNS1/PIHOLE_DNS_1=/}" setupDNS2="${setupDNS2/PIHOLE_DNS_2=/}" if [[ -n "$DNS1" && -n "$setupDNS1" ]] || \ - [[ -n "$DNS2" && -n "$setupDNS2" ]] ; then + [[ -n "$DNS2" && -n "$setupDNS2" ]] ; then echo "Docker DNS variables not used" fi echo "Existing DNS servers used (${setupDNS1:-unset} & ${setupDNS2:-unset})" @@ -174,9 +158,9 @@ setup_dnsmasq() { local dns2="$2" local interface="$3" local dnsmasq_listening_behaviour="$4" - # Coordinates + # Coordinates setup_dnsmasq_config_if_missing - setup_dnsmasq_dns "$dns1" "$dns2" + setup_dnsmasq_dns "$dns1" "$dns2" setup_dnsmasq_interface "$interface" setup_dnsmasq_listening_behaviour "$dnsmasq_listening_behaviour" setup_dnsmasq_user "${DNSMASQ_USER}" @@ -269,7 +253,7 @@ setup_web_port() { # Quietly exit early for empty or default if [[ -z "${1}" || "${1}" == '80' ]] ; then return ; fi - if ! echo $1 | grep -q '^[0-9][0-9]*$' ; then + if ! echo $1 | grep -q '^[0-9][0-9]*$' ; then echo "$warning - $1 is not an integer" return fi @@ -340,9 +324,8 @@ test_configs() { echo "::: All config checks passed, cleared for startup ..." } - setup_blocklists() { - local blocklists="$1" + local blocklists="$1" # Exit/return early without setting up adlists with defaults for any of the following conditions: # 1. skip_setup_blocklists env is set exit_string="(exiting ${FUNCNAME[0]} early)" @@ -380,3 +363,32 @@ setup_var_exists() { fi } +setup_temp_unit() { + local UNIT="$1" + # check if var is empty + if [[ "$UNIT" != "" ]] ; then + # check if we have valid units + if [[ "$UNIT" == "c" || "$UNIT" == "k" || $UNIT == "f" ]] ; then + pihole -a -${UNIT} + fi + fi +} + +setup_ui_layout() { + local LO=$1 + # check if var is empty + if [[ "$LO" != "" ]] ; then + # check if we have valid types boxed | traditional + if [[ "$LO" == "traditional" || "$LO" == "boxed" ]] ; then + change_setting "WEBUIBOXEDLAYOUT" "$WEBUIBOXEDLAYOUT" + fi + fi +} + +setup_admin_email() { + local EMAIL=$1 + # check if var is empty + if [[ "$EMAIL" != "" ]] ; then + pihole -a -e "$EMAIL" + fi +} diff --git a/start.sh b/start.sh index 9802985..5f18560 100755 --- a/start.sh +++ b/start.sh @@ -5,7 +5,7 @@ export ServerIP export ServerIPv6 export PYTEST export PHP_ENV_CONFIG -export PHP_ERROR_LOG +export PHP_ERROR_LOG export HOSTNAME export WEBLOGDIR export DNS1 @@ -21,12 +21,18 @@ export CONDITIONAL_FORWARDING export CONDITIONAL_FORWARDING_IP export CONDITIONAL_FORWARDING_DOMAIN export CONDITIONAL_FORWARDING_REVERSE +export TEMPERATUREUNIT +export ADMIN_EMAIL +export WEBUIBOXEDLAYOUT export adlistFile='/etc/pihole/adlists.list' # The below functions are all contained in bash_functions.sh . /bash_functions.sh +# Ensure we have all functions available to update our configurations +. /opt/pihole/webpage.sh + # PH_TEST prevents the install from actually running (someone should rename that) PH_TEST=true . $PIHOLE_INSTALL @@ -45,6 +51,12 @@ fix_capabilities generate_password validate_env || exit 1 prepare_configs +change_setting "PIHOLE_INTERFACE" "$PIHOLE_INTERFACE" +change_setting "IPV4_ADDRESS" "$IPV4_ADDRESS" +change_setting "QUERY_LOGGING" "$QUERY_LOGGING" +change_setting "INSTALL_WEB_SERVER" "$INSTALL_WEB_SERVER" +change_setting "INSTALL_WEB_INTERFACE" "$INSTALL_WEB_INTERFACE" +change_setting "LIGHTTPD_ENABLED" "$LIGHTTPD_ENABLED" change_setting "IPV4_ADDRESS" "$ServerIP" change_setting "IPV6_ADDRESS" "$ServerIPv6" change_setting "DNS_BOGUS_PRIV" "$DNS_BOGUS_PRIV" @@ -56,6 +68,9 @@ change_setting "CONDITIONAL_FORWARDING_DOMAIN" "$CONDITIONAL_FORWARDING_DOMAIN" change_setting "CONDITIONAL_FORWARDING_REVERSE" "$CONDITIONAL_FORWARDING_REVERSE" setup_web_port "$WEB_PORT" setup_web_password "$WEBPASSWORD" +setup_temp_unit "$TEMPERATUREUNIT" +setup_ui_layout "$WEBUIBOXEDLAYOUT" +setup_admin_email "$ADMIN_EMAIL" setup_dnsmasq "$DNS1" "$DNS2" "$INTERFACE" "$DNSMASQ_LISTENING_BEHAVIOUR" setup_php_env setup_dnsmasq_hostnames "$ServerIP" "$ServerIPv6" "$HOSTNAME" From 740248f18bf71b751a4ab9e44d504bc0c8cfb81b Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Mon, 11 May 2020 18:38:19 -0500 Subject: [PATCH 03/24] Add quote for path --- docker_run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker_run.sh b/docker_run.sh index 0197f34..7c7751e 100755 --- a/docker_run.sh +++ b/docker_run.sh @@ -2,7 +2,7 @@ # https://github.com/pi-hole/docker-pi-hole/blob/master/README.md -PIHOLE_BASE=${PIHOLE_BASE:-$(pwd)} +PIHOLE_BASE="${PIHOLE_BASE:-$(pwd)}" [[ -d "$PIHOLE_BASE" ]] || mkdir -p "$PIHOLE_BASE" || { echo "Couldn't create storage directory: $PIHOLE_BASE"; exit 1; } docker run -d \ From 6939ea024fe0583f472465a9314deca0d4c5f2ba Mon Sep 17 00:00:00 2001 From: Daniel Date: Sun, 29 Mar 2020 23:07:46 -0400 Subject: [PATCH 04/24] Simplify Dockerfile configuration. * Simplify docker builds by consolidating all arch's into a single Dockerfile and using ARGS for various differences * Introduce docker-compose based builds (build.yml) for simple management of the various args differences Signed-off-by: Daniel --- Dockerfile.template => Dockerfile | 25 ++++++--- Dockerfile.py | 85 ++++--------------------------- Dockerfile_amd64 | 49 ------------------ Dockerfile_arm64 | 49 ------------------ Dockerfile_armel | 49 ------------------ Dockerfile_armhf | 49 ------------------ README.md | 2 +- build.yml | 57 +++++++++++++++++++++ 8 files changed, 85 insertions(+), 280 deletions(-) rename Dockerfile.template => Dockerfile (62%) delete mode 100644 Dockerfile_amd64 delete mode 100644 Dockerfile_arm64 delete mode 100644 Dockerfile_armel delete mode 100644 Dockerfile_armhf create mode 100644 build.yml diff --git a/Dockerfile.template b/Dockerfile similarity index 62% rename from Dockerfile.template rename to Dockerfile index cce6af4..9fff81c 100644 --- a/Dockerfile.template +++ b/Dockerfile @@ -1,7 +1,11 @@ -FROM {{ pihole.base }} +ARG PIHOLE_BASE +FROM $PIHOLE_BASE -ENV ARCH {{ pihole.arch }} -ENV S6OVERLAY_RELEASE https://github.com/just-containers/s6-overlay/releases/download/{{ pihole.s6_version }}/s6-overlay-{{ pihole.s6arch }}.tar.gz +ARG PIHOLE_ARCH +ENV PIHOLE_ARCH "${PIHOLE_ARCH}" +ARG S6_ARCH +ARG S6_VERSION +ENV S6OVERLAY_RELEASE "https://github.com/just-containers/s6-overlay/releases/download/${S6_VERSION}/s6-overlay-${S6_ARCH}.tar.gz" COPY install.sh /usr/local/bin/install.sh COPY VERSION /etc/docker-pi-hole-version @@ -16,8 +20,10 @@ ADD s6/debian-root / COPY s6/service /usr/local/bin/service # php config start passes special ENVs into -ENV PHP_ENV_CONFIG '{{ pihole.php_env_config }}' -ENV PHP_ERROR_LOG '{{ pihole.php_error_log }}' +ARG PHP_ENV_CONFIG +ENV PHP_ENV_CONFIG "${PHP_ENV_CONFIG}" +ARG PHP_ERROR_LOG +ENV PHP_ERROR_LOG "${PHP_ERROR_LOG}" COPY ./start.sh / COPY ./bash_functions.sh / @@ -37,11 +43,14 @@ ENV ServerIP 0.0.0.0 ENV FTL_CMD no-daemon ENV DNSMASQ_USER root -ENV VERSION {{ pihole.version }} +ARG PIHOLE_VERSION +ENV VERSION "${PIHOLE_VERSION}" ENV PATH /opt/pihole:${PATH} -LABEL image="{{ pihole.name }}:{{ pihole.version }}_{{ pihole.arch }}" -LABEL maintainer="{{ pihole.maintainer }}" +ARG NAME +LABEL image="${NAME}:${PIHOLE_VERSION}_${PIHOLE_ARCH}" +ARG MAINTAINER +LABEL maintainer="${MAINTAINER}" LABEL url="https://www.github.com/pi-hole/docker-pi-hole" HEALTHCHECK CMD dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1 diff --git a/Dockerfile.py b/Dockerfile.py index 0aa1e1b..7fc1363 100755 --- a/Dockerfile.py +++ b/Dockerfile.py @@ -2,12 +2,11 @@ """ Dockerfile.py - generates and build dockerfiles Usage: - Dockerfile.py [--hub_tag=] [--arch= ...] [-v] [-t] [--no-build | --no-generate] [--no-cache] + Dockerfile.py [--hub_tag=] [--arch= ...] [-v] [-t] [--no-build] [--no-cache] Options: --no-build Skip building the docker images --no-cache Build without using any cache data - --no-generate Skip generating Dockerfiles from template --hub_tag= What the Docker Hub Image should be tagged as [default: None] --arch= What Architecture(s) to build [default: amd64 armel armhf arm64] -v Print docker's command output [default: False] @@ -17,81 +16,18 @@ Examples: """ -from jinja2 import Environment, FileSystemLoader from docopt import docopt import os import subprocess -import sys THIS_DIR = os.path.dirname(os.path.abspath(__file__)) -base_vars = { - 'name': 'pihole/pihole', - 'maintainer' : 'adam@diginc.us', - 's6_version' : 'v1.22.1.0', -} - -os_base_vars = { - 'php_env_config': '/etc/lighttpd/conf-enabled/15-fastcgi-php.conf', - 'php_error_log': '/var/log/lighttpd/error.log' -} - __version__ = None dot = os.path.abspath('.') with open('{}/VERSION'.format(dot), 'r') as v: raw_version = v.read().strip() __version__ = raw_version.replace('release/', 'release-') -images = { - __version__: [ - { - 'base': 'pihole/debian-base:latest', - 'arch': 'amd64', - 's6arch': 'amd64', - }, - { - 'base': 'multiarch/debian-debootstrap:armel-stretch-slim', - 'arch': 'armel', - 's6arch': 'arm', - }, - { - 'base': 'multiarch/debian-debootstrap:armhf-stretch-slim', - 'arch': 'armhf', - 's6arch' : 'arm', - }, - { - 'base': 'multiarch/debian-debootstrap:arm64-stretch-slim', - 'arch': 'arm64', - 's6arch' : 'aarch64', - } - ] -} - -def generate_dockerfiles(args): - if args['--no-generate']: - print(" ::: Skipping Dockerfile generation") - return - - for version, archs in images.items(): - for image in archs: - if image['arch'] not in args['--arch']: - continue - s6arch = image['s6arch'] if image['s6arch'] else image['arch'] - merged_data = dict( - list({ 'version': version }.items()) + - list(base_vars.items()) + - list(os_base_vars.items()) + - list(image.items()) + - list({ 's6arch': s6arch }.items()) - ) - j2_env = Environment(loader=FileSystemLoader(THIS_DIR), - trim_blocks=True) - template = j2_env.get_template('Dockerfile.template') - - dockerfile = 'Dockerfile_{}'.format(image['arch']) - with open(dockerfile, 'w') as f: - f.write(template.render(pihole=merged_data)) - def build_dockerfiles(args): if args['--no-build']: @@ -104,7 +40,9 @@ def build_dockerfiles(args): def run_and_stream_command_output(command, args): print("Running", command) - build_result = subprocess.Popen(command.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + build_env = os.environ.copy() + build_env['PIHOLE_VERSION'] = __version__ + build_result = subprocess.Popen(command.split(), env=build_env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True) if args['-v']: while build_result.poll() is None: @@ -117,19 +55,17 @@ def run_and_stream_command_output(command, args): def build(docker_repo, arch, args): - dockerfile = 'Dockerfile_{}'.format(arch) - repo_tag = '{}:{}_{}'.format(docker_repo, __version__, arch) - cached_image = '{}/{}'.format('pihole', repo_tag) + repo_tag = '{}:{}-{}'.format(docker_repo, __version__, arch) print(" ::: Building {}".format(repo_tag)) - time='' + time = '' if args['-t']: - time='time ' + time = 'time ' no_cache = '' if args['--no-cache']: no_cache = '--no-cache' - build_command = '{time}docker build {no_cache} --pull --cache-from="{cache},{create_tag}" -f {dockerfile} -t {create_tag} .'\ - .format(time=time, no_cache=no_cache, cache=cached_image, dockerfile=dockerfile, create_tag=repo_tag) - print(" ::: Building {} into {}".format(dockerfile, repo_tag)) + build_command = '{time}docker-compose -f build.yml build {no_cache} --pull {arch}'\ + .format(time=time, no_cache=no_cache, arch=arch) + print(" ::: Building {} into {}".format(arch, repo_tag)) run_and_stream_command_output(build_command, args) if args['-v']: print(build_command, '\n') @@ -145,5 +81,4 @@ if __name__ == '__main__': if args['-v']: print(args) - generate_dockerfiles(args) build_dockerfiles(args) diff --git a/Dockerfile_amd64 b/Dockerfile_amd64 deleted file mode 100644 index 121adde..0000000 --- a/Dockerfile_amd64 +++ /dev/null @@ -1,49 +0,0 @@ -FROM pihole/debian-base:latest - -ENV ARCH amd64 -ENV S6OVERLAY_RELEASE https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-amd64.tar.gz - -COPY install.sh /usr/local/bin/install.sh -COPY VERSION /etc/docker-pi-hole-version -ENV PIHOLE_INSTALL /root/ph_install.sh - -RUN bash -ex install.sh 2>&1 && \ - rm -rf /var/cache/apt/archives /var/lib/apt/lists/* - -ENTRYPOINT [ "/s6-init" ] - -ADD s6/debian-root / -COPY s6/service /usr/local/bin/service - -# php config start passes special ENVs into -ENV PHP_ENV_CONFIG '/etc/lighttpd/conf-enabled/15-fastcgi-php.conf' -ENV PHP_ERROR_LOG '/var/log/lighttpd/error.log' -COPY ./start.sh / -COPY ./bash_functions.sh / - -# IPv6 disable flag for networks/devices that do not support it -ENV IPv6 True - -EXPOSE 53 53/udp -EXPOSE 67/udp -EXPOSE 80 -EXPOSE 443 - -ENV S6_LOGGING 0 -ENV S6_KEEP_ENV 1 -ENV S6_BEHAVIOUR_IF_STAGE2_FAILS 2 - -ENV ServerIP 0.0.0.0 -ENV FTL_CMD no-daemon -ENV DNSMASQ_USER root - -ENV VERSION v5.0 -ENV PATH /opt/pihole:${PATH} - -LABEL image="pihole/pihole:v5.0_amd64" -LABEL maintainer="adam@diginc.us" -LABEL url="https://www.github.com/pi-hole/docker-pi-hole" - -HEALTHCHECK CMD dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1 - -SHELL ["/bin/bash", "-c"] \ No newline at end of file diff --git a/Dockerfile_arm64 b/Dockerfile_arm64 deleted file mode 100644 index c3f6ad4..0000000 --- a/Dockerfile_arm64 +++ /dev/null @@ -1,49 +0,0 @@ -FROM multiarch/debian-debootstrap:arm64-stretch-slim - -ENV ARCH arm64 -ENV S6OVERLAY_RELEASE https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz - -COPY install.sh /usr/local/bin/install.sh -COPY VERSION /etc/docker-pi-hole-version -ENV PIHOLE_INSTALL /root/ph_install.sh - -RUN bash -ex install.sh 2>&1 && \ - rm -rf /var/cache/apt/archives /var/lib/apt/lists/* - -ENTRYPOINT [ "/s6-init" ] - -ADD s6/debian-root / -COPY s6/service /usr/local/bin/service - -# php config start passes special ENVs into -ENV PHP_ENV_CONFIG '/etc/lighttpd/conf-enabled/15-fastcgi-php.conf' -ENV PHP_ERROR_LOG '/var/log/lighttpd/error.log' -COPY ./start.sh / -COPY ./bash_functions.sh / - -# IPv6 disable flag for networks/devices that do not support it -ENV IPv6 True - -EXPOSE 53 53/udp -EXPOSE 67/udp -EXPOSE 80 -EXPOSE 443 - -ENV S6_LOGGING 0 -ENV S6_KEEP_ENV 1 -ENV S6_BEHAVIOUR_IF_STAGE2_FAILS 2 - -ENV ServerIP 0.0.0.0 -ENV FTL_CMD no-daemon -ENV DNSMASQ_USER root - -ENV VERSION v5.0 -ENV PATH /opt/pihole:${PATH} - -LABEL image="pihole/pihole:v5.0_arm64" -LABEL maintainer="adam@diginc.us" -LABEL url="https://www.github.com/pi-hole/docker-pi-hole" - -HEALTHCHECK CMD dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1 - -SHELL ["/bin/bash", "-c"] \ No newline at end of file diff --git a/Dockerfile_armel b/Dockerfile_armel deleted file mode 100644 index dc35b3a..0000000 --- a/Dockerfile_armel +++ /dev/null @@ -1,49 +0,0 @@ -FROM multiarch/debian-debootstrap:armel-stretch-slim - -ENV ARCH armel -ENV S6OVERLAY_RELEASE https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-arm.tar.gz - -COPY install.sh /usr/local/bin/install.sh -COPY VERSION /etc/docker-pi-hole-version -ENV PIHOLE_INSTALL /root/ph_install.sh - -RUN bash -ex install.sh 2>&1 && \ - rm -rf /var/cache/apt/archives /var/lib/apt/lists/* - -ENTRYPOINT [ "/s6-init" ] - -ADD s6/debian-root / -COPY s6/service /usr/local/bin/service - -# php config start passes special ENVs into -ENV PHP_ENV_CONFIG '/etc/lighttpd/conf-enabled/15-fastcgi-php.conf' -ENV PHP_ERROR_LOG '/var/log/lighttpd/error.log' -COPY ./start.sh / -COPY ./bash_functions.sh / - -# IPv6 disable flag for networks/devices that do not support it -ENV IPv6 True - -EXPOSE 53 53/udp -EXPOSE 67/udp -EXPOSE 80 -EXPOSE 443 - -ENV S6_LOGGING 0 -ENV S6_KEEP_ENV 1 -ENV S6_BEHAVIOUR_IF_STAGE2_FAILS 2 - -ENV ServerIP 0.0.0.0 -ENV FTL_CMD no-daemon -ENV DNSMASQ_USER root - -ENV VERSION v5.0 -ENV PATH /opt/pihole:${PATH} - -LABEL image="pihole/pihole:v5.0_armel" -LABEL maintainer="adam@diginc.us" -LABEL url="https://www.github.com/pi-hole/docker-pi-hole" - -HEALTHCHECK CMD dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1 - -SHELL ["/bin/bash", "-c"] \ No newline at end of file diff --git a/Dockerfile_armhf b/Dockerfile_armhf deleted file mode 100644 index 1862462..0000000 --- a/Dockerfile_armhf +++ /dev/null @@ -1,49 +0,0 @@ -FROM multiarch/debian-debootstrap:armhf-stretch-slim - -ENV ARCH armhf -ENV S6OVERLAY_RELEASE https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-arm.tar.gz - -COPY install.sh /usr/local/bin/install.sh -COPY VERSION /etc/docker-pi-hole-version -ENV PIHOLE_INSTALL /root/ph_install.sh - -RUN bash -ex install.sh 2>&1 && \ - rm -rf /var/cache/apt/archives /var/lib/apt/lists/* - -ENTRYPOINT [ "/s6-init" ] - -ADD s6/debian-root / -COPY s6/service /usr/local/bin/service - -# php config start passes special ENVs into -ENV PHP_ENV_CONFIG '/etc/lighttpd/conf-enabled/15-fastcgi-php.conf' -ENV PHP_ERROR_LOG '/var/log/lighttpd/error.log' -COPY ./start.sh / -COPY ./bash_functions.sh / - -# IPv6 disable flag for networks/devices that do not support it -ENV IPv6 True - -EXPOSE 53 53/udp -EXPOSE 67/udp -EXPOSE 80 -EXPOSE 443 - -ENV S6_LOGGING 0 -ENV S6_KEEP_ENV 1 -ENV S6_BEHAVIOUR_IF_STAGE2_FAILS 2 - -ENV ServerIP 0.0.0.0 -ENV FTL_CMD no-daemon -ENV DNSMASQ_USER root - -ENV VERSION v5.0 -ENV PATH /opt/pihole:${PATH} - -LABEL image="pihole/pihole:v5.0_armhf" -LABEL maintainer="adam@diginc.us" -LABEL url="https://www.github.com/pi-hole/docker-pi-hole" - -HEALTHCHECK CMD dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1 - -SHELL ["/bin/bash", "-c"] \ No newline at end of file diff --git a/README.md b/README.md index cd86e54..31183ce 100644 --- a/README.md +++ b/README.md @@ -180,7 +180,7 @@ The primary docker tags / versions are explained in the following table. [Click | tag | architecture | description | Dockerfile | | --- | ------------ | ----------- | ---------- | -| `latest` | auto detect | x86, arm, or arm64 container, docker auto detects your architecture. | [Dockerfile](https://github.com/pi-hole/docker-pi-hole/blob/master/Dockerfile_amd64) | +| `latest` | auto detect | x86, arm, or arm64 container, docker auto detects your architecture. | [Dockerfile](https://github.com/pi-hole/docker-pi-hole/blob/master/Dockerfile) | | `v4.0.0-1` | auto detect | Versioned tags, if you want to pin against a specific version, use one of these | | | `v4.0.0-1_` | based on tag | Specific architectures tags | | | `dev` | auto detect | like latest tag, but for the development branch (pushed occasionally) | | diff --git a/build.yml b/build.yml new file mode 100644 index 0000000..ab78653 --- /dev/null +++ b/build.yml @@ -0,0 +1,57 @@ +# Docker Compose build file: docker-compose -f build.yml build +version: "3.7" + +x-common-args: &common-args + PIHOLE_VERSION: ${PIHOLE_VERSION} + NAME: pihole/pihole + MAINTAINER: adam@diginc.us + S6_VERSION: v1.22.1.0 + PHP_ENV_CONFIG: /etc/lighttpd/conf-enabled/15-fastcgi-php.conf + PHP_ERROR_LOG: /var/log/lighttpd/error.log + + +services: + amd64: + image: pihole:${PIHOLE_VERSION}-amd64 + build: + context: . + cache_from: + - pihole/pihole:${PIHOLE_VERSION}-amd64 + args: + <<: *common-args + PIHOLE_BASE: pihole/debian-base:latest + PIHOLE_ARCH: amd64 + S6_ARCH: amd64 + armel: + image: pihole:${PIHOLE_VERSION}-armel + build: + context: . + cache_from: + - pihole/pihole:${PIHOLE_VERSION}-armel + args: + <<: *common-args + PIHOLE_BASE: multiarch/debian-debootstrap:armel-stretch-slim + PIHOLE_ARCH: armel + S6_ARCH: arm + armhf: + image: pihole:${PIHOLE_VERSION}-armhf + build: + context: . + cache_from: + - pihole/pihole:${PIHOLE_VERSION}-armhf + args: + <<: *common-args + PIHOLE_BASE: multiarch/debian-debootstrap:armhf-stretch-slim + PIHOLE_ARCH: arm + S6_ARCH: arm + arm64: + image: pihole:${PIHOLE_VERSION}-arm64 + build: + context: . + cache_from: + - pihole/pihole:${PIHOLE_VERSION}-arm64 + args: + <<: *common-args + PIHOLE_BASE: multiarch/debian-debootstrap:arm64-stretch-slim + PIHOLE_ARCH: arm64 + S6_ARCH: aarch64 \ No newline at end of file From d7fdb188ed38bd2cbe820ebb8289b11808bf82d4 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Tue, 26 May 2020 10:45:48 -0500 Subject: [PATCH 05/24] remove resolv conf checks Signed-off-by: Adam Hill --- README.md | 3 --- bash_functions.sh | 25 ------------------------- docker-compose-jwilder-proxy.yml | 5 +---- docker-compose-traefik-proxy.md | 3 --- docker-compose.yml | 3 --- docker_run.sh | 1 - start.sh | 2 -- test/conftest.py | 12 ++++-------- test/test_bash_functions.py | 17 ----------------- 9 files changed, 5 insertions(+), 66 deletions(-) diff --git a/README.md b/README.md index 81d9f5a..c756752 100644 --- a/README.md +++ b/README.md @@ -30,9 +30,6 @@ services: volumes: - './etc-pihole/:/etc/pihole/' - './etc-dnsmasq.d/:/etc/dnsmasq.d/' - dns: - - 127.0.0.1 - - 1.1.1.1 # Recommended but not required (DHCP needs NET_ADMIN) # https://github.com/pi-hole/docker-pi-hole#note-on-capabilities cap_add: diff --git a/bash_functions.sh b/bash_functions.sh index f2567dc..b1d00ff 100644 --- a/bash_functions.sh +++ b/bash_functions.sh @@ -2,31 +2,6 @@ # Some of the bash_functions use variables these core pi-hole/web scripts . /opt/pihole/webpage.sh -docker_checks() { - warn_msg='WARNING Misconfigured DNS in /etc/resolv.conf' - ns_count="$(grep -c nameserver /etc/resolv.conf)" - ns_primary="$(grep nameserver /etc/resolv.conf | head -1)" - ns_primary="${ns_primary/nameserver /}" - warned=false - - if [ "$ns_count" -lt 2 ] ; then - echo "$warn_msg: Two DNS servers are recommended, 127.0.0.1 and any backup server" - warned=true - fi - - if [ "$ns_primary" != "127.0.0.1" ] ; then - echo "$warn_msg: Primary DNS should be 127.0.0.1 (found ${ns_primary})" - warned=true - fi - - if ! $warned ; then - echo "OK: Checks passed for /etc/resolv.conf DNS servers" - fi - - echo - cat /etc/resolv.conf -} - fix_capabilities() { setcap CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_NET_ADMIN+ei $(which pihole-FTL) || ret=$? diff --git a/docker-compose-jwilder-proxy.yml b/docker-compose-jwilder-proxy.yml index 47ab2f7..ee9c9e4 100644 --- a/docker-compose-jwilder-proxy.yml +++ b/docker-compose-jwilder-proxy.yml @@ -15,9 +15,6 @@ services: pihole: image: pihole/pihole:latest - dns: - - 127.0.0.1 - - 1.1.1.1 ports: - '53:53/tcp' - '53:53/udp' @@ -60,4 +57,4 @@ services: # PROXY_LOCATION: ghost # VIRTUAL_HOST: ghost.yourDomain.lan # VIRTUAL_PORT: 2368 -# restart: always \ No newline at end of file +# restart: always diff --git a/docker-compose-traefik-proxy.md b/docker-compose-traefik-proxy.md index 87335c5..8e3e6f3 100644 --- a/docker-compose-traefik-proxy.md +++ b/docker-compose-traefik-proxy.md @@ -44,9 +44,6 @@ services: domainname: homedomain.lan image: pihole/pihole:latest - dns: - - 127.0.0.1 - - 1.1.1.1 ports: - '0.0.0.0:53:53/tcp' - '0.0.0.0:53:53/udp' diff --git a/docker-compose.yml b/docker-compose.yml index 06cd155..031eb49 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -22,9 +22,6 @@ services: - './etc-dnsmasq.d/:/etc/dnsmasq.d/' # run `touch ./var-log/pihole.log` first unless you like errors # - './var-log/pihole.log:/var/log/pihole.log' - dns: - - 127.0.0.1 - - 1.1.1.1 # Recommended but not required (DHCP needs NET_ADMIN) # https://github.com/pi-hole/docker-pi-hole#note-on-capabilities cap_add: diff --git a/docker_run.sh b/docker_run.sh index e14d33d..ab733cc 100755 --- a/docker_run.sh +++ b/docker_run.sh @@ -10,7 +10,6 @@ docker run -d \ -e TZ="America/Chicago" \ -v "$(pwd)/etc-pihole/:/etc/pihole/" \ -v "$(pwd)/etc-dnsmasq.d/:/etc/dnsmasq.d/" \ - --dns=127.0.0.1 --dns=1.1.1.1 \ --restart=unless-stopped \ --hostname pi.hole \ -e VIRTUAL_HOST="pi.hole" \ diff --git a/start.sh b/start.sh index 815da70..6f8f7d8 100755 --- a/start.sh +++ b/start.sh @@ -32,8 +32,6 @@ PH_TEST=true . $PIHOLE_INSTALL echo " ::: Starting docker specific checks & setup for docker pihole/pihole" -docker_checks - # TODO: #if [ ! -f /.piholeFirstBoot ] ; then # echo " ::: Not first container startup so not running docker's setup, re-create container to run setup again" diff --git a/test/conftest.py b/test/conftest.py index 546fde6..12cdb99 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -14,10 +14,6 @@ with open('{}/VERSION'.format(dotdot), 'r') as v: raw_version = v.read().strip() __version__ = raw_version.replace('release/', 'release-') -@pytest.fixture() -def args_dns(): - return '--dns 127.0.0.1 --dns 1.1.1.1' - @pytest.fixture() def args_volumes(): return '-v /dev/null:/etc/pihole/adlists.list' @@ -27,8 +23,8 @@ def args_env(): return '-e ServerIP="127.0.0.1"' @pytest.fixture() -def args(args_dns, args_volumes, args_env): - return "{} {} {}".format(args_dns, args_volumes, args_env) +def args(args_volumes, args_env): + return "{} {}".format(args_volumes, args_env) @pytest.fixture() def test_args(): @@ -123,8 +119,8 @@ def persist_args_env(): return '-e ServerIP="127.0.0.1"' @pytest.fixture(scope='module') -def persist_args(persist_args_dns, persist_args_volumes, persist_args_env): - return "{} {} {}".format(args_dns, args_volumes, args_env) +def persist_args(persist_args_volumes, persist_args_env): + return "{} {}".format(persist_args_volumes, persist_args_env) @pytest.fixture(scope='module') def persist_test_args(): diff --git a/test/test_bash_functions.py b/test/test_bash_functions.py index d917a50..8773d41 100644 --- a/test/test_bash_functions.py +++ b/test/test_bash_functions.py @@ -179,20 +179,3 @@ def test_webPassword_pre_existing_trumps_all_envs(Docker, args_env, test_args): assert '::: Pre existing WEBPASSWORD found' in function.stdout assert Docker.run('grep -q \'{}\' {}'.format('WEBPASSWORD=volumepass', '/etc/pihole/setupVars.conf')).rc == 0 - - -@pytest.mark.skip('broke, needs investigation in v5.0 beta') -@pytest.mark.parametrize('args_dns, expected_stdout', [ - # No DNS passed will vary by the host this is ran on, bad idea for a test - #('', 'WARNING Misconfigured DNS in /etc/resolv.conf: Primary DNS should be 127.0.0.1'), - ('--dns 1.1.1.1', 'WARNING Misconfigured DNS in /etc/resolv.conf: Two DNS servers are recommended, 127.0.0.1 and any backup server\n' - 'WARNING Misconfigured DNS in /etc/resolv.conf: Primary DNS should be 127.0.0.1 (found 1.1.1.1)'), - ('--dns 127.0.0.1', 'WARNING Misconfigured DNS in /etc/resolv.conf: Two DNS servers are recommended, 127.0.0.1 and any backup server'), - ('--dns 1.1.1.1 --dns 127.0.0.1', 'WARNING Misconfigured DNS in /etc/resolv.conf: Primary DNS should be 127.0.0.1 (found 1.1.1.1)'), - ('--dns 127.0.0.1 --dns 1.1.1.1', 'OK: Checks passed for /etc/resolv.conf DNS servers'), -]) -def test_docker_checks_for_resolvconf_misconfiguration(Docker, args_dns, expected_stdout): - ''' The container checks for misconfigured resolv.conf ''' - function = Docker.run('. /bash_functions.sh ; eval `grep docker_checks /start.sh`') - print(function.stdout) - assert expected_stdout in function.stdout From caad778b0c4bfc36129a9d67a42fb557c3440962 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Wed, 27 May 2020 00:10:40 -0500 Subject: [PATCH 06/24] install docker-compose; Signed-off-by: Adam Hill --- circle-test.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/circle-test.sh b/circle-test.sh index e0a8b93..ede2285 100755 --- a/circle-test.sh +++ b/circle-test.sh @@ -2,6 +2,10 @@ set -ex # Circle CI Job for single architecture +if ! command -v docker-compose; then + curl -L https://github.com/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose + chmod +x /usr/local/bin/docker-compose +fi # setup qemu/variables docker run --rm --privileged multiarch/qemu-user-static:register --reset > /dev/null From 92c19555bd4b439335c0d01f23ca05fe69c1c7de Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Wed, 27 May 2020 00:15:53 -0500 Subject: [PATCH 07/24] install docker-compose on circle in build image Signed-off-by: Adam Hill --- Pipfile | 1 + Pipfile.lock | 54 +++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 50 insertions(+), 5 deletions(-) diff --git a/Pipfile b/Pipfile index 4aadaea..3b8e586 100644 --- a/Pipfile +++ b/Pipfile @@ -58,6 +58,7 @@ Jinja2 = "==2.10.3" MarkupSafe = "==1.1.1" PyYAML = "==5.2" websocket_client = "==0.57.0" +docker-compose = "*" [requires] python_version = "3.8" diff --git a/Pipfile.lock b/Pipfile.lock index d6cfad0..9e089be 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "ee7705112b315cad899e08bd6eac8f47e9a200a0d47a1920cc192995b79f8673" + "sha256": "ae4ddd6be37a69be0783db52c8c18b6edbc831979af084ac329ed54b4fe7e72c" }, "pipfile-spec": 6, "requires": { @@ -223,6 +223,9 @@ "version": "==2.8" }, "docker": { + "extras": [ + "ssh" + ], "hashes": [ "sha256:6e06c5e70ba4fad73e35f00c55a895a448398f3ada7faae072e2bb01348bafc1", "sha256:8f93775b8bdae3a2df6bc9a5312cce564cade58d6555f2c2570165a1270cd8a7" @@ -230,6 +233,14 @@ "index": "pypi", "version": "==4.1.0" }, + "docker-compose": { + "hashes": [ + "sha256:7a2eb6d8173fdf408e505e6f7d497ac0b777388719542be9e49a0efd477a50c6", + "sha256:9d33520ae976f524968a64226516ec631dce09fba0974ce5366ad403e203eb5d" + ], + "index": "pypi", + "version": "==1.25.5" + }, "dockerpty": { "hashes": [ "sha256:69a9d69d573a0daa31bcd1c0774eeed5c15c295fe719c61aca550ed1393156ce" @@ -324,13 +335,16 @@ "sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161", "sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235", "sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5", + "sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42", "sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff", "sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b", "sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1", "sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e", "sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183", "sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66", + "sha256:596510de112c685489095da617b5bcbbac7dd6384aeebeda4df6025d0256a81b", "sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1", + "sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15", "sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1", "sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e", "sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b", @@ -347,7 +361,9 @@ "sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6", "sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f", "sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f", - "sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7" + "sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2", + "sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7", + "sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be" ], "index": "pypi", "version": "==1.1.1" @@ -363,10 +379,17 @@ }, "packaging": { "hashes": [ - "sha256:aec3fdbb8bc9e4bb65f0634b9f551ced63983a529d6a8931817d52fdd0816ddb", - "sha256:fe1d8331dfa7cc0a883b49d75fc76380b2ab2734b220fbb87d774e4fd4b851f8" + "sha256:4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8", + "sha256:998416ba6962ae7fbd6596850b80e17859a5753ba17c32284f67bfff33784181" ], - "version": "==20.0" + "version": "==20.4" + }, + "paramiko": { + "hashes": [ + "sha256:920492895db8013f6cc0179293147f830b8c7b21fdfc839b6bad760c27459d9f", + "sha256:9c980875fa4d2cb751604664e9a2d0f69096643f5be4db1b99599fe114a97b2f" + ], + "version": "==2.7.1" }, "pathlib2": { "hashes": [ @@ -399,6 +422,27 @@ "index": "pypi", "version": "==2.19" }, + "pynacl": { + "hashes": [ + "sha256:06cbb4d9b2c4bd3c8dc0d267416aaed79906e7b33f114ddbf0911969794b1cc4", + "sha256:11335f09060af52c97137d4ac54285bcb7df0cef29014a1a4efe64ac065434c4", + "sha256:2fe0fc5a2480361dcaf4e6e7cea00e078fcda07ba45f811b167e3f99e8cff574", + "sha256:30f9b96db44e09b3304f9ea95079b1b7316b2b4f3744fe3aaecccd95d547063d", + "sha256:511d269ee845037b95c9781aa702f90ccc36036f95d0f31373a6a79bd8242e25", + "sha256:537a7ccbea22905a0ab36ea58577b39d1fa9b1884869d173b5cf111f006f689f", + "sha256:54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505", + "sha256:757250ddb3bff1eecd7e41e65f7f833a8405fede0194319f87899690624f2122", + "sha256:7757ae33dae81c300487591c68790dfb5145c7d03324000433d9a2c141f82af7", + "sha256:7c6092102219f59ff29788860ccb021e80fffd953920c4a8653889c029b2d420", + "sha256:8122ba5f2a2169ca5da936b2e5a511740ffb73979381b4229d9188f6dcb22f1f", + "sha256:9c4a7ea4fb81536c1b1f5cc44d54a296f96ae78c1ebd2311bd0b60be45a48d96", + "sha256:cd401ccbc2a249a47a3a1724c2918fcd04be1f7b54eb2a5a71ff915db0ac51c6", + "sha256:d452a6746f0a7e11121e64625109bc4468fc3100452817001dbe018bb8b08514", + "sha256:ea6841bc3a76fa4942ce00f3bda7d436fda21e2d91602b9e21b7ca9ecab8f3ff", + "sha256:f8851ab9041756003119368c1e6cd0b9c631f46d686b3904b18c0139f4419f80" + ], + "version": "==1.4.0" + }, "pyparsing": { "hashes": [ "sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f", From 04ab6d447fb96cd9f674b855c1d21781d23c24fb Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Wed, 27 May 2020 00:42:32 -0500 Subject: [PATCH 08/24] docker-compose pip is garbage to install Signed-off-by: Adam Hill --- Dockerfile_build | 3 ++- Pipfile | 1 - Pipfile.lock | 41 +---------------------------------------- circle-test.sh | 4 ---- 4 files changed, 3 insertions(+), 46 deletions(-) diff --git a/Dockerfile_build b/Dockerfile_build index 550a050..ed7a178 100644 --- a/Dockerfile_build +++ b/Dockerfile_build @@ -6,7 +6,8 @@ RUN apk --update add python3 python3-dev curl gcc make \ musl-dev libffi-dev openssl-dev ${packages} \ && rm -rf /var/cache/apk/* \ && pip3 install -U pip pipenv - +RUN curl -L https://github.com/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose && \ + chmod +x /usr/local/bin/docker-compose # -v "$(pwd):/$(pwd)" -w "$(pwd)" to prevent nested docker path confusion COPY ./Dockerfile.sh /usr/local/bin/ diff --git a/Pipfile b/Pipfile index 3b8e586..4aadaea 100644 --- a/Pipfile +++ b/Pipfile @@ -58,7 +58,6 @@ Jinja2 = "==2.10.3" MarkupSafe = "==1.1.1" PyYAML = "==5.2" websocket_client = "==0.57.0" -docker-compose = "*" [requires] python_version = "3.8" diff --git a/Pipfile.lock b/Pipfile.lock index 9e089be..80ca574 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "ae4ddd6be37a69be0783db52c8c18b6edbc831979af084ac329ed54b4fe7e72c" + "sha256": "ee7705112b315cad899e08bd6eac8f47e9a200a0d47a1920cc192995b79f8673" }, "pipfile-spec": 6, "requires": { @@ -223,9 +223,6 @@ "version": "==2.8" }, "docker": { - "extras": [ - "ssh" - ], "hashes": [ "sha256:6e06c5e70ba4fad73e35f00c55a895a448398f3ada7faae072e2bb01348bafc1", "sha256:8f93775b8bdae3a2df6bc9a5312cce564cade58d6555f2c2570165a1270cd8a7" @@ -233,14 +230,6 @@ "index": "pypi", "version": "==4.1.0" }, - "docker-compose": { - "hashes": [ - "sha256:7a2eb6d8173fdf408e505e6f7d497ac0b777388719542be9e49a0efd477a50c6", - "sha256:9d33520ae976f524968a64226516ec631dce09fba0974ce5366ad403e203eb5d" - ], - "index": "pypi", - "version": "==1.25.5" - }, "dockerpty": { "hashes": [ "sha256:69a9d69d573a0daa31bcd1c0774eeed5c15c295fe719c61aca550ed1393156ce" @@ -384,13 +373,6 @@ ], "version": "==20.4" }, - "paramiko": { - "hashes": [ - "sha256:920492895db8013f6cc0179293147f830b8c7b21fdfc839b6bad760c27459d9f", - "sha256:9c980875fa4d2cb751604664e9a2d0f69096643f5be4db1b99599fe114a97b2f" - ], - "version": "==2.7.1" - }, "pathlib2": { "hashes": [ "sha256:0ec8205a157c80d7acc301c0b18fbd5d44fe655968f5d947b6ecef5290fc35db", @@ -422,27 +404,6 @@ "index": "pypi", "version": "==2.19" }, - "pynacl": { - "hashes": [ - "sha256:06cbb4d9b2c4bd3c8dc0d267416aaed79906e7b33f114ddbf0911969794b1cc4", - "sha256:11335f09060af52c97137d4ac54285bcb7df0cef29014a1a4efe64ac065434c4", - "sha256:2fe0fc5a2480361dcaf4e6e7cea00e078fcda07ba45f811b167e3f99e8cff574", - "sha256:30f9b96db44e09b3304f9ea95079b1b7316b2b4f3744fe3aaecccd95d547063d", - "sha256:511d269ee845037b95c9781aa702f90ccc36036f95d0f31373a6a79bd8242e25", - "sha256:537a7ccbea22905a0ab36ea58577b39d1fa9b1884869d173b5cf111f006f689f", - "sha256:54e9a2c849c742006516ad56a88f5c74bf2ce92c9f67435187c3c5953b346505", - "sha256:757250ddb3bff1eecd7e41e65f7f833a8405fede0194319f87899690624f2122", - "sha256:7757ae33dae81c300487591c68790dfb5145c7d03324000433d9a2c141f82af7", - "sha256:7c6092102219f59ff29788860ccb021e80fffd953920c4a8653889c029b2d420", - "sha256:8122ba5f2a2169ca5da936b2e5a511740ffb73979381b4229d9188f6dcb22f1f", - "sha256:9c4a7ea4fb81536c1b1f5cc44d54a296f96ae78c1ebd2311bd0b60be45a48d96", - "sha256:cd401ccbc2a249a47a3a1724c2918fcd04be1f7b54eb2a5a71ff915db0ac51c6", - "sha256:d452a6746f0a7e11121e64625109bc4468fc3100452817001dbe018bb8b08514", - "sha256:ea6841bc3a76fa4942ce00f3bda7d436fda21e2d91602b9e21b7ca9ecab8f3ff", - "sha256:f8851ab9041756003119368c1e6cd0b9c631f46d686b3904b18c0139f4419f80" - ], - "version": "==1.4.0" - }, "pyparsing": { "hashes": [ "sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f", diff --git a/circle-test.sh b/circle-test.sh index ede2285..e0a8b93 100755 --- a/circle-test.sh +++ b/circle-test.sh @@ -2,10 +2,6 @@ set -ex # Circle CI Job for single architecture -if ! command -v docker-compose; then - curl -L https://github.com/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose - chmod +x /usr/local/bin/docker-compose -fi # setup qemu/variables docker run --rm --privileged multiarch/qemu-user-static:register --reset > /dev/null From e621670df4ba4940cff8f6bd03763d6cc0cf6280 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Wed, 27 May 2020 19:26:53 -0500 Subject: [PATCH 09/24] debian, our one true savior from bad docker build base images Signed-off-by: Adam Hill --- Dockerfile_build | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/Dockerfile_build b/Dockerfile_build index ed7a178..5327d8d 100644 --- a/Dockerfile_build +++ b/Dockerfile_build @@ -1,10 +1,13 @@ -FROM docker:latest +FROM python:buster + +# Only works for docker CLIENT (bind mounted socket) +COPY --from=docker:18.09.3 /usr/local/bin/docker /usr/local/bin/ # Based on https://github.com/Ilhicas/alpine-pipenv ARG packages -RUN apk --update add python3 python3-dev curl gcc make \ - musl-dev libffi-dev openssl-dev ${packages} \ - && rm -rf /var/cache/apk/* \ +RUN apt-get update && \ + apt-get install -y python3-dev curl gcc make \ + libffi-dev libssl-dev ${packages} \ && pip3 install -U pip pipenv RUN curl -L https://github.com/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose && \ chmod +x /usr/local/bin/docker-compose @@ -15,7 +18,7 @@ COPY Pipfile* /root/ WORKDIR /root RUN pipenv install --system \ - && sed -i 's|/bin/sh|/bin/bash|g' /usr/lib/python3.8/site-packages/testinfra/backend/docker.py + && sed -i 's|/bin/sh|/bin/bash|g' /usr/local/lib/python3.8/site-packages/testinfra/backend/docker.py RUN echo "set -ex && Dockerfile.sh && \$@" > /usr/local/bin/entrypoint.sh From 0da5afcd610ef10d8ce596acb3261382aaf2c181 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Thu, 28 May 2020 08:15:01 -0500 Subject: [PATCH 10/24] print images in build image Signed-off-by: Adam Hill --- Dockerfile.sh | 2 ++ circle-test.sh | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile.sh b/Dockerfile.sh index e48a64f..3338491 100755 --- a/Dockerfile.sh +++ b/Dockerfile.sh @@ -3,6 +3,8 @@ set -eux ./Dockerfile.py -v --arch="${ARCH}" --hub_tag="${ARCH_IMAGE}" +docker images + # TODO: Add junitxml output and have circleci consume it # 2 parallel max b/c race condition with docker fixture (I think?) py.test -vv -n 2 -k "${ARCH}" ./test/ diff --git a/circle-test.sh b/circle-test.sh index e0a8b93..38ff736 100755 --- a/circle-test.sh +++ b/circle-test.sh @@ -22,8 +22,6 @@ docker run --rm \ --env-file /tmp/env \ $enter image_pipenv -docker images - test -z "${CIRCLE_PROJECT_REPONAME}" && exit 0 # The rest is circle-ci only echo $DOCKERHUB_PASS | docker login --username=$DOCKERHUB_USER --password-stdin From 72f5730d5643904b544e2eaff4747e44f416e72c Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Thu, 28 May 2020 08:25:49 -0500 Subject: [PATCH 11/24] have tests use - not _ in tag Signed-off-by: Adam Hill --- test/conftest.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/conftest.py b/test/conftest.py index 546fde6..6ff8b44 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -85,7 +85,7 @@ def version(): @pytest.fixture() def tag(version, arch): - return '{}_{}'.format(version, arch) + return '{}-{}'.format(version, arch) @pytest.fixture def webserver(tag): From 55ee5d0e60499f4f0cb3e70d23448d46652c8410 Mon Sep 17 00:00:00 2001 From: Daniel Date: Thu, 28 May 2020 23:00:55 -0400 Subject: [PATCH 12/24] Run circle-test.sh in Github Actions on push or PR Signed-off-by: Daniel --- .github/workflows/test.yaml | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 .github/workflows/test.yaml diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml new file mode 100644 index 0000000..69f2cc3 --- /dev/null +++ b/.github/workflows/test.yaml @@ -0,0 +1,26 @@ +name: Tests +on: + pull_request: + push: + +jobs: + build: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + ARCH: + - amd64 + # https://github.com/pi-hole/docker-pi-hole/issues/587#issuecomment-617180631 + #- armel + - armhf + - arm64 + env: + ARCH: ${{matrix.ARCH}} + steps: + - name: Checkout Repo + uses: actions/checkout@v2 + - name: Run Circle Test + run: | + echo "Building ${ARCH}" + ./circle-test.sh \ No newline at end of file From ecb0aaf805d693fb6f1da2d57c908111b3b556e8 Mon Sep 17 00:00:00 2001 From: Sandro Date: Sat, 30 May 2020 06:56:32 +0200 Subject: [PATCH 13/24] Don't bind to 127.0.0.1 when listening on all ports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #495 Signed-off-by: Sandro Jäckel --- bash_functions.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bash_functions.sh b/bash_functions.sh index f2567dc..3117db7 100644 --- a/bash_functions.sh +++ b/bash_functions.sh @@ -237,7 +237,7 @@ setup_dnsmasq_hostnames() { setup_lighttpd_bind() { local serverip="$1" # if using '--net=host' only bind lighttpd on $ServerIP and localhost - if grep -q "docker" /proc/net/dev ; then #docker (docker0 by default) should only be present on the host system + if grep -q "docker" /proc/net/dev && [[ $serverip != 0.0.0.0 ]]; then #docker (docker0 by default) should only be present on the host system if ! grep -q "server.bind" /etc/lighttpd/lighttpd.conf ; then # if the declaration is already there, don't add it again sed -i -E "s/server\.port\s+\=\s+([0-9]+)/server.bind\t\t = \"${serverip}\"\nserver.port\t\t = \1\n"\$SERVER"\[\"socket\"\] == \"127\.0\.0\.1:\1\" \{\}/" /etc/lighttpd/lighttpd.conf fi From 5b75953d9180dfd7dbb8199188c9ff0ba82a3e28 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Sat, 30 May 2020 18:01:43 -0500 Subject: [PATCH 14/24] New bash integration style test for volume data Signed-off-by: Adam Hill --- test/test_volume_data.sh | 85 ++++++++++++++++++++++++++++++++++++++++ test/test_volumes.py | 25 ++++++++++++ 2 files changed, 110 insertions(+) create mode 100755 test/test_volume_data.sh create mode 100644 test/test_volumes.py diff --git a/test/test_volume_data.sh b/test/test_volume_data.sh new file mode 100755 index 0000000..97a71ce --- /dev/null +++ b/test/test_volume_data.sh @@ -0,0 +1,85 @@ +#!/bin/bash +set -ex +# Trying something different from the python test, this is a big integration test in bash +# Tests multiple volume settings and how they are impacted by the complete startup scripts + restart/re-creation of container +# Maybe a bit easier to read the workflow/debug in bash than python for others? +# This workflow is VERY similar to python's tests, but in bash so not object-oriented/pytest fixture based + +# Debug can be added anywhere to check current state mid-test +RED='\033[0;31m' +NC='\033[0m' # No Color +debug() { + sudo grep -r . "$VOL_PH" + sudo grep -r . "$VOL_DM" +} +# Cleanup at the end, print debug on fail +cleanup() { + retcode=$? + { set +x; } 2>/dev/null + if [ $retcode != 0 ] ; then + printf "${RED}ERROR / FAILURE${NC} - printing all volume info" + debug + fi + docker rm -f $CONTAINER + sudo rm -rf $VOLUMES + exit $retcode +} +trap "cleanup" INT TERM EXIT + + +# VOLUME TESTS + +# Given... +IMAGE="pihole:v5.0-amd64" # The latest build test image (generic, non release/branch tag) +VOLUMES="$(mktemp -d)" # A fresh volume directory +VOL_PH="$VOLUMES/pihole" +VOL_DM="$VOLUMES/dnsmasq.d" +tty -s && TTY='-t' || TTY='' + +echo "Testing $IMAGE with volumes base path $VOLUMES" + +# When +# Running stock+empty volumes (no ports to avoid conflicts) +CONTAINER="$( + docker run -d \ + -v "$VOL_PH:/etc/pihole/" \ + -v "$VOL_DM:/etc/dnsmasq.d/" \ + -v "/dev/null:/etc/pihole/adlists.list" \ + --entrypoint='' \ + $IMAGE \ + tail -f /dev/null +)" # container backgrounded for multipiple operations over time + +EXEC() { + # Must quote for complex commands + docker exec $TTY $CONTAINER bash -c "$@" +} +EXEC /start.sh + +# Then default are present +grep "PIHOLE_DNS_1=8.8.8.8" "$VOL_PH/setupVars.conf" +grep "PIHOLE_DNS_2=8.8.4.4" "$VOL_PH/setupVars.conf" +grep "IPV4_ADDRESS=0.0.0.0" "$VOL_PH/setupVars.conf" +grep -E "WEBPASSWORD=.+" "$VOL_PH/setupVars.conf" + +# Given the settings are manually changed (not good settings, just for testing changes) +EXEC 'pihole -a setdns 127.1.1.1,127.2.2.2,127.3.3.3,127.4.4.4' +EXEC '. /opt/pihole/webpage.sh ; change_setting IPV4_ADDRESS 10.0.0.0' +EXEC 'pihole -a -p login' +assert_new_settings() { + grep "PIHOLE_DNS_1=127.1.1.1" "$VOL_PH/setupVars.conf" + grep "PIHOLE_DNS_2=127.2.2.2" "$VOL_PH/setupVars.conf" + grep "PIHOLE_DNS_3=127.3.3.3" "$VOL_PH/setupVars.conf" + grep "PIHOLE_DNS_4=127.4.4.4" "$VOL_PH/setupVars.conf" + grep "IPV4_ADDRESS=10.0.0.0" "$VOL_PH/setupVars.conf" + grep "WEBPASSWORD=6060d59351e8c2f48140f01b2c3f3b61652f396c53a5300ae239ebfbe7d5ff08" "$VOL_PH/setupVars.conf" + grep "server=127.1.1.1" $VOL_DM/01-pihole.conf + grep "server=127.2.2.2" $VOL_DM/01-pihole.conf +} +assert_new_settings + +# When Restarting +docker restart $CONTAINER + +# Then settings are still manual changed values +assert_new_settings diff --git a/test/test_volumes.py b/test/test_volumes.py new file mode 100644 index 0000000..d703bf1 --- /dev/null +++ b/test/test_volumes.py @@ -0,0 +1,25 @@ +import subprocess + +def run_and_stream_command_output(command, verbose=False): + print("Running", command) + build_env = os.environ.copy() + build_env['PIHOLE_VERSION'] = __version__ + build_result = subprocess.Popen(command.split(), env=build_env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + bufsize=1, universal_newlines=True) + if verbose: + while build_result.poll() is None: + for line in build_result.stdout: + print(line, end='') + build_result.wait() + if build_result.returncode != 0: + print(" ::: Error running".format(command)) + print(build_result.stderr) + +def test_volume_shell_script(arch): + # only one arch should be necessary + if arch == 'amd64': + run_and_stream_command_output('./test/test_volume_data.sh') + + +def test_fail(): + assert 1 == 2 From d832213e3c863f00b23e8510db1fd115b49076bb Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Tue, 2 Jun 2020 08:59:37 -0500 Subject: [PATCH 15/24] limit branches to be built Signed-off-by: Adam Hill --- .github/workflows/test.yaml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 69f2cc3..102482d 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -1,7 +1,12 @@ name: Tests on: - pull_request: push: + branches: + - master + - dev + - v* + - beta-v* + pull_request: jobs: build: @@ -23,4 +28,4 @@ jobs: - name: Run Circle Test run: | echo "Building ${ARCH}" - ./circle-test.sh \ No newline at end of file + ./circle-test.sh From 022b03971818b6609492b4cf535a6c96b547e2d2 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Tue, 2 Jun 2020 12:27:09 -0500 Subject: [PATCH 16/24] added remove + recreate container test Signed-off-by: Adam Hill --- test/test_volume_data.sh | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/test/test_volume_data.sh b/test/test_volume_data.sh index de854b6..1261ed0 100755 --- a/test/test_volume_data.sh +++ b/test/test_volume_data.sh @@ -54,10 +54,11 @@ CONTAINER="$( )" # container backgrounded for multipiple operations over time EXEC() { + local container="$1" # Must quote for complex commands - docker exec $TTY $CONTAINER bash -c "$@" + docker exec $TTY $container bash -c "$2" } -EXEC /start.sh +EXEC $CONTAINER /start.sh # run all the startup scripts # Then default are present grep "PIHOLE_DNS_1=8.8.8.8" "$VOL_PH/setupVars.conf" @@ -66,9 +67,9 @@ grep "IPV4_ADDRESS=0.0.0.0" "$VOL_PH/setupVars.conf" grep -E "WEBPASSWORD=.+" "$VOL_PH/setupVars.conf" # Given the settings are manually changed (not good settings, just for testing changes) -EXEC 'pihole -a setdns 127.1.1.1,127.2.2.2,127.3.3.3,127.4.4.4' -EXEC '. /opt/pihole/webpage.sh ; change_setting IPV4_ADDRESS 10.0.0.0' -EXEC 'pihole -a -p login' +EXEC $CONTAINER 'pihole -a setdns 127.1.1.1,127.2.2.2,127.3.3.3,127.4.4.4' +EXEC $CONTAINER '. /opt/pihole/webpage.sh ; change_setting IPV4_ADDRESS 10.0.0.0' +EXEC $CONTAINER 'pihole -a -p login' assert_new_settings() { grep "PIHOLE_DNS_1=127.1.1.1" "$VOL_PH/setupVars.conf" grep "PIHOLE_DNS_2=127.2.2.2" "$VOL_PH/setupVars.conf" @@ -83,6 +84,20 @@ assert_new_settings # When Restarting docker restart $CONTAINER +# Then settings are still manual changed values +assert_new_settings + +# When removing/re-creating the container +docker rm -f $CONTAINER +CONTAINER="$( + docker run -d \ + -v "$VOL_PH:/etc/pihole/" \ + -v "$VOL_DM:/etc/dnsmasq.d/" \ + -v "/dev/null:/etc/pihole/adlists.list" \ + --entrypoint='' \ + $IMAGE \ + tail -f /dev/null +)" # container backgrounded for multipiple operations over time # Then settings are still manual changed values assert_new_settings From 3b8227c29da6424c1e443ac71380e39be5dd41c9 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Wed, 3 Jun 2020 08:34:28 -0500 Subject: [PATCH 17/24] Make image a parameter to allow running with any image --- test/test_volume_data.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/test_volume_data.sh b/test/test_volume_data.sh index 1261ed0..2c87a63 100755 --- a/test/test_volume_data.sh +++ b/test/test_volume_data.sh @@ -33,8 +33,8 @@ trap "cleanup" INT TERM EXIT # VOLUME TESTS # Given... -IMAGE="pihole:v5.0-amd64" # The latest build test image (generic, non release/branch tag) -VOLUMES="$(mktemp -d)" # A fresh volume directory +IMAGE="${1:-pihole:v5.0-amd64}" # Default is latest build test image (generic, non release/branch tag) +VOLUMES="$(mktemp -d)" # A fresh volume directory VOL_PH="$VOLUMES/pihole" VOL_DM="$VOLUMES/dnsmasq.d" tty -s && TTY='-t' || TTY='' From 752d83aeaa5d2500c81d375c5ea4f3acd7fd329a Mon Sep 17 00:00:00 2001 From: Daniel Date: Thu, 4 Jun 2020 17:03:48 -0400 Subject: [PATCH 18/24] Introduce additional docker tags for the debian version. * Added new docker tag variations to specify the debian version ('stretch', and 'buster'). * Arch images are alway as specific as possible: pihole/pihole:master-amd64-stretch * Multiarch images have both the specific debian version tags as well as the generic non-debian tags: pihole/pihole:master-stretch & pihole/pihole:master * Currently, the non-specific tags point to the 'stretch' images. Eventaully it can be migrated to 'buster'. * Use GitHub actions to do the builds. Although the script names include 'gh-actions' to differentiate them from the 'circle' scripts, there is zero logic that is specific to Github (ie. no Github environment variables). * 'armhf:buster' & 'arm64:buster' has an issue with `ip route get`. I think the issue is related to 'qemu', but I'm not sure. Update the `validate_env` function to only use `ip route get` if `nc` reports something strange. Signed-off-by: Daniel --- .github/workflows/test-and-build.yaml | 62 +++++++++++++++++++++++ .github/workflows/test.yaml | 31 ------------ .gitignore | 1 + Dockerfile.py | 70 ++++++++++++------------- Dockerfile.sh | 11 ++-- Dockerfile_build | 7 +-- README.md | 17 ++++--- TESTING.md | 2 +- bash_functions.sh | 4 +- build.yml | 26 ++++------ gh-actions-deploy.sh | 73 +++++++++++++++++++++++++++ gh-actions-test.sh | 31 ++++++++++++ gh-actions-vars.sh | 53 +++++++++++++++++++ install.sh | 3 ++ test/conftest.py | 17 +++++-- test/test_volume_data.sh | 5 +- tox.ini | 2 +- 17 files changed, 302 insertions(+), 113 deletions(-) create mode 100644 .github/workflows/test-and-build.yaml delete mode 100644 .github/workflows/test.yaml create mode 100755 gh-actions-deploy.sh create mode 100755 gh-actions-test.sh create mode 100755 gh-actions-vars.sh diff --git a/.github/workflows/test-and-build.yaml b/.github/workflows/test-and-build.yaml new file mode 100644 index 0000000..f518e6a --- /dev/null +++ b/.github/workflows/test-and-build.yaml @@ -0,0 +1,62 @@ +name: Test & Build +on: + push: + branches: + - master + - dev + - v* + - beta-v* + pull_request: + +#env: +# DOCKER_HUB_REPO: pihole + +jobs: + test-and-build: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + ARCH: [amd64, armhf, arm64] + DEBIAN_VERSION: [stretch, buster] + env: + ARCH: ${{matrix.ARCH}} + DEBIAN_VERSION: ${{matrix.DEBIAN_VERSION}} + steps: + - name: Checkout Repo + uses: actions/checkout@v2 + - name: Run Tests + run: | + echo "Building ${ARCH}-${DEBIAN_VERSION}" + ./gh-actions-test.sh + - name: Push the ARCH image + if: github.event_name != 'pull_request' + run: | + . gh-actions-vars.sh + echo "${{ secrets.DOCKERHUB_PASS }}" | docker login --username="${{ secrets.DOCKERHUB_USER }}" --password-stdin + docker push "${ARCH_IMAGE}" + - name: Upload gh-workspace + if: github.event_name != 'pull_request' + uses: actions/upload-artifact@v1 + with: + name: gh-workspace + path: .gh-workspace + + publish: + if: github.event_name != 'pull_request' + runs-on: ubuntu-latest + needs: test-and-build + steps: + - name: Checkout Repo + uses: actions/checkout@v2 + - name: Download workspace files + uses: actions/download-artifact@v1 + with: + name: gh-workspace + path: .gh-workspace + - name: Tag and Publish multi-arch images + env: + DOCKERHUB_PASS: ${{ secrets.DOCKERHUB_PASS }} + DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }} + run: | + ./gh-actions-deploy.sh diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml deleted file mode 100644 index 102482d..0000000 --- a/.github/workflows/test.yaml +++ /dev/null @@ -1,31 +0,0 @@ -name: Tests -on: - push: - branches: - - master - - dev - - v* - - beta-v* - pull_request: - -jobs: - build: - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - ARCH: - - amd64 - # https://github.com/pi-hole/docker-pi-hole/issues/587#issuecomment-617180631 - #- armel - - armhf - - arm64 - env: - ARCH: ${{matrix.ARCH}} - steps: - - name: Checkout Repo - uses: actions/checkout@v2 - - name: Run Circle Test - run: | - echo "Building ${ARCH}" - ./circle-test.sh diff --git a/.gitignore b/.gitignore index fd32835..4e7bc94 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ __pycache__ UNKNOWN.egg-info .env ci-workspace +.gh-workspace # WIP/test stuff doco.yml diff --git a/Dockerfile.py b/Dockerfile.py index 7fc1363..749db9a 100755 --- a/Dockerfile.py +++ b/Dockerfile.py @@ -2,15 +2,16 @@ """ Dockerfile.py - generates and build dockerfiles Usage: - Dockerfile.py [--hub_tag=] [--arch= ...] [-v] [-t] [--no-build] [--no-cache] + Dockerfile.py [--hub_tag=] [--arch= ...] [--debian= ...] [-v] [-t] [--no-build] [--no-cache] Options: - --no-build Skip building the docker images - --no-cache Build without using any cache data - --hub_tag= What the Docker Hub Image should be tagged as [default: None] - --arch= What Architecture(s) to build [default: amd64 armel armhf arm64] - -v Print docker's command output [default: False] - -t Print docker's build time [default: False] + --no-build Skip building the docker images + --no-cache Build without using any cache data + --hub_tag= What the Docker Hub Image should be tagged as [default: None] + --arch= What Architecture(s) to build [default: amd64 armel armhf arm64] + --debian= What debian version(s) to build [default: stretch buster] + -v Print docker's command output [default: False] + -t Print docker's build time [default: False] Examples: """ @@ -20,8 +21,6 @@ from docopt import docopt import os import subprocess -THIS_DIR = os.path.dirname(os.path.abspath(__file__)) - __version__ = None dot = os.path.abspath('.') with open('{}/VERSION'.format(dot), 'r') as v: @@ -30,21 +29,22 @@ with open('{}/VERSION'.format(dot), 'r') as v: def build_dockerfiles(args): + if args['-v']: + print(args) if args['--no-build']: print(" ::: Skipping Dockerfile building") return for arch in args['--arch']: - build('pihole', arch, args) + for debian_version in args['--debian']: + build('pihole', arch, debian_version, args['--hub_tag'], args['-t'], args['--no-cache'], args['-v']) -def run_and_stream_command_output(command, args): +def run_and_stream_command_output(command, environment_vars, verbose): print("Running", command) - build_env = os.environ.copy() - build_env['PIHOLE_VERSION'] = __version__ - build_result = subprocess.Popen(command.split(), env=build_env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, - bufsize=1, universal_newlines=True) - if args['-v']: + build_result = subprocess.Popen(command.split(), env=environment_vars, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True) + if verbose: while build_result.poll() is None: for line in build_result.stdout: print(line, end='') @@ -54,31 +54,25 @@ def run_and_stream_command_output(command, args): print(build_result.stderr) -def build(docker_repo, arch, args): - repo_tag = '{}:{}-{}'.format(docker_repo, __version__, arch) - print(" ::: Building {}".format(repo_tag)) - time = '' - if args['-t']: - time = 'time ' - no_cache = '' - if args['--no-cache']: - no_cache = '--no-cache' - build_command = '{time}docker-compose -f build.yml build {no_cache} --pull {arch}'\ - .format(time=time, no_cache=no_cache, arch=arch) - print(" ::: Building {} into {}".format(arch, repo_tag)) - run_and_stream_command_output(build_command, args) - if args['-v']: +def build(docker_repo: str, arch: str, debian_version: str, hub_tag: str, show_time: bool, no_cache: bool, verbose: bool): + create_tag = f'{docker_repo}:{__version__}-{arch}-{debian_version}' + print(f' ::: Building {create_tag}') + time_arg = 'time' if show_time else '' + cache_arg = '--no-cache' if no_cache else '' + build_env = os.environ.copy() + build_env['PIHOLE_VERSION'] = __version__ + build_env['DEBIAN_VERSION'] = debian_version + build_command = f'{time_arg} docker-compose -f build.yml build {cache_arg} --pull {arch}' + print(f' ::: Building {arch} into {create_tag}') + run_and_stream_command_output(build_command, build_env, verbose) + if verbose: print(build_command, '\n') - if args['--hub_tag']: - hub_tag_command = "{time}docker tag {create_tag} {hub_tag}"\ - .format(time=time, create_tag=repo_tag, hub_tag=args['--hub_tag']) - print(" ::: Tagging {} into {}".format(repo_tag, args['--hub_tag'])) - run_and_stream_command_output(hub_tag_command, args) + if hub_tag: + hub_tag_command = f'{time_arg} docker tag {create_tag} {hub_tag}' + print(f' ::: Tagging {create_tag} into {hub_tag}') + run_and_stream_command_output(hub_tag_command, build_env, verbose) if __name__ == '__main__': args = docopt(__doc__, version='Dockerfile 1.1') - if args['-v']: - print(args) - build_dockerfiles(args) diff --git a/Dockerfile.sh b/Dockerfile.sh index 3338491..9f9d8d7 100755 --- a/Dockerfile.sh +++ b/Dockerfile.sh @@ -1,10 +1,13 @@ -#!/usr/bin/env sh -# alpine sh only +#!/usr/bin/env bash + +# @param ${ARCH} The architecture to build. Example: amd64 +# @param ${DEBIAN_VERSION} The debian version to build. Example: buster +# @param ${ARCH_IMAGE} What the Docker Hub Image should be tagged as [default: None] set -eux -./Dockerfile.py -v --arch="${ARCH}" --hub_tag="${ARCH_IMAGE}" +./Dockerfile.py -v --arch="${ARCH}" --debian="${DEBIAN_VERSION}" --hub_tag="${ARCH_IMAGE}" docker images -# TODO: Add junitxml output and have circleci consume it +# TODO: Add junitxml output and have something consume it # 2 parallel max b/c race condition with docker fixture (I think?) py.test -vv -n 2 -k "${ARCH}" ./test/ diff --git a/Dockerfile_build b/Dockerfile_build index 5327d8d..e40f551 100644 --- a/Dockerfile_build +++ b/Dockerfile_build @@ -3,16 +3,14 @@ FROM python:buster # Only works for docker CLIENT (bind mounted socket) COPY --from=docker:18.09.3 /usr/local/bin/docker /usr/local/bin/ -# Based on https://github.com/Ilhicas/alpine-pipenv -ARG packages RUN apt-get update && \ apt-get install -y python3-dev curl gcc make \ - libffi-dev libssl-dev ${packages} \ + libffi-dev libssl-dev \ && pip3 install -U pip pipenv + RUN curl -L https://github.com/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose && \ chmod +x /usr/local/bin/docker-compose -# -v "$(pwd):/$(pwd)" -w "$(pwd)" to prevent nested docker path confusion COPY ./Dockerfile.sh /usr/local/bin/ COPY Pipfile* /root/ WORKDIR /root @@ -20,7 +18,6 @@ WORKDIR /root RUN pipenv install --system \ && sed -i 's|/bin/sh|/bin/bash|g' /usr/local/lib/python3.8/site-packages/testinfra/backend/docker.py - RUN echo "set -ex && Dockerfile.sh && \$@" > /usr/local/bin/entrypoint.sh RUN chmod +x /usr/local/bin/entrypoint.sh ENTRYPOINT entrypoint.sh diff --git a/README.md b/README.md index 1751c4a..9a05db6 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ A [Docker](https://www.docker.com/what-docker) project to make a lightweight x86 2) Use the above quick start example, customize if desired. 3) Enjoy! -[![Build Status](https://api.travis-ci.org/pi-hole/docker-pi-hole.svg?branch=master)](https://travis-ci.org/pi-hole/docker-pi-hole) [![Docker Stars](https://img.shields.io/docker/stars/pihole/pihole.svg?maxAge=604800)](https://store.docker.com/community/images/pihole/pihole) [![Docker Pulls](https://img.shields.io/docker/pulls/pihole/pihole.svg?maxAge=604800)](https://store.docker.com/community/images/pihole/pihole) +[![Build Status](https://github.com/pi-hole/docker-pi-hole/workflows/Test%20&%20Build/badge.svg)](https://github.com/pi-hole/docker-pi-hole/actions?query=workflow%3A%22Test+%26+Build%22) [![Docker Stars](https://img.shields.io/docker/stars/pihole/pihole.svg?maxAge=604800)](https://store.docker.com/community/images/pihole/pihole) [![Docker Pulls](https://img.shields.io/docker/pulls/pihole/pihole.svg?maxAge=604800)](https://store.docker.com/community/images/pihole/pihole) ## Running Pi-hole Docker @@ -176,14 +176,15 @@ Users of older Ubuntu releases (circa 17.04) will need to disable dnsmasq. ## Docker tags and versioning -The primary docker tags / versions are explained in the following table. [Click here to see the full list of tags](https://store.docker.com/community/images/pihole/pihole/tags) ([arm tags are here](https://store.docker.com/community/images/pihole/pihole/tags)), I also try to tag with the specific version of Pi-hole Core for version archival purposes, the web version that comes with the core releases should be in the [GitHub Release notes](https://github.com/pi-hole/docker-pi-hole/releases). +The primary docker tags / versions are explained in the following table. [Click here to see the full list of tags](https://store.docker.com/community/images/pihole/pihole/tags), I also try to tag with the specific version of Pi-hole Core for version archival purposes, the web version that comes with the core releases should be in the [GitHub Release notes](https://github.com/pi-hole/docker-pi-hole/releases). -| tag | architecture | description | Dockerfile | -| --- | ------------ | ----------- | ---------- | -| `latest` | auto detect | x86, arm, or arm64 container, docker auto detects your architecture. | [Dockerfile](https://github.com/pi-hole/docker-pi-hole/blob/master/Dockerfile) | -| `v4.0.0-1` | auto detect | Versioned tags, if you want to pin against a specific version, use one of these | | -| `v4.0.0-1_` | based on tag | Specific architectures tags | | -| `dev` | auto detect | like latest tag, but for the development branch (pushed occasionally) | | +| tag | architecture | description | Dockerfile | +| --- | ------------ | ----------- | ---------- | +| `latest` | auto detect | x86, arm, or arm64 container, docker auto detects your architecture. | [Dockerfile](https://github.com/pi-hole/docker-pi-hole/blob/master/Dockerfile) | +| `v5.0` | auto detect | Versioned tags, if you want to pin against a specific Pi-hole version, use one of these | | +| `v5.0-stretch` | auto detect | Versioned tags, if you want to pin against a specific Pi-hole and Debian version, use one of these | | +| `v5.0--stretch` | based on tag | Specific architectures and Debian version tags | | +| `dev` | auto detect | like latest tag, but for the development branch (pushed occasionally) | | ### `pihole/pihole:latest` [![](https://images.microbadger.com/badges/image/pihole/pihole:latest.svg)](https://microbadger.com/images/pihole/pihole "Get your own image badge on microbadger.com") [![](https://images.microbadger.com/badges/version/pihole/pihole:latest.svg)](https://microbadger.com/images/pihole/pihole "Get your own version badge on microbadger.com") [![](https://images.microbadger.com/badges/version/pihole/pihole:latest.svg)](https://microbadger.com/images/pihole/pihole "Get your own version badge on microbadger.com") diff --git a/TESTING.md b/TESTING.md index 88d10cc..75e8090 100644 --- a/TESTING.md +++ b/TESTING.md @@ -4,7 +4,7 @@ Make sure you have bash, docker. Python and some test hacks are crammed into th # Running tests locally -`ARCH=amd64 ./circle-test.sh` +`ARCH=amd64 ./gh-actions-test.sh` Should result in : diff --git a/bash_functions.sh b/bash_functions.sh index 32a0d32..025f845 100644 --- a/bash_functions.sh +++ b/bash_functions.sh @@ -46,7 +46,7 @@ validate_env() { # Optional ServerIP is a valid IP # nc won't throw any text based errors when it times out connecting to a valid IP, otherwise it complains about the DNS name being garbage # if nc doesn't behave as we expect on a valid IP the routing table should be able to look it up and return a 0 retcode - if [[ "$(nc -4 -w1 -z "$ServerIP" 53 2>&1)" != "" ]] || ! ip route get "$ServerIP" > /dev/null ; then + if [[ "$(nc -4 -w1 -z "$ServerIP" 53 2>&1)" != "" ]] && ! ip route get "$ServerIP" > /dev/null ; then echo "ERROR: ServerIP Environment variable ($ServerIP) doesn't appear to be a valid IPv4 address" exit 1 fi @@ -58,7 +58,7 @@ validate_env() { unset ServerIPv6 exit 1 fi - if [[ "$(nc -6 -w1 -z "$ServerIPv6" 53 2>&1)" != "" ]] || ! ip route get "$ServerIPv6" > /dev/null ; then + if [[ "$(nc -6 -w1 -z "$ServerIPv6" 53 2>&1)" != "" ]] && ! ip route get "$ServerIPv6" > /dev/null ; then echo "ERROR: ServerIPv6 Environment variable ($ServerIPv6) doesn't appear to be a valid IPv6 address" echo " TIP: If your server is not IPv6 enabled just remove '-e ServerIPv6' from your docker container" exit 1 diff --git a/build.yml b/build.yml index ab78653..0a33a7c 100644 --- a/build.yml +++ b/build.yml @@ -12,46 +12,38 @@ x-common-args: &common-args services: amd64: - image: pihole:${PIHOLE_VERSION}-amd64 + image: pihole:${PIHOLE_VERSION}-amd64-${DEBIAN_VERSION:-stretch} build: context: . - cache_from: - - pihole/pihole:${PIHOLE_VERSION}-amd64 args: <<: *common-args - PIHOLE_BASE: pihole/debian-base:latest + PIHOLE_BASE: pihole/debian-base:${DEBIAN_VERSION:-stretch} PIHOLE_ARCH: amd64 S6_ARCH: amd64 armel: - image: pihole:${PIHOLE_VERSION}-armel + image: pihole:${PIHOLE_VERSION}-armel-${DEBIAN_VERSION:-stretch} build: context: . - cache_from: - - pihole/pihole:${PIHOLE_VERSION}-armel args: <<: *common-args - PIHOLE_BASE: multiarch/debian-debootstrap:armel-stretch-slim + PIHOLE_BASE: multiarch/debian-debootstrap:armel-${DEBIAN_VERSION:-stretch}-slim PIHOLE_ARCH: armel S6_ARCH: arm armhf: - image: pihole:${PIHOLE_VERSION}-armhf + image: pihole:${PIHOLE_VERSION}-armhf-${DEBIAN_VERSION:-stretch} build: context: . - cache_from: - - pihole/pihole:${PIHOLE_VERSION}-armhf args: <<: *common-args - PIHOLE_BASE: multiarch/debian-debootstrap:armhf-stretch-slim + PIHOLE_BASE: multiarch/debian-debootstrap:armhf-${DEBIAN_VERSION:-stretch}-slim PIHOLE_ARCH: arm S6_ARCH: arm arm64: - image: pihole:${PIHOLE_VERSION}-arm64 + image: pihole:${PIHOLE_VERSION}-arm64-${DEBIAN_VERSION:-stretch} build: context: . - cache_from: - - pihole/pihole:${PIHOLE_VERSION}-arm64 args: <<: *common-args - PIHOLE_BASE: multiarch/debian-debootstrap:arm64-stretch-slim + PIHOLE_BASE: multiarch/debian-debootstrap:arm64-${DEBIAN_VERSION:-stretch}-slim PIHOLE_ARCH: arm64 - S6_ARCH: aarch64 \ No newline at end of file + S6_ARCH: aarch64 diff --git a/gh-actions-deploy.sh b/gh-actions-deploy.sh new file mode 100755 index 0000000..66f0d01 --- /dev/null +++ b/gh-actions-deploy.sh @@ -0,0 +1,73 @@ +#!/usr/bin/env bash +set -ex +# Github Actions Job for merging/deploying all architectures (post-test passing) +. gh-actions-vars.sh + +function annotate() { + local base=$1 + local image=$2 + local arch=$3 + local annotate_flags="${annotate_map[$arch]}" + + $dry docker manifest annotate ${base} ${image} --os linux ${annotate_flags} +} + +function create_manifest() { + local debian_version=$1 + cd "${debian_version}" + + for arch in *; do + arch_image=$(cat "${arch}") + docker pull "${arch_image}" + images+=("${arch_image}") + done + + multiarch_images=$(get_multiarch_images) + for docker_tag in ${multiarch_images}; do + docker manifest create ${docker_tag} ${images[*]} + for arch in *; do + arch_image=$(cat "${arch}") + annotate "${docker_tag}" "${arch_image}" "${arch}" + done + + docker manifest inspect "${docker_tag}" + docker manifest push --purge "${docker_tag}" + done + cd ../ +} + +function get_multiarch_images() { + multiarch_images="${MULTIARCH_IMAGE}-${debian_version}" + if [[ "${debian_version}" == "${DEFAULT_DEBIAN_VERSION}" ]] ; then + # default debian version gets a non-debian tag as well as latest tag + multiarch_images="${multiarch_images} ${MULTIARCH_IMAGE} ${LATEST_IMAGE}" + fi + echo "${multiarch_images}" +} + + +# Keep in sync with build.yml names +declare -A annotate_map=( + ["amd64"]="--arch amd64" + ["armel"]="--arch arm --variant v6" + ["armhf"]="--arch arm --variant v7" + ["arm64"]="--arch arm64 --variant v8" +) + +mkdir -p ~/.docker +export DOCKER_CLI_EXPERIMENTAL='enabled' +echo "{}" | jq '.experimental="enabled"' | tee ~/.docker/config.json +# I tried to keep this login command outside of this script +# but for some reason auth would always fail in Github Actions. +# I think setting up a cred store would fix it +# https://docs.docker.com/engine/reference/commandline/login/#credentials-store +echo "${DOCKERHUB_PASS}" | docker login --username="${DOCKERHUB_USER}" --password-stdin +docker info + +images=() +ls -lat ./.gh-workspace/ +cd .gh-workspace + +for debian_version in *; do + create_manifest "${debian_version}" +done diff --git a/gh-actions-test.sh b/gh-actions-test.sh new file mode 100755 index 0000000..691ab16 --- /dev/null +++ b/gh-actions-test.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -ex + +# Script ran by Github actions for tests +# +# @environment ${ARCH} The architecture to build. Example: amd64. +# @environment ${DEBIAN_VERSION} Debian version to build. ('buster' or 'stretch'). +# @environment ${ARCH_IMAGE} What the Docker Hub Image should be tagged as. Example: pihole/pihole:master-amd64-stretch + +# setup qemu/variables +docker run --rm --privileged multiarch/qemu-user-static:register --reset > /dev/null +. gh-actions-vars.sh + +if [[ "$1" == "enter" ]]; then + enter="-it --entrypoint=sh" +fi + +# generate and build dockerfile +docker build --tag image_pipenv --file Dockerfile_build . +docker run --rm \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume "$(pwd):/$(pwd)" \ + --workdir "$(pwd)" \ + --env PIPENV_CACHE_DIR="$(pwd)/.pipenv" \ + --env ARCH="${ARCH}" \ + --env ARCH_IMAGE="${ARCH_IMAGE}" \ + --env DEBIAN_VERSION="${DEBIAN_VERSION}" \ + ${enter} image_pipenv + +mkdir -p ".gh-workspace/${DEBIAN_VERSION}/" +echo "${ARCH_IMAGE}" | tee "./.gh-workspace/${DEBIAN_VERSION}/${ARCH}" diff --git a/gh-actions-vars.sh b/gh-actions-vars.sh new file mode 100755 index 0000000..6361698 --- /dev/null +++ b/gh-actions-vars.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +set -a + +# @environment ${ARCH} The architecture to build. Defaults to 'amd64'. +# @environment ${DEBIAN_VERSION} Debian version to build. Defaults to 'stretch'. +# @environment ${DOCKER_HUB_REPO} The docker hub repo to tag images for. Defaults to 'pihole'. +# @environment ${DOCKER_HUB_IMAGE_NAME} The name of the resulting image. Defaults to 'pihole'. + +GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD | sed "s/\//-/g") +GIT_TAG=$(git describe --tags --exact-match 2> /dev/null || true) + +DEFAULT_DEBIAN_VERSION="stretch" + +if [[ -z "${ARCH}" ]]; then + ARCH="amd64" + echo "Defaulting arch to ${ARCH}" +fi + +if [[ -z "${DEBIAN_VERSION}" ]]; then + DEBIAN_VERSION="${DEFAULT_DEBIAN_VERSION}" + echo "Defaulting DEBIAN_VERSION to ${DEBIAN_VERSION}" +fi + +if [[ -z "${DOCKER_HUB_REPO}" ]]; then + DOCKER_HUB_REPO="pihole" + echo "Defaulting DOCKER_HUB_REPO to ${DOCKER_HUB_REPO}" +fi + +if [[ -z "${DOCKER_HUB_IMAGE_NAME}" ]]; then + DOCKER_HUB_IMAGE_NAME="pihole" + echo "Defaulting DOCKER_HUB_IMAGE_NAME to ${DOCKER_HUB_IMAGE_NAME}" +fi + +BASE_IMAGE="${DOCKER_HUB_REPO}/${DOCKER_HUB_IMAGE_NAME}" + +GIT_TAG="${GIT_TAG:-$GIT_BRANCH}" +ARCH_IMAGE="${BASE_IMAGE}:${GIT_TAG}-${ARCH}-${DEBIAN_VERSION}" +MULTIARCH_IMAGE="${BASE_IMAGE}:${GIT_TAG}" + + + +# To get latest released, cut a release on https://github.com/pi-hole/docker-pi-hole/releases (manually gated for quality control) +latest_tag='UNKNOWN' +if ! latest_tag=$(curl -sI https://github.com/pi-hole/docker-pi-hole/releases/latest | grep --color=never -i Location | awk -F / '{print $NF}' | tr -d '[:cntrl:]'); then + print "Failed to retrieve latest docker-pi-hole release metadata" +else + if [[ "${GIT_TAG}" == "${latest_tag}" ]] ; then + LATEST_IMAGE="${BASE_IMAGE}:latest" + fi +fi + + +set +a diff --git a/install.sh b/install.sh index 2df2275..8b8f40f 100644 --- a/install.sh +++ b/install.sh @@ -15,6 +15,9 @@ fi apt-get update apt-get install --no-install-recommends -y curl procps ca-certificates +# curl in armhf-buster's image has SSL issues. Running c_rehash fixes it. +# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923479 +c_rehash curl -L -s $S6OVERLAY_RELEASE | tar xvzf - -C / mv /init /s6-init diff --git a/test/conftest.py b/test/conftest.py index bb82273..61d68fd 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -9,6 +9,7 @@ import types local_host = testinfra.get_host('local://') check_output = local_host.check_output +DEBIAN_VERSION = os.environ.get('DEBIAN_VERSION', 'stretch') __version__ = None dotdot = os.path.abspath(os.path.join(os.path.abspath(__file__), os.pardir, os.pardir)) with open('{}/VERSION'.format(dotdot), 'r') as v: @@ -99,8 +100,12 @@ def version(): return __version__ @pytest.fixture() -def tag(version, arch): - return '{}-{}'.format(version, arch) +def debian_version(): + return DEBIAN_VERSION + +@pytest.fixture() +def tag(version, arch, debian_version): + return '{}-{}-{}'.format(version, arch, debian_version) @pytest.fixture def webserver(tag): @@ -125,6 +130,10 @@ def persist_arch(): def persist_version(): return __version__ +@pytest.fixture(scope='module') +def persist_debian_version(): + return DEBIAN_VERSION + @pytest.fixture(scope='module') def persist_args_dns(): return '--dns 127.0.0.1 --dns 1.1.1.1' @@ -147,8 +156,8 @@ def persist_test_args(): return '' @pytest.fixture(scope='module') -def persist_tag(persist_version, persist_arch): - return '{}_{}'.format(persist_version, persist_arch) +def persist_tag(persist_version, persist_arch, persist_debian_version): + return '{}_{}_{}'.format(persist_version, persist_arch, persist_debian_version) @pytest.fixture(scope='module') def persist_webserver(persist_tag): diff --git a/test/test_volume_data.sh b/test/test_volume_data.sh index 2c87a63..0acf9b2 100755 --- a/test/test_volume_data.sh +++ b/test/test_volume_data.sh @@ -33,8 +33,9 @@ trap "cleanup" INT TERM EXIT # VOLUME TESTS # Given... -IMAGE="${1:-pihole:v5.0-amd64}" # Default is latest build test image (generic, non release/branch tag) -VOLUMES="$(mktemp -d)" # A fresh volume directory +DEBIAN_VERSION="$(DEBIAN_VERSION:-stretch)" +IMAGE="${1:-pihole:v5.0-amd64}-${DEBIAN_VERSION}" # Default is latest build test image (generic, non release/branch tag) +VOLUMES="$(mktemp -d)" # A fresh volume directory VOL_PH="$VOLUMES/pihole" VOL_DM="$VOLUMES/dnsmasq.d" tty -s && TTY='-t' || TTY='' diff --git a/tox.ini b/tox.ini index 9d3164d..9342d6b 100644 --- a/tox.ini +++ b/tox.ini @@ -2,7 +2,7 @@ envlist = py38 [testenv] -commands = echo "Use ./circle-test.sh instead for now" +commands = echo "Use ./gh-actions-test.sh instead for now" # Currently out of comission post-python3 upgrade due to failed monkey patch of testinfra sh -> bash #[testenv] From c8781ee4278fb99672bf102461919833438ff1a6 Mon Sep 17 00:00:00 2001 From: Daniel Date: Thu, 16 Jul 2020 14:06:07 -0400 Subject: [PATCH 19/24] Remove circle-ci since it was not updated to work with multi-debian builds and is replaced with Github Actions Signed-off-by: Daniel --- .circleci/config.yml | 98 -------------------------------------------- circle-deploy.sh | 51 ----------------------- circle-test.sh | 31 -------------- circle-vars.sh | 48 ---------------------- 4 files changed, 228 deletions(-) delete mode 100644 .circleci/config.yml delete mode 100755 circle-deploy.sh delete mode 100755 circle-test.sh delete mode 100755 circle-vars.sh diff --git a/.circleci/config.yml b/.circleci/config.yml deleted file mode 100644 index 7ef1f32..0000000 --- a/.circleci/config.yml +++ /dev/null @@ -1,98 +0,0 @@ -version: 2 - -.job_template: &job_template - machine: - enabled: true - steps: - - checkout - - run: - command: ./circle-test.sh - - persist_to_workspace: - root: . - paths: [ 'ci-workspace' ] - -jobs: - amd64: - <<: *job_template - arm64: - <<: *job_template - armhf: - <<: *job_template - armel: - <<: *job_template - deploy: - docker: - - image: circleci/python:latest - steps: - - setup_remote_docker: - version: 18.06.0-ce - - checkout - - attach_workspace: - at: . - - run: - command: ./circle-deploy.sh - - - -workflows: - version: 2 - build: - jobs: - - amd64: - filters: - tags: - only: /^v.*/ - - arm64: - filters: - tags: - only: /^v.*/ - - armhf: - filters: - tags: - only: /^v.*/ - #- armel: - # filters: - # tags: - # only: /^v.*/ - - deploy: - requires: - - amd64 - - arm64 - - armhf - #- armel - filters: - tags: - only: /^v.*/ - nightly_build: - triggers: - - schedule: - cron: "0 0 * * *" - filters: - branches: - only: 'beta-v5.0' - jobs: - - amd64: - filters: - tags: - only: /^v.*/ - - arm64: - filters: - tags: - only: /^v.*/ - - armhf: - filters: - tags: - only: /^v.*/ - #- armel: - # filters: - # tags: - # only: /^v.*/ - - deploy: - requires: - - amd64 - - arm64 - - armhf - #- armel - filters: - tags: - only: /^v.*/ diff --git a/circle-deploy.sh b/circle-deploy.sh deleted file mode 100755 index 884383a..0000000 --- a/circle-deploy.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/usr/bin/env bash -set -ex -# Circle CI Job for merging/deploying all architectures (post-test passing) -. circle-vars.sh - -annotate() { - local base=$1 - local image=$2 - local arch=$3 - local annotate_flags="${annotate_map[$arch]}" - - $dry docker manifest annotate ${base} ${image} --os linux ${annotate_flags} -} - -# Keep in sync with circle-ci job names -declare -A annotate_map=( - ["amd64"]="--arch amd64" - ["armel"]="--arch arm --variant v6" - ["armhf"]="--arch arm --variant v7" - ["arm64"]="--arch arm64 --variant v8" -) - -# push image when not running a PR -mkdir -p ~/.docker -export DOCKER_CLI_EXPERIMENTAL='enabled' -echo "{}" | jq '.experimental="enabled"' | tee ~/.docker/config.json -docker info -if [[ "$CIRCLE_PR_NUMBER" == "" ]]; then - images=() - echo $DOCKERHUB_PASS | docker login --username=$DOCKERHUB_USER --password-stdin - ls -lat ./ci-workspace/ - cd ci-workspace - - for arch in *; do - arch_image=$(cat $arch) - docker pull $arch_image - images+=($arch_image) - done - - for docker_tag in $MULTIARCH_IMAGE $LATEST_IMAGE; do - docker manifest create $docker_tag ${images[*]} - for arch in *; do - arch_image=$(cat $arch) - docker pull $arch_image - annotate "$docker_tag" "$arch_image" "$arch" - done - - docker manifest inspect "$docker_tag" - docker manifest push --purge "$docker_tag" - done; -fi diff --git a/circle-test.sh b/circle-test.sh deleted file mode 100755 index 8b61499..0000000 --- a/circle-test.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/usr/bin/env bash -set -ex - -# Circle CI Job for single architecture - -# setup qemu/variables -docker run --rm --privileged multiarch/qemu-user-static:register --reset > /dev/null -. circle-vars.sh - -if [[ "$1" == "enter" ]]; then - enter="-it --entrypoint=sh" -fi - -# generate and build dockerfile -docker pull python:buster -docker build -t image_pipenv -f Dockerfile_build . -env > /tmp/env -docker run --rm \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v "$(pwd):/$(pwd)" \ - -w "$(pwd)" \ - -e PIPENV_CACHE_DIR="$(pwd)/.pipenv" \ - --env-file /tmp/env \ - $enter image_pipenv - -test -z "${CIRCLE_PROJECT_REPONAME}" && exit 0 -# The rest is circle-ci only -echo $DOCKERHUB_PASS | docker login --username=$DOCKERHUB_USER --password-stdin -docker push $ARCH_IMAGE -mkdir -p ci-workspace -echo "$ARCH_IMAGE" | tee ./ci-workspace/$ARCH diff --git a/circle-vars.sh b/circle-vars.sh deleted file mode 100755 index fbe4db1..0000000 --- a/circle-vars.sh +++ /dev/null @@ -1,48 +0,0 @@ -set -a - -CIRCLE_JOB="${CIRCLE_JOB:-}" -ARCH="${ARCH:-$CIRCLE_JOB}" -if [[ -z "$ARCH" ]] ; then - echo "Defaulting arch to amd64" - ARCH="amd64" -fi -BASE_IMAGE="${BASE_IMAGE:-${CIRCLE_PROJECT_REPONAME}}" -if [[ -z "$BASE_IMAGE" ]] ; then - echo "Defaulting image name to pihole" - BASE_IMAGE="pihole" -fi - -# The docker image will match the github repo path by default but is overrideable with CircleCI environment -# BASE_IMAGE Overridable by Circle environment, including namespace (e.g. BASE_IMAGE=bobsmith/test-img:latest) -CIRCLE_PROJECT_USERNAME="${CIRCLE_PROJECT_USERNAME:-unset}" -HUB_NAMESPACE="${HUB_NAMESPACE:-$CIRCLE_PROJECT_USERNAME}" -[[ $CIRCLE_PROJECT_USERNAME == "pi-hole" ]] && HUB_NAMESPACE="pihole" # Custom mapping for namespace -[[ $BASE_IMAGE != *"/"* ]] && BASE_IMAGE="${HUB_NAMESPACE}/${BASE_IMAGE}" # If missing namespace, add one - -# Secondary docker tag info (origin github branch/tag) will get prepended also -ARCH_IMAGE="$BASE_IMAGE" -[[ $ARCH_IMAGE != *":"* ]] && ARCH_IMAGE="${BASE_IMAGE}:$ARCH" # If tag missing, add circle job name as a tag (architecture here) - -DOCKER_TAG="${CIRCLE_TAG:-$CIRCLE_BRANCH}" -if [[ -n "$DOCKER_TAG" ]]; then - # remove latest tag if used (as part of a user provided image variable) - ARCH_IMAGE="${ARCH_IMAGE/:latest/:}" - # Prepend the github tag(version) or branch. image:arch = image:v1.0-arch - ARCH_IMAGE="${ARCH_IMAGE/:/:${DOCKER_TAG}-}" - # latest- sometimes has a trailing slash, remove it - ARCH_IMAGE="${ARCH_IMAGE/%-/}" -fi - -# To get latest released, cut a release on https://github.com/pi-hole/docker-pi-hole/releases (manually gated for quality control) -latest_tag='' -if ! latest_tag=$(curl -sI https://github.com/pi-hole/docker-pi-hole/releases/latest | grep --color=never -i Location | awk -F / '{print $NF}' | tr -d '[:cntrl:]'); then - print "Failed to retrieve latest docker-pi-hole release metadata" -else - if [[ "$DOCKER_TAG" == "$latest_tag" ]] ; then - LATEST_IMAGE="$BASE_IMAGE:latest" - fi -fi - -MULTIARCH_IMAGE="$BASE_IMAGE:$DOCKER_TAG" - -set +a From 7ad7b0824f469f69026f39c5398451245e60cfd4 Mon Sep 17 00:00:00 2001 From: Daniel Date: Thu, 16 Jul 2020 14:59:59 -0400 Subject: [PATCH 20/24] Report docker build and tag exit code to halt the tests on failure to build Signed-off-by: Daniel --- Dockerfile.py | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/Dockerfile.py b/Dockerfile.py index b4e36d8..44efac4 100755 --- a/Dockerfile.py +++ b/Dockerfile.py @@ -2,11 +2,12 @@ """ Dockerfile.py - generates and build dockerfiles Usage: - Dockerfile.py [--hub_tag=] [--arch= ...] [--debian= ...] [-v] [-t] [--no-build] [--no-cache] + Dockerfile.py [--hub_tag=] [--arch= ...] [--debian= ...] [-v] [-t] [--no-build] [--no-cache] [--fail-fast] Options: --no-build Skip building the docker images --no-cache Build without using any cache data + --fail-fast Exit on first build error --hub_tag= What the Docker Hub Image should be tagged as [default: None] --arch= What Architecture(s) to build [default: amd64 armel armhf arm64] --debian= What debian version(s) to build [default: stretch buster] @@ -15,10 +16,9 @@ Options: Examples: """ - - from docopt import docopt import os +import sys import subprocess __version__ = None @@ -28,19 +28,23 @@ with open('{}/VERSION'.format(dot), 'r') as v: __version__ = raw_version.replace('release/', 'release-') -def build_dockerfiles(args): +def build_dockerfiles(args) -> bool: + all_success = True if args['-v']: print(args) if args['--no-build']: print(" ::: Skipping Dockerfile building") - return + return all_success for arch in args['--arch']: for debian_version in args['--debian']: - build('pihole', arch, debian_version, args['--hub_tag'], args['-t'], args['--no-cache'], args['-v']) + all_success = build('pihole', arch, debian_version, args['--hub_tag'], args['-t'], args['--no-cache'], args['-v']) and all_success + if not all_success and args['--fail-fast']: + return False + return all_success -def run_and_stream_command_output(command, environment_vars, verbose): +def run_and_stream_command_output(command, environment_vars, verbose) -> bool: print("Running", command) build_result = subprocess.Popen(command.split(), env=environment_vars, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True) @@ -50,12 +54,12 @@ def run_and_stream_command_output(command, environment_vars, verbose): print(line, end='') build_result.wait() if build_result.returncode != 0: - print(" ::: Error running {}".format(command)) + print(" ::: Error running".format(command)) print(build_result.stderr) - sys.exit(build_resuilt.returncode) + return build_result.returncode == 0 -def build(docker_repo: str, arch: str, debian_version: str, hub_tag: str, show_time: bool, no_cache: bool, verbose: bool): +def build(docker_repo: str, arch: str, debian_version: str, hub_tag: str, show_time: bool, no_cache: bool, verbose: bool) -> bool: create_tag = f'{docker_repo}:{__version__}-{arch}-{debian_version}' print(f' ::: Building {create_tag}') time_arg = 'time' if show_time else '' @@ -65,15 +69,18 @@ def build(docker_repo: str, arch: str, debian_version: str, hub_tag: str, show_t build_env['DEBIAN_VERSION'] = debian_version build_command = f'{time_arg} docker-compose -f build.yml build {cache_arg} --pull {arch}' print(f' ::: Building {arch} into {create_tag}') - run_and_stream_command_output(build_command, build_env, verbose) + success = run_and_stream_command_output(build_command, build_env, verbose) if verbose: print(build_command, '\n') - if hub_tag: + if success and hub_tag: hub_tag_command = f'{time_arg} docker tag {create_tag} {hub_tag}' print(f' ::: Tagging {create_tag} into {hub_tag}') - run_and_stream_command_output(hub_tag_command, build_env, verbose) + success = run_and_stream_command_output(hub_tag_command, build_env, verbose) + return success if __name__ == '__main__': args = docopt(__doc__, version='Dockerfile 1.1') - build_dockerfiles(args) + success = build_dockerfiles(args) + exit_code = 0 if success else 1 + sys.exit(exit_code) From 9c791f099f152c77383e3e02d5196d8973708573 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Fri, 17 Jul 2020 20:35:55 -0500 Subject: [PATCH 21/24] Allow internal PRs to push Signed-off-by: Adam Hill --- .github/workflows/test-and-build.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test-and-build.yaml b/.github/workflows/test-and-build.yaml index f518e6a..af400d8 100644 --- a/.github/workflows/test-and-build.yaml +++ b/.github/workflows/test-and-build.yaml @@ -6,6 +6,8 @@ on: - dev - v* - beta-v* + - release/* + tags: pull_request: #env: @@ -30,20 +32,20 @@ jobs: echo "Building ${ARCH}-${DEBIAN_VERSION}" ./gh-actions-test.sh - name: Push the ARCH image - if: github.event_name != 'pull_request' + if: github.repository == 'pihole/docker-pi-hole' run: | . gh-actions-vars.sh echo "${{ secrets.DOCKERHUB_PASS }}" | docker login --username="${{ secrets.DOCKERHUB_USER }}" --password-stdin docker push "${ARCH_IMAGE}" - name: Upload gh-workspace - if: github.event_name != 'pull_request' + if: github.repository == 'pihole/docker-pi-hole' uses: actions/upload-artifact@v1 with: name: gh-workspace path: .gh-workspace publish: - if: github.event_name != 'pull_request' + if: github.repository == 'pihole/docker-pi-hole' runs-on: ubuntu-latest needs: test-and-build steps: From 28bec779847a7373d1c7ecc6c867e57829f0618c Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Fri, 17 Jul 2020 20:51:19 -0500 Subject: [PATCH 22/24] fix repo name Signed-off-by: Adam Hill --- .github/workflows/test-and-build.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test-and-build.yaml b/.github/workflows/test-and-build.yaml index af400d8..6b92374 100644 --- a/.github/workflows/test-and-build.yaml +++ b/.github/workflows/test-and-build.yaml @@ -32,20 +32,20 @@ jobs: echo "Building ${ARCH}-${DEBIAN_VERSION}" ./gh-actions-test.sh - name: Push the ARCH image - if: github.repository == 'pihole/docker-pi-hole' + if: github.repository == 'pi-hole/docker-pi-hole' run: | . gh-actions-vars.sh echo "${{ secrets.DOCKERHUB_PASS }}" | docker login --username="${{ secrets.DOCKERHUB_USER }}" --password-stdin docker push "${ARCH_IMAGE}" - name: Upload gh-workspace - if: github.repository == 'pihole/docker-pi-hole' + if: github.repository == 'pi-hole/docker-pi-hole' uses: actions/upload-artifact@v1 with: name: gh-workspace path: .gh-workspace publish: - if: github.repository == 'pihole/docker-pi-hole' + if: github.repository == 'pi-hole/docker-pi-hole' runs-on: ubuntu-latest needs: test-and-build steps: From d96a7fe88740245a3f3203cea67500eef40676ac Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Fri, 17 Jul 2020 21:41:57 -0500 Subject: [PATCH 23/24] return to original to double check if it build branch pushes separately from PRs or not Signed-off-by: Adam Hill --- .github/workflows/test-and-build.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test-and-build.yaml b/.github/workflows/test-and-build.yaml index 6b92374..d25362e 100644 --- a/.github/workflows/test-and-build.yaml +++ b/.github/workflows/test-and-build.yaml @@ -32,20 +32,20 @@ jobs: echo "Building ${ARCH}-${DEBIAN_VERSION}" ./gh-actions-test.sh - name: Push the ARCH image - if: github.repository == 'pi-hole/docker-pi-hole' + if: github.event_name != 'pull_request' run: | . gh-actions-vars.sh echo "${{ secrets.DOCKERHUB_PASS }}" | docker login --username="${{ secrets.DOCKERHUB_USER }}" --password-stdin docker push "${ARCH_IMAGE}" - name: Upload gh-workspace - if: github.repository == 'pi-hole/docker-pi-hole' + if: github.event_name != 'pull_request' uses: actions/upload-artifact@v1 with: name: gh-workspace path: .gh-workspace publish: - if: github.repository == 'pi-hole/docker-pi-hole' + if: github.event_name != 'pull_request' runs-on: ubuntu-latest needs: test-and-build steps: From bba93dea3e33a0f95f7e61d6860ca4713fd13c99 Mon Sep 17 00:00:00 2001 From: Adam Hill Date: Fri, 17 Jul 2020 22:10:10 -0500 Subject: [PATCH 24/24] reset the arch image array between debian versions Signed-off-by: Adam Hill --- gh-actions-deploy.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gh-actions-deploy.sh b/gh-actions-deploy.sh index 66f0d01..51e3240 100755 --- a/gh-actions-deploy.sh +++ b/gh-actions-deploy.sh @@ -14,6 +14,7 @@ function annotate() { function create_manifest() { local debian_version=$1 + local images=() cd "${debian_version}" for arch in *; do @@ -64,7 +65,6 @@ echo "{}" | jq '.experimental="enabled"' | tee ~/.docker/config.json echo "${DOCKERHUB_PASS}" | docker login --username="${DOCKERHUB_USER}" --password-stdin docker info -images=() ls -lat ./.gh-workspace/ cd .gh-workspace