diff --git a/CHANGELOG.md b/CHANGELOG.md index fc1a0dd83..2701ea832 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,500 @@ +Changelog for restic 0.12.0 (2021-02-14) +======================================= + +The following sections list the changes in restic 0.12.0 relevant to +restic users. The changes are ordered by importance. + +Summary +------- + + * Fix #1681: Make `mount` not create missing mount point directory + * Fix #1800: Ignore `no data available` filesystem error during backup + * Fix #2563: Report the correct owner of directories in FUSE mounts + * Fix #2688: Make `backup` and `tag` commands separate tags by comma + * Fix #2739: Make the `cat` command respect the `--no-lock` option + * Fix #3087: The `--use-fs-snapshot` option now works on windows/386 + * Fix #3100: Do not require gs bucket permissions when running `init` + * Fix #3111: Correctly detect output redirection for `backup` command on Windows + * Fix #3151: Don't create invalid snapshots when `backup` is interrupted + * Fix #3166: Improve error handling in the `restore` command + * Fix #3232: Correct statistics for overlapping targets + * Fix #3014: Fix sporadic stream reset between rclone and restic + * Fix #3152: Do not hang until foregrounded when completed in background + * Fix #3249: Improve error handling in `gs` backend + * Chg #3095: Deleting files on Google Drive now moves them to the trash + * Enh #2186: Allow specifying percentage in `check --read-data-subset` + * Enh #2453: Report permanent/fatal backend errors earlier + * Enh #2528: Add Alibaba/Aliyun OSS support in the `s3` backend + * Enh #2706: Configurable progress reports for non-interactive terminals + * Enh #2944: Add `backup` options `--files-from-{verbatim,raw}` + * Enh #3083: Allow usage of deprecated S3 `ListObjects` API + * Enh #3147: Support additional environment variables for Swift authentication + * Enh #3191: Add release binaries for MIPS architectures + * Enh #909: Back up mountpoints as empty directories + * Enh #3250: Add several more error checks + * Enh #2718: Improve `prune` performance and make it more customizable + * Enh #2495: Add option to let `backup` trust mtime without checking ctime + * Enh #2941: Speed up the repacking step of the `prune` command + * Enh #3006: Speed up the `rebuild-index` command + * Enh #3048: Add more checks for index and pack files in the `check` command + * Enh #2433: Make the `dump` command support `zip` format + * Enh #3099: Reduce memory usage of `check` command + * Enh #3106: Parallelize scan of snapshot content in `copy` and `prune` + * Enh #3130: Parallelize reading of locks and snapshots + * Enh #3254: Enable HTTP/2 for backend connections + +Details +------- + + * Bugfix #1681: Make `mount` not create missing mount point directory + + When specifying a non-existent directory as mount point for the `mount` command, restic used + to create the specified directory automatically. + + This has now changed such that restic instead gives an error when the specified directory for + the mount point does not exist. + + https://github.com/restic/restic/issues/1681 + https://github.com/restic/restic/pull/3008 + + * Bugfix #1800: Ignore `no data available` filesystem error during backup + + Restic was unable to backup files on some filesystems, for example certain configurations of + CIFS on Linux which return a `no data available` error when reading extended attributes. These + errors are now ignored. + + https://github.com/restic/restic/issues/1800 + https://github.com/restic/restic/pull/3034 + + * Bugfix #2563: Report the correct owner of directories in FUSE mounts + + Restic 0.10.0 changed the FUSE mount to always report the current user as the owner of + directories within the FUSE mount, which is incorrect. + + This is now changed back to reporting the correct owner of a directory. + + https://github.com/restic/restic/issues/2563 + https://github.com/restic/restic/pull/3141 + + * Bugfix #2688: Make `backup` and `tag` commands separate tags by comma + + Running `restic backup --tag foo,bar` previously created snapshots with one single tag + containing a comma (`foo,bar`) instead of two tags (`foo`, `bar`). + + Similarly, the `tag` command's `--set`, `--add` and `--remove` options would treat + `foo,bar` as one tag instead of two tags. This was inconsistent with other commands and often + unexpected when one intended `foo,bar` to mean two tags. + + To be consistent in all commands, restic now interprets `foo,bar` to mean two separate tags + (`foo` and `bar`) instead of one tag (`foo,bar`) everywhere, including in the `backup` and + `tag` commands. + + NOTE: This change might result in unexpected behavior in cases where you use the `forget` + command and filter on tags like `foo,bar`. Snapshots previously backed up with `--tag + foo,bar` will still not match that filter, but snapshots saved from now on will match that + filter. + + To replace `foo,bar` tags with `foo` and `bar` tags in old snapshots, you can first generate a + list of the relevant snapshots using a command like: + + Restic snapshots --json --quiet | jq '.[] | select(contains({tags: ["foo,bar"]})) | .id' + + And then use `restic tag --set foo --set bar snapshotID [...]` to set the new tags. Please adjust + the commands to include real tag names and any additional tags, as well as the list of snapshots + to process. + + https://github.com/restic/restic/issues/2688 + https://github.com/restic/restic/pull/2690 + https://github.com/restic/restic/pull/3197 + + * Bugfix #2739: Make the `cat` command respect the `--no-lock` option + + The `cat` command would not respect the `--no-lock` flag. This is now fixed. + + https://github.com/restic/restic/issues/2739 + + * Bugfix #3087: The `--use-fs-snapshot` option now works on windows/386 + + Restic failed to create VSS snapshots on windows/386 with the following error: + + GetSnapshotProperties() failed: E_INVALIDARG (0x80070057) + + This is now fixed. + + https://github.com/restic/restic/issues/3087 + https://github.com/restic/restic/pull/3090 + + * Bugfix #3100: Do not require gs bucket permissions when running `init` + + Restic used to require bucket level permissions for the `gs` backend in order to initialize a + restic repository. + + It now allows a `gs` service account to initialize a repository if the bucket does exist and the + service account has permissions to write/read to that bucket. + + https://github.com/restic/restic/issues/3100 + + * Bugfix #3111: Correctly detect output redirection for `backup` command on Windows + + On Windows, since restic 0.10.0 the `backup` command did not properly detect when the output + was redirected to a file. This caused restic to output terminal control characters. This has + been fixed by correcting the terminal detection. + + https://github.com/restic/restic/issues/3111 + https://github.com/restic/restic/pull/3150 + + * Bugfix #3151: Don't create invalid snapshots when `backup` is interrupted + + When canceling a backup run at a certain moment it was possible that restic created a snapshot + with an invalid "null" tree. This caused `check` and other operations to fail. The `backup` + command now properly handles interruptions and never saves a snapshot when interrupted. + + https://github.com/restic/restic/issues/3151 + https://github.com/restic/restic/pull/3164 + + * Bugfix #3166: Improve error handling in the `restore` command + + The `restore` command used to not print errors while downloading file contents from the + repository. It also incorrectly exited with a zero error code even when there were errors + during the restore process. This has all been fixed and `restore` now returns with a non-zero + exit code when there's an error. + + https://github.com/restic/restic/issues/3166 + https://github.com/restic/restic/pull/3207 + + * Bugfix #3232: Correct statistics for overlapping targets + + A user reported that restic's statistics and progress information during backup was not + correctly calculated when the backup targets (files/dirs to save) overlap. For example, + consider a directory `foo` which contains (among others) a file `foo/bar`. When `restic + backup foo foo/bar` was run, restic counted the size of the file `foo/bar` twice, so the + completeness percentage as well as the number of files was wrong. This is now corrected. + + https://github.com/restic/restic/issues/3232 + https://github.com/restic/restic/pull/3243 + + * Bugfix #3014: Fix sporadic stream reset between rclone and restic + + Sometimes when using restic with the `rclone` backend, an error message similar to the + following would be printed: + + Didn't finish writing GET request (wrote 0/xxx): http2: stream closed + + It was found that this was caused by restic closing the connection to rclone to soon when + downloading data. A workaround has been added which waits for the end of the download before + closing the connection. + + https://github.com/rclone/rclone/issues/2598 + https://github.com/restic/restic/pull/3014 + + * Bugfix #3152: Do not hang until foregrounded when completed in background + + On Linux, when running in the background restic failed to stop the terminal output of the + `backup` command after it had completed. This caused restic to hang until moved to the + foreground. This has now been fixed. + + https://github.com/restic/restic/pull/3152 + https://forum.restic.net/t/restic-alpine-container-cron-hangs-epoll-pwait/3334 + + * Bugfix #3249: Improve error handling in `gs` backend + + The `gs` backend did not notice when the last step of completing a file upload failed. Under rare + circumstances, this could cause missing files in the backup repository. This has now been + fixed. + + https://github.com/restic/restic/pull/3249 + + * Change #3095: Deleting files on Google Drive now moves them to the trash + + When deleting files on Google Drive via the `rclone` backend, restic used to bypass the trash + folder required that one used the `-o rclone.args` option to enable usage of the trash folder. + This ensured that deleted files in Google Drive were not kept indefinitely in the trash folder. + However, since Google Drive's trash retention policy changed to deleting trashed files after + 30 days, this is no longer needed. + + Restic now leaves it up to rclone and its configuration to use or not use the trash folder when + deleting files. The default is to use the trash folder, as of rclone 1.53.2. To re-enable the + restic 0.11 behavior, set the `RCLONE_DRIVE_USE_TRASH` environment variable or change the + rclone configuration. See the rclone documentation for more details. + + https://github.com/restic/restic/issues/3095 + https://github.com/restic/restic/pull/3102 + + * Enhancement #2186: Allow specifying percentage in `check --read-data-subset` + + We've enhanced the `check` command's `--read-data-subset` option to also accept a + percentage (e.g. `2.5%` or `10%`). This will check the given percentage of pack files (which + are randomly selected on each run). + + https://github.com/restic/restic/issues/2186 + https://github.com/restic/restic/pull/3038 + + * Enhancement #2453: Report permanent/fatal backend errors earlier + + When encountering errors in reading from or writing to storage backends, restic retries the + failing operation up to nine times (for a total of ten attempts). It used to retry all backend + operations, but now detects some permanent error conditions so that it can report fatal errors + earlier. + + Permanent failures include local disks being full, SSH connections dropping and permission + errors. + + https://github.com/restic/restic/issues/2453 + https://github.com/restic/restic/issues/3180 + https://github.com/restic/restic/pull/3170 + https://github.com/restic/restic/pull/3181 + + * Enhancement #2528: Add Alibaba/Aliyun OSS support in the `s3` backend + + A new extended option `s3.bucket-lookup` has been added to support Alibaba/Aliyun OSS in the + `s3` backend. The option can be set to one of the following values: + + - `auto` - Existing behaviour - `dns` - Use DNS style bucket access - `path` - Use path style + bucket access + + To make the `s3` backend work with Alibaba/Aliyun OSS you must set `s3.bucket-lookup` to `dns` + and set the `s3.region` parameter. For example: + + Restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r + s3:https://oss-eu-west-1.aliyuncs.com/bucketname init + + Note that `s3.region` must be set, otherwise the MinIO SDK tries to look it up and it seems that + Alibaba doesn't support that properly. + + https://github.com/restic/restic/issues/2528 + https://github.com/restic/restic/pull/2535 + + * Enhancement #2706: Configurable progress reports for non-interactive terminals + + The `backup`, `check` and `prune` commands never printed any progress reports on + non-interactive terminals. This behavior is now configurable using the + `RESTIC_PROGRESS_FPS` environment variable. Use for example a value of `1` for an update + every second, or `0.01666` for an update every minute. + + The `backup` command now also prints the current progress when restic receives a `SIGUSR1` + signal. + + Setting the `RESTIC_PROGRESS_FPS` environment variable or sending a `SIGUSR1` signal + prints a status report even when `--quiet` was specified. + + https://github.com/restic/restic/issues/2706 + https://github.com/restic/restic/issues/3194 + https://github.com/restic/restic/pull/3199 + + * Enhancement #2944: Add `backup` options `--files-from-{verbatim,raw}` + + The new `backup` options `--files-from-verbatim` and `--files-from-raw` read a list of + files to back up from a file. Unlike the existing `--files-from` option, these options do not + interpret the listed filenames as glob patterns; instead, whitespace in filenames is + preserved as-is and no pattern expansion is done. Please see the documentation for specifics. + + These new options are highly recommended over `--files-from`, when using a script to generate + the list of files to back up. + + https://github.com/restic/restic/issues/2944 + https://github.com/restic/restic/issues/3013 + + * Enhancement #3083: Allow usage of deprecated S3 `ListObjects` API + + Some S3 API implementations, e.g. Ceph before version 14.2.5, have a broken `ListObjectsV2` + implementation which causes problems for restic when using their API endpoints. When a broken + server implementation is used, restic prints errors similar to the following: + + List() returned error: Truncated response should have continuation token set + + As a temporary workaround, restic now allows using the older `ListObjects` endpoint by + setting the `s3.list-objects-v1` extended option, for instance: + + Restic -o s3.list-objects-v1=true snapshots + + Please note that this option may be removed in future versions of restic. + + https://github.com/restic/restic/issues/3083 + https://github.com/restic/restic/pull/3085 + + * Enhancement #3147: Support additional environment variables for Swift authentication + + The `swift` backend now supports the following additional environment variables for passing + authentication details to restic: `OS_USER_ID`, `OS_USER_DOMAIN_ID`, + `OS_PROJECT_DOMAIN_ID` and `OS_TRUST_ID` + + Depending on the `openrc` configuration file these might be required when the user and project + domains differ from one another. + + https://github.com/restic/restic/issues/3147 + https://github.com/restic/restic/pull/3158 + + * Enhancement #3191: Add release binaries for MIPS architectures + + We've added a few new architectures for Linux to the release binaries: `mips`, `mipsle`, + `mips64`, and `mip64le`. MIPS is mostly used for low-end embedded systems. + + https://github.com/restic/restic/issues/3191 + https://github.com/restic/restic/pull/3208 + + * Enhancement #909: Back up mountpoints as empty directories + + When the `--one-file-system` option is specified to `restic backup`, it ignores all file + systems mounted below one of the target directories. This means that when a snapshot is + restored, users needed to manually recreate the mountpoint directories. + + Restic now backs up mountpoints as empty directories and therefore implements the same + approach as `tar`. + + https://github.com/restic/restic/issues/909 + https://github.com/restic/restic/pull/3119 + + * Enhancement #3250: Add several more error checks + + We've added a lot more error checks in places where errors were previously ignored (as hinted by + the static analysis program `errcheck` via `golangci-lint`). + + https://github.com/restic/restic/pull/3250 + + * Enhancement #2718: Improve `prune` performance and make it more customizable + + The `prune` command is now much faster. This is especially the case for remote repositories or + repositories with not much data to remove. Also the memory usage of the `prune` command is now + reduced. + + Restic used to rebuild the index from scratch after pruning. This could lead to missing packs in + the index in some cases for eventually consistent backends such as e.g. AWS S3. This behavior is + now changed and the index rebuilding uses the information already known by `prune`. + + By default, the `prune` command no longer removes all unused data. This behavior can be + fine-tuned by new options, like the acceptable amount of unused space or the maximum size of + data to reorganize. For more details, please see + https://restic.readthedocs.io/en/stable/060_forget.html . + + Moreover, `prune` now accepts the `--dry-run` option and also running `forget --dry-run + --prune` will show what `prune` would do. + + This enhancement also fixes several open issues, e.g.: - + https://github.com/restic/restic/issues/1140 - + https://github.com/restic/restic/issues/1599 - + https://github.com/restic/restic/issues/1985 - + https://github.com/restic/restic/issues/2112 - + https://github.com/restic/restic/issues/2227 - + https://github.com/restic/restic/issues/2305 + + https://github.com/restic/restic/pull/2718 + https://github.com/restic/restic/pull/2842 + + * Enhancement #2495: Add option to let `backup` trust mtime without checking ctime + + The `backup` command used to require that both `ctime` and `mtime` of a file matched with a + previously backed up version to determine that the file was unchanged. In other words, if + either `ctime` or `mtime` of the file had changed, it would be considered changed and restic + would read the file's content again to back up the relevant (changed) parts of it. + + The new option `--ignore-ctime` makes restic look at `mtime` only, such that `ctime` changes + for a file does not cause restic to read the file's contents again. + + The check for both `ctime` and `mtime` was introduced in restic 0.9.6 to make backups more + reliable in the face of programs that reset `mtime` (some Unix archivers do that), but it turned + out to often be expensive because it made restic read file contents even if only the metadata + (owner, permissions) of a file had changed. The new `--ignore-ctime` option lets the user + restore the 0.9.5 behavior when needed. The existing `--ignore-inode` option already turned + off this behavior, but also removed a different check. + + Please note that changes in files' metadata are still recorded, regardless of the command line + options provided to the backup command. + + https://github.com/restic/restic/issues/2495 + https://github.com/restic/restic/issues/2558 + https://github.com/restic/restic/issues/2819 + https://github.com/restic/restic/pull/2823 + + * Enhancement #2941: Speed up the repacking step of the `prune` command + + The repack step of the `prune` command, which moves still used file parts into new pack files + such that the old ones can be garbage collected later on, now processes multiple pack files in + parallel. This is especially beneficial for high latency backends or when using a fast network + connection. + + https://github.com/restic/restic/pull/2941 + + * Enhancement #3006: Speed up the `rebuild-index` command + + We've optimized the `rebuild-index` command. Now, existing index entries are used to + minimize the number of pack files that must be read. This speeds up the index rebuild a lot. + + Additionally, the option `--read-all-packs` has been added, implementing the previous + behavior. + + https://github.com/restic/restic/pull/3006 + https://github.com/restic/restic/issue/2547 + + * Enhancement #3048: Add more checks for index and pack files in the `check` command + + The `check` command run with the `--read-data` or `--read-data-subset` options used to only + verify only the pack file content - it did not check if the blobs within the pack are correctly + contained in the index. + + A check for the latter is now in place, which can print the following error: + + Blob ID is not contained in index or position is incorrect + + Another test is also added, which compares pack file sizes computed from the index and the pack + header with the actual file size. This test is able to detect truncated pack files. + + If the index is not correct, it can be rebuilt by using the `rebuild-index` command. + + Having added these tests, `restic check` is now able to detect non-existing blobs which are + wrongly referenced in the index. This situation could have lead to missing data. + + https://github.com/restic/restic/pull/3048 + https://github.com/restic/restic/pull/3082 + + * Enhancement #2433: Make the `dump` command support `zip` format + + Previously, restic could dump the contents of a whole folder structure only in the `tar` + format. The `dump` command now has a new flag to change output format to `zip`. Just pass + `--archive zip` as an option to `restic dump`. + + https://github.com/restic/restic/pull/2433 + https://github.com/restic/restic/pull/3081 + + * Enhancement #3099: Reduce memory usage of `check` command + + The `check` command now requires less memory if it is run without the `--check-unused` option. + + https://github.com/restic/restic/pull/3099 + + * Enhancement #3106: Parallelize scan of snapshot content in `copy` and `prune` + + The `copy` and `prune` commands used to traverse the directories of snapshots one by one to find + used data. This snapshot traversal is now parallized which can speed up this step several + times. + + In addition the `check` command now reports how many snapshots have already been processed. + + https://github.com/restic/restic/pull/3106 + + * Enhancement #3130: Parallelize reading of locks and snapshots + + Restic used to read snapshots sequentially. For repositories containing many snapshots this + slowed down commands which have to read all snapshots. + + Now the reading of snapshots is parallelized. This speeds up for example `prune`, `backup` and + other commands that search for snapshots with certain properties or which have to find the + `latest` snapshot. + + The speed up also applies to locks stored in the backup repository. + + https://github.com/restic/restic/pull/3130 + https://github.com/restic/restic/pull/3174 + + * Enhancement #3254: Enable HTTP/2 for backend connections + + Go's HTTP library usually automatically chooses between HTTP/1.x and HTTP/2 depending on + what the server supports. But for compatibility this mechanism is disabled if DialContext is + used (which is the case for restic). This change allows restic's HTTP client to negotiate + HTTP/2 if supported by the server. + + https://github.com/restic/restic/pull/3254 + + Changelog for restic 0.11.0 (2020-11-05) =======================================