clitools
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
7,871 Commits 9 Branches 44 Tags 201 MiB
Commit Graph

1648 Commits

Author SHA1 Message Date
Sebastian Hoß a8657bde68 Add --stdin-from-command option 
This new flag is added to the backup subcommand in order to allow restic to control the execution of a command and determine whether to save a snapshot if the given command succeeds.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
 2023-10-27 23:58:51 +02:00
Michael Eischer 731b3a4357 stats: fix hardlink tracking in a snapshot 
inodes are only unique within a device. Use the HardlinkIndex from the
restorer instead of the custom (broken) hashmap to correctly account for
both inode and deviceID.
 2023-10-27 23:40:42 +02:00
Michael Eischer 249605843b prune: get backend connection count via repository 2023-10-25 23:01:54 +02:00
Michael Eischer c7b770eb1f convert MemorizeList to be repository based 
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
 2023-10-25 23:01:35 +02:00
Michael Eischer 1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Alexander Neumann 27ec320eae Set development version for 0.16.1 2023-10-24 20:02:58 +02:00
Alexander Neumann baca3f6303 Add version for 0.16.1 2023-10-24 20:02:55 +02:00
Michael Eischer ead8dd0173
Merge pull request #4524 from MichaelEischer/fix-autocompletion 
Don't load password when autocompleting CLI commands
 2023-10-23 20:06:53 +00:00
Michael Eischer cd09ef4614 don't load password when autocompleting CLI commands 2023-10-23 21:44:22 +02:00
Michael Eischer 54a4034ec0 Only show index progress bar if stdout is a terminal 
This ensures that there is no interference when using restic for
scripting use cases.
 2023-10-23 21:16:52 +02:00
Michael Eischer d1d4510974 repair packs: Add stub feature flag implementation 2023-10-23 18:36:28 +02:00
Michael Eischer a28940ea29 check: Suggest usage of `restic repair packs` for corrupted blobs 
For now, the guide is only shown if the blob content does not match its
hash. The main intended usage is to handle data corruption errors when
using maximum compression in restic 0.16.0
 2023-10-23 18:36:28 +02:00
Michael Eischer db26dc75e1 repair packs: add experimental command 
This allows recovering a repository from several cases of damaged blobs.
 2023-10-23 18:36:28 +02:00
Michael Eischer ceff4af1ac Disable log output from automaxprocs library 2023-10-22 17:09:46 +02:00
Michael Gmelin 34f3b13b7c Make `key list` command honor `--no-lock` 
Fixes #4513
 2023-10-19 22:09:35 +02:00
Michael Eischer 495982232c
Merge pull request #4511 from dnnr/inode-in-json 
Include inode in JSON output for find and ls
 2023-10-17 21:06:34 +00:00
Michael Eischer f5d5e8fd0a automatically adjust GOMAXPROCS when run in a cpu-constrained container 2023-10-17 20:31:57 +02:00
Daniel Danner a858ab254b Include inode in JSON output for find and ls 
This introduces the inode attribute to the JSON output emitted for nodes
in `ls` and matches in `find`. There doesn't seem to be any discernible
reason to omit the inode and it can be useful in scripting scenarios.
 2023-10-11 21:10:01 +02:00
Michael Eischer 91aef00df3 check: add index loading progress bar 2023-10-01 19:55:29 +02:00
Michael Eischer b0da0f152f deduplicate index progress bar setup 2023-10-01 19:53:26 +02:00
Michael Eischer 75f6bd89ed hide index loading progress for JSON output 2023-10-01 19:53:26 +02:00
Michael Eischer 3fd0ad7448 repository: list index files only once 2023-10-01 19:53:26 +02:00
arjunajesh ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Michael Eischer e60c5b2d7f
Merge pull request #4436 from MichaelEischer/add-snapshot-subfolder-syntax-to-help-text 
Add snapshot subfolder syntax in help output of diff/dump/restore
 2023-08-13 19:36:38 +02:00
Michael Eischer a04964bb86
Merge pull request #4446 from 2000yeshu/feat-cat 
Check for arguments before opening remote connection in cat command
 2023-08-13 17:25:37 +00:00
Michael Eischer fe54912a46 cat: extract parameter validation and add a test 2023-08-13 19:17:46 +02:00
Yakul Garg 705556f134
Check for arguments before opening remote connection in cat command 
Signed-off-by: Yakul Garg <2000yeshu@gmail.com>
 2023-08-12 23:54:40 +05:30
Michael Eischer 0fca028491 restore: Fix help text for `--iinclude` and `--iexclude` 2023-08-05 18:21:03 +02:00
Michael Eischer 3a478bc522 unify snapshotID spelling in command parameters 2023-08-05 12:35:45 +02:00
Michael Eischer 9a7704fa2b diff/dump/restore: describe snapshot:subfolder syntax in help text 2023-08-05 12:35:08 +02:00
Alexander Neumann 82e6e28781 Set development version for 0.16.0 2023-07-31 20:24:29 +02:00
Alexander Neumann eb33e564c9 Add version for 0.16.0 2023-07-31 20:23:24 +02:00
Michael Eischer 3f919f2371 forget: simplify test 2023-07-28 19:21:40 +02:00
Michael Eischer 50b43fbac0 forget: replace `--keep-* -1` with `--keep-* unlimited` 
This ensures consistency with the `prune --max-unused unlimited` option.
 2023-07-28 19:21:40 +02:00
Michael Eischer 6c5fc32967
Merge pull request #3225 from wojas/restic-cmd-more-help 
Extend help output with short overview
 2023-07-23 15:43:03 +00:00
Michael Eischer ae13cf15c1 doc: move quickstart to docs 2023-07-23 17:32:37 +02:00
arjunajesh 32f5ee6f4e snapshots sorted by timestamp 2023-07-23 12:49:23 +02:00
Konrad Wojas 5bd8a6d7eb Extend help output with short overview 
The list of subcommands can be confusing for first time users. This
adds a short overview of the most important commands to help get people
started.
 2023-07-23 12:01:40 +02:00
Michael Eischer 25ff9fa893
Merge pull request #4334 from MichaelEischer/snapshot-subtree-syntax 
Add support for snapshot:path syntax
 2023-07-22 23:59:26 +02:00
Michael Eischer 76253b2a20 Hopefully fix flaky TestLockWaitCancel test 2023-07-22 23:23:17 +02:00
Michael Eischer b8f4267a36 lock: add more debug information to flaky test 2023-07-22 23:23:17 +02:00
Michael Eischer 1f6883a05c lock: test stale lock refresh 2023-07-22 23:23:17 +02:00
Michael Eischer 8154f6a77a Ensure consistent naming for `<snapshot>:<subfolder>` syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 233b841ad9 Add cat tree snapshot:subpath command 
This command prints the tree for the given path.
 2023-07-22 19:55:57 +02:00
Michael Eischer 85860e6e97 Add support for snapshot:subpath syntax 
This snapshot specification syntax is supported by the cat, diff, dump,
ls and restore command.
 2023-07-22 19:55:57 +02:00
Michael Eischer 2dd6769429 lock: Fix possible deadlock during refresh of stale lock 
A delayed lock refresh could send a signal on the `refreshed` channel
while the `monitorLockRefresh` goroutine waits for a reply to its
`refreshLockRequest`. As the channels are unbuffered, this resulted in a
deadlock.
 2023-07-22 19:55:44 +02:00
Michael Eischer 399f8e84a1 lock: use mem backend for locking tests 
This hopefully fixes the timing related test failures which appear to be
caused by very slow fsync calls.
 2023-07-22 19:55:44 +02:00
Michael Eischer 6a436d731d lock: rename test helper function 2023-07-22 19:55:44 +02:00
Michael Eischer b2fcbc21cb lock: rework stale lock refresh to avoid data race 2023-07-22 19:55:44 +02:00
Michael Eischer 05e5e29a8c backup: don't pretend to lock repo for dry run 
The dry run did not actually create lock files, but still ran all other
parts of the lock processing.
 2023-07-22 19:55:44 +02:00
Michael Eischer f490288738 lock: freeze backend operations while refreshing stale lock 
Freeze new backend operations while trying to refresh a stale lock.
 2023-07-22 19:55:44 +02:00
Michael Eischer 11eb88a2ea lock: cleanup 2023-07-22 19:55:44 +02:00
Michael Eischer bee3231ed4 lock: try refreshing of stale locks 
A stale lock may be refreshed if it continues to exist until after a
replacement lock has been created. This ensures that a repository was
not unlocked in the meantime.
 2023-07-22 19:55:44 +02:00
Michael Eischer 47206a6579
Merge pull request #4351 from kenny-y-dev/human-readable-ls 
Add --human-readable to ls and find output
 2023-07-16 11:44:41 +00:00
Michael Eischer 89fbd39e59 Don't print stacktrace on invalid cacert option 2023-07-09 14:23:54 +02:00
Michael Eischer 1ce599d2ae Fix handling of empty cacert environment variable 
This resulted in a "empty filename for root certificate supplied" error.
 2023-07-09 14:23:54 +02:00
Michael Eischer 325fa916b5 stats: Add debug mode to collect repository statistics 2023-07-08 23:04:47 +02:00
Michael Eischer cc3c218baf small cleanups for certificate environment variables 2023-07-08 09:44:59 +02:00
arjunajesh c1578a2035 certificates can be passed through env vars 2023-07-08 09:44:59 +02:00
Michael Eischer 389f6ee74c backup: add minimal test for program versioni 2023-07-07 23:33:23 +02:00
Michael Eischer bbac74b172 add program version to snapshot 2023-07-07 23:27:10 +02:00
greatroar 41a5bf357f cmd, ui: Move size parsing code and make it more robust 2023-07-02 21:44:50 +02:00
Michael Eischer cbf87fbdb3 init: don't include password in debug log 2023-06-17 15:15:58 +02:00
Michael Eischer 50e0d5e6b5 backend: Hardcode backend scheme in Factory 
Our ParseConfig implementations always expect a specific scheme, thus no
other scheme would work.
 2023-06-17 15:15:58 +02:00
Michael Eischer 3a3cf608f5 b2/s3: Move config validation from ApplyEnvironment to Open/Create 
Conceptually the backend configuration should be validated when creating
or opening the backend, but not when filling in information from
environment variables into the configuration.
 2023-06-17 15:15:58 +02:00
Michael Eischer 9aa9e0d1ec local/sftp: move limiter setup into backend 2023-06-17 15:15:58 +02:00
Michael Eischer 7d12c29286 backend: Unify backend construction using factory and registry 
This unified construction removes most backend-specific code from
global.go. The backend registry will also enable integration tests to
use custom backends if necessary.
 2023-06-17 15:15:57 +02:00
Michael Eischer 56836364a4 backend: pass context into every backend constructor 2023-06-17 15:15:57 +02:00
Michael Eischer 4df77e9f26
Merge pull request #4314 from MichaelEischer/restore-json 
Add JSON support for the restore progress bar
 2023-06-16 23:53:17 +02:00
Michael Eischer 191c47d30e
Merge pull request #4353 from MichaelEischer/tune-gc 
Tune Go garbage collector
 2023-06-16 23:24:39 +02:00
Michael Eischer cf16239058 restore: print output via termStatus 2023-06-09 12:03:33 +02:00
Michael Eischer d54176ce5d restore: add basic json progress 2023-06-09 12:03:33 +02:00
Michael Eischer a9aff885d6 restore: reorganize progress bar code 
The structure is now much more similar to that of the backup command.
 2023-06-09 12:03:33 +02:00
Michael Eischer bb20078641 restore: pass termStatus to restore in tests 2023-06-09 12:03:33 +02:00
Michael Eischer 9464c63550 Make formatNode test timezone independent 
formatNode formats the timestamp according to the current time zone. Pin
the local timezone to UTC to ensure the test works everywhere.
 2023-06-08 19:18:30 +02:00
Michael Eischer 6ebf2dd235 Reduce duplicate code in test for fomatNode 2023-06-08 19:16:16 +02:00
Michael Eischer 5f153109ba Refactor formatNode 2023-06-08 19:12:49 +02:00
Michael Eischer 32a6b66267 backend: add standardized Config.ApplyEnvironment 
This removes the backend specific special cases while parsing the
configuration in `global.go`.
 2023-06-07 22:31:15 +02:00
Michael Eischer f903db492c backend: let ParseConfig return a Config pointer 
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.

An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
 2023-06-07 22:31:15 +02:00
Michael Eischer 2f7b4ceae1 backend: Move environment based configuration into backend 2023-06-07 22:30:35 +02:00
Michael Eischer 49a6a4f5bf
Merge pull request #4356 from MichaelEischer/fix-unlock-hint 
lock: fix missing hint to unlock command if repository is locked
 2023-06-06 21:03:06 +02:00
Michael Eischer 237d00000e lock: fix missing hint to unlock command if repository is locked 2023-06-02 23:16:49 +02:00
Michael Eischer 2fcb3947df prune: trigger GC after prune planning 2023-06-02 21:57:40 +02:00
Michael Eischer f1b73c9301 Reduce GOGC to 50 
The index used by restic consumes a major part of the total memory
usage. This leads to an unnecessarily large amount of memory that
contains ephemeral objects that are only used for a short time.
 2023-06-02 21:51:50 +02:00
Kenny Y 8812dcd56a
Add --human-readable to ls and find output 
Modifies format module to add options for human readable storage size formatting, using size parsing already in ui/format.
Cmd flag --human-readable added to ls and find commands.
Additional option added to formatNode to support printing size in regular or new human readable format
 2023-06-01 21:18:18 -04:00
Michael Eischer 88a10a368f
Merge pull request #4342 from greatroar/errors 
cmd: Don't check for errno == 0
 2023-05-31 22:17:07 +02:00
greatroar 8f1ca8fabe cmd: Don't check for errno == 0 
Maybe term.Restore's predecessor could return this, but the current
implementation never does.
 2023-05-27 08:52:37 +02:00
Michael Eischer 9747cef338 fix linter warnings 2023-05-26 19:38:44 +02:00
Michael Eischer 8e913e6d3a repair index: always read Quiet flags from GlobalOptions passed as parameter 2023-05-26 19:38:44 +02:00
Michael Eischer b93459cbb0 repair snapshots: use local copy of globalOptions to open repository 2023-05-26 19:38:44 +02:00
Michael Eischer 692f81ede8 cleanup prune integration test 2023-05-26 19:38:44 +02:00
Michael Eischer 7a268e4aba always access stdout/stderr via globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer 4b3a0b4104 read JSON/Quiet flag from the passed in globalOptions 2023-05-26 19:38:44 +02:00
Michael Eischer cebce52c16 test: add helper to capture stdout for integration tests 2023-05-26 19:38:44 +02:00
Michael Eischer 675a49a95b Restructure integration tests 
The tests are now split into individual files for each command. The
separation isn't perfect as many tests make use of multiple commands. In
particular `init`, `backup`, `check` and `list` are used by a larger
number of test cases.

Most tests now reside in files name cmd_<name>_integration_test.go. This
provides a certain indication which commands have significant test
coverage.
 2023-05-26 19:38:43 +02:00
Michael Eischer e2dba9f5c7 test: cleanup a some check calls 2023-05-26 19:34:57 +02:00
Michael Eischer 06fd6b54d7 test: print log output if testRunCheck fails 2023-05-26 19:34:57 +02:00
Michael Eischer 419e6f26b1 tests: Simplify checks that a specific number of snapshots exists 2023-05-26 19:34:57 +02:00
Michael Eischer c3212ab6a6 test: use standard logging methods from testing for the test helpers 
Use the logging methods from testing.TB to make use of tb.Helper(). This
allows the tests to log the filename and line number in which the test
helper was called. Previously the test helper was logged which is rarely
useful.
 2023-05-26 19:34:57 +02:00
Michael Eischer a466e945d9 stats: pass StatsOptions via parameter 2023-05-18 21:17:53 +02:00
Michael Eischer 03b9764bce init/generate: check that no parameters are passed 2023-05-18 21:17:53 +02:00
Michael Eischer 1e33b285c1 lock: remove unused parameter 2023-05-18 21:17:53 +02:00
Michael Eischer c05f96e6b9 repair index: remove no longer used paramter 2023-05-18 21:17:53 +02:00
Michael Eischer 94752b7ee2 backup: Remove unused parameters from rejectFunc collection 2023-05-18 21:17:53 +02:00
Michael Eischer 472bf5184f Replace lots of unused parameters with _ 
The parameters are required by the implemented function signature or interface.
 2023-05-18 21:17:53 +02:00
Michael Eischer 1514593f22 Remove unused context or testing parameters 2023-05-18 21:17:53 +02:00
Michael Eischer 5e4e268bdc Use _ as parameter name for unused Context 
The context is required by the implemented interface.
 2023-05-18 21:15:45 +02:00
Michael Eischer 319087c056 Remove redundant if ...; err != nil; return pattern 2023-05-18 21:15:45 +02:00
Michael Eischer e01baeabba Use either test or rtest to refer to internal test helpers 
A single test file should not use both names.
 2023-05-18 21:15:45 +02:00
Michael Eischer 5773b86d02 repository: Push all usage of errors.Fatal out of the package 
As the `Fatal` error type only includes a string, it becomes impossible
to inspect the contained error. This is for a example a problem for the
fuse implementation, which must be able to detect context.Canceled
errors.

Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
 2023-05-18 17:27:41 +02:00
Michael Eischer ee3c55ea3d
Merge pull request #2876 from aawsome/new-repair-command 
Add repair command
 2023-05-05 23:22:24 +02:00
Michael Eischer 78e5aa6d30 repair snapshots: add basic tests 2023-05-04 23:00:46 +02:00
Michael Eischer 1cb11ad8ad mount: enable debug logging for the flaky TestMount test 
The test case fails from time to time with an Input/Output error while
trying to access the snapshots directory.
 2023-05-01 18:03:17 +02:00
Michael Eischer 5aa37acdaa repair snapshots: cleanup command help 2023-05-01 16:06:17 +02:00
Michael Eischer 7c8dd61e8c repair snapshots: cleanup warnings 2023-05-01 15:22:30 +02:00
Michael Eischer f6cc10578d repair snapshots: Always sanitize file nodes 
If the node for a file is intact, this is a no-op.
 2023-05-01 15:22:18 +02:00
Michael Eischer 4ce87a7f64 repair snapshots: port to filterAndReplaceSnapshot 
The previous approach of rewriting all snapshots first, then flushing
the repository data and finally removing old snapshots has the downside
that an interrupted command execution leaves behind broken snapshots as
not all new data is already flushed.
 2023-05-01 15:22:03 +02:00
Michael Eischer e17ee40a31 repair snapshots: Port to use walker.TreeRewriter 2023-05-01 15:20:36 +02:00
Michael Eischer 1bd1f3008d walker: extend TreeRewriter to support snapshot repairing 
This adds support for caching already rewritten trees, handling of load
errors and disabling the check that the serialization doesn't lead to
data loss.
 2023-05-01 15:20:24 +02:00
Michael Eischer 38dac78180 walker: restructure FilterTree into TreeRewriter 
The more generic RewriteNode callback replaces the SelectByName and
PrintExclude functions. The main part of this change is a preparation to
allow using the TreeRewriter for the `repair snapshots` command.
 2023-05-01 15:20:12 +02:00
Michael Eischer 8c4caf09a8 repair snapshots: Do not rename repaired files 
The files in a tree must be sorted in lexical order. However, this
cannot be guaranteed when appending a filename suffix. For two files

file, file.rep

where "file" is broken, this would result in

file.repaired, file.rep

which is no longer sorted.

In addition, adding a filename suffix is also prone to filename
collisions which would require a rather complex search for a
collision-free name in order to work reliably.
 2023-05-01 15:19:36 +02:00
Michael Eischer 375189488c rewrite: prepare for code sharing with rewrite snapshots 2023-05-01 15:19:24 +02:00
Michael Eischer 903651c719 repair snapshots: partially synchronize code with rewrite command 
Simplify CLI options:
* Rename "DeleteSnapshots" to "Forget"
* Replace "AddTag" and "Append" with hardcoded values

Change output and snapshot modifications to be more in line with the
"rewrite" command.
 2023-05-01 15:19:11 +02:00
Michael Eischer 118d599d0a Rename 'rebuild-index' to 'repair index' 
The old name still works, but is deprecated.
 2023-05-01 15:16:44 +02:00
Michael Eischer db459eda21 move to subcommand 2023-05-01 15:15:48 +02:00
Michael Eischer a14a63cd29 modernize code 2023-05-01 15:12:09 +02:00
Alexander Weiss 947f0c345e correct typos 2023-05-01 14:56:42 +02:00
Alexander Weiss d23a2e1925 better error handling and correct nil tree behavior 2023-05-01 14:56:42 +02:00
Alexander Weiss 08ae708b3b make linter happy 2023-05-01 14:56:42 +02:00
Alexander Weiss 5f58797ba7 Add repair command 2023-05-01 14:56:42 +02:00
Michael Eischer 8d971172c4
Merge pull request #4306 from MichaelEischer/document-restic-compression-env 
Document that the compression mode can be set via $RESTIC_COMPRESSION
 2023-04-30 18:26:10 +02:00
Michael Eischer face5bd7f7 Document that the compression mode can be set via $RESTIC_COMPRESSION 2023-04-30 16:11:53 +02:00
Michael Eischer 1daf928a77
Merge pull request #4305 from MichaelEischer/stracktrace-for-windows 
Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
 2023-04-30 16:08:58 +02:00
Michael Eischer c8641f4479 Merge branch 'patch-release' 2023-04-24 20:44:38 +02:00
Alexander Neumann ac7ac0cb97 Set development version for 0.15.2 2023-04-24 20:28:37 +02:00
Alexander Neumann be8be3397c Add version for 0.15.2 2023-04-24 20:28:37 +02:00
Michael Eischer 306a29980a Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set 
The builtin mechanism to capture a stacktrace in Go is to send a SIGQUIT
to the running process. However, this mechanism is not avaiable on
Windows. Thus, tweak the SIGINT handler to dump a stacktrace if the
environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT` is set.
 2023-04-23 15:50:40 +02:00
Michael Eischer 22562d2132
Merge pull request #4300 from MichaelEischer/less-flaky-tests 
Increase timeouts for lock refresh tests
 2023-04-23 11:24:59 +02:00
Michael Eischer 51d823348d
Merge pull request #4286 from MichaelEischer/backend-cleanup-logging 
Normalize backend logging and connection limiting
 2023-04-23 11:24:26 +02:00
Michael Eischer 179e11c2ae Increase timeouts for lock refresh tests 
When saving files to the local backend, in some cases the used fsync
calls are slow enough to cause the tests to time out. Thus, increase the
test timeouts as a stopgap measure until we can use the mem backend for
these tests.
 2023-04-22 12:45:59 +02:00
Michael Eischer ebba233a3a backend/sema: rename constructor to NewBackend 2023-04-22 12:32:57 +02:00
Michael Eischer 1dd873b706
Merge pull request #4293 from MichaelEischer/fix-groups-by 
Fix parent snapshot selection for relative paths
 2023-04-21 22:47:03 +02:00
Michael Eischer 7a60d9e54f
Merge pull request #4288 from MichaelEischer/log-warnings-to-debug-log 
Add warnings via Warnf to the debug log
 2023-04-21 22:39:00 +02:00
Michael Eischer 3001dd8c2b Add test to verify that the backup parent is correctly selected 2023-04-21 22:35:02 +02:00
Michael Eischer 09cddb8927 rewrite: log snapshot saved before removal of the old snapshot 
The snapshot was already saved before removing the old snapshot. Only
the log messages were printed in the wrong order.
 2023-04-17 21:00:45 +02:00
Michael Eischer 913eab3361
Merge pull request #4234 from thndrbrrr/forget-opts-neg1-means-forever-issue-2565 
restic forget --keep-* options will interpret -1 as "forever"
 2023-04-14 23:18:47 +02:00
Michael Eischer a9c7c12276
Merge pull request #4265 from HeikoSchlittermann/chg/progress-to-stderr 
generate: do not write progress to STDOUT
 2023-04-14 23:18:23 +02:00
Michael Eischer 85eef232e6
Merge pull request #4232 from thndrbrrr/copy-chunker-params-verbose-msg 
init: Add --copy-chunker-params verbose msg
 2023-04-14 23:17:56 +02:00
Michael Eischer c934c99d41 gs: replace usage of context.Background() 2023-04-14 22:32:15 +02:00
Michael Eischer 8e1e3844aa backend: factor out connection limiting and parameter validation 
The SemaphoreBackend now uniformly enforces the limit of concurrent
backend operations. In addition, it unifies the parameter validation.

The List() methods no longer uses a semaphore. Restic already never runs
multiple list operations in parallel.

By managing the semaphore in a wrapper backend, the sections that hold a
semaphore token grow slightly. However, the main bottleneck is IO, so
this shouldn't make much of a difference.

The key insight that enables the SemaphoreBackend is that all of the
complex semaphore handling in `openReader()` still happens within the
original call to `Load()`. Thus, getting and releasing the semaphore
tokens can be refactored to happen directly in `Load()`. This eliminates
the need for wrapping the reader in `openReader()` to release the token.
 2023-04-14 22:32:15 +02:00
Michael Eischer 4703473ec5 backend: extract most debug logs into logger backend 2023-04-14 22:32:15 +02:00
Michael Eischer ba16904eed backup: Add test to verify parent snapshot selection for relative paths 2023-04-14 22:21:43 +02:00
Michael Eischer fab4a8a4d2 Properly initialize the --group-by option for backup tests 2023-04-14 21:53:01 +02:00
Torben Giesselmann 6d6c04abef forget: Simplify usage text 2023-04-14 10:05:23 -07:00
greatroar 97274ecabd cmd, restic: Refactor and fix snapshot filtering 
This turns snapshotFilterOptions from cmd into a restic.SnapshotFilter
type and makes restic.FindFilteredSnapshot and FindFilteredSnapshots
methods on that type. This fixes #4211 by ensuring that hosts and paths
are named struct fields instead of unnamed function arguments in long
lists of such.

Timestamp limits are also included in the new type. To avoid too much
pointer handling, the convention is that time zero means no limit.
That's January 1st, year 1, 00:00 UTC, which is so unlikely a date that
we can sacrifice it for simpler code.
 2023-04-13 22:51:45 +02:00
Michael Eischer 74f7dd0b38 Make help for --verbose less confusing 
The output is now
```
-v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
instead of
```
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
 2023-04-13 22:46:37 +02:00
Heiko Schlittermann (HS12-RIPE) 087cf7e114
generate: write progress to STDOUT if this is a terminal 
This allows usage as:

  eval "$(restic generated --bash-completion /dev/stdout)"
 2023-04-12 22:39:59 +02:00
Michael Eischer e604939e72 Debug log status code if execution is interrupted 
Currently, there is no clear indication in the debug log whether restic
exited or not.
 2023-04-12 21:58:35 +02:00
Michael Eischer 37aca6bec0 Add warnings via Warnf to the debug log 2023-04-12 21:37:37 +02:00
Torben Giesselmann 71537da4b0 init: Add --copy-chunker-params verbose msg 2023-04-09 14:59:07 -07:00
Torben Giesselmann ce51d2f3c0 forget: Update usage text 2023-04-09 12:59:15 -07:00
Michael Eischer 1f43003cc1 Merge branch 'master' into feature-wait-for-unlock 2023-04-07 20:32:21 +02:00
Michael Eischer 8ce5f29758 lock: increase test timeout tolerances to avoid test failures 2023-04-07 20:07:05 +02:00
Michael Eischer cf1cc1fb72 lock: Print stacktrace if TestLockSuccessfulRefresh fails 2023-04-07 19:51:31 +02:00
Michael Eischer 64233ca0a7 lock: Improve debug logging in the test 2023-04-07 19:43:00 +02:00
jo ea59896bd6
Add a global option --retry-lock 
Fixes restic#719

If the option is passed, restic will wait the specified duration of time
and retry locking the repo every 10 seconds (or more often if the total
timeout is relatively small).

- Play nice with json output
- Reduce wait time in lock tests
- Rework timeout last attempt
- Reduce test wait time to 0.1s
- Use exponential back off for the retry lock
- Don't pass gopts to lockRepo functions
- Use global variable for retry sleep setup
- Exit retry lock on cancel
- Better wording for flag help
- Reorder debug statement
- Refactor tests
- Lower max sleep time to 1m
- Test that we cancel/timeout in time
- Use non blocking sleep function
- Refactor into minDuration func

Co-authored-by: Julian Brost <julian@0x4a42.net>
 2023-04-07 16:24:14 +02:00
Mark Herrmann f875a8843d restore: Add progress bar 
Co-authored-by: Mark Herrmann <mark.herrmann@mailbox.org>
 2023-04-07 12:08:23 +02:00
Torben Giesselmann 84ede6ad7a forget: Prevent neg. values in --keep-within* opts 2023-03-14 19:20:03 -07:00
Torben Giesselmann 667536cea4 forget: Allow neg. values in "--keep-within*" opts 2023-03-05 14:18:08 -08:00
Torben Giesselmann b77b0749fa forget: Treat -1 as forever for all "last n" opts 2023-03-04 19:01:37 -08:00
Torben Giesselmann 6aca7dac21 forget: Verify forget opts 2023-03-03 19:10:11 -08:00
Ben Wiederhake 9888443f5c check: Document behavior of --with-cache a bit better 
Also see https://forum.restic.net/t/how-to-speed-up-tiny-incremental-checks/5905
 2023-02-19 17:45:47 +01:00
greatroar a7786c67f1 cmd, restic: Refactor and fix snapshot filtering 
This turns snapshotFilterOptions from cmd into a restic.SnapshotFilter
type and makes restic.FindFilteredSnapshot and FindFilteredSnapshots
methods on that type. This fixes #4211 by ensuring that hosts and paths
are named struct fields instead of unnamed function arguments in long
lists of such.

Timestamp limits are also included in the new type. To avoid too much
pointer handling, the convention is that time zero means no limit.
That's January 1st, year 1, 00:00 UTC, which is so unlikely a date that
we can sacrifice it for simpler code.
 2023-02-19 15:04:25 +01:00
Michael Eischer 0ce182f044 document backup --group-by 2023-02-11 00:55:11 +01:00
Michael Eischer 2885db7902 backup: add group-by option 2023-02-10 23:18:14 +01:00
Michael Eischer acb40d2b94 Refactor group-by to parse options into a struct 2023-02-10 23:18:14 +01:00
Michael Eischer 2d700c3887 Make help for --verbose less confusing 
The output is now
```
-v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
instead of
```
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
 2023-02-06 22:11:21 +01:00
greatroar d129baba7a repository: Reuse buffers in Repository.LoadUnpacked 
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
 2023-01-30 22:01:01 +01:00
Alexander Neumann febb32b5b4 Set development version for 0.15.1 2023-01-30 20:43:52 +01:00
Alexander Neumann 7d4b7ad9cb Add version for 0.15.1 2023-01-30 20:43:47 +01:00
Michael Eischer 3b24e0ac55
Merge pull request #4154 from MichaelEischer/cleanup-prune-tests 
deduplicate prune tests a bit
 2023-01-26 22:39:15 +01:00
Alexander Neumann 65923e9c26
Merge pull request #4152 from MichaelEischer/invalid-locks 
Improve handling of invalid locks
 2023-01-25 08:19:53 +01:00
Michael Eischer 00216d54a1 deduplicate prune tests a bit 2023-01-14 19:05:45 +01:00
Michael Eischer 1f3f68b2c0 prune: Don't show "packs processed" for quiet runs 2023-01-14 18:48:32 +01:00
Michael Eischer 20ad14e362 lock: add help message how to recover from invalid locks 2023-01-14 18:04:22 +01:00
Michael Eischer c995b5be52 lock: cleanup error message 
The error message is now `Fatal: unable to create lock in backend:
[...]` instead of `unable to create lock in backend: Fatal: [...]`.
 2023-01-14 17:57:02 +01:00
Michael Eischer 4a7a6b06af ui/backup: Use progress.Updater for progress updates 2023-01-14 01:20:43 +01:00
Michael Eischer e499bbe3ae progress: extract progress updating into Updater struct 
This allows reusing the code to create periodic progress updates.
 2023-01-14 01:13:08 +01:00
Michael Eischer c15b4bceae backup: extract StdioWrapper from ProgressPrinters 
The StdioWrapper is not used at all by the ProgressPrinters. It is
called a bit earlier than previously. However, as the password prompt
directly accessed stdin/stdout this doesn't cause problems.
 2023-01-14 00:58:13 +01:00
Alexander Neumann 74348be3fa Set development version for 0.15.0 2023-01-12 20:51:23 +01:00
Alexander Neumann 37d0e1fe58 Add version for 0.15.0 2023-01-12 20:51:19 +01:00
Michael Eischer c354b55e62
Merge pull request #4136 from restic/doc-small-files 
doc: Clarify text about tuning backups for small files
 2023-01-08 22:00:09 +01:00
Leo R. Lundgren 6306797238 doc: Clarify text about tuning backups for small files 2023-01-08 21:49:55 +01:00
Michael Eischer 732184a849 Correct maximum verbosity level in help message 
The maximum for `--verbose=n` is n=2. Internally it is translated into a
scale from 0 to 3. However, the default (without verbose) is 1, thus the
verbosity level can only be increased two times.
 2023-01-07 22:02:13 +01:00
Michael Eischer 7a36306901 forget: Clarify log message for `--dry-run --prune` 2023-01-04 00:44:46 +01:00
Michael Eischer b404ad4eaa prune: make it clearer when prune is used in dry-run mode 2023-01-04 00:44:46 +01:00
Michael Eischer e02a10c58a
Merge pull request #4109 from MichaelEischer/fix-prune-uncompressed-accounting 
prune: Fix calculation of remaining uncompressed data
 2023-01-03 23:28:10 +01:00
Michael Eischer 81dc8c8d13 prune: Fix calculation of remaining uncompressed data 
Only the repacking of *un*compressed packs reduces the amount of
uncompressed data. Previously the counter even overflowed for fully
compressed repositories.
 2023-01-03 22:34:36 +01:00
Michael Eischer 0de3b24756
Merge pull request #4110 from MichaelEischer/remove-exitf 
Remove Exitf function
 2022-12-29 12:07:51 +01:00
Michael Eischer 68b1f30733 Remove Exitf function 
Commands should use the normal shutdown path. In addition, the Exitf
function was only used by `dump` and `restore` but not any other command
which introduces the risk of inconsistent behavior.
 2022-12-28 21:42:38 +01:00