clitools
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
5,024 Commits 11 Branches 44 Tags 203 MiB
Commit Graph

5024 Commits

Author SHA1 Message Date
Alexander Neumann c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00
Alexander Neumann f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo 
Replace most usages of context.TODO()
 2020-10-10 15:03:44 +02:00
MichaelEischer 164d8af3dd
Merge pull request #2906 from kitone/fix-inconsistent-timestamps-permissions 
Restore inconsistent timestamps permissions
 2020-10-10 14:56:43 +02:00
kitone 052564007a Add changelog 2020-10-10 14:37:51 +02:00
kitone 6099f81692 Fix #1212 restore code produces inconsistent timestamps/permissions. 
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
 2020-10-10 13:46:44 +02:00
kitone 9d7f616190 Improve restorer debug log information 2020-10-10 13:46:43 +02:00
kitone 295ddb9e57 Add test case for inconsistent timestamps and permissions restoration 
Reproduce from https://github.com/restic/restic/issues/1212
 2020-10-10 13:46:43 +02:00
Michael Eischer e638b46a13 Embed context into ReaderAt 
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
 2020-10-09 22:39:07 +02:00
Michael Eischer d6cfe857b7 pass proper context into MasterIndex.RebuildIndex 2020-10-09 22:39:07 +02:00
Michael Eischer 2964d2ad15 Skip unlocking for nil locks 
Now that lockRepo receives a context, it is possible that it is canceled
before a lock was created. Thus `unlockRepo` must be able to handle this
case.
 2020-10-09 22:39:06 +02:00
Michael Eischer 0c9efa9c2a Pass context to lockRepo 2020-10-09 22:39:06 +02:00
Michael Eischer 37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer 4b0fcaed45 unlock: use proper context for locks cleanup 
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
 2020-10-09 22:37:56 +02:00
Michael Eischer 645a6efaf2 restorer: remove redundant type specification 2020-10-09 22:37:56 +02:00
Michael Eischer dc2e664209 integration_fuse_test: use global context 
No need to use the TODO context.
 2020-10-09 22:37:56 +02:00
Michael Eischer a449450021 init: pass proper context to master key generation 
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
 2020-10-09 22:37:56 +02:00
Michael Eischer 603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer 27456f6545 debug: use proper context 
This allows the debug commands to be properly interrupted.
 2020-10-09 22:37:56 +02:00
Michael Eischer c458e114d4 pass context to Find / FindSnapshot 
This allows proper interruption of restic while it searches for
snapshots or key files.
 2020-10-09 22:37:56 +02:00
Michael Eischer 45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer 307a6ba3a3 Upgrade minio sdk to v7 
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
 2020-10-09 22:37:24 +02:00
Michael Eischer 50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann 5fd3dbccb7
Merge pull request #2963 from MichaelEischer/fix-status-deadlock 
backup: Fix possible deadlock of scanner goroutine
 2020-10-09 21:35:50 +02:00
Alexander Neumann 35655a481b
Merge pull request #2931 from MichaelEischer/reduce-debug-overhead 
Reduce debug log overhead
 2020-10-09 21:35:11 +02:00
Alexander Neumann 30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create 
Call rclone.Create to create a new repository for the rclone backend
 2020-10-09 21:29:15 +02:00
Alexander Neumann 2f3eeff2e7
Merge pull request #2964 from MichaelEischer/fix-repro-instructions 
Fix instructions to reproduce the release binaries
 2020-10-09 21:28:24 +02:00
MichaelEischer 356ac404cd
Merge pull request #2996 from greatroar/setconsolecursorposition 
Use windows.SetConsoleCursorPosition in ui/termstatus
 2020-10-09 18:19:02 +02:00
rawtaz ab769abeaf
Merge pull request #3002 from MichaelEischer/update-contributing 
Update links to help wanted labels in CONTRIBUTING.md and link to forum
 2020-10-09 17:24:25 +02:00
Michael Eischer 4036991c91 Update links to help wanted labels in CONTRIBUTING.md and link to forum 2020-10-09 16:15:43 +02:00
Michael Eischer 88c8e903d2 filter: Fix glob matching on absolute path marker on windows 
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
 2020-10-09 16:11:05 +02:00
greatroar 740758a5fa Optimize filter pattern matching 
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.

name                          old time/op    new time/op    delta
FilterLines-8                   44.7ms ± 5%    44.9ms ± 5%     ~     (p=0.631 n=10+10)
FilterPatterns/Relative-8       13.6ms ± 4%    13.4ms ± 5%     ~     (p=0.165 n=10+10)
FilterPatterns/Absolute-8       10.9ms ± 5%    10.7ms ± 4%     ~     (p=0.052 n=10+10)
FilterPatterns/Wildcard-8       53.7ms ± 5%    50.4ms ± 5%   -6.00%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8     128ms ± 2%      95ms ± 1%  -25.54%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=1.000 n=9+8)
FilterPatterns/Absolute-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.903 n=9+8)
FilterPatterns/Wildcard-8       19.7MB ± 0%    19.7MB ± 0%   -0.00%  (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-8        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-8     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-09 15:51:14 +02:00
MichaelEischer 862ee4b2c9
Merge pull request #2970 from labkode/fopenskip 
Skip fopen for file change check to avoid network penalty
 2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador 958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
MichaelEischer 88cc444779
Merge pull request #2934 from MichaelEischer/upgrade-backoff 
Upgrade github.com/cenkalti/backoff module
 2020-10-08 20:33:30 +02:00
MichaelEischer 4a424af1d5
Merge pull request #2989 from MichaelEischer/remove-local-chmod 
local: mark repository files as read-only and handle chmod errors
 2020-10-08 19:04:13 +02:00
Michael Eischer bc65da2baf Add changelog 2020-10-08 14:45:26 +02:00
Michael Eischer b79f18209f Upgrade github.com/cenkalti/backoff module 
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
 2020-10-07 22:04:59 +02:00
Michael Eischer 8388e67c4b filter: cleanup path separator conversion 2020-10-07 21:14:07 +02:00
Michael Eischer 0acc3c5923 filter: special case patterns without globbing characters 
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.

name                          old time/op    new time/op    delta
FilterLines-4                   34.8ms ±17%    41.2ms ±23%  +18.36%  (p=0.000 n=10+10)
FilterPatterns/Relative-4       21.7ms ± 6%    12.1ms ±23%  -44.46%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       10.0ms ± 5%     9.1ms ±11%   -9.80%  (p=0.006 n=10+9)
FilterPatterns/Wildcard-4       47.0ms ± 7%    42.2ms ± 5%  -10.19%  (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4     190ms ± 1%     131ms ±20%  -31.47%  (p=0.000 n=8+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.870 n=9+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.145 n=10+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    19.7MB ± 0%  +37.91%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.421 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:55:43 +02:00
Michael Eischer 54a124de3b filter: explicitly test separate ListWithChild function 2020-10-07 20:47:52 +02:00
Michael Eischer bcc3bddcf4 filter: only check whether a child path could match when necessary 
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:52 +02:00
Michael Eischer 17c53efb0d filter: Optimize double wildcard expansion 
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       22.7ms ± 5%    23.3ms ± 9%     ~     (p=0.353 n=10+10)
FilterPatterns/Absolute-4       14.2ms ±13%    13.9ms ± 7%     ~     (p=0.853 n=10+10)
FilterPatterns/Wildcard-4        266ms ±16%      51ms ± 7%  -80.67%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4     554ms ± 6%     551ms ± 9%     ~     (p=0.436 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.349 n=10+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.073 n=10+9)
FilterPatterns/Wildcard-4        141MB ± 0%      14MB ± 0%  -89.89%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     0.09M ± 0%  -94.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:48 +02:00
Michael Eischer 7959796269 filter: Special case for absolute paths 
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.6ms ±20%    22.7ms ± 5%     ~     (p=0.684 n=10+10)
FilterPatterns/Absolute-4       32.3ms ± 8%    14.2ms ±13%  -56.01%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        334ms ±17%     266ms ±16%  -20.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     709ms ± 7%     554ms ± 6%  -21.89%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%   +0.00%  (p=0.046 n=9+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.464 n=10+10)
FilterPatterns/Wildcard-4        141MB ± 0%     141MB ± 0%     ~     (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     1.63M ± 0%     ~     (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:29 +02:00
Michael Eischer 375c2a56de filter: Parse filter patterns only once 
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
 2020-10-07 20:47:27 +02:00
Michael Eischer b8eacd1364 filter: Reduce redundant path and pattern splitting 
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       62.1ms ±15%    30.3ms ±10%  -51.22%  (p=0.000 n=9+10)
FilterPatterns/Absolute-4        111ms ±10%      49ms ± 3%  -56.08%  (p=0.000 n=10+8)
FilterPatterns/Wildcard-4        393ms ±15%     345ms ± 9%  -12.30%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     10.0s ± 3%      3.9s ± 2%  -60.53%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       16.4MB ± 0%     4.6MB ± 0%  -71.76%  (p=0.000 n=10+9)
FilterPatterns/Absolute-4       31.4MB ± 0%     8.5MB ± 0%  -72.77%  (p=0.000 n=9+10)
FilterPatterns/Wildcard-4        168MB ± 0%     146MB ± 0%  -13.19%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.23GB ± 0%    0.91GB ± 0%  -71.96%  (p=0.000 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4         178k ± 0%       67k ± 0%  -62.50%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4         266k ± 0%       89k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.87M ± 0%     1.70M ± 0%   -9.47%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     17.7M ± 0%      4.5M ± 0%  -74.87%  (p=0.000 n=9+10)
 2020-10-07 18:13:19 +02:00
Michael Eischer e73c281142 filter: Benchmark absolute paths, wildcards and long filter lists 2020-10-07 17:54:36 +02:00
Michael Eischer fdd3b14db3 filter: test some corner cases 2020-10-07 17:09:44 +02:00
Michael Eischer f4282aa6fd local: mark repository files as read-only 
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
 2020-10-07 12:29:37 +02:00
Michael Eischer 40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation 
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
 2020-10-07 12:29:37 +02:00
greatroar f3e933f0c1 Use windows.SetConsoleCursorPosition in ui/termstatus 2020-10-07 11:18:54 +02:00