Commit Graph

1567 Commits

Author SHA1 Message Date
Alexander Neumann 65923e9c26
Merge pull request #4152 from MichaelEischer/invalid-locks
Improve handling of invalid locks
2023-01-25 08:19:53 +01:00
Michael Eischer 0f398b82e3
Merge pull request #4145 from greatroar/index-encode
index: Optimize generatePackList
2023-01-21 01:01:12 +01:00
greatroar 99755c634b index: Optimize generatePackList
name                 old time/op    new time/op    delta
EncodeIndex/100-8      1.56ms ± 2%    1.48ms ± 3%   -5.37%  (p=0.000 n=10+10)
EncodeIndex/1000-8     14.5ms ± 2%    13.1ms ± 2%   -9.49%  (p=0.000 n=9+10)
EncodeIndex/10000-8     120ms ± 2%     116ms ± 2%   -3.58%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
EncodeIndex/100-8       306kB ± 1%     275kB ± 1%  -10.28%  (p=0.000 n=10+10)
EncodeIndex/1000-8     3.69MB ±11%    2.88MB ± 5%  -22.07%  (p=0.000 n=10+9)
EncodeIndex/10000-8    35.9MB ±11%    31.9MB ±10%  -11.13%  (p=0.005 n=10+10)

name                 old allocs/op  new allocs/op  delta
EncodeIndex/100-8       3.39k ± 0%     2.39k ± 0%  -29.61%  (p=0.000 n=10+10)
EncodeIndex/1000-8      32.6k ± 0%     22.9k ± 0%  -29.63%  (p=0.000 n=10+9)
EncodeIndex/10000-8      326k ± 0%      229k ± 0%  -29.71%  (p=0.000 n=10+10)

The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
2023-01-14 20:41:07 +01:00
Michael Eischer 57acc769b4 lock: Ignore empty lock files
These may be left behind by backends which do not guarantee atomic
uploads.
2023-01-14 18:15:46 +01:00
Michael Eischer 20ad14e362 lock: add help message how to recover from invalid locks 2023-01-14 18:04:22 +01:00
Michael Eischer c995b5be52 lock: cleanup error message
The error message is now `Fatal: unable to create lock in backend:
[...]` instead of `unable to create lock in backend: Fatal: [...]`.
2023-01-14 17:57:02 +01:00
Michael Eischer 1adf28a2b5 repository: properly return invalid data error in LoadUnpacked
The retry backend does not return the original error, if its execution
is interrupted by canceling the context. Thus, we have to manually
ensure that the invalid data error gets returned.

Additionally, use the retry backend for some of the repository tests, as
this is the configuration which will be used by restic.
2023-01-14 17:57:02 +01:00
Michael Eischer 6d9675c323 repository: cleanup error message on invalid data
The retry printed the filename twice:
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: load(<lock/04804cba82>): invalid data returned
```
now the warning has changed to
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: invalid data returned
```
2023-01-14 17:57:02 +01:00
Michael Eischer fb43cbab49
Merge pull request #4129 from greatroar/cleanup
cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser
2023-01-14 12:17:25 +01:00
Michael Eischer 4a7a6b06af ui/backup: Use progress.Updater for progress updates 2023-01-14 01:20:43 +01:00
Michael Eischer e499bbe3ae progress: extract progress updating into Updater struct
This allows reusing the code to create periodic progress updates.
2023-01-14 01:13:08 +01:00
Michael Eischer c15b4bceae backup: extract StdioWrapper from ProgressPrinters
The StdioWrapper is not used at all by the ProgressPrinters. It is
called a bit earlier than previously. However, as the password prompt
directly accessed stdin/stdout this doesn't cause problems.
2023-01-14 00:58:13 +01:00
greatroar 72922a79ed cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser 2023-01-03 19:03:36 +01:00
Panagiotis Cheilaris 3b516d4b70 convert uid/gid -1 to 0 only in 32-bit tar dump
Only for a 32-bit build of restic, convert a uid or gid value of -1 to 0.
2022-12-30 18:12:12 +01:00
Panagiotis Cheilaris a86a56cf3b fix lint issue with function name 'tarId'
See https://github.com/golang/lint/issues/89 and
https://github.com/golang/lint/issues/124
2022-12-28 18:46:58 +01:00
Panagiotis Cheilaris 050ed616ae be more explicit with uid or gid of value -1 2022-12-28 18:44:36 +01:00
Panagiotis Cheilaris 10fa5cde0a in tar dump, convert uid, gid of value -1 to zero 2022-12-27 16:36:04 +01:00
Michael Eischer 90fb6f70b4
Merge pull request #4089 from greatroar/errors
Clean up error handling further
2022-12-24 10:41:56 +01:00
greatroar 1678392a6d checker: Make ErrLegacyLayout a value, not a type 2022-12-17 09:41:07 +01:00
greatroar d9002f050e backend: Don't Wrap errors from url.Parse
The messages from url.Error.Error already start with the word "parse".
2022-12-17 09:41:07 +01:00
greatroar b150dd0235 all: Replace some errors.Wrap calls by errors.WithStack
Mostly changed the ones that repeat the name of a system call, which is
already contained in os.PathError.Op. internal/fs.Reader had to be
changed to actually return such errors.
2022-12-17 09:41:07 +01:00
Michael Eischer cccc17e4e9
Merge pull request #4086 from blackpiglet/modify_access_denied_code
Fix: change error code in function isAccessDenied to AccessDenied
2022-12-16 21:55:52 +01:00
Xun Jiang cc5325d22b Fix: change error code in function isAccessDenied to AccessDenied
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-12-16 21:41:16 +01:00
Michael Eischer da0e45cf40
Merge pull request #4083 from greatroar/cleanup
repository: Remove empty cleanup functions in tests
2022-12-16 21:39:30 +01:00
Michael Eischer 1bfe98bdc0
Merge pull request #2398 from DanielG/b2-hide-file
b2: Fallback to b2_hide_file when delete returns unauthorized
2022-12-13 22:52:23 +01:00
Michael Eischer 1c071a462e
Merge pull request #4084 from ekarlso/azure-stat-fix
fix: Make create not error out when ContainerNotFound
2022-12-13 22:49:05 +01:00
Endre Karlson 7dd33c0ecc azure: Make create not error out when ContainerNotFound 2022-12-11 22:57:23 +01:00
greatroar c0b5ec55ab repository: Remove empty cleanup functions in tests
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
2022-12-11 11:06:25 +01:00
Michael Eischer 049a105ba5
Merge pull request #4077 from greatroar/cleanup
test: Use testing.T.Cleanup to remove tempdirs
2022-12-09 22:17:46 +01:00
Michael Eischer 4b98b5562d
Merge pull request #4075 from greatroar/sftp-enospc
sftp: Fix ENOSPC check
2022-12-09 22:00:13 +01:00
greatroar f90bf84ba7 test: Use testing.T.Cleanup to remove tempdirs 2022-12-09 14:23:55 +01:00
Endre Karlson 25648e2501 azure: Switch to azblob sdk 2022-12-07 21:46:07 +01:00
greatroar 62520bb7b4 sftp: Fix ENOSPC check
We now check for space that is not reserved for the root user on the
remote, and the check is no longer in a defer block because it wouldn't
fire. Some change in the surrounding code may have led the deferred
function to capture the wrong err variable.

Fixes #3336.
2022-12-07 21:06:46 +01:00
Michael Eischer 2b67862420 backend/test: check that IsNotExist actually works 2022-12-03 18:56:55 +01:00
Michael Eischer 2f934f5803 gs: check against the correct error in IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer 04d101fa94 gs/s3: remove useless os.IsNotExist check 2022-12-03 18:49:54 +01:00
Michael Eischer 579cd6dc64 azure: fix totally broken IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer 3ebdadc58f
Merge pull request #4069 from greatroar/cleanup
cache, prune, restic: Cleanup
2022-12-03 17:50:52 +01:00
Michael Eischer bc8b2455b9
Merge pull request #4064 from MichaelEischer/flaky-abort-early-on-error
archiver: Fix flaky TestArchiverAbortEarlyOnError
2022-12-03 17:43:44 +01:00
Michael Eischer 60c6a09324
Merge pull request #4065 from MichaelEischer/flaky-rclone-failed-start
rclone: treat "file already closed" as command startup error
2022-12-03 17:42:56 +01:00
greatroar 63bed34608 restic: Clean up restic.IDs type
IDs.Less can be rewritten as

	string(list[i][:]) < string(list[j][:])

Note that this does not copy the ID's.

The Uniq method was no longer used.

The String method has been reimplemented without first copying into a
separate slice of a custom type.
2022-12-03 12:38:20 +01:00
greatroar d45a2475e1 cache: Rewrite unnecessary if-else 2022-12-03 12:13:54 +01:00
Michael Eischer 648edeca40 retry: Do not retry Stat() if file does not exist
In non test/debug code, Stat() is used exclusively to check whether a
file exists. Thus, do not retry if a file is reported as not existing.
2022-12-03 11:42:48 +01:00
Michael Eischer 40ac678252 backend: remove Test method
The Test method was only used in exactly one place, namely when trying
to create a new repository it was used to check whether a config file
already exists.

Use a combination of Stat() and IsNotExist() instead.
2022-12-03 11:28:10 +01:00
Michael Eischer 57d8eedb88
Merge pull request #4020 from greatroar/fuse-inode
fuse: Better inode generation
2022-12-02 22:28:15 +01:00
Michael Eischer ca1803cacb
Merge pull request #4063 from MichaelEischer/replace-ioutil-usage
Replace ioutil usage
2022-12-02 21:49:40 +01:00
Michael Eischer f3d964a8c1 rclone: treat "file already closed" as command startup error
Since #3940 the rclone backend returns the commands exit code if it
fails to start. The list of expected errors was missing the "file
already closed"-error which can occur if the http test request first
learns about the closed pipe to rclone before noticing the canceled
context.

Go internally makes sure that a file descriptor is unusable once it was
closed, thus this cannot have unintended side effects (like accidentally
reading from the wrong file due to a reused file descriptor).
2022-12-02 20:46:02 +01:00
Michael Eischer a9972dbe7d archiver: Fix flaky TestArchiverAbortEarlyOnError
Saving the blobs of a file by now happens asynchronously to the
processing in the FileSaver. Thus we have to account for the blobs
queued for saving.
2022-12-02 20:07:34 +01:00
Michael Eischer fa20a78bb6
Merge pull request #4056 from greatroar/cleanup
backend, fs, options: Minor cleanup
2022-12-02 19:44:54 +01:00
Michael Eischer ff7ef5007e Replace most usages of ioutil with the underlying function
The ioutil functions are deprecated since Go 1.17 and only wrap another
library function. Thus directly call the underlying function.

This commit only mechanically replaces the function calls.
2022-12-02 19:36:43 +01:00