Commit Graph

1786 Commits

Author SHA1 Message Date
Michael Eischer e4bbde7036 rclone: Workaround for incorrect "not found" errors while listing files
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:28:17 +01:00
Michael Eischer 103beb96bc restore: separately restore blobs that are frequently referenced
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-09 18:27:48 +01:00
Michael Eischer f0f89d7f27 restore: split error reporting from downloadPack 2024-01-09 18:27:48 +01:00
Michael Eischer cf352ccafb restore: cleanup downloadPack 2024-01-09 18:27:48 +01:00
Michael Eischer b856e9489a restore: split downloadPack into smaller methods 2024-01-09 18:27:48 +01:00
Michael Eischer f2fafbffaa restore: only report errors for blobs that actually failed to load
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-08 21:33:05 +01:00
Michael Eischer 7a3a884874 repository: test that StreamPack only delivers blobs once 2024-01-08 21:33:05 +01:00
Michael Eischer 772a907533 repository: StreamPack delivers blobs at most once
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-08 21:33:05 +01:00
Michael Eischer 1bab29c336 archiver: Add filepath to error message if it is not included yet 2024-01-08 21:33:05 +01:00
Michael Eischer e886c3f6b2 archiver: improve error message for irregular files
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.

Add the filepath to the error message and state that the file type is
not supported.
2024-01-08 21:33:05 +01:00
Michael Eischer c95de54726 restic: cleanup node type determination
os.ModeCharDevice is already included in os.ModeType
2024-01-08 21:33:05 +01:00
Michael Eischer 5cd000f4b0 CI: update golangci-lint
Necessary to properly support Go 1.21.
2024-01-08 21:33:05 +01:00
Leo R. Lundgren 8c1125fe13 doc: Correct two typos 2023-10-27 19:48:00 +02:00
Martin Michlmayr 0b6ccea461 Fix typos 2023-10-27 19:48:00 +02:00
Michael Eischer a28940ea29 check: Suggest usage of `restic repair packs` for corrupted blobs
For now, the guide is only shown if the blob content does not match its
hash. The main intended usage is to handle data corruption errors when
using maximum compression in restic 0.16.0
2023-10-23 18:36:28 +02:00
Michael Eischer 7f05af02b9
Merge pull request #4480 from AgathaSorceress/add-rest-auth-env
Support reading basic auth credentials for REST server from environment variables
2023-10-21 17:41:08 +00:00
Michael Eischer ab6defbace
Merge pull request #4519 from adamantike/backend/sftp/add-args-option
backend/sftp: Add sftp.args option
2023-10-21 17:37:46 +00:00
Michael Eischer 6a4d6d5da4 sftp: test that Args and Command options cannot be set at the same time 2023-10-21 19:26:39 +02:00
Michael Manganiello 41f70f1f4f backend/sftp: Add sftp.args option
Allow setting custom arguments for the `sftp` backend, by using the
`sftp.args` option. This is similar to the approach already implemented
in the `rclone` backend, to support new arguments without requiring
future code changes for each different SSH argument.

Closes #4241
2023-10-21 19:25:44 +02:00
Michael Eischer 6c7560e537 backend/b2: Switch to official library `Backblaze/blazer`
According to https://github.com/kurin/blazer Readme, the repository is
no longer maintained, and the official one at
https://github.com/Backblaze/blazer took over.

Closes #4148
2023-10-21 19:18:13 +02:00
Agatha V. Lovelace f1877e721e
feat: support reading REST credentials from env 2023-10-20 13:04:58 +02:00
Michael Eischer d27cfd10a9
Merge pull request #4487 from lbausch/tweak-restore-summary
Tweak restore summary
2023-10-19 20:14:14 +00:00
Michael Eischer d173d1c532
Merge pull request #4486 from MichaelEischer/separate-large-blobs
repository: store oversized blobs in separate pack files
2023-10-17 23:03:52 +02:00
Michael Eischer cb9cbe55d9 repository: store oversized blobs in separate pack files
Store oversized blobs in separate pack files as the blobs is large
enough to warrant its own pack file. This simplifies the garbage
collection of such blobs and keeps the cache smaller, as oversize (tree)
blobs only have to be downloaded if they are actually used.
2023-10-17 22:52:16 +02:00
Michael Eischer f750aa8dfb backend/limiter: replace juju/ratelimit with x/time/rate 2023-10-17 22:49:25 +02:00
Michael Eischer 91aef00df3 check: add index loading progress bar 2023-10-01 19:55:29 +02:00
Michael Eischer 3fd0ad7448 repository: list index files only once 2023-10-01 19:53:26 +02:00
arjunajesh ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Lorenz Bausch 2657217574
Tweak restore summary 2023-09-20 10:59:55 +02:00
Michael Eischer 6e586b64e4
Merge pull request #4450 from MichaelEischer/cleanup-master-index-test
master_index: fix inconsistent length blob length in test
2023-09-06 22:55:48 +02:00
greatroar 691c01963b internal/restic: Return summary from BlobSet.String
Fixes #4449: error messages from prune were too long to fit in scroll
buffers.
2023-08-25 21:41:30 +02:00
Michael Eischer 2bec99dc6f master_index: fix inconsistent length blob length in test
Two blobs with the same hash must always have the same content length.
2023-08-19 20:04:25 +02:00
Michael Eischer feb6abb7bb
Merge pull request #4394 from greatroar/acl
dump: Rewrite Linux ACL handling
2023-08-13 17:01:31 +00:00
greatroar aaf5254e26 dump: Rewrite Linux ACL handling
The old version was taken from an MPL-licensed library. This is a
cleanroom implementation. The code is shorter and it's now explicit that
only Linux ACLs are supported.
2023-08-13 13:37:11 +02:00
Michael Eischer 4a33370072 backend: Fix unreliable TestListCancel/Timeout test
The test uses `WithTimeout` to create a context that cancels the List
operation after a given delay. Several backends internally use a derived
child context created using WithCancel.

The cancellation of a context first closes the done channel of the
context (here: the `WithTimeout` context) and _afterwards_ propagates
the cancellation to child contexts (here: the `WithCancel` context).
Therefor if the List implementation uses a child context, then it may
take a moment until that context is also cancelled. Thus give the
context cancellation a moment to propagate.
2023-07-23 11:47:56 +02:00
Michael Eischer f12bbd9229 restic: check that Node.LinkTarget can handle non-utf8 characters 2023-07-23 00:05:14 +02:00
Michael Eischer 6adb629608 Add support for non-utf8 symlink targets 2023-07-23 00:05:14 +02:00
Michael Eischer 25ff9fa893
Merge pull request #4334 from MichaelEischer/snapshot-subtree-syntax
Add support for snapshot:path syntax
2023-07-22 23:59:26 +02:00
Michael Eischer 3f63b53090 lock: Shrink critical section in RefreshStaleLock
A cleaning up after a failed lock refresh attempt does not require a
mutex.
2023-07-22 23:29:14 +02:00
Michael Eischer 96eada3d5f backend/sema: add test for freeze functionality 2023-07-22 23:23:17 +02:00
Michael Eischer 8154f6a77a Ensure consistent naming for `<snapshot>:<subfolder>` syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 090f9d6237 restic: Cleanup and simplify TestCreateSnapshot 2023-07-22 19:55:57 +02:00
Michael Eischer 321cc35cde restic: add test for FindTreeDirectory 2023-07-22 19:55:57 +02:00
Michael Eischer a97915642c restic: Ensure snapshots created by TestCreateSnapshot are valid
All nodes in a tree must be sorted by node name. Otherwise functionality
like searching for a specific tree node will break.
2023-07-22 19:55:57 +02:00
Michael Eischer c64d81063e test snapshot filtering with snapshot:path syntax 2023-07-22 19:55:57 +02:00
Michael Eischer 85860e6e97 Add support for snapshot:subpath syntax
This snapshot specification syntax is supported by the cat, diff, dump,
ls and restore command.
2023-07-22 19:55:57 +02:00
Michael Eischer 5d9b0d894e lock: add unit test for RefreshStaleLock 2023-07-22 19:55:44 +02:00
Michael Eischer d4bf7a3cb1 restic: cleanup lock test 2023-07-22 19:55:44 +02:00
Michael Eischer 24c8a33da9 restic: reduce sleeps in lock tests 2023-07-22 19:55:44 +02:00
Michael Eischer f490288738 lock: freeze backend operations while refreshing stale lock
Freeze new backend operations while trying to refresh a stale lock.
2023-07-22 19:55:44 +02:00