74bc7141c1
Merge pull request #2781 from aawsome/reduce-index-memory
...
Reduce index memory
2020-06-14 10:58:28 +02:00
1361341c58
don't save duplicate packIDs when using internal/repository/Index.Store
2020-06-14 07:56:24 +02:00
ce4a2f4ca6
save packIDs and duplicates separately
...
A side remark to the definition of Index.blob:
Another possibility would have been to use:
blob map[restic.BlobHandle]*indexEntry
This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry: 8 + 4 + 4 = 16 bytes
each packID: 32 bytes
To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes
More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
2020-06-14 07:56:21 +02:00
cf979e2b81
make offset and length uint32
2020-06-14 07:50:19 +02:00
d92e2c5769
simplify index code
2020-06-14 07:50:19 +02:00
7419844885
add changelog, benchmark, memory calculation
2020-06-14 07:50:15 +02:00
1d66bb9e62
Merge pull request #2784 from MichaelEischer/minio-sha256-changelog
...
Add changelog for #2709
2020-06-13 18:33:54 +02:00
0b2c31b05b
Add changelog for #2709
2020-06-13 18:13:36 +02:00
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256
...
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
6896c6449b
Merge pull request #2779 from greatroar/archiver-comment
...
Fix up comment on archiver.BlobSaver.Save
2020-06-12 23:04:21 +02:00
735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs
...
Fix non-intuitive repo behavior
2020-06-12 22:41:04 +02:00
70347e95d5
disable index uploads for prune command
...
+ modifications of changelog
2020-06-12 09:24:38 +02:00
0fa3091c78
Fix up comment on archiver.BlobSaver.Save
2020-06-11 13:40:30 +02:00
91906911b0
Fix non-intuitive repository behavior
...
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
-> also cleans up pending index entries when they are saved in the index
-> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
-> removes the need of an index uploader
-> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
fae7f78057
Merge pull request #2755 from greatroar/solaris-workaround
...
Remove Solaris build workaround in integration tests
2020-06-10 23:30:51 +02:00
ac9ec4b990
Merge pull request #2778 from MichaelEischer/fix-unlock-hint
...
Revive hint to the unlock command if a repository is locked
2020-06-10 21:26:38 +02:00
087c770161
Revive hint to the unlock command if a repository is locked
...
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.
Just add an additional remark to the error using `errors.WithMessage`.
2020-06-10 20:58:56 +02:00
6856d1e422
Merge pull request #2749 from aawsome/fix-fullindex
...
Change condition for full index
2020-06-10 20:40:19 +02:00
8c1261ff02
changed condition for full index
2020-06-07 22:00:49 +02:00
26704be17f
Merge pull request #2776 from nunoperalta/patch-1
...
Fix #2774 - make unable to umount message clearer
2020-06-07 17:19:50 +02:00
2c3360db98
Fix #2774 - make unable to umount message clearer
2020-06-07 15:57:39 +01:00
cba6ad8d8e
Merge pull request #2711 from greatroar/update-x-net
...
Update golang.org/x/net for fewer allocations in http2
2020-05-25 20:43:29 +02:00
2a3312ac35
Remove Solaris build workaround in integration tests
...
This was introduced in #1821 because of issues with Go 1.9, but that
compiler version is no longer supported.
2020-05-25 10:33:36 +02:00
c35c4e0cbf
Update golang.org/x/net for fewer allocations in http2
...
name old allocs/op new allocs/op delta
BackendREST/BenchmarkLoadFile-8 2.15k ± 0% 2.14k ± 0% -0.27% (p=0.002 n=20+20)
BackendREST/BenchmarkLoadPartialFile-8 587 ± 0% 582 ± 0% -0.88% (p=0.000 n=20+20)
BackendREST/BenchmarkLoadPartialFileOffset-8 584 ± 0% 579 ± 0% -0.90% (p=0.000 n=19+16)
The throughput isn't significantly higher or lower.
2020-05-24 16:39:17 +02:00
84475aa3a8
Merge pull request #2730 from greatroar/mount-build-tags
...
Simplify build tags for restic mount
2020-05-23 20:33:43 +02:00
f12f9ae240
Merge pull request #2741 from tbm/typos
...
Fix typos
2020-05-21 23:40:12 +02:00
5cc1760fdf
Fix typos
2020-05-16 14:05:26 +08:00
32ac5486e9
Merge pull request #2732 from restic/rawtaz-update-contributing
...
Reword parts of Providing Patches in CONTRIBUTING.md
2020-05-12 21:37:08 +02:00
c4336978eb
Reword parts of Providing Patches in CONTRIBUTION.md
...
Primarily makes the request that contributors create a new branch for their changes stand out more.
2020-05-12 14:03:57 +02:00
649cbec6c5
Simplify build tags for restic mount
...
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224 ).
Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
2020-05-12 11:30:41 +02:00
b17bd7f860
Merge pull request #2719 from greatroar/update-x-crypto
...
Update x/crypto to stop poly1305 segfaulting on ARM
2020-05-09 00:02:18 +02:00
68f1e9c524
Update x/crypto to fix poly1305 on ARM
...
The broken poly1305 implementation for arm64 was removed, fixing the
segfault issue on arm64 (gh-2618). The version for amd64 has improved
performance, which shows up in the internal/repository benchmark:
name old speed new speed delta
SaveAndEncrypt-8 110MB/s ± 2% 113MB/s ± 1% +2.24% (p=0.000 n=20+20)
2020-05-05 11:05:35 +02:00
1ee2306033
Merge pull request #2717 from MichaelEischer/fix-flaky-password-prompt
...
backup: Avoid race between password prompt and open repository message
2020-05-01 23:58:55 +02:00
c882a92cd6
backup: Avoid race between password prompt and open repository message
...
`term.Print` sends the output via a channel to a goroutine which
actually prints the message. This may race with the password prompt
printed by `OpenRepository` resulting in a missing prompt.
2020-05-01 23:42:16 +02:00
f54db5d796
Merge pull request #2713 from greatroar/unused
...
Move Index.FindBlob to tests
2020-05-01 20:41:59 +02:00
843e7f404e
Merge pull request #2716 from MichaelEischer/revert-apfs-test-workaround
...
Revert "Darwin test fix: allow 1μs timestamp difference"
2020-05-01 18:13:38 +02:00
d465b5b9ad
Revert "Darwin test fix: allow 1μs timestamp difference"
...
This reverts commit f3016a9096
2020-05-01 17:11:30 +02:00
9f7cd69f13
Move Index.FindBlob to tests
2020-04-29 10:57:01 +02:00
f97a680887
Fix repository benchmarks
...
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
2020-04-28 07:57:29 +02:00
42a3db05b0
Use Minio's optimized SHA-256
...
internal/repository benchmarks on an Intel i7-3770k:
name old speed new speed delta
PackerManager-8 209MB/s ± 1% 291MB/s ± 1% +38.94% (p=0.008 n=5+5)
SaveAndEncrypt-8 112MB/s ± 1% 135MB/s ± 1% +20.25% (p=0.008 n=5+5)
2020-04-28 07:57:18 +02:00
070d43e290
Merge pull request #2704 from azak-azkaran/patch-1
...
Update 030_preparing_a_new_repo.rst
2020-04-23 20:48:34 +02:00
d4bd32a37e
Merge pull request #2640 from greatroar/simplify-loadblob-usage
...
Simplify Repository.LoadBlob usage
2020-04-23 20:23:35 +02:00
e7d7b85d59
Merge Repository.{LoadBlob,loadBlob}
...
Pushing the allocation logic down into the former loadBlob body means
that fewer allocations have to be performed:
name old time/op new time/op delta
LoadTree-8 478µs ± 1% 481µs ± 2% ~ (p=0.315 n=9+10)
LoadBlob-8 11.6ms ± 1% 11.6ms ± 2% ~ (p=0.393 n=10+10)
LoadAndDecrypt-8 13.3ms ± 3% 13.3ms ± 3% ~ (p=0.905 n=10+9)
LoadIndex-8 33.6ms ± 2% 33.2ms ± 1% -1.15% (p=0.028 n=10+9)
name old alloc/op new alloc/op delta
LoadTree-8 41.2kB ± 0% 41.1kB ± 0% -0.23% (p=0.000 n=10+10)
LoadBlob-8 2.28kB ± 0% 2.18kB ± 0% -4.21% (p=0.000 n=10+10)
LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal)
LoadIndex-8 5.22MB ± 0% 5.22MB ± 0% ~ (p=0.631 n=10+10)
name old allocs/op new allocs/op delta
LoadTree-8 652 ± 0% 651 ± 0% -0.15% (p=0.000 n=10+10)
LoadBlob-8 24.0 ± 0% 23.0 ± 0% -4.17% (p=0.000 n=10+10)
LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal)
LoadIndex-8 30.2k ± 0% 30.2k ± 0% ~ (p=0.610 n=10+10)
name old speed new speed delta
LoadBlob-8 86.4MB/s ± 1% 85.9MB/s ± 2% ~ (p=0.393 n=10+10)
LoadAndDecrypt-8 75.4MB/s ± 3% 75.4MB/s ± 3% ~ (p=0.858 n=10+9)
2020-04-23 10:04:20 +02:00
be5a0ff59f
Centralize buffer allocation and size checking in Repository.LoadBlob
...
Benchmark results for internal/repository:
name old time/op new time/op delta
LoadTree-8 479µs ± 2% 478µs ± 1% ~ (p=0.780 n=10+9)
LoadBlob-8 11.6ms ± 2% 11.6ms ± 1% ~ (p=0.631 n=10+10)
LoadAndDecrypt-8 13.2ms ± 2% 13.3ms ± 3% ~ (p=0.631 n=10+10)
name old alloc/op new alloc/op delta
LoadTree-8 41.2kB ± 0% 41.2kB ± 0% ~ (all equal)
LoadBlob-8 2.28kB ± 0% 2.28kB ± 0% ~ (all equal)
LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
LoadTree-8 652 ± 0% 652 ± 0% ~ (all equal)
LoadBlob-8 24.0 ± 0% 24.0 ± 0% ~ (all equal)
LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal)
name old speed new speed delta
LoadBlob-8 86.2MB/s ± 2% 86.4MB/s ± 1% ~ (p=0.594 n=10+10)
LoadAndDecrypt-8 75.7MB/s ± 2% 75.4MB/s ± 3% ~ (p=0.617 n=10+10)
2020-04-23 10:04:20 +02:00
c5100d5632
Update 030_preparing_a_new_repo.rst
...
fixed markdown for wasabi region link
2020-04-23 09:24:19 +02:00
956a1b0f96
Merge pull request #2294 from BenWiederhake/debugsyms
...
build: Preserve debug symbols in debug build
2020-04-19 17:26:03 +02:00
fab626a3df
build: Preserve debug symbols in debug and profile build
...
Signed-off-by: Ben Wiederhake <BenWiederhake.GitHub@gmx.de>
2020-04-19 14:48:40 +02:00
4f00564574
Merge pull request #2621 from greatroar/fixes
...
Some small fixes
2020-04-18 18:07:08 +02:00
f77477129f
Merge pull request #2584 from greatroar/mount-cache-uid-gid
...
Cache uid and gid for top directories in internal/fuse
2020-04-18 17:45:14 +02:00
2e31120f89
Remove unused argument to restic.fakeFile
2020-04-18 17:40:13 +02:00
MichaelEischer
Alexander Weiss
rawtaz
greatroar
nunoperalta
Martin Michlmayr
azak-azkaran
Ben Wiederhake