Commit Graph

25 Commits

Author SHA1 Message Date
greatroar de5516a90e Fix sync.WaitGroup usage in restorer.fileRestorer 2020-02-27 00:07:49 +01:00
Igor Fedorenko f17ffa0283 restorer: Allow writing target file blobs out of order
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:14:45 +01:00
Igor Fedorenko bf9a507148 Fix rare 'file already closed' during restore
Fixes #2183

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2019-03-16 17:01:20 +01:00
Alexander Neumann c0fca3f50a restore: Close files tested with --verify
Before, the target files were opened once per blob and never closed,
this commit fixes that.
2018-10-14 21:00:14 +02:00
Igor Fedorenko bda8d7722e restorer: Optimize empty file restore
don't create fileInfo structs for empty files. this saves memory.
this also avoids extra serial scan of all fileInfo, which should
make restore faster and more consistent.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko c2bcb764cd restorer: Add a note on hardlink metadata
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko 9e24154ec9 restore: significantly reduce memory footprint
reworked restore error callback to use file location
path instead of much heavier Node. this reduced restore
memory usage by as much as 50% in some of my tests.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko 9f3ca97ee8 restore: Chang fileInfo to use snapshot location instead of target path
* uses less memory as common prefix is only stored once
* stepping stone for simpler error callback api, which
  will allow further memory footprint reduction

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko 32d5ceba87 restore: Fix packcache capacity math with failed downloads
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko e010f3b884 restore: Fix small memory leak in filesWriter, add tests
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Alexander Neumann 941202c119 restore: Rename things, match the rest of the project 2018-10-14 17:39:37 +02:00
Alexander Neumann c021ad2334 restore: Move documentation to godoc 2018-10-14 17:39:35 +02:00
Igor Fedorenko da57302fca restore: Removed legacy restore implementation
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
Igor Fedorenko 1869930d95 restore: New optimized multithreaded implementation
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
Igor Fedorenko 1213d8fef4 Cleanup: more realistic restorer test data setup
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
Alexander Neumann 44924ba043 restorer: Fix traverseTree
traverseTree() was meant to call enterDir() whenever a directory is
selected for restore, either explicitly or implicitly (=contains a file
which is to be restored). After restoring a file, leaveDir() is called
in reverse order for all intermediate directories so that the metadata
can be restored.

When a directory is selected implicitly, the metadata for it is
restored. This is different from the previous restorer behavior, which
created implicitly selected intermediate directories with permissions
0700 (only user can read/write it).

This commit changes the behavior back to the old one. Only a directory
is explicitly selected for restore, enterDir()/leaveDir() are called for
it. Otherwise, only visitNode() is called, so visitNode() needs to make
sure the parent directory exists. If the directory is explicitly
included, leaveDir() will then restore the metadata correctly.

When we decide to change the behavior (restore metadata for all
intermediate directories, even if selected implicitly), we should do
that in the selection functions, not here.

This finally resolves #1870
2018-07-21 23:24:40 +02:00
Alexander Neumann ce19f26948 restorer: Add tests for traverseTree 2018-07-21 23:24:40 +02:00
Alexander Neumann 74016d5981 restorer: Fix return of saveSnapshot 2018-07-21 23:24:40 +02:00
Alexander Neumann 57636a4573 restorer: Run tests in the same package 2018-07-21 23:24:40 +02:00
Alexander Neumann 4f6d2502f7 restorer: Add test for restore with include filter 2018-07-21 23:24:40 +02:00
Alexander Neumann 92421ec47f restore: Make sure target directory exists 2018-07-15 16:02:04 +02:00
Alexander Neumann 9acc9243ba Add test for not-existing top-level dir and top-level file 2018-07-15 16:00:26 +02:00
Igor Fedorenko e206680947 restore: New --verify flag to verify restored files content
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00
Igor Fedorenko 5fa6dc53cb Refactor: introduced restorer tree visitor
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00
Igor Fedorenko 26be094f28 Refactor: moved restorer to separate package
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00