Commit Graph

53 Commits

Author SHA1 Message Date
MichaelEischer cb81ee9396
Merge pull request #3474 from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
kitone 1b23675f21 cache --cleanup should handle directories created by restic check.
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
kitone 95eb859b54 Honor RESTIC_CACHE_DIR environment variable
Fix #3382: restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Alexander Weiss 81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
Michael Eischer 9aa2eff384 Add plumbing to calculate backend specific file hash for upload
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
greatroar 6586e90acf Modernize internal/cache error handling 2021-08-04 22:02:42 +02:00
greatroar ea04f40eb3 Save cached files to a temporary location first 2021-08-04 22:02:42 +02:00
greatroar f9b6f8fd45 Replace duplicate type checking in cache with a function 2021-08-04 22:02:42 +02:00
Alexander Neumann 0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
greatroar 673dda77c0 Less repetitive error messages in internal/cache
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
greatroar 7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
greatroar cb3f531050 Make Cache.Path private
It's only used inside the package.
2020-09-18 11:17:29 +02:00
greatroar 23fcbb275a Remove restic.Cache interface
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
greatroar 6e3215a80d Remove unused error type from internal/cache 2020-09-18 10:38:14 +02:00
aawsome 0fed6a8dfc
Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
greatroar 5cd0bce452 Honor RESTIC_CACHE_DIR on Mac and Windows 2020-02-28 15:44:32 +01:00
Stephane Thiell 0e897ef7b8 Add support for $RESTIC_CACHE_DIR
Add support for restic-specific $RESTIC_CACHE_DIR environment variable
to override the cache directory like --cache-dir would have.
2019-09-26 15:59:56 -07:00
Alexandr Bruyako e19622e4b1 start using fileMode const 2019-07-01 00:26:00 +03:00
Alexandr Bruyako 02014be76c simplified prefix removal, removed unnecessary if-else statements 2019-06-30 23:34:47 +03:00
Alexander Neumann d51e9d1b98 Add []byte to repo.LoadAndDecrypt and utils.LoadAll
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.

In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
2019-04-13 13:38:39 +02:00
Alexander Neumann 00597284de cache: Don't return an error for truncated files
Closes #1229
Closes #1328
2018-11-25 14:18:02 +01:00
Alexander Neumann 5b95bb7059 Fix error message on Windows 2018-11-11 14:06:32 +01:00
Alexander Neumann 5971650f77 cache: Fix recheck logic, remove channel from inProgress 2018-10-04 17:09:43 +02:00
Alexander Neumann 19725954ee cache: Recheck before downloading 2018-10-04 14:31:18 +02:00
Alexander Neumann 6d53e767d5 cache: Print message when new cache is created
Sometimes, users run restic without retaining the local cache
directories. This was reported several times in the past.

Restic will now print a message whenever a new cache directory is
created from scratch (i.e. it did not exist before), so users have a
chance to recognize when the cache is not kept between different runs of
restic.
2018-08-29 22:01:10 +02:00
Alexander Neumann 0f83fea007 cache: Fix test for new behavior
Accessing beyond the end of the file now removes the file from the cache
because it is assumed to be truncated. Usually, this means that the data
is fetched directly from the backend instead.
2018-08-11 23:11:51 +02:00
Alexander Neumann 5dd0df0162 cache: Remove files from cache which are too small 2018-08-11 22:47:01 +02:00
Alexander Neumann bd742ddb69 cache: Don't recreate CACHEDIR.TAG 2018-07-08 12:05:12 +02:00
Alexander Neumann fc5439a37a cache: Ensure failed downloads are retried
This fixes #1833, which consists of two different bugs:

 * The `defer` in `cacheFile()` may remove a channel from the
   `inProgress` map although it is not responsible for downloading the
   file

 * If the download fails, goroutines waiting for the file to be cached
   assumed that the file was there, there was no way to signal the
   error.
2018-06-09 17:50:56 +02:00
Gaige Lama a63989afcd
Ran gofmt on backend.go
I ran gofmt on backend.go, this appears to valid edit.
2018-05-11 11:07:16 -04:00
Alexander Neumann f928aeec34 Add 'cache' command 2018-05-01 16:27:44 +02:00
Alexander Neumann 99f7fd74e3 backend: Improve Save()
As mentioned in issue [#1560](https://github.com/restic/restic/pull/1560#issuecomment-364689346)
this changes the signature for `backend.Save()`. It now takes a
parameter of interface type `RewindReader`, so that the backend
implementations or our `RetryBackend` middleware can reset the reader to
the beginning and then retry an upload operation.

The `RewindReader` interface also provides a `Length()` method, which is
used in the backend to get the size of the data to be saved. This
removes several ugly hacks we had to do to pull the size back out of the
`io.Reader` passed to `Save()` before. In the `s3` and `rest` backend
this is actively used.
2018-03-03 15:49:44 +01:00
Igor Fedorenko d58ae43317 Reworked Backend.Load API to retry errors during ongoing download
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-16 21:12:14 -05:00
Alexander Neumann 909d9273cc Close backend files in case of errors 2018-01-25 21:05:57 +01:00
Edward Betts b5c23695c9
Correct a spelling mistake. 2018-01-24 12:09:41 +00:00
Alexander Neumann ea593fca1b cache: Correctly return dir for Windows/darwin 2017-11-20 06:11:18 +01:00
Alexander Neumann fe1f151ae1 cache: Return error during default dir detection 2017-11-20 06:10:42 +01:00
Alexander Neumann f4bab789b8 cache: Simplify cache dir creation 2017-11-24 20:53:26 +01:00
Alexander Neumann fa893ee477 cache: Add detection code for old cache dirs 2017-11-24 20:53:26 +01:00
George Armhold bcdebfb84e small cleanup:
- be explicit when discarding returned errors from .Close(), etc.
- remove named return values from funcs when naked return not used
- fix some "err" shadowing when redeclaration not needed
2017-10-25 12:03:55 -04:00
Alexander Neumann 8e2ef3f38b cache: Store first, then store in the backend
Store the file in the backend, then rewind the reader and store it
in the cache afterwards.
2017-10-17 22:01:20 +02:00
Alexander Neumann cebee0b8fa cache: Refuse to cache truncated files 2017-10-05 20:40:02 +02:00
Alexander Neumann d81adcfaa5 cache: Add file name to error message 2017-10-05 19:30:56 +02:00
Michael Pratt 2133869127 cache: OS-specific cache directories
Windows, and to a lesser extent OS X, don't conform to XDG and have
their own preferred locations for caches.

On Windows, use %LOCALAPPDATA%/restic (i.e., ~/AppData/Local/restic). I
can't find authoritative documentation from Microsoft recommending
specifically which of %APPDATA%, %LOCALAPPDATA%, and %TEMP% should be
used for caches, but %LOCALAPPDATA% is where browsers store their
caches, so it seems like a good fit.

On OS X, use ~/Library/Caches/restic, which is recommended by the Apple
documentation. They do suggest using the application "bundle identifier"
as the base folder name, but restic doesn't have one, so I just used
"restic".
2017-09-27 21:16:22 -07:00
Alexander Neumann 0bb2a8e0d0 cache: Synchronize downloading
This commit adds code to synchronize downloading files to the cache.
Before, requests that came in for files currently downloading would fail
because the file was not completed in the cache. Now, the code waits
until the download is completed.

Closes #1278
2017-09-25 15:58:20 +02:00
Alexander Neumann e262f35d0a cache: Auto-remove invalid files 2017-09-24 23:11:47 +02:00
Alexander Neumann 240c4cf2fd cache: In case of an error, fall back backend 2017-09-24 23:11:23 +02:00
Alexander Neumann e1dfaf5d87 cache: Allow proactive caching of tree packs
This commit adds a function to the cache which can decide to proactively
load the complete pack file and store it in the cache. This is helpful
for pack files containing only tree blobs, as it is likely that the same
file is accessed again in the future.
2017-09-24 23:11:23 +02:00
Alexander Neumann 5436154f0d cache: Add PerformReadahead 2017-09-24 23:11:23 +02:00
Alexander Neumann 809e218d20 cache: Improve debug logs 2017-09-24 23:11:23 +02:00