Commit Graph

1648 Commits

Author SHA1 Message Date
Michael Eischer 5974a79497 features: add basic feature flag implementation 2024-03-09 17:29:52 +01:00
Michael Eischer b6520038fd snapshots: Print snapshot size stored in snapshots 2024-02-23 20:27:13 +01:00
Michael Eischer 38f91d3b5e backup: store statistics in snapshot 2024-02-23 20:27:13 +01:00
Michael Eischer a59f654fa6 archiver: refactor summary collection from ui into the archiver 2024-02-23 20:27:13 +01:00
Aneesh Nireshwalia eeb1aa5388
Add ability to report warnings to terminal
Report warnings to terminal when unrecognized generic attributes are found in the repository.
2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia 0962917974
Support windows metadata using generic attribs
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
2024-02-22 17:31:20 -07:00
Michael Eischer 0a65a0f94f update comment 2024-02-22 22:00:42 +01:00
Brian Harring 30e979d252
Catch SIGTERM, run cleanup
The previous code only ran cleanup (lock release for example) on SIGINT.  For
anyone running restic in a container, the signal is going to be SIGTERM which
means containerized execution would leave locks behind.

While this could be addressed via interposing dumb-init to translate the signal,
a `kill` invocation is going to default to SIGTERM, so the same problem exists
for non container users.

Signed-off-by: Brian Harring <ferringb@gmail.com>
2024-02-19 11:12:15 +01:00
Michael Eischer 80754dbf0c
Merge pull request #4664 from MichaelEischer/ls-unified-json-output
ls: include standard `message_type` field in output
2024-02-18 15:47:41 +00:00
Michael Eischer 527a3ff2b2 check: link to troubleshooting guide 2024-02-12 20:25:15 +01:00
Michael Eischer 544fe38786 check: suggest repair pack for all damaged packs 2024-02-12 20:20:13 +01:00
Michael Eischer 772e3416d1 repair pack: drop feature flag 2024-02-12 20:20:12 +01:00
Michael Eischer ee1ff3c1d0 backup: improve error message if exclude file cannot be loaded 2024-02-11 22:26:13 +01:00
Alexander Neumann c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
lou 80db02fc35 dump flag --target should be allowed to write existing file
Signed-off-by: lou <alex1988@outlook.com>
2024-02-10 18:39:06 +01:00
Michael Eischer e46b21ab80 key: fix integration test for invalid arguments 2024-02-08 20:52:30 +01:00
Srigovind Nayak 795d33b3ee key: move add, list, remove, passwd to sub-commands
docs: improve the sub-command docs

changelog: add the unreleased changelog for the key command updates

key: update integration tests
2024-02-06 01:47:43 +05:30
Michael Eischer f5ffa40652 dump: minor cleanups 2024-02-05 20:10:52 +01:00
Srigovind Nayak 175c14b5c9 dump: add --target option 2024-02-05 20:10:52 +01:00
Michael Eischer b63bfd2257 Merge branch 'patch-release' 2024-02-04 20:21:42 +01:00
Alexander Neumann 0f9fa44de5 Set development version for 0.16.4 2024-02-04 19:50:56 +01:00
Alexander Neumann 3786536dc1 Add version for 0.16.4 2024-02-04 19:50:52 +01:00
Michael Eischer a72c2b74f3 Apply changelog entry / documentation improvements from review 2024-02-04 19:10:06 +01:00
Michael Eischer 2a0bd2b637 rename `--no-verify-pack` to `--no-extra-verify` 2024-02-04 19:10:05 +01:00
Michael Eischer dc11d012bb Make --no-verify-pack globally available
Verifying all blobs before upload comes with a notable performance
impact. Allow users to skip it if necessary.
2024-02-04 19:09:49 +01:00
Michael Eischer 5957417b1f Apply changelog entry / documentation improvements from review 2024-02-04 18:55:41 +01:00
Michael Eischer 86b38a0b17 rename `--no-verify-pack` to `--no-extra-verify` 2024-02-04 17:01:05 +01:00
Michael Eischer 66e8971659 Make --no-verify-pack globally available
Verifying all blobs before upload comes with a notable performance
impact. Allow users to skip it if necessary.
2024-02-04 15:31:42 +01:00
Michael Eischer 8b08b522c9
Merge pull request #4669 from MichaelEischer/fix-rewrite-typo
rewrite: fix typo in dry-run output
2024-01-31 22:08:12 +01:00
Michael Eischer eaf9659efc
Merge pull request #4657 from numerigraphe/fix-key-add-id
Properly report the ID of newly added keys
2024-01-31 20:59:41 +00:00
Michael Eischer ba136ff60c rewrite: fix typo in dry-run output 2024-01-31 21:48:37 +01:00
Lionel Sausin 8fbe328371 Properly report the ID of newly added keys
Other commands like key list and key remove show the key's ID.

Showing the ID here lets users easily reuse the ID as a key hint for subsequent
commands.
In particular, a key hint is needed when the repository has many keys - otherwise
opening the repository may fail with "Fatal: maximum number of keys reached" even
when a proper password is provided.

Fixes #4656
2024-01-29 10:12:49 +01:00
Michael Eischer 4273e06a43
Merge pull request #4662 from MichaelEischer/clarify-backup-ignore-inode
backup: clarify that --ignore-inode also ignores ctime
2024-01-27 18:25:11 +01:00
Michael Eischer 5dca8a70d5 ls: include standard `message_type` field in output 2024-01-27 15:48:24 +01:00
Michael Eischer 765729d009 key: return an error if subcommand is unknown 2024-01-27 15:33:49 +01:00
Michael Eischer a09d51d96c backup: clarify that --ignore-inode also ignores ctime 2024-01-27 13:42:29 +01:00
Michael Eischer e44e4b00a6
Merge pull request #4550 from ndecker/ls-ncdu
Ls ncdu
2024-01-27 12:27:35 +00:00
Michael Eischer 261737abc8 ls: only allow either --json or --ncdu 2024-01-27 13:17:33 +01:00
Michael Eischer a2f2f8fb4c fix linter warning 2024-01-27 13:17:33 +01:00
Michael Eischer 4bae54d040 ls: test ncdu output format 2024-01-27 13:17:33 +01:00
Michael Eischer 509b339d54 ls: correctly handle setuid/setgit/sticky bit in ncdu output 2024-01-27 13:17:33 +01:00
Michael Eischer a2fe337610 ls: unify printer implementations 2024-01-27 13:17:33 +01:00
Michael Eischer 1b008c92d3 ls: rework ncdu output to use walker.LeaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer 9ecbda059c walker: add callback to inform about leaving a directory 2024-01-27 13:17:32 +01:00
Nils Decker a9310948cf command ls: add option for ncdu output
NCDU (NCurses Disk Usage) is a tool to analyse disk usage of directories.
It has an option to save a directory tree and analyse it later.
This patch adds an output option to the ls command.

A snapshot can be seen with

`restic ls latest --ncdu | ncdu -f -`

- https://dev.yorhel.nl/ncdu
2024-01-27 13:06:52 +01:00
Michael Eischer bfb56b78e1 replace some usages of restic.Repository with more specific interface
This should eventually make it easier to test the code.
2024-01-27 13:02:02 +01:00
Michael Eischer 3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs
Refactor and test `repair packs`
2024-01-27 13:00:51 +01:00
Michael Eischer 724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey
repository: Introduce RemoveKey function
2024-01-27 13:00:42 +01:00
Michael Eischer feeab84204 repair pack: extract the repair logic into the repository package
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
2024-01-27 12:51:45 +01:00
Michael Eischer d7a50fe739 properly show termstatus progress bar if visible less than one frame
If a progress bar using termstatus was only visible for less than one
frame, then its output could be lost.
2024-01-27 12:51:40 +01:00
Michael Eischer 6b65a495b1 backup/restore: fix termstatus initialization
The termstatus must only be canceled once the command has returned.
Otherwise output may be lost when the context gets canceled.
2024-01-27 12:51:08 +01:00
Michael Eischer d26d2d41f8 backup/restore: extract termstatus initialization 2024-01-27 12:51:08 +01:00
Michael Eischer cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00
Michael Eischer bedff1ed6d split deleteFiles into UI and logic parts 2024-01-27 12:51:08 +01:00
Michael Eischer c13bf0b607 repository: Introduce RemoveKey function
This replaces directly removing keys via the backend.
2024-01-27 12:42:58 +01:00
Michael Eischer 5b36c4eb5f
Merge pull request #4647 from MichaelEischer/reduce-globals
Remove all usages of the global command-specific options
2024-01-23 19:46:15 +01:00
Michael Eischer 66103aea3d Remove all usages of the global command-specific options
Now, every command uses an options struct, which is passed to the run*
function by the command.RunE method.
2024-01-23 19:21:39 +01:00
Michael Eischer 767c2539a0 backup: Improve help text for `--stdin-from-command` 2024-01-21 22:06:54 +01:00
Michael Eischer 2c310a526e repository: Replace StreamPack function with LoadBlobsFromPack method
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.

The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer 0b39940fdb walker: Remove ignoreTrees functionality
It was only used in two places:
- stats: apparently as a minor performance optimization, which is
  unlikely to be important
- find: filtered directories would be ignored. However, this
  optimization missed that it is possible that two directories have the
  exact same content. Such directories would be incorrectly ignored too.
  Example:
```
mkdir test test/a test/b
restic backup test
restic find latest test/b
-> incorrectly does not return anything
```

Thus, remove the functionality as it's apparently too complex to use
correctly.
2024-01-19 21:16:06 +01:00
Michael Eischer 03e06d0797 Merge branch 'patch-release' 2024-01-14 21:38:17 +01:00
Alexander Neumann 0ec9383ba2 Set development version for 0.16.3 2024-01-14 20:21:45 +01:00
Alexander Neumann abca112404 Add version for 0.16.3 2024-01-14 20:21:45 +01:00
Michael Kuhn 3fa6b2de4a Fix repository not being printed when using repository file
When using `RESTIC_REPOSITORY_FILE` in combination with `restic init`,
the repository is missing in the output:
```
$ restic init
created restic repository 3c872be20f at
[...]
```
This is due to the code using `gopts.Repo`, which is empty in this case.
2024-01-08 21:33:05 +01:00
Michael Eischer 5cd000f4b0 CI: update golangci-lint
Necessary to properly support Go 1.21.
2024-01-08 21:33:05 +01:00
Michael Eischer 1604922360
Merge pull request #4527 from adamantike/cmd/copy/prefix-hostname-to-snapshot-paths
cmd: Add hostname to snapshot display output
2024-01-06 19:32:14 +00:00
Michael Eischer 33b7c84a7a deduplicate string formatting of snapshot metadata
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
2024-01-06 20:20:51 +01:00
Michael Manganiello 045aa64558 cmd/copy: Prefix hostname to snapshot display output
This change better resembles the output generated by `Snapshot.String()`,
which includes both username and hostname.

Closes #4506

Before:

```
$ restic copy --from-repo /srv/restic-repo
repository 3666882b opened (version 2, compression level auto)
repository 0085c387 opened (version 2, compression level auto)
created new cache in /home/mike/.cache/restic
[0:00] 100.00%  1 / 1 index files loaded
[0:00]          0 index files loaded

snapshot 32b39a20 of [/home/mike/data] at 2023-10-21 16:01:13.979948154 -0300 -03)
  copy started, this may take a while...
[0:00] 100.00%  1 / 1 packs copied
snapshot 10331fdd saved
```

After:

```
$ restic copy --from-repo /srv/restic-repo
repository 3666882b opened (version 2, compression level auto)
repository 0085c387 opened (version 2, compression level auto)
[0:00] 100.00%  1 / 1 index files loaded
[0:00]          0 index files loaded

snapshot 32b39a20 of [/home/mike/data] at 2023-10-21 16:01:13.979948154 -0300 -03 by mike@desktop)
  copy started, this may take a while...
[0:00] 100.00%  1 / 1 packs copied
snapshot a67bd1ee saved
```
2024-01-06 20:20:46 +01:00
Michael Eischer 4f6b1bb6f6 diff: document limitations regarding metadata 2024-01-06 20:12:47 +01:00
Michael Eischer 3549635243 diff: copy nodes before modifying them for bitrot detection 2024-01-06 20:12:47 +01:00
Daniel Danner a7dc18e697 Add bitrot detection to "diff" command
This introduces a new modifier to the output of the diff command. It
appears whenever two files being compared only differ in their content
but not in their metadata. As far as we know, under normal
circumstances, this should only ever happen if some kind of bitrot has
happened in the source file. The prerequisite for this detection to work
is that the right-side snapshot of the comparison has been created with
"backup --force".
2024-01-06 20:12:47 +01:00
Michael Eischer 649a6409ee rewrite: cleanup tests 2023-12-24 15:36:22 +01:00
Michael Eischer c31f5f986c rewrite: Minor cleanups 2023-12-24 15:36:22 +01:00
Michael Eischer 2730d05fce rewrite: Don't walk snapshot content if only metadata is modified 2023-12-24 15:36:22 +01:00
Michael Eischer 893d0d6325 rewrite: cleanup new metadata options and fix no parameters check 2023-12-24 15:36:22 +01:00
Gabriel Kabbe a02d8d75c2 rewrite: Implement rewriting metadata 2023-12-24 15:36:22 +01:00
Gabriel Kabbe 7bf38b6c50 rewrite: Add test TestRewriteMetadata 2023-12-24 15:36:22 +01:00
Gabriel Kabbe da1704b2d5 rewrite: Add tests
Pass nil instead of metadata to existing tests
2023-12-24 15:36:19 +01:00
Gabriel Kabbe 3026baea07 rewrite: Add structs for tracking metadata changes
Adds

  * snapshotMetadataArgs, which holds the new metadata as strings parsed from
    the command line

  * snapshotMetadata, which holds the new metadata converted to the
    correct types
2023-12-24 14:43:07 +01:00
Sébastien Gross f7587be28f mount: detect mountpoint does not exist before opening the repository
Bug #1681 suggests that restic should not be nice to user and should
refrain from creating a mountpoint if it does not exist. Nevertheless,
it currently opens the repository before checking for the mountpoint's
existence. In the case of large or remote repositories, this process
can be time-consuming, delaying the inevitable outcome.

    /restic mount --repo=REMOTE --verbose /tmp/backup
    repository 33f14e42 opened (version 2, compression level max)
    [0:38] 100.00%  162 / 162 index files loaded
    Mountpoint /tmp/backup doesn't exist
    stat /tmp/backup: no such file or directory

    real	0m39.534s
    user	1m53.961s
    sys	0m3.044s

In this scenario, 40 seconds could have been saved if the nonexistence
of the path had been verified beforehand.

This patch relocates the mountpoint check to the beginning of the
runMount function, preceding the opening of the repository.

    /restic mount --repo=REMOTE --verbose /tmp/backup
    Mountpoint /tmp/backup doesn't exist
    stat /tmp/backup: no such file or directory

    real	0m0.136s
    user	0m0.018s
    sys	0m0.027s

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2023-12-24 11:54:18 +01:00
Michael Eischer 30e6ed038c
Merge pull request #4586 from Gelma/typos
Fix typos
2023-12-23 13:21:50 +00:00
Michael Kuhn fd2fb233aa Fix repository not being printed when using repository file
When using `RESTIC_REPOSITORY_FILE` in combination with `restic init`,
the repository is missing in the output:
```
$ restic init
created restic repository 3c872be20f at
[...]
```
This is due to the code using `gopts.Repo`, which is empty in this case.
2023-12-23 13:49:22 +01:00
Michael Eischer dbbd31bc3a CI: update golangci-lint
Necessary to properly support Go 1.21.
2023-12-23 13:41:30 +01:00
Andrea Gelmini 241916d55b
Fix typos 2023-12-06 13:11:55 +01:00
Michael Eischer b72de5a883 Merge branch 'patch-release' 2023-11-12 11:52:21 +01:00
Tobias Speicher ce53ea32c6
Split `go_target` into `go_os` and `go_arch` 2023-11-01 22:43:38 +01:00
Tobias Speicher 10cbc169c1
Use different function to be more consistent with other code 2023-11-01 22:18:37 +01:00
Tobias Speicher d45cc52468
command version: add json output option 2023-10-31 19:31:07 +01:00
Alexander Neumann 59fe24cb2b Set development version for 0.16.2 2023-10-29 19:50:12 +01:00
Alexander Neumann 1a5efcf680 Add version for 0.16.2 2023-10-29 19:50:07 +01:00
Michael Eischer ee305e6041 backup: rework error reporting for subcommand 2023-10-27 23:58:52 +02:00
Michael Eischer 317144c1d6 fs: merge command startup into CommandReader 2023-10-27 23:58:51 +02:00
Michael Eischer 7d879705ad fs: cleanup CommandReader implementation 2023-10-27 23:58:51 +02:00
Enrico204 37a312e505 restic-from-command: use standard behavior when no output and exit code 0 from command
The behavior of the new option should reflect the behavior of normal backups: when the command exit code is zero and there is no output in the stdout, emit a warning but create the snapshot. This commit fixes the integration tests and the ReadCloserCommand struct.
2023-10-27 23:58:51 +02:00
Enrico204 c0ca54dc8a restic-from-command: add tests 2023-10-27 23:58:51 +02:00
Enrico204 81f8d473df restic-from-command: abort snapshot on non-zero exit codes 2023-10-27 23:58:51 +02:00
Enrico204 4e5caab114 stdin-from-command: implemented suggestions in #4254
The code has been refactored so that the archiver is back to the original code, and the stderr is handled using a go routine to avoid deadlock.
2023-10-27 23:58:51 +02:00
Sebastian Hoß 25350a9c55 Extend SnapshotOptions w/ command data
In order to determine whether to save a snapshot, we need to capture the exit code returned by a command. In order to provide a nice error message, we supply stderr as well.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß a2b76ff34f Start command from --stdin-from-command
It acts similar to --stdin but reads its data from the stdout of the given command instead of os.Stdin.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00
Sebastian Hoß 333fe1c3cf Align Stdin and StdinCommand in conditionals
In order to run with --stdin-from-command we need to short-circuit some functions similar to how it is handled for the --stdin flag. The only difference here is that --stdin-from-command actually expects that len(args) should be greater 0 whereas --stdin does not expect any args at all.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
2023-10-27 23:58:51 +02:00