clitools
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
1,991 Commits 7 Branches 44 Tags 203 MiB
Commit Graph

1991 Commits

Author SHA1 Message Date
Alexander Neumann b1ed74eb43 Merge pull request #669 from zcalusic/master 
Fix REST backend HTTP keepalive
 2016-11-10 21:05:14 +01:00
Zlatko Čalušić d8f0e7cbd1 Fix REST backend HTTP keepalive 
This is subtle.  A combination od fast client disk (read: SSD) with lots
of files and fast network connection to restic-server would suddenly
start getting lots of "dial tcp: connect: cannot assign requested
address" errors during backup stage.  Further inspection revealed that
client machine was plagued with TCP sockets in TIME_WAIT state.  When
ephemeral port range was finally exhausted, no more sockets could be
opened, so restic would freak out.

To understand the magnitude of this problem, with ~18k ports and default
timeout of 60 seconds, it means more than 300 HTTP connections per
seconds were created and teared down.  Yeah, restic-server is that
fast. :)

As it turns out, this behavior was product of 2 subtle issues:

1) The body of HTTP response wasn't read completely with io.ReadFull()
   at the end of the Load() function.  This deactivated HTTP keepalive,
   so already open connections were not reused, but closed instead, and
   new ones opened for every new request.  io.Copy(ioutil.Discard,
   resp.Body) before resp.Body.Close() remedies this.

2) Even with the above fix, somehow having MaxIdleConnsPerHost at its
   default value of 2 wasn't enough to stop reconnecting.  It is hard to
   understand why this would be so detrimental, it could even be some
   subtle Go runtime bug.  Anyhow, setting this value to match the
   connection limit, as set by connLimit global variable, finally nails
   this ugly bug.

I fixed several other places where the response body wasn't read in
full (or at all).  For example, json.NewDecoder() is also known not to
read the whole body of response.

Unfortunately, this is not over yet. :( The check command is firing up
to 40 simultaneous connections to the restic-server.  Then, once again,
MaxIdleConnsPerHost is too low to support keepalive, and sockets in the
TIME_WAIT state pile up.  But, as this kind of concurrency absolutely
kill the poor disk on the server side, this is a completely different
bug then.
 2016-11-10 09:32:07 +01:00
Alexander Neumann 5e721afb5d doc/mkdocs: Improve code hilighting 
Additionally, refresh the restic sample output.
 2016-11-08 20:23:39 +01:00
Alexander Neumann 149c01a86a Merge pull request #659 from restic/device-freebsd 
fs.DeviceID(): Return errors whehn fi is nil
 2016-11-05 13:35:16 +01:00
Alexander Neumann 51322a1055 selectFunc: handle nil 2016-11-05 12:38:33 +01:00
Alexander Neumann c5bc802ff0 fs.DeviceID(): Return errors when fi is nil 2016-11-05 12:38:17 +01:00
Alexander Neumann 6b88d3b5d0 Merge pull request #651 from justinclift/issue649v1 
Remove redundant check of error var e
 2016-10-26 16:06:14 +02:00
Justin Clift ecc1f92787 Remove redundant check of error var e 
As per #649
 2016-10-25 18:10:53 +01:00
Alexander Neumann d4f76fbe26 Merge pull request #650 from restic/forget-remove-index-load 
forget: do not load index
 2016-10-24 14:33:11 +02:00
Alexander Neumann 1dd72693f9 forget: Remove unneeded index loading 2016-10-24 14:01:23 +02:00
Alexander Neumann fe1013e779 cmds/ls: Format timestamp 2016-10-19 22:11:37 +02:00
Alexander Neumann 84ca5172f0 Remove Debian UID from GPG key printout 2016-10-17 13:10:16 +02:00
Alexander Neumann 7c49255c2a Add hints how to use the `go` tool and direnv 2016-10-17 13:09:56 +02:00
Alexander Neumann a5a9c42185 Merge pull request #646 from stakewinner00/master 
don't print status info when running in the background
 2016-10-15 20:09:42 +02:00
David 5f8a6cea6f don't print status info if running in the background 
clean

fix OS issues & format code

fix issues
 2016-10-15 18:12:19 +00:00
Alexander Neumann 50212805aa Merge pull request #643 from restic/update-poly1305 
Update golang.org/x/crypto/poly1305
 2016-10-14 15:51:57 +02:00
Alexander Neumann cd7feb0148 Update golang.org/x/crypto/poly1305 2016-10-14 12:44:06 +02:00
Alexander Neumann 974f2f78a9 Merge pull request #641 from restic/fix-640 
Improve error message for 'forget'
 2016-10-13 20:41:29 +02:00
Alexander Neumann 250b36eeb1 Improve error message for 'forget' 
$ bin/restic forget /d 7 /w 4 /m 12
    argument "/d" is not a snapshot ID, ignoring
    argument "7" is not a snapshot ID, ignoring
    argument "/w" is not a snapshot ID, ignoring
    argument "4" is not a snapshot ID, ignoring
    argument "/m" is not a snapshot ID, ignoring
    cound not find a snapshot for ID "12", ignoring
 2016-10-10 20:55:02 +02:00
Alexander Neumann 6f72164bbe Merge pull request #638 from hmsdao/patch-fixpath 
Added long paths fix for samba network shares
 2016-10-05 17:07:53 +02:00
Daniel Örn ba8d960c8f using backtics instead of doublequotes 2016-10-05 08:26:32 +02:00
Daniel Örn 84421a7c68 structured file with gofmt 2016-10-05 07:30:46 +02:00
Daniel Örn 5c7325f44a Added long paths fix for samba network shares 2016-10-05 07:09:56 +02:00
Alexander Neumann c45b498a8b Merge pull request #637 from ckemper67/s3-join 
Use path.Join to create the s3 object name within the bucket.
 2016-10-03 22:38:28 +02:00
Christian Kemper a4261dcc9c Use path.Join to create the s3 object name within the bucket. 
path.Join already automatically skips empty path segments when
joining, so this simplifies the s3Path code.
 2016-10-02 16:56:07 -07:00
Alexander Neumann d1ecdf7441 Add VERSION file for 0.3.0 2016-10-02 18:24:14 +02:00
Alexander Neumann 088ca033f8 Reword README 2016-10-02 16:08:05 +02:00
Alexander Neumann 5b7dd32c20 Manual: Reword section about fuse support 2016-10-02 16:03:02 +02:00
Alexander Neumann eb94395f3d Merge pull request #635 from restic/fix-633 
Fix short-hand option clash
 2016-09-29 21:39:05 +02:00
Alexander Neumann 22f5fc5739 Improve help text for slice options 2016-09-29 20:39:55 +02:00
Alexander Neumann e994cacbfe Fix short-hand option clash 2016-09-29 20:37:45 +02:00
Alexander Neumann 3114d41cb7 Merge pull request #632 from restic/rework-debug 
Rework debug message printing
 2016-09-28 21:11:02 +02:00
Alexander Neumann 968b2ece43 Add section to the manual about debug message filters 2016-09-28 20:22:22 +02:00
Alexander Neumann feed54caef Remove timing, simplify function matching 2016-09-28 20:10:40 +02:00
Alexander Neumann 4eddcb344e Update calls to debug.Log() 2016-09-28 19:56:03 +02:00
Alexander Neumann 2ae06a7a01 Rework debug log function 2016-09-28 19:56:03 +02:00
Alexander Neumann 25945718a1 Fix recursive call to debug.Log 2016-09-28 19:56:03 +02:00
Alexander Neumann 254188f38f Merge pull request #631 from restic/switch-to-cobra 
Switch to cobra/pflag for CLI
 2016-09-28 19:54:59 +02:00
Alexander Neumann 3601c39177 Add comments 2016-09-27 20:22:01 +02:00
Alexander Neumann 02f7bb0d4c Add mousetrap library for Windows 2016-09-27 20:13:22 +02:00
Alexander Neumann 565d72ef36 Use cobra for all commands 2016-09-27 19:53:03 +02:00
Alexander Neumann 3806623c23 Vendor cobra and pflag 2016-09-27 19:52:48 +02:00
Alexander Neumann 0fa12839a5 Remove go-flags 2016-09-27 19:52:48 +02:00
Alexander Neumann a257a613d7 Fix debug log 2016-09-27 19:52:48 +02:00
Alexander Neumann 0a752b9fab test helpers: Always print stack trace 2016-09-27 19:50:26 +02:00
Alexander Neumann eeec0d63c2 Merge pull request #630 from restic/remove-unused 
Remove unused bits and pieces
 2016-09-24 12:03:26 +02:00
Alexander Neumann 04d6b5da2f Remove more unused bits 2016-09-21 20:45:18 +02:00
Alexander Neumann 1dfd3b8aa3 Remove unused bits and pieces 
Reported by https://github.com/dominikh/go-unused
 2016-09-21 20:22:32 +02:00
Alexander Neumann 0873821b98 Add section about `--one-file-system` to manual 2016-09-18 20:18:52 +02:00
Alexander Neumann 0a9cbd47c7 Merge pull request #626 from rfjakob/master 
Add "-x", "--one-file-system" option
 2016-09-18 20:03:58 +02:00