mu4e: improve mail-retrieval error handling

Add two new customization variables:
  mu4e-index-update-error-continue
  mu4e-index-update-error-warning

With these, we can configure what happens when the mail-retrieval
program finishes with a non-zero exit code.

Make the default to warn but continue; it seems quite some users got
bitten by the old behavior of not updating after an error (which may
only be a pseudo-error). offlineimap/mbsync do not document their exit
codes very well, unlike fetchmail.

Also update manual for this.

mu4e/mu4e-utils.el

@@ -90,7 +90,6 @@ User's addresses are set in `mu4e-user-mail-address-list')."
 
 
 
-
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; the standard folders can be functions too
 (defun mu4e~get-folder (foldervar msg)
@@ -834,11 +833,15 @@ frame to display buffer BUF."
  	  (maybe-error (or (not (eq status 'exit)) (/= code 0)))
 	  (buf (and (buffer-live-p mu4e~update-buffer) mu4e~update-buffer))
 	  (win (and buf (get-buffer-window buf))))
-    ;; there may be an error, give the user up to 5 seconds to check
     (message nil)
     (if maybe-error
-      (sit-for 5)
+      (progn 
-      (mu4e-update-index))
+	(when mu4e-index-update-error-warning
+	  (mu4e-message "Update process returned with non-zero exit code")
+	  (sit-for 5))
+	(when mu4e-index-update-error-continue 
+	  (mu4e-update-index))) 
+      (mu4e-update-index))  
     (if (window-live-p win)
       (with-selected-window win (kill-buffer-and-window))
       (when (buffer-live-p buf) (kill-buffer buf)))))
@@ -858,9 +861,10 @@ in the background; otherwise, pop up a window."
   (run-hooks 'mu4e-update-pre-hook)
   (unless mu4e-get-mail-command
     (mu4e-error "`mu4e-get-mail-command' is not defined"))
+
   (let* ((process-connection-type t)
 	  (proc (start-process-shell-command
-		  "mu4e-update" "mu4e-update"
+		  "mu4e-update" " *mu4e-update*"
 		  mu4e-get-mail-command))
 	  (buf (process-buffer proc))
 	  (win (or run-in-background

mu4e/mu4e-vars.el

@@ -57,18 +57,31 @@ link."
 
 (defcustom mu4e-get-mail-command "true"
   "Shell command to run to retrieve new mail.
-Common values are \"offlineimap\" and \"fetchmail\", but you use
+Common values are \"offlineimap\", \"fetchmail\" and \"mbsync\",
-arbitrary shell-commands.
+but you use arbitrary shell-commands. If you set it to
-
+\"true\" (the default), the command won't don't anything, which is
-If you set it to \"true\" (the default), the command won't don't
+useful if you get your mail without the need to explicitly run any
-anything, which is useful if you get your mail without the need to
+scripts, for example when running yout own mail-server."
-explicitly run any scripts, for example when running yout own
-mail-server.
-"
   :type 'string
   :group 'mu4e
   :safe 'stringp)
 
+(defcustom mu4e-index-update-error-warning t
+  "Whether to display warnings when we the retrieval process (as
+  per `mu4e-get-mail-command') finished with a non-zero exit code."
+  :type 'boolean
+  :group 'mu4e
+  :safe 'booleanp)
+
+(defcustom mu4e-index-update-error-continue t
+  "Whether to continue with indexing when we the retrieval
+  process (as per `mu4e-get-mail-command') finished with a non-zero
+  exit code."
+  :type 'boolean
+  :group 'mu4e
+  :safe 'booleanp)
+
+
 (defcustom mu4e-update-interval nil
   "Number of seconds between automatic calls to retrieve mail and
 update the database. If nil, don't update automatically. Note,

mu4e/mu4e.texi

@@ -55,8 +55,8 @@ Documentation License.''
 Welcome to @t{mu4e} @value{mu-version}!
 
 @t{mu4e} (@t{mu}-for-emacs) is an e-mail client for GNU-Emacs version 24
-(version 23 may works as well, but is tested as much), built on top of
+(version 23 may works as well, but is not tested as much), built on top
-the @t{mu}@footnote{@url{http://www.djcbsoftware.nl/code/mu}} e-mail
+of the @t{mu}@footnote{@url{http://www.djcbsoftware.nl/code/mu}} e-mail
 search engine. @t{mu4e} is optimized for fast handling of large amounts
 of e-mail.
 
@@ -123,24 +123,27 @@ Appendices
 
 Fair question.
 
-I'm not sure the world needs yet another e-mail client, but perhaps @emph{I}
+I'm not sure the world needs yet another e-mail client, but perhaps
-do! I (the author) spend a @emph{lot} of time dealing with e-mail, both
+@emph{I} do!
-professionally and privately. Having an efficient e-mail client is essential.
+
-Since none of the existing ones worked the way I wanted, I created my
+I (the author) spend a @emph{lot} of time dealing with e-mail, both
-own. @command{emacs} is an integral part of my workflow, so it made a lot of
+professionally and privately. Having an efficient e-mail client is
-sense to use it for e-mail as well. And as I already had written an e-mail
+essential.  Since none of the existing ones worked the way I wanted, I
-search engine (@t{mu}), it seemed only logical to use that as a basis.
+created my own. @command{emacs} is an integral part of my workflow, so
+it made a lot of sense to use it for e-mail as well. And as I already
+had written an e-mail search engine (@t{mu}), it seemed only logical to
+use that as a basis.
 
 @node Other mail clients
 @section Other mail clients
 
 Under the hood, @t{mu4e} is fully search-based, similar to programs like
 @t{notmuch}@footnote{@url{http://notmuchmail.org}},
-@t{md}@footnote{@url{https://github.com/nicferrier/md}} and
+@t{md}@footnote{@url{https://github.com/nicferrier/md} (inactive)} and
-@t{sup}@footnote{@url{http://sup.rubyforge.org/}}. However, @t{mu4e}'s
+@t{sup}@footnote{@url{http://sup.rubyforge.org/}
-user-interface is quite different. @t{mu4e}'s mail handling (deleting, moving
+(unreachable)}. However, @t{mu4e}'s user-interface is quite
-etc.) is inspired by
+different. @t{mu4e}'s mail handling (deleting, moving etc.) is inspired
-@emph{Wanderlust}@footnote{@url{http://www.gohome.org/wl/}} (another
+by @emph{Wanderlust}@footnote{@url{http://www.gohome.org/wl/}} (another
 @code{emacs}-based e-mail client),
 @t{mutt}@footnote{@url{http://www.mutt.org/}} and @t{dired}.
 
@@ -156,9 +159,10 @@ There are a number of things that @t{mu4e} does @emph{not} do:
 @item @t{mu}/@t{mu4e} do @emph{not} get your e-mail messages from
 a mail server. That task is delegated to other tools, such as
 @t{offlineimap}@footnote{@url{http://offlineimap.org/}},
-@t{isync}@footnote{@url{http://isync.sourceforge.net/}} or
+@t{isync/mbsync}@footnote{@url{http://isync.sourceforge.net/}} or
 @t{fetchmail}@footnote{@url{http://www.fetchmail.info/}}. As long as the
-messages end up in a maildir, @t{mu4e} and @t{mu} are happy to deal with them.
+messages end up in a maildir, @t{mu4e} and @t{mu} are happy to deal with
+them.
 @item @t{mu4e} also does @emph{not} implement sending of messages; instead, it
 depends on @t{smptmail} (@inforef{Top,,smtpmail}), which is part of
 @command{emacs}. In addition, @t{mu4e} piggybacks on Gnus' message editor;
@@ -173,23 +177,23 @@ efficiently as possible.
 @node Becoming a mu4e user
 @section Becoming a @t{mu4e} user
 
-If @t{mu4e} looks like something for you, give it a shot! We've been trying
+If @t{mu4e} looks like something for you, give it a shot! We've been
-hard to make it as easy as possible to set up and use; and while you can use
+trying hard to make it as easy as possible to set up and use; and while
-elisp in various places to augment @t{mu4e}, a lot of knowledge about
+you can use elisp in various places to augment @t{mu4e}, a lot of
-programming or elisp shouldn't be required. The idea is always to provide
+knowledge about programming or elisp shouldn't be required. The idea is
-sensible defaults.
+to provide sensible defaults, and allow for customization.
 
 When you take @t{mu4e} into use, it's a good idea to subscribe to the
 @t{mu}/@t{mu4e}-mailing
 list@footnote{@url{http://groups.google.com/group/mu-discuss}}.
 
-If you have suggestions for improvements or bug reports, please use the GitHub
+If you have suggestions for improvements or bug reports, please use the
-issues list@footnote{@url{https://github.com/djcb/mu/issues}}. In bug reports,
+GitHub issues list@footnote{@url{https://github.com/djcb/mu/issues}}. In
-please clearly specify the versions of @t{mu}/@t{mu4e} and @command{emacs} you
+bug reports, please clearly specify the versions of @t{mu}/@t{mu4e} and
-are using, as well as any other relevant details. Also, if it is about the
+@command{emacs} you are using, as well as any other relevant
-behavior for specific messages, please attach the raw message (that is, the
+details. Also, if it is about the behavior for specific messages, please
-message file as it exists in your maildir); you can of course strip it of any
+attach the raw message (that is, the message file as it exists in your
-personal informatiion.
+maildir); you can of course strip off any personal information.
 
 If you are new to all this, the somewhat paternalistic @emph{``How to ask
 questions the smart
@@ -199,9 +203,9 @@ be a good read.
 @node Getting started
 @chapter Getting started
 
-In this chapter, we go through the installation of @t{mu4e} and its basic
+In this chapter, we go through the installation of @t{mu4e} and its
-setup. After we have succeeded in @ref{Getting mail}, and @pxref{Indexing your
+basic setup. After we have succeeded in @ref{Getting mail}, and
-messages}, we discuss @ref{Basic configuration}.
+@pxref{Indexing your messages}, we discuss @ref{Basic configuration}.
 
 After these steps, @t{mu4e} should be ready to go!
 
@@ -228,7 +232,8 @@ Xapian@footnote{@url{http://xapian.org/}} and
 GMime@footnote{@url{http://spruce.sourceforge.net/gmime/}}.
 
 @t{mu} has optional support for the Guile 2.x (Scheme) programming
-language. There are also some GUI-tools, which require GTK+ 3.x and Webkit.
+language. There are also some GUI-tools, which require GTK+ 3.x and
+Webkit.
 
 If you intend to compile @t{mu} yourself, you need to have the typical
 development tools, such as C and C++ compilers (both @command{gcc} and
@@ -388,10 +393,10 @@ need to provide the @t{--maildir=~/Maildir} since it is the default; see the
 @t{mu-index} man-page for details} and fill the database, and give progress
 information while doing so.
 
-The indexing process may take a few minutes the first time you do it (for
+The indexing process may take a few minutes the first time you do it
-thousands of e-mails); afterwards it is much faster, since @t{mu} only scans
+(for thousands of e-mails); afterwards it is much faster, since @t{mu}
-messages that are new or have changed. Indexing is discussed in full detail in
+only scans messages that are new or have changed. Indexing is discussed
-the @t{mu-index} man-page.
+in full detail in the @t{mu-index} man-page.
 
 After the indexing process has finished, you can quickly test if everything
 worked, by trying some command-line searches, for example
@@ -428,12 +433,12 @@ with @t{MU4E-PATH} replaced with the actual path.
 @node Folders
 @section Folders
 
-The next step is to tell @t{mu4e} where it can find your Maildir, and some
+The next step is to tell @t{mu4e} where it can find your Maildir, and
-special folders. So, for example@footnote{Note that the folders
+some special folders. So, for example@footnote{Note that the folders
 (@t{mu4e-sent-folder}, @t{mu4e-drafts-folder}, @t{mu4e-trash-folder} and
-@t{mu4e-refile-folder}) can also be @emph{functions} that are evaluated at
+@t{mu4e-refile-folder}) can also be @emph{functions} that are evaluated
-runtime. This allows for dynamically changing them depending on context. See
+at runtime. This allows for dynamically changing them depending on the
-@ref{Dynamic folders} for details.}:
+context. See @ref{Dynamic folders} for details.}:
 @lisp
   ;; these are actually the defaults
   (setq
@@ -449,34 +454,59 @@ folder names are all relative to @code{mu4e-maildir}. Also note that
 this must @emph{not} be a symbolic link.
 
 @node Retrieval and indexing
-@section Retrieval and indexing
+@section Retrieval and indexing with mu4e
 
 As we have seen, we can do all of the mail retrieval @emph{outside} of
 @command{emacs}/@t{mu4e}. However, you can also do it from within
-@t{mu4e}. For that, set the variable @code{mu4e-get-mail-command} to the
+@t{mu4e}. 
-program or shell command you want to use for retrieving mail. You can
-then retrieve your e-mail using @kbd{M-x mu4e-update-mail-and-index}, or
-@kbd{C-S-u} in all @t{mu4e}-views; alternatively, you can use @kbd{C-c
-C-u}, which may be more convenient if you use emacs in a
-terminal. 
 
-Please note that indexing only takes place if getting mail finished
+@subsection Basics
-successfully, that is, if the mail-retrieval program ended with
+ 
-exit-code 0. If you're mail-retrieval program does not do that, you need
+To set up mail-retrieval from withing @t{mu4e}, set the variable
-to wrap it in a shell-script.
+@code{mu4e-get-mail-command} to the program or shell command you want to
-
+use for retrieving mail. You can then get your e-mail using @kbd{M-x
-If you don't have a specific command for getting mail, for example because you
+mu4e-update-mail-and-index}, or @kbd{C-S-u} in all @t{mu4e}-views;
-are running your own mail-server, you can set @code{mu4e-get-mail-command} to
+alternatively, you can use @kbd{C-c C-u}, which may be more convenient
-@t{"true"}, in which case @t{mu4e} won't try to get new mail, but still
+if you use emacs in a terminal.
-re-index your messages.
 
 You can interrupt the (foreground) update process with @kbd{q}.
 
-You can also update your mail and index periodically in the background,
+It is possible to update your mail and index periodically in the
-by setting the variable @code{mu4e-update-interval} to the number of
+background, by setting the variable @code{mu4e-update-interval} to the
-seconds between these updates. If set to @code{nil}, it won't update at
+number of seconds between these updates. If set to @code{nil}, it won't
-all. After you make changes to @code{mu4e-update-interval}, @t{mu4e}
+update at all. After you make changes to @code{mu4e-update-interval},
-must be restarted before the changes take effect.
+@t{mu4e} must be restarted before the changes take effect.
+
+@subsection Handling errors during mail retrieval
+
+If the mail-retrieval process returns with a non-zero exit code,
+@t{mu4e} will show a warning (unless
+@code{mu4e-index-update-error-warning} is set to @code{nil}), but then
+try to index your maildirs anyway (unlesds
+@code{mu4e-index-update-error-continue} is set to @code{nil}).
+
+Reason for these defaults is that some of the mail-retrieval programs
+may return non-zero, even when the updating process succeeded; however,
+it is hard to tell such pseudo-errors from real ones like 'login
+failed'.
+
+If you need more refinement, it may be useful to wrap the mail-retrieval
+program in a shell-script, for example @t{fetchmail} returns 1 to
+indicate 'no mail'; we can handle that with:
+@lisp
+  (setq mu4e-get-mail-command "fetchmail -v || [ $? -eq 1 ]")
+@end lisp
+A similar approach can be used with other mail retrieval programs,
+although not all of them have their exit codes documented.
+
+@subsection Implicit mail retrieval 
+
+If you don't have a specific command for getting mail, for example
+because you are running your own mail-server, you can leave
+@code{mu4e-get-mail-command} at @t{"true"} (the default), in which case
+@t{mu4e} won't try to get new mail, but still re-index your messages.
+
+@subsection Example setup
 
 A simple setup could look something like:
 
@@ -486,14 +516,15 @@ A simple setup could look something like:
    mu4e-update-interval 300)             ;; update every 5 minutes
 @end lisp
 
-A hook @code{mu4e-update-pre-hook} is available which is run right before
+A hook @code{mu4e-update-pre-hook} is available which is run right
-starting the process, which you can for example to influence
+before starting the process. That can be useful, for example, to
-@code{mu4e-get-mail-command} based on the the current situation (location,
+influence, @code{mu4e-get-mail-command} based on the the current
-time of day, ...).
+situation (location, time of day, ...).
 
-It is possible to get notifications when the indexing process does any updates
+It is possible to get notifications when the indexing process does any
-- for example when receiving new mail. See @code{mu4e-index-updated-hook} and
+updates - for example when receiving new mail. See
-some tips on its usage in the @ref{FAQ}.
+@code{mu4e-index-updated-hook} and some tips on its usage in the
+@ref{FAQ}.
 
 @node Sending mail
 @section Sending mail
@@ -3206,7 +3237,7 @@ see @ref{(emacs) Mail Aliases}.
 Once more, see @ref{Compose hooks}.
 @item @emph{How can I influence the way the original message looks when
 replying or forwarding?} Since @code{mu4e-compose-mode} derives from
-@code{message-mode}, you can re-use many of the latter's facilities.
+@code{message-mode}, you can re-use many of its facilities.
 @inforef{Insertion Variables,,message}.
 @item @emph{How can I easily include attachments in the messages I write?}
 You can drag-and-drop from your desktop; alternatively, you can use @t{dired}
@@ -3264,7 +3295,7 @@ as a todo-list.
 
 @itemize
 @item @emph{mu4e does not work well if the @command{emacs} language environment is not
-UTF-8}; so, if you problems with encodings, be sure to have
+UTF-8}; so, if you encounter problems with encodings, be sure to have
 @code{(set-language-environment "UTF-8")} in your @file{~/.emacs}.
 @item @emph{Thread handling is incomplete.} While threads are calculated and are
 visible in the headers buffer, you cannot collapse/open them.
@@ -3277,9 +3308,10 @@ conflicts with @t{mu4e}}. @t{notmuch} running in parallel with
 error in process filter: mu4e-error-handler: Error 70: cannot read
 ~/Maildir/...
 @end verbatim
-when sending a reply to a new e-mail. This seems to be caused by @t{notmuch}
+when sending a reply to some e-mail. This seems to be caused by
-changing the name of the original message file while @t{mu4e} is working in on
+@t{notmuch} changing the name of the original message file while
-it. To prevent this, deactivate @t{notmuch} in your Emacs setup.
+@t{mu4e} is working on it. To prevent this, deactivate @t{notmuch} in
+your Emacs setup.
 @item @emph{The PDF-version of the manual does not show any of the non-ASCII
 characters} - this is because, sadly, @t{texi2pdf} documentation system
 does not support those. There is not much we can do about that.