mu/guile/mu-guile.texi

\input texinfo.tex    @c -*-texinfo-*-
@c %**start of header
@setfilename mu-guile.info
@settitle mu-guile user manual
@documentencoding utf-8
@c %**end of header

@dircategory The Algorithmic Language Scheme
@direntry
* mu-guile manual: (mu-guile).  Guile bindings for the @t{mu} e-mail indexer/searcher.
@end direntry

@copying
Copyright @copyright{} 2012 Dirk-Jan C. Binnema

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A
copy of the license is included in the section entitled ``GNU Free
Documentation License.''
@end quotation
@end copying

@node Top
@top mu4e Manual

Welcome to @t{mu-guile}!

@t{mu-guile} is a binding of the @t{mu} email search engine and the @t{guile}
programming language.

@menu
* Introduction::
* Getting started::
* Initializing mu-guile::
* Messages::
* Contacts::
* Attachments and other parts::
* Statistics::
* Plotting data::

Appendices

* GNU Free Documentation License::  The license of this manual.
@end menu

@node Introduction
@chapter Introduction

@t{mu4e} is an e-mail program for @emph{GNU/Emacs}; it uses the @t{mu} maildir
search engine as its backend, making @t{mu} fully search-based.
@t{mu} is a program for indexing / searching e-mails, as well as an
@t{emacs}-based email-client (@t{mu4e}.

@t{guile} is the @emph{GNU Ubiquitous Intelligent Language for Extensions} - a
version of the @emph{Scheme} programming language and the official GNU
extension language.

@t{mu-guile} connects @t{mu} and @t{guile}, and allows you to easily write
programs to do things with your e-mails.

@node Getting started
@chapter Getting started

@menu
* Installation::
* First steps::
@end menu

This chapter walks you through the installation and some basic steps to ensure
things work correctly.

@node Installation
@section Installation

@t{mu-guile} is part of @t{mu} - by installing the latter, the former will be
installed as well. Note, however, that @t{mu-guile} requires you to have
@t{guile} version 2.0 installed, otherwise @t{mu-guile} will not be
built/installed.

At the time of writing, there are no distribution packages for @t{mu-guile}
yet, so we are assuming installation from source packages.

Installation follows the normal sequence of:
@example
$ tar xvfz mu-<version>.tar.gz  # use the specific version
$ cd mu-<version>
$./configure
@end example

The output of @t{./configure} should end with a little text describing the
detected versions of various libraries @t{mu} depends on. In particular, it
should mention the @t{guile} version, e.g.

@example
Guile version                        : 2.0.3.82-a2c66
@end example

If you don't see any line referring to @t{guile}, please install it, and run
@t{configure} again. Note once more, @t{mu-guile} requires @t{guile} version
2.0.

After a succesfull @t{./configure}, we can make and install the package:

@example
$ make && sudo make install
@end example

After this, @t{mu} and @t{mu-guile} should be installed. Note that the above
instructions will normally install things under @t{/usr/local}; you may need
to update @t{guile}'s @t{%load-path} to find it there.

You can check the current load-path with the following:

@example
guile -c '(display %load-path)(newline)'
@end example

If necessary, you can add the @t{%load-path} by adding something like the
following to your @file{~/.guile}:

@lisp
(set! %load-path (cons "/usr/local/share/guile/site/2.0" %load-path))
@end lisp

After this, you should be ready to go.

@node First steps
@section First steps

Assuming @t{mu-guile} has been installed correctly (@ref{Installation}), and
also assuming that you have already indexed your e-mail messages (if
necessary, see the @t{mu-index} man-page), we are ready to start @t{mu-guile};
a session may look something like this:

@verbatim
$ guile
GNU Guile 2.0.3.82-a2c66
Copyright (C) 1995-2011 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)>
@end verbatim

Now, we need to load the @t{mu-guile} module:

@verbatim
scheme@(guile-user)> (use-modules (mu) (mu message))
@end verbatim

This will load the basic modules for dealing with messages. After we have
loaded the modules, we need to initialize the @t{mu-guile} system:

@verbatim
scheme@(guile-user)> (mu:initialize)
@end verbatim

When this is done, we can start querying the database. We'll go into various
functions later in this manual, but just to give an example, let's get a list
of the subjects of all messages that mention @emph{hello}:

@verbatim
scheme@(guile-user)> (for-each
                        (lambda(msg)
                          (format #t "Subject: ~a\n" (subject msg)))
                        (mu:message-list "hello"))
@end verbatim

@node Initializing mu-guile
@chapter Initializing mu-guile

It is of course possible to write separate programs with @t{mu-guile}, but for
now we'll do things @emph{interactively}, i.e., from the Guile-prompt
(``@abbr{REPL}'').

We start our @t{mu-guile} session by starting @t{guile}:

@verbatim
$ guile
GNU Guile 2.0.3.82-a2c66
Copyright (C) 1995-2011 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)>
@end verbatim

Now, the first thing we need to do is load the @t{mu-guile} modules;
currently, there are three available:

@itemize
@item @code{mu} - initialization, functions to get messages, contacts
@item @code{mu message} - functions to deal with messages
@item @code{mu contact} - functions to deal with contacts
@end itemize

Let's simply load all of them:

@verbatim
scheme@(guile-user)> (use-modules (mu) (mu message) (mu contact))
@end verbatim

Assuming you have installed everything correctly, the first time you do this,
@t{guile} will probably respond by showing some message about compiling the
modules, and then end with another prompt.

Before we can do anything with @t{mu guile}, we need to initialize the
system. The reason as to not do this automatically is to enable people to use
non-default places to keep there @t{mu} data files.

We can initialize the system with:

@verbatim
scheme@(guile-user)> (mu:initialize)
@end verbatim

Which will use the default location of @file{~/.mu}. Or, instead, if you keep
your @t{mu} data in a non-standard place:

@verbatim
scheme@(guile-user)> (mu:initialize "/path/to/my/mu/")
@end verbatim

If all worked up until here, we're ready to go with @t{mu-guile}.

@node Messages
@chapter Messages

In this chapter, we discuss how to find messages, and then how to do various
things with them.

@menu
* Finding messages::
* Message functions::
* Example - the longest subject::
@end menu

@node Finding messages
@section Finding messages
Now we are ready to retrieve some messages from the system. There are two
principle functions to do this:

@itemize
@item @code{(mu:message-list [<search-expression>])}
@item @code{(mu:for-each-message <function> [<search-expression>])}
@end itemize

The first function, @code{mu:message-list} returns a list of all messages
matching @t{<search-expression>}; if you leave @t{<search-expression>} out, it
returns @emph{all} messages.

For example, to get all messages with @emph{coffee} in the subject line, you
could do:

@verbatim
scheme@(guile-user)> (mu:message-list "subject:coffee")
$1 = (#<<mu-message> 9040640> #<<mu-message> 9040630>
      #<<mu-message> 9040570>)
@end verbatim

So, we get a list with three @t{<mu-message>} objects. We'll discuss them in a
bit more detail in the next section, but let's just use the @code{subject}
function ('method') provided by @t{<mu-message>} objects to retrieve the
subject-field.

For your convenience, @t{guile} has saved the result in @t{$1}, so to get the
subject of the first message in the list, we can do:

@verbatim
scheme@(guile-user)> (subject (car $1))
$2 = "Re: best coffee ever!"
@end verbatim

The second function we mentioned, @code{mu:for-each-message}, executes some
function for each message matched by the search expression (or @emph{all}
message if the search expression is omitted).

@verbatim
scheme@(guile-user)> (mu:for-each-message
                       (lambda(msg)
                         (display (subject msg))
                         (newline))
                      "subject:coffee")
Re: best coffee ever!
best coffee ever!
Coffee beans
scheme@(guile-user)>
@end verbatim

Using @code{mu:message-list} and/or
@code{mu:for-each-message}@footnote{Implementation node:
@code{mu:message-list} is implemented in terms of @code{mu:for-each-message},
not the other way around. Due to the way @t{mu} works,
@code{mu:for-each-message} is rather more efficient than a combination for
@code{for-each} and @code{mu:message-list}} and a couple of @t{<mu-message>}
methods, together with that Guile/Scheme provides should allow for many
interesting programs.

@node Message functions
@section Message functions

Now that we've seen how to retrieve lists of message objects
(@code{<mu-message>}), let's see what we can do with such an object.

@code{<mu-message>} defines the following methods
@footnote{A note on naming: functions we have seen before --
@code{mu:initialize}, @code{mu:message-list} and @code{mu:for-each-message}
are prefixed with @t{mu:}. This is not the case for the @code{<mu-message>}
methods to be discussed next, such as the methods @code{subject} and
@code{from}. Reason for this is that it is not @emph{needed}, since these
methods only recognized for @code{<mu-message>} objects, and do not affect
anything else, while the @code{mu:}-prefixed are 'globally visible' and thus
we need to be careful about naming conflicts}
 that all only take a single
@code{<mu-message>} object as a parameter. We won't go into the exact meanings
for all of these functions here - for the details about various flags /
properties, please refer to the @t{mu-find} man-page.

@itemize
@item @code{bcc}: the @t{Bcc} field of the message, or @t{#f} if there is none
@item @code{body-html}: : the html body of the message, or @t{#f} if there is none
@item @code{body-txt}: the plain-text body of the message, or @t{#f} if there is none
@item @code{cc}: the @t{Bcc} field of the message, or @t{#f} if there is none
@item @code{date}: the @t{Date} field of the message, or 0 if there is none
@item @code{flags}: list of message-flags for this message
@item @code{from}: the @t{From} field of the message, or @t{#f} if there is none
@item @code{maildir}: the maildir this message lives in, or @t{#f} if there is none
@item @code{message-id}: the @t{Message-Id} field of the message, or @t{#f} if there is none
@item @code{path}: the file system path for this message
@item @code{priority}: the priority of this message (either @t{mu:low}, @t{mu:normal}
or @t{mu:high}
@item @code{references}: the list of messages (message-ids) this message
refers to in the @t{References:} header
@item @code{size}: size of the message in bytes
@item @code{subject}: the @t{Subject} field of the message, or @t{#f} if there is none.
@item @code{tags}: list of tags for this message
@item @code{to}: the sender of the message, or @t{#f} if there is none.
@end itemize

With these functions, we can query messages for their properties; for example:

@verbatim
scheme@(guile-user)> (define msg (car (mu:message-list "snow")))
scheme@(guile-user)> (subject msg)
$1 = "Re: Running in the snow is beautiful"
scheme@(guile-user)> (flags msg)
$2 = (mu:replied mu:seen)
scheme@(guile-user)> (strftime "%F" (localtime (date msg)))
$3 = "2011-01-15"
@end verbatim

There are a couple more functions:
@itemize
@item @code{(header <mu-message> "<header-name>")} returns an arbitrary message
header (or @t{#f} if not found) -- e.g. @code{(header msg "User-Agent")}
@item @code{(contacts <mu-message> contact-type)} which returns a list
of contacts (names/e-mail addresses in the To/From/Cc/Bcc-fields). @xref{Contacts}.
@end itemize

@node Example - the longest subject
@section Example - the longest subject

Now, let's write a little example -- let's find out what is the @emph{longest
subject} of any e-mail messages we received in the year 2011.  If you put the
following in a separate file, make it executable, and run it like any program.

@verbatim
#!/bin/sh
exec guile -s $0 $@
!#

(use-modules (mu) (mu message))
(use-modules (srfi srfi-1))

(mu:initialize)

;; note: (subject msg) => #f if there is no subject
(define list-of-subjects
  (map (lambda (msg)
      (or (subject msg) "")) (mu:message-list "date:2011..2011")))
;; see the mu-find manpage for the date syntax

(define longest-subject
  (fold (lambda (subj1 subj2)
      (if (> (string-length subj1) (string-length subj2))
	subj1 subj2))
    "" list-of-subjects))

(format #t "Longest subject: ~s" longest-subject)
(newline)
@end verbatim

There are many other ways to solve the same problem, for example by using an
iterative approach with @code{mu:for-each-message}, but it should show how one
can easily write little programs to answer specific questions about an e-mail
corpus.

@node Contacts
@chapter Contacts

We can retrieve the sender and recipients of an e-mail message using methods
like @code{from}, @code{to}, @code{cc} and @code{bcc}; @xref{Message
functions}. These functions return the list of recipients as a single string;
however, often it is more useful to deal with recipients as separate objects.

@t{mu-guile} offers some functionality for this in the @code{(mu contact)}
module.

@menu
* Contact functions and objects::
* All contacts::
* Example - mutt export::
@end menu


@node Contact functions and objects
@section Contact functions and objects

@verbatim
(use-modules (mu contact))
@end verbatim

After loading the @code{(mu contact)}, message objects (@pxref{Messages}) gain
the the @t{contacts}-methods:

    @code{(contacts <message-object> [<contact-type>])}

The @t{<contact-type>} is a symbol, one of @code{mu:to}, @code{mu:from},
@code{mu:cc} or @code{mu:bcc}; this will then get the contact objects for the
contacts of the corresponding type. If you leave out the contact-type (or
specify @t{#t} for it, you will get a list of @emph{all} contact objects for
the message.

A contact object (@code{<mu-contact>}) has two methods:
@itemize
@item @code{name} returns the name of the contact, or #f if there is none
@item @code{email} returns the e-mail address of the contact, or #f if there is none
@end itemize

Let's get a list of all names and e-mail addresses in the 'To:' field, of
messages matching 'book':

@lisp
(use-modules (mu) (mu message) (mu contact))
(mu:initialize)
(mu:for-each-message
   (lambda (msg)
      (for-each
        (lambda (contact)
           (format #t "~a => ~a\n"
             (or (email contact) "") (or (name contact) "no-name")))
        (contacts msg mu:to)))
    "book")
@end lisp

This shows what the methods do, but for many uses, it would be more useful to
have each of the contacts only show up @emph{once} - for that, please refer to
@xref{All contacts}.

@node All contacts
@section All contacts

Sometimes it may also be useful to look at @emph{all} the different contacts
in the @t{mu} database -- that is, all the different contacts. This is useful,
for example, when exporting contacts to some external format that can then be
important in an e-mail program.

To enable this, there is the function @code{mu:for-each-contact}, defined as

   @code{(mu:for-each-contact <function> [<search-expression>])}.

This will aggregate the unique contacts from @emph{all} messages matching
@t{<search-expression>} (when it is left empty, it will match all messages in
the database), and execute @t{<function>} for each of these contacts.

The @t{<function>} receives an object of the type @t{<contact-with-stats>},
which is a @emph{subclass} of the @t{<contact>} class discussed in
@xref{Contact functions and objects}. @t{<contact-with-stats>} objects expose
the following methods:

@itemize
@item @code{frequency}: returns the @emph{number of times} this contact occured in
one of the address fields
@item @code{last-seen}: returns the @emph{most recent time} the contact was
seen in one of the address fields, as a @t{time_t} value
@end itemize

The function aggregates per e-mail address; if a certain e-mail address occurs
with different names, it uses the most recent non-empty name.

@node Example - mutt export
@section Example - mutt export

Let's see how we could export the addresses in the @t{mu} database to the
addressbook format of the venerable
@t{mutt}@footnote{@url{http://www.mutt.org/}} e-mail client.

The addressbook format that @t{mutt} uses is a sequence of lines that look
something like:
@verbatim
alias <nick> [<name>] "<" <email> ">"
@end verbatim

Many of the names in our database could be random people writing things in
mailing lists, so we may want to limit it to people we have seen at least 10
times in the last year.

It is a bit hard to @emph{guess} the nick name for e-mail contacts, so we are
going to assume it is the lowercase version of the first word in
@t{<name>}. You can always adjust them later by hand, obviously.

@lisp
#!/bin/sh
exec guile -s $0 $@
!#
(use-modules (mu) (mu message) (mu contact))
(mu:initialize)

(define (selected-contacts)
  "Get a list of contacts that were seen at least 20 times since
2010."
  (let  ((addrs '())
	  (start (car (mktime (car (strptime "%F" "2010-01-01")))))
	  (minfreq 20))
    (mu:for-each-contact
      (lambda (contact)
	(if (and (email contact)
	      (>= (frequency contact) minfreq)
	      (>= (last-seen contact) start))
	  (set! addrs (cons contact addrs)))))
      addrs))

(define (guess-nick contact)
   "Guess a nick name for CONTACT."
    (string-map
      (lambda(kar)
	(if (char-alphabetic? kar) kar #\_))
      (string-downcase (or (name contact) (email contact)))))

(for-each
  (lambda (contact)
    (format #t "alias ~a ~a <~a>\n"
      (guess-nick contact) (name contact) (email contact)))
  (selected-contacts))
@end lisp

This simple program could be improved in many ways; this is left as an
excercise to the reader.

@node Attachments and other parts
@chapter Attachments and other parts

To deal with @emph{attachments}, or, more in general @emph{MIME-parts}, there
is the @t{mu part} module.

@menu
* Parts and their methods::
* Attachment example::
@end menu

@node Parts and their methods
@section Parts and their methods
The module defines the @code{<mu-part>} class, and adds two methods to
@code{<mu-message>} objects:
@itemize
@item @code{(parts msg)} - returns a list @code{<mu-part>} objects, one for
each MIME-parts in the message.
@item @code{(attachments)} - like @code{parts}, but only list those MIME-parts
that look like proper attachments.
@end itemize

A @code{<mu-part>} object exposes a few methods to get information about the
part:
@itemize
@item @code{name} - returns the file name of the mime-part, or @code{#f} if
there is none.
@item @code{mime-type} - returns the mime-type of the mime-part, or @code{#f}
if there is none.
@item @code{size} - returns the size in bytes of the mime-part
@end itemize

Then, we may want to save the part to a file; this can be done using either:
@itemize
@item @code{(save part)} - save a part to a temporary file, return the file
name@footnote{the temporary filename is a predictable function of (user-id,
msg-path, part-index)}
@item @code{(save-as part path)} - save part to file at path
@end itemize

@node Attachment example
@section Attachment example

Let's look at some small examples.

First, let's get a list of the biggest attachments in messages about
Luxemburg:

@lisp
#!/bin/sh
exec guile -s $0 $@
!#

(use-modules (mu) (mu message) (mu part))
(mu:initialize)

(define (all-attachments expr)
  "Return a list of (name . size) for all attachments in messages
matching EXPR."
  (let ((pairs '()))
    (mu:for-each-message
      (lambda (msg)
	(for-each
	  (lambda (att) ;; add (filename . size) to the list
	    (set! pairs (cons (cons (name att) (or (size att) 0)) pairs)))
	  (attachments msg)))
      expr)
    pairs))

(for-each
  (lambda (att)
    (format #t "~a: ~,1fKb\n"
      (car att) (exact->inexact (/ (cdr att) 1024))))
  (sort (all-attachments "Luxemburg")
    (lambda (att1 att2)
      (< (cdr att1) (cdr att2)))))
@end lisp

As an exercise for the reader, you might want to re-rewrite the
@code{all-attachments} in terms of @code{mu:message-list}, which would
probably be a bit more elegant.


@node Statistics
@chapter Statistics

@t{mu-guile} offers some convenience functions to determine various statistics
about the messages in the database.

First, there is @code{(mu:tabulate-messages <function> [<search-expr>])}. This
function applies @t{<function>} to each message matching @t{<search-expr>}
(leave empty to match @emph{all} messages), and returns a associative list
with each of the different results of @t{<function>} and their frequencies.

This can best be demonstrated with a little example. Suppose we want to know
how many messages we receive per weekday:

@lisp
#!/bin/sh
exec guile -s $0 $@
!#

(use-modules (mu) (mu message) (mu stats))
(mu:initialize)

(define (weekday-table)
  "Returns something like
 '((0 . 12) (5 . 20) (2 . 16) ... )
 that is, an unsorted list of (<weekday> . <frequency>)."
  (mu:tabulate-messages
    (lambda (msg)
      (tm:wday (localtime (date msg))))))

;; sort & display
(let ((table (weekday-table)))
 (for-each
  (lambda (pair)
    (let ((days '("Sun" "Mon" "Tue" "Wed" "Thu" "Fri" "Sat")))
      (format #t "~a: ~a\n"
	(list-ref days (car pair)) (cdr pair))))
  (sort (weekday-table)(lambda (a b) (< (car a) (car b))))))
@end lisp

The function @code{weekday-table} use @code{mu:tabulate-message} to get the
frequencies per hour -- this returns a list of pairs:
@verbatim
((5 . 2339) (0 . 2278) (4 . 2800) (2 . 3184) (6 . 1856) (3 . 2833) (1 . 2993))
@end verbatim

The script then output these numbers in following form:

@verbatim
Sun: 2278
Mon: 2993
Tue: 3184
Wed: 2833
Thu: 2800
Fri: 2339
Sat: 1856
@end verbatim

Clearly, Saturday is a slow day for e-mail...

@node Plotting data
@chapter Plotting data

@lisp
#!/bin/sh
exec guile -s $0 $@
!#

(use-modules (mu) (mu message) (mu contact) (mu stats) (mu plot))
(mu:initialize)

(define (mail-per-hour-table)
  (sort
    (mu:tabulate-messages
      (lambda (msg)
	(tm:hour (localtime (date msg)))))
    (lambda (x y) (< (car x) (car y)))))

(mu:plot-ascii (mail-per-hour-table) "Mail per hour" "Hour" "Frequency")
@end lisp

@verbatim
                                  Mail per hour
 Frequency
   1200 ++--+--+--+--+-+--+--+--+--+-+--+--+--+-+--+--+--+--+-+--+--+--+--++
        |+  +  +  +  + +  + "/tmp/fileHz7D2u" using 2:xticlabels(1) ********
   1100 ++                                                            *** +*
        ****                                                          * *  *
   1000 *+ *                                                       **** * +*
        *  *                           ******  ****                * ** *  *
    900 *+ *                           * ** ****  *             **** ** * +*
        *  *                           * ** *  *  *********     * ** ** *  *
    800 *+ *                        **** ** *  *  *  * ** *     * ** ** * +*
    700 *+ ***                   ****  * ** *  *  *  * ** ****  * ** ** * +*
        *  * *                ****  *  * ** *  *  *  * ** *  **** ** ** *  *
    600 *+ * ****             *  *  *  * ** *  *  *  * ** *  *  * ** ** * +*
        *  * ** *             *  *  *  * ** *  *  *  * ** *  *  * ** ** *  *
    500 *+ * ** *             *  *  *  * ** *  *  *  * ** *  *  * ** ** * +*
        *  * ** ****        ***  *  *  * ** *  *  *  * ** *  *  * ** ** *  *
    400 *+ * ** ** ****     * *  *  *  * ** *  *  *  * ** *  *  * ** ** * +*
        *+ *+**+**+* +*******+* +* +*+ *+**+* +*+ *+ *+**+* +*+ *+**+**+* +*
    300 ********************************************************************
         0  1  2  3  4 5  6  7  8  910 11 12 1314 15 16 17 1819 20 21 22 23
                                       Hour
@end verbatim



@node GNU Free Documentation License
@appendix GNU Free Documentation License

@include fdl.texi
@bye