The default was 250000 but that led to problems on some systems with
limited memory, esp. since mu's indexing does quite a bit more than in
the olden days (e.g. html mail).
Fixes#2529.
When passing messages to mu, often we got a (parsed from string)
message-sexp from the message document; then appended some more
properties ("build_message_sexp").
Instead, we can do it in terms of the strings; this is _a little_
inelegant, but also much faster; compare:
(base)
[mu4e] Found 500 matching messages; 0 hidden; search: 1298.0 ms (2.60 ms/msg); render: 642.1 ms (1.28 ms/msg)
(with temp-file optimization (earlier commit)
[mu4e] Found 500 matching messages; 0 hidden; search: 1152.7 ms (2.31 ms/msg); render: 270.1 ms (0.54 ms/msg)
(with temp file optimize _and_ the string opt (this commit)
[mu4e] Found 500 matching messages; 0 hidden; search: 266.0 ms (0.53 ms/msg); render: 199.7 ms (0.40 ms/msg)
This makes queries where we don't need the sexp much faster; e.g.
before:
mu find "a" --include-related 47,51s user 2,68s system 99% cpu 50,651 total
after:
mu find "a" --include-related 7,12s user 1,97s system 87% cpu 10,363 total
Add a new configuration variable mu4e-mu-allow-temp-file, t by default,
which triggers the same optimization in the mu server.
It improves message rendering speed:
I.e. without:
[mu4e] Found 500 matching messages; 0 hidden; search: 1298.0 ms (2.60 ms/msg); render: 642.1 ms (1.28 ms/msg)
with:
[mu4e] Found 500 matching messages; 0 hidden; search: 1152.7 ms (2.31 ms/msg); render: 270.1 ms (0.54 ms/msg)
As a useful side-effect, this also:
Fixes#1802.
It can be faster to feed big mu -> mu4e data, such as contacts are
message headers through a temp-file instead directly though stdout;
implement this, and add the server parameter --allow-temp-file.
Implement this the "contacts" and "find" commands.
For expanding shell options (with expand_path / wordexp)
Note that e.g. in zsh: --maildir=~/Maildir is handled (program receives
--maildir=/home/user/Maildir) but e.g. bash does not do that, and the
program receives the literal '~/Maildir'
We expanded this in mu earlier, so let's do that again.
*Usually* we need Xapian's replace_document() API, but when we know a
document (message) is completely new, we can use the faster
add_document(). That is the case with the initial (re)indexing, when
start with an empty database.
Also a few smaller cleanups.
On rotational devices (HDD) processing direntries is much faster when
doing so sorted by i-node for the dir-entries. This is an old
optimization (perhaps mu <= 1.6 or so?) that was implemented yet after
indexing changed, likely because my systems use SDDs instead!
But, let's restore that optimization; the sorting is fast enough that we
don't care for SDDs; on HDD it should be quite a bit faster.
We were dumping the HTML-parts as-is in the Xapian indexer; however,
it's better to remove the html decoration first, and just pass the text.
We use the new built-in html->text scraper for that.
Dirk-Jan C. Binnema