We got many reports where the 'lazy check' didn't work too well for
people... so make it a bit less lazy, so it'll just work for more
people.
In practice, never skip _directories_ unless they're leaf directories;
this avoids the mtime-does-not-bubble-up problem.
Not only check for duplicate subjects in *siblings*, also recurse into
the children. This remove some clutter from deeply nested threads.
Fixes: #2078.
When marking a message a read, do the same for the duplicates; this
was the old behavior and the intention of the new behavior but didn't
quite work.
Fixes: #2071.
Some #includes were missing for the latter (but only noticeable on some
systems - e.g., build breaks on Cygwin).
So let's replace with something that works equally everywhere.
Fixes: #2060
The scanner had a timeout for scanning, which doesn't work too well with
machine with rel. fast disks / rel. slow machines. Which I don't happen
to have!
Let's remove the timeout, should help with getting unwanted timeouts
which would cut short the indexing.
mu-query.cc:
- make_related_enquire: don't include first query in qvec, we already have all
thread IDs we need to query in thread_ds.
- run_related: always sort first query by date, explained by the comment.
- run_related: include qflags (in particular ascending vs descending) in
leader_qflags.
- run_theaded: don't limit results to maxnum, that results in threads
potentially being cut off.
mu-server.cc:
- output_sexp: don't limit results to maxnum so as to match the behaviour of
mu find (and avoid cuttong off threads).
Fixes#1924 and #1911.
Calculate the thread subject, that is, the subject of the (sub)thread _or_
empty if it's the same as the previous subject.
This is for the UI feature of _not_ showing the subject when it's just
repeating from the previous.
For threading, we still get the _full_ set of messages (since the mset is
limited, but not the enquire); so no need to warn about docids we
haven't seen before.
Also, ensure the unwanted docids are sorted after the wanted ones.
Fixes: #1926.
Rewrite the query machinery in c++:
- use an MSet decorator instead of the mu-msg-iter stuff
- use mu-query-decider to mark duplicates/unreadable/related messages
- use mu-query-threader to replace the older container/thread code
Algorithm did not substantially change, but the implementation details
did.
Add some Rust-style Result/Option types, based on TartanLlama's
expected, optional classes.
There's std::optional of course, but we can't depend on C++17 yet.
- Move the lib/query/ stuff up a level into lib/
- Associate directly with the Query object
- Rework the Query object to be C++ rather than mixed with C
- Update all dependencies, tests
Using deque gives compilation errors when compiling on
MacOS/clang (where it defaults to libc++ rather than gcc's libstdc++)
```
#include <deque>
struct Foo { std::deque<Foo> foos; };
int main() { Foo foo; }
```
So, let's use a vector instead; this is a drop-in replacement here, but
unfortunately in some future code...
Seems there are problems compiling mu with XCode 11.6 (see build tests);
apparently because of libc++ being different from libstdc++.
clang++ builds works fine as long as we're using libstdc++.
If the user has wants to postpone clean-up we shouldn't lock the
indexer waiting for something that will never happen. Clear the flag
event though we are actually skipping cleanup.
When this function is declared const or pure, clang at -O1 or higher optimizes
away the call to mu_str_size_s() inside mu_str_size(), so that it ignores its
argument and returns whatever is in mu_str_size_s()'s static buffer.
Found when test-mu-str failed while testing an update of mu in OpenBSD's ports tree.
Implement a new message indexer consisting of a single-threaded scanner
and a multi-threaded indexer.
This allows for a number of optimizations as well as background
indexing, though this initial version should be behave similar to the
old indexer.
reimplement the old mu-log.[ch] into mu-logging.{cc,hh}
If available (and using an appropriately equipped glib), log to the
systemd journal
Only g_criticals have stderr output, all the other g_* go to the log
file / journal.
For the new symlink-support, it's better to use the *canonical* path than
the *realpath(3)* for files, so removing a symlinked maildir will work as
expected.
Until now, mu would _not_ follow symlinks; with these changes, we do.
There were some complications with that ~10 years ago, but I forgot the
details. So let's re-enable. At least one thing is in place now: moving
between file systems.
Fixes#1489Fixes#1628 (technically, this came with slightly earlier commit)
When calling mu_maildir_move_message with the new_name
options (workaround for mbsync's), do the src=target check *without* first
creating that new name.
This avoids some unnecessary moves.
Isync uses this by default on Windows where ':' is an invalid character
in file names. Also try to preserve the existing separator character
when generating a new file name.
We were verifying signatures when this was not needed; it seems that
gpgme is a bit slow (?), and on some people's machine the extra
verification made opening messages slow (with the non-gnus view esp.)
Today when we query a find cmd with the `--threads` option, all the
childs of each thread are sorted according to their leader based on
the sortfield.
This patch change the way of how the childs of a thread are sorted.
The threads are still sorted according to their leader but all the
childs of each thread are now sorted based on the sortfield only.
Here is an example of what happened with the previous sorting:
Example with random kernel thread sorted by date:
[PATCH 0/4] drm/panel: jh057n0090: Add regulators and drop magic value in init
┣━▶[PATCH 1/4] MAINTAINERS: Add Purism mail alias as reviewer for their devkit's panel
┣━▶[PATCH 2/4] drm/panel: jh057n0090: Don't use magic constant
┣━▶[PATCH 3/4] dt-bindings: display/panel: jh057n0090: Document power supply properties
┗━▶[PATCH 4/4] drm/panel: jh057n0090: Add regulator support
If someone reply to one of these emails in the middle, this email
become the leader and the thread is displayed like this:
[PATCH 0/4] drm/panel: jh057n0090: Add regulators and drop magic value in init
┣━▶[PATCH 2/4] drm/panel: jh057n0090: Don't use magic constant
┃ ┗━▶ Re: [PATCH 2/4] drm/panel: jh057n0090: Don't use magic constant
┣━▶[PATCH 1/4] MAINTAINERS: Add Purism mail alias as reviewer for their devkit's panel
┣━▶[PATCH 3/4] dt-bindings: display/panel: jh057n0090: Document power supply properties
┗━▶[PATCH 4/4] drm/panel: jh057n0090: Add regulator support
With this patch, we will have the following output:
[PATCH 0/4] drm/panel: jh057n0090: Add regulators and drop magic value in init
┣━▶[PATCH 1/4] MAINTAINERS: Add Purism mail alias as reviewer for their devkit's panel
┣━▶[PATCH 2/4] drm/panel: jh057n0090: Don't use magic constant
┃ ┗━▶ Re: [PATCH 2/4] drm/panel: jh057n0090: Don't use magic constant
┣━▶[PATCH 3/4] dt-bindings: display/panel: jh057n0090: Document power supply properties
┗━▶[PATCH 4/4] drm/panel: jh057n0090: Add regulator support
The tests cases concerning threads have also been updated.
Signed-off-by: Julien Masson <massonju.eseo@gmail.com>
* mu-store.h, mu-store-read.cc, mu-store-write.cc, mu-store-priv.hh have been reworked
in mu-store.{cc,hh}, it the mix of c/c++ improved
* update all the dependent modules
* make it easier to upgrade an database in place (without user intervention)
* remove the xbatch-size option
Instead of using ~/.mu, use the XDG Base Directory Specification, typically:
~/.cache/xapian
~/.cache/mu.log
~/.cache/parts
~/.config/bookmarks
Update dependencies, documentation.
Seems gmime passes them on; and it causes havoc with our contacts cache.
Bump database schema version to force an rebuild (since that's what's
required.)
Rewrite the contacts-cache backend in c++
Store the contacts as metadata in the xapian database, rather than in a
separate file.
Update the Store to deal with this.
Consider all 'inline' text parts attachments too, unless they're
'text/plain' or something that looks like a signature.
It's a heuristic so we might get some new corner-cases.. let's see.
Some mailing lists do _not_ set reply-to, see e.g.,
https://github.com/djcb/mu/pull/1278
In that case, use the 'List-Post' address instead, so the behavior is
the same (in mu4e) as for other mailing lists.
We got some errors when some of the key values exceeded the Xapian
maximum; in particular the message-id.
So make all the key-methods check, and truncate the message-id if
necessary.
The current threading algorithm is applied to the entire result of a query, even
if maxnum is specified, and then the result of the threading algorithm is
truncated to maxnum. The improves threading results by returning the entire
thread even when only a single message makes it into the top maxnum results.
This commit applies the threading algorithm to the related message set of the
maxnum-truncated query result instead of to the entire query result. For a given
set of messages, the set of messages which will share threads with any of the
original messages is exactly the related message sets. Put another way, either
any messages returned by the original query but removed by the maxnum truncation
will also be returned by the related message query, or they would not have been
needed anyway because they would not be members of any visible thread.
To maintain backward compatibility and allow threading to be used without
including related messages, the related message set is found for the threading
calculation, but any messages which would not have matched the original query
are then pruned, resulting in a superset of the truncated query, but a subset of
the untruncated query.
This does not improve (or degrade) the run time of a threading calculation when
maxnum is not set, but significant improves it when maxnum is set by making it
scale (roughly) linearly in terms of maxnum. On a maildir with ~200k messages
and maxnum set to 500 (the default), the run time of a threading calculation is
lowered from ~1m to ~0.1s.
Perform threading calculation on related set instead of entire result.
The current threading algorithm is applied to the entire result of a query, even
if maxnum is specified, and then the result of the threading algorithm is
truncated to maxnum. The improves threading results by returning the entire
thread even when only a single message makes it into the top maxnum results.
This commit applies the threading algorithm to the related message set of the
maxnum-truncated query result instead of to the entire query result. For a given
set of messages, the set of messages which will share threads with any of the
original messages is exactly the related message sets. Put another way, either
any messages returned by the original query but removed by the maxnum truncation
will also be returned by the related message query, or they would not have been
needed anyway because they would not be members of any visible thread.
To maintain backward compatibility and allow threading to be used without
including related messages, the related message set is found for the threading
calculation, but any messages which would not have matched the original query
are then pruned, resulting in a superset of the truncated query, but a subset of
the untruncated query.
This does not improve (or degrade) the run time of a threading calculation when
maxnum is not set, but significant improves it when maxnum is set by making it
scale (roughly) linearly in terms of maxnum. On a maildir with ~200k messages
and maxnum set to 500 (the default), the run time of a threading calculation is
lowered from ~1m to ~0.1s.
We were transforming wild-card searches into regular-expression
searches; while that works, it's also significantly slower.
So, instead, special-case wildcards, and use the Xapian machinery for
wildcard queries.
In the olden days, we stored dates like e.g. 20180131121234, and do a
lexicographical check. With that, we could use e.g. upper-limits
201802312359 for "all dates in Feb 2018", even if Feb doesn't have 31
days.
However, nowadays we use time_t values, and g_date_time_new_local raises
errors for non-existent days; easiest fix is to massage things a bit; so
let's do that.
Fixes issue #1197.
For now, don't treat "and not" specially; this gets us back into a
somewhat working state. At some point, we probably _do_ want to
special-case and_not though (since Xapian supports it).
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
When you have multiple mu home directories, e.g. for the use case
detailed in my "Changing mu4e-{maildir,mu-home} from a context hook"
post to the mailing list it's quite inconvenient to have to hammer out
"mu --muhome=.. find .." every time you want to run some ad-hoc
command.
This allows me to set up a screen session where I do searches in mu
directory A in some screen panes, and searches in directory B in
others.
I initially called this MU_MUHOME but then I noticed that the perl
plugin has MUP_MU_HOME for analogous functionality, so I'm just
following its example.
The code I'm adding in mu-util.c is just a copy/paste & adjustment of
the same sort of already tested functionality in
mu_util_guess_maildir() just a few lines earlier.
If not, when the session of mu is killed, these child processes are also
killed. This scenario shows up when using mu4e: a PDF attachment, for
example, is opened by Evince, but as soon as Emacs exits, Evince is also
killed.
clear_links as used for the --clear-links option had some broken
filename generation, causing garbage data at the end.
Clean up this old code, and fix this problem as a side-effect.
Fixes issue #951.
mu_util_fputs_encode was aborting on behalf of the stack-guard on
OpenBSD (seemingly only when compile with optimization). It appears as
if the root cause of this was a differences in sizes of the parameters
to g_locale_from_utf8. Fix this.
The callbacks for the contacts functions should return TRUE (or be
terminated early), but were void. Seems on Linux this usually still
worked, not so on OpenBSD at least (unit test broke). So, fix this.
Can't say I fully understand what's going on, but it seems gpg-before-2
has some trouble with its agent, at least when using
gnome-session (which stopped using gnome-keyring as a gpg-agent since
Fedora 23 at least).
Sanity seems to be restored when preferring gpg2 instead. "gpg" is used
when gpg2 isn't there; and there's the MU_GPG_PATH env variable to
override all of that.
Add an option --lazy-check to ignore any directories that don't have
their ctime changed since the last indexing operation.
There are a few corner-cases (such as editing a message outside mu's
control) where this might miss a change, but apart from that, makes
indexing in for a maildir (and its sub-maildirs) almost a no-op if there
were no changes.
Improve the function ``cleanup_filename()`` of ``lib/mu-msg-part.c`` to
use Unicode characters when replacing the control characters, slashes
and colons with ``-``.
Originally, this function just use plain C characters (i.e., assuming
ASCII string) when checking each character is or not a control character,
slash or colon. However, when the attachment filename contains non-ASCII
(e.g., Chinese characters), all the non-ASCII characters are replaced
with ``-``.
For example:
* Before:
```
> mu view test_chinese_attachment_filename.eml
From: Tester <tester@example.com>
To: Example <example@example.com>
Subject: Test email with attachment of Chinese filename
Date: Mon 23 May 2016 05:22:09 PM CST
Attachments: 'attachment-test.txt', '------------.txt', '-------test.txt'
Hello,
This is a simple test email with three attachments:
1. `attachment:test.txt`: filename is all English;
2. `测试附件.txt`: filename is all Chinese (exclude the extension);
3. `附件-test.txt`: filename mixes Chinese and English.
```
* After:
```
> ./build/mu/mu/mu view test_chinese_attachment_filename.eml
From: Tester <tester@example.com>
To: Example <example@example.com>
Subject: Test email with attachment of Chinese filename
Date: Mon 23 May 2016 05:22:09 PM CST
Attachments: 'attachment-test.txt', '测试附件.txt', '附件-test.txt'
Hello,
This is a simple test email with three attachments:
1. `attachment:test.txt`: filename is all English;
2. `测试附件.txt`: filename is all Chinese (exclude the extension);
3. `附件-test.txt`: filename mixes Chinese and English.
```
Add a user-agent property to the full message sexps (i.e., the ones
available in mu4e-view). This property contains either the User-Agent or
X-Mailer string (and is absent otherwise)
Seems people are getting really big mails these days, so let's up the
default (which is also what mu4e uses) to 500 Mb (which should be enough
for everyone, always)
mu: cleanup server side; make sure not to loose 'personal' flag when
seeing same contact in non-personal context
mu4e: tweak the sorting algorithm a bit to take the personal flag into
account
Doing:
!access(...) == 0
Is equivalent to:
(!access(...)) == 0
Not:
!(access(...) == 0)
And throws this warning under clang:
mu-store.cc:77:6: warning: logical not is only applied to the left hand
side of this comparison [-Wlogical-not-parentheses]
if (!access(xpath, F_OK) == 0) {
^ ~~
mu-store.cc:77:6: note: add parentheses after the '!' to evaluate the
comparison first
if (!access(xpath, F_OK) == 0) {
^
( )
mu-store.cc:77:6: note: add parentheses around left hand side expression
to silence this warning
if (!access(xpath, F_OK) == 0) {
^
( )
It ends up doing what the author intended anyway since access() returns
-1 on error, and !-1 == 0, but just do the more obvious check and check
that we don't get 0 here with !=.
Some users were report seeing get_uid_term high in the profiles; so
optimize this:
- make mu_util_get_hash a static inline function (used by get_uid_term)
- don't use 'realpath' in get_uid_term, seem that's the main culprit
- some slight faster string handling there too.
It seems some tools try to interpret the filename of message files,
even though they shouldn't:
"Do not try to extract information from unique names."
In particular, they seem to interpret the first part of the name (before
the first dot) as the # of seconds since the Unix epoch (ie.,
time(NULL)). That's not what mu/mu4e put there.
So, let's conform a bit more to the expected filename (as per the
maildir spec), so we're not confusing those tools.
The core dump only seems to occur if mu4e-headers-include-related is
set to t.
Apparently, std::string's c_str() method is confusing to many
people, c.f.
http://stackoverflow.com/questions/22330250/how-to-return-a-stdstring-c-str
The answer seems to be that the pointer c_str() returns may not be
valid past the current statement; returning it, or even using it
subsequently can have you sending a wild pointer into e.g. g_strdup().
In short, it seems idioms like this are okay:
return g_strcmp0 (s1.c_str(), s2.c_str()) < 0;
Whereas idioms like this are not:
const char *msgid (iter->msgid().c_str());
return msgid ? g_strdup (msgid) : NULL;
At least in my environment by the time we get to g_strdup() the
pointer returned by c_str() is wild and points at garbage. Since
g_strdup() returns NULL if passed NULL, it seems collapsing it into a
single line is not only possible but necessary.
I've looked at all of the calls to c_str() in mu and it appears to
me this was the one remaining one that was bad.
The test fails in some cases with interesting directory setups, although
the function does work. So de-activate the test for now, until we come
up with a better one.
Since `parent` is not really used as a parent, I use it as the last
visited encrypted part while going down the parts-tree.
At the decryption of a part (`mu_msg_crypto_decrypt_part`) I check,
through the GMimeDecryptResult, for signatures (`check_decrypt_result`)
and add them to the part (`tag_with_sig_status`). Any nested parts hold
that encrypted part as their parent. Finally at `handle_part`, for each
part I check if it a descendent of an encrypted part. If so, I proceed
checking for signatures and adding them to the `msgpart`.
This reverts commit 6e9b9ad2d0.
Unfortunately the reverted commit breaks the Signature field for
encrypted and, at the same time, signed messages.
TODO: details button in the Signatures field does not work for such
cases because the signature is encrypted.
Conflicts:
lib/mu-msg-part.c
Add a decryption field of the form
Decryption: 2 part(s) decrypted 1 part(s) failed
Meaning that 2 encrypted mime parts where successfully decrypted and 1
part failed. Note that the number 2 refers to the number of
successfully decrypted mime parts and not the number of successfully
decrypted encryptes multiparts, i.e., if an encrypted multipart
contains 4 parts and decryption is successful the field will be
Decryption: 4 part(s) decrypted
TODO: Add details button listing the names and indexes of the
decrypted (or not) mime-parts
