aeb6d44172
mu-store/indexer: consume messages from workers
...
Add store::consume_message, which is like add message but std::move from
the caller such that the messages longer has copies (with
Xapian::Document) on the caller side; this is to avoid threading issues.
2024-05-08 19:11:40 +03:00
5a48664033
lib: centralize xapian include, enable move semantics
...
Only include xapian.h in one place, so we can have consistent options.
With that in place, we can enable C++ move semantics.
We don't do anything with that yet, but we check in the meson.build file
to see if we have the required xapian version.
2024-05-01 17:02:24 +03:00
dcbcd697f4
mu-init: insist that --maildir is absolute
2024-02-26 01:08:44 +02:00
146b80113f
lib: move transaction handling to mu-xapian
...
Instead of handling transactions in the store, handle it in xapian-db.
Make the code a bit more natural / cleaner-out
Handle transaction automatically (with a batch-size) and add some RAII
Transaction object, which makes all database interaction transactable
for the duration. So, no more need for explicit parameters to
add_message while indexing.
2023-12-22 21:24:41 +02:00
b0dca49dc0
store: better handling moving messages
...
sort results (for testing)
with the ChangeName flag, also apply to dup messages.
2023-09-24 17:29:56 +03:00
2d20074b99
mu-move: add new move sub command
...
Add sub-command to move messages; add tests and docs.
Fixes #157
2023-09-24 17:29:08 +03:00
9dcbe1d96c
lib: unit tests: improve / better coverage
2023-09-13 23:02:53 +03:00
53c7381929
lib: move index/ into main lib/
...
simplify things a bit
2023-09-10 08:55:25 +03:00
89ed21e0c5
lib: improve printability for some types
...
A little fmt pixie dust
2023-09-09 17:26:20 +03:00
264bb092f0
support xapian ngrams
...
Xapian supports an "ngrams" option to help with languages/scripts
without explicit wordbreaks, such as Chinese / Japanese / Korean.
Add some plumbing for supporting this in mu as well. Experimental for
now.
2023-09-09 17:26:20 +03:00
e52030c049
store: expose maildirs() method
...
This gets the current list of maildirs by asking the scanner to do a
file-system search.
2023-08-17 22:42:25 +03:00
8caf504381
store: update "move" and some related APIs
...
Update test cases as well.
2023-08-17 22:42:25 +03:00
4c0b7db3d8
store: add 'add_document' optimization, use it
...
*Usually* we need Xapian's replace_document() API, but when we know a
document (message) is completely new, we can use the faster
add_document(). That is the case with the initial (re)indexing, when
start with an empty database.
Also a few smaller cleanups.
2023-07-25 23:56:19 +03:00
72f43f11df
lib: improve store error messages
...
Use xapian_try_result
2023-07-23 21:04:26 +03:00
18490a818d
store/server: centralize docids-for-msgid
...
No need for two near-identical impls
Remove some dead declarations.
2023-07-10 23:17:06 +03:00
3791d0c375
lib/store: rework to use xapian-db / config
...
Simplifies the implementation.
2023-07-02 00:06:56 +03:00
a3538e5ef1
store: small cleanups
...
Remove readonly/inmemory from properties. Add read_only() accessor.
2023-04-23 20:26:16 +03:00
87c3ceb7b1
store: update move_message API
...
Update the move_message API so to allow for updating duplicate messages
too (not implemented yet), and return all updated messages.
2022-12-07 18:22:29 +02:00
5367122c08
store: support reinit
...
Support reinitializing, based on some current store. This is useful for
upgrading. Note that this is only the backend implementation + tests.
2022-10-02 18:24:23 +03:00
c1f66059a8
store: fix typo
2022-06-09 00:39:34 +03:00
13f0e24241
lib: improve test coverage
...
Add a bunch of tests
2022-06-02 21:07:41 +03:00
9af9d0fa94
indexer/store: avoid completed callback
...
This cause havoc / race conditions.
2022-05-18 20:16:48 +03:00
2e9666af0b
store/info: Gather some usage statistics
...
Keep track of the latest-change/latest-index.
2022-05-10 08:16:47 +03:00
80d84bf635
store: use Result<Store> builder, add auto upgrade
...
Make it a Result type, and add auto-upgrade (not enabled yet)
Update dependents.
2022-05-09 22:25:28 +03:00
7470a5f8ea
store/fields: Use one MaxTermLength everywhere
...
One should be enough!
2022-05-09 22:25:28 +03:00
6126d7ee62
store: ensure updates update message sexp too
...
And turn all "add" into "replace" so old messages get removed.
Update tests too.
2022-05-05 01:40:17 +03:00
fea596ae3b
message/field: cache the message's sexp
...
Keep it in the store; much faster than generating on the fly.
2022-05-05 01:38:25 +03:00
525fef479a
store: update to use Message; big cleanup
...
Remove much of the message processing from the store
2022-04-30 10:40:45 +03:00
9b8353fc37
lib: update Store & QueryResults in terms of Mu::Message
2022-04-30 10:40:45 +03:00
27ebfb3b3c
lib+guile: use Mu::Option, not std::optional
...
We need the extensions, and/or let's use _one_ optional implementation everywhere.
2022-03-28 08:58:04 +03:00
8ed09a9a82
store: cosmetics
2022-03-26 17:19:10 +02:00
f7c84006d7
many: update for lib/message updates
...
Adapt to the new names / directory. Big commit, but mostly just very boring renaming.
2022-03-26 17:19:10 +02:00
056fecd6aa
store::for_each_term: use field-id
...
We were trying to convert a field (string) to a xapian prefix back to a
field (enum). That's unnecessarily complicated and worse, step 2 won't work.
2022-03-14 09:38:59 +02:00
a8654a616a
lib/store: use mu-message-fields
2022-03-07 06:50:18 +02:00
a82bd77d09
lib: rename contacts into contacts-cache
...
Plus dependents.
2022-02-21 23:22:42 +02:00
831d26052a
store: expose the mutex so we can lock query-results
...
The QueryResults must not outlive the lock (when in a MT context), so expose for
clients (mu-server) to handle it.
2022-02-17 23:49:17 +02:00
3086238b33
store: expose metadata()/set_metadata()
...
Allow for storing metadata in the database, and consider the cache.
2022-02-14 11:12:26 +02:00
3820118246
store: rename "metadata" into "properties"
...
properties are the constant (for the duration) values for a store; metadata may
change, so reserve that name for that.
2022-02-14 11:12:26 +02:00
05393ba797
index: save/commit metadata after messages
...
Ensure the metadata (dirstamps) for messages are only written / committed _after_
the accompanying message have been written / committed.
This avoids missing updates when indexing gets terminated unexpectedly.
2022-02-03 23:04:18 +02:00
5fc8a8f83e
store/query: access query only through store
...
Make Mu::Query only accessible through store, so we can lock the db for the
duration of a (full, multipass) query.
2022-01-30 14:52:41 +02:00
4c0d8572d8
store/indexer: move transaction handling to store
...
Move the transaction handling code inside Store, simplifying the indexer.
2021-11-09 22:43:11 +02:00
8028f88a51
mu: rework xapian dependencies a bit
...
Reduce the places where we need to include Xapian, and add a few places where
the (meson build) didn't explicitly have a Xapian dep where needed.
2021-11-03 12:34:08 +02:00
89014ecd06
index/store: simplify transaction handling
...
only have transactions for indexing, and make it opportunistic. All other ops do
not use transactions.
2021-10-20 20:14:25 +03:00
3dd721d5a3
clang-format: update c/cc coding style
...
Update all cc code using .clang-format; please do so as well for future PRs
etc.; emacs has a handy 'clang-format' mode to make this automatic.
For comparing old changes with git blame, we can disregard this one using
--ignore-rev
(see https://www.moxio.com/blog/43/ignoring-bulk-change-commits-with-git-blame )
2021-10-20 12:26:16 +03:00
98abcf8e84
store: support in-memory database
...
For testing, this is faster / and doesn't pollute the file system.
2021-02-16 19:32:15 +02:00
a3c6e74869
lib: replace some #define with constexpr
...
minor cleanup.
2021-02-11 18:54:25 +02:00
98744c66f7
remove some dead code
2021-01-22 23:24:53 +02:00
e282d80bc0
lib: Update for new querying machinery
...
Port to c++ and use new APIs.
2021-01-22 22:51:33 +02:00
ed4a640c39
lib/query: refactor & rework
...
- Move the lib/query/ stuff up a level into lib/
- Associate directly with the Query object
- Rework the Query object to be C++ rather than mixed with C
- Update all dependencies, tests
2020-11-04 23:35:30 +02:00
dbff5671dd
lib: support 'personal' regexp, move to mu-contacts
...
Move the determination of "personal" to MuContacts; add support for
regexps (POSIX-basic, in //)
2020-10-17 10:36:25 +03:00
Dirk-Jan C. Binnema