2020-01-01 14:44:39 +01:00
|
|
|
## Copyright (C) 2017-2020 Dirk-Jan C. Binnema <djcb@djcbsoftware.nl>
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
##
|
|
|
|
|
## This program is free software; you can redistribute it and/or modify
|
|
|
|
|
## it under the terms of the GNU General Public License as published by
|
|
|
|
|
## the Free Software Foundation; either version 3 of the License, or
|
|
|
|
|
## (at your option) any later version.
|
|
|
|
|
##
|
|
|
|
|
## This program is distributed in the hope that it will be useful,
|
|
|
|
|
## but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
## GNU General Public License for more details.
|
|
|
|
|
##
|
|
|
|
|
## You should have received a copy of the GNU General Public License
|
|
|
|
|
## along with this program; if not, write to the Free Software Foundation,
|
|
|
|
|
## Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
|
|
|
|
|
|
|
|
|
include $(top_srcdir)/gtest.mk
|
|
|
|
|
|
|
|
|
|
@VALGRIND_CHECK_RULES@
|
|
|
|
|
|
2017-10-28 13:12:50 +02:00
|
|
|
AM_CXXFLAGS= \
|
|
|
|
|
-I$(srcdir)/.. \
|
|
|
|
|
-I$(top_srcdir)/lib \
|
|
|
|
|
$(GLIB_CFLAGS) \
|
|
|
|
|
$(XAPIAN_CXXFLAGS) \
|
|
|
|
|
$(WARN_CXXFLAGS) \
|
2018-11-03 14:47:39 +01:00
|
|
|
$(ASAN_CXXFLAGS) \
|
2020-01-01 14:44:39 +01:00
|
|
|
$(CODE_COVERAGE_CFLAGS) \
|
2017-10-28 13:12:50 +02:00
|
|
|
-Wno-inline \
|
|
|
|
|
-Wno-switch-enum
|
|
|
|
|
|
|
|
|
|
AM_CPPFLAGS= \
|
2020-01-01 14:44:39 +01:00
|
|
|
$(CODE_COVERAGE_CPPFLAGS)
|
2017-10-28 13:12:50 +02:00
|
|
|
|
|
|
|
|
AM_LDFLAGS= \
|
2018-11-03 14:47:39 +01:00
|
|
|
$(ASAN_LDFLAGS) \
|
2020-01-01 14:44:39 +01:00
|
|
|
$(WARN_LDFLAGS)
|
2017-10-28 13:12:50 +02:00
|
|
|
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
noinst_PROGRAMS= \
|
|
|
|
|
tokenize \
|
|
|
|
|
parse
|
|
|
|
|
|
|
|
|
|
noinst_LTLIBRARIES= \
|
2019-12-16 21:41:17 +01:00
|
|
|
libmu-parser.la
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
|
2019-12-16 21:41:17 +01:00
|
|
|
libmu_parser_la_SOURCES= \
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
data.hh \
|
|
|
|
|
parser.cc \
|
|
|
|
|
parser.hh \
|
|
|
|
|
proc-iface.hh \
|
|
|
|
|
tokenizer.cc \
|
|
|
|
|
tokenizer.hh \
|
|
|
|
|
tree.hh \
|
|
|
|
|
xapian.cc \
|
|
|
|
|
xapian.hh
|
|
|
|
|
|
2019-12-16 21:41:17 +01:00
|
|
|
libmu_parser_la_LIBADD= \
|
2017-10-28 13:12:50 +02:00
|
|
|
$(WARN_LDFLAGS) \
|
|
|
|
|
$(GLIB_LIBS) \
|
2020-01-01 14:44:39 +01:00
|
|
|
$(XAPIAN_LIBS) \
|
|
|
|
|
../utils/libmu-utils.la \
|
|
|
|
|
$(CODE_COVERAGE_LIBS)
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
|
2019-12-16 21:41:17 +01:00
|
|
|
VALGRIND_SUPPRESSIONS_FILES= \
|
|
|
|
|
${top_srcdir}/mu.supp
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
|
2020-01-01 14:44:39 +01:00
|
|
|
tokenize_SOURCES= \
|
|
|
|
|
tokenize.cc
|
|
|
|
|
|
|
|
|
|
tokenize_LDADD= \
|
|
|
|
|
$(WARN_LDFLAGS) \
|
|
|
|
|
libmu-parser.la
|
|
|
|
|
|
|
|
|
|
parse_SOURCES= \
|
|
|
|
|
parse.cc
|
|
|
|
|
|
|
|
|
|
parse_LDADD= \
|
|
|
|
|
$(WARN_LDFLAGS) \
|
|
|
|
|
libmu-parser.la
|
|
|
|
|
|
2019-12-16 21:41:17 +01:00
|
|
|
noinst_PROGRAMS+=$(TEST_PROGS)
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
|
2019-12-16 21:41:17 +01:00
|
|
|
TEST_PROGS+= \
|
|
|
|
|
test-tokenizer
|
|
|
|
|
test_tokenizer_SOURCES= \
|
|
|
|
|
test-tokenizer.cc
|
|
|
|
|
test_tokenizer_LDADD= \
|
2020-01-01 14:44:39 +01:00
|
|
|
libmu-parser.la
|
2019-12-16 21:41:17 +01:00
|
|
|
|
|
|
|
|
TEST_PROGS+= \
|
|
|
|
|
test-parser
|
|
|
|
|
test_parser_SOURCES= \
|
|
|
|
|
test-parser.cc
|
|
|
|
|
test_parser_LDADD= \
|
2020-01-01 14:44:39 +01:00
|
|
|
libmu-parser.la
|
lib: implement new query parser
mu's query parser is the piece of software that turns your queries
into something the Xapian database can understand. So, if you query
"maildir:/inbox and subject:bla" this must be translated into a
Xapian::Query object which will retrieve the sought after messages.
Since mu's beginning, almost a decade ago, this parser was based on
Xapian's default Xapian::QueryParser. It works okay, but wasn't really
designed for the mu use-case, and had a bit of trouble with anything
that's not A..Z (think: spaces, special characters, unicode etc.).
Over the years, mu added quite a bit of pre-processing trickery to
deal with that. Still, there were corner cases and bugs that were
practically unfixable.
The solution to all of this is to have a custom query processor that
replaces Xapian's, and write it from the ground up to deal with the
special characters etc. I wrote one, as part of my "future, post-1.0
mu" reseach project, and I have now backported it to the mu 0.9.19.
From a technical perspective, this is a major cleanup, and allows us
to get rid of much of the fragile preprocessing both for indexing and
querying. From and end-user perspective this (hopefully) means that
many of the little parsing issues are gone, and it opens the way for
some new features.
From an end-user perspective:
- better support for special characters.
- regexp search! yes, you can now search for regular expressions, e.g.
subject:/h.ll?o/
will find subjects with hallo, hello, halo, philosophy, ...
As you can imagine, this can be a _heavy_ operation on the database,
and might take quite a bit longer than a normal query; but it can be
quite useful.
2017-10-24 21:55:35 +02:00
|
|
|
|
|
|
|
|
TESTS=$(TEST_PROGS)
|
2020-01-01 14:44:39 +01:00
|
|
|
|
|
|
|
|
include $(top_srcdir)/aminclude_static.am
|