mirror of
https://github.com/djcb/mu.git
synced 2024-07-01 08:11:06 +02:00
b75f9f508b
mu's query parser is the piece of software that turns your queries into something the Xapian database can understand. So, if you query "maildir:/inbox and subject:bla" this must be translated into a Xapian::Query object which will retrieve the sought after messages. Since mu's beginning, almost a decade ago, this parser was based on Xapian's default Xapian::QueryParser. It works okay, but wasn't really designed for the mu use-case, and had a bit of trouble with anything that's not A..Z (think: spaces, special characters, unicode etc.). Over the years, mu added quite a bit of pre-processing trickery to deal with that. Still, there were corner cases and bugs that were practically unfixable. The solution to all of this is to have a custom query processor that replaces Xapian's, and write it from the ground up to deal with the special characters etc. I wrote one, as part of my "future, post-1.0 mu" reseach project, and I have now backported it to the mu 0.9.19. From a technical perspective, this is a major cleanup, and allows us to get rid of much of the fragile preprocessing both for indexing and querying. From and end-user perspective this (hopefully) means that many of the little parsing issues are gone, and it opens the way for some new features. From an end-user perspective: - better support for special characters. - regexp search! yes, you can now search for regular expressions, e.g. subject:/h.ll?o/ will find subjects with hallo, hello, halo, philosophy, ... As you can imagine, this can be a _heavy_ operation on the database, and might take quite a bit longer than a normal query; but it can be quite useful.
141 lines
3.3 KiB
C++
141 lines
3.3 KiB
C++
/*
|
|
** Copyright (C) 2017 Dirk-Jan C. Binnema <djcb@djcbsoftware.nl>
|
|
**
|
|
** This library is free software; you can redistribute it and/or
|
|
** modify it under the terms of the GNU Lesser General Public License
|
|
** as published by the Free Software Foundation; either version 2.1
|
|
** of the License, or (at your option) any later version.
|
|
**
|
|
** This library is distributed in the hope that it will be useful,
|
|
** but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
|
** Lesser General Public License for more details.
|
|
**
|
|
** You should have received a copy of the GNU Lesser General Public
|
|
** License along with this library; if not, write to the Free
|
|
** Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA
|
|
** 02110-1301, USA.
|
|
*/
|
|
|
|
#ifndef __TOKENIZER_HH__
|
|
#define __TOKENIZER_HH__
|
|
|
|
#include <string>
|
|
#include <vector>
|
|
#include <deque>
|
|
#include <ostream>
|
|
#include <stdexcept>
|
|
|
|
// A simple tokenizer, which turns a string into a deque of tokens
|
|
//
|
|
// It recognizes '(', ')', '*' 'and', 'or', 'xor', 'not'
|
|
//
|
|
// Note that even if we recognizes those at the lexical level, they might be demoted to mere strings
|
|
// when we're creating the parse tree.
|
|
//
|
|
// Furthermore, we detect ranges ("a..b") and regexps (/../) at the parser level, since we need a
|
|
// bit more context to resolve ambiguities.
|
|
|
|
namespace Mux {
|
|
|
|
// A token
|
|
struct Token {
|
|
enum class Type {
|
|
Data, /**< e .g., banana or date:..456 */
|
|
|
|
// Brackets
|
|
Open, /**< ( */
|
|
Close, /**< ) */
|
|
|
|
// Unops
|
|
Not, /**< logical not*/
|
|
|
|
// Binops
|
|
And, /**< logical and */
|
|
Or, /**< logical not */
|
|
Xor, /**< logical xor */
|
|
|
|
Empty, /**< nothing */
|
|
};
|
|
|
|
size_t pos{}; /**< position in string */
|
|
Type type{}; /**< token type */
|
|
const std::string str{}; /**< data for this token */
|
|
|
|
/**
|
|
* operator==
|
|
*
|
|
* @param rhs right-hand side
|
|
*
|
|
* @return true if rhs is equal to this; false otherwise
|
|
*/
|
|
bool operator==(const Token& rhs) const {
|
|
return pos == rhs.pos &&
|
|
type == rhs.type &&
|
|
str == rhs.str;
|
|
}
|
|
};
|
|
|
|
/**
|
|
* operator<<
|
|
*
|
|
* @param os an output stream
|
|
* @param t a token type
|
|
*
|
|
* @return the updated output stream
|
|
*/
|
|
inline std::ostream&
|
|
operator<< (std::ostream& os, Token::Type t)
|
|
{
|
|
switch (t) {
|
|
case Token::Type::Data: os << "<data>"; break;
|
|
|
|
case Token::Type::Open: os << "<open>"; break;
|
|
case Token::Type::Close: os << "<close>";break;
|
|
|
|
case Token::Type::Not: os << "<not>"; break;
|
|
case Token::Type::And: os << "<and>"; break;
|
|
case Token::Type::Or: os << "<or>"; break;
|
|
case Token::Type::Xor: os << "<xor>"; break;
|
|
|
|
default: // can't happen, but pacify compiler
|
|
throw std::runtime_error ("<<bug>>");
|
|
}
|
|
|
|
return os;
|
|
}
|
|
|
|
/**
|
|
* operator<<
|
|
*
|
|
* @param os an output stream
|
|
* @param t a token
|
|
*
|
|
* @return the updated output stream
|
|
*/
|
|
inline std::ostream&
|
|
operator<< (std::ostream& os, const Token& t)
|
|
{
|
|
os << t.pos << ": " << t.type;
|
|
|
|
if (!t.str.empty())
|
|
os << " [" << t.str << "]";
|
|
|
|
return os;
|
|
}
|
|
|
|
/**
|
|
* Tokenize a string into a vector of tokens. The tokenization always succeeds, ie., ignoring errors
|
|
* such a missing end-".
|
|
*
|
|
* @param s a string
|
|
*
|
|
* @return a deque of tokens
|
|
*/
|
|
using Tokens = std::deque<Token>;
|
|
Tokens tokenize (const std::string& s);
|
|
|
|
} // namespace Mux
|
|
|
|
#endif /* __TOKENIZER_HH__ */
|