Create initializer function that puts default values to all fields
of message list item. Fix all code that directly assigns some hash
to the elements of messagelist: for direct assignments only initializer
is now permitted, all other modification are done in-place.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.
It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.
We also introduce GmailMaildir repository that adds functionality to
change message labels. It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Make external API of class/module to be smaller, explicitely mark
all internal functions. Also annotate methods that are implemented
as the part of the parent class interface.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
There is no need to cast 0 to 'long' even if we want to compare it to long
numbers in modern pythons.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.
1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
from the imaplib2 result stack, so subsequent calls would return "None".
As various functions can invoke this, this led to some errors that we
avoid by caching the current UIDVALIDITY value in the Folder instance.
There are more simplifications and improvements to be made.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Do not read in custom maildir flags, or we would try to sync them over
the wire. The next step will be to merge flag writes with existing
custom flags, so we don't lose information.
The long term goal will be to attempt to sync flags to the other side,
of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than always parsing the filename, we only need to do so if the flags
have actually changed, otherwise we can keep the filename.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, assigning a new UID to a mapped IMAP or Maildir repository
was done by loading the "local" item, saving it under a new UID and
deleting the old one. This involved lots of disk activity for nothing
more than an effective file rename in Maildirs, and lots of network
usage in the MappedUID cases.
We do this on every upload from a local to a remote item, so that can
potentially be quite expensive. This patch lets backends that support it
(Maildir, MappedUID) efficiently rename the file rather than having to
read the mail content, write it out as a new file and delete the old
file. This speeds up uploads from Maildir and the MappedUID server.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Various functions (such as change_message_uid) will want to construct
maildir filenames, so factor out the code into a helper.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Create a helper function that retrieves the UID, folder MD5, and Flags from
a message filename.
We need these items when we simply want to rename (=new UID) a Maildir
message file later. The new function can give us these components.
Rework, so we cache the calculation of the folder's md5 value once, it
never changes and we call it a lot.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If someone had a custom :2,a flag, adding a new flag would lead to the
invalid maildir filename ...a:2,... due to regex deficiencies not coping
with this. Fix this so we alway produce valid maildir names.
Note that custom flags are still problematic: as the syncing to the
remote IMAP server will fail, the next sync will assume that they have
been removed from the remote IMAP side and they will be removed from the
local Maildir then. We will need to think about how to handle this. At
least, with this patch we won't lose standard flags and won't produce
invalid maildir names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The regex for catching Maildir message flags was
self.infosep + '.*2,([A-Z]+)' (infosep being ':').
The .* is bogus, as there is nothing between the : and the 2, per
maildir name specification, so remove that unneeded piece.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The Message UID is already the key to self.messagelist, so we have that
information. It is redundant to save the UID again as
self.messagelist[uid]{'uid': uid} and we never made use of the
information anyway.
The same thing should be done with the other 2 backends.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A more pythonic and less verbose way to do the same. Add a comment what the
variable is all about.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
open() and os.open() lead to different file permissions by default, and
while we have not changed the os.open that had been used, some code
changes led to these permissions slipping through. Fix this by setting
the permissions explicitly to 0666 (minus the users umask).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It is possible to get the config parameter from the Repository() which is
set in BaseFolder, so we set self.config there and remove the various
methods and 'config' parameters that are superfluous.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We passed in the accountname to all derivatives of BaseFolder, such as
IMAPFolder(...,repository,...,accountname), although it is perfectly
possible to get the accountname from the Repository(). So remove this
unneeded parameter. Each backend had to define getaccountname() (although
the function is hardly used and most accessed .accountname directly).
On the other hand BaseFolder was using getaccountname but it never defined
the function. So make the sane thing, remove all definitions from backends
and define accountname() once in Basefolder. It was made a property and not
just a (public) attribute, so it will show up in our developer
documentation as public API.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As all Folders share these parameters, we can safely handle them in
BaseFolder. This makes sense, as BaseFolder has a getname() function
that returns self.name but nothing actually set self.name.
It also saves a few lines of code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit e023f190b0 changed the storing of
file paths in the messagelist variable to be relative paths, but we were
using the full absolute path anyway as we missed one spot.
Adapt this and construct the full file path in the one place where we
need it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:
for flag in newflags:
if flag not in oldflags:
oldflags.append(flag)
Also some more code documenting.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
That makes OfflineIMAP to use exclamation mark (!) instead of colon for storing
messages. Such files can be written to windows partitions. But you will probably
loose compatibility with other programs trying to read the same Maildir.
Signed-off-by: Vladimir Marek <vladimir.marek@oracle.com>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we were attempting to save out mails according to
http://www.qmail.org/man/man5/maildir.html in 4 steps:
1 Create a unique filename
2 Do stat(tmp/<filename>). If it found a file, wait 2 sec and go back to 1.
3 Create and write the message to the tmp/<filename>.
4 Link from tmp/* to new/*
(we did step 2 up to 15 times) But as stated by
http://wiki1.dovecot.org/MailboxFormat/Maildir (see section 'Issues with
the specification'), this is a pointless approach, e.g. there are race
issues between stating that the filename does not exist and the actual
moving (when it might exist).
So, we can simplify the steps as suggested in the dovecot wiki and
tighten up our safety at the same time.
One improvement that we do is to open the file, guaranteeing that it did
not exist before in an atomic manner, thus our simplified approach is
really more secure than what we had before.
Also, we throw an OfflineImapError at MESSAGE level when the supposedly
unique filename already exists, so that we can skip this message and
still continue with other messages.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
MaildirFolder.messagelist[*]['filename'] was storing the absolute file
paths for all stored emails. While this is convenient, it wastes much
space, as the folder prefix is always the same and it is known to the
MaildirFolder. Just 40 chars in a folder with 100k mails waste >4MB of
space. Adapt the few locations where we need the full path to construct
it dynamically.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We use getfullname() very often (thousands to millions), yet we
dynamically calculate the very same value over and over. Optimize this
by caching the value once and be done with it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make the folder classes use uidexists() more. Add some code
documentation while going through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
quickchanged() was iterating a lot, make use of some of the helper
functions that had been introduced recently and document the function a
bit better. No functional change.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than regetting the ui with UIBase.getglobalui() all the time, we get it once in the base class and let all derivative classes just make use of self.ui rather than refetching the ui all the time, this makes for a bit less code and shorter lines.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Dear All,
Attached is the patch that I have developed to provide maxage and
maxsize options. You can thus sync only the last x days of messages and
exclude large messages. All details in the attached git file.
Regards,
-Mike
-- Attached file included as plaintext by Ecartis --
-- File: submit
From 04fead2b46a79675a5b29de6f2b4088b9c9448e5 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Sun, 16 Aug 2009 17:00:49 +0430
Subject: [PATCH] Patch to provide maxage and maxsize account options to exclude old/large messages
This is designed to make offlineimap even better for low bandwidth connections.
maxage allows you to specify a number of days and only messages within
that range will be considered by offlineimap for the sync. This can be
useful if you would like to start using offlineimap with a large
existing account and do not want to import large archives of mail.
maxsize allows you to specify the maximum size of a message to consider so
that you can exclude messages with large attachments etc.
In both cases the cachemessagelist function of the folder was modified to ignore
messages that do not meet the criteria. If the criteria are not specified
then the existing code will be executed the same as before. If a message
does not meet the criteria it will be as though this message does not exist
- offlineimap will completely ignore it. It will not have flags updated,
it will not be deleted, it will not be considered at all.
When operating against an IMAP repository a server side search function
is used. This of course requires support for server side search.
I have tested this with either option, no options etc. against IMAP, Maildir
and Gmail. I have run variations of this patch here for the last 3 weeks or
so syncing about 4 accounts normally.
Closes: #535160
Bug#535160: offlineimap fails on encfs (FUSE) due to hard linking
Package: offlineimap
Version: 6.0.3
Severity: normal
Tags: patch
Hi,
if the localrepository is stored on an encfs, offlineimap fails as follows:
OfflineIMAP 6.0.3
Copyright (C) 2002 - 2008 John Goerzen <jgoerzen@complete.org>
This software comes with ABSOLUTELY NO WARRANTY; see the file
COPYING for details. This is free software, and you are welcome
to distribute it under the conditions laid out in COPYING.
***** Processing account pleione
Copying folder structure from IMAP to Maildir
Establishing connection to tunnel:eval `keychain --quiet --eval` && ssh -q mymailhost /usr/sbin/dovecot --exec-mail imap.
Syncing Drafts: IMAP -> Maildir
Syncing INBOX: IMAP -> Maildir
----------8<----------
Syncing net: IMAP -> Maildir
Copy message 360 IMAP[net] -> Maildir[net], LocalStatus[net]
Syncing personal: IMAP -> Maildir
Thread 'Copy message 360 from net' terminated with exception:
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/offlineimap/threadutil.py", line 149, in run
Thread.run(self)
File "threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "/var/lib/python-support/python2.4/offlineimap/folder/Base.py", line 282, in copymessageto
newuid = object.savemessage(uid, message, flags, rtime)
File "/var/lib/python-support/python2.4/offlineimap/folder/Maildir.py", line 198, in savemessage
os.path.join(tmpdir, messagename))
OSError: [Errno 1] Operation not permitted
Last 4 debug messages logged for Copy message 360 from net prior to exception:
----------8<----------
As encfs does not support hard linking, the rename of the temporary
message file fails.
This is the only occurrence of hard linking in offlineimap and can be
fixed by using an atomic `os.rename` instead of `os.link`+`os.unlink`.
I have been using the attached patch for more than 1.5 years and have
never again had any problems with offlineimap on encfs.
Regards,
Peter
Paul Hinze <paul.t.hinze@gmail.com> on 2008-12-22 at 19:16:
> I did a bit of debugging and asking around in #python and it turns out
> that because Maildir.py uses a conditional import of md5 from either
> hashlib or the md5 module, the md5.new() call is not always correct.
> The attached patch should fix this problem.
Thanks to Daniel Rall for pointing out that the attachment got stripped
off my last message. Patch follows.
Paul
----
From e01fdfbf62c61c31b514e826e6cc00fb98d88808 Mon Sep 17 00:00:00 2001
From: Paul Hinze <paul.t.hinze@gmail.com>
Date: Mon, 22 Dec 2008 18:42:44 -0600
Subject: [PATCH] Fixing md5 import when hashlib is available
Thanks to habnabit and Rhamphoryncus in #python for help debugging
Nicolas Sebrecht
Eygene Ryabinkin
Abdo Roig-Maranges
Sebastian Spaeth
Vladimir Marek
Vladimir Marek
Mike Dawson
John Goerzen
Peter Colberg
Paul Hinze
Loui Chang