PIM/offlineimap
1
0
Fork 0
mirror of https://github.com/OfflineIMAP/offlineimap.git synced 2024-07-03 08:30:56 +02:00
Code Issues Releases Wiki Activity

Merge branch 'next'

Browse Source
This commit is contained in:
Sebastian Spaeth 2011-09-30 09:44:02 +02:00
commit e63302395e
14 changed files with 338 additions and 207 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
Changelog.draft.rst
View File

@ -15,6 +15,17 @@ New Features
Changes Changes
------- -------
* Indicate progress when copying many messages (slightly change log format)
* Output how long an account sync took (min:sec).
Bug Fixes Bug Fixes
--------- ---------
* Syncing multiple accounts in single-threaded mode would fail as we try
to "register" a thread as belonging to two accounts which was
fatal. Make it non-fatal (it can be legitimate).
* New folders on the remote would be skipped on the very sync run they
are created and only by synced in subsequent runs. Fixed.

153
docs/FAQ.rst
View File

@ -49,50 +49,40 @@ increase performance of the sync.
Don’t set the number too high. If you do that, things might actually slow down Don’t set the number too high. If you do that, things might actually slow down
as your link gets saturated. Also, too many connections can cause mail servers as your link gets saturated. Also, too many connections can cause mail servers
to have excessive load. Administrators might take unkindly to this, and the to have excessive load. Administrators might take unkindly to this, and the
server might bog down. There are many variables in the optimal setting; server might bog down. There are many variables in the optimal setting; experimentation may help.
experimentation may help.
An informal benchmark yields these results for my setup:: See the Performance section in the MANUAL for some tips.
* 10 minutes with MacOS X Mail.app “manual cache”
* 5 minutes with GNUS agent sync
* 20 seconds with OfflineIMAP 1.x
* 9 seconds with OfflineIMAP 2.x
* 3 seconds with OfflineIMAP 3.x “cold start”
* 2 seconds with OfflineIMAP 3.x “held connection”
What platforms does OfflineIMAP support? What platforms does OfflineIMAP support?
---------------------------------------- ----------------------------------------
It should run on most platforms supported by Python, which are quite a few. I It should run on most platforms supported by Python, with one exception: we do not support Windows, but some have made it work there.
do not support Windows myself, but some have made it work there. Use on
Windows
These answers have been reported by OfflineIMAP users. I do not run OfflineIMAP The following has been reported by OfflineIMAP users. We do not test
on Windows myself, so I can’t directly address their accuracy. OfflineIMAP on Windows, so we can’t directly address their accuracy.
The basic answer is that it’s possible and doesn’t require hacking OfflineIMAP The basic answer is that it’s possible and doesn’t require hacking OfflineIMAP
source code. However, it’s not necessarily trivial. The information below is source code. However, it’s not necessarily trivial. The information below is
based in instructions submitted by Chris Walker. based in instructions submitted by Chris Walker::
First, you must run OfflineIMAP in the Cygwin environment. The Windows First, you must run OfflineIMAP in the Cygwin environment. The Windows
filesystem is not powerful enough to accomodate Maildir by itself. filesystem is not powerful enough to accomodate Maildir by itself.
Next, you’ll need to mount your Maildir directory in a special way. There is Next, you’ll need to mount your Maildir directory in a special
information for doing that at http://barnson.org/node/295. That site gives this way. There is information for doing that at
example:: http://barnson.org/node/295. That site gives this example::
mount -f -s -b -o managed "d:/tmp/mail" "/home/of/mail" mount -f -s -b -o managed "d:/tmp/mail" "/home/of/mail"
That URL also has more details on making OfflineIMAP work with Windows. That URL also has more details on making OfflineIMAP work with Windows.
Does OfflineIMAP support mbox, mh, or anything else other than Maildir? Does OfflineIMAP support mbox, mh, or anything else other than Maildir?
----------------------------------------------------------------------- -----------------------------------------------------------------------
Not directly. Maildir was the easiest to implement. I’m not planning to write Not directly. Maildir was the easiest to implement. We are not planning
mbox code for OfflineIMAP, though if someone sent me well-written mbox support to write an mbox-backend, though if someone sent me well-written mbox
and pledged to support it, I’d commit it to the tree. support and pledged to support it, it would be committed it to the tree.
However, OfflineIMAP can directly sync accounts on two different IMAP servers However, OfflineIMAP can directly sync accounts on two different IMAP servers
together. So you could install an IMAP server on your local machine that together. So you could install an IMAP server on your local machine that
@ -105,22 +95,25 @@ point your mail readers to that IMAP server on localhost.
What is the UID validity problem for folder? What is the UID validity problem for folder?
-------------------------------------------- --------------------------------------------
IMAP servers use a unique ID (UID) to refer to a specific message. This number IMAP servers use a folders UIDVALIDITY value in combination with a
is guaranteed to be unique to a particular message forever. No other message in unique ID (UID) to refer to a specific message. This is guaranteed to
the same folder will ever get the same UID. UIDs are an integral part of be unique to a particular message forever. No other message in the same
`OfflineIMAP`_'s synchronization scheme; they are used to match up messages on folder will ever get the same UID as long as UIDVALIDITY remains
your computer to messages on the server. unchanged. UIDs are an integral part of `OfflineIMAP`_'s
synchronization scheme; they are used to match up messages on your
computer to messages on the server.
Sometimes, the UIDs on the server might get reset. Usually this will happen if Sometimes, the UIDs on the server might get reset. Usually this will
you delete and then recreate a folder. When you create a folder, the server happen if you delete and then recreate a folder. When you create a
will often start the UID back from 1. But `OfflineIMAP`_ might still have the folder, the server will often start the UID back from 1. But
UIDs from the previous folder by the same name stored. `OfflineIMAP`_ will `OfflineIMAP`_ might still have the UIDs from the previous folder by the
detect this condition and skip the folder. This is GOOD, because it prevents same name stored. `OfflineIMAP`_ will detect this condition because of
data loss. the changed UIDVALIDITY value and skip the folder. This is GOOD,
because it prevents data loss.
You can fix it by removing your local folder and cache data. For instance, if In the IMAP<->Maildir case, you can fix it by removing your local folder
your folders are under `~/Folders` and the folder with the problem is INBOX, and cache data. For instance, if your folders are under `~/Folders` and
you'd type this:: the folder with the problem is INBOX, you'd type this::
rm -r ~/Folders/INBOX rm -r ~/Folders/INBOX
rm -r ~/.offlineimap/Account-AccountName/LocalStatus/INBOX rm -r ~/.offlineimap/Account-AccountName/LocalStatus/INBOX
@ -146,12 +139,10 @@ This question comes up frequently on the `mailing list`_. You can find a detail
discussion of the problem there discussion of the problem there
http://lists.complete.org/offlineimap@complete.org/2003/04/msg00012.html.gz. http://lists.complete.org/offlineimap@complete.org/2003/04/msg00012.html.gz.
How do I add or delete a folder? How do I automatically delete a folder?
-------------------------------- ---------------------------------------
OfflineIMAP does not currently provide this feature. However, if you create a OfflineIMAP does not currently provide this feature. You will have to delete folders manually. See next entry too.
new folder on the remote server, OfflineIMAP will detect this and create the
corresponding folder locally automatically.
May I delete local folders? May I delete local folders?
--------------------------- ---------------------------
@ -175,6 +166,7 @@ It will perform a check on startup and abort if another `OfflineIMAP`_ is
already running. If you need to schedule synchronizations, you'll probably already running. If you need to schedule synchronizations, you'll probably
find autorefresh settings more convenient than cron. Alternatively, you can find autorefresh settings more convenient than cron. Alternatively, you can
set a separate metadata directory for each instance. set a separate metadata directory for each instance.
In the future, we will lock each account individually rather than having one global lock.
Can I copy messages between folders? Can I copy messages between folders?
--------------------------------------- ---------------------------------------
@ -197,10 +189,7 @@ next run it will upload the message to second server and delete on first, etc.
Does OfflineIMAP support POP? Does OfflineIMAP support POP?
----------------------------- -----------------------------
No. POP is not robust enough to do a completely reliable multi-machine sync No.
like OfflineIMAP can do.
OfflineIMAP will never support POP.
How is OfflineIMAP conformance? How is OfflineIMAP conformance?
------------------------------- -------------------------------
@ -214,17 +203,21 @@ How is OfflineIMAP conformance?
Can I force OfflineIMAP to sync a folder right now? Can I force OfflineIMAP to sync a folder right now?
--------------------------------------------------- ---------------------------------------------------
Yes, if you use the `Blinkenlights` UI. That UI shows the active accounts Yes,
1) if you use the `Blinkenlights` UI. That UI shows the active accounts
as follows:: as follows::
4: [active] *Control: . 4: [active] *Control: .
3: [ 4:36] personal: 3: [ 4:36] personal:
2: [ 3:37] work: . 2: [ 3:37] work: .
1: [ 6:28] uni: 1: [ 6:28] uni:
Simply press the appropriate digit (`3` for `personal`, etc.) to resync that Simply press the appropriate digit (`3` for `personal`, etc.) to
account immediately. This will be ignored if a resync is already in progress resync that account immediately. This will be ignored if a resync is
for that account. already in progress for that account.
2) while in sleep mode, you can also send a SIGUSR1. See the `Signals
on UNIX`_ section in the MANUAL for details.
Configuration Questions Configuration Questions
======================= =======================
@ -248,10 +241,12 @@ what folders are present on the IMAP server and synchronize them. You can use
the folderfilter and nametrans configuration file options to request only the folderfilter and nametrans configuration file options to request only
certain folders and rename them as they come in if you like. certain folders and rename them as they come in if you like.
Also you can configure OfflineImap to only synchronize "subscribed" folders.
How do I prevent certain folders from being synced? How do I prevent certain folders from being synced?
--------------------------------------------------- ---------------------------------------------------
Use the folderfilter option. Use the folderfilter option. See the MANUAL for details and examples.
What is the mailbox name recorder (mbnames) for? What is the mailbox name recorder (mbnames) for?
------------------------------------------------ ------------------------------------------------
@ -261,9 +256,11 @@ Some mail readers, such as mutt, are not capable of automatically determining th
Does OfflineIMAP verify SSL certificates? Does OfflineIMAP verify SSL certificates?
----------------------------------------- -----------------------------------------
By default, no. However, as of version 6.3.2, it is possible to enforce verification You can verify an imapserver's certificate by specifying the CA
of SSL certificate on a per-repository basis by setting the `sslcacertfile` option in the certificate on a per-repository basis by setting the `sslcacertfile`
config file. (See the example offlineimap.conf for details.) option in the config file. (See the example offlineimap.conf for
details.) If you do not specify any CA certificate, you will be presented with the server's certificate fingerprint and add that to the configuration file, to make sure it remains unchanged.
No verification happens if connecting via STARTTLS.
How do I generate an `sslcacertfile` file? How do I generate an `sslcacertfile` file?
------------------------------------------ ------------------------------------------
@ -293,23 +290,6 @@ with the IMAP RFCs. Some servers provide imperfect compatibility that may be
good enough for general clients. OfflineIMAP needs more features, specifically good enough for general clients. OfflineIMAP needs more features, specifically
support for UIDs, in order to do its job accurately and completely. support for UIDs, in order to do its job accurately and completely.
Microsoft Exchange
------------------
Several users have reported problems with Microsoft Exchange servers in
conjunction with OfflineIMAP. This generally seems to be related to the
Exchange servers not properly following the IMAP standards.
Mark Biggers has posted some information to the OfflineIMAP `mailing list`_
about how he made it work.
Other users have indicated that older (5.5) releases of Exchange are so bad
that they will likely not work at all.
I do not have access to Exchange servers for testing, so any problems with it,
if they can even be solved at all, will require help from OfflineIMAP users to
find and fix.
Client Notes Client Notes
============ ============
@ -494,12 +474,13 @@ To send a patch, we recommend using 'git send-email'.
Where from should my patches be based on? Where from should my patches be based on?
----------------------------------------- -----------------------------------------
Depends. If you're not sure, it should start off of the master branch. master is Depends. If you're not sure, it should start off of the master
the branch where new patches should be based on by default. branch. master is the branch where new patches should be based on by
default.
Obvious materials for next release (e.g. new features) start off of current Obvious materials for next release (e.g. new features) start off of
next. Also, next is the natural branch to write patches on top of commits not current next. Also, next is the natural branch to write patches on top
already in master. of commits not already in master.
A fix for a very old bug or security issue may start off of maint. This isn't A fix for a very old bug or security issue may start off of maint. This isn't
needed since such fix are backported by the maintainer, though. needed since such fix are backported by the maintainer, though.

136
docs/MANUAL.rst
View File

@ -6,7 +6,7 @@
Powerful IMAP/Maildir synchronization and reader support Powerful IMAP/Maildir synchronization and reader support
-------------------------------------------------------- --------------------------------------------------------
:Author: John Goerzen <jgoerzen@complete.org> :Author: John Goerzen <jgoerzen@complete.org> & contributors
:Date: 2011-01-15 :Date: 2011-01-15
:Copyright: GPL v2 :Copyright: GPL v2
:Manual section: 1 :Manual section: 1
@ -22,10 +22,8 @@ emails between them, so that you can read the same mailbox from multiple
computers. The REMOTE repository is some IMAP server, while LOCAL can be computers. The REMOTE repository is some IMAP server, while LOCAL can be
either a local Maildir or another IMAP server. either a local Maildir or another IMAP server.
Missing folders will be automatically created on the LOCAL side, however Missing folders will be automatically created on both sides if
NO folders will currently be created on the REMOTE repository needed. No folders will be deleted at the moment.
automatically (it will sync your emails from local folders if
corresponding REMOTE folders already exist).
Configuring OfflineImap in basic mode is quite easy, however it provides Configuring OfflineImap in basic mode is quite easy, however it provides
an amazing amount of flexibility for those with special needs. You can an amazing amount of flexibility for those with special needs. You can
@ -159,12 +157,10 @@ Blinkenlights
Blinkenlights is an interface designed to be sleek, fun to watch, and Blinkenlights is an interface designed to be sleek, fun to watch, and
informative of the overall picture of what OfflineIMAP is doing. informative of the overall picture of what OfflineIMAP is doing.
Blinkenlights contains a row of "LEDs" with command buttons and a log. Blinkenlights contains a row of "LEDs" with command buttons and a log.
The log shows more detail about what is happening and is color-coded to match The log shows more detail about what is happening and is color-coded to match
the color of the lights. the color of the lights.
Each light in the Blinkenlights interface represents a thread of execution -- Each light in the Blinkenlights interface represents a thread of execution --
that is, a particular task that OfflineIMAP is performing right now. The colors that is, a particular task that OfflineIMAP is performing right now. The colors
indicate what task the particular thread is performing, and are as follows: indicate what task the particular thread is performing, and are as follows:
@ -232,7 +228,7 @@ English-speaking world. One version ran in its entirety as follows:
TTYUI TTYUI
--------- ------
TTYUI interface is for people running in terminals. It prints out basic TTYUI interface is for people running in terminals. It prints out basic
status messages and is generally friendly to use on a console or xterm. status messages and is generally friendly to use on a console or xterm.
@ -245,7 +241,7 @@ Basic is designed for situations in which OfflineIMAP will be run
non-attended and the status of its execution will be logged. This user non-attended and the status of its execution will be logged. This user
interface is not capable of reading a password from the keyboard; interface is not capable of reading a password from the keyboard;
account passwords must be specified using one of the configuration file account passwords must be specified using one of the configuration file
options. options. For example, it will not print periodic sleep announcements and tends to be a tad less verbose, in general.
Quiet Quiet
@ -367,6 +363,128 @@ accounts will abort any current sleep and will exit after a currently running
synchronization has finished. This signal can be used to gracefully exit out of synchronization has finished. This signal can be used to gracefully exit out of
a running offlineimap "daemon". a running offlineimap "daemon".
Folder filtering and Name translation
=====================================
OfflineImap provides advanced and potentially complex possibilities for
filtering and translating folder names. If you don't need this, you can
safely skip this section.
folderfilter
------------
If you do not want to synchronize all your filters, you can specify a folderfilter function that determines which folders to include in a sync and which to exclude. Typically, you would set a folderfilter option on the remote repository only, and it would be a lambda or any other python function.
If the filter function returns True, the folder will be synced, if it
returns False, it. The folderfilter operates on the *UNTRANSLATED* name
(before any nametrans translation takes place).
Example 1: synchronizing only INBOX and Sent::
folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']
Example 2: synchronizing everything except Trash::
folderfilter = lambda foldername: foldername not in ['Trash']
Example 3: Using a regular expression to exclude Trash and all folders
containing the characters "Del"::
folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)
If folderfilter is not specified, ALL remote folders will be
synchronized.
You can span multiple lines by indenting the others. (Use backslashes
at the end when required by Python syntax) For instance::
folderfilter = lambda foldername: foldername in
['INBOX', 'Sent Mail', 'Deleted Items',
'Received']
You only need a folderfilter option on the local repository if you want to prevent some folders on the local repository to be created on the remote one.
Even if you filtered out folders, You can specify folderincludes to
include additional folders. It should return a Python list. This might
be used to include a folder that was excluded by your folderfilter rule,
to include a folder that your server does not specify with its LIST
option, or to include a folder that is outside your basic reference. The
'reference' value will not be prefixed to this folder name, even if you
have specified one. For example::
folderincludes = ['debian.user', 'debian.personal']
nametrans
----------
Sometimes, folders need to have different names on the remote and the
local repositories. To achieve this you can specify a folder name
translator. This must be a eval-able Python expression that takes a
foldername arg and returns the new value. I suggest a lambda. This
example below will remove "INBOX." from the leading edge of folders
(great for Courier IMAP users)::
nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)
Using Courier remotely and want to duplicate its mailbox naming
locally? Try this::
nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
WARNING: you MUST construct nametrans rules such that it NEVER returns
the same value for two folders, UNLESS the second values are
filtered out by folderfilter below. That is, two filters on one side may never point to the same folder on the other side. Failure to follow this rule
will result in undefined behavior. See also *Sharing a maildir with multiple IMAP servers* in the `PITFALLS & ISSUES`_ section.
Where to put nametrans rules, on the remote and/or local repository?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
If you never intend to create new folders on the LOCAL repository that
need to be synced to the REMOTE repository, it is sufficient to create a
nametrans rule on the remote Repository section. This will be used to
determine the names of new folder names on the LOCAL repository, and to
match existing folders that correspond.
*IF* you create folders on the local repository, that are supposed to be
automatically created on the remote repository, you will need to create
a nametrans rule that provides the reverse name translation.
(A nametrans rule provides only a one-way translation of names and in
order to know which names folders on the LOCAL side would have on the
REMOTE side, you need to specify the reverse nametrans rule on the local
repository)
OfflineImap will complain if it needs to create a new folder on the
remote side and a back-and-forth nametrans-lation does not yield the
original foldername (as that could potentially lead to infinite folder
creation cycles).
What folder separators do I need to use in nametrans rules?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
**Q:** If I sync from an IMAP server with folder separator '/' to a
Maildir using the default folder separator '.' which do I need to use
in nametrans rules?::
nametrans = lambda f: "INBOX/" + f
or::
nametrans = lambda f: "INBOX." + f
**A:** Generally use the folder separator as defined in the repository
you write the nametrans rule for. That is, use '/' in the above
case. We will pass in the untranslated name of the IMAP folder as
parameter (here `f`). The translated name will ultimately have all
folder separators be replaced with the destination repositories'
folder separator.
So if ̀f` was "Sent", the first nametrans yields the translated name
"INBOX/Sent" to be used on the other side. As that repository uses the
folder separator '.' rather than '/', the ultimate name to be used will
be "INBOX.Sent".
(As a final note, the smart will see that both variants of the above
nametrans rule would have worked identically in this case)
KNOWN BUGS KNOWN BUGS
========== ==========

50
offlineimap/accounts.py
View File

@ -1,5 +1,4 @@
# Copyright (C) 2003 John Goerzen # Copyright (C) 2003-2011 John Goerzen & contributors
# <jgoerzen@complete.org>
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by
@ -196,7 +195,6 @@ class SyncableAccount(Account):
def syncrunner(self): def syncrunner(self):
self.ui.registerthread(self.name) self.ui.registerthread(self.name)
self.ui.acct(self.name)
accountmetadata = self.getaccountmeta() accountmetadata = self.getaccountmeta()
if not os.path.exists(accountmetadata): if not os.path.exists(accountmetadata):
os.mkdir(accountmetadata, 0700) os.mkdir(accountmetadata, 0700)
@ -208,32 +206,32 @@ class SyncableAccount(Account):
# Loop account sync if needed (bail out after 3 failures) # Loop account sync if needed (bail out after 3 failures)
looping = 3 looping = 3
while looping: while looping:
self.ui.acct(self)
try: try:
try: self.lock()
self.lock() self.sync()
self.sync() except (KeyboardInterrupt, SystemExit):
except (KeyboardInterrupt, SystemExit): raise
raise except OfflineImapError, e:
except OfflineImapError, e: # Stop looping and bubble up Exception if needed.
# Stop looping and bubble up Exception if needed. if e.severity >= OfflineImapError.ERROR.REPO:
if e.severity >= OfflineImapError.ERROR.REPO: if looping:
if looping: looping -= 1
looping -= 1 if e.severity >= OfflineImapError.ERROR.CRITICAL:
if e.severity >= OfflineImapError.ERROR.CRITICAL: raise
raise self.ui.error(e, exc_info()[2])
self.ui.error(e, exc_info()[2]) except Exception, e:
except Exception, e: self.ui.error(e, exc_info()[2], msg = "While attempting to sync"
self.ui.error(e, msg = "While attempting to sync " " account '%s'" % self)
"account %s:\n %s"% (self, traceback.format_exc())) else:
else: # after success sync, reset the looping counter to 3
# after success sync, reset the looping counter to 3 if self.refreshperiod:
if self.refreshperiod: looping = 3
looping = 3
finally: finally:
self.ui.acctdone(self)
self.unlock() self.unlock()
if looping and self.sleeper() >= 2: if looping and self.sleeper() >= 2:
looping = 0 looping = 0
self.ui.acctdone(self.name)
def getaccountmeta(self): def getaccountmeta(self):
return os.path.join(self.metadatadir, 'Account-' + self.name) return os.path.join(self.metadatadir, 'Account-' + self.name)
@ -267,7 +265,7 @@ class SyncableAccount(Account):
remoterepos = self.remoterepos remoterepos = self.remoterepos
localrepos = self.localrepos localrepos = self.localrepos
statusrepos = self.statusrepos statusrepos = self.statusrepos
# replicate the folderstructure from REMOTE to LOCAL # replicate the folderstructure between REMOTE to LOCAL
if not localrepos.getconfboolean('readonly', False): if not localrepos.getconfboolean('readonly', False):
self.ui.syncfolders(remoterepos, localrepos) self.ui.syncfolders(remoterepos, localrepos)
remoterepos.syncfoldersto(localrepos, statusrepos) remoterepos.syncfoldersto(localrepos, statusrepos)
@ -281,7 +279,7 @@ class SyncableAccount(Account):
thread = InstanceLimitedThread(\ thread = InstanceLimitedThread(\
instancename = 'FOLDER_' + self.remoterepos.getname(), instancename = 'FOLDER_' + self.remoterepos.getname(),
target = syncfolder, target = syncfolder,
name = "Folder sync [%s]" % self, name = "Folder %s [acc: %s]" % (remotefolder, self),
args = (self.name, remoterepos, remotefolder, localrepos, args = (self.name, remoterepos, remotefolder, localrepos,
statusrepos, quick)) statusrepos, quick))
thread.setDaemon(1) thread.setDaemon(1)

8
offlineimap/folder/Base.py
View File

@ -266,7 +266,6 @@ class BaseFolder(object):
statusfolder.savemessage(uid, None, flags, rtime) statusfolder.savemessage(uid, None, flags, rtime)
return return
self.ui.copyingmessage(uid, self, dstfolder)
# If any of the destinations actually stores the message body, # If any of the destinations actually stores the message body,
# load it up. # load it up.
if dstfolder.storesmessages(): if dstfolder.storesmessages():
@ -331,15 +330,16 @@ class BaseFolder(object):
copylist = filter(lambda uid: not \ copylist = filter(lambda uid: not \
statusfolder.uidexists(uid), statusfolder.uidexists(uid),
self.getmessageuidlist()) self.getmessageuidlist())
for uid in copylist: num_to_copy = len(copylist)
for num, uid in enumerate(copylist):
self.ui.copyingmessage(uid, num+1, num_to_copy, self, dstfolder)
# exceptions are caught in copymessageto() # exceptions are caught in copymessageto()
if self.suggeststhreads(): if self.suggeststhreads():
self.waitforthread() self.waitforthread()
thread = threadutil.InstanceLimitedThread(\ thread = threadutil.InstanceLimitedThread(\
self.getcopyinstancelimit(), self.getcopyinstancelimit(),
target = self.copymessageto, target = self.copymessageto,
name = "Copy message %d from %s" % (uid, name = "Copy message from %s:%s" % (self.repository, self),
self.getvisiblename()),
args = (uid, dstfolder, statusfolder)) args = (uid, dstfolder, statusfolder))
thread.setDaemon(1) thread.setDaemon(1)
thread.start() thread.start()

14
offlineimap/init.py
View File

@ -1,6 +1,5 @@
# OfflineIMAP initialization code # OfflineIMAP initialization code
# Copyright (C) 2002-2007 John Goerzen # Copyright (C) 2002-2011 John Goerzen & contributors
# <jgoerzen@complete.org>
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by
@ -157,11 +156,14 @@ class OfflineImap:
if not options.singlethreading: if not options.singlethreading:
logging.warn("Profile mode: Forcing to singlethreaded.") logging.warn("Profile mode: Forcing to singlethreaded.")
options.singlethreading = True options.singlethreading = True
profiledir = options.profiledir if os.path.exists(options.profiledir):
os.mkdir(profiledir) logging.warn("Profile mode: Directory '%s' already exists!" %
threadutil.setprofiledir(profiledir) options.profiledir)
else:
os.mkdir(options.profiledir)
threadutil.ExitNotifyThread.set_profiledir(options.profiledir)
logging.warn("Profile mode: Potentially large data will be " logging.warn("Profile mode: Potentially large data will be "
"created in '%s'" % profiledir) "created in '%s'" % options.profiledir)
#override a config value #override a config value
if options.configoverride: if options.configoverride:

13
offlineimap/repository/Base.py
View File

@ -144,7 +144,8 @@ class BaseRepository(object, CustomConfig.ConfigHelperMixin):
src_repo = self src_repo = self
src_folders = src_repo.getfolders() src_folders = src_repo.getfolders()
dst_folders = dst_repo.getfolders() dst_folders = dst_repo.getfolders()
# Do we need to refresh the folder list afterwards?
src_haschanged, dst_haschanged = False, False
# Create hashes with the names, but convert the source folders # Create hashes with the names, but convert the source folders
# to the dest folder's sep. # to the dest folder's sep.
src_hash = {} src_hash = {}
@ -160,6 +161,7 @@ class BaseRepository(object, CustomConfig.ConfigHelperMixin):
if src_folder.sync_this and not src_name in dst_hash: if src_folder.sync_this and not src_name in dst_hash:
try: try:
dst_repo.makefolder(src_name) dst_repo.makefolder(src_name)
dst_haschanged = True # Need to refresh list
except OfflineImapError, e: except OfflineImapError, e:
self.ui.error(e, exc_info()[2], self.ui.error(e, exc_info()[2],
"Creating folder %s on repository %s" %\ "Creating folder %s on repository %s" %\
@ -203,6 +205,7 @@ class BaseRepository(object, CustomConfig.ConfigHelperMixin):
try: try:
src_repo.makefolder(newsrc_name) src_repo.makefolder(newsrc_name)
src_haschanged = True # Need to refresh list
except OfflineImapError, e: except OfflineImapError, e:
self.ui.error(e, exc_info()[2], self.ui.error(e, exc_info()[2],
"Creating folder %s on repository %s" %\ "Creating folder %s on repository %s" %\
@ -211,7 +214,13 @@ class BaseRepository(object, CustomConfig.ConfigHelperMixin):
status_repo.makefolder(newsrc_name.replace( status_repo.makefolder(newsrc_name.replace(
src_repo.getsep(), status_repo.getsep())) src_repo.getsep(), status_repo.getsep()))
# Find deleted folders. # Find deleted folders.
# We don't delete folders right now. # TODO: We don't delete folders right now.
#Forget old list of cached folders so we get new ones if needed
if src_haschanged:
self.forgetfolders()
if dst_haschanged:
dst_repo.forgetfolders()
def startkeepalive(self): def startkeepalive(self):
"""The default implementation will do nothing.""" """The default implementation will do nothing."""

4
offlineimap/repository/IMAP.py
View File

@ -311,6 +311,10 @@ class IMAPRepository(BaseRepository):
def makefolder(self, foldername): def makefolder(self, foldername):
"""Create a folder on the IMAP server """Create a folder on the IMAP server
This will not update the list cached in :meth:`getfolders`. You
will need to invoke :meth:`forgetfolders` to force new caching
when you are done creating folders yourself.
:param foldername: Full path of the folder to be created.""" :param foldername: Full path of the folder to be created."""
#TODO: IMHO this existing commented out code is correct and #TODO: IMHO this existing commented out code is correct and
#should be enabled, but this would change the behavior for #should be enabled, but this would change the behavior for

6
offlineimap/repository/Maildir.py
View File

@ -72,6 +72,10 @@ class MaildirRepository(BaseRepository):
def makefolder(self, foldername): def makefolder(self, foldername):
"""Create new Maildir folder if necessary """Create new Maildir folder if necessary
This will not update the list cached in getfolders(). You will
need to invoke :meth:`forgetfolders` to force new caching when
you are done creating folders yourself.
:param foldername: A relative mailbox name. The maildir will be :param foldername: A relative mailbox name. The maildir will be
created in self.root+'/'+foldername. All intermediate folder created in self.root+'/'+foldername. All intermediate folder
levels will be created if they do not exist yet. 'cur', levels will be created if they do not exist yet. 'cur',
@ -108,8 +112,6 @@ class MaildirRepository(BaseRepository):
(foldername, subdir)) (foldername, subdir))
else: else:
raise raise
# Invalidate the folder cache
self.folders = None
def deletefolder(self, foldername): def deletefolder(self, foldername):
self.ui.warn("NOT YET IMPLEMENTED: DELETE FOLDER %s" % foldername) self.ui.warn("NOT YET IMPLEMENTED: DELETE FOLDER %s" % foldername)

29
offlineimap/threadutil.py
View File

@ -1,6 +1,5 @@
# Copyright (C) 2002, 2003 John Goerzen # Copyright (C) 2002-2011 John Goerzen & contributors
# Thread support module # Thread support module
# <jgoerzen@complete.org>
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by
@ -20,15 +19,10 @@ from threading import Lock, Thread, BoundedSemaphore
from Queue import Queue, Empty from Queue import Queue, Empty
import traceback import traceback
from thread import get_ident # python < 2.6 support from thread import get_ident # python < 2.6 support
import os.path
import sys import sys
from offlineimap.ui import getglobalui from offlineimap.ui import getglobalui
profiledir = None
def setprofiledir(newdir):
global profiledir
profiledir = newdir
###################################################################### ######################################################################
# General utilities # General utilities
###################################################################### ######################################################################
@ -132,11 +126,14 @@ def threadexited(thread):
class ExitNotifyThread(Thread): class ExitNotifyThread(Thread):
"""This class is designed to alert a "monitor" to the fact that a thread has """This class is designed to alert a "monitor" to the fact that a thread has
exited and to provide for the ability for it to find out why.""" exited and to provide for the ability for it to find out why."""
profiledir = None
"""class variable that is set to the profile directory if required"""
def run(self): def run(self):
global exitthreads, profiledir global exitthreads
self.threadid = get_ident() self.threadid = get_ident()
try: try:
if not profiledir: # normal case if not ExitNotifyThread.profiledir: # normal case
Thread.run(self) Thread.run(self)
else: else:
try: try:
@ -148,9 +145,8 @@ class ExitNotifyThread(Thread):
prof = prof.runctx("Thread.run(self)", globals(), locals()) prof = prof.runctx("Thread.run(self)", globals(), locals())
except SystemExit: except SystemExit:
pass pass
prof.dump_stats( \ prof.dump_stats(os.path.join(ExitNotifyThread.profiledir,
profiledir + "/" + str(self.threadid) + "_" + \ "%s_%s.prof" % (self.threadid, self.getName())))
self.getName() + ".prof")
except: except:
self.setExitCause('EXCEPTION') self.setExitCause('EXCEPTION')
if sys: if sys:
@ -194,7 +190,12 @@ class ExitNotifyThread(Thread):
a call to setExitMessage(), or None if there was no such message a call to setExitMessage(), or None if there was no such message
set.""" set."""
return self.exitmessage return self.exitmessage
@classmethod
def set_profiledir(cls, directory):
"""If set, will output profile information to 'directory'"""
cls.profiledir = directory
###################################################################### ######################################################################
# Instance-limited threads # Instance-limited threads

5
offlineimap/ui/Blinkenlights.py
View File

@ -54,9 +54,10 @@ class BlinkenBase:
s.gettf().setcolor('blue') s.gettf().setcolor('blue')
s.__class__.__bases__[-1].syncingmessages(s, sr, sf, dr, df) s.__class__.__bases__[-1].syncingmessages(s, sr, sf, dr, df)
def copyingmessage(s, uid, src, destfolder): def copyingmessage(s, uid, num, num_to_copy, src, destfolder):
s.gettf().setcolor('orange') s.gettf().setcolor('orange')
s.__class__.__bases__[-1].copyingmessage(s, uid, src, destfolder) s.__class__.__bases__[-1].copyingmessage(s, uid, num, num_to_copy, src,
destfolder)
def deletingmessages(s, uidlist, destlist): def deletingmessages(s, uidlist, destlist):
s.gettf().setcolor('red') s.gettf().setcolor('red')

2
offlineimap/ui/Machine.py
View File

@ -108,7 +108,7 @@ class MachineUI(UIBase):
(s.getnicename(sr), sf.getname(), s.getnicename(dr), (s.getnicename(sr), sf.getname(), s.getnicename(dr),
df.getname())) df.getname()))
def copyingmessage(self, uid, srcfolder, destfolder): def copyingmessage(self, uid, num, num_to_copy, srcfolder, destfolder):
self._printData('copyingmessage', "%d\n%s\n%s\n%s[%s]" % \ self._printData('copyingmessage', "%d\n%s\n%s\n%s[%s]" % \
(uid, self.getnicename(srcfolder), srcfolder.getname(), (uid, self.getnicename(srcfolder), srcfolder.getname(),
self.getnicename(destfolder), destfolder)) self.getnicename(destfolder), destfolder))

6
offlineimap/ui/TTY.py
View File

@ -15,7 +15,6 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software # along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
from UIBase import UIBase from UIBase import UIBase
from getpass import getpass from getpass import getpass
import sys import sys
@ -37,10 +36,7 @@ class TTYUI(UIBase):
#if the next output comes from a different thread than our last one #if the next output comes from a different thread than our last one
#add the info. #add the info.
#Most look like 'account sync foo' or 'Folder sync foo'. #Most look like 'account sync foo' or 'Folder sync foo'.
try: threadname = currentThread().getName()
threadname = currentThread().name
except AttributeError:
threadname = currentThread().getName()
if (threadname == s._lastThreaddisplay \ if (threadname == s._lastThreaddisplay \
or threadname == 'MainThread'): or threadname == 'MainThread'):
print " %s" % msg print " %s" % msg

106
offlineimap/ui/UIBase.py
View File

@ -1,6 +1,5 @@
# UI base class # UI base class
# Copyright (C) 2002 John Goerzen # Copyright (C) 2002-2011 John Goerzen & contributors
# <jgoerzen@complete.org>
# #
# This program is free software; you can redistribute it and/or modify # This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by # it under the terms of the GNU General Public License as published by
@ -47,6 +46,9 @@ class UIBase:
s.debugmessages = {} s.debugmessages = {}
s.debugmsglen = 50 s.debugmsglen = 50
s.threadaccounts = {} s.threadaccounts = {}
"""dict linking active threads (k) to account names (v)"""
s.acct_startimes = {}
"""linking active accounts with the time.time() when sync started"""
s.logfile = None s.logfile = None
s.exc_queue = Queue() s.exc_queue = Queue()
"""saves all occuring exceptions, so we can output them at the end""" """saves all occuring exceptions, so we can output them at the end"""
@ -117,29 +119,32 @@ class UIBase:
if exc_traceback: if exc_traceback:
self._msg(traceback.format_tb(exc_traceback)) self._msg(traceback.format_tb(exc_traceback))
def registerthread(s, account): def registerthread(self, account):
"""Provides a hint to UIs about which account this particular """Register current thread as being associated with an account name"""
thread is processing.""" cur_thread = threading.currentThread()
if s.threadaccounts.has_key(threading.currentThread()): if cur_thread in self.threadaccounts:
raise ValueError, "Thread %s already registered (old %s, new %s)" %\ # was already associated with an old account, update info
(threading.currentThread().getName(), self.debug('thread', "Register thread '%s' (previously '%s', now "
s.getthreadaccount(s), account) "'%s')" % (cur_thread.getName(),
s.threadaccounts[threading.currentThread()] = account self.getthreadaccount(cur_thread), account))
s.debug('thread', "Register new thread '%s' (account '%s')" %\ else:
(threading.currentThread().getName(), account)) self.debug('thread', "Register new thread '%s' (account '%s')" %\
(cur_thread.getName(), account))
self.threadaccounts[cur_thread] = account
def unregisterthread(s, thr): def unregisterthread(self, thr):
"""Recognizes a thread has exited.""" """Unregister a thread as being associated with an account name"""
if s.threadaccounts.has_key(thr): if self.threadaccounts.has_key(thr):
del s.threadaccounts[thr] del self.threadaccounts[thr]
s.debug('thread', "Unregister thread '%s'" % thr.getName()) self.debug('thread', "Unregister thread '%s'" % thr.getName())
def getthreadaccount(s, thr = None): def getthreadaccount(self, thr = None):
"""Get name of account for a thread (current if None)"""
if not thr: if not thr:
thr = threading.currentThread() thr = threading.currentThread()
if s.threadaccounts.has_key(thr): if thr in self.threadaccounts:
return s.threadaccounts[thr] return self.threadaccounts[thr]
return '*Control' return '*Control' # unregistered thread is '*Control'
def debug(s, debugtype, msg): def debug(s, debugtype, msg):
thisthread = threading.currentThread() thisthread = threading.currentThread()
@ -223,29 +228,33 @@ class UIBase:
s._msg(offlineimap.banner) s._msg(offlineimap.banner)
def connecting(s, hostname, port): def connecting(s, hostname, port):
if s.verbose < 0: """Log 'Establishing connection to'"""
return if s.verbose < 0: return
if hostname == None: displaystr = ''
hostname = '' hostname = hostname if hostname else ''
if port != None: port = "%d" % port if port else ''
port = ":%s" % str(port) if hostname:
displaystr = ' to %s%s.' % (hostname, port) displaystr = ' to %s:%s' % (hostname, port)
if hostname == '' and port == None: s._msg("Establishing connection%s" % displaystr)
displaystr = '.'
s._msg("Establishing connection" + displaystr)
def acct(s, accountname): def acct(self, account):
if s.verbose >= 0: """Output that we start syncing an account (and start counting)"""
s._msg("***** Processing account %s" % accountname) self.acct_startimes[account] = time.time()
if self.verbose >= 0:
self._msg("*** Processing account %s" % account)
def acctdone(s, accountname): def acctdone(self, account):
if s.verbose >= 0: """Output that we finished syncing an account (in which time)"""
s._msg("***** Finished processing account " + accountname) sec = time.time() - self.acct_startimes[account]
del self.acct_startimes[account]
self._msg("*** Finished account '%s' in %d:%02d" %
(account, sec // 60, sec % 60))
def syncfolders(s, srcrepos, destrepos): def syncfolders(self, src_repo, dst_repo):
if s.verbose >= 0: """Log 'Copying folder structure...'"""
s._msg("Copying folder structure from %s to %s" % \ if self.verbose < 0: return
(s.getnicename(srcrepos), s.getnicename(destrepos))) self.debug('', "Copying folder structure from %s to %s" % \
(src_repo, dst_repo))
############################## Folder syncing ############################## Folder syncing
def syncingfolder(s, srcrepos, srcfolder, destrepos, destfolder): def syncingfolder(s, srcrepos, srcfolder, destrepos, destfolder):
@ -284,12 +293,11 @@ class UIBase:
s.getnicename(dr), s.getnicename(dr),
df.getname())) df.getname()))
def copyingmessage(self, uid, src, destfolder): def copyingmessage(self, uid, num, num_to_copy, src, destfolder):
"""Output a log line stating which message we copy""" """Output a log line stating which message we copy"""
if self.verbose >= 0: if self.verbose < 0: return
self._msg("Copy message %d %s[%s] -> %s[%s]" % \ self._msg("Copy message %s (%d of %d) %s:%s -> %s" % (uid, num,
(uid, self.getnicename(src), src, num_to_copy, src.repository, src, destfolder.repository))
self.getnicename(destfolder), destfolder))
def deletingmessage(s, uid, destlist): def deletingmessage(s, uid, destlist):
if s.verbose >= 0: if s.verbose >= 0:
@ -400,14 +408,14 @@ class UIBase:
"""Sleep for sleepsecs, display remainingsecs to go. """Sleep for sleepsecs, display remainingsecs to go.
Does nothing if sleepsecs <= 0. Does nothing if sleepsecs <= 0.
Display a message on the screen every 10 seconds. Display a message on the screen if we pass a full minute.
This implementation in UIBase does not support this, but some This implementation in UIBase does not support this, but some
implementations return 0 for successful sleep and 1 for an implementations return 0 for successful sleep and 1 for an
'abort', ie a request to sync immediately. 'abort', ie a request to sync immediately.
""" """
if sleepsecs > 0: if sleepsecs > 0:
if remainingsecs % 10 == 0: if remainingsecs//60 != (remainingsecs-sleepsecs)//60:
s._msg("Next refresh in %d seconds" % remainingsecs) s._msg("Next refresh in %.1f minutes" % (remainingsecs/60.0))
time.sleep(sleepsecs) time.sleep(sleepsecs)
return 0 return 0