excluding subscribed folders

[anarcat]

One of the problems I have migrating from OfflineIMAP is that I restrict the sync to subscribed folders, which makes it much more efficient. It also keeps archives from propagating everywhere and seems like generally good hygiene.

Implementing this with smd seems hard: while there is an EXCLUDE configuration option, it's not very flexible: it's just a static list of files. I guess I could make it looks something like:

EXCLUDE=$(ssh example.net cat Maildir/subscriptions)

... but I doubt that would actually work. In fact, I had trouble making excludes work at all. I was able to exclude the .notmuch/hooks/ directory (otherwise smd would just crash on the symlinks that are there), but not much else. I used to have, for example, a Maildir/Koumbit folder on the remote server. I was never able to make smd ignore that folder.

Maybe that's another bug report, but it makes me feel exclude pattern matching is brittle and hard to implement. I feel it might be better to delegate this to mddif which could parse that file more reasonably...

What do you think?

[gares]

I'm fine with an --exclude-from-file option to mddiff, but I also think the Koumbit problem should be understood. Did you use just EXCLUDE or EXCLUDE_LOCAL/REMOTE?

mddiff does nothing special for exclude, it just calls glib glob pattern matcher IIRC, so it should be possible to reproduce it. Do you mind trying to reproduce the bug by editing tests.d/mddiff/08-exclude ? Then make test runs it.

[anarcat]

i tried EXCLUDE and EXCLUDE_LOCAL/REMOTE. I'll file a different bug report regarding that.

[anarcat]

I'm fine with an --exclude-from-file option to mddiff

So how should I go forward here? I'm a little hesitant in trying to implement such an option considering how much time I've spent just tweaking things so they work at all (and coercing my offlineimap maildirs to be compatible :).

I'm thinking maybe i should just archive those unsubscribed folders out of the way, or sync them with offlineimap once and be done with them... What do you think?

[gares]

I'm thinking maybe i should just archive those unsubscribed folders out of the way

I think this is a good option for you. But re-reading your report I also think that EXCLUDE is not meant for "dynamic data". It was added to let one skip stuff like the .notmuch directory (i.e. a static path), not a dynamic list of subscribed folders. What would be the problem in synchronizing them all (with smd)?

Thanks for the other reports, I'll look into them ASAP.

[anarcat]

the problem is archival: those are very old emails and it's not necessarily relevant to carry them around all the time. they were also excluded because they were slowing down offlineimap quite a bit, but maybe that won't be a problem with SMD? :)

[gares]

I now notice that the data in the README is a bit outdated ;-)

With my laptop (cheap hp with an SSD) and a mailbox of 11G it takes around 2 seconds to smd-pull (the server is a tiny kvm instance, but does not do much, so probably the disk cache is hot).
How large is your mailbox?

[anarcat]

the mailbox is about 4GB with 200k emails with a hopefully hot cache, but i'm not sure it is.

[anarcat]

FWIW, I have disabled any filters on offlineimap's side and just downloaded everything. this is my workstation, so it's reasonably fast and will provide a good testing ground. I'll see what i will do with the laptop once I deploy there...

unfortunately, i'm traveling next week and won't have time to finish this setup by then, but i should resume working on this somewhere in May.

thanks for the support!