Fetchmail (Debian)

This article describes different use cases for Fetchmail as well as how they can be implemented with Dovecot IMAP Server on Debian 10.

Introduction

There are four major use cases for downloading mails with Fetchmail:

  • Download manually via IMAP or POP protocol
  • Download periodically via IMAP or POP protocol by daemon
  • Download periodically via IMAP or POP protocol by cronjob
  • Download instantly via IMAP IDLE connection

IMAP IDLE can be used by default only for one account. There is a script on the internet which creates multiple instances which enables the administrator to create multiple IMAP IDLE connections each to a different account. It's use and how to alter it to work on Debian is described below.

Install Fetchmail on Debian

Install:

apt-get install fetchmail

Locations and Permissions of Configuration Files

File Content Owner Permissions
/etc/fetchmailrc System config fetchmail:root 600
~/.fetchmailrc User config user:user 600
~/.netrc Passwords for user config user:user 600

As configuration files contain user names and passwords their permissions should be set to 600.

Manual Download

Download INBOX to Default Inbox

Start Fetchmail in user context either using user config with:

fetchmail

Or with a specific config:

fetchmail -f </path/to/specific/fetchmailrc-config-file>

Example for ~/.fetchmailrc which downloads all mails from INBOX folder at the specified host:

poll host.domain.tld protocol IMAP
     user "<username@host.domain.tld>" with pass "<password>" is "<username@localhost>" here
     ssl fetchall

Remember to restrict permissions for that file in order to prevent other users to see your password.

There is also another variant of storing passwords which is described in section Passwords in Configuration Files

Download Selected Remote Folder to Specific Local Folder

An email fetched by fetchmail will be delivered to SMTP port 25 on the local machine by default. That prevents a specification to which local folder an email should be delivered to. In order do deliver emails from a selected remote folder to a specific local folder one has to use the MDA parameter.

When using the MDA parameter the MTA for SMTP running at port 25 will be bypassed and the email will be directly handled by the specified MDA.

Example for delivering emails from the remote folder called "Sent" to the local folder called "Sent":

poll host.domain.tld protocol IMAP
     user "<username@host.domain.tld>" is "<username@localhost>" here
     folder Sent
     ssl fetchall
     mda "/usr/lib/dovecot/deliver -d <username@localhost> -m Sent"

Periodical Download

Fetchmail Running as Daemon

Enable Daemon in config file /etc/default/fetchmail by changing the following line to:

START_DAEMON=yes

In the system specific config file /etc/fetchmailrc a polling interval in seconds will be set for the daemon with "set daemon". Following that all mailboxes will be specified. It is possible to skip a interval for a mailbox by specifing the "interval" parameter followed by a number which says how many intervals should be skipped. Example:

set daemon 60

poll host1.domain.tld protocol IMAP interval 3
     user "<username@host1.domain.tld>" with pass "<password>" is "<username@localhost>" here
     ssl fetchall
     smtp localhost

poll host2.domain.tld protocol IMAP interval 10
     user "<username@host2.domain.tld>" with pass "<password>" is "<username@localhost>" here
     ssl fetchall
     smtp localhost

Fetchmail Running as Cronjob

Fetchmail can also be run in user context by specifing a user specific cronjob with crontab -e

Content can be somewhat like this for running the user specific config every day at 2:30 am:

30 2 * * * fetchmail

or like this for running a specific config:

30 2 * * * fetchmail -f </path/to/specific/fetchmailrc-config-file>

For details see section Manual Download as well as Cronjob (Debian)

Immediate Download via IMAP IDLE

Single IMAP IDLE Instance for Single Account

IMAP IDLE can be used in the system config /etc/fetchmailrc in order to have it active after a reboot. Example:

poll host.domain.tld protocol IMAP
     user "<username@host.domain.tld>" with pass "<password>" is "<username@localhost>" here
     idle
     ssl

Hint: This works only with one Fetchmail instance and therefore only one IMAP IDLE connection per server as the connection persists as soon as it was started.

Workaround: In order to have multiple Fetchmail instances with multiple IMAP IDLE connections for different mailboxes each Fetchmail instance has to have it's own process ID:

Multiple IMAP IDLE Connections for Multiple Mailboxes

Source: http://fnxweb.com/blog/2012/07/14/using-multiple-fetchmail-instances-for-instant-gratification/

The description above is altered to match Debian folders and security.

Files:

http://www.fnxweb.com/data/fetchmail-service --> Save as fetchmail-imap-idle at /usr/local/bin

http://www.fnxweb.com/data/fetchmail.service --> Save as fetchmail-imap-idle.service at /usr/local/bin

Add execute permissions:

chmod +x /usr/local/bin/fetchmail-imap-idle

Create symbolic link to fetchmail-imap-idle.service in /etc/systemd/system:

ln -s /usr/local/bin/fetchmail-imap-idle.service /etc/systemd/system/fetchmail-imap-idle.service

Place config files at: /etc/fetchmail.conf.d/*.conf

Each config file should contain only one IMAP connection.

Example /etc/fetchmail.conf.d/my-imap-account1-inbox.conf

poll host.domain.tld protocol IMAP
     user "<username@host.domain.tld>" with pass "<password>" is "<username@localhost>" here
     idle
     ssl

Avoid parameter fetchall in IMAP IDLE config files as this leads to somewhat intransparent behavior of the local dovecot LDA and emails don't appear at once rather than somewhat delayed.

Fix: Try to send another mail to remote mailbox. After that it seems that all missing mails appear magically in Dovecot mailbox.

Alter /usr/local/bin/fetchmail-imap-idle:

Change lines 10 and 11 as follows:

fetchmailuser=fetchmail
fetchmailgroup=nogroup

Alter paths and executable names in /etc/systemd/system/fetchmail.service as follows:

[Unit]
Description=A remote mail retrieval and forwarding utility
After=network-online.target sendmail.service 

[Service]
ExecStart=/usr/local/bin/fetchmail-imap-idle start
ExecReload=/usr/local/bin/fetchmail-imap-idle restart
ExecStop=/usr/local/bin/fetchmail-imap-idle stop
Type=forking
GuessMainPID=no

[Install]
WantedBy=multi-user.target

Activate service as follows:

systemctl enable fetchmail-imap-idle
systemctl start fetchmail-imap-idle

Service can be removed if necessary with:

systemctl stop fetchmail-imap-idle
systemctl disable fetchmail-imap-idle

Initial Download of Entire Mailbox

First download of an entire mailbox should be done via manual downlod rather than via IMAP IDLE with fetchall.

For further information see Manual Download

Further Information

Passwords in Configuration Files

Parameter with pass <password> can be removed from file ~/.fetchmailrc for security reasons if the file ~/.netrc exists in the user profile. It contains passwords and will be read by fetchmail at runtime. Permissions for this file see section Locations and Permissions of Configuration Files

This does not work with system config file /etc/fetchmailrc

Example for ~/.netrc:

machine <host1.domain.tld>
login <username@host1.domain.tld>
password <pass>

machine <host2.domain.tld>
login <username@host2.domain.tld>
password <pass>

Keep Emails on Source Server

Edit config file and replace parameter fetchall with keep.

Use of Additional Files

fetchmail -f <config-filename>        # Specify config file
fetchmail --pidfile <pidfile>         # Specify process ID file

Remove Unseen Flag in Sent Folder

When using Fetchmail to download Sent folder every downloaded mail will be flagged as unseen.

In order to mark them seen create following cronjob with crontab -e.

*/3 * * * * doveadm flags add -u <username@localhost> '\Seen' mailbox Sent unseen 

Sources

IMAP IDLE: http://fnxweb.com/blog/2012/07/14/using-multiple-fetchmail-instances-for-instant-gratification/

Manpage: http://www.fetchmail.info/fetchmail-man.html