Main e-mail protocols: SMTP, POP, and IMAP

Why are we discussing basic network communication protocols in this book? Are we not running advanced software? Indeed we are, but knowing one's way around the protocols cannot only assist debugging a possibly non-working system but also increases the understanding of a mail system's behavior. We will start with a rather non-technical overview of the protocols, after which we will focus on the protocol details.

Overview

In the UNIX environment, traditional mail applications did not use any network protocol at all. They have instead accessed the locally stored mailbox files directly through the file system. Typically, the inbox of each user is stored in a single file in either the /var/mail or the /var/spool/mail directory with the same name as that of the user (for example, /var/spool/mail/joe). The focus of this book is to discuss Linux based e-mail solutions for a small office where users do not wish to log on to a central server with a terminal application in order to access their mail, so local mail storage will be covered only briefly.

The most important protocol in Internet mailing is the Simple Mail Transfer Protocol (SMTP). Its purpose is to transport e-mail messages between two systems. Both these computers may either be servers, or one of them may be a client machine on which the user runs the mail application-Outlook, Thunderbird, Eudora, or whatever. To collect new messages, the end user does not utilize SMTP. This is where the Post Office Protocol (POP) and the Internet Message Access Protocol (IMAP) come in.

Some proprietary systems such as Microsoft Exchange and Lotus Notes use their own protocols to access messages, and we will not discuss them here.

POP protocol

POP is the older and more widely used protocol of the two. It focuses on giving the users access to their inboxes, from which the users can download the new messages to their local computers and then delete them from the server. POP servers are not meant to be used for permanent storage of messages. The POP services of some Internet providers even prohibit users from leaving messages on the server after they have been downloaded once. The chief disadvantage of POP is that it only provides an intermediary storage medium and the users must store their messages permanently someplace else (for example, on their local hard drives). This is not only impractical for users who want to access their e-mail messages from multiple locations, but it is also a hassle for the System Administrator who may have to implement a backup solution for the users' messages on their local hard drives. POP also does not have any notion of providing multiple folders for every user; with POP a user can access his/her inbox only.

IMAP protocol

IMAP is meant as an access method to a first class mail store, that is, it is designed to allow the user to store the messages permanently on the server. This solves the System Administrator's backup problem and allows the user to access all messages from any place in the world (firewall restrictions aside). IMAP also has a more widespread implementation of TLS-secured connections, making IMAP safe to use in hostile environments. To improve performance and allow users to work with their mailboxes while not being connected to the mail server, most mail applications with IMAP support caching the downloaded mailboxes and messages in the local hard drive.

Unlike POP, IMAP supports multiple folders and stores message state information (whether or not the message has been read, replied to, or deleted) on the server. This means that a user accessing their message store from different locations, with possibly different e-mail clients, will be presented with a consistent, up-to-date view of their messages. IMAP also supports server-side searching, so the client application does not need to download all the messages to search for an e-mail.

The SMTP protocol

SMTP is a line-oriented text protocol that runs over TCP, which makes it trivial to decode SMTP transcripts and to initiate SMTP sessions using the regular telnet client found on just about any computer. An SMTP client starts a session by connecting to port 25 on the SMTP server. After the server has greeted the client, the client must respond by saying hello, or actually HELO or EHLO, followed by the client's hostname. If the server accepts the cordial greeting, the client may begin the first mail transaction.

An SMTP mail transaction consists of three parts-a sender, one or more recipients, and the actual message contents. The sender is specified with the MAIL FROM command, each recipient with an RCPT TO command, and the start of the message contents with a DATA command. If the server accepts the message, the client may continue with additional transactions or issue the QUIT command to terminate the SMTP session.

Let's be less abstract and look at an actual SMTP session to illustrate the protocol. The bold face print represents what the client sends to the server.

This example shows a host that claims to be named gw.example.net connecting to an SMTP server that calls itself mail.example.com. Because the server's first response contains ESMTP, the client decides to try Enhanced SMTP (ESMTP) and greets the server with EHLO instead of HELO. The server accepts this greeting and responds with a list of the supported ESMTP extensions.

Together with the sender address, the client sends the SIZE attribute to indicate the size of the message to the server. This is allowed because the server has stated that it supports the SIZE extension. If the size specified by the client exceeds the message size limit set by the server, the message can be rejected at once rather than after the whole message has been received and the server can assess the size.

An SMTP message can obviously have more than one recipient. This has a few consequences that must be remembered while implementing a mail system and inventing policies. In the previous example, the mail server accepts the first two recipients but rejects the third one. As two recipients have been accepted by the server, the client will try to send the message contents. Here the message is accepted by the server and queued for delivery (250 Ok: queued as B059D3C2B), which means that the SMTP server has taken over the responsibility for the delivery of the message to the accepted recipients. If the message cannot be delivered, the server will send a non-delivery message (bounce) back to the sender. The server could also have chosen to reject the whole message. If so, it would have rejected it for all recipients and not delivered it at all. In other words, in response to the message contents the server must either reject the message for all recipients or accept it for all recipients.

It is vital to understand the difference between the envelope and the header. The envelope of a message consists of the information given in the MAIL FROM and RCPT TO commands, that is, the sender and recipient information that are used to deliver the message. An SMTP server pays no attention what so ever to the From, To, and Cc message headers. In our example the To header contains just a single address with no other relation to the actual recipient addresses than the domain, but that is just a coincidence. Bounces are always sent to the envelope sender address, in this case jack@example.net. The sender address of bounce messages is the empty sender address, often called the null sender. However tempting it may be for some people, the null sender address must not be blocked.

So far, we have not commented on the numerical codes given by the server at the beginning of each line. Each number has a specific meaning and it is important to learn the correct interpretation of the first digit.

Digit

Meaning

2

Server has accepted the previous command and is awaiting your next...