How email really works
Pauls: Ahlquist & Visscher
hackers of sorts
Press "T" to toggle the outline view which contains
additional information.
Email is…
- old
- misunderstood
- trusted too much
- abused
- has little to do with the from: & to:
seen in your mail client
-why most folks have little clue on spam
[any material that should appear in print but not on the slide]
What we will cover
- basic SMTP exchange/protocol at each hop
- how spammers & phishers abuse/forge mail
- simple LDA
- message structure
- POPage, "bring it home"
MIME goodies are touched on in a bonus section authored by paulv.
Some Acronyms
Common acronyms used when talking email--the RFCs are rife with these.
- MX
- Mail eXchanger
A server that handles mail.
- MUA
- Mail User Agent
Your email client application. Examples: mutt, elm, pine, Thunderbird,
kmail, Evolution and to a limited extent Outlook.
- MTA
- Mail Transfer Agent
A mail relay. A server that shuffles mail from location to location.
- MSA
- Mail Submisison Agent
A mail server that is speciffically tasked with receiving new mail messages.
- LDA
- Local Delivery Agent
An application that places mail into the user's mail box.
- DSN
- Delivery Status Notification
Result codes used by mail servers.
"Standards"
RFCs
These are just the starting points
RFC: Request for Comment. These are the standards drafts that form the
consensus specifications of the internet. The RFCs for mail start with the basic
roots above and build into an exquisitely detailed branched tree.
A full collection of RFCs can be found at
Ohio State.
Tour Map
We will follow a mail message from sender to server to server...
| paulv @ |
|
pea @ |
|
|
|
|
|
|
^ |
| |
|
|
|
|
| |
v |
|
| canonical.org |
|
inet |
> |
dma.org |
|
| |
|
User --> SMTP relay [0]
First, I fire up telnet or netcat (cat for sockets)
to connect to the SMTP relay. This is the same exchange as your mail client
or local mail relay would use.
Open a connection to the server and start the exchange
Purple is what paulv types. This is the client side. Yellow is the
server's response.
- pea@birch $ nc canonical.org smtp
This is slightly simplified.
The domain and port (canonical.org smtp) are not exactly
as they would be entered at the command line. In practice, we would need
to determine and specify the server that handles mail for canonical.org.
This is often at a different address than canonical.org proper.
"smtp" is normally port 25--that is, normally.
- 220 panacea.canonical.org ESMTP Postfix Canonicality is not a panacea
canonical acknowleges our connection and then we continue with the exchange.
- helo ahlquist.org
- 250 panacea.canonical.org
- mail from: <pea@ahlquist.org>
- 250 Ok
- rcpt to: <pea@dma.org>
- 554 >pea@dma.org<: Relay access denied
My intention was to create the examples for this set of slides by pretending
to be paulv, pushing a message into canoncal and catching copies as they moved
through DMA's servers. Well, that didn't work, I'm not allowed to relay from
canonical. Canonical is not an open relay (a mail server that takes all comers).
User --> SMTP relay [1]
So, paulv will have to start the example mail himself. He has privledges on
panacea.canonical.org. For those thinking ahead: no AUTH is shown here
just to keep this example very simple.
A simple submission:
- $ paulv@schrodinger $ nc canonical.org smtp
- 220 panacea.canonical.org ESMTP Postfix Canonicality is not a panacea<
- helo schrodinger.canonical.org
- 250 panacea.canonical.org
- mail from: <paulv@canonical.org>
- 250 Ok
- rcpt to: <pea@dma.org>
- 250 Ok
User --> SMTP relay [2]
Additional recepients are entered one by one....
- rcpt to: <some-one-else@example.com>
- 250 Sender OK
- rcpt to: <yet-another-someone@example.com>
- 250 Sender OK
…and so on (within reason)
There are no extra recepients; this just illistrates where they would be
entered.
User --> SMTP relay [3] data
Notice that the data section contains whatever
you feel like entering. To: ___; From: ___; Subject: ___;
Date: ____; are just courtesies extended to those downline.
Many relays will move the mail on without any of these lines being
present. Lesson: don't blindly trust the To and From lines.
- > data
- 354 Enter mail, end with "." on a line by itself
-
Date: Wed, 14 Feb 2007 21:01:26 -0500
From: Paul Visscher <paulv@canonical.org>
To: Paul Ahlquist <pea@ahlquist.org>
Subject: LinuxSig-SMTP talk
Message-ID: <20070215020126.GA30178@canonical.org>
User-Agent: Mutt/1.5.9i
We should totally have candy at the meeting. It'll be great.
--paulv
- .
- 250 2.0.0 l14LPbNQ012741 Message accepted for delivery
- quit
- 221 panacea.canonical.org closing connection
Data section
Let's take another look at the data section. This is important-- the whole
data section is entered by the client. They can put anything they desire here.
This information is a courtesy
Date: Wed, 14 Feb 2007 21:01:26 -0500
From: Paul Visscher <paulv@canonical.org>
To: Paul Ahlquist <pea@ahlquist.org>
Notice the To: Paul Ahlquist <pea@ahlquist.org>.
The message is actually being sent to pea@dma.org. This shows the
insignificance of the data section To:/From: information.
even the message body is optional
We should totally have candy at the meeting. It'll be great.
--paulv
What did this show?
- plain text
- pattern:
- HELO
- MAIL FROM
- RCPT TO
- DATA
- Numeric server response codes (DSN)
- Data section:
to: from: subject: date: are data, not header!
Are you beginning to see how a spammer or phisher can fake
all sorts of things in the email?
Server Respose Codes
- 2xx --> OK, normal
- 4xx --> error: temporary. try again later
5xx --> error: permanent. too bad, so sad
- no unauthorized relaying
- no such user
- no spam from you
This type error will cause an immediate bounce. There is no reason
to queue the message because whatever went wrong is not expected to
get better in a typical queuing period.
see
RFC-1893
for the details
Server admins may display some creativity in the human readable response
messages, but the response codes--the ^[245]\d\d--will be
consistant.
Progress Map : accross the 'Net
On the the next hop, accross the internet…
| paulv @ |
|
pea @ |
|
|
|
|
|
|
^ |
| |
|
|
|
|
| |
v |
|
| panacea.canonical.org |
|
inet |
> |
kathalla.dma.org |
|
| |
|
SMTP relay-->destination [1]
This exchange is much like the initial injection of the mail above:
HELO, MAIL FROM, RCPT TO
[do MX lookups and open the connection...]
To move the mail to the destination, the MX of dma.org needs to be
determined. This is kathalla.dma.org. This is the server the mail
will be move to next.
- a long pause
If you connect to kathalla you will discover that this pause is on
the order of 20-30 seconds.
- 220 kathalla.dma.org ESMTP Sendmail 8.13.4/8.13.4/Debian-3sarge3; Sun, 4 Feb 2007 16:25:37 -0500; (No UCE/UBE) logging access from: panacea.canonical.org. [66.193.87.113]
- ehlo canonical.org
-
The ENVELOPE
The "MAIL FROM" and the "RCPT TO" are the
envelope sender/receiver. "RCPT TO" is who will
actually get the message; "MAIL FROM" is who will get the
bounce. Notice that is "who will get the bounce" not "who
sent";this, too, could be forged information.
- 250 kathalla.dma.org Hello panacea.canonical.org. [66.193.87.113], pleased to meet you
- mail from: <paulv@canonical.org>
- 250 2.1.0 <paulv@canonical.org>... Sender ok
- rcpt to: <pea@dma.org>
- 250 2.1.5 <pea@dma.org>... Recipient ok
If there were any, what happened to the additional recepients?
They were relayed on to "example.com". in a separate exchange
just like this one with "dma.org".
SMTP relay-->destination [2] Data
The data section has grown a bit, it collects an extra line.
The extra line shows information about the last server passed.
- data
- 354 Enter mail, end with "." on a line by itself
-
Received: by panacea.canonical.org (Postfix, from userid 1001)
id 61571E340B8; Wed, 14 Feb 2007 21:01:26 -0500 (EST)
Above is the extra line describing the message's passge through
panacea.canonical.org
From: Paul Visscher <paulv@canonical.org>
To: "Paul Ahlquist" <pea@ahlquist.org>
Subject: LinuxSig-SMTP talk
Message-ID: <20070213123456.GB5973@canonical.org>
User-Agent: Mutt/1.5.9
We should totally have candy at the meeting. It'll be great.
--paulv
- .
- 250 2.0.0 l1F21II2031491 Message accepted for delivery
- quit
- 221 kathalla.dma.org closing connection
What happened on the iNet hop
Progress Map : Last Stop
We find that there is another step to go....
dma.org uses an outside MX that communicates with the internet and an internal
server, dmapub, that handles the user's mail box.
| paulv @ |
|
pea @ |
|
|
|
|
|
|
^ |
| |
|
|
|
|
|
| |
|
|
|
dma.org |
|
|
|
|
| |
v |
| panacea.canonical.org |
|
inet |
> |
|
kathalla |
|
|
|
> |
|
dmapub |
|
|
| |
|
|
|
|
|
|
|
|
|
|
The mail will now work its way to LDA, the Local Delivery Agent, procmail.
Last Stop…
- same exchange repeats kathalla-->dmapub
…aaaand yet another line is added to the data section. This line shows
the mail's passage through kathalla
- another line is added to the header
- Sendmail on dmapub pipes message to procmail
- procmail stuffs (appends) pea's mbox
Getting your mail: POP3
- Post Office Protocol, version 3
- TCP port 110
- plain text
- RFC 1939
Successful Authentication
$ nc localhost 110
- +OK
- USER pea
- +OK
- PASS my_sekret_password_in_cleartext
- +OK
Failed Authentication
$ nc localhost 110
- +OK
- USER pea
- +OK
- PASS the_wrong_password
- -ERR Authentication Failed (invalid password?)
Connection closed by remote host.
Look at the messages we have
- LIST
- +OK
1 414
2 416
3 418
4 994
.
- 4 messages, 2242 bytes (414 + 416 + 418 + 994)
- Why?
- Receiving message 1 of 4 (2 seconds remaining)
This allows the MUA, your mail client, to display how much mail is available
and, perhaps, generate a progress meter.
Download one of the messages
- RETR 4
- +OK
Return-Path: <paulv@canonical.org>
Received: from kathalla.dma.org (kathalla.dma.org [64.56.115.34])
by dmapub.dma.org (8.12.11.20060308/8.12.11) with ESMTP id
l1F21laL027154 for <pea@dma.org>; Wed, 14 Feb 2007 21:01:48
-0500
Received: from panacea.canonical.org (panacea.canonical.org [66.193.87.113])
by kathalla.dma.org (8.13.4/8.13.4/Debian-3sarge3) with ESMTP id l1F21II2031491
for <pea@dma.org>; Wed, 14 Feb 2007 21:01:39 -0500
Received: by panacea.canonical.org (Postfix, from userid 1001)
id 61571E340B8; Wed, 14 Feb 2007 21:01:26 -0500 (EST)
Date: Wed, 14 Feb 2007 21:01:26 -0500
From: Paul Visscher <paulv@canonical.org>
To: Paul Ahlquist <pea@ahlquist.org>
...continues...
Download one of the messages
- Subject: LinuxSig-SMTP talk
Message-ID: <20070215020126.GA30178@canonical.org>
Mime-Version: 1.0
Content-Type: text/plain;
charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.9i
X-UID: 120197
We should totally have candy at the meeting. It'll be great.
--paulv
.
Delete a message
- DELE 4
- +OK
- LIST
- +OK
1 414
2 416
3 418
.
POP3: undelete
- RSET
- +OK
- LIST
- +OK
1 414
2 416
3 418
4 994
.
- Only works in a given session; when you disconnect, the message is gone!
Server Graph vs Messaage Headers
Colors on graph corrispond to highlights on
the message.
Open the message linked above (it will open in a new window/tab).
Compare the highlights on the message with this graph. The colors
corrispond to which server added that part of the message.
| paulv @ |
|
pea @ |
|
|
|
|
|
|
^ |
| |
|
|
|
|
|
| |
|
|
|
dma.org |
|
|
|
|
| |
v |
| panacea.canonical.org |
|
inet |
> |
|
kathalla |
|
|
|
> |
|
dmapub |
|
|
| |
|
|
|
|
|
|
|
|
|
|
The only past you can really trust are the blue sections. The rest is
in the data section of the mail--it can be forged. The added header lines are a
courtesy provided by the servers along the way.
Standards compliant mail servers adhere to the courtesy of providing valid tracking
headers …and a normal user mail client will properly provide rational To: From:
Date: and Subject:information.
Final message
Return-Path:
Received: from kathalla.dma.org (kathalla.dma.org [64.56.115.34])
by dmapub.dma.org (8.12.11.20060308/8.12.11) with ESMTP id l1F21laL027154
for ; Wed, 14 Feb 2007 21:01:48 -0500
Received: from panacea.canonical.org (66-193-87-113.static.twtelecom.net [66.193.87.113])
by kathalla.dma.org (8.13.4/8.13.4/Debian-3sarge3) with ESMTP id l1F21II2031491
for ; Wed, 14 Feb 2007 21:01:39 -0500
Received: by panacea.canonical.org (Postfix, from userid 1001)
id 61571E340B8; Wed, 14 Feb 2007 21:01:26 -0500 (EST)
Date: Wed, 14 Feb 2007 21:01:26 -0500
From: Paul Visscher
To: Paul Ahlquist
Subject: LinuxSig-SMTP talk
...continues...
Final message
Message-ID: <20070215020126.GA30178@canonical.org>
Mime-Version: 1.0
Content-Type: text/plain;
charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.9i
X-UID: 120197
We should totally have candy at the meeting. It'll be great.
--paulv
FIN
Questions?
Questions can be posted to the DMA LinuxSig mailing list. Of course, you will
need to be subscribed
Try it yourself
If you have a mail account on a smaller mailserver you can try sending yourself
a message. Just repeat the early SMTP exchange from this presentation (where paulv
first posts the message) making appropriate changes in the addresses. A dmapub,
account, which is available to all DMA members, is perfect for this.
How do we send binary files?
Bonus material
paulv put together some extra material not presented at the LinuxSig meeting. It
includes info on sending attachments, SMTP relay authorization and and secure auth.
- We said SMTP and POP3 are plain text
- So how would you send binary files?
- Binary files can't always be represented in printable ASCII
- We need to encode it somehow
MIME
- Multipurpose Internet Mail Extensions
- RFC 2045
- The MIME RFC defines base64 as an acceptable encoding
- base64 is a way to represent 0-64 with A-Z, a-z, 0-9, +, /, and =
Example
- A 5x5 blue gif in hex:
47 49 46 38 37 61 05 00 05 00 80 01 00 00 00 ff
ff ff ff 2c 00 00 00 00 05 00 05 00 00 02 04 84
8f a9 58 00 3b
- Base64 encoded:
R0lGODdhBQAFAIABAAAA/////ywAAAAABQAFAAACBISPqVgAOw==
- Send this as the attachment. The MUA will then know how to decode it.
Authenticated SMTP
- What if we want to relay mail from trusted users from arbitrary IP addresses?
- SMTP Auth + Simple Authentication and Security Layer (SASL)
- SMTP Auth - RFC 2554
- SASL - RFC 4422
SASL AUTH
- $ nc localhost 25
- 220 schrodinger ESMTP Postfix (Ubuntu)
- EHLO planck
- 250-planck
250-AUTH LOGIN PLAIN
- AUTH PLAIN
- 334
- AHBhdWx2AG15X3Nla3JldF9wYXNzd29yZF9pbl9jbGVhcnRleHQ=
- 235 Authentication Successful
- mail from: <paulv@canonical.org>
- 250 Ok
- [...]
SMTP Auth
- What is AHBhdWx2AG15X3Nla3JldF9wYXNzd29yZF9pbl9jbGVhcnRleHQ= ?
- base64 for "\0paulv\0my_sekret_password_in_cleartext"
- BASE64 IS NOT ENCRYPTION!
- This is why we combine AUTH PLAIN with STARTTLS!
- STARTTLS is SMTP over TLS.
- RFC 3207
FIN
Questions?
Questions can be posted to the DMA LinuxSig mailing list. Of course, you will
need to be subscribed