[Maia-users] lost connection after CONNECT (solved?)

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Sims wrote:

>... because of the wait states associated with the TCP
> connections.... i.e., if you have more traffic than available listening
> processes to deal with it, the traffic doesn't seem to serialize too
> well... I am concerned with the idea of having lots of smtp listeners and
> few amavis processes though...

The usual advice you'll receive is to match the number of SMTP listeners
with the number of amavisd-maia listeners, and I believe the docs
suggest exactly that.  As you've discovered, however, there are some
circumstances in which it can make sense to scale these two parameters
independently.

First, though, it's instructive to understand the reasoning for locking
the parameters together in the first place.  The theory is that every
email being received needs service from one SMTP listener and one
amavisd-maia listener, so having more of one than the other is
potentially a waste of resources.  You certainly don't want more
amavisd-maia listeners than SMTP listeners, for example, since those
excess amavisd-maia listeners could never receive any connections and
would just take up memory needlessly.

That reasoning is oversimplified somewhat, though.  It assumes, for
instance, that every connection to an SMTP listener is going to result
in a connection to an amavisd-maia listener.  If you use your SMTP
listeners to reject mail, though--say with DNSBLs, or based on some
other criteria (e.g. invalid recipients, greylisting, etc.)--then that
assumption is no longer strictly valid, as you've discovered.  This is
also true during DDoS attacks on your SMTP listeners--i.e. attempts to
initiate connections without ever completing them.  When this happens,
you need more SMTP listeners than amavisd-maia listeners, because some
of those SMTP listeners will be busy handling the "nuisance" connections.

What it comes down to, then, is this: what percentage of the incoming
SMTP connections is actually mail that amavisd-maia needs to process?
Or, to invert the question, what percentage of the connections to your
SMTP listeners are "nuisance" connections?  You could presumably monitor
and measure this sort of thing from your Postfix logs and come up with a
reasonable number, but even so, during something like a DDoS attack this
is going to rise considerably, then drop down again after the attack
subsides.

Right now, in other words, at the height of the DDoS attack you're
weathering, you've found that 50 SMTP listeners are required in order to
let 5 valid connections get through to your 5 amavisd-maia listeners.
The other 45 SMTP listeners are presumably busy handling the bogus
connections from the botnet (i.e. there are probably 45 machines
involved in this attack).  If the attack intensifies you may need to
increase that figure even more, but you shouldn't need to increase the
number of amavisd-maia listeners (unless of course your volume of
non-nuisance SMTP connections is also increasing).

Coming up with a hard-and-fast metric isn't easy, alas, because even
among the nuisance connections the delays vary widely.  Rejecting based
on invalid recipients, for example, happens early in the SMTP
transaction (at the RCPT TO stage), and is very fast, so that kind of
nuisance connection doesn't really take an SMTP listener out of service
long enough to impact mail throughput.  On the other hand a DDoS attack
is like a reverse tarpit, in that it keeps an SMTP listener waiting, and
waiting, and waiting...until it finally times out and gives up.  That
most definitely takes an SMTP listener out of service long enough to
impact throughput.  That's why scaling up the number of SMTP listeners
works to help mail get through in that case--for every concurrent fake
connection attempt the attackers send at you, you need to sacrifice one
SMTP listener, while having as many additional SMTP listeners as you
need to process the non-nuisance connections you would ordinarily be
receiving.  Or if you prefer an analogy from (American) football, you
need to have enough blockers to match the other team's tacklers, in
order to free up your quarterback and receiver to make their connecting
pass.

Obviously there are limits to this; if the botnet targeting you had
10,000 machines, you couldn't reasonably run 10,005 SMTP listeners in
order to weather the storm (if you had any bandwidth left at that point
anyway).  When you can no longer manage an effective defense on your
own, it's time to turn to your ISP for help--if for no other reason than
to make sure they don't bill you for the traffic! :)

In the end, I suppose the rule of thumb ought to be that you must run at
/least/ as many SMTP listeners as amavisd-maia listeners.  Under normal
circumstances (i.e. when not under a DDoS attack) you may benefit from
running a few additional SMTP listeners if you do a lot of rejections,
but you shouldn't need many more--certainly no more than double the
number of amavisd-maia listeners.  In an attack situation you should
feel free to temporarily scale this higher, since the number of nuisance
connections is dramatically higher.  After the attack is over you should
reduce the number of SMTP listeners to its normal level, to avoid
wasting resources unnecessarily.  There's no need to change the number
of amavisd-maia listeners in any case, unless the volume of non-nuisance
connections changes.

- --
Robert LeBlanc <rjl at renaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.maiamailguard.com/>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)

iD8DBQFGu2ILGmqOER2NHewRAkKIAKCOs9PjXyw/NIVEOq6CELihpI7xxACgk6aQ
yczu0yshPtbJIQ8cFv4EtX4=
=+4pt
-----END PGP SIGNATURE-----