Basic Performace Statistics
From MailWasher Server
Benchmarking the MailWasher Server Process
Introduction
This document investigates the effect that the Firetrust MailWasher Server product will have on the performance of an existing mail server. It concentrates on the ability to receive mail messages from the outside world, to deliver them within an organisation, and the added load to system resources that is imposed on the mail server.
Test Environment
All tests described in this document are run on the same hardware platform, in the same environment.
1. The Originator. This comprises a bespoke mail client whose sole purpose is to deliver mails as quickly as possible. So, there’s no fancy stuff here at all. If thew server decides not to accept it for any reason, it just goes on processing.
2. Communications Infrastructure. Client and server are connected via a 100Mbit network running at full duplex. This ensures that there is no possibility of a lack of bandwidth skewing the results.
3. The Recipient. This machine has a removable hard disk, allowing for different configurations / operating systems to be compared on a level playing field.
The client machine for these tests is a Sun V120 server, running Solaris 9. The mail client has been written in C, and is run many times in parallel to ensure a heavy load on the server. The light load that this places on the Sun server ensures that there is no skewing of the results due to a shortfall in resources on the client end.
The server machine is based on industry standard Intel hardware. A 3.0GHz hyperthreaded Pentium P4 CPU, with 512MB of memory, and an 80 GB SATA disk is used to support various versions of linux and Windows Server 2003. The relevant mail servers from Exchange 2003, Sendmail and Qmail are used to receive messages.
The delivery of mail can be thought of as being in two separate modules. First, the incoming message needs to be accepted by the mail server, and then this message needs to be delivered locally to the final recipient. Both the performance of the external mail receipt and the local delivery are monitored in this benchmarking. The choice of backend mail store can greatly affect the speed of local delivery, so the poorest performing option is used.
All benchmarking is necessarily an artificial look at what will happen in reality. No attempt has been made to model real world loadings, rather the aim has been to provide worst-case scenarios. To this end, no attempt has been made to tune the mail server (other than reducing constraints to allow this abnormal load to run in the first place), so it is hoped that real time performance will always be better than these results.
As the client’s mail server hardware will vary greatly, these benchmarks strive to provide an indication of the extra load and performance degradation caused by the installation of the MailWasher Server software on and existing mail server platform.
Static Test Conditions
For each configuration, three x three tests are performed. Mails with a payload of 1k, 10k and 100k are sent to a mail server configured with MailWasher Server accepting all or half the incoming mails, with a check run where MailWasher Server is disabled. Each test is run 3 times, and the results averaged.
The client Sun server is configured to run 50 mail clients concurrently, each sending a total of 200 emails, so 10,000 emails are send in total for each test. Each mail client sends to 5 email addresses on a round-robin basis.
Test Results
1. Debian Linux / Sendmail v.8.13.1
By default, the sendmail server is configured to connect only a maximum of 10 processes concurrently, so the loads used to stress this configuration should never be encountered in the real world.
| Email Size | Spam Level | Accepted | Emails/sec | Delivered | Emails/sec |
| 1KB | 0% | 04:45 | 35.1 | 04:48 | 34.7 |
| 1KB | 50% | 07:41 | 21.7 | 07:47 | 21.4 |
| 1KB | Disabled | 01:59 | 84.0 | 05:04 | 32.9 |
| 10KB | 0% | 04:56 | 33.8 | 06:34 | 29.9 |
| 10KB | 50% | 05:29 | 30.4 | 05:40 | 29.4 |
| 10KB | Disabled | 02:18 | 72.5 | 06:17 | 26.5 |
| 100KB | 0% | 05:32 | 30.1 | 15:39 | 10.6 |
| 100KB | 50% | 08:54 | 18.7 | 12:39 | 13.1 |
| 100KB | Disabled | 02:59 | 55.6 | 12:05 | 13.8 |
Taking the worst case statistics from the table above, email acceptance can be degraded by up to 4 times, but the complete email delivery process by approximately 50%. The degradation in acceptance speed can be thought of as a hit on the ability to handle short-term loads, whereas the latter figure is the performance loss in real terms.
2. Microsoft Windows 2003 / Exchange 2003 / Outlook POP3 clients
[Due to the system performance, the number of messages delivered has been dropped from 10,000 to 1,000… the first trial took 28 minutes to deliver the mail, and over 2 hours to deliver it locally!]
With these benchmarks, it should be noted that the configuration of SATA disk and 512MB memory causes a major disk bottleneck which affects the delivery times of the larger emails.
In an attempt to compare like with like ( well, fruit with fruit at least ), I have set up an exchange server with 5 email addresses, and I’m looking at these through a single outlook 2003 client using pop3 clients.
| Email Size | Spam Level | Accepted | Emails/sec | Delivered | Emails/sec |
| 1KB | 0% | 00:52 | 19.2 | 06:06 | 2.7 |
| 1KB | 50% | 00:49 | 20.4 | 04:07 | 4.0 |
| 1KB | Disabled | 00:53 | 18.7 | 06:02 | 2.8 |
| 10KB | 0% | 00:53 | 18.7 | 06:25 | 2.6 |
| 10KB | 50% | 00:52 | 19.2 | 05:04 | 3.3 |
| 10KB | Disabled | 00:52 | 19.2 | 06:26 | 2.6 |
| 100KB | 0% | 01:20 | 12.5 | 15:08 | 1.1 |
| 100KB | 50% | 01:09 | 14.5 | 09:25 | 1.8 |
| 100KB | Disabled | 01:19 | 12.7 | 13:51 | 1.2 |
The difference in internal design can clearly be seen when comparing the variance in delivery rates between this architecture and that of Linux/Sendmail. The acceptance rate of emails with the Exchange Server receiving the mail shows no difference whether the MailWasher Server conduit is installed or not, and there is a barely noticeable ( less than 10%, and unlikely to be statistically relevant, given that each test was only repeated 3 times ) difference in the delivery time to the ultimate recipient.
3. RedHat Linux 9.0 / qmail 1.05
By default, the qmail server is configured to connect only a maximum of 20 processes concurrently, so the loads used to stress this configuration should never be encountered in the real world. The server was reconfigured to accept the 50 processes loading the system.
| Email Size | Spam Level | Accepted | Emails/sec | Delivered | Emails/sec |
| 1KB | 0% | 03.14 | 51.5 | 06.38 | 25.1 |
| 1KB | 50% | 09:48 | 17.0 | 09:48 | 17.0 |
| 1KB | Disabled | 01:09 | 144.9 | 05:35 | 29.8 |
| 10KB | 0% | 03:18 | 50.5 | 07:07 | 23.4 |
| 10KB | 50% | 11:59 | 13.9 | 11:59 | 13.9 |
| 10KB | Disabled | 01:12 | 138.9 | 06:07 | 23.4 |
| 100KB | 0% | 04:21 | 38.3 | 15:09 | 11.0 |
| 100KB | 50% | 17:05 | 9.8 | 17:05 | 9.8 |
| 100KB | Disabled | 02:21 | 70.9 | 16:02 | 10.4 |
Taking the worst case statistics from the table above, email acceptance can be degraded by up to 10 times, but the complete email delivery process by approximately 50% The degradation in acceptance speed can be thought of as a hit on the ability to handle short-term loads, whereas the latter figure is the performance loss in real terms.
Conclusions
Although this study is in no way an attempt to model real life, it is an attempt to investigate a ‘worst case’ scenario, and get a feel of how MailWasher Server affects the characteristics of a mail server.
The Windows/Exchange installation seems to require more powerful hardware to support the main server function, and disk access bottlenecks masked any potential performance cost that MailWasher Server has.
The linux based installations show a considerable degradation in the ability to accept emails. This will show itself as a lower ability to accept a short term load of incoming emails. However, when measuring the complete email delivery, which includes local deliver to the users’ mailbox, then the performance is cut by about 50%.
The filtering of emails was performed using the statistical analysis feature. When this actually has to quarantine an email, it seems to add a significant delay to the delivery of mail, and this may need to be addressed in the future.
To put the above figures into perspective and taking the worst case delivery rates, the ( unrepresentative ) windows performance relates to over 100,000 emails/day ( or 10GB ), and the equivalent linux figure is over 800,000 emails/day ( or 80GB ). It would take a pretty large company to have a problem with a drop from 200,000 to 100,000 emails per day.
