There are various forms of intrusion attacks on mail servers: there are attacks using buffer overflow vulnerabilities, denial of service attacks, directory collection attacks, etc. Measures such as hardening mail servers, using mail filtering tools, using managed services, and installing integrated software can all stop attacks on mail servers from different sides. This article describes these measures in detail.
Hardening your mail server, first installing a mail filtering network tool in front of it, or using a managed mail filtering service will help mitigate attacks from spammers and other sources.
As attacks against end users and their desktops increase, direct attacks on mail servers have decreased (although this decrease is relative). However, servers are still vulnerable, as attackers continue to find vulnerabilities in Microsoft's EXChange server and even Sendmail. Here's a look at two common attacks and ways to reduce or eliminate your mail server's exposure to these attacks.
One of the root causes: buffer overflow vulnerability
A buffer overflow occurs when a software program, such as mail server software, stores more data in a data buffer than was originally allowed and fails to guard against unexpected input. An attacker could exploit this flaw to cause the mail server to perform other procedures than it intended. If the mail server is running with privileges, the security of the entire system will be compromised. Even if the mail server is not privileged, an attacker can still compromise its security and gain full control over its resources.
Although buffer overflows are caused by accidental programming errors, they are a very common security vulnerability in terms of data integrity. When a buffer overflow occurs, the excess data can contain code designed to trigger specific actions, such as sending new instructions to the compromised server that could corrupt user files, modify data, or expose top-secret information.
Attackers have proven their skills in the past by exploiting buffer overflow vulnerabilities to allow worms to travel between different servers on the Internet. But recently, buffer overflow vulnerabilities have taken on a more specific target. They allow attackers to compromise a mail server, which they can then use to send spam.
This attack has two serious consequences. First, a compromised email server means an attacker can read the company's incoming and outgoing emails. The results can be catastrophic. Second, attackers can use a company's server resources to send spam. This situation can bring a bad name to the company, violate the ISP contract, and often means termination of service.
It is important to harden your mail server (and any other public server) against buffer overflow vulnerabilities and other forms of attacks. There are other protective measures that can be taken.
One response: server hardening
The best way to reduce the chance of your mail server's security being compromised is to harden the mail server itself. In any case, reinforcement is worth the effort. On hardened servers, especially those on the Internet, few services are vulnerable to vulnerabilities, and those services are usually treated "differently." Reinforcement usually requires the following measures:
• Physically secure computers;
• Update operating systems and application software;
• Enable logging to record administrators’ operations on accessing and using resources;
• Remove unnecessary applications, services and tools;
• Enable local firewall service;
• Restrict the use of privileged accounts.
By hardening your servers, your weak points can be greatly reduced. But simply hardening your mail server is often not enough. A better solution would be to harden the server while also providing additional filtering of email traffic before the email actually reaches the server.
Email traffic can be pre-filtered through the use of network tools, management services, and software integrated into existing email systems (such as Microsoft's EXChange). Remember to have different layers of defense—for example, hardening your internal email servers and deploying vendor-hardened network tools to protect the surrounding environment.
Response 2: Network tools
Mail filtering network tools are deployed in front of internal mail servers. These tools typically provide two types of firewalls: packet filtering firewalls and application-level firewalls. Network tools that act as packet filtering firewalls only allow valid TCP/IP traffic to ports used by mail services (such as SMTP, usually POP3 and IMAP). The tool as an application-level firewall ensures that the sending server uses SMTP correctly and follows relevant IEEE Requests for Comments (RFCS) and conventions (e.g., supports reverse DNS settings).
Network tools are not vulnerable to attacks for several reasons. First, the vast majority of tools run on highly customized operating systems. These operating systems have disabled most of the additional services that would allow attackers to gain a foothold (or were customized from the outset specifically for the tools to be used).
Second, engineers strictly adhere to best practices when hardening tools.
Finally, a tool allows only a limited type of communication to and from the mail server (i.e., communication related to mail transport), and even this type of communication is subject to careful inspection.
Response 3: Managed services
With managed services, all email is first sent to an offsite service that filters email, which then forwards valid email to the company's mail server.
To use this strategy to effectively prevent attacks using direct mail protocols, the internal mail server must only accept connections initiated by the managed service and not any other connections. But these services are only available for incoming email communications. Outbound email traffic is still sent directly to other servers on the Internet, thereby activating possible vulnerabilities in the use of email protocols (for example, a receiving email server could exploit a buffer overflow vulnerability in the sending email server software during SMTP transmission).
Response 4: Integrated software
Finally, integrated software can be installed to help protect your mail server. This locally installed software protects against network attacks and makes the server more robust. Integrated software typically runs at the application layer (i.e. SMTP) to protect servers from exploits. Some integration software replaces the server's native TCP/IP stack with a custom hardened version.
However, it is more common for local filtering software to work with the email software rather than building a wall between the email software and the external system. Integrated software using this approach can be useful when an attacker has direct access to the mail server (for example, if a trusted internal user launches the attack).
Response 5: Denial of Service Attacks and Directory Collection Attacks
Denia1 of Service (DoS) attacks reduce the capabilities of the target system. Let's say a mail server, for example, and an attacker is trying to slow it down or disable it. Attackers launch denial-of-service attacks in several ways, including consuming network resources and launching directory harvest attacks.
When an attacker conducts a denial-of-service attack through network resource consumption, the attack often focuses on consuming all available incoming connections to the target machine. Because SMTP is a TCP protocol, a successful exploit only requires that the attacker requests more TCP connections than is available. That is, the attacker creates more connections to the mail server than the mail server can handle. This way the mail server can no longer accept valid incoming connections from legitimate mail servers.
There are few server-based solutions to prevent denial-of-service attacks. Most mail servers run on general-purpose operating systems that are not tuned to protect against denial-of-service attacks. Even on a hardened UNIX system, increasing the server's ability to withstand a large number of denial-of-service attacks requires different network settings. As a result, companies often purchase systems that are specifically created to detect and prevent denial-of-service attacks, or hardened filtering tools that can accept many more simultaneous connections than a general-purpose mail server. Such filtering devices are often better able to detect denial-of-service attacks and take defensive measures.
Directory harvest attacks are resource-intensive attacks launched by spammers to identify valid addresses available for future spam. When a directory collection attack occurs, the load on the mail server will be greatly increased, affecting the transmission of effective mail. In addition, the local mail server will return non-delivery reports for invalid addresses attempting to the From address used by the spammer.
Returning non-delivery reports generates additional outbound email traffic, consuming expensive bandwidth and thus increasing the load on the mail server. Because most From addresses used by spammers are fake, transmission non-delivery reports always time out, requiring the mail server to try the transmission again at a later time. In summary, a directory harvest attack is an expensive form of attack on a mail server.
Unfortunately, there are few ways to mitigate the dangers of directory collection attacks. One solution is to use managed services. Typically managed services maintain many more mail servers than a company can provide, so directory harvest attacks do not impact mail delivery to a large extent.
Another solution is to install front-end filtering tools that are optimized for this type of attack. Maintain a list of legitimate email users in the tool (either via a static list or Light Directory Access Protocol access to an internal directory) so that filters do not send emails to invalid users