Improving SSL Security for Forefront Threat Management Gateway (TMG) 2010 Published Web Sites

Introduction

For many years readers here at ISAserver.org have heard me and many others tout the formidable security that Forefront Threat Management Gateway (TMG) 2010 provides. Forefront TMG 2010 has an impeccable track record of proven reliability and security. It leverages the mature, well understood patch management methodology of the underlying Windows Server operating system, ensuring that vulnerabilities in TMG (extremely rare) and the OS (more common) are mitigated in a timely manner. Out of the box, a default installation of Forefront TMG on a default Windows Server 2008 R2 provides a fairly high level of security for itself and TMG-protected networks. You can further improve the security level of your TMG firewall by performing system hardening and following configuration and administration best practices. One area in which the Forefront TMG firewall seems to fall painfully short is in providing essential protection for SSL published web sites. Special attention is needed in this area in order to ensure that SSL published web sites remain secure and available.

Published SSL Web Site Challenges

On a fully-patched (OS and TMG) firewall with baseline security hardening and attack surface reduction in place, in a default configuration a Forefront TMG 2010 published SSL web site will receive an F rating from the Qualys SSL Server Test web site.

ImageFigure 1

Immediately you’ll notice that this site rates well in terms of the certificate used, along with the key exchange and cipher strength. However, in terms of protocol support there are some very serious issues with this default implementation. To be fair, this is no particular fault of Forefront TMG, as TMG simply leverages the underlying operating system for certificate and cryptographic services. For example, by default Windows Server 2008 R2 supports SSL v2, which is very insecure and should not be used at all. It is also vulnerable to man-in-the-middle (MitM) attacks because it supports insecure SSL renegotiation. It is also susceptible to denial-of-service (DoS) attacks as it supports client-initiated renegotiation. All of these issues need to be addressed before allowing access to Forefront TMG published SSL web sites.

Improving SSL Security

The first step in improving the security of published SSL websites with Forefront TMG is to disable the use of SSL v2. To do this, open the registry, navigate to HKLM\System\CurrentControlSet\Control\SecurityProviders\SChannel\Protocols\SSL 2.0 and create a new key called Server. Under the Server key create a DWORD value called Enabled set to 0. Restart the TMG firewall for this change to take effect.

ImageFigure 2

To address the insecure renegotiation issue, security update MS10-049 must first be installed. Since this security update is now more than three years old, it should already be installed on your TMG firewall (if it is not, you probably have bigger security issues than just weakly protected SSL published web sites!). Even with the update installed, it will be necessary change SChannel’s mode from compatible to strict. To do this, open the registry, navigate to HKLM\System\CurrentControlSet\Control\SecurityProviders\SChannel\ and create a new DWORD value called AllowInsecureRenegoClients set to 0. Restart the TMG firewall for this change to take effect.

ImageFigure 3

At this point, running the SSL server test again will now yield an A rating.

ImageFigure 4

Additional SSL Security Enhancements

Although the changes we’ve so far have significantly improved our security rating, there are a few more things that should be addressed in order to provide the highest level of security for our SSL published web sites. To prevent denial-of-service (DoS) attacks, it’s a good idea to disable client-initiated SSL renegotiation. To do this, open the registry, navigate to HKLM\System\CurrentControlSet\Control\SecurityProviders\SChannel\ and create a new DWORD value called DisableRenegoOnServer set to 1. Restart the TMG firewall for this change to take effect.

ImageFigure 5

It’s also a good idea to enable new protocols such as Transport Layer Security (TLS) v1.1 and v1.2 for modern clients that support them. To do this, open the registry and navigate to HKLM\System\CurrentControlSet\Control\SecurityProviders\SChannel\Protocols and create two new keys called TLS 1.1 and TLS 1.2. Under each of these keys create new keys called Client and Server. Within each Client and Server key under TLS 1.1 and TLS 1.2 create DWORD values called DisabledByDefault set to 0 and Enabled set to 1. Restart the TMG firewall for this change to take effect.

ImageFigure 6

Mitigating the BEAST

In the output of the report generated by the Qualys SSL server test you may notice that the Browser Exploit Against SSL/TLS (BEAST) attack is not mitigated on the server side.

ImageFigure 7

The way to address this is to alter the order of SSL cipher suites on the TMG firewall to prefer cipher suites that use RC4 as outlined here. However, this presents a real conundrum because the RC4 encryption algorithm has proven to be weak and vulnerable to attack, and has even been disabled by default in Windows 8.1. Since the beast attack has mostly been mitigated in modern browsers, it is not recommended to make this change.

Standard Disclaimer

There is always a tradeoff between security and usability, so it is important to understand that while the changes I’ve described in this article will dramatically improve the overall security of your SSL websites published using Forefront TMG 2010, there is always the possibility of interoperability issues. When making these changes, I suggest making them incrementally and testing thoroughly at each point to minimize any disruption that may arise due to your SSL hardening efforts.

Summary

Forefront TMG 2010 provides excellent security in its default configuration in most deployment scenarios. Security can be further enhanced by ensuring that the underlying operating system and Forefront TMG are fully up to date, and by performing surface hardening and attack surface reduction along with following some common configuration and administration best practices. If you’re publishing SSL web sites using Forefront TMG 2010 however, you’ll need to perform the steps outlined in this article to achieve the highest level of protection that TMG has to offer. SSL security starts with choosing a strong certificate with a minimum key length of 2048 bits. Disabling deprecated protocols and enabling newer ones is also essential, as is enforcing appropriate SSL renegotiation policies. Once you’ve completed these tasks you will have significantly improved TMG’s overall security posture and you can have full confidence that you’re published SSL web sites are well protected.

1 thought on “Improving SSL Security for Forefront Threat Management Gateway (TMG) 2010 Published Web Sites”

  1. Hi Richard,

    Thank you for this useful article. Just wanted to know whether there will any adverse effect for MAC clients when they will be accessing OWA published through TMG 2010.

    Thanks

Leave a Comment

Your email address will not be published.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Scroll to Top