A question about how to block applications that tunnel inside SSL came up the other day. This ISA firewall admin was interested in blocking GoToMeeting and wondered if the HTTP Security Filter on the ISA firewall might be useful in blocking the connections. He specifically was interested in any specific signatures that he could craft to block the application.
One suggestion was that he use outbound SSL to SSL bridging, a feature that enables the firewall to inspect outbound SSL connections. This feature isn't included with the ISA firewall, but you can get outbound SSL inspection by using Collective Software's ClearTunnel. TMG firewalls will be able to do this right out of the box, and this functionality is included with the TMG Beta 2 firewall.
The reason why outbound SSL inspection might work is that many of these applications (such as GoToMyPC) work by using SSL on port 443. But inside the SSL tunnel they usually don't use HTTP. So with outbound inspection, the ISA Web proxy filter will try to digest the internal protocol, and block it with "400 bad request" since it's expecting an HTTP request.
Now, if the app does use HTTP inside the SSL tunnel, then it's possible the ISA Web proxy will indeed allow it. In this case, the original question of "signatures" becomes relevant; you can define an HTTP Security Filter entry to block the traffic by signature in the normal fashion. Of course you need to see the HTTP traffic to get an idea of what a good signature might be. Since it's encrypted everywhere except inside of ISA, you'll need some help to do that. One good option is to use a free debugging filter, such as Collective Software's TrafficLog to snoop on that traffic.
Thomas W Shinder, M.D., MCSE
Sr. Consultant / Technical Writer