Let’s see what we can do to get the Mac Mail to connect successfully to Exchange servers.
Since we addressed the 100-continue handling in Part 1 of this article, ExRCA tests did not indicate any more issues. Microsoft Outlook for Mac did not have issue connecting to Exchange even before we changed 100-continue handling to be RFC-7231 compliant.
At this point, I decided to look at the network trace. Looking at the trace captured using Wireshark on the client machine, I noticed the following HTTP dialog:
POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 200 OK
POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 400 Bad Request
Notice how initial dialog included challenge response for NTLM authentication and ended with 200 OK. At this point Mac Mail client started with another POST operation to EWS. However, this time the pattern followed multiple 401 responses from server and ended with 400.
When comparing that with Outlook for Mac, the pattern looked like the following:
POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 401 Unauthorized <-> POST /ews/exchange.asmx <-> 200 OK
Notice how the pattern looks similar to Mac Mail’s second POST request, however, it ends with 200 OK. The response included data that was requested by POST operation.
I needed to correlate this with the data being received on the Exchange server. Due to newer ciphers which use PFS (Perfect Forward Secrecy) I couldn’t decrypt data being received from load balancer by the server despite having the private key used to encrypt the information. I was able to decrypt client side traffic due to load balancer configuration of ciphers that didn’t use PFS. I don’t recommend this configuration in production and KEMP LoadMaster default setup uses more secure ciphers. I had it configured this way only for troubleshooting purpose in my lab and no production data was in flight during testing.
I was now facing the issue of capturing the headers from the server side. One way to address this is to configure list of ciphers used by server similar to what I had done on KEMP LoadMaster. However, in my test that did not work for some reason. I decided to use failed request tracing in IIS (big thanks to my Microsoft colleague Brad Hughes for the hint). Here’s how to set it up:
– Using IIS Manager, select the “Default Web Site” and select “Failed Request Tracing Rules”
– Create a new rule and select “All Contents” in the “Specify Content to Trace” dialog
– In “Trace Conditions” dialog, select “Status Code(s)” and type 400
– Accept the default selection for trace providers and click finish
– Click “Edit Site Tracing” option under Actions and click “Enable”
– Change trace location or accept default and click “OK”
Now that the trace is enabled and rule configured to capture HTTP 400 error condition, I launched Mac Mail client and reproduced the issue. When the error occurred, numerous xml files were created by IIS. Here’s the most relevant piece of data from the trace file (I have removed unnecessary headers for brevity):
Via: HTTPS/1.1 load.balancer.ip.address:443
Authorization: NTLMCookie: X-BackEndCookie=S-1-5-21-…;
ClientId=GALCMQFEEUEGNHCEWFGEVQ; exchangecookie=196574d09f7f419897032917045ee4fb
Host: mail.contoso.com
User-Agent: Mac OS X/10.11.1 (15B42); ExchangeWebServices/6.0 (242); Mail/9.1 (3096.5)
X-ClientSide: some.public.ip.address:49792 -> load.balancer.ip.address:443
While the headers gives you idea of what in included in a normal POST request from the client, for me, the first and last headers were very important. The “Via” header indicates presence of a proxy between client and server. In this case that is the IP address of the load balancer. The X-ClientSide (or X-Forwarded-For) header is used by proxies and load balancers to preserve original client IP address when SNAT is in use. SNAT is recommended when load balancing Exchange servers. KEMP LoadMaster uses X-ClientSide header by default.
When I saw these headers, I remembered the discussion I had with my talented support engineers at KEMP. At some point in our discussions, someone had mentioned that inserting “Via” headers are known to negatively impact Mac Mail client.
As the saying goes, it takes a while to find the root cause, fix is usually quick. Once I got to this point where I was fairly confident that the issue could be presence of “Via” header, I set “Additional L7 Header” from default value of “X-ClientSide” to “None”:
All there was left to do at this point, was to launch Mail client on Mac and watch it connect. And connect it did!
There are couple of important things to note here:
– Outlook for Mac didn’t have any issues with default settings of KEMP LoadMaster (100-Continue handling and/or additional L7 headers).
– When testing for EWS connectivity, ExRCA tests failed with “501” error. This error was fixed by changing 100-Continue handling from RFC-2616 compliant (default setting) to RFC-7231 compliant. This, however, did not fix the issues with Mac Mail.
– Changing additional L7 headers from “X-ClientSide” (default setting) to “None” stopped KEMP LoadMaster from inserting any additional headers including “Via” headers. This did fix the issue for built-in Mail client on Mac.
However, it leaves us with a few open questions:
Why did Outlook for Mac worked all along?
Why is Outlook for Mac not affected by presence of “Via” headers?
The “Via” and other additional headers are inserted by load balancer but are only visible to the server. They were not present in network captures on the client side. Why did the Mail client on Mac behave differently when the “Via” header was present and visible only to the server?
Last but not the least, what is the real issue here? It would be easy to blame the load balancer but since Outlook for Mac worked fine, is it programmed to work around an EWS issue that Mac Mail isn’t? Or is it a Mac Mail issue as Outlook worked fine?
I am working with Microsoft to find the answers but don’t hold your breath. While doctors work on the root cause, use the handy band aid to stop the bleeding! 