I’ve seen this issue a couple of times after applying Exchange 2010 SP1 on Client Access servers. I’m presented with the OWA FBA page and enter my credentials and then boom it throws an “HTTP 500” error instead of opening the mailbox.
You tried to issue an “IISReset /NoForce” and even rebooted the server and still see the symptom.
So the issues is typically caused by the “Microsoft Exchange Forms-Based Authentication” service being in a stopped state. Starting the service immediately fixes the issue.
Why it sometimes doesn’t start automatically after a reboot of the servers is a very good question. I’ll do some more research in order to find the root cause.
MCM: Exchange 2007 | MVP: Exchange Architecture