Just a quick note to let everyone know that I've had no problems with in place upgrades of ISA 2004 SE to ISA 2006 SE. The upgrade process is very simple, and the installation wizard walks through any tasks you might need to carry out before performing the upgrade. For example, you need to uninstall both the Firewall Client Share and the SMTP Message Screener from ISA 2004 before upgrading to ISA 2006.
The only problem with the upgrade is that the System Performance graphs in the Dashboard of the Monitoring node do not work. I noticed that this was a problem in some of the beta versions of ISA 2006, but was fixed in the last beta. However, I think they rolled back from the last beta to a previous beta for RTM, so the same issues appeared again, the only one being this problem with the System Performance graphs.
Let me know if you run into the same problem. Here's what they look like when they're not working: