If you would like to read the other parts in this article series please go to:
Introduction
In the previous article in this series, I showed you some common Live Migration errors, as well as some techniques for troubleshooting those errors. In this article, I want to continue the discussion by showing you some more common problems with live migrations.
Failed to Establish a Connection with the Host
One of the most common Hyper-V live migration error messages indicates that Hyper-V failed to establish a connection to the host. The actual error message is:
Virtual machine migration operation failed at migration source.
Failed to establish a connection with host. No credentials are available in the security package (0x8009030E).
The Virtual Machine Management Service failed to authenticate the connection for a virtual machine migration at the source host: no suitable credentials available. Make sure the operation is initiated on the source host of the migration, or the source host is configured to use Kerberos for the authentication of migration connections and Constrained Delegation is enabled for the host in Active Directory.
You can see the actual error message in Figure A.
Figure A: This is what the error message looks like.
At first, this error message would seem to indicate a password problem or some sort of problem with a set of authentication credentials. In actuality however, the error message is triggered as a result of improperly configured authentication mechanism.
To show you what I mean, open the Hyper-V Manager and then right click on the container representing your host server, and select the Properties command from the shortcut menu. When you do, the Hyper-V Settings dialog box will appear. Now, expand the Live Migration container and click on Advanced Features. When you do, you will see the screen shown in Figure B.
Figure B: Hyper-V offers two different authentication options for live migrations.
As you can see in the figure above, Hyper-V offers two different types of authentication for live migrations. The servers can authenticate with one another using either Credential Security Support Provider (CredSSP) or Kerberos. Many administrators prefer to use CredSSP because doing so does not require any additional configuration. Even so, CredSSP authentication is often inadequate.
The reason for this has to do with hop limitations. The CredSSP protocol only works across a single hop. This means that if you want to live migrate a virtual machine using CredSSP authentication then you will have to be logged directly into the host server on which the virtual machine is currently running. If you attempt to login and initiate the migration process from elsewhere, then the live migration will fail and you will see the error message shown in Figure A. The error message shown in Figure A will also be displayed if you attempt to live migrate a virtual machine to a host that is more than one hop away, regardless of whether or not you are logged directly into the host.
As you have probably already figured out, the solution for avoiding this problem is to use Kerberos authentication instead of CredSSP authentication. Unlike CredSSP, Kerberos is not subject to the single hop limitation.
The problem with using Kerberos authentication, is that by using Kerberos for Hyper-V live migration, you are essentially granting the Hyper-V host server permission to act on your behalf. That isn’t a big deal if you are just performing a live migration, but imagine that a Hyper-V host became infected with malware. You wouldn’t want the host to be able to use Kerberos credentials to perform potentially destructive operations.
The solution to this problem is to use constrained delegation for Kerberos. Constrained delegation allows the Hyper-V host to act on your behalf, but only for specific purposes (in this case, performing live migrations).
In order to configure constrained delegation, the Hyper-V hosts must be domain members. With that said, go ahead and log into a domain controller or a server that has the AD Management tools installed, and then open the Server Manager. Now, launch the Active Directory Users and Computers console by selecting it from the Server Manager’s Tools menu.
When the Active Directory Users and Computers console opens, select the Computers container. Now, right click on the listing for your Hyper-V host server, and select the Properties command from the shortcut menu. When the host’s properties sheet appears, select the Delegation tab.
At this point, you must select the Trust this Computer for Delegation to Specified Services Only option, as shown in Figure C. You must also make sure that the Use Kerberos Only option is selected.
Figure C: Choose the option to trust the computer for delegation to specific services.
Now, click the Add button and Windows will reveal the Add Services dialog box. When this dialog box appears, click on the Users or Computers button. This will cause Windows to display the Select Users of Computers dialog box. You must enter the name of the Hyper-V Server to which you want to live migrate the virtual machine, and then click OK. You will now see a list of all of the services that are available on the destination host, as shown in Figure D.
Figure D: Windows displays the services that are available on the destination host.
Select the CIFS service and click OK. The CIFS service is associated with storage live migrations, and it will need to be enabled if you plan to move a virtual machine’s storage as a part of a live migration.
Now, click the Add button and click the Users and Computers button once again. Enter the name of the destination host server and click OK. You will once again see the list of services that are available on the destination host. Select the Microsoft Virtual System Migration Service, and click OK. You should now see the CIFS service and the Microsoft Virtual System Migration Service listed for the destination host, as shown in Figure E. Click OK to complete the process.
Figure E: You should see the CIFS service and the Microsoft Virtual System Migration Service listed for the destination host.
So far we have given Kerberos permission to be used for live migration from one specific host server to one specific destination server. As it stands right now, Kerberos does not even have permission to live migrate the virtual machine back to the original host.
In order to make live migrations work for all of your Hyper-V hosts, each host server must have constrained delegation enabled for every other Hyper-V host. Suppose for instance that I had three servers named Lab1, Lab22, and Lab3. Lab1 would need permission to migrate to Lab2 and Lab3. Similarly, Lab2 would need permission to live migrate virtual machines to Lab1 and Lab3. Finally, Lab3 would require permissions to live migrate virtual machines to Lab1 and Lab2. Needless to say, the setup process can be quite tedious if you have a lot of host servers.
Conclusion
In this article, I have shown you how live migrations can fail as a result of the CredSSP protocol’s single hop limitation, and how you can fix the problem by using Kerberos with constrained delegation. In the next article in this series, I will show you some more things that can cause live migrations to break down.
If you would like to read the other parts in this article series please go to:
