Khaled Salameh's Blog

NOTE: a Story with a solution, if you don’t have time and just need the solution for the error, jump to “To Sum it up for you section below”, but if you want to learn more, read the whole post

I know, you hate error codes as much as I do, especially when you paste the code on a search-engine and get stupid results!.

Here is my very long story:

The office communications server in my company suffered from an unexpected failure, we spent hours and hours trying to fix it up but with no hope, so we decided to add an additional server to the pool and uninstall the failed one, but…… we chose the wrong server to install OCS on!, a server running VMM 2008 R2, HyperV R2, SCSM Beta 2, WDS, and a couple of eggrolls …. and of course, we forgot to check Microsoft website to see if OCS R2 is supported on Win2008 R2! (its not, and it should come soon) , spending more than 24 hours trying to fit OCS over 2008 R2, it didn’t work , so we moved the services to another server (OCS is not the point now ) and things worked like a charm.

2 weeks later, our systems administrator comes in, says : Hey, VMM 2008 R2 is not working!, did you do anything?, Me : Hell No, anyway, check it out.

The guy goes, 4 hours later, he comes back saying: man , it’s impossible! I tried to uninstall VMM 2008 R2 and reinstall it again, but now, I receive the following error during the installation:

“Virtual Machine Manager cannot process the request because an error occurred while authenticating VMMServer.Domainname.local. Possible causes are:

1) The specified user name or password are not valid.    <– My Comment was CRAP!

2) The Service Principal Name (SPN) for the remote computer name and port does not exist. <– My comment was what remote computer is he talking about?

3) The client and remote computers are in different domains and there is not a two-way full trust between the two domains. Log in by using an account on the same domain as the VMM server, or by using an account on a domain that has a two-way full trust with the domain of the VMM server, and then try the operation again. If this does not work, purge the Kerberos tickets on the VMM server by using kerbtray.exe, available at http://go.microsoft.com/fwlink/?LinkId=93709. Then, reset the SPN for VMMServer.Domainname.local by using setspn.exe available from http://go.microsoft.com/fwlink/?LinkId=93710. If this still does not fix the problem, make VMMServer.Domainname.local a member of a workgroup instead of a domain, restart the computer, rejoin the domain, and then try the operation again. <– My comment was Two way trust? We have a single domain!, Kerberos tickets, SPNs , blah blah blah…. There must be something else.

ID: 2917. Details: The network path was not found (0×80070035)” <– Here, I commented: Damn, I’ll never be able to find a resolution for this, network path was not found relates to millions of issues!

Shortly, after doing other tries, we received another error:

Failed to configure the WS-Management service. In the Local Group Policy Editor (gpedit.msc), navigate to Computer Configuration\Administrative Templates\Windows Components\Windows Remote Management (WinRM), and then ensure that there are no policy settings configured for WinRM Client or WinRM Service.

And here, we both looked at the screen…. And laughed! We haven’t touched WinRM configuration! NEVER!

So we decided to be smart, and tried to disable and enable WinRM (suspecting that WinRM has failed for some reason on the local computer) when we tried to disable WinRM from Server Manager, we received the following error:

WinRM already is set up to receive requests on this machine.
WSManFault
    Message = WinRM cannot process the request. The following error occurred while using Negotiate authentication: An unknown security error occurred.
 Possible causes are:
  -The user name or password specified are invalid.
  -Kerberos is used when no authentication method and no user name are specified.
  -Kerberos accepts domain user names, but not local user names.
  -The Service Principal Name (SPN) for the remote computer name and port does not exist.
  -The client and remote computers are in different domains and there is no trust between the two domains.
 After checking for the above issues, try the following:
  -Check the Event Viewer for events related to authentication.
  -Change the authentication method; add the destination computer to the WinRM T
rustedHosts configuration setting or use HTTPS transport.
 Note that computers in the TrustedHosts list might not be authenticated.
   -For more information about WinRM configuration, run the following command: w
inrm help config.

Error number:  -2144108387 0x8033809D
An unknown security error occurred.

mmmmmm. CRAP!, we’ve done every possible thing to fix this issue, tried to disjoin and rejoin the machine but no hope, and there we stopped saying that we need to reformat the VMM Server. (remember, hosting SCSM, WDS, HyperV and the eggrolls )

so, yesterday, the guy reformats the machine after copying and backing up some data and send me an SMS that things are ready for installation.

I was so excited that VMM will be back, managing a virtual infrastructure without VMM is just like a network with 1000 User without Active Directory…….SUCKS!.

First thing to do, let’s make sure that WinRM is enabled….Check!

Started the VMM installation process and…….. :( the same error appears again!, Damn it, what the **** Is this ? again, I googled everywhere, searched every single place, no sign of any help, things referred to SPNs and stuff, tried them all, but didn’t work! , after around 4 Hours of thinking , I tried the following command:

SetSPN –X                <–  This commands queries against active directory to see if there are any duplicate SPNs, and BAM! Returned a duplicate, the fun part is that even if you disjoin a server from the domain, SPNs remain in active directory (bound with the server netbios and FQDN!), when I saw that I decided to change the computer name and when I did that, VMM Installed properly and WinRM worked!, but……, when I tried to install Systm Center Service Manager……… It failed to install! Why? Because SCSM database was built using the old server name, I tried to modify the database and change the records but it didn’t work, so my only option was changing the computer name again to the old one” VMMServer”,  and VMM was already installed on the other name…. tried to run VMM….. it didn’t work because the computer name was changed.

Well, you see, the life of IT Pros is really really hard and sometimes you will really want to bang your head in the wall.

Anyway, my only option was to fix that VMM installation issue, so I reinstalled SCSM and it worked fine, tried to install VMM and got the same old error.

Tried to manually delete all computer name SPNs using :

Setspn –D %Value%/Value    e.g. setspn –D HOST/VMMServer.DomainName.Local

Note: you can query for all SPNs related to a computer name using SetSPN -Q SPN or Hostname

And deleted all SPNs, then, disjoined the server from the domain and rejoined it, and it didn’t work!

I was really frustrated, I didn’t know what to do, then, an angel came by and said: try to use SetSPN –X , it shall set you free!, again! i’ve already ran the -X and saw duplicates that i didn’t know what to do with!

So, I ran SetSPN –X    looked at the duplicates and thought of reading the help thoroughly, and found that the server name was bound to a USER ACCOUNT called RTCComponentService (YEAH, Office Communications Server damn it).

I ran the following command and added “-U” which means user account:

SetSPN –U –D http/VMMServer.DomainName.Local “RTCComponentService”

Waited for 2 minutes, tried to disable and enable WinRM, and it worked, then, tried to install VMM and …. IT WORKED!

So that account was blocking and making all the SPN crappy issues!

To sum it up for you:

If you are having the error code above or any error code related to SPNs during VMM Installation or any product that queries SPNs during installation, try changing the computer name (if you don’t have other services that depends on it) and things should work, however, if you do have services depending on your computer name, check the duplicate SPNs using SetSPN –X, then, try to remove those duplicates using setspn –U or –C (-U for user accounts binding the SPN and –C for computer accounts) followed by the –D (which means Delete) , then the service type (WinRM uses HTTP or HTTPS) so expect it to start with http/servername, finally, the user account or computer account name which is usually detailed in setspn –X, here is a sample setspn –X with duplicates reported:

===========================

C:\Users\Myuser>setspn -x

Checking domain DC=MyDomain,DC=local

Processing entry 2

http/TestServer.MyDomain.local is registered on these accounts:

CN=Testuser,DC=MyDomain,DC=local

found 1 group of duplicate SPNs.

=========================

As you can see, I have one group of duplicates on the server, first verify that the CN=TestUser is a user account not a computer account using Active Directory Users and Computers, then your setspn command will be as follows:

SetSPN –U –D http/testserver.domain.local “TestUser”

SetSPN should report that the record has been updated.

Your life should be so nice after that!

 

I wrote this post because I know that there are only few resources for VMM errors, in my case it was OCS that is causing the problems because it was installed on the server, in your case, it could be anything else that uses SPNs (SharePoint for example, or anything else, there are many) and many guys had such problems with no answers in forums!

If you have any questions, feel free to comment below.

I take no responsibility in any damage to your infrastructure that is caused by information in this post, USE IT AT YOUR OWN RISK, and be careful, setspn can do many nasty things.

P.S. : VMM Agent could also fail if you have WinRM issues, so it could be an SPN that is causing your VMM Agent Installation Failures on HyperV Servers.

Thanks for having the patience to read all of this and hope things work with you,

Cheers,