Kerberos and VMs and NLB

[wraith808]

Anyone know quite a bit about Kerberos and VMs and NLB that I can bounce some stuff off of?  I'll post it here... but didn't want to go through formulating the post if there wasn't a sufficient mindshare to even have a hope of getting an answer 

[Stoic Joker]

Good chance 40 and I could be of some help.

[wraith808]

For each service, I have two VMs clustered- one in a Colo, and one Local.  I have one user that I'm using on the web service for that particular cluster.  I have an SPN setup for HTTP that contains the name of each machine and the user.  I access the machine through the clustername, and it goes to one server or another.

So for later reference, we have machine A, machine B (in the colo), cluster AB spanning the machines, and user C.  I connect to the test endpoint, and everything is fine.

I replicate that- let's say Machine X, machine Y (in the colo), cluster XY, and user Z.  I connect to the test endpoint and everything is fine.

So I have a client, that connects to cluster AB, that then connects to cluster XY to use its service.

I irregularly get 401s.  When it connects, it's fine... on the second hop, I can see that it's the client, and have it's credentials.  But sometimes... it  just fails for no reason.  I look in the logs, and there's a 401 there.  This never happens when I connect direct to the machines.

Our network guy said he thinks it's an interaction of VMware VMs with Hypervisor VMs... but I had to call BS on that.  There's no rhyme nor reason in the machines that it's connecting to creating a pattern (I added a key to pass through the machines so I can see the exact path it's taking, and there's just no pattern).

I think it's passing the kerberos ticket through the NLB... but he said that it just passes it along, so it's only passing what it's sent.  The other part of this is we made another endpoint just to test authorization, because it seems like if we're getting a 401, then we always get a 401, but if that first hop works, it will work from then on (we have affinity set, so the path stays the same through multiple calls), but that started failing today.  The other dev got a 201... then when he called the real endpoint, got a 401. (All of this is on the second hop- the first hop always works)

I know more about Kerberos than I wanted to know now after looking at this for a couple of months.  And I think we finally have the code right.  But the netops says that there is no way that NLB is causing this issue.

Have you ever seen NLB cause issues with kerberos tickets?

[Shades]

StackOverflow mentions:
in IIS > 7, I had to add "useAppPoolCredentials = true" to the application's configuration in order to have everything work as expected. Without that configuration element, all authentication was broken.

The situation described at the link has an image indicating the problem...which on first glance matches with your description. Perhaps the suggestion above could be of any help?

[wraith808]

StackOverflow mentions:
in IIS > 7, I had to add "useAppPoolCredentials = true" to the application's configuration in order to have everything work as expected. Without that configuration element, all authentication was broken.

The situation described at the link has an image indicating the problem...which on first glance matches with your description. Perhaps the suggestion above could be of any help?

I've already done that.  My setup on the app server side instructions that I created:

Setup of Web Application for use with Kerberos

• Add App Pool Account to the Administrators group
• Set up app pools. Use a service account that has an SPN set to be able to decrypt the Kerberos ticket.
• Turn on 32-bit application support in app pool
• Copy files. Make sure that file location has Read and Read & Execute (not sure if it actually needs List Folder Contents- I doubt it).  Give App pool accounts full access.
• Set up sites
• a. Turn off Anonymous Access if it is on.
• b. Turn on Windows Authentication
• c. Use Negotiate 1 - not the new Negotiate 2
• d. Do not need to turn off Kernel Mode Authentication
• e. Go into Config Editor
• e1. Open webserver/security/authentication/windows auth
• e2. set useAppPoolIdentity=True
• e3. Make sure that Enabled=True
• e4. Make sure that there is only one provider.

A lot of that is to remove red herrings from the mix rather than needed.  As I said, it works.  Just not reliably through the NLB, and I know nothing about the interaction of NLB with Kerberos, and can't find anything specifically in that regard.  I also don't know about the interactions of hypervisor and VMware, and that was one of the areas the neteng talked about, also.  But after reading books on kerberos and staying up many nights on that point, I'm not really looking to try to become an expert on hypervisor vs. vmware or NLB- so that becomes the point- trusting him that they have little to do with kerberos when everything else is set up correctly.

[Stoic Joker]

What was the 401 sub status code - 401.? if any?

I could be totally out in the weeds here, but I just ran into a rather annoy authentication issue with a RemoteApp server that was refusing to authenticate anything from the outside. After some digging I ran across the EnforceChannelBinding registry setting which - when set on the target IIS8 server -  got things going finally. The article snippet I put in my notes is below.

External RDP Logon Failures: Audit Failure Event 4625 – Status Code: 0x000035B
Resolution Options:
Method 1:
Adjust the LmCompatibility registry value not to force NTLMv1 by setting it to a value of 3 or larger.
For more information about the LmCompatibility registry value, see included file: LmCompatibilityLevel.pdf
Method 2 (Currently Used in Lab Configuration):
Set the EnforceChannelBinding registry value to 0 (zero) to ignore missing channel bindings on the Gateway server. To do this, locate the following registry subkey, and use the given specifications:
HKLM\Software\Microsoft\Windows NT\CurrentVersion\TerminalServerGateway\Config\Core
Type: REG_DWORD
Name: EnforceChannelBinding
Value: 0 (Decimal)
Note: By default, the EnforceChannelBinding value does not exist on the Gateway server. You must create this value.

Are there any corresponding entries in the event logs of any of the systems crossed that may shed some light on who is dropping the ball why?

[wraith808]

Thanks for that SJ!  I'll check that on Monday- I'd not heard any of that before.  I'll also check the 401 - I think it was just a plain 401 still, which has been the problem from the beginning- no context.  I was able to get past all of the others with that setup... but it's been a pain without context.  The wireshark even returned just a 401, but I was able to get it from the Event logs.