Linux by Trial and Error

A repository of the things I learn about Linux

Kerberos Authentication on Load Balanced Web Servers

Today is simply a follow-on from yesterday’s post about Kerberos authentication for a Desktop application and a web application. In yesterday’s post, I went over what we did to set up Kerberos authentication for our Dev and QA environments, each of which have a single web server and a single application server.

In our production environment, however, we have two web servers and two application servers. The web servers are handled by a load balancer which provides a virtual IP address (VIP) and determines which web server to send HTTP requests to. The application servers are clustered servers.

Rather than re-hashing a lot of the same steps, it is worth noting that the actual setup of the different Service Principals for the production environment, please read my previous post for creating the Service Principal user accounts in AD and setting up the krb5.conf, krb5_ccache and krb5.keytab files on the server. Basically, all of that remains largely the same except for a couple of things that I will detail out in this post.

Before we get to that, however, we’ll get to how the challenge with this environment manifested and how we went about resolving it…

After having gone through setting up our Dev and QA environments, I moved on to set up the production environment using the same method. Service Principal accounts were created in AD and set the same way we had set the Dev and QA SP’s. The Kerberos config file, cache and keytab files were all created using exactly the same steps as the other servers.

Also, the jaas.conf and tomcat5.conf files were set the same as the other servers. Everything was exactly the same…except for the names of the servers/Service Principals on each system.

Unfortunately, when tomcat was started, we would go to the http://webapp.domain.com and would get “Service Temporarily Unavailable.” If I commented out the following line from the tomcat5.conf file, it would work fine (although then I would have to enter my credentials):

JAVA_OPTS=ā€${JAVA_OPTS} -Xmx1024m -Djava.security.auth.login.config=/path/to/jaas.conf -Djava.security.krb5.conf=/path/to/krb5/files/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=falseā€

The only difference between production and Dev/QA was the load balancer. So, off I went to the networking guy…

We found that our load balancer was doing a health check periodically and was using an HTTP call, then looking for specific text in the response. When the above line was uncommented, the load balancer was not authenticated and was therefore not receiving the expected string. Therefore, the load balancer was shutting down the VIP because the server was “unavailable” as far as the health check was concerned.

After changing the health check to a TCP call rather than HTTP, we were able to get to the main page of the web server, though SSO was still not working correctly.

With much fuss and going back and forth about the fact that, still, the only difference seemed to be the load balancer, we decided to look at how our DNS structure was set up.

We had a HOST(A) entry for web-dev.domain.com as well as for web-qa.domain.com and web-prod1.domain.com and web-prod2.domain.com. Then, we had an Alias set up for web-dev.domain.com called webapp.dev.domain.com. We also had an Alias for web-qa.domain.com called webapp.qa.domain.com.

However, when I looked at the production set up, we had a HOST(A) entry for webapp.domain.com which pointed to the VIP.

Hmmm….

Here is what I did:

Following the steps from the previous post, I created a user in AD called HTTP/webapp-prod.domain.com and set up the Service Principal Name.

Next, I got on web-prod1.domain.com and ran:

# kinit HTTP/webapp-prod.domain.com

This added the ticket to the krb5_ccache file. Next, I ran:

# ktutil
ktutil: addent -password -p HTTP/webapp-prod.domain.com -k 2 -e rc4-hmac
Password for HTTP/webapp-prod.domain.com@DOMAIN.COM:
ktutil: addent -password -p HTTP/webapp-prod -k 2 -e rc4-hmac
Password for HTTP/webapp-prod@DOMAIN.COM:
ktutil: wkt /path/to/krb5/files/krb5.keytab
ktutil: quit
#

Then, I did the same thing on web-prod2.domain.com. Now, each production web server had four Kerberos tickets in the keytab file. Two for the local server itself (one fully qualified and the other a short name) and two for HTTP/webapp-prod (one fully qualified and the other a short name).

In my DNS server, I removed the HOST(A) entry for webapp.domain.com and created a new DNS entry for webapp-prod.domain.com which pointed to the VIP, and an Alias for that entry called webapp.domain.com.

Voila!!

I cleared all my temp files and cookies from IE, relaunched the browser and went to http://webapp.domain.com and found myself already logged into the web application and looking at my Dashboard!

Perhaps there is a better way to accomplish what I was trying to do, but since I was not able to find anything that would resolve this issue, this was the best thing I came up with.

If you know of a better way to use Kerberos authentication on a load balanced VIP to a web server, please share your experience. However, if you are struggling with something similar, hopefully this helps!

June 13, 2012 Posted by | kerberos | , , , , , | Leave a comment