Linux by Trial and Error

A repository of the things I learn about Linux

Clobbering Java

This may not happen often, but perhaps because I learned two new things in the same day, that is probably what prompted me to start this blog to begin with. And this one also has to do with the recent errata updates that I recently applied on several of my RHEL servers.

In this case, we have a server that runs a web interface that runs on Java. After applying the errata, the web page wouldn’t come up. Went through old documentation and ran the scripts to stop and restart the necessary services, but to no avail.

I was able to verify that the main application was running. Even verified that Tomcat was running. But no web site!

In this case, there is also a Desktop application to interface with the system and that worked just fine. Just couldn’t get to things from the browser.

Once again, after a few hours of investigation, we found that the application had been using a 32-bit version of Java. Well…since the Satellite server only had the latest version of the 64-bit Java, it was kind enough to remove the 32-bit version for us and install the new 64-bit version instead. Isn’t that considerate?

We didn’t stop there, however. After manually re-installing the 32-bit version, we synced up our Satellite and verified that the 32-bit version was now available. So, as a test, we pushed out both 32-bit and 64-bit versions of Java (the same release) and this time, nothing broke!

Something to keep in mind if you, like many others in the world of technology today, are heavily reliant on Java to run applications.

Advertisements

March 29, 2012 Posted by | java | | 1 Comment

It’s the Little Things

So, I recently deployed the latest errata via our Red Hat Satellite server to a handful of RHEL boxes. The updates included a minor OS revision, so we went from RHEL 5.7 to RHEL 5.8. This was my first deployment of errata onto production servers, so I was a bit nervous.

Because of the kernel update, a reboot was required. So, I scheduled the errata to deploy at 7:00pm in order to give the time to install before I started bouncing servers at 9:00pm. Before I got to 9:00, however, I started getting alert e-mails from system monitors about systems not responding.

Most of the restarts went fine. However, several of them were in our DMZ, so instead of authenticating to our Active Directory server, we have a directory server in the DMZ.

Five of the servers being restarted were in the DMZ, including DNS and Directory Servers. After the application of the errata, I found that I could not log in using LDAP user accounts. Good thing they were VMs and I had access to the console so I could log in as root!

After fighting with this for about 2 1/2 hours in the middle of the night, I decided that since the mission critical applications running in the DMZ did not seem to be adversely affected, I’d get some rest and tackle this first thing in the morning.

Another couple hours of sifting through logs and working with one of my senior server admins (who actually knows how to use tools like strace and such), we found that LDAP was throwing an error regarding “Unauthorized connections” to the directory server. That led us to take a look at the /etc/ldap.secret file.

One thing we have noticed about RHEL is that a lot of files, for some reason, require a blank line at the end of the file in order to be recognized. Our /etc/ldap.secret file did not have a carriage-return at the end of the first line…so there was one and only one line in the file and the EOF was at the end of the first line.

# vi /etc/ldap.secret
<ldap password>
~
~
~

became

# vi /etc/ldap.secret
<ldap password>

~
~
:wq

[Note the lack of the ‘~’ character on the second line.]

Viola!!!

# getent passwd johndoe
johndoe:*:10001:10001:johndoe:/home/johndoe:/bin/bash

In less than 15 seconds, we fixed a problem that I had spent several hours working on.

Don’t you hate when that happens?

March 29, 2012 Posted by | ldap | , , | 2 Comments