Linux by Trial and Error

A repository of the things I learn about Linux

It’s the Little Things

So, I recently deployed the latest errata via our Red Hat Satellite server to a handful of RHEL boxes. The updates included a minor OS revision, so we went from RHEL 5.7 to RHEL 5.8. This was my first deployment of errata onto production servers, so I was a bit nervous.

Because of the kernel update, a reboot was required. So, I scheduled the errata to deploy at 7:00pm in order to give the time to install before I started bouncing servers at 9:00pm. Before I got to 9:00, however, I started getting alert e-mails from system monitors about systems not responding.

Most of the restarts went fine. However, several of them were in our DMZ, so instead of authenticating to our Active Directory server, we have a directory server in the DMZ.

Five of the servers being restarted were in the DMZ, including DNS and Directory Servers. After the application of the errata, I found that I could not log in using LDAP user accounts. Good thing they were VMs and I had access to the console so I could log in as root!

After fighting with this for about 2 1/2 hours in the middle of the night, I decided that since the mission critical applications running in the DMZ did not seem to be adversely affected, I’d get some rest and tackle this first thing in the morning.

Another couple hours of sifting through logs and working with one of my senior server admins (who actually knows how to use tools like strace and such), we found that LDAP was throwing an error regarding “Unauthorized connections” to the directory server. That led us to take a look at the /etc/ldap.secret file.

One thing we have noticed about RHEL is that a lot of files, for some reason, require a blank line at the end of the file in order to be recognized. Our /etc/ldap.secret file did not have a carriage-return at the end of the first line…so there was one and only one line in the file and the EOF was at the end of the first line.

# vi /etc/ldap.secret
<ldap password>
~
~
~

became

# vi /etc/ldap.secret
<ldap password>

~
~
:wq

[Note the lack of the ‘~’ character on the second line.]

Viola!!!

# getent passwd johndoe
johndoe:*:10001:10001:johndoe:/home/johndoe:/bin/bash

In less than 15 seconds, we fixed a problem that I had spent several hours working on.

Don’t you hate when that happens?

March 29, 2012 Posted by | ldap | , , | 2 Comments