HMC v7r3.1 - v7r.3.2 commands return error "Connection to the Command Server failed"

Problem(Abstract)

Defect in HMC v7r3.1 - v7r3.2 dealing with HMC log rotation scripts will eventually render HMC unusable. While problem is fixed at a later release level, you still might have to manually repair some key files or reinstall the HMC.

Symptom

Access to the V7 HMC GUI is not available (initializing) and CLI functions accessed via ssh are extremely limited, Most CLI commands return error , "Connection to the Command Server failed."

Cause

This is a known problem on that can occur if you were at HMC v7r3.1 or v7r3.2 when the file /var/hsc/log/hmclogger.log grows beyond 10MB.

Environment

HMC v7r3.1 or v7r3.2

Diagnosing the problem

Determine if the hmclogger.log file size exceeds 10MB by running command
ls -la /var/hsc/log/hmclogger.log

Determine if the cimserver.log file exists by running command
ls -la /var/hsc/log/cimserver.log

Most likely there will not be a cimserver.log file and hmclogger.log file will exceed 10MB in size.

Resolving the problem

Either reinstall the HMC and update to v7r3.3 or higher or contact IBM support to obtain a pesh password so you can manually repair key files.

Obtain a pesh password from IBM support. You must know the serial number of the HMC. Unfortunately, the command that gives you the serial#, lshmc -v, will not work as it is one of the symptoms. Hopefully you have an accurate list of serial numbers, otherwise on-site viewing of the HMC for the correct serial number will be needed so that support can give a pesh password that works. The other commands needed to become root usually work without problems.

Create an hscpe account if it does not exist (use lshmcusr to list all HMC user accounts)

$ mkhmcusr -u hscpe -a hmcpe -d "HMC PE user"

enter a password of seven characters or more.

If the hscpe user ID exist already, and you do not know the password, you may change the password as hscroot

$ chhmcusr -u hscpe -t passwd

Also, the password for the root user needs to be known as well. By default, that password is "passw0rd", but you can also change root's password as hscroot.

$ chhmcusr -u root -t passwd

Once you know the passwords for hscpe and root, and you have obtained a pesh password from IBM support, the following commands should resolve the problem.

- SSH to the HMC and login as hscpe and run following command

$ pesh

You will be prompted to enter a password after you enter the command above. Note that the serial number is seven characters long with alphabetical characters in upper case. The pesh password provided by IBM support will be eight characters long with alphabetical characters in lower case.

Once you enter the pesh password and return to the prompt, you will have access to run command "su -" so you can become root. After entering roots password perform the following.

# cat /dev/null > /var/hsc/log/hmclogger.log

Now the hmc needs to be rebooted and the normal hmcshutdown command will not work. Use reboot command as root to restart the HMC.

# reboot

This should reboot the hmc and once its back up you should have access to both the GUI web interface as well as the CLI commands when you ssh to the HMC.

This problem is resolved at 7.3.3, but if you had upgraded and the hmclogger.log file was already past the 10MB limit you may still experience this problem and have to perform this manual repair operation.