It’s Our Blog's 1 Year Anniversary ...!

Its  been an year ago I have started this blog , I was  thrilled and motivated  by  the tremendous  response you gave to me with over 2.6+ lakhs patronage.

I hope ,we  produce more and more informative articles  moving forward.

One thing I can say ,  Thank you All ;)

Our Moto is  " Stay Hungry, Stay Foolish"

Happy Unixing 

Stay In Touch:  feeds, twitter ,facebook & google+.

Thanks,
Ram

Read More

A Simple Way to Send Multiple Line Commands Over SSH

Below are three methods to send multiple line commands over SSH. The first method is a quick overview of running remote commands over SSH, the second method uses the bash command to run remote commands over SSH, and the third method uses HERE documents to run remote commands over SSH. Each have their limitations which I will cover.

Contents

1. Running Remote Commands Over SSH 
2. Using SSH with the BASH Command 
3. Using SSH with HERE Documents

Running Remote Commands Over SSH

To run one command on a remote server over SSH:

ssh $HOST ls

To run two commands on a remote server over SSH:

ssh $HOST 'ls; pwd'

To run the third, fourth, fifth, etc. commands on a remote server over SSH keep appending commands with a semicolon inside the single quotes.

But, what if you want to remotely run many more commands, or if statements, or while loops, etc., and make it all readable?

#!/bin/bash
ssh $HOST '
ls

pwd

if true; then
    echo "This is true"
else
    echo "This is false"
fi

echo "Hello world"
'

The above shell script works but begins to break if local variables are added.

For example, the following shell script will run, but the local variable HELLO will not be parsed inside the remote if statement:

#!/bin/bash

HELLO="world"

ssh $HOST '
ls

pwd

if true; then
    echo $HELLO
else
    echo "This is false"
fi

echo "Hello world"
'

In order to parse the local variable HELLO so it is used in the remote if statement, read onto the next section.

Using SSH with the BASH Command

As mentioned above, in order to parse the local variable HELLO so it is used in the remote if statement, the bash command can to be used:

#!/bin/bash

HELLO="world"

ssh $HOST bash -c "'
ls

pwd

if true; then
    echo $HELLO
else
    echo "This is false"
fi

echo "Hello world"
'"

Perhaps you want to use a remote sudo command within the shell script:

#!/bin/bash

HELLO="world"

ssh $HOST bash -c "'
ls

pwd

if true; then
    echo $HELLO
else
    echo "This is false"
fi

echo "Hello world"

sudo ls /root
'"

When the above shell script is run, everything will work as intended until the remote sudo command which will throw the following error:

sudo: sorry, you must have a tty to run sudo

This error is thrown because the remote sudo command is prompting for a password which needs an interactive tty/shell. To force a pseudo interactive tty/shell, add the -t command line switch to the ssh command:

#!/bin/bash

HELLO="world"

ssh -t $HOST bash -c "'
ls

pwd

if true; then
    echo $HELLO
else
    echo "This is false"
fi

echo "Hello world"

sudo ls /root
'"

With a pseudo interactive tty/shell available, the remote sudo command’s password prompt will be displayed, the remote sudo password can then be entered, and the contents of the remote root’s home directory will be displayed.

However, recently I needed to run a specific remote sed command over SSH to find and delete one line and the subsequent three lines and another specific remote sed command over SSH to find a line and insert another line with some text above it, so I naturally tried using the bash method mentioned above:

#!/bin/bash

ssh $HOST bash -c "'
cat << EOFTEST1 > /tmp/test1
line one
line two
line three
line four
EOFTEST1

cat << EOFTEST2 > /tmp/test2
line two
EOFTEST2

sed -i -e '/line one/,+3 d' /tmp/test1

sed -i -e '/^line two$/i line one' /tmp/test2
'"

Everytime I would run the above shell script, I would get the following error:

sed: -e expression #1, char 5: unterminated address regex

However, the same commands work when run by themselves:

ssh $HOST "sed -i -e '/line one/,+3 d' /tmp/test1"
ssh $HOST "sed -i -e '/^line two$/i line one' /tmp/test2"

I thought the problem may be because of single quotes within single quotes. The bash command above requires everything to be wrapped in single quotes and a sed command requires the regular expression to be wrapped in single quotes as well. As mentioned in the BASH manual, “a single quote may not occur between single quotes, even when preceded by a backslash”.

However, I debunked this single quote theory being my problem because running a simple remote sed search and replace command inside of the bash command worked just fine:

#!/bin/bash

ssh $HOST bash -c "'

echo "Hello" >> /tmp/test3

sed -i -e 's/Hello/World/g' /tmp/test3
'"

I can only assume the problem with the specific remote sed commands is something with the syntax that I have not yet figured out.

Despite all this, I eventually figured out that the specific remote sed commands I wanted to run would work when using SSH with HERE documents.

Using SSH with HERE Documents

As mentioned above, the specific remote sed commands I wanted to run did work when using SSH with HERE documents:

ssh $HOST << EOF
cat << EOFTEST1 > /tmp/test1
line one
line two
line three
line four
EOFTEST1

cat << EOFTEST2 > /tmp/test2
line two
EOFTEST2

sed -i -e '/line one/,+3 d' /tmp/test1

sed -i -e '/^line two$/i line one' /tmp/test2
EOF

Despite the remote sed commands working, the following warning message was thrown:

Pseudo-terminal will not be allocated because stdin is not a terminal.

To stop this warning message from appearing, add the -T command line switch to the ssh command to disable pseudo-tty allocation (a pseudo-terminal can never be allocated when using HERE documents because it is reading from standard input):

ssh -T $HOST << EOF
cat << EOFTEST1 > /tmp/test1
line one
line two
line three
line four
EOFTEST1

cat << EOFTEST2 > /tmp/test2
line two
EOFTEST2

sed -i -e '/line one/,+3 d' /tmp/test1

sed -i -e '/^line two$/i line one' /tmp/test2
EOF

With this working, I later discovered remote sudo commands that require a password prompt will not work with HERE documents over SSH.

ssh $HOST << EOF
sudo ls /root
EOF

The above ssh command will throw the following error if the SSH user you are logging into requires a password when using the remote sudo command:

Pseudo-terminal will not be allocated because stdin is not a terminal.
user@host's password: 
sudo: no tty present and no askpass program specified

However, the remote sudo command will work if the SSH user’s sudo settings allow that user to use sudo without a password by setting user ALL=(ALL) NOPASSWD:ALL in /etc/sudoers.

References

What’s the Cleanest Way to SSH and Run Multiple Commands in Bash?
Chapter 19. Here Documents

Read More

Starting and Stopping Software via RC Directories

Question

How can I start up and stop my software on AIX?

Answer:

Starting and Stopping Software via RC Directories

This document describes how to start and stop software using run level directories via /etc/inittab. A run level is a software configuration that allows only a selected group of processes to exist.

For another method to start or stop an application during a reboot or shutdown, refer to the document, Automated Startup and Shutdown of Custom Software. 

The objective of run level script feature is to allow customers to start and stop selected applications by changing the run level. The directories are provided for customers to place their own stop and start scripts.

Background
Scripts to stop or start applications

---

Background:

---

During system startup, after the root file system has been mounted in the pre-initialization process, the init command is run as the last step of the startup process. The init command attempts to read the /etc/inittab file. If the file exists, init attempts to locate an initdefault in /etc/inittab. If initdefault entry exists, the init command uses the specified run level as the initial system run level. Run level 2 is defined by default to contain all of the terminal processes and daemons that are run in the multiuser environment. This can be seen in the /etc/inittab file:

 # lsitab init:
init:2:initdefault:

NOTE: Booting the system into mulitiuser mode using a runlevel other than the default, 2, is not supported by IBM. For a list of valid run levels, see man pages for init or telinit.

Upon the installation of the feature, the following are added to your system:

1. Nine directories:

   /etc/rc.d
   /etc/rc.d/rc2.d
   /etc/rc.d/rc3.d
   /etc/rc.d/rc4.d
   /etc/rc.d/rc5.d
   /etc/rc.d/rc6.d
   /etc/rc.d/rc7.d
   /etc/rc.d/rc8.d
   /etc/rc.d/rc9.d

2. Eight new entries are added to your /etc/inittab

   l2:2:wait:/etc/rc.d/rc 2
   l3:3:wait:/etc/rc.d/rc 3
   l4:4:wait:/etc/rc.d/rc 4
   l5:5:wait:/etc/rc.d/rc 5
   l6:6:wait:/etc/rc.d/rc 6
   l7:7:wait:/etc/rc.d/rc 7
   l8:8:wait:/etc/rc.d/rc 8
   l9:9:wait:/etc/rc.d/rc 9

   The system will automatically run the "K" or kill scripts when entering a given run level, then proceed to run all "S" or start scripts to start up the applications necessary at that level. In this manner, some applications could be stopped while others started when entering a run level.


3. When shutting down the system or rebooting using the /usr/sbin/shutdown command, all "K" or kill scripts for every run level will be run. This ensures all custom applications are finished before fully shutting down AIX. 


4. /etc/rc.d/rc script - This script is designed to use the input run level to visit the appropriate /etc/rc.d/rc.d. It first executes scripts in this directory starting with K to stop the applications. Then, it executes scripts starting with S to start the applications.

Sample scripts are provided in the /etc/rc.d/samples directory. See the Commands Reference for information about /etc/inittab, telinit, and init command.

---

Scripts to stop or start applications

---

1. Create a shell script that includes the commands (provided by the application vendor) to stop or start that program. To use the scripts, the user must copy them to the appropriate /etc/rc.d/rc.d directory. The /etc/rc.d/rc shell script will only vist the directory structures under rc.d.


NOTE: The script name must start with a K to stop or an S to start the application.


2. Make the script executable by running the chmod command.
In both cases, it is adviseable to use a file naming convention with a numeric after the K or S, and a short description of the process or service to be killed or started. The controlling RC script will run these in numeric order as it finds them. For example, a script to stop and start lpd daemon can be named K70lpd and S70lpd, respectively.


3. The run level can be changed by running:

 telinit 

This tells the init command to place the system in one of the run levels. When the init command requests a change to run levels 0-9, it kills all processes at the current run levels and then restarts any processes associated with the new run levels.


To check current run level, run who -r. It will return something similar to the following output:

 run level 2 Oct 4 14:23  2 0 S

In this example, the system is running at the default run level 2.

Read More

Power Systems Capacity on demand(CoD)

Capacity on demand

Capacity on Demand (CoD) offerings allow you to dynamically activate one or more resources on your server as your business peaks dictate. You can activate inactive processors or memory units that are already installed on your server on a temporary and permanent basis.

This topic describes Capacity on demand capabilities for IBM® POWER6™ or above  servers managed by the Hardware Management Console (HMC) Version 7 and later, and explains how to order and use each offering.

Capacity on Demand offerings

We are going to discuss about the  "Capacity on Demand offerings" and learn basic information about each offering.The following table provides a brief description of each CoD offering. Consult your IBM Business Partner or IBM sales representative to select the CoD offering most appropriate for your environment.

Offering	Description
Capacity Upgrade on Demand	You can permanently activate inactive processors and memory units with code without restart of server
Trial Capacity on Demand	You can evaluate the use of inactive processors, memory, or both, at no charge using Trial CoD. After it is started, the trial period is available for 30 power-on days.
On/Off Capacity on Demand	You can activate processors or memory units for a number of days by using the HMC to activate resources on a temporary basis.
Utility Capacity on Demand	Utility CoD is used when you have unpredictable, short workload spikes.
Capacity BackUp	You can use Capacity BackUp to provide an off-site, disaster recovery server using On/Off CoD capabilities.
PowerVM™ Editions (PowerVM)	PowerVM Editions (PowerVM Editions) deliver advanced virtualization functions for AIX®, Linux® and IBM i™ clients. PowerVM Editions (PowerVM Editions) include the following offerings: Ø  Micro-Partitioning™ Ø  Virtual I/O Server Ø  Integrated Virtualization Manager Ø  Live Partition Mobility Ø  The ability to run x86 Linux applications on Power Systems PowerVM Editions (Express, Standard, and Enterprise) offer different capabilities.

Capacity on Demand software licensing considerations:

Typically a tool, such as a license manager, is used to manage the licenses. A license manager detects use of the software, compares it to the entitlement, and then takes action based on the results. A licensemanager can be provided by IBM or can be made available by the software provider.

This table shows Capacity on Demand software licensing considerations.

Table 2. Capacity on Demand software licensing considerations

*Note: It is possible that you use a combination of these licensing types.

Capacity Upgrade on Demand activation codes:

After you decide to permanently activate some or all of your resources, you must order and purchase one or more activation features. When you order and purchase activation features, you are provided with one or more activation codes that you use to activate resources on your server.

The activation codes are posted on an IBM Web site for quick access,usually it will take 24 hrs upon purchase to reflect.   Here is the link http://www-912.ibm.com/pod/pod.

Ordering Capacity Upgrade on Demand activation features:

You can order activation features for a new server, a server model upgrade, or an installed server. After you place your order, you will receive a code that activates inactive processors or memory units.

Notes:

•  It can take several days to process an order. You can use a one-time no-charge Trial Capacity on Demand for 30 days to satisfy workload requirements while your order for permanent activation of additional capacity is being fulfilled. 
•  An order for activation features will process more quickly if you do not include any miscellaneous features with the order.

To order one or more CUoD activation features:

1. Determine the number of inactive processors or memory units that you want to activate.
2. Contact your IBM Business Partner or IBM sales representative to place your order for one or more activation features.

After ordering, see “Activating Capacity Upgrade on Demand” to activate inactive resources permanently.

Using Capacity Upgrade on Demand from ASMI

You can use the Hardware Management Console (HMC) or the Advanced System Management Interface (ASMI) to manage Capacity Upgrade on Demand (CUoD).

Most Capacity on Demand (CoD) tasks on the HMC require the HMC Super Administrator user role.
If you are not using the HMC, you can use the ASMI.

Activating Capacity Upgrade on Demand

When you purchase one or more activation features, you will receive corresponding activation codes to permanently activate your inactive processors or memory units.

To permanently activate your inactive resources by retrieving and entering your activation code:

1. Retrieve the activation code by going to http://www-912.ibm.com/pod/pod .
2. Enter the system type and serial number of your server.
3. Record the activation code that is displayed on the Web site.
4. Enter your activation code on your server using the HMC. To enter your code:
      a. In the navigation area of the HMC window, expand Systems Management.
      b. Select Servers.
      c. In the contents area, select the server on which you want enter your activation code.
      d. Select Tasks > Capacity on Demand (CoD) > Enter CoD Code.
      e. Type your activation code in the Code field.
      f. Click OK.

Any newly activated processors are now available for use by uncapped logical partitions. If there are no uncapped logical partitions, you must assign the processors to one or more logical partitions in order to begin using the processors. Any newly activated memory must be assigned to one or more logical partitions to begin using the newly activated memory.

Viewing settings for Capacity on Demand resources

You can use the Hardware Management Console (HMC) to view Capacity on Demand (CoD) settings.You can see how many processors or memory units you have, how many are active, and how many are available for activation using CoD with these settings. You can also view information about your On/Off CoD processors and memory units, Trial CoD processors and memory units, and your Utility CoD processors.

To view the capacity settings for processors or memory, do the following:

  1. In the navigation area of the HMC window, expand Systems Management.
  2. Select Servers.
  3. In the contents area, select the server on which you want to view capacity settings.
  4. Select Capacity on Demand.
  5. Select either Processor or Memory.
  6. Select the CoD offering that you want to view.
  7. Select View Capacity Settings

Viewing and saving Capacity on Demand code-generation information:

   1. In the navigation area of the HMC window, expand Systems Management.
   2. Select Servers.
   3. In the contents area, select the server on which you want to view and save the CoD code
        information.
4. Select Tasks.
5. Select Capacity on Demand (CoD).
6. Select Processors (or Memory).
7. Select the CoD offering you want to view or save.
8. Select View Code Information.
9. In the CoD Code Information window, click Save to save the CoD code information to a file on a
     remote system or to a file on removable media.
10. In the Save CoD Code Information panel, select one of these options, and then perform the tasks
     associated with that option.

Trial Capacity on Demand:

Trial Capacity on Demand (CoD) provides no-charge temporary capacity to enable you to test new function on your server.

 Activating Trial Capacity on Demand:

You can activate your inactive processors or memory for a trial period by obtaining and entering a trial processor code or a trial memory code.

To activate Trial CoD, do the following:
1. Retrieve the activation code by going to the following Web address: http://www-912.ibm.com/pod/pod
2. Enter your activation code on your server using the HMC. To enter your code:
   a. In the navigation area of the HMC window, expand Systems Management.
   b. Select Servers.
   c. In the contents area, select the server on which you want enter your activation code.
   d. Select Tasks > Capacity on Demand (CoD) > Enter CoD Code.
   e. Type your activation code in the Code field.
   f. Click OK. 

Stopping Trial Capacity on Demand

Trial Capacity on Demand ends when the trial period is over and the resources have been reclaimed by the server. You must return the resources before the trial period has ended.

 

To stop a current Trial Capacity on Demand, follow these steps:

1. Return the trial resources. 

2. In the navigation area of the HMC window, expand Systems Management.
3. Select Servers.
4. In the contents area, select the server on which you want to stop Trial Capacity on Demand.
5. Select Tasks.
6. Select Capacity on Demand (CoD).
7. Select either Processor or Memory.
8. Select Trial CoD.
9. Select Stop.
10. In the confirmation window, click Yes to stop the trial. Click No to cancel the request to stop the trial (the trial will remain active).

   Trial Capacity on Demand is now stopped and cannot be restarted.

On/Off Capacity on Demand:

On/Off Capacity on Demand (CoD) allows you to temporarily activate and deactivate processor cores and memory units to help meet the demands of business peaks. After you request that a number of processor cores or memory units are to be made temporarily available for a specified number of days,those processor cores and memory units are available immediately. You can start and stop requests for On/Off CoD, and you are billed for usage at the end of each quarter.

Using On/Off Capacity on Demand

You must use the Hardware Management Console (HMC) to use and manage On/Off Capacity on Demand (CoD).Most Capacity on Demand (CoD) tasks on the HMC require the HMC Super Administrator user role.After you have enabled and activated On/Off CoD, minimal day-to-day management of your temporary capacity is required.

Enabling On/Off Capacity on Demand

Before requesting temporary capacity on your server, you must enable your server for On/Off Capacity on Demand (CoD). To enable your server for On/Off CoD, you must use the Hardware Management Console (HMC).

Most CoD tasks on the HMC require the HMC super administrator user role.
To enable your server for On/Off CoD:

1. Retrieve the On/Off CoD enablement code by going to http://www-912.ibm.com/pod/pod .
2. Enter your activation code on your server using the HMC. To enter your code:
a. In the navigation area of the HMC window, expand Systems Management.
b. Select Servers.
c. In the contents area, select the server on which you want enter your activation code.
d. Select Tasks > Capacity on Demand (CoD) > Enter CoD Code.
e. Type your activation code in the Code field.
f. Click OK.

Your server is now enabled for On/Off CoD.

Activating On/Off Capacity on Demand

After you have ordered On/Off CoD and enabled On/Off CoD, you can request temporary activation of On/Off CoD resources.

To request activation of On/Off CoD resources:

1. In the navigation area of the Hardware Management Console window, expand Systems      Management.
2. Select Servers.
3. In the contents area, select the server on which you want to activate processors or memory temporarily.
4. Select Tasks.
5. Select Capacity on Demand.
6. Select either Processor or Memory.
7. Select On/Off CoD.
8. Select Manage.
9. Type the number of On/Off CoD resources you want and the number of days you want them for, and then click OK.

Any newly activated processors are now available for use by uncapped logical partitions. If there are no uncapped logical partitions, you must assign the processors to one or more logical partitions in order to begin using the processors. Any newly activated memory must be assigned to one or more logical partitions to begin using the newly activated memory.

 Utility Capacity on Demand

Utility Capacity on Demand automatically delivers additional processor capacity on a temporary basis within the system’s default Shared Processor Pool.

Utility Capacity on Demand concepts

The Utility Capacity on Demand (CoD) offering is for customers with unpredictable, short workload increases who need an automated and affordable way to help assure that adequate server resource is
available as needed.
When you add utility CoD processors, they are automatically placed in the default shared processor pool. These processors are available to any uncapped partition in any shared processor pool.
Entering Utility CoD enablement codes and reporting codes

Learn more about how to use the HMC to enter enablement and reporting codes.
To enter the Utility CoD enablement and reporting codes, do the following:
1. In the navigation area of the HMC window, expand Systems Management.
2. Select Servers.
3. In the contents area, select the server on which you want to enter your Utility CoD enablement or reporting code.
4. Select Tasks > Capacity on Demand > Enter CoD Code.
5. Type your enablement or reporting code in the Code field.
6. Click OK

Capacity BackUp:

Capacity BackUp uses On/Off Capacity on Demand (CoD) capabilities to provide an off-site,disaster-recovery server.

The Capacity BackUp offering has a minimum set of active processor cores that can be used for any workload and a large number of inactive processor cores that can be activated using On/Off CoD in the event of a disaster. A specified number of no-charge On/Off CoD processor days is provided with Capacity BackUp.

Make sure that you have prepared your server before continuing. 

You can test your Capacity BackUp activations while not incurring duplicate charges. 

PowerVM Editions (PowerVM)

PowerVM Editions (also referred to asPowerVM) is activated with a code, similar to the way that capacity is activated on IBM Systems and IBM eServer hardware.

When you purchase a PowerVM Editions feature, a code is provided that can be entered on the Hardware Management Console (HMC) to activate the technology. You can enter PowerVM activation codes by using the Integrated Virtualization Manager (IVM).

PowerVM Editions concepts

This information describes the virtualization technologies that are available.

The following virtualization technologies are available:

PowerVM is a Virtualization Engine technology that enables the system for the following features:

– Micro-Partitioning

– Virtual I/O Server

– Integrated Virtualization Manager

– Live Partition Mobility

– The ability to run x86 Linux applications on Power Systems

The following table describes the features each PowerVM Edition offers:

Read More

How to Install Wine 1.6.2 in RHEL/Fedora/CentOS/Ubuntu/Linux Mint ?

"Wine" is a free  open source application  for Linux operating system which will  enable running Windows applications on several POSIX-compliant operating systems, such as Linux, Mac OSX, & BSD.

Instead of simulating internal Windows logic like a virtual machine or emulator, Wine translates Windows API calls into POSIX calls on-the-fly, eliminating the performance and memory penalties of other methods and allowing you to cleanly integrate Windows applications into your desktop.

Benefits:

• Its uses all  possibilities of Unix Operating systems like stability, flexibility, remote administration
• Wine allows  to call windows application from shell scripts ,so as to make best usage of  Unix Systems.
• Wine makes it possible to access Windows applications remotely.
• Wine makes it economical to use thin clients
• Wine is Open Source Software, you can customise it as per your requirements

Wine comes with a software library Winelibthat lets the coders to compile Windows applications on Linux/Unix platform. Wine has now reached it's latest stable version- Wine 1.6.2.

Going forward , we will walk you through the procedure to install wine.

Download S/W: http://www.winehq.org/announce/1.6.2

Installing Wine 1.6.2 in RHEL, CentOS and Fedora:

1) Installing Dependency Packages
# yum -y groupinstall 'Development Tools'
# yum -y install libX11-devel freetype-devel

2) Downloading Wine 1.6.2
#wget http://prdownloads.sourceforge.net/wine/wine-1.6.2.tar.bz2

3)Extract & Installing Wine 1.6.2
$ tar -xvjf wine-1.6.2.tar.bz2

if its 32 bit

#cd wine-1.6.2/
#./tools/wineinstall

On 64-Bit Systems

$ cd wine-1.6.2/
$ ./configure --enable-win64
$ make
# make installRunning Wine 1.6.2

4) Run wine
$ wine notepad++.exe

Install Wine 1.6.2 in Ubuntu/Linux Mint

1) Add the PPA
# add-apt-repository ppa:ubuntu-wine/ppa

2) Update the local repository.
# apt-get update

3) Install the package.
# apt-get install wine1.6

Read More

Troubleshooting GPFS Issues

In this article ,we are going to discuss about most general methods of  GPFS issues troubleshooting.

When you got GPFS issue?

Got a problem? Don’t panic! 

Check for possible basic problems: 

• Is Network OK? 
• Check status of the cluster: “mmgetstate–a” 
• Check status of NSDs: “mmlsdisk fsname” 

Take a 5 min break  

• In major cases GPFS will recover by it self without need of any intervention from the administrator 

If not recovered 

• Ensure that you are the only person who is doing the work! 
• check gpfslogs (first on cluster manager, then on FS manager, then on NSD servers) 
• check syslog(/var/log/messages) for eventual errors 
• Check disks availability (mmlsdisk fsname) 
• Consult “Problem determination guide” 

Some usefull commands:

• “mmfsadm dump waiters” will help to find long lasting processes 
• “mmdiag --network|grep pending” helps to individuate non-responsive node 
• “mmdiag --iohist” lists last 512 I/O operations performed by GPFS on current node (helps to find malfunctioning disk) 
• “gpfs.snap” will garter all logs and configurations from all nodes in the cluster 
• the first thing to send to IBM support when opening service reques

GPFS V3.4 Problem Determination Guide:

• http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp

NFS stale file handle:

When a GPFS mount point is in the "NFS stale file handle" status, example 

[root@um-gpfs1 root]# df 

Filesystem 1K-blocks Used Available Use% Mounted on !

/dev/gpfs_um1 8125032448 8023801088 101231360 99% /storage/gpfs_um

df: `/storage/gpfs_um': Stale NFS file handle 

Then check if there is any NSD with status "down" 

[root@um-gpfs1 root]# mmlsdisk gpfs_um 

disk driver sector failure holds holds 

name type size group metadata data status availability 

------------ -------- ------ ------- -------- ----- ------------- ------------ 

disk21 nsd 512 4015 yes yes ready up !

disk22 nsd 512 4015 yes yes ready down !

disk23 nsd 512 4015 yes yes ready down !

disk24 nsd 512 4013 yes yes ready up !

restart the NSDs (important: do it for all NSD with status "down" in one command): 

[root@um-gpfs1 root]# mmchdisk gpfs_um start -d "disk21;disk24”

re-mount filesystems

Recovery of GPFS configuration:

If a node of the cluster lost its configuration (has been re-installed) but still present as member of this cluster

(“mmgetstate” lists it in “unknown” state) use this command to recover the node: 

/usr/lpp/mmfs/bin/mmsdrrestore -p diskserv-san-5 -R /usr/bin/scp

Checking existing NSD:

• If get this warning while creating new nsd Disk descriptor xxx system refers to an existing NSD 

Use this command to verify if this device is actually used in one of the file systems 

mmfsadm test readdescraw /dev/emcpowerax

Read More