Populating gwhost in GridWay

Source of information when executing gwhost command in GridWay comes from two major sources:
From $GLOBUS_LOCATION/libexec/globus-scheduler-provider-[sge OR fork] and $GLOBUS_LOCATION/etc/globus_wsrf_mds_usefulrp/gluerp.xml. The gluerp.xml script is the one that provides info about static resource info and it obtains infrom from a provider such as ganlgia (gmond process) or hawkeye. Information available from ganglia can be checked by executing: telnet localhost 8649
gluerp.xml needs to have a line such as the following enabled in order for gwhost to be properly populated:
.../...
java org.globus.mds.usefulrp.glue.GangliaElementProducer
.../...

The globus-scheduler-provider-[sge OR fork] script provides mostly dynamic information used for scheduling.

GRAM Authentication test failure

If getting this:
[afgane@everest00 afgane]$ globusrun -r everest.cis.uab.edu -a

GRAM Authentication test failure: connecting to the job manager failed. Possible reasons: job terminated, invalid job contact, network problems, ...

After making sure you have current grid proxy (through grid-proxy-init), check that the globus-gatekeeper is running by telneting to port 2119 by executing:
telnet [hostname] 2119

If you get an error such as the following, read on...
telnet localhost 2119
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused


2119 is globus-gatekeeper default port. If a different port is being used, you can check it by examining file usr/local/globus-4.0.2/etc/globus-gatekeeper.conf first.
Continue by examining /etc/xinetd.d/globus-gatekeeper. This is service startup file that looks something like this (make sure first line is giving the name of the service (i.e., globus-gatekeeper, in this case)):
service globus-gatekeeper
{
socket_type = stream
protocol = tcp
wait = no
user = root
env = LD_LIBRARY_PATH=/usr/local/globus-4.0.2/lib
server = /usr/local/globus-4.0.2/sbin/globus-gatekeeper
server_args = -conf /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf
disable = no
}

In this file find the globus-gatekeeper config file (e.g., line: server_args = -conf /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf) and then examine it next. This file looks something like this:
[root@everest00 xinetd.d]# cat /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf
-x509_cert_dir /etc/grid-security/certificates
-x509_user_cert /etc/grid-security/hostcert.pem
-x509_user_key /etc/grid-security/hostkey.pem
-gridmap /etc/grid-security/grid-mapfile
-home /usr/local/globus-4.0.2/
-e libexec
-logfile var/globus-gatekeeper.log
-port 2119
-grid_services etc/grid-services
-inetd

Here you can find which port the gatekeeper is running on and then go back to telnet.

If any changes were made to the two files just mentioned, you must restart xinetd. This is done as root by executing:
/etc/rc.d/init.d/xinetd restart
If this still does not work, execute: netstat -lt
This will print a list of all service currentl running. You can also try starting globus-gatekeeper manually by starting the server. Path and parameters for the server can be found in /etc/xinetd.d/globus-gatekeeper again under server and server_args (e.g., $ /usr/local/globus-4.0.2/sbin/globus-gatekeeper -conf /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf)

If you can succesfully telnet into to machine, gatekeeper is running and next steps would include checking host certificates and making sure permissions are set correctly and that they are still valid (in /etc/grid-security/):
-rw-r--r-- 1 root root 1.4K Mar 8 13:50 hostcert.pem
-r-------- 1 root root 887 Mar 8 13:49 hostkey.pem

gridmap file needs to hold distinguished names of individual users that map to local user names. (e.g.: "/C=US/ST=Alabama/L=Birmingham/O=University of Alabama at Birmingham/OU=UABgrid/CN=jpr/emailAddress=jpr@uab.edu" afgane).

Finally, /etc/grid-security/certificates directory must hold currently valid CA certificates for participating resources/organizations.


Additional (excellent) documentation can be gotten from Georgia Tech at http://www.hpcc.ttu.edu/Globus.html and http://www.sdsc.edu/~tkaiser/globus/build/. Information on globus-personal-gatekeeper is also included at the second link.

Globus simple tests

Test GRAM authentication only:
globusrun -r everest.cis.uab.edu -a
GRAM Authentication test successful

Submit simple (non-WS) job:
globusrun -o -r everest.cis.uab.edu/jobmanager '&(executable=/bin/date)'

Trying to submit a simple job to new installation of Globus, I got the following error:

globusrun-ws -F wave.cis.uab.edu -submit -s -c /bin/date
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:a8109b36-c203-11db-a449-00123f2a7564
Termination time: 02/22/2007 23:31 GMT
Current job state: Failed
Destroying job...Done.
Cleaning up any delegated credentials...Done.
globusrun-ws: Job failed: Error code: 201
Script stderr:
sudo: sorry, you must have a tty to run sudo


If sudoers file has properly been edited and contains necessary lines and this happend, it is due to default Fedora sudoers setup. Edit sudoers file (Command: /usr/sbin/visudo -f /etc/sudoers) and comment out the following line:
Defaults requiretty

Installing Globus Toolkit (4.0.3) Gotchas

After following the installation documentation for installing GT 4.0.3 from source, I kept getting the following error during the make part of installation: db.c:40:17: error: sql.h: No such file or directory

(all this as root, except GT install of course)
This required installing iODBC driver (including driver manager RPM and RPM Developers Kit) from here (use command: rpm -ivv rpmName).
Also, psqlODBC driver needs to be installed. Download the source and follow the standard installation procedure:

tar xvzf psqlodbc-[version].tar.gz
cd psqlodbc-[version]
./configure --with-iodbc --enable-pthreads
make
make install

Then, run globus related ./configure and make

Trusting UAB CA on a UABGrid resource

There are four (4) things that need to be changed on every resource to get the UAB CA certificate where given resource can be used for job submission and as a job submission source (NOTE: all the steps except step 3 need to be done for every user on every resource):

1. Download userkey.pem and usercert.pem from http://uabgrid.uab.edu (click on Other Tools -> Manage Certificates)
2. Save these two files to given resource in ~/.globus directory and change userkey.pem permissions to read only.
3. As root, save the 56498486.signing_policy and UAB-root.crt
   to /etc/grid-security/certificates directory on given resource.
   Those files can be obtained with the following set of commands:
   cd /etc/grid-security/certificates
   wget http://webapp.lab.ac.uab.edu/UAB-root.crt
   wget https://www.pki.virginia.edu/nmi-bridge/certs/56498486.signing_policy
   ln -s UAB-root.crt 56498486.0
4. As root, add an entry to grid-mapfile - this entry can be obtained by executing grid-proxy-info by particular user (after the previous steps have been completed) and then copy the issuer line into grid-mapfile (e.g.: /C=US/ST=Alabama/L=Birmingham/O=University of Alabama at Birmingham/OU=UABgrid/CN=afgane/emailAddress=afgane@uab.edu)

Submit a simple WS-GT4 job

Must have valid proxy and execute the following command: globusrun-ws -s -submit -F olympus.cis.uab.edu -c /bin/date

Setting up GT4 MDS hierarchy

As user globus, edit file $GLOBUS_LOCATION/etc/globus_wsrf_mds_index/hierarchy.xml
to include the line with the server higher up in the hierarchy (i.e., below ...upstream... element). For example, to have olympus as head of the hierarchy, add the following line on everest:
https://olympus.cis.uab.edu:8443/wsrf/services/DefaultIndexService
This needs to be done on every machine you want to have register with the higher level resource.
There is also an option to register resources from higher levels and not have to edit files on all individual resources but rather only in one place. To use this option, add the following line (i.e., below ...downstream... element in above stated file) to the machine acting as the head of hierarchy (e.g., olympus): https://everest.cis.uab.edu:8443/wsrf/services/DefaultIndexService

Once the hierarchy file has been updated, globus needs to be restarted on that machine. To restart globus, as root, execute: /etc/init.d/globus-4.0.2 restart


To check if everything works, execute: wsrf-query -s https://olympus.cis.uab.edu:8443/wsrf/services/DefaultIndexService '/*' grep 138.26.64.200 wc -l7 as regular user from any machine (e.g., as user afgane - Note that grid-proxy must be valid to work!). Above command checks for everest's MDS report through olympus.

To check if WebMDS works too, as globus start tomcat on olympus: $CATALINA_HOME/bin/startup.sh and point the browser to: http://olympus.cis.uab.edu:8080/webmds/

Full information on how do to this can be obtained form here.