Adding host info to ganglia and gmond

GridWay was not reporting OS name and OS version for mileva. After making sure gmond was running and MDS was actually being populated by available information (info on this is available in earlier post), you have to make sure all the information that is needed by GridWay is being provided by ganglia.
For example, I was noticing on the problem host that telnet localhost 8649 was giving the followgin output:
[METRIC NAME="sample-metric" VAL="Linux mileva-0-3.local 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
[METRIC NAME="sample-metric" VAL="Linux mileva-0-2.local 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

While output on working host was giving the following output:
METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="401" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/]
[METRIC NAME="sample-metric" VAL="Linux everest-0-3.local 2.4.21-20.ELsmp #1 SMP Sat Sep 18 18:28:16 PDT 2004 x86_64 x86_64 x86_64 GNU/Linux
[METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="943" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/]
[METRIC NAME="sample-metric" VAL="Linux everest-0-13.local 2.4.21-20.ELsmp #1 SMP Sat Sep 18 18:28:16 PDT 2004 x86_64 x86_64 x86_64 GNU/Linux

This means ganglia is not providing needed information, so do the following to include needed info. (Following was done on Ganglia v3.0.4)
Edit /etc/gmond.conf (this is the default location) file and add the following text toward the bottom of the document (among all the other similar entries):
metric {
name = "os_name"
value_threshold = 10.0
}

It is likely that os_release will need to be added in the same fashion as was described for os_name.

Edited config file (/etc/gmond.conf) needs to be copied to all the nodes of the cluster before it will take effect.

Then restart gmond (/etc/init.d/gmond restart) on all the nodes and wait for GridWay info to be updated...

Populating gwhost in GridWay

Source of information when executing gwhost command in GridWay comes from two major sources:
From $GLOBUS_LOCATION/libexec/globus-scheduler-provider-[sge OR fork] and $GLOBUS_LOCATION/etc/globus_wsrf_mds_usefulrp/gluerp.xml. The gluerp.xml script is the one that provides info about static resource info and it obtains infrom from a provider such as ganlgia (gmond process) or hawkeye. Information available from ganglia can be checked by executing: telnet localhost 8649
gluerp.xml needs to have a line such as the following enabled in order for gwhost to be properly populated:
.../...
java org.globus.mds.usefulrp.glue.GangliaElementProducer
.../...

The globus-scheduler-provider-[sge OR fork] script provides mostly dynamic information used for scheduling.

CLASSPATH on Windows with paths including spaces

For example:
javac -classpath ".;C:\Documents and Settings\afgane\My Documents\Enis\Projects\JaCoP\JaCoP.jar" *.java

More info/examples available here.

GRAM Authentication test failure

If getting this:
[afgane@everest00 afgane]$ globusrun -r everest.cis.uab.edu -a

GRAM Authentication test failure: connecting to the job manager failed. Possible reasons: job terminated, invalid job contact, network problems, ...

After making sure you have current grid proxy (through grid-proxy-init), check that the globus-gatekeeper is running by telneting to port 2119 by executing:
telnet [hostname] 2119

If you get an error such as the following, read on...
telnet localhost 2119
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused


2119 is globus-gatekeeper default port. If a different port is being used, you can check it by examining file usr/local/globus-4.0.2/etc/globus-gatekeeper.conf first.
Continue by examining /etc/xinetd.d/globus-gatekeeper. This is service startup file that looks something like this (make sure first line is giving the name of the service (i.e., globus-gatekeeper, in this case)):
service globus-gatekeeper
{
socket_type = stream
protocol = tcp
wait = no
user = root
env = LD_LIBRARY_PATH=/usr/local/globus-4.0.2/lib
server = /usr/local/globus-4.0.2/sbin/globus-gatekeeper
server_args = -conf /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf
disable = no
}

In this file find the globus-gatekeeper config file (e.g., line: server_args = -conf /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf) and then examine it next. This file looks something like this:
[root@everest00 xinetd.d]# cat /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf
-x509_cert_dir /etc/grid-security/certificates
-x509_user_cert /etc/grid-security/hostcert.pem
-x509_user_key /etc/grid-security/hostkey.pem
-gridmap /etc/grid-security/grid-mapfile
-home /usr/local/globus-4.0.2/
-e libexec
-logfile var/globus-gatekeeper.log
-port 2119
-grid_services etc/grid-services
-inetd

Here you can find which port the gatekeeper is running on and then go back to telnet.

If any changes were made to the two files just mentioned, you must restart xinetd. This is done as root by executing:
/etc/rc.d/init.d/xinetd restart
If this still does not work, execute: netstat -lt
This will print a list of all service currentl running. You can also try starting globus-gatekeeper manually by starting the server. Path and parameters for the server can be found in /etc/xinetd.d/globus-gatekeeper again under server and server_args (e.g., $ /usr/local/globus-4.0.2/sbin/globus-gatekeeper -conf /usr/local/globus-4.0.2/etc/globus-gatekeeper.conf)

If you can succesfully telnet into to machine, gatekeeper is running and next steps would include checking host certificates and making sure permissions are set correctly and that they are still valid (in /etc/grid-security/):
-rw-r--r-- 1 root root 1.4K Mar 8 13:50 hostcert.pem
-r-------- 1 root root 887 Mar 8 13:49 hostkey.pem

gridmap file needs to hold distinguished names of individual users that map to local user names. (e.g.: "/C=US/ST=Alabama/L=Birmingham/O=University of Alabama at Birmingham/OU=UABgrid/CN=jpr/emailAddress=jpr@uab.edu" afgane).

Finally, /etc/grid-security/certificates directory must hold currently valid CA certificates for participating resources/organizations.


Additional (excellent) documentation can be gotten from Georgia Tech at http://www.hpcc.ttu.edu/Globus.html and http://www.sdsc.edu/~tkaiser/globus/build/. Information on globus-personal-gatekeeper is also included at the second link.

Globus simple tests

Test GRAM authentication only:
globusrun -r everest.cis.uab.edu -a
GRAM Authentication test successful

Submit simple (non-WS) job:
globusrun -o -r everest.cis.uab.edu/jobmanager '&(executable=/bin/date)'

Using Threads in Java

The way threads work in Java is that there has to be a class implementing either Runnable interface of extends Thread class. Either way, run method must be overridden wihh desired functionality. From the client class, a new thread must be created (as below) where the first argument passed is an an instance of the class you want executed as a separate thread. Once this is done, the thread can be started by invoking start method.

Thread thrd = new Thread(new ClassImplementingThreadFunc(param1,param2));
Thread thrd2 = new Thread(new ClassImplementingThreadFunc(newParam1,newParam2));
thrd.start();
thrd2.start();

Numerical sort in Perl

If trying to sort a hash based on key where keys are numerical, using just
foreach $Key (sort keys %Output_Hash) will sort/output values as follows:
1
10
11
...
2
20
21
...

In order to get proper numerical sort, modify above code as this:
foreach $Key (sort {$a <=> $b} keys %Output_Hash)
This will make a comparison on each of the keys and result will be what is needed:
1
2
3
...
10
11

Change reference indentation in Endnote

To change indentation in references when using EndNote (v8). For example:
Change from this:
[1] The Grid: Blueprint for a New Computing Infrastructure, 1st ed: Morgan Kaufmann Publishers, 1998.
To this:
[1] [TAB] The Grid: Blueprint for a New Computing Infrastructure, 1st ed: Morgan Kaufmann Publishers, 1998.

Go to EndNote, Edit->Output style->Edit "style"
Go to Bibliography->Templates and change EVERY entry to include tab in front of Author field (insert tab by clicking on 'Insert Filed' on the top right and selecting 'Tab'). For example:
Journal Article
[TAB] Author, "Title," Journal|, vol. Volume|, pp. Pages|, Year|.

As a side note, in EndNote style editing window, Citations are referring to in-text citations, while Bibliography is referring to the References/Bibliography at the end of the document.

Using my code with Java packages

1. After having created Java packages, copy over the entire directory structure to desired machine (e.g., source files are in /home/afgane/dBLAST/src/com/dBLAST/util).
2. In order to compile the code, in base dir (e.g., /home/afgane/dBLAST), execute
javac -d bin/ src/com/dBLAST/util/*.java
The -d option specifies where to store class files. It will create all the package subdirectories from the specified folder (e.g., by specifying bin/, it will create bin/com/dBLAST/util/ and put all class files there).
3. In order to execute the code, first you must export directory where class files are. This can be done either by editing .bashrc file and adding a line such as the following one to it
export CLASSPATH=/home/afgane/dBLAST/bin/:$CLASSPATH
Alternatively, you can do it manually and executing that same line for each login session.
4. Finally, run the code by executing java com.dBLAST.util.DynamicBLAST_6 props.properties, for example from any directory.

Dynamic BLAST installation and execution requirements (v0.6)

Dynamic BLAST (v0.6)

Required libraries (these must be specified in CLASSPATH)
drmaa.jar (located in $dBLAST_LOCATION/lib)
JSAP-21.jar (located in $dBLAST_LOCATION/lib)
mysql-connector-java-5.0.7-bin.jar (located in $dBLAST_LOCATION/lib)

Required source files (package com.dBLAST.util)
DynamicBLAST_6.java
ReadProperties.java

Properties files
propsLS.properties
analysis.properties
[props.properties] (pre-configured properties file for BLAST runs)

External applications/scripts
analyze.pl
runCat.pl
runGWHOST.pl
querySplittingOnly_v2.pl

Input files
first100.fas


Compiling the application
While in base directory (e.g., /home/afgane/dBLAST), execute:
javac -d bin/ src/com/dBLAST/util/*.java
(-d option specifies where to store class files. It will create all the package subdirectories from the specified folder (e.g., by specifying bin/, it will
create bin/com/dBLAST/util/ and put all class files there))
There is also a compile.sh script that automates above process.

Invoking the application
Must first execute (if not provided in .bashrc):
export dBLAST_LOCATION=/home/afgane/dBLAST
export CLASSPATH=$dBLAST_LOCATION/bin/:$dBLAST_LOCATION/lib/JSAP-2.1.jar:$CLASSPATH
Usage: java com.dBLAST.util.DynamicBLAST_6 [(-p|--propertiesFile) ] [-g|--modeGrid] [(-q|--queryFile) ] [(-d|--database) ] [-f ] [-c|--clean] [-h|--help]

[(-p|--propertiesFile) ]
Propeties file provding job arguments and options. (default:
propsLS.properties)

[-g|--modeGrid]
If switch present, analysis and fragmentation are performed on GRID
resources rather than using LOCAL resource.

[(-q|--queryFile) ]
Query input file to be used for the job (if another value is present in
the job properties file, value associated with this switch overrides
it).

[(-d|--database) ]
Query input file to be used for the job (if another value is present in
the job properties file, value associated with this switch overrides
it).

[-f ]
Defines number of fragments to be created. Once created, they are
randomly distributed to chose resources.

[-c|--clean]
Clean temporary files (most of them).

[-h|--help]
Displays this help.

Porting EndNote references from ver. 7 to ver. 8

The source of the problem is that EndNote 8 and later export the data as a UTF-8 Unicode document, not as traditional ANSI text. Reference Manager, ProCite, and earlier versions of EndNote are not Unicode compatible, so they need the document to be plain ANSI text. To resolve this problem, do the following:

1. Export the EndNote 8 or later library to a plain text file (.txt) using the Refer Export format:
a. Open the library in EndNote 8 or later and make sure all references are showing.Select "Refer Export" as the output style by going to Edit->Output Styles (if Refer Export is not available, add it by clicking on Open Style Manager and selecting it for edit. Then, instead of editing it, just exit that form)
b. Select File->Export..., enter a file name, and save the file to a convenient location such as your desktop as plain .txt file.

2. Open the text file you exported from EndNote in Microsoft Word. When Word prompts you for the encoding option, select "Other Encoding" and "Unicode (UTF-8)" as the options.

3. In Word select "File > Save As" and save the document with a new name as plain text with Windows-default as the encoding option.

4. Import the Refer Export text file into the earlier version of EndNote:
a. In the older version of EndNote create a new library or open an existing one. Select File->Import. Click the "Choose File" button and select the text file you just created. Under "Import Option" select "Refer/BibIX"
b. Click the "Import" button.

More documentation available here and here.

Installing Mozart Oz and Cygwin on Windows XP

1. Mozart-Oz needs to be installed using typical Windows installation procedure
2. Cygwin needs to be installed using typical Windows installation (this is bare bones, so install packages such as vi and gcc - also, 'more' command does not work in cygwin so create alias in .bashrc file to use 'less' command then typing 'more'). Windows drives can be accessed from cygwin by going to /cygdrive directory.
3. Emacs needs to be installed. Install Windows version (not the cygwin package).
4. Run addpm.exe in emacs bin subdirectory from windows explorer. This will integrate/install emacs into the Windows environment.
5. Create symbolic link in /usr/bin to point to emacs (e.g., ls -s /cygdrive/c/Program\ Files/emacs/bin/emacs.exe emacs)
6. Define environment variable OZEMACS in Windows (Ctlr Panel->System->Advanced->Env. Var) to point to emacs.exe.
7. Start cygwin and type oz to get going...

Add second y axis on a chart in Excel

In Excel 2007:
Plot two sets of data on a single chart.
Then, you must select data series you wish to plot on the second axis by either clicking on it on the chart or selecting it from a drop down menu in top left corner under Layout tab. Then click 'Format Selection' under Layout tab and select "Plot on Secondary Axis".

In Excel 2003 and earlier:
Select chart, press Ctrl+1 and proceed with data selection...

Note: that not all charts support this feature.

Format elapsed time in excel

I have a set of times saved in seconds and fractions of a second. I want to be able to transpose that into something that can be understood easily, such as hours:minutes:seconds
Because Excel does not provide automatic support to measure elapsed time, this unfortunately needs to be done by hand.
Sample input data set (in columns A1-A4):

4508.507

4452.111

4423.425

4592.724

Sample output:

1:15:8.507

1:14:12.111

1:13:43.425

1:16:32.724

Formula (for first value only (i.e., in cell A1)
=INT(A1/3600)&":"&RIGHT("00"&MOD(INT(A1/60),60),2)&":"&ROUND(MOD(A1, 60),3)

Cell value (e.g., A1) can be replaced with things such as SUM (cell range), AVERAGE (cell range), MAX (cell range) to get formatting of the result of processing a range of cell values.

Formatting database for BLAST

Unformatted databases can be downloaded from ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/, while formatted ones from ftp://ftp.ncbi.nlm.nih.gov/blast/db/.
Command to use to format a database:
formatdb -i input_db -p F -o T for nucleotide
formatdb -i input_db -p T -o T for protein

For yeast.nt use: formatdb -i yeast.nt -p T -o T
For nr use: formatdb -i nr -p T -o T

Trying to submit a simple job to new installation of Globus, I got the following error:

globusrun-ws -F wave.cis.uab.edu -submit -s -c /bin/date
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:a8109b36-c203-11db-a449-00123f2a7564
Termination time: 02/22/2007 23:31 GMT
Current job state: Failed
Destroying job...Done.
Cleaning up any delegated credentials...Done.
globusrun-ws: Job failed: Error code: 201
Script stderr:
sudo: sorry, you must have a tty to run sudo


If sudoers file has properly been edited and contains necessary lines and this happend, it is due to default Fedora sudoers setup. Edit sudoers file (Command: /usr/sbin/visudo -f /etc/sudoers) and comment out the following line:
Defaults requiretty

Installing Globus Toolkit (4.0.3) Gotchas

After following the installation documentation for installing GT 4.0.3 from source, I kept getting the following error during the make part of installation: db.c:40:17: error: sql.h: No such file or directory

(all this as root, except GT install of course)
This required installing iODBC driver (including driver manager RPM and RPM Developers Kit) from here (use command: rpm -ivv rpmName).
Also, psqlODBC driver needs to be installed. Download the source and follow the standard installation procedure:

tar xvzf psqlodbc-[version].tar.gz
cd psqlodbc-[version]
./configure --with-iodbc --enable-pthreads
make
make install

Then, run globus related ./configure and make

MediaWiki - changing sidebar menu

Found solution here by Tom Sherman.

Gist of it is the following though (in case original page moves):
"On the off chance that you’re here because you actually want to modify your wiki’s navigation links (and not just read my complaining), I recommend that you modify the existing MediaWiki URLs and not try to create new ones. That’s what I did. Simple instructions:

1. Go to Special:Allmessages.
2. Choose a link you’ll modify. I decided I didn’t want the “Community portal” link, so that meant I would modify the:
* portal page [MediaWiki:portal] (do an in-page find on the Special:Allmessages page). Modify the existing text to change what will be rendered in the menu as the link text.
* portal-url page [MediaWiki:portal-url]. Modify the existing text to change what will be rendered in the menu as the link URL (can be internal wiki page or external URL).
3. After you’ve edited, make sure to do a full-refresh of your the page or clear your browser cache. MediaWiki is very finicky about this.

Remember, you have the choice of any of the existing links to modify. Here they are:
link text | link URL
mainpage | mainpage
portal | portal-url
currentevents | currentevents-url
recentchanges | recentchanges-url
randompage | randompage-url
help | helppage
sitesupport | sitesupport-url

Even though the pages I modified were “portal” and “portal-url,” they now have nothing to do with that topic. Those are simply the references in the database."