Use wget to recursively get contents of a directory

Sometimes, it's just necessary to explore the directory space on a remote dir to know what are the specific keys that can be queried individually.
This is particularly convenient in the case of cloud instance metadata where one would like to find out what are all the metadata keys that are available for a given instance on a given cloud.

So, to download the complete directory tree recursively, use the following command (this example uses the 'default' cloud instance metadata server):

$ wget -rk Is http://169.254.169.254/latest/meta-data/

OS X top command: sort by column

 After starting top, sort by column names by typing:

o<key>         Set primary sort key to : [+-]{command|cpu|pid
               |prt|reg|rprvt|rshrd|rsize|th|time|uid|username|vprvt
               |vsize}
O<skey>        Set secondary sort key to 

Field descriptions
command: command name
cpu: CPU usage
prt: number of Mach ports
reg: number of memory regions
rpvrt: resident (physical) private address space
rshrd: resident (physical) shared memory space
rsize: resident (physical) memory size
th: number of threads
time: execution time
uid: user ID
vprvt: private address space size
vsize: total virtual memory size

Create a .mo file from .po file for i18n

Quite simple (on a OS X 10.6, at least), simply run:
msgfmt -o [output_file].mo [input_file].po

Profile SGE queues

As root, run the command "qconf -tsm" on your head node. That will cause a one-time scheduler profiling run to be dumped to a text file called "schedd_runlog" in your $SGE_ROOT/ $SGE_CELL/common/ directory. This file can then be examined for any anomalies if things are otherwise not running smoothly.

Replace entire line that contains given text

I need to replace an entire line that contains a certain string with another line. This is especially useful in case of editing configuration files that have predefined fields but values may be different.


Say the string you are looking for is STRING_TO_BE_REPLACED.
And the line you want to insert is LINE_TO_REPLACE_WITH.
 
sed 's/^.*STRING_TO_BE_REPLACED.*$/LINE_TO_REPLACE_WITH/'  input_file > new_file

The ^ is a start of line anchor, the '.*' is a wildcard, the $ is a end of line anchor.

Prepend every line in a file with some text

Say you have a file 'files.txt' looking like this:
p1-c-b-1.fastq.gz
p1-c-b-2.fastq.gz
p1-c-c-1.fastq.gz

Now, you would like to add a full URL path to each of those lines, prepending existing text.
Use awk as follows:
$ cat files.txt | awk '{ print "http://www.complete.url.com/", $0; }' > links.txt

You get a file names 'links.txt' looking like this:
http://www.complete.url.com/ p1-c-b-1.fastq.gz
http://www.complete.url.com/ p1-c-b-2.fastq.gz
http://www.complete.url.com/ p1-c-c-1.fastq.gz


To get rid of introduced extra white space, use sed:
$ cat links.txt | sed 's/ //g'

Monitor network traffic on OpenSolaris from command line

To check a summary of network activity for a NIC, use:
# netstat -i 1
    input   xnf0      output       input  (Total)    output
packets errs  packets errs  colls  packets errs  packets errs  colls
1111615 0     745655  0     0      1111684 0     745724  0     0    
399     0     306     0     0      399     0     306     0     0    
416     0     343     0     0      416     0     343     0     0 

---

To expolore actual packets crossing the network, use snoop command. It captures both TCP and UDP traffic. It is a tool that is shipped with Solaris.

Here is sample output:
# snoop
Using device xnf0 (promiscuous mode)
cbcb-vs.umiacs.umd.edu -> domU-12-31-39-04-EC-47.compute-1.internal TCP D=60088 S=8021 Ack=4204280156 Seq=2401233531 Len=1460 Win=92
domU-12-31-39-04-EC-47.compute-1.internal -> cbcb-vs.umiacs.umd.edu TCP D=8021 S=60088 Ack=2401246671 Seq=4204280156 Len=0 Win=49640 Options=
domU-12-31-39-04-EC-47.compute-1.internal -> dhcp243113.rollins.emory.edu TCP D=49769 S=22 Push Ack=4234202219 Seq=2467318705 Len=192 Win=49232



It shows source and destination servers, type of traffic, source and destination ports, as well as some packet info.

Add folder to Places sidebar in Mac Finder

Very simple, navigate to the folder you desire to have listed under Places in the Finder, select it and press Cmd+T. That's it!
To remove the folder from the list, right click it in the sidebar and select "Remove from sidebar".

Replace multiple spaces/tabs with single space

Use following UNIX command to replace multiple spaces or tabs with a single space, thus enabling easier work with cut command:

tr -s ''

Adding new disk to ZFS

This is a short and adjusted version of article available here.
When initially instantiating and setting up accounts on ZFS filesystem (in my case, under OpenSolaris 2009.06), it is beneficial to store user data on a pool other than root pool (i.e., rpool). This is because rpool cannot be composed from multiple hard disks and thus cannot be changed in size (unless it is mirrored) [ref].

So, start by adding a disk to the machine and creating a pool from it.
The following commands lists names of disks available in the system:
afgane@opensolaris:~$ pfexec format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c7d0
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c7d1
/pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0
2. c8d1
/pci@0,0/pci-ide@1,1/ide@1/cmdk@1,0
Specify disk (enter its number): ^C
afgane@opensolaris:~$

From this, we see that the above system has three disks: c7d0, c7d1 and c8d1
NOTE: Instead of creating a filesystem on a physical disk, you may create it from a simple file that exists in current file system. To do this, create a file on the system of desired size and use it instead of the disk name. Use the following command:

afgane@opensolaris:~$ dd if=/dev/zero of=myfile bs=1024 count=10

This command creates a file called "myfile" filled with zeros with a block size of 1000 multiplied 10 tens or 10K (i.e., created file of size 10K bytes).

Before seeing which of the disks is being used by which pool, we need to see what pools are available on the machine. Do so by issuing the following command:
root@opensolaris:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
rpool 34.8G 3.79G 31.0G 10% ONLINE -
root@opensolaris:~#

From this output we can see that there is only one pool available on the machine, namely rpool.

Next, in order to see which disk is being used by rpool, issue the following command:
afgane@opensolaris:~$ zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c7d0s0 ONLINE 0 0 0

errors: No known data errors
afgane@opensolaris:~$

This tells us that disk named c7d0s0 is used by rpool.

We can now create a new pool and add one of the two extra disks that are available in the machine to this newly created pool. Do so by executing following command (note that you must be root to issue this command):
root@opensolaris:~$ zpool create myPool c7d1

We can now see our newly created pool and can access it as a regular directory on the machine:
root@opensolaris:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
myPool 1.95G 32.7M 1.92G 1% ONLINE -
rpool 34.8G 3.79G 31.0G 10% ONLINE -
root@opensolaris:~#
root@opensolaris:~# cd /myPool

If you ever add another disk to this machine, it is very simple to add it to the overall pool of available space. We will use the third disk that is available in this machine, as follows (remember that you need to be root to issue this command):
root@opensolaris:~# zpool add myPool c8d1

And here is the list of the new pool arrangement, showing the increased pool size:
root@opensolaris:~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
myPool 2.95G 32.7M 2.91G 1% ONLINE -
rpool 34.8G 3.79G 31.0G 10% ONLINE -
root@opensolaris:~#

Add multiple linux timing results (output in seconds)

If there are multiple files with timing results resulting from Linux 'time' command:
$ cat outputFile* | grep real
real 4m5.047s
real 2m51.264s
real 2m52.414s
real 2m52.293s

And these timings need to be converted into seconds and added into a single number (e.g., 761.018), use the following command:
cat outputFile* | grep real | cut -f 2 | sed 's/m/*60+/' | sed 's/s//g' | bc | awk '{ printf "%s", $0 "+" }' | sed '$s/.$/\n/' | bc

Convert output of linux 'time' command to show seconds only

After executing Linux 'time' command, the following output is obtained:
real 2m52.293s
user 1m39.670s
sys 0m4.370s

Use the following command to extract value for 'real' time only and convert the output to seconds only (i.e., convert 2m52.293s into 172.293):
cat outputFile* | grep real | cut -f 2 | sed 's/m/*60+/' | sed 's/s//g' | bc

Running JaCoP

From command prompt, while residing in the same directory as all the JaCoP java files, compile all java files using the following command:
C:\Documents and Settings\afgane\My Documents\Enis\Projects\JaCoP>java
c -classpath ".;C:\Documents and Settings\afgane\My Documents\Enis\Projects\JaCo
P\JaCoP.jar" *.java

Run any of the files using the following command, for example:
C:\Documents and Settings\afgane\My Documents\Enis\Projects\JaCoP>java
-classpath ".;C:\Documents and Settings\afgane\My Documents\Enis\Projects\JaCoP
\JaCoP.jar" Queens

After compiling it from command line, it can be run directly from Eclipse too.

Formatting database for BLAST

Unformatted databases can be downloaded from ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/, while formatted ones from ftp://ftp.ncbi.nlm.nih.gov/blast/db/.
Command to use to format a database:
formatdb -i input_db -p F -o T for nucleotide
formatdb -i input_db -p T -o T for protein

For yeast.nt use: formatdb -i yeast.nt -p T -o T
For nr use: formatdb -i nr -p T -o T

Acces MySQL on titanic

Most of my databases are stored under user mpiblast. To access one, type mysql -d (e.g., test)