bio386, Tue, salary example

Go over salary example. Ask students to follow along. Some students were able to modify my code and apply it to different variables.
The EdX R course also prepared the students for R syntax.

Correlationhttps://class.coursera.org/rprog-032/forum/thread?thread_id=265

Announcement: Skip "program 2 assignment" on Coursera's R course.

youtube:
https://youtu.be/wrM2uA0NGNs

bio233 gram stain lab, Monday

Gram stain lab.

Sunday: 
I prepared the lab on Sunday.

Items: 
Distilled water

·      Slides, cover slips

·      Bunsen burners and lighter

·      Inoculating loops

·      Microscope (optional oil-immersion lenses, oil cleaner)

·      Lens wipers

·      Bibulous papers (Absorbent papers)

·      Wax pencil

·      Crystal violet

·      Gram iodine

·      Safranin 

·      95% EtOH

·      75% EthOH

·      Dropping pipettes

·      Shape collection box

·      Staining rack

·      Clips for staining

·      White paper towel to absorb the runoff

·      Kim wipes

Lab coat is recommended

Microscope with vertical adaptoer, smart phone stages. 

Monday  gram stain lab, 2pm-4:40pm

Many students did not know how to use microscope. 

There is a shortage of microscope with vertical adapters. 

flow cytometer models

http://www.aceabio.com/novocyte/novocyte-index.html

Accuri C6

http://www.emdmillipore.com/US/en/life-science-research/cell-analysis/flow-cytometry-instrumentation/9Iyb.qB.31gAAAFMiMw0iy0I,nav

bio233 20150917Thu go over problem sets

bio233 20150917Thu go over problem sets

Go over youtube tutorial on screen cast. Bonus points will be given to students for self-recording during exam.

bio386 20150917Thu, coursera coding assignment part1

Problem: Almost all students did not watch the coursera videos. they seem to expect to finish the assignment without going over the lecture videos.

bio386 20150917Thu, coursera coding assignment part1

MByant 20150916

Convert PDF into WORD.
MB manually copy-pasted Table 2 into an Excel file.

Get FASTA sequences using NCBI Gene ID

echo -e "19084\n112407\n18113" | while read G; do curl -s "http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=gene&db=protein&id=${G}" | grep -A 1 "<Link>" | grep "<Id>" | cut -d '>' -f 2 | cut -d '<' -f 1 | while read S ; do curl -s "http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=protein&id=${S}&retmode=text&rettype=fasta" ; done;  done

See https://www.biostars.org/p/52652/

HQ tested with 1029 and it returned the correct sequences. 

aging genomics resource

http://genomics.senescence.info/

http://pcwww.liv.ac.uk/~aging/publications.html

*** greenfield, netaging.pbs running results, each run is 6.5 SU

Conclusions: Each run is 8:30pm - 3am = 6.5 hours. 

20140914 8:30pm :: qsub netaging.pbs 

[hqin2@greenfield ms02GINPPI]$ cat netaging.pbs

#!/bin/bash

# Request 15 cores

#PBS -l nodes=1:ppn=15

#  Request 7days  of cpu time

#PBS -l walltime=7:00:00:00

module load R/3.2.1-mkl

echo hostname

pwd

cd /crucible/mc48o9p/hqin2/mactower-network-failure-simulation-master/ms02GINPPI

pwd

numactl -C +0 R --vanilla --slave -f net-aging-sim-20150915.R --args 92 92 0.02 1.0 2000  &

numactl -C +1 R --vanilla --slave -f net-aging-sim-20150915.R --args 93 93 0.02 1.0 2000  &

numactl -C +2 R --vanilla --slave -f net-aging-sim-20150915.R --args 94 94 0.02 1.0 2000  &

numactl -C +3 R --vanilla --slave -f net-aging-sim-20150915.R --args 95 95 0.02 1.0 2000  &

numactl -C +4 R --vanilla --slave -f net-aging-sim-20150915.R --args 96 96 0.02 1.0 2000  &

numactl -C +5 R --vanilla --slave -f net-aging-sim-20150915.R --args 97 97 0.02 1.0 2000  &

numactl -C +6 R --vanilla --slave -f net-aging-sim-20150915.R --args 98 98 0.02 1.0 2000  &

numactl -C +7 R --vanilla --slave -f net-aging-sim-20150915.R --args 99 99 0.02 1.0 2000  &

numactl -C +8 R --vanilla --slave -f net-aging-sim-20150915.R --args 100 100 0.02 1.0 2000  &

numactl -C +9 R --vanilla --slave -f net-aging-sim-20150915.R --args 101 101 0.02 1.0 2000  &

numactl -C +10 R --vanilla --slave -f net-aging-sim-20150915.R --args 102 102 0.02 1.0 2000  &

numactl -C +11 R --vanilla --slave -f net-aging-sim-20150915.R --args 103 103 0.02 1.0 2000  &

numactl -C +12 R --vanilla --slave -f net-aging-sim-20150915.R --args 92 92 0.01 1.0 2000  &

wait

[hqin2@greenfield ms02GINPPI]$ ll -th dipgin.ms02.output/9[2-9]/popages/*2000*

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:55 dipgin.ms02.output/96/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025515.txt

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:55 dipgin.ms02.output/95/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025509.txt

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:54 dipgin.ms02.output/98/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025456.txt

-rw-r--r-- 1 hqin2 mc48o9p 34K Sep 15 02:54 dipgin.ms02.output/97/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025451.txt

-rw-r--r-- 1 hqin2 mc48o9p 34K Sep 15 02:54 dipgin.ms02.output/94/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025442.txt

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:54 dipgin.ms02.output/99/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025409.txt

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:52 dipgin.ms02.output/92/popages/cutoff.4.p.1.lambda.0.01.popsize.2000.time.2015Sep15_025240.txt

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:52 dipgin.ms02.output/93/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025211.txt

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:50 dipgin.ms02.output/92/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025058.txt

[hqin2@greenfield ms02GINPPI]$ ll -th dipgin.ms02.output/10[234]/popages/*2000*

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:54 dipgin.ms02.output/102/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025438.txt

-rw-r--r-- 1 hqin2 mc48o9p 33K Sep 15 02:53 dipgin.ms02.output/103/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Sep15_025353.txt

[hqin2@greenfield ms02GINPPI]$ cat netaging.pbs.o2350 | grep 2000 | grep "1.0"

[1] "101"  "101"  "0.02" "1.0"  "2000"

 "100"  "100"  "0.02" "1.0"  "2000"

[1] "94"   "94"   "0.02" "1.0"  "2000"

[1] "92"   "92"   "0.02" "1.0"  "2000"

 "103"  "0.02" "1.0"  "2000"

 "102"  "0.02" "1.0"  "2000"

 "1.0"  "2000"

[1] "97"   "97"   "0.02" "1.0"  "2000"

 "1.0"  "2000"

[1] "95"   "95"   "0.02" "1.0"  "2000"

[1] "96"   "96"   "0.02" "1.0"  "2000"

 "99"   "99"   "0.02" "1.0"  "2000"

 "92"   "92"   "0.01" "1.0"  "2000"

Genie technology

Genie technology and broaden participation

bio233 20150915Tue ch4 go over exam 1, part2 of fall 2014

Retrurn central dogma drawing back to students

Exam preparation

http://hongqinlab.blogspot.com/2015/09/bio233-ch4-mole-biol.html
http://hongqinlab.blogspot.com/2014/03/bio233-ch6-dna-helices-bases-3d.html
http://hongqinlab.blogspot.com/2014/09/bio233-20140930-ch5-continued-and-ch6.html
http://hongqinlab.blogspot.com/2015/01/bio125-tue-jan-20-2015-dna-structure.html

bio386, week 1, programming assignment

Problem: One student try to read in a file in a directory. "directory/file_name"

The first programming assignment of the Coursera is very challenging. Students started to have trouble to complete the programming assignment.

(in progress)(to do) network aging popSize=2000

[hqin2@greenfield ms02GINPPI]$ cat netaging.pbs 

#!/bin/bash

# Request 15 cores

#PBS -l nodes=1:ppn=15

#  Request 2:30 minutes of cpu time

#PBS -l walltime=7:00:00:00

module load R/3.2.1-mkl

echo hostname

pwd

cd /crucible/mc48o9p/hqin2/mactower-network-failure-simulation-master/ms02GINPPI

pwd

numactl -C +0 R --vanilla --slave -f net-aging-sim-20150915.R --args 92 92 0.02 1.0 2000  &

numactl -C +1 R --vanilla --slave -f net-aging-sim-20150915.R --args 93 93 0.02 1.0 2000  &

numactl -C +2 R --vanilla --slave -f net-aging-sim-20150915.R --args 94 94 0.02 1.0 2000  &

numactl -C +3 R --vanilla --slave -f net-aging-sim-20150915.R --args 95 95 0.02 1.0 2000  &

numactl -C +4 R --vanilla --slave -f net-aging-sim-20150915.R --args 96 96 0.02 1.0 2000  &

numactl -C +5 R --vanilla --slave -f net-aging-sim-20150915.R --args 97 97 0.02 1.0 2000  &

numactl -C +6 R --vanilla --slave -f net-aging-sim-20150915.R --args 98 98 0.02 1.0 2000  &

numactl -C +7 R --vanilla --slave -f net-aging-sim-20150915.R --args 99 99 0.02 1.0 2000  &

numactl -C +8 R --vanilla --slave -f net-aging-sim-20150915.R --args 100 100 0.02 1.0 2000  &

numactl -C +9 R --vanilla --slave -f net-aging-sim-20150915.R --args 101 101 0.02 1.0 2000  &

numactl -C +10 R --vanilla --slave -f net-aging-sim-20150915.R --args 102 102 0.02 1.0 2000  &

numactl -C +11 R --vanilla --slave -f net-aging-sim-20150915.R --args 103 103 0.02 1.0 2000  &

numactl -C +12 R --vanilla --slave -f net-aging-sim-20150915.R --args 92 92 0.01 1.0 2000  &

wait

[hqin2@greenfield ms02GINPPI]$ qsub netaging.pbs 

2350.greenfield.psc.xsede.org

[hqin2@greenfield ms02GINPPI]$ qstat

Job ID                    Name             User            Time Use S Queue

------------------------- ---------------- --------------- -------- - -----

2104.greenfield            newRunNew.job    ismini          134:54:1 R batch          

2314.greenfield            runjob           hchen7          21:41:25 R batch          

2350.greenfield            netaging.pbs     hqin2                  0 R batch          

[hqin2@greenfield ms02GINPPI]$ qstat

Job ID                    Name             User            Time Use S Queue

------------------------- ---------------- --------------- -------- - -----

2104.greenfield            newRunNew.job    ismini          134:54:1 R batch          

2314.greenfield            runjob           hchen7          21:41:25 R batch          

2350.greenfield            netaging.pbs     hqin2                  0 R batch          

[hqin2@greenfield ms02GINPPI]$ 

[hqin2@greenfield ms02GINPPI]$ 

[hqin2@greenfield ms02GINPPI]$ 

[hqin2@greenfield ms02GINPPI]$ qstat -a

greenfield.psc.xsede.org: 

                                                                                  Req'd    Req'd       Elap

Job ID                  Username    Queue    Jobname          SessID  NDS   TSK   Memory   Time    S   Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2104.greenfield.psc.xs  ismini      batch    newRunNew.job     83468     1     30    --  150:00:00 R 116:11:48

2314.greenfield.psc.xs  hchen7      batch    runjob           139085     1     15    --   24:00:00 R  21:41:55

2350.greenfield.psc.xs  hqin2       batch    netaging.pbs      48203     1     15    --  168:00:00 R  00:00:12

[hqin2@greenfield ms02GINPPI]$ 

backup files from crucial to my data directory (in progress)

$ tar cf /arc/users/hqin2/ms02GINPPI.tar ms02GINPPI/ & 

[1] 17265

[hqin2@greenfield mactower-network-failure-simulation-master]$ ps

  PID TTY          TIME CMD

 1519 pts/10   00:00:00 bash

17265 pts/10   00:00:00 tar

17280 pts/10   00:00:00 ps

There some runtime errors during the tar operation. Maybe some simulation jobs were killed prematurely. 

$ tar: ms02GINPPI/dipgin.ms02.output/43/ms02_43.tab: Read error at byte 0, while reading 7680 bytes: Connection timed out

$ tar: ms02GINPPI/dipgin.ms02.output/49/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2014Oct22_043941.txt: Read error at byte 0, while reading 8192 bytes: Connection timed out

$ tar: ms02GINPPI/dipgin.ms02.output/133/ms02_133.tab: Read error at byte 0, while reading 1536 bytes: Connection timed out

$ tar: ms02GINPPI/dipgin.ms02.output/372/ms02_372.tab: Read error at byte 0, while reading 1536 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/71/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Feb17_113337.txt: Read error at byte 0, while reading 6656 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/71/ms02_71.tab: Read error at byte 0, while reading 8704 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/645/ms02_645.tab: Read error at byte 0, while reading 6656 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/324/ms02_324.tab: Read error at byte 0, while reading 9216 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/50/ms02_50.tab: Read error at byte 0, while reading 5632 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/68/popages/cutoff.4.p.1.lambda.0.02.popsize.2000.time.2015Feb05_212237.txt: Read error at byte 0, while reading 2048 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/86/popages/cutoff.4.p.0.6.lambda.0.02.popsize.2000.time.2015Feb01_021709.txt: Read error at byte 0, while reading 512 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/62/popages/cutoff.4.p.0.6.lambda.0.01.popsize.2000.time.2014Dec04_123712.txt: Read error at byte 0, while reading 7168 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/528/ms02_528.tab: Read error at byte 0, while reading 512 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/7/ms02_7.tab: Read error at byte 0, while reading 8704 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/428/ms02_428.tab: Read error at byte 0, while reading 1024 bytes: Connection timed out

tar: ms02GINPPI/dipgin.ms02.output/18/ms02_18.tab: Read error at byte 0, while reading 4096 bytes: Connection timed out

*** parallel runs of net-aging-sim-20150915.R -> net-aging-sim-2015Sep15.R

[hqin2@greenfield ms02GINPPI]$ cat netaging.pbs

#!/bin/bash

# Request 15 cores

#PBS -l nodes=1:ppn=15

#  Request 2:30 minutes of cpu time

#PBS -l walltime=7:00:00:00

module load R/3.2.1-mkl

echo hostname

pwd

cd /crucible/mc48o9p/hqin2/mactower-network-failure-simulation-master/ms02GINPPI

pwd

numactl -C +0 R --vanilla --slave -f net-aging-sim-20150915.R --args 95 95 0.02 1.0 50  &

numactl -C +1 R --vanilla --slave -f net-aging-sim-20150915.R --args 96 96 0.02 1.0 50  &

numactl -C +2 R --vanilla --slave -f net-aging-sim-20150915.R --args 97 97 0.02 1.0 50  &

wait

[hqin2@greenfield ms02GINPPI]$ qstat -u hqin2 -a

greenfield.psc.xsede.org: 

                                                                                  Req'd    Req'd       Elap

Job ID                  Username    Queue    Jobname          SessID  NDS   TSK   Memory   Time    S   Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2343.greenfield.psc.xs  hqin2       batch    netaging.pbs      44687     1     15    --  168:00:00 R  00:03:56

[hqin2@greenfield ms02GINPPI]$ ll -t dipgin.ms02.output/9[567]/popages/*0.02*

-rw-r--r-- 1 hqin2 mc48o9p 848 Sep 14 19:26 dipgin.ms02.output/97/popages/cutoff.4.p.1.lambda.0.02.popsize.50.time.2015Sep14_192617.txt

-rw-r--r-- 1 hqin2 mc48o9p 845 Sep 14 19:26 dipgin.ms02.output/95/popages/cutoff.4.p.1.lambda.0.02.popsize.50.time.2015Sep14_192616.txt

-rw-r--r-- 1 hqin2 mc48o9p 846 Sep 14 19:26 dipgin.ms02.output/96/popages/cutoff.4.p.1.lambda.0.02.popsize.50.time.2015Sep14_192613.txt

[hqin2@greenfield ms02GINPPI]$ cat netaging.pbs

#!/bin/bash

# Request 15 cores

#PBS -l nodes=1:ppn=15

#  Request 2:30 minutes of cpu time

#PBS -l walltime=7:00:00:00

module load R/3.2.1-mkl

echo hostname

pwd

cd /crucible/mc48o9p/hqin2/mactower-network-failure-simulation-master/ms02GINPPI

pwd

numactl -C +0 R --vanilla --slave -f net-aging-sim-20150915.R --args 95 95 0.02 1.0 150  &

numactl -C +1 R --vanilla --slave -f net-aging-sim-20150915.R --args 96 96 0.02 1.0 150  &

numactl -C +2 R --vanilla --slave -f net-aging-sim-20150915.R --args 97 97 0.02 1.0 150  &

wait

[hqin2@greenfield ms02GINPPI]$ qstat -u hqin2

greenfield.psc.xsede.org: 

                                                                                  Req'd    Req'd       Elap

Job ID                  Username    Queue    Jobname          SessID  NDS   TSK   Memory   Time    S   Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2344.greenfield.psc.xs  hqin2       batch    netaging.pbs      45653     1     15    --  168:00:00 R  00:21:47

[hqin2@greenfield ms02GINPPI]$ qstat

Job ID                    Name             User            Time Use S Queue

------------------------- ---------------- --------------- -------- - -----

2344.greenfield            netaging.pbs     hqin2           01:07:54 R batch 

/*Great. The Elapse time is 21 minutes but my CPU time is 3 times of it. So, I am running job in paralle. */

[hqin2@greenfield ms02GINPPI]$ ll -th dipgin.ms02.output/9[567]/popages/*txt

-rw-r--r-- 1 hqin2 mc48o9p 2.5K Sep 14 20:05 dipgin.ms02.output/95/popages/cutoff.4.p.1.lambda.0.02.popsize.150.time.2015Sep14_200529.txt

-rw-r--r-- 1 hqin2 mc48o9p 2.5K Sep 14 20:05 dipgin.ms02.output/96/popages/cutoff.4.p.1.lambda.0.02.popsize.150.time.2015Sep14_200528.txt

-rw-r--r-- 1 hqin2 mc48o9p 2.5K Sep 14 20:05 dipgin.ms02.output/97/popages/cutoff.4.p.1.lambda.0.02.popsize.150.time.2015Sep14_200527.txt

-rw-r--r-- 1 hqin2 mc48o9p  848 Sep 14 19:26 dipgin.ms02.output/97/popages/cutoff.4.p.1.lambda.0.02.popsize.50.time.2015Sep14_192617.txt

-rw-r--r-- 1 hqin2 mc48o9p  845 Sep 14 19:26 dipgin.ms02.output/95/popages/cutoff.4.p.1.lambda.0.02.popsize.50.time.2015Sep14_192616.txt

-rw-r--r-- 1 hqin2 mc48o9p  846 Sep 14 19:26 dipgin.ms02.output/96/popages/cutoff.4.p.1.lambda.0.02.popsize.50.time.2015Sep14_192613.txt

/*The three popsize.150 files are generated simultaneously. */

*** parallel runs on greenfield, example.R and example.pbs

helen:ms02GINPPI hqin$ cat example.R 

#20150914 testing parallel jobs on greenfield

# R CMD BATCH ./example.R --args 2 35 red

rm(list=ls())

source("lifespan.r")

source("network.r")

#R -f file --args start end

options(echo=TRUE) # if you want see commands in output file

args <- commandArgs(trailingOnly = TRUE)

print(args)

# trailingOnly=TRUE means that only your arguments are returned, check:

# print(commandsArgs(trailingOnly=FALSE))

start = as.integer(args[1]); start; 

end = as.integer(args[2]); end; 

tag = args[3]

  

myhost = 'greenfield'  # 'byte' 'blacklight' 'mactower'

myhost = 'byte'  # 'byte' 'blacklight' 'mactower'

myhost = 'helen'  # 'byte' 'blacklight' 'helen'

mydir = "/crucible/mc48o9p/hqin2/mactower-network-failure-simulation-master/ms02GINPPI"

if (myhost == 'byte') {  mydir = "/Users/hqin/github/mactower-network-failure-simulation/ms02GINPPI"

} else if (myhost == 'helen') { mydir = "/Users/hqin/github/mactower-network-failure-simulation/ms02GINPPI";  

}

print(paste("current dir is:", mydir))

list.files()

getwd()

print(paste("Now setwd"))

setwd(mydir)

list.files()

debug = 0; 

x = start:end 

outfile = paste( tag, start, end, "tab", sep='.')

write.csv( x, outfile, row.names=F)

[hqin2@greenfield ms02GINPPI]$ cat example.pbs

#!/bin/bash

# Request 15 cores

#PBS -l nodes=1:ppn=15

#  Request 1hr of cpu time

#PBS -l walltime=0:00:01:00

module load R/3.2.1-mkl

echo hostname

pwd

cd /crucible/mc48o9p/hqin2/mactower-network-failure-simulation-master/ms02GINPPI

pwd

numactl -C +0 R --vanilla --slave -f ./example.R --args 1 199999 '_zero' &

numactl -C +1 R --vanilla --slave -f ./example.R --args 2 299999 '_one' &

numactl -C +2 R --vanilla --slave -f ./example.R --args 3 399999 '_two' &

numactl -C +3 R --vanilla --slave -f ./example.R --args 4 499999 '_three' &

numactl -C +4 R --vanilla --slave -f ./example.R --args 4 499999 '_four' &

numactl -C +5 R --vanilla --slave -f ./example.R --args 4 499999 '_five' &

numactl -C +6 R --vanilla --slave -f ./example.R --args 4 499999 '_six' &

numactl -C +7 R --vanilla --slave -f ./example.R --args 4 499999 '_seven' &

numactl -C +8 R --vanilla --slave -f ./example.R --args 4 499999 '_eight' &

numactl -C +9 R --vanilla --slave -f ./example.R --args 4 499999 '_nine' &

wait