Open Notebook: gordon

Showing posts with label gordon. Show all posts

Monday, July 4, 2016

(todo) grid net2 gordon run (in progress)

Net2 is a 5-link grid network model.

#test on byte

Byte-4:1.grid.network.2015Oct hqin$ pwd

/Users/hqin/github/mactower-network-failure-simulation/1.grid.network.2015Oct

Rscript netwk_aging_sim.v0.1b.R -if1 net2/Degree5N1000_network.csv -if2 net2/Degree5N1000_EssenLookupTb.csv -l1 0.02 -l2 0 -dt 0 -p 0.98 -n 50 -op net2 -od net2 -d 1 -iC 16 -h /Users/hqin/github/mactower-network-failure-simulation/1.grid.network.2015Oct

@gordon

[hongqin@gordon-ln1 1.grid.network.2015Oct]$ cat grid_net2_aging_loopgordon.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

pwd

for p in 1.0 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86 0.85 0.84 0.82 0.81 0.8

for LOne in 0.002 0.0025 0.003 0.0035 0.004 0.0045 0.005 0.0055 0.006 0.0065 0.007 0.0075 0.008

echo "$p $LOne"

#R --vanilla --slave -f 20151101-net-sim-ginppi.R --args $LOne 0.0002 5 $p 1000

Rscript netwk_aging_sim.v0.1b.R -if1 net2/Degree5N1000_network.csv -if2 net2/Degree5N1000_EssenLookupTb.csv -l1 $LOne -l2 0 -dt 0 -p $p -n 5000 -op net2 -od net2 -d 1 -iC 16 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

done

July 4, 11:26am

[hongqin@gordon-ln1 1.grid.network.2015Oct]$ qstat -u hongqin

gordon-fe2.local:

Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2620348.gordon-fe2.loc hongqin normal grid_net2_aging_ 47508 1 16 -- 168:00:00 R 00:02:46 - --

July 5: 0:52am:

[hongqin@gordon-ln2 1.grid.network.2015Oct]$ qstat -u hongqin

gordon-fe2.local:

Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2620348.gordon-fe2.loc hongqin normal grid_net2_aging_ 47508 1 16 -- 168:00:00 R 13:29:18

July 5, 2016. 12pm. Loginto Gordon and found the job completed around 3am.
Using Globus, I transferred the 'new and changed" files back to Byte's mactow-simulation github repository.

Thursday, January 7, 2016

How fast is Gordon? 2.6GHz 2x8 cores, compare with Byte Intel core i7 6 cores

@Byte, simulate aging for 1000 cells using 4 cores.
11:02am -11:05
time Rscript netwk_aging_sim.v0.1b.R -if1 merged_PPIGIN_Factorized2015Oct132Cols.csv -if2 essentialGeneLookupTable_20151013.csv -l1 0.0023 -l2 0 -dt 4 -p 0.9 -n 1000 -op __TEST -od tmp -iC 4 -d 1 -h /Users/hqin/github/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

real 2m23.106s

user 8m25.295s

sys 0m37.280s

@gordon, job with 16 cores

[hongqin@gordon-ln2 1.ori.ginppi.2015Oct]$ cat ori_aging_loopgordon.3.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

pwd

#for p in 1.0 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

for p in 1.0 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9

#for LOne in 0.002 0.0021 0.0022 0.0023 0.0024 0.0025 0.0026 0.0028 0.0028 0.0029 0.003 0.0035 0.004 0.0045

for LOne in 0.001 0.0011 0.0012 0.0013 0.0014 0.0015 00016 0.0017 0.0018 0.0019 0.002

echo "$p $LOne"

time Rscript netwk_aging_sim.v0.1b.R -if1 merged_PPIGIN_Factorized2015Oct132Cols.csv -if2 essentialGeneLookupTable_20151013.csv -l1 $LOne -l2 0 -dt 5 -p $p -n 5000 -op ginppi -od ginppi.ori.out -iC 16 -d 1 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

done

real 3m35.381s

user 54m18.786s

sys 0m50.109s

[hongqin@gordon-ln1 1.ori.ginppi.2015Oct]$ cat ori_aging_loopgordon.3.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

pwd

#for p in 1.0 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

for p in 0.99 0.98

#for LOne in 0.002 0.0021 0.0022 0.0023 0.0024 0.0025 0.0026 0.0028 0.0028 0.0029 0.003 0.0035 0.004 0.0045

for LOne in 0.001 0.0011

echo "$p $LOne"

time Rscript netwk_aging_sim.v0.1b.R -if1 merged_PPIGIN_Factorized2015Oct132Cols.csv -if2 essentialGeneLookupTable_20151013.csv -l1 $LOne -l2 0 -dt 5 -p $p -n 1000 -op ginppi -od ginppi.ori.out -iC 16 -d 1 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

done

real 0m45.861s

user 11m8.614s

sys 0m10.168s

Conclusions: Gordon single core seems to be similar or even slower than Byte. However, its numbers of cores saves time.

"Each compute node contains two 8-core 2.6 GHz Intel EM64T Xeon E5 (Sandy Bridge) processors and 64 GB of DDR3-1333 memory. " By is Intel 2.6 GHz Corei7 (6 cores?)

Wednesday, January 6, 2016

(in progress) PBS run 1.ori.ginppi.2015Oct

/home/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ cat ori_aging_loopgordon.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

pwd

for p in 1.0 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

#for LOne in 0.002 0.0021 0.0022 0.0023 0.0024 0.0025 0.0026 0.0028 0.0028 0.0029 0.003 0.0035 0.004 0.0045

for LOne in 0.001 0.0011 0.0012 0.0013 0.0014 0.0015 00016 0.0017 0.0018 0.0019 0.002

echo "$p $LOne"

Rscript netwk_aging_sim.v0.1b.R -if1 merged_PPIGIN_Factorized2015Oct132Cols.csv -if2 essentialGeneLookupTable_20151013.csv -l1 $LOne -l2 0 -dt 4 -p $p -n 5000 -op ginppi -od ginppi.ori.out -iC 16 -d 1 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

done

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ qstat -u hongqin

gordon-fe2.local:

Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2394355.gordon-fe2.loc hongqin normal ori_aging_loopgo 113139 1 16 -- 168:00:00 R 00:10:54

Job started at 10:30am -->

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ cat ori_aging_loopgordon.2.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

pwd

for p in 1.0 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

for LOne in 0.001 0.0011 0.0012 0.0013 0.0014 0.0015 00016 0.0017 0.0018 0.0019 0.002 0.0021 0.0022 0.0023 0.0024 0.0025 0.0026 0.0028 0.0028 0.0029 0.003 0.0035 0.004 0.0045

echo "$p $LOne"

Rscript netwk_aging_sim.v0.1b.R -if1 merged_PPIGIN_Factorized2015Oct132Cols.csv -if2 essentialGeneLookupTable_20151013.csv -l1 $LOne -l2 0 -dt 5 -p $p -n 2500 -op ginppi -od ginppi.ori.out -iC 16 -d 1 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

done

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ qstat -u hongqin

gordon-fe2.local:

Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2394355.gordon-fe2.loc hongqin normal ori_aging_loopgo 113139 1 16 -- 168:00:00 R 00:19:15

2394382.gordon-fe2.loc hongqin normal ori_aging_loopgo 103945 1 16 -- 168:00:00 R 00:01:09

Saturday, January 2, 2016

Gordon, netwk_aging_0.1b.R /1.ori.ginppi.2015Oct

=> Linked R files from /1.grid.network.2015Oct/ to /1.ori.ginppi.2015Oct/

=> Did some test and editing on byte, and transfer files to Gordon using Globus.

Sat Jan 2 22:26:12 PST 2016, Try a testing pbs

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ cat ori_aging_loopgordon.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

pwd

#for p in 1.0 0.98 0.95 0.94 0.93 0.92 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

for p in 1.0 0.98

#for LOne in 0.003 0.0035 0.004 0.0045 0.005 0.0055 0.006 0.0065 0.007 0.0075 0.008

for LOne in 0.003 0.035

echo "$p $LOne"

#R --vanilla --slave -f 20151101-net-sim-ginppi.R --args $LOne 0.0002 5 $p 1000

Rscript netwk_aging_sim.v0.1b.R -if1 merged_PPIGIN_Factorized2015Oct132Cols.csv -if2 essentialGeneLookupTable_20151013.csv -l1 $LOne -l2 0 -dt 4 -p $p -n 49 -op ginppi -od ginppi.ori.out -iC 4 -d 1 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

done

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ date

Sat Jan 2 22:27:30 PST 2016

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ qsub ori_aging_loopgordon.pbs

2391092.gordon-fe2.local

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ qstat -u hongqin

gordon-fe2.local:

Req'd Req'd Elap

Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

----------------------- ----------- -------- ---------------- ------ ----- ------ ------ --------- - ---------

2390576.gordon-fe2.loc hongqin normal grid_aging_loopg 110732 1 16 -- 168:00:00 R 06:25:40

2391092.gordon-fe2.loc hongqin normal ori_aging_loopgo 81499 1 16 -- 168:00:00 R 00:00:02

This pbs works.

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ date

Sat Jan 2 22:31:48 PST 2016

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ cat ori_aging_loopgordon.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.ori.ginppi.2015Oct

pwd

for p in 1.0 0.98 0.95 0.94 0.93 0.92 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

for LOne in 0.002 0.0021 0.0022 0.0023 0.0024 0.0025 0.0026 0.0028 0.0028 0.0029 0.003 0.0035 0.004 0.0045

echo "$p $LOne"

done

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ date

Sat Jan 2 22:32:07 PST 2016

[hongqin@gordon-ln3 1.ori.ginppi.2015Oct]$ qsub ori_aging_loopgordon.pbs

2391118.gordon-fe2.local

This job run from 22:34 to 10:30am PST for a total of 12 hours for 197 runs.

Gordon, foreach doMC network aging run, 1.grid.network.2015Oct/net1

[hongqin@gordon-ln3 1.grid.network.2015Oct]$ cat grid_aging_gordon.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=8

#PBS -l walltime=12:30:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

pwd

Rscript netwk_aging_sim.v0.1b.R -if1 net1/Degree4N1000_network.csv -if2 net1/Degree4N1000_EssenLookupTb.csv -l1 0.006 -l2 0.0002 -dt 0 -p 0.95 -n 50 -op net1 -od net1 -d 1 -iC 4 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

Rscript netwk_aging_sim.v0.1b.R -if1 net1/Degree4N1000_network.csv -if2 net1/Degree4N1000_EssenLookupTb.csv -l1 0.006 -l2 0.0002 -dt 0 -p 0.89 -n 50 -op net1 -od net1 -d 1 -iC 4 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

[hongqin@gordon-ln3 1.grid.network.2015Oct]$ cat grid_aging_gordon.pbs.e2390416

Loading required package: iterators

Loading required package: parallel

Error in library(GetoptLong) : there is no package called 'GetoptLong'

Execution halted

Loading required package: iterators

Loading required package: parallel

Error in library(GetoptLong) : there is no package called 'GetoptLong'

Execution halted

Try to install GetoptLong

[hongqin@gordon-ln3 1.grid.network.2015Oct]$ module load R

[hongqin@gordon-ln3 1.grid.network.2015Oct]$

5:22pm. I can run the R-foreach-doMC code locally.

$ Rscript netwk_aging_sim.v0.1b.R -if1 net1/Degree4N1000_network.csv -if2 net1/Degree4N1000_EssenLookupTb.csv -l1 0.006 -l2 0.0002 -dt 0 -p 0.95 -n 50 -op net1 -od net1 -d 1 -iC 4 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

5:30pm qsub grid_aging_gordon.pbs

Finally worked.

Try shell loop

[hongqin@gordon-ln3 1.grid.network.2015Oct]$ cat grid_aging_loopgordon.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=8

#PBS -l walltime=12:30:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

pwd

for p in 1.0 0.98 0.95 0.94 0.93 0.92 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

for LOne in 0.003 0.0035 0.004 0.0045 0.005 0.0055 0.006 0.0065 0.007 0.0075 0.008

echo "$p $LOne"

#R --vanilla --slave -f 20151101-net-sim-ginppi.R --args $LOne 0.0002 5 $p 1000

Rscript netwk_aging_sim.v0.1b.R -if1 net1/Degree4N1000_network.csv -if2 net1/Degree4N1000_EssenLookupTb.csv -l1 $LOne -l2 $LOne/10 -dt 0 -p $p -n 50 -op net1 -od net1 -d 1 -iC 4 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

done

Note: -l2 $LOne/10 DID NOT WORK!

5:57pm

[hongqin@gordon-ln3 1.grid.network.2015Oct]$ qsub grid_aging_loopgordon.pbs

2390490.gordon-fe2.local

This job ran for 37 minutes much longer than I thought.

=> modified netwk_aging_sim.v0.1b.R to take default lambda2.

=>by 6:57pm, modified grid_aging_loopgordon.pbs for a test run.

[hongqin@gordon-ln3 1.grid.network.2015Oct]$ cat grid_aging_loopgordon.pbs

#!/bin/bash

#PBS -l nodes=1:ppn=16

#PBS -l walltime=07:00:00:00

module load R

echo hostname

cd /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

pwd

for p in 1.0 0.98 0.95 0.94 0.93 0.92 0.9 0.89 0.88 0.87 0.86 0.85 0.82 0.8

for LOne in 0.003 0.0035 0.004 0.0045 0.005 0.0055 0.006 0.0065 0.007 0.0075 0.008

echo "$p $LOne"

#R --vanilla --slave -f 20151101-net-sim-ginppi.R --args $LOne 0.0002 5 $p 1000

Rscript netwk_aging_sim.v0.1b.R -if1 net1/Degree4N1000_network.csv -if2 net1/Degree4N1000_EssenLookupTb.csv -l1 $LOne -l2 0 -dt 0 -p $p -n 5000 -op net1 -od net1 -d 1 -iC 16 -h /oasis/scratch/hongqin/temp_project/mactower-network-failure-simulation/1.grid.network.2015Oct

done

7:01pm

[hongqin@gordon-ln3 1.grid.network.2015Oct]$ qsub grid_aging_loopgordon.pbs

2390576.gordon-fe2.local

This job could run for ~450 minutes. Each n.5000 simulation only take about 4 minutes.

It turns out that this job from 16:05 to 05:43 PST, which is almost 14 hours (840 minutes) for 157 runs.

Thursday, November 5, 2015

*** bio386, Gordon exercise 20151105Thu

Gordon, file transfer

Procedure to transfer bio386.tar archives from QIN's directory to student directory:

You can loginto Gordon using ssh or Putty directly

$ ssh username@gordon.sdsc.xsede.org

Then go to your file directory:

$ cd /oasis/scratch/username/temp_project/

Then copy bio386.tar to your file directory

$ cp /oasis/scratch/hongqin/temp_project/bio386.tar .

Then

$ tar xvf bio386.tar

Then

$ cd Slim-network-aging-bio386-fall-2015

Then

$ nano -w 20151101-net-sim-ginppi.R

Inside of nano, you should change the username in the following line:

mydir = '/oasis/scratch/username/temp_project/Slim-network-aging-bio386-fall-2015'

Then control-X save and exit nano

Now, we can need to change the working directory in PBS job script.

$ nano -w netaging.pbs

Then change username in the following line

cd /oasis/scratch/hongqin/temp_project/Slim-network-aging-bio386-fall-2015

Then exit and save.

Then submit the job request

$ qsub netaging.pbs

If this runs, you are one of the first Spelman students to use a Federally funded super-computer.

To check the status of your computing jobs

$ qstat -u username

Thursday, September 24, 2015

XSEDE GORDON

https://portal.xsede.org/sdsc-gordon#running

Monday, September 7, 2015

estimate greenfield SU for Gordon

[hqin2@greenfield ms02GINPPI]$ cat ms02.pbs
#!/bin/bash
# Request 15 cores
#PBS -l nodes=1:ppn=15
# Request 2:30 minutes of cpu time
#PBS -l walltime=02:30:00

module load R/3.2.1-mkl

echo hostname

pwd
cd /crucible/mc48o9p/hqin2/mactower-network-failure-simulation-master/ms02GINPPI
pwd

R --vanilla --slave -f ms02-2015Aug27.R --args 900 999

10:10am

Using "ls -l", find out that each ms02 network took 3 minutes on greenfield.
100 permutation means 300minutes = 5 SUs.