https://stephens999.github.io/fiveMinuteStats/MH-examples1.html
Wednesday, November 17, 2021
bayesian statistics, MCMC, UCSC, coursera
https://www.coursera.org/learn/mcmc-bayesian-statistics/home/week/3
R code
4180 data science project discussion
== pre-class to do:
calendar email invitation: including guests; done.
socrative questions (midterm exam, questions on contents from last lecture ). done
update Canvas course materials, update learning objectives. assignments as needed. done
* make sure grades are NOT shown on Canvas.
Test-run code: ipynb. NA
learning objectives: updated on canvas, NA
== In-class to do:
clean up desktop space, calendars":
ZOOM, live transcript (start video recording). Turn the computer speaker on.
Socrative sign in
Give time for students to take course surveys
* ask students to share projects. Need to work on writing.
* discussion covid19 transmission
https://www.nature.com/articles/s41467-021-21358-2
Easy MCMC
https://stephens999.github.io/fiveMinuteStats/MH-examples1.html
Tuesday, November 16, 2021
Monday, November 15, 2021
secure python CEDI
Socrative
Slides
ask students to share their screen
Last times, I let students worked on replacingExample1.py
=======lesson plan ==========
15 minutes:
Go over Socrative review questions.
10 minutes:
whileExample1.py :: line.strip()
5 minutes:
whileExample2.py :: line.split()
10 minutes:
=> ask student to modify whileExample2 to process gleep2. This is whileExample3.py
15 minutes
csvHandler.py
20 minutes:
=> Lab exercise 1. datetime.datetime
=> datePython.py
cpsc 4180 data science, project discussion
== pre-class to do:
calendar email invitation: including guests; done.
socrative questions (midterm exam, questions on contents from last lecture ). done
update Canvas course materials, update learning objectives. assignments as needed. done
* make sure grades are NOT shown on Canvas.
Test-run code: ipynb. NA
learning objectives: updated on canvas, NA
== In-class to do:
clean up desktop space, calendars":
ZOOM, live transcript (start video recording). Turn the computer speaker on.
Socrative sign in
Give time for students to take course surveys
* ask students to share projects. Need to work on writing.
* Show my ERA5 land weather data
CAE showcase workshop
https://cyberedwiki.org/index.php?title=Welcome_to_CyberEd_Wiki
https://www.caeresource.directory/home
Sunday, November 14, 2021
parse ERA5 GRIB for JHU counties on ts117, Nov 14-17, 7 days runs
The template pbs is:
-bash-4.2$ cat ts_grib_job_template.pbs
#!/bin/bash -l
#$ -S /bin/bash
#$ -N pygrib_NUMBER
#$ -cwd
. /etc/profile.d/modules.sh
module load anaconda
source activate grib
python parse_grib_4JHU-debug.py STARTINDEX ENDINDEX
-bash-4.2$
Qin used a python code to generate pbs jobs, and then submit them using shell script.
-bash-4.2$ cat generate_PBS_bash.py
import os
import re
filebatchshell = "submit_PBS.sh"
fshell = open(filebatchshell, "w")
step = 15
for i in range( 0, 600, step):
fin = open("ts_grib_job_template.pbs", "r")
LinesIn = fin.readlines()
fin.close()
LinesIn[2] = "#$ -N pygrib_" + str(i) + "\n"
LinesIn[7] = "python parse_grib_4JHU.py " + str(i) + " " + str(i + step) + " \n"
fileout = "ts_" + str(i) + ".pbs"
fout = open(fileout, "w")
for line in LinesIn:
fout.write(line)
fout.close()
buffer = "qsub " + fileout + "\n"
fshell.write(buffer)
fshell.close()
To parse the 2m-dew point, I used generate_PBS_bash_dewpoint.py
This is a 7-day run. The last t2m output is on Nov 20, and the last dewpoint file output is on Nov 21.
On Nov 21, I use github to backup running logs, and output csv files
ERA5 grib file parsing
grbs = pygrib.open('2020-2021NovT2m-dewpoint.grib') #does not work
grbs = pygrib.open('2019-2020June10.grib') #works
It would be strange if grib format changed. I need to redownload the grib file. again
6pm, Qin found out that remove the time selection, and the codes runs on 2021 GRIB file.
Saturday, November 13, 2021
DoD 8140 certification
8140 certification replace 8570 certification.
https://www.comptia.org/blog/what-is-dod-8140-cybersecurity-certifications-and-requirements
Friday, November 12, 2021
data science journals, conferences
https://www.computer.org/csdl/journal/bd
https://conferenceindex.org/disciplines
Springer: International Journal of Data Science and Analytics
ACM Transactions on Knowledge Discovery from Data
Thursday, November 11, 2021
CPSC4900 software engineering
UTC course scheduling, time slots
Aquarium fish tank video with Yolo
https://github.com/Developer-Y/cs-video-courses#software-engineering
Tuesday, November 9, 2021
Rt batch run on ts
Generate PBS submission jobs
* python generate_PBS_bash.py
* sh submit_PBS_small.sh
-bash-4.2$ cat submit_PBS_small.sh
qsub PBS/ts_1_2.pbs
qsub PBS/ts_2_3.pbs
qsub PBS/ts_3_4.pbs
qsub PBS/ts_4_5.pbs
qsub PBS/ts_5_6.pbs
qsub PBS/ts_6_7.pbs
qsub PBS/ts_7_8.pbs
qsub PBS/ts_8_9.pbs
qsub PBS/ts_9_10.pbs
qsub PBS/ts_10_11.pbs
These jobs were done within minutes.
23:50 sh submit_PBS.sh ==> 3342 jobs, one for each USA county. small test run.
00:14 Use generate_PBS_bash.py for time from Feb 1, 2020 to Nov 1, 2021. Sub 3342 jobs again.
-bash-4.2$ qstat | head
job-ID prior name user state submit/start at queue slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
40904 0.50500 epi_job1 hqin r 11/10/2021 01:16:14 all.q@ts26.cm.cluster 1
40905 0.50500 epi_job2 hqin r 11/10/2021 01:16:14 all.q@ts19.cm.cluster 1
40906 0.50500 epi_job3 hqin r 11/10/2021 01:16:14 all.q@ts01.cm.cluster 1
40907 0.50500 epi_job4 hqin r 11/10/2021 01:16:14 all.q@ts33.cm.cluster 1
40909 0.50500 epi_job6 hqin r 11/10/2021 01:16:14 all.q@ts19.cm.cluster 1
40910 0.50500 epi_job7 hqin r 11/10/2021 01:16:14 all.q@ts01.cm.cluster 1
40912 0.50500 epi_job9 hqin r 11/10/2021 01:16:14 all.q@ts29.cm.cluster 1
40914 0.50500 epi_job11 hqin r 11/10/2021 01:16:14 all.q@ts26.cm.cluster 1
The last output file was generated on Nov 14, 2021 at 04:43. So the job run for more than 5 days.
total 209792
-rw-r--r-- 1 hqin simctr 23771 Nov 14 04:43 Suffolk_NewYork_US_v2.csv
Subscribe to:
Posts (Atom)