deep supervised learning

https://twitter.com/jacobmbuckman/status/1475305972616900612?s=20

 

probability for data science

 

https://probability4datascience.com/

Notice of Special Interest (NOSI): Methods Development for Genomic Studies of Genetic Variation, Function, and Disease

 Notice of Special Interest (NOSI): Methods Development for Genomic Studies of Genetic Variation, Function, and Disease

https://grants.nih.gov/grants/guide/notice-files/NOT-HG-22-007.html

historical twitter data

 

https://gwu-libraries.github.io/sfm-ui/posts/2017-09-14-twitter-data

cointegration on all counties 1-3151 on ridgeside.

 

hqin@ridgeside:~/anaconda3/bin$ source  activate /home/hqin/.conda/envs/r2021nov

(r2021nov) hqin@ridgeside:~/github/COVID19_transmission_MS$ nohup  R -f auto_county_cajo.R --args 1 3151 1 &

job started on 13:33 Dec 17, 2021

Anaconda tensorflow on IBM powerpc architecture

This is for LOOKOUT cluster at UTC SimCenter

The key step is to specify the channel for IBM PowerPC. More details can be found at the following link.

https://stackoverflow.com/questions/35330084/installing-tensorflow-on-the-ibm-power8

The following link might also be helpful.

https://gauravm.gitbook.io/about/blogs/installing-pytorch-and-transformers-on-ibm-powerpc-architecture

If you need additional information, please let me know.

ICML

 

https://icml.cc/Sponsors/sponsorinfo

RH is close to a stationary signal for Hamilton, TN

 

adf.test(tb4$RH)

	Augmented Dickey-Fuller Test

data:  tb4$RH
Dickey-Fuller = -6.996, Lag order = 8, p-value = 0.01
alternative hypothesis: stationary

update Labda from 18 to LTS 20.4.x

In Lambda server root account. 

$sudo apt update

install the necessary packages (python, nvidia docker etc)

$sudo apt upgrade

This runs for a while, with many heading on focal shown on screen

chose N to keep ssh_config

13:20, a gui-window popped-up, chose English US as keyboard

13:22, Yes to restart service

13:29 close all chromium browser window update

INFO waiting for automatic snapd to restart

13:31, passed, and moving forward, many lines printed on screen

13:39, updating database for manual pages ........

13:48 searching for obsolete software

13:58 fetch and download can take several hours, continue? chose Yes. 

13:59. system update is complete. Restart? Yes. 

14:02, user log in window appeared. 

Run partial update using "software & update" tool as lambda root. 

TODO: need to test cuda and GPU support. 

auto cointegration Johasen test on ECS323GPUStation and macbook pro

20211211Sat: 

22:04, executed (error occured). 

hqin@ECS323GPUStation:~/COVID19_transmission_MS$ 

hqin@ECS323GPUStation:~/COVID19_transmission_MS$ cat commands_cajo.txt 

nohup  R -f auto_county_cajo.R --args 1 800  & 

nohup  R -f auto_county_cajo.R --args 801 1600  & 

nohup  R -f auto_county_cajo.R --args 1601 2400  & 

nohup  R -f auto_county_cajo.R --args 2401 3151 & 

Back at MacPro

22:40. I modified the code to take debug as input, and ran on macpro. Many jobs get done quickly, but something take very long time. 

22:55, move to ECS323GPUStation: 

hqin@ECS323GPUStation:~/COVID19_transmission_MS$ cat commands_cajo.txt 

nohup  R -f auto_county_cajo.R --args 1 800  1 & 

nohup  R -f auto_county_cajo.R --args 801 1600  1 & 

nohup  R -f auto_county_cajo.R --args 1601 2400 1  & 

nohup  R -f auto_county_cajo.R --args 2401 3151 1 &

hqin@ECS323GPUStation:~/COVID19_transmission_MS$ ps -ef | grep county

hqin      156203       1 99 22:03 pts/0    00:51:08 /usr/lib/R/bin/exec/R -f auto_county_cajo.R --args 801 1600

hqin      591653   13303  0 22:54 pts/0    00:00:00 grep county

hqin     3595313       1 99 22:54 pts/0    00:00:38 /usr/lib/R/bin/exec/R -f auto_county_cajo.R --args 1 800 1

hqin     3595314       1 99 22:54 pts/0    00:00:38 /usr/lib/R/bin/exec/R -f auto_county_cajo.R --args 801 1600 1

hqin     3595315       1 99 22:54 pts/0    00:00:38 /usr/lib/R/bin/exec/R -f auto_county_cajo.R --args 1601 2400 1

hqin     3595316       1 99 22:54 pts/0    00:00:38 /usr/lib/R/bin/exec/R -f auto_county_cajo.R --args 2401 3151 1

The job 801-1600 took almost 55 minutes on ECS313GPUStation and is the longest job. 

Re-run commands_cajo.txt and all jobs are done quickly. So, co.ja() run have stochastic elements.  

(base) hqin@CS313BQin COVID19_transmission_MS % ls -tlh *csv

-rw-r--r--  1 hqin  staff   266K Dec 11 23:15 _autocajo_801-1600.csv

-rw-r--r--  1 hqin  staff   257K Dec 11 23:15 _autocajo_2401-3151.csv

-rw-r--r--  1 hqin  staff   266K Dec 11 23:15 _autocajo_1601-2400.csv

-rw-r--r--  1 hqin  staff   523K Dec 11 23:15 _autocajo_1-3151.csv

TODO: merge these four temporary files. 

Alternatively, run the whole process in workstation. 

auto cajo pbs on ts117

 job in EQW status? !

This does NOT worked for batch cointegration using Johansen test

-bash-4.2$ 

-bash-4.2$ cat cajo.pbs 

#!/bin/bash -l

#$ -S /bin/bash

#$ -N epinow_job

#$ -V

#$ -cwd

. /etc/profile.d/modules.sh

module load anaconda/5.2.0

source activate r2021nov

5000 study guide for waiver test

Study Guide for CPSC 5000 Waiver Test 

Programming Language: Java 

1. Number Types, Variables, Constants, Operators 

• Commonly used number types, declaration and use of variables, declaration and use of  constants 

• Arithmetic operators, increment vs. decrement, integer division and remainder, power and  roots, cast operators; be able to determine the value of a given expression • Strings, string concatenation, escape sequences, strings and characters, substrings  • Arrays, two-dimensional arrays, array lists 

2. Input, Output, Exception Handling 

• Reading input, formatted output 

• Reading and writing text files, text input and output 

• Exception handling (throwing exceptions, catching exceptions, checked exceptions, closing  resources), designing your own exception types 

3. If statement 

• Relational operators for comparing values (integers, floating-point numbers, strings, objects) • Boolean variables and operators, logical conditions 

• Multiple alternatives using if, switch statement 

• Nested branches/if 

4. Loops 

• The while loop 

• The for loop 

• The do loop 

• Sentinel values 

• Common loop algorithms 

• Nested loops 

5. Classes and Objects 

• Concepts of classes and objects, concept of encapsulation, advantages of encapsulation

• Instance variables, instance variables vs. local variables, constructors, methods, public interface  of a class 

• Implementing and designing classes, constructing objects 

• Discovering classes for programming problems 

• Inheritance: implementing subclasses, overriding methods 

• Interfaces: declaring an interface type, implementing an interface type, working with interface  variables (converting from classes to interfaces, invoking methods on interface variables, casting  from interfaces to classes), the Comparable interface 

• Relationships between classes (dependency, aggregation, inheritance) 

6. Algorithms 

• Common loop algorithms (sum and average value, counting matches, finding the first match,  prompting until a match is found, maximum and minimum, comparing adjacent values) • Common array algorithms (filling, sum and average value, maximum and minimum, element  separators, linear search, removing an element, inserting an element, swapping elements,  copying arrays, reading input) 

• Random numbers and simulations: generating random numbers, the monte Carlo method • Recursion, the efficiency of recursion 

7. Other topics 

• Method (signature, overloading, and pass by value/reference, recursive programing) • Polymorphism (introduced by abstract class, interface, and generic programming) • GUI and event-driven programming 

• Applied programming (threading, socket, and exception handling)

 

quantum computer and neural networks

 
Quantum processor swapped in for a neural network
Quantum processor used to model weather based on sparse data.

https://arstechnica.com/science/2021/12/quantum-processor-swapped-in-for-a-neural-network/

UTC sample bylaws

 

https://www.utc.edu/sites/default/files/2020-06/dpa-bylaws-v2.pdf

Rt ~ g_workplace ERA_dewpoint ERA_K, air_temp

 early 2020 to spring 2021. 

Hamilton, TN

 sub = tb4[, c('Rt', 'g_workplace',  'ERA5_dewpoint', 'K', 'air_temp')] # r <= 2

dewpoint weakly correlates with RH

 NOAA dewpoint  correlate weakly with RH, R2=0.11 for Hamilton, TN

ERA5 dewpoint correlates with NOAA dewpoint well. R2 = 0.97

ERA5 Grib t2m correlates well with NOAA air_temp at Hamilton, TN

 source code:  
https://github.com/QinLab/COVID19_transmission_MS/blob/main/hamiltonTN_compare_T2m_NOAA.Rmd 

Call:
lm(formula = tb_comp$K ~ tb_comp$air_temp)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.7346 -0.9967 -0.0165  0.9527  7.6325 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)      271.41583    0.14303  1897.6   <2e-16 ***
tb_comp$air_temp   0.96368    0.00688   140.1   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.492 on 597 degrees of freedom
Multiple R-squared:  0.9705,	Adjusted R-squared:  0.9704 
F-statistic: 1.962e+04 on 1 and 597 DF,  p-value: < 2.2e-16

data science, last lecture, Q&A about final project and final presentation video

 

== pre-class to do: 

calendar email invitation: including guests;  done. 

socrative questions (midterm exam, questions on contents from last lecture ).  done

update Canvas course materials, update learning objectives. assignments as needed.   done

* make sure grades are NOT shown on Canvas. 

Test-run code:  ipynb. NA

learning objectives:  updated on canvas, NA

== In-class to do: 

clean up desktop space, calendars": 

ZOOM, live transcript (start video recording).  Turn the computer speaker on. 

Socrative sign in 

Give time for students to take course surveys:     

* Show my ERA5 land weather data, PBS jobs, run errors and outputs. 

 Arts and Science Dean cited some books on STEM in general education. 

Robot-Proof

Higher Education in the Age of Artificial Intelligence

https://mitpress.mit.edu/books/robot-proof

李宏毅对抗生成网络(GAN)国语教程(2018)

https://www.bilibili.com/video/BV1Up411R7Lk?from=search&seid=11284843292198463798&spm_id_from=333.337.0.0

awesome public data sets

 

https://github.com/awesomedata/awesome-public-datasets

APEER by ZEISS

deep learning image analysis provided by ZEISS

https://www.apeer.com/blog

training set, validation set, and test test for machine learing / deep learning

 

from: https://machinelearningmastery.com/difference-test-validation-datasets/

– Training set: A set of examples used for learning, that is to fit the parameters of the classifier.

– Validation set: A set of examples used to tune the parameters of a classifier, for example to choose the number of hidden units in a neural network.

– Test set: A set of examples used only to assess the performance of a fully-specified classifier.

secure python, unit 14, 15, regular expression

Socrative: review of while loop

Slides
ask students to share their screen 
Last times, I let students worked on 
whileExample1.py
csvHandler.py
Lab exercise 1.

=======lesson plan ==========

15 minutes: 

Go over Socrative review questions. while loop, csv handler

=> datePython.py 

regular expression in Python

https://docs.python.org/3/howto/regex.html

https://github.com/AstunTechnology/python-basics/blob/master/regular-expressions.md

HCA AIDA pilot

19-year-old human stage, 21-year-old human stage, 22-year-old human stage, 23-year-old human stage, 24-year-old human stage, 25-year-old human stage, 26-year-old human stage, 27-year-old human stage, 28-year-old human stage, 29-year-old human stage, 30-year-old human stage, 31-year-old human stage, 32-year-old human stage, 33-year-old human stage, 34-year-old human stage, 35-year-old human stage, 36-year-old human stage, 37-year-old human stage, 38-year-old human stage, 39-year-old human stage, 40-year-old human stage, 41-year-old human stage, 42-year-old human stage, 43-year-old human stage, 44-year-old human stage, 45-year-old human stage, 46-year-old human stage, 47-year-old human stage, 48-year-old human stage, 49-year-old human stage, 50-year-old human stage, 51-year-old human stage, 52-year-old human stage, 53-year-old human stage, 54-year-old human stage, 55-year-old human stage, 56-year-old human stage, 57-year-old human stage, 58-year-old human stage, 59-year-old human stage, 60-year-old human stage, 61-year-old human stage, 62-year-old human stage, 63-year-old human stage, 64-year-old human stage, 65-year-old human stage, 66-year-old human stage, 68-year-old human stage, 71-year-old human stage, 73-year-old human stage

 

https://data.humancellatlas.org/explore/projects/f0f89c14-7460-4bab-9d42-22228a91f185

single cell aging mouse brain （error）

 37069 cells

https://singlecell.broadinstitute.org/single_cell/study/SCP263/aging-mouse-brain

455M is too big to download by browser. 

Tring to get certificate at local server

openssl s_client -showcerts -servername singlecell.broadinstitute.org/single_cell/api/v1 -connect server:443 > cacert.pem

https://singlecell.broadinstitute.org/

singlecell.broadinstitute.org/single_cell/api/v1

(base) hqin@CS313BQin q-sandbox % which openssl

/Users/hqin/opt/anaconda3/bin/openssl

(base) hqin@CS313BQin q-sandbox % openssl s_client -showcerts -servername https://singlecell.broadinstitute.org -connect server:443 > cacert.pem

4504972800:error:2008F002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/b_addr.c:730:nodename nor servname provided, or not known

connect:errno=0

CURL download error

curl: (60) SSL certificate problem: unable to get local issuer certificate

More details here: https://curl.haxx.se/docs/sslcerts.html

design for inference : scRNA-seq, Aviv Regev

 

Schrodinger Equation

 i and h are constant

H hat is the Hamiltonian operator that includes the total energy. 

So, Schrodinger Equation indicates that change of a quantum system is ~ ih of the total energy?! 

quantum reading notes, QAOA

superposition is a linear combination of states

quantum gates are matrices

Hadamard gate creates superposition. 

H on |0> gives to sqrt(2)( |0> + |1> ), whereas H on |1> gives sqrt(2) ( |0> - |1> ) 

Controlled NOT gate is on two qubit systems: 

Quantum Approximate Optimization Algorithm (QAOA) can solve Max-Cut of graphs, an NP-complete problem. 

For n vertices and m edges, the vertices can be either k(i) = +1 or -1. A cut can be defined as k(i)xk(j)= -1. 

The n-vertices can implemented as n qubits. The 2^n assignments for the vertices will correspond to the 2^n dimenstional Hilbert space generated by n qubits. So, if <i,j> is a cut, the related qubit will land onto different bases. 

QIN: In the above implementation, each vertice is allowed either 1 or 0. So, expand this for my network aging model, I may be able to expand the vertice to sub-vertices, which allows for 1 or 0 separately. Can I? 

Reference: 

Quantgates training materials

greek symbols pronunciations

 

https://www.thoughtco.com/the-greek-alphabet-1705558

COVID variants B.1.1.529 Omicron

betacoronavirus

GR

B.1.1.529

2021-11-18

(NSP5_P132H,Spike_H69del,Spike_T95I,Spike_A67V,Spike_S373P,Spike_Q493R,Spike_H655Y,N_R203K,Spike_N969K,Spike_N856K,Spike_G142D,NSP3_A1892T,Spike_Q954H,N_P13L,NSP3_L1266I,Spike_N501Y,N_R32del,M_Q19E,Spike_N440K,NSP4_T492I,NSP6_L105del,Spike_N679K,Spike_N764K,Spike_L212I,Spike_Y505H,NSP6_G107del,NSP6_I189V,Spike_T547K,M_D3G,Spike_D796Y,N_G204R,Spike_T478K,Spike_V143del,M_A63T,Spike_G496S,NSP3_V1069I,Spike_K417N,NSP6_S106del,Spike_S371L,Spike_G339D,NSP3_S1265del,NSP14_I42V,Spike_P681H,Spike_Y144del,Spike_ins214EPE,N_S33del,Spike_S375F,Spike_Q498R,Spike_G446S,Spike_S477N,N_E31del,NSP3_K38R,Spike_N211del,Spike_E484A,E_T9I,Spike_V70del,Spike_L981F,NSP12_P323L,Spike_D614G,Spike_Y145del)

VOC Omicron GR/484A (B.1.1.529) first detected in Botswana/Hong Kong/South Africa

2021-11-11

Africa / Botswana / South East / Greater Gaborone / Gaborone

Human

2021-11-23

TRUE

0.0131383910524

0.379838874855

29684

 

B1.1.529 immuno comprosied person of origin

 

https://english.alarabiya.net/coronavirus/2021/11/26/Explainer-What-is-the-new-B-1-1-529-COVID-19-variant-

remote jupyter-notebook on ts117

Follow https://github.com/justinmclark/utc_lookout_remote_jupyter

I used 'grib' environment, and reinstalled jupyter in 'grib'. 

I then used a browser on local:8888 in ECS313 desktop, and it worked. 

 

CPSC 4180 data science

== pre-class to do: 

calendar email invitation: including guests;  done. 

socrative questions (midterm exam, questions on contents from last lecture ).  done

update Canvas course materials, update learning objectives. assignments as needed.   done

* make sure grades are NOT shown on Canvas. 

Test-run code:  ipynb. NA

learning objectives:  updated on canvas, NA

== In-class to do: 

clean up desktop space, calendars": 

ZOOM, live transcript (start video recording).  Turn the computer speaker on. 

Socrative sign in 

Give time for students to take course surveys

* ask students to share projects. Need to work on writing. 

* Show my ERA5 land weather data, PBS jobs, run errors and outputs. 

Application instruction for UTC Fall 2022 PhD program

 

To apply for Fall 2022 enrollment of PhD program, please apply at https://www.utc.edu/research/graduate-school/prospective-students/graduate-school-admissions

 

The deadline for fall 2022 application is February 1 (for students interested in graduate assistantship)

https://www.utc.edu/engineering-and-computer-science/academic-programs/electrical-engineering/graduate-curriculum/graduate-admission

 

Our graduate school application requires GRE and TOEFL/IELTS scores. 

 

Thanks again for your interest. Please let me know if I can answer some of your questions,

 

15 Apple counties not found in JHU county

 

There are  32  Apple counties not found in counties2/

Out[83]:

{'AguadillaMunicipio_PuertoRico_US_v2.csv',
 'AnchorageMunicipality_Alaska_US_v2.csv',
 'AreciboMunicipio_PuertoRico_US_v2.csv',
 'BayamónMunicipio_PuertoRico_US_v2.csv',
 'BoxElder_Utah_US_v2.csv',
 'CaguasMunicipio_PuertoRico_US_v2.csv',
 'CarolinaMunicipio_PuertoRico_US_v2.csv',
 'Carson_Nevada_US_v2.csv',
 'DoradoMunicipio_PuertoRico_US_v2.csv',
 'DoñaAna_NewMexico_US_v2.csv',
 'Dukes_Massachusetts_US_v2.csv',
 'FairbanksNorthStarBorough_Alaska_US_v2.csv',
 'Guam_Guam_US_v2.csv',
 'GuaynaboMunicipio_PuertoRico_US_v2.csv',
 'GuraboMunicipio_PuertoRico_US_v2.csv',
 'HumacaoMunicipio_PuertoRico_US_v2.csv',
 'James_Virginia_US_v2.csv',
 'Juab_Utah_US_v2.csv',
 'JuneauandBorough_Alaska_US_v2.csv',
 'KenaiPeninsulaBorough_Alaska_US_v2.csv',
 'Matanuska-SusitnaBorough_Alaska_US_v2.csv',
 'MayagüezMunicipio_PuertoRico_US_v2.csv',
 'Millard_Utah_US_v2.csv',
 'PonceMunicipio_PuertoRico_US_v2.csv',
 'RíoGrandeMunicipio_PuertoRico_US_v2.csv',
 'SanJuanMunicipio_PuertoRico_US_v2.csv',
 'Sevier_Utah_US_v2.csv',
 'St.CroixIsland_VirginIslands_US_v2.csv',
 'St.ThomasIsland_VirginIslands_US_v2.csv',
 'ToaBajaMunicipio_PuertoRico_US_v2.csv',
 'TrujilloAltoMunicipio_PuertoRico_US_v2.csv',
 'Weber_Utah_US_v2.csv'}

Sevier Utah is an unincoporated community. 

After remove some "and", "City", "Municipio", "Borough",  there are 18 left. 

There are  15  Apple counties not found in counties2/
Out[13]:
{'Bayamón_PuertoRico_US_v2.csv',
 'BoxElder_Utah_US_v2.csv',
 'Carson_Nevada_US_v2.csv',
 'DoñaAna_NewMexico_US_v2.csv',
 'Dukes_Massachusetts_US_v2.csv',
 'Guam_Guam_US_v2.csv',
 'James_Virginia_US_v2.csv',
 'Juab_Utah_US_v2.csv',
 'Mayagüez_PuertoRico_US_v2.csv',
 'Millard_Utah_US_v2.csv',
 'RíoGrande_PuertoRico_US_v2.csv',
 'Sevier_Utah_US_v2.csv',
 'St.CroixIsland_VirginIslands_US_v2.csv',
 'St.ThomasIsland_VirginIslands_US_v2.csv',
 'Weber_Utah_US_v2.csv'}

Dukes_Massachusetts_US_v2.csv is in Apple. 
In JHU, this seems to be DukesandNantucket_Massachusetts_US_v2.csv 

deepl

 

https://www.deepl.com/en/translator

Deviance information criterion DIC

From https://en.wikipedia.org/wiki/Deviance_information_criterion : 

There are two calculations in common usage for the effective number of parameters of the model. The first, as described in Spiegelhalter et al. (2002, p. 587), is ${\displaystyle p_{D}={\overline {D(\theta )}}-D({\bar {\theta }})}$, where ${\displaystyle {\bar {\theta }}}$ is the expectation of ${\displaystyle \theta }$. The second, as described in Gelman et al. (2004, p. 182), is ${\displaystyle p_{D}=p_{V}={\frac {1}{2}}{\overline {\operatorname {var} \left(D(\theta )\right)}}}$. The larger the effective number of parameters is, the easier it is for the model to fit the data, and so the deviance needs to be penalized.

The deviance information criterion is calculated as

${\displaystyle \mathrm {DIC} =p_{D}+{\overline {D(\theta )}},}$

or equivalently as

${\displaystyle \mathrm {DIC} =D({\bar {\theta }})+2p_{D}.}$

,