https://arbor-analytics.com/post/mixed-models-a-primer/
Saturday, October 31, 2020
Fourier Neural Operator for Parametric Partial Differential Equations
Fourier Neural Operator for Parametric Partial Differential Equations
Zongyi Li∗ , Nikola Kovachki∗ , Kamyar Azizzadenesheli† , Burigede Liu∗ , Kaushik Bhattacharya∗ , Andrew Stuart∗ , Anima Anandkumar∗ October 20, 2020
"The classical development of neural networks has primarily focused on learning mappings between finite-dimensional Euclidean spaces. Recently, this has been generalized to neural operators that learn mappings between function spaces. For partial differential equations (PDEs), neural operators directly learn the mapping from any functional parametric dependence to the solution. Thus, they learn an entire family of PDEs, in contrast to classical methods which solve one instance of the equation. In this work, we formulate a new neural operator by parameterizing the integral kernel directly in Fourier space, allowing for an expressive and efficient architecture. We perform experiments on Burgers’ equation, Darcy flow, and the Navier-Stokes equation (including the turbulent regime). Our Fourier neural operator shows state-of-the-art performance compared to existing neural network methodologies and it is up to three orders of magnitude faster compared to traditional PDE solvers."
https://www.technologyreview.com/2020/10/30/1011435/ai-fourier-neural-network-cracks-navier-stokes-and-partial-differential-equations/?utm_term=Autofeed&utm_campaign=site_visitor.unpaid.engagement&utm_medium=tr_social&utm_source=Facebook&fbclid=IwAR3HPDxmTVGrYLdfUzBFL7KeHiSlfN57dmZx2IStwA4dNpywqccY6Ip_9sk#Echobox=1604049241
ts conda create condaR403
# Hong will install anaconda R403 in a conda environment on ts117. This strategy worked.
conda create --name condaR403
environment location: /home/hqin/.conda/envs/condaR403
-bash-4.2$ conda activate condaR403
conda install -c r r-base #??
# which R shows an R403 inside an conda environment.
R
install.packages('tidyverse') #this seems worked.
install.packages('EpiNow2') #this run for a while
non-zero exit again due to V8.
(condaR403) -bash-4.2$ conda install -c conda-forge libv8
$ conda install -c conda-forge r-randomcolor
R
install.packages('EpiNow2') #this worked!!!!
> library(EpiNow2)
>
module load sge
qsub epinow2.pbs #this runs!!!!
(condaR403) -bash-4.2$ cat epinow2.pbs
#!/bin/bash -l
#$ -S /bin/bash
#$ -N epinow_job
#$ -V
#$ -cwd
. /etc/profile.d/modules.sh
module load anaconda/5.2.0
source activate condaR403
R -f batch_Rt_by_county.R --args 900 901 1 4/1/2020 5/1/2020
Friday, October 30, 2020
ts conda tsR403, tidyverse, EpiNow2 installation
# THIS DID NOT WORK
-bash-4.2$ module load anaconda/5.2.0
-bash-4.2$ conda create --name tsR403
Collecting package metadata: done
Solving environment: done
## Package Plan ##
environment location: /home/hqin/.conda/envs/tsR403
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate tsR403
#
# To deactivate an active environment, use
#
# $ conda deactivate
-bash-4.2$
-bash-4.2$ conda activate tsR403
(tsR403) -bash-4.2$ conda install -c conda-forge libv8
Collecting package metadata: done
Hong then run R
install.packages('tidyverse') #this seem to worked.
-----------------------------[ ANTICONF ]-------------------------------
Configuration failed to find the libv8 engine library. Try installing:
* deb: libv8-dev or libnode-dev (Debian / Ubuntu)
* rpm: v8-devel (Fedora, EPEL)
* brew: v8 (OSX)
* csw: libv8_dev (Solaris)
To use a custom libv8, set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
---------------------------[ ERROR MESSAGE ]----------------------------
<stdin>:1:16: fatal error: v8.h: No such file or directory
compilation terminated.
-------------------------------------------------------
(tsR403) -bash-4.2$ conda install -c conda-forge r-randomcolor
Collecting package metadata: \
#this install many packages
Ref: https://github.com/iaconogi/bigSCale2/issues/19
I then tried:
R: install.packages('EpiNow2') #this seems to be running now.
library(EpiNow2) #it worked!
install.packages('woldmet')
#The entire install seem to take almost 2 hours.
Checked a few hour later, EpiNow2 installation did not work.
list of MSI
minority serving instutitons, rugters
https://cmsi.gse.rutgers.edu/sites/default/files/MSI%20List.pdf
Thursday, October 29, 2020
Ubuntu 18.04 LTS apt get install R4.0.3
see:
https://askubuntu.com/questions/1237102/problem-installing-r-4-0-on-ubuntu-18-04
sudo apt remove r-base
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
sudo add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/'
sudo apt updatesudo apt install r-base
Motely on geo-tagged tweets
COVID19 tweets with geo tag at IEEE
I have been pulling data from here daily since then, https://ieee-dataport.org/open-access/coronavirus-covid-19-geo-tagged-tweets-dataset because I thought we wanted to find a tweet source with the geolocation included.
Lambda Quad specifics
2565 3rd St, Suite 244
San Francisco, CA 94107
enterprise@lambdalabs.com
Operating System: Ubuntu 18.04 + Lambda Stack
- Software: TensorFlow, PyTorch, Caffe, Keras, CUDA, cuDNN
- Processor: Intel Core i9-9820X (10 Cores, 3.30 GHz)
- CPU Cooler: Air Cooling
- GPUs: 4x RTX 2080 Ti
- Memory: 64 GB
- Operating System Drive: 2 TB NVMe (3,500 MB/s Read)
- Data Drive: No Data Drive
- Warranty & Support: Three years of hardware coverage, plus technical support from a Lambda engineer.
- EDU: Academic Discount Applied
1 $8,300.00 $8,300.00
Wednesday, October 28, 2020
ts117 usage
module load sge
$ module load sge/2011.11p1
For local R package installation, it seems to go to
info man
-bash-4.2$ ls /cm/shared/apps/R/R-3.4.3/share/info/
I ran R/3.5.3
install.packages('tidyverse') # it went to /cm/shared/apps/R-3.5.3
Tuesday, October 27, 2020
non-typical locations in JHU covid data sets
Out of 'AL' style
cruise ships
federal correction facilities
unassigned cases (to counties or states),
FIPS of 99999
"For records corresponding to parts of or entire county entities that do not overlap any 2010 urban area, the urban area code is 99999, the urban area name is “Not in a 2010 urban area”, and the urban area population, housing unit count, total area, and land area values are null. The percent values relating to the urban area are also null."
Ref: https://www.census.gov/programs-surveys/geography/technical-documentation/records-layout/urban-area-record-layouts.html
Sunday, October 25, 2020
humidity and virus transmission
Low ambient humidity impairs barrier function and innate resistance against influenza infection
https://www.pnas.org/content/116/22/10905
Saturday, October 24, 2020
Thursday, October 22, 2020
CITI RCR training,
Go to CITProgram.org, register using UTC emails,
https://about.citiprogram.org/en/homepage/
Choose the University of Tennessee at Chattanooga as "Affiliation". If you do not see UTC, choose "Add affiliation".
Click "Add a course" (see image below)
Scroll down to Question 4,
For biology undergraduates, click "Biomedical RCR Course"
For computer science undergraduates, click "Computer Science RCR Course"
For graduate students, in addition to the undergraduate courses, please also click
"Advanced topics in RCR courses" (see image below)
Answer any requested question (for example Covid19 questions)
At the main page, choose a relevant module, click "Start Now".
Wednesday, October 21, 2020
Tuesday, October 20, 2020
2020 SACNAS – The National Diversity in STEM Virtual Conference
2020 SACNAS – The National Diversity in STEM Virtual Conference
Dates: October 19 – 24
Location: Online!
https://www.2020sacnas.org/
guest lecture at U of Arkansas bioinformatics seriees
https://ualr.edu/bioinformatics/education-series/
Friday, October 16 at 2:00 pm CST
Recorded Session available at https://youtu.be/6q9xHV5VznM
Speaker Topic
Dr. Hong Qin is an Associate Professor in the Department of Computer Science and Engineering at the University of Tennessee – Chattanooga who uses computational and mathematical approaches to investigate biomedical and biological questions. One focus is to develop probabilistic gene network models to infer network changes during cellular aging. We build gene network models from heterogeneous genomics data sets, including protein interactions, gene expression data sets, RNAseq data sets, protein mass-spec data sets, high-throughput phenotypic screens, and gene annotations. We are developing machine-learning methods to automatically estimate cellular lifespan from time-lapsed images. We are also applying engineering principles to study molecular, biological, and ecological networks. We are developing deep-learning methods for better classification and prediction using heterogeneous biomedical and biological large data sets. Dr. Hong Qin is a recipient of a NSF CAREER award 2015-2020. Qin’s expertise: Graph reliability modeling; Bioinformatics; Computational genomics; Mathematical modeling; Systems Biology; Cellular aging; Gene network analysis and modeling Dr. Qin will present how to use R to analyze COVID 19 data. R (along with bio-python and bio-perl) is one of the top choices for analyzing life science data. R is open source, runs on multiple platforms (Windows, Linux, MacOS, and cloud), has excellent packages for analyzing genomic data (e.g., Bioconductor), and has several nice interfaces for developing and running code (R Studio and Jupyter Notebook). The R code designed for this demo is available from https://github.com/hongqin/Use-R-in-CoLab/blob/master/Learn_R_UALR_CoLab.ipynb. This tutorial runs in Google’s CoLab cloud so no local installation is needed.
Tuesday, October 13, 2020
Thursday, October 8, 2020
Wednesday, October 7, 2020
Heckman correction
https://en.wikipedia.org/wiki/Heckman_correction
"Conceptually, this is achieved by explicitly modelling the individual sampling probability of each observation (the so-called selection equation) together with the conditional expectation of the dependent variable (the so-called outcome equation)"
The Heckman used Mills ratio to model right-censored data as 'selection bias'.
selection bias corrected by Mills ratio (inverse of Gompertz mortality rate?)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3659766
`selection bias' using the inverse Mills ratio neglected by epidemiologists.
https://en.wikipedia.org/wiki/Mills_ratio
Mills ratio is the Survival function / pdf,
The Gompertz function is u = - dS/dt * 1/S, so, HQ thinks Mills ratio is the inversion of the Gompertz mortality rate.
related to
https://en.wikipedia.org/wiki/Heckman_correction
smoothing parameter
over-smoothed curves may lose some details
rough or over-sensitive smoothed curse may be too noisy. An ideal choice of smoothing parameter can be obtained by appropriately contructed error measures, such as residuals.
Ref: Sharma, 2000, J of hydrology, seasonal to interannaul rainfall probabilistic forecasts for improved water supply management.
mutual information criterion for time series analysis
mutual information criterion for time series analysis
cran
https://rdrr.io/cran/tseriesChaos/man/mutual.html
Tuesday, October 6, 2020
Monday, October 5, 2020
how to cite ERS5-land
How to cite ERA5-Land
(1) Please acknowledge the use of ERA5-Land as stated in the Copernicus C3S/CAMS License agreement:
"5.1.2 Where the Licensee communicates or distributes Copernicus Products to the public, the Licensee shall inform the recipients of the source by using the following or any similar notice: 'Generated using Copernicus Climate Change Service Information [Year]'.
5.1.3 Where the Licensee makes or contributes to a publication or distribution containing adapted or modified Copernicus Products, the Licensee shall provide the following or any similar notice: 'Contains modified Copernicus Climate Change Service Information [Year]';
Any such publication or distribution covered by clauses 5.1.1 and 5.1.2 shall state that neither the European Commission nor ECMWF is responsible for any use that may be made of the Copernicus Information or Data it contains."
(2) cite the ERA5-Land dataset (as part of the bibliography) as follows:
- For "ERA5-Land hourly data from 1981 to present" downloaded from the Climate Data Store:
Muñoz Sabater, J., (2019): ERA5-Land hourly data from 1981 to present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS). (<date of access>), 10.24381/cds.e2161bac
- For "ERA5-Land monthly averaged data from 1981 to present" downloaded from the Climate Data Store:
Muñoz Sabater, J., (2019): ERA5-Land monthly averaged data from 1981 to present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS). (<date of access>), 10.24381/cds.68d2bb30
Saturday, October 3, 2020
Thursday, October 1, 2020
cross correlation between two time series
correlation is a linear measure of similarity between two signals.
Cross-correlation is somewhat a generalization of the correlation measure as it
takes into account the lag of one signal relative to the other. If lag == 0,
then correlation = cross-correlation. Cross-correlation is particularly
important to assess the causal relationship between two signals in time. If you
suspect that there is a non-linear relationship between the two signals, then
you should consider measures such as mutual information and partial mutual
information, which are the information-theoretic equivalent of correlation and
cross-correlation.
You can refer to the comment above for information on how to
compute correlation and cross-correlation. Check this paper instead (and
references therein) for details on mutual information and partial mutual
information http://www.sciencedirect.com/science/article/pii/S0022169400003462
https://youtu.be/6ao9-39zw40 https://youtu.be/L6YJqhbsuFY
ccf in R Ziwei Ma:
I checked the ccf function which calculates the correlation for x_{t+h} and y_t, so in our case, the ccf( dailyCases (x), dew_points (y) ) report they are positively relative, and the peak happen at arround 7 which says dailyCases lags dew_points or dew_points predict dailyCases in 7 days.
I checked the ccf function which calculates the correlation for x_{t+h} and y_t, so in our case, the ccf( dailyCases (x), dew_points (y) ) report they are positively relative, and the peak happen at arround 7 which says dailyCases lags dew_points or dew_points predict dailyCases in 7 days.
The
following website have more details there.
https://online.stat.psu.edu/stat510/lesson/8/8.2
Subscribe to:
Posts (Atom)