cajo results summary for publication, error found!

working code:   _generate_tables_for_publication.Rmd

10% critical value

Rt ~T2m 

# 35.6% counties has positive ca.jo Rt ~ T2m at 10% critical value, 28.2% at 5%, and 14.9% at 1% critical value. 

cajo Rt - Apple driving

cajo Rt - google workplace 

# cajo Rt, Apple driving, T2m

ERROR DISCOVERed. 

Based on https://www.quantstart.com/articles/Johansen-Test-for-Cointegrating-Time-Series-Analysis-in-R/,  when three time series are used, r = 3 means three time series are needed to generate a stationary series!!! 

So, its seems eigen values can be used to evaluate the combination of time series for stationary combinations. 

dewpoint weakly correlates with RH

 NOAA dewpoint  correlate weakly with RH, R2=0.11 for Hamilton, TN

ERA5 dewpoint correlates with NOAA dewpoint well. R2 = 0.97

ERA5 Grib t2m correlates well with NOAA air_temp at Hamilton, TN

 source code:  
https://github.com/QinLab/COVID19_transmission_MS/blob/main/hamiltonTN_compare_T2m_NOAA.Rmd 

Call:
lm(formula = tb_comp$K ~ tb_comp$air_temp)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.7346 -0.9967 -0.0165  0.9527  7.6325 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)      271.41583    0.14303  1897.6   <2e-16 ***
tb_comp$air_temp   0.96368    0.00688   140.1   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.492 on 597 degrees of freedom
Multiple R-squared:  0.9705,	Adjusted R-squared:  0.9704 
F-statistic: 1.962e+04 on 1 and 597 DF,  p-value: < 2.2e-16

parse ERA5 GRIB for JHU counties on ts117, Nov 14-17, 7 days runs

 The template pbs is: 

-bash-4.2$ cat ts_grib_job_template.pbs 

#!/bin/bash -l

#$ -S /bin/bash

#$ -N pygrib_NUMBER

#$ -cwd

. /etc/profile.d/modules.sh

module load anaconda

source activate grib 

python parse_grib_4JHU-debug.py STARTINDEX  ENDINDEX 

-bash-4.2$ 

Qin used a python code to generate pbs jobs, and then submit them using shell script. 

-bash-4.2$ cat generate_PBS_bash.py 

import os

import re

filebatchshell = "submit_PBS.sh"

fshell = open(filebatchshell, "w")

step = 15

for i in range( 0, 600, step):

    fin = open("ts_grib_job_template.pbs", "r")

    LinesIn = fin.readlines() 

    fin.close()

    LinesIn[2] = "#$ -N pygrib_" + str(i) + "\n"

    LinesIn[7] = "python parse_grib_4JHU.py " + str(i) + " " + str(i + step) + "  \n"

            

    fileout = "ts_" + str(i) + ".pbs"

    fout = open(fileout, "w")

    for line in LinesIn: 

        fout.write(line)

    fout.close()

    buffer = "qsub " + fileout + "\n"

    fshell.write(buffer)

fshell.close()

To parse the 2m-dew point, I used generate_PBS_bash_dewpoint.py

This is a 7-day run. The last t2m output is on Nov 20, and the last dewpoint file output is on Nov 21. 

On Nov 21, I use github to backup running logs, and output csv files 

ERA5 grib file parsing

grbs = pygrib.open('2020-2021NovT2m-dewpoint.grib') #does not work

grbs = pygrib.open('2019-2020June10.grib') #works

It would be strange if grib format changed. I need to redownload the grib file. again

6pm, Qin found out that remove the time selection, and the codes runs on 2021 GRIB file. 

down load ERA5 land hourly data

 

ERA5 land hours data was last updated on July 12. 

I selected all 2020, all day, and all hours, it is 45.8 GB, which will take 12 hours to download at home.

I ssh into my simcenter workstation, 

wget url, 

which runs with an estimated time of a little over 2 hours. 

https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-land?tab=overview

ERA5-Land hourly data from 1981 to present

Hong can only found Dec 1, 2020 data on Feb 19, 2021. 

2022-02-02. Hong found the title changed to "1950 to present", and resolution changed to 0.1 degree with a resolution about 9km.