parse ERA5 GRIB for JHU counties on ts117, Nov 14-17, 7 days runs

 The template pbs is: 

-bash-4.2$ cat ts_grib_job_template.pbs 

#!/bin/bash -l

#$ -S /bin/bash

#$ -N pygrib_NUMBER

#$ -cwd

. /etc/profile.d/modules.sh

module load anaconda

source activate grib 

python parse_grib_4JHU-debug.py STARTINDEX  ENDINDEX 

-bash-4.2$ 

Qin used a python code to generate pbs jobs, and then submit them using shell script. 

-bash-4.2$ cat generate_PBS_bash.py 

import os

import re

filebatchshell = "submit_PBS.sh"

fshell = open(filebatchshell, "w")

step = 15

for i in range( 0, 600, step):

    fin = open("ts_grib_job_template.pbs", "r")

    LinesIn = fin.readlines() 

    fin.close()

    LinesIn[2] = "#$ -N pygrib_" + str(i) + "\n"

    LinesIn[7] = "python parse_grib_4JHU.py " + str(i) + " " + str(i + step) + "  \n"

            

    fileout = "ts_" + str(i) + ".pbs"

    fout = open(fileout, "w")

    for line in LinesIn: 

        fout.write(line)

    fout.close()

    buffer = "qsub " + fileout + "\n"

    fshell.write(buffer)

fshell.close()

To parse the 2m-dew point, I used generate_PBS_bash_dewpoint.py

This is a 7-day run. The last t2m output is on Nov 20, and the last dewpoint file output is on Nov 21. 

On Nov 21, I use github to backup running logs, and output csv files 

ERA5 grib file parsing

grbs = pygrib.open('2020-2021NovT2m-dewpoint.grib') #does not work

grbs = pygrib.open('2019-2020June10.grib') #works

It would be strange if grib format changed. I need to redownload the grib file. again

6pm, Qin found out that remove the time selection, and the codes runs on 2021 GRIB file. 

down load ERA5 land hourly data

 

ERA5 land hours data was last updated on July 12. 

I selected all 2020, all day, and all hours, it is 45.8 GB, which will take 12 hours to download at home.

I ssh into my simcenter workstation, 

wget url, 

which runs with an estimated time of a little over 2 hours. 

GRIB paring check, Beijing, passed

code: parse_grib.ipynb

20200501 [76.24124573 77.13069885]. OK
20200502 [72.6345166  74.09350098]. OK
20200503 [66.69898804 68.96656616] OK
20200504 [54.08201416 53.69529541] OK
20200505 [61.49418762 61.181297  ] OK
20200506 [64.57176025 65.886604  ] OK
20200507 [61.92833984 61.68576172] OK
20200508 [54.80175598 56.51738098] OK
20200509 [58.18392456 59.06986206] OK
20200510 [61.55351379 64.23945129] OK
20200511 [64.92865112 64.3063855 ] OK
20200512 [57.41510132 61.64439819] OK
20200513 [69.78394897 73.4155896 ] OK
20200514 [65.61595581 67.13822144] OK
20200515 [64.39757202 64.8581189 ] OK
20200516 [57.69824646 60.99238708] OK
20200517 [59.46649597 61.04852722] OK
20200518 [59.9135022  63.11623657] OK
20200519 [63.12549255 65.50557068] OK
20200520 [69.51305359 70.70836609] OK
20200521 [62.20648621 63.47914246] OK
20200522 [72.32934387 73.32778137] OK
20200523 [65.23289001 67.20867126] OK
20200524 [64.14851196 67.15437134] OK
20200525 [60.65716675 63.04779175] OK
20200526 [64.33808105 68.1419873 ] OK
20200527 [61.88840454 63.31223267] OK
20200528 [69.24611328 71.53478516] OK
20200529 [70.45276917 72.82933167] OK
20200530 [68.43469055 71.78508118] OK
20200531 [56.83400696 59.86447571] OK
20200601 [63.65937317 64.65781067] OK
20200602 [72.12255371 77.76864746] OK
20200603 [73.47376221 74.66204346] OK
20200604 [71.11302002 70.94778564] OK

https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-land?tab=overview

ERA5-Land hourly data from 1981 to present

Hong can only found Dec 1, 2020 data on Feb 19, 2021. 

2022-02-02. Hong found the title changed to "1950 to present", and resolution changed to 0.1 degree with a resolution about 9km. 

How to extract specific information from GRIB files?

https://gis.stackexchange.com/questions/173775/how-to-extract-specific-information-from-grib-files



https://github.com/jswhit/pygrib/issues/27

Coordinates, spatial grids
https://confluence.ecmwf.int/display/CKB/ERA5%3A+data+documentation#ERA5:datadocumentation-Spatialgrid

To find specific coordinates within this 720 x 360 array i just extrapolate the coordinates to the array. Ex: If I want to find data for New York New York coordinates ~40.5N 74 W I'll map those coordinates to the array to by doing 2x(90-40.5) and 2x (180-74) to find the following result:

grbs_old = pygrib.open('nomads.ncep.noaa.gov/cgi-bin/'+str(grib_filename))
data=grbs_old[6].values
data[99][212]

Grid box
https://confluence.ecmwf.int/display/CKB/Model+grid+box+and+time+step
Horizontal

In the horizontal the discrete points are arranged in a two dimensional grid and hence are referred to as grid points. The grid can be regular or irregular. An example of a regular latitude/longitude grid would be where the grid points are located every 1 degree of longitude in the east-west direction and every 1 degree of latitude in the north-south direction. Each grid point is associated with an area that either surrounds the grid point or lies between the grid points. This area is referred to as the "grid box".

Grid point values cannot properly represent variability on spatial scales smaller than the grid box.

(In addition to using grid points, the ECMWF Integrated Forecasting System (IFS) also uses an additional mathematical concept, spectral space, to represent horizontal space. This concept uses a set of wavy basis functions, spherical harmonics, to describe variations in the horizontal. The IFS switches between spectral space and grid point space, in order to perform specific computations.)

For Hong's Pygrib file,
grbs = pygrib.open('2019-2020June10.grib')
valuesTmp = grbs[1].values
valuesTmp.shape #return (721, 1440)

so, 720/180 = 4

1440/360 = 4

It turns out that GRIB use 0-360 longitude and many software use -180W, 180E longitude. 

Hong tried ( - 180W mod 360), and temperature results from DC matches AccuWeather results,