shap summry plot by chatGPT

 chatGPT repeatly make mistake for shap summary plot. 

for different class label, the index should be the 3rd position:  shap_vals[:,:,idx],

Mendely preprint error

 In Mendely, if an article has unspecified type, it often list it as "preprint". 

To fix this, just change the document type to 'jounral article' or 'conference proceedings' or other appropriate type. 

compare two pdf files for IEEE PDF Express conference submission

 chatgpt: To compare two PDF files for differences, you have several options:

1. Using a PDF Comparison Tool

There are tools specifically designed for PDF comparison, such as:

• Adobe Acrobat Pro DC: It has a built-in "Compare Files" feature.
• DiffPDF: A lightweight application to compare two PDFs visually or textually.
• Beyond Compare: A general comparison tool with PDF support via add-ons.
• PDF-XChange Editor: Offers PDF comparison as part of its suite.

2. Online PDF Comparison Tools

There are free and paid online services for comparing PDFs:

• Draftable: Offers a detailed side-by-side view of differences.
• PDF24 Tools: A straightforward online PDF comparison tool.

3. Python for PDF Comparison

If you'd like to script the comparison, you can use Python libraries like PyPDF2, pdfplumber, or pdfdiff. Here's an example:

Example Python Code

from PyPDF2 import PdfReader

# Function to extract text from a PDF file
def extract_text_from_pdf(pdf_path):
    reader = PdfReader(pdf_path)
    text = ""
    for page in reader.pages:
        text += page.extract_text()
    return text

# Compare two PDFs
pdf1_text = extract_text_from_pdf("file1.pdf")
pdf2_text = extract_text_from_pdf("file2.pdf")

if pdf1_text == pdf2_text:
    print("The PDF files are identical.")
else:
    print("The PDF files have differences.")

4. Using Command-Line Tools

• diffpdf: A command-line tool to compare PDFs.
• pdftotext: Convert both PDFs to text and then use a standard text comparison tool like diff or vimdiff.

hqin@Hong-MBP2 Downloads % diff-pdf -v file1.pdf file2.pdf 

page 0 has 3756 pixels that differ

page 0 differs

page 1 has 369491 pixels that differ

page 1 differs

page 2 has 0 pixels that differ

page 3 has 0 pixels that differ

page 4 has 0 pixels that differ

page 5 has 0 pixels that differ

page 6 has 0 pixels that differ

page 7 has 0 pixels that differ

page 8 has 0 pixels that differ

2 of 9 pages differ.

Visual comparison shows that IEEE PDF Express put an extract figure name in Figure 3. This png file name has extra space. So, I renamed this png file by replace white space with '_'s. This seems to fix the error. 

hqin@Hong-MBP2 Downloads % diff-pdf -v overleaf-MABEL_neuralnetwork_SHAP_IEEE2024Fall.pdf /Users/hqin/Downloads/2024402761.pdf 

page 0 has 3756 pixels that differ

page 0 differs

page 1 has 0 pixels that differ

page 2 has 0 pixels that differ

page 3 has 0 pixels that differ

page 4 has 0 pixels that differ

page 5 has 0 pixels that differ

page 6 has 0 pixels that differ

page 7 has 0 pixels that differ

page 8 has 0 pixels that differ

1 of 9 pages differ.

Visual check the first page to find they are consistent. 

alphafold-2 firefly run, error

[hqin@firefly02 alphafold-2]$ bash alphafold-singularity-run.sh --fasta_paths T1050.fasta 

Mounting /scr -> /mnt/data_dir

Mounting /scr/alphafold-data/uniref90 -> /mnt/uniref90_database_path

Mounting /scr/alphafold-data/mgnify -> /mnt/mgnify_database_path

Mounting /scr/alphafold-data/bfd -> /mnt/bfd_database_path

Mounting /scr/alphafold-data/uniclust30/uniclust30_2018_08 -> /mnt/uniclust30_database_path

Mounting 

-> /mnt/pdb70_database_path

Mounting /scr/alphafold-data/pdb_mmcif -> /mnt/template_mmcif_dir

Mounting /scr/alphafold-data/pdb_mmcif -> /mnt/obsolete_pdbs_path

Mounting /home/hqin/alphafold-2 -> /mnt/fasta_path_0

--data_dir=/mnt/data_dir/alphafold-data --uniref90_database_path=/mnt/uniref90_database_path/uniref90.fasta --mgnify_database_path=/mnt/mgnify_database_path/mgy_clusters_2018_12.fa --bfd_database_path=/mnt/bfd_database_path/bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt --uniclust30_database_path=/mnt/uniclust30_database_path/uniclust30_2018_08 --pdb70_database_path=/mnt/pdb70_database_path/pdb70 --template_mmcif_dir=/mnt/template_mmcif_dir/mmcif_files --obsolete_pdbs_path=/mnt/obsolete_pdbs_path/obsolete.dat --fasta_paths=/mnt/fasta_path_0/T1050.fasta --output_dir=/mnt/output --benchmark=0 --logtostderr --max_template_date=2021-12-31

/scr:/mnt/data_dir,/scr/alphafold-data/uniref90:/mnt/uniref90_database_path,/scr/alphafold-data/mgnify:/mnt/mgnify_database_path,/scr/alphafold-data/bfd:/mnt/bfd_database_path,/scr/alphafold-data/uniclust30/uniclust30_2018_08:/mnt/uniclust30_database_path,/scr/alphafold-data/pdb70:/mnt/pdb70_database_path,/scr/alphafold-data/pdb_mmcif:/mnt/template_mmcif_dir,/scr/alphafold-data/pdb_mmcif:/mnt/obsolete_pdbs_path,/home/hqin/alphafold-2:/mnt/fasta_path_0,/home/hqin/alphafold-2:/mnt/output

I0804 20:07:16.598631 140258065744832 templates.py:857] Using precomputed obsolete pdbs /mnt/obsolete_pdbs_path/obsolete.dat.

I0804 20:07:16.809164 140258065744832 xla_bridge.py:244] Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker: 

I0804 20:07:17.299607 140258065744832 xla_bridge.py:244] Unable to initialize backend 'tpu': INVALID_ARGUMENT: TpuPlatform is not available.

I0804 20:07:21.937146 140258065744832 run_alphafold.py:385] Have 5 models: ['model_1', 'model_2', 'model_3', 'model_4', 'model_5']

I0804 20:07:21.937372 140258065744832 run_alphafold.py:397] Using random seed 1266311393757702950 for the data pipeline

I0804 20:07:21.937706 140258065744832 run_alphafold.py:150] Predicting T1050

I0804 20:07:21.938605 140258065744832 jackhmmer.py:130] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmp8lfb38hd/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /mnt/fasta_path_0/T1050.fasta /mnt/uniref90_database_path/uniref90.fasta"

I0804 20:07:21.984835 140258065744832 utils.py:36] Started Jackhmmer (uniref90.fasta) query

I0804 20:13:41.682038 140258065744832 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 379.697 seconds

I0804 20:13:44.308897 140258065744832 jackhmmer.py:130] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmpqgyb_oib/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /mnt/fasta_path_0/T1050.fasta /mnt/mgnify_database_path/mgy_clusters_2018_12.fa"

I0804 20:13:44.343975 140258065744832 utils.py:36] Started Jackhmmer (mgy_clusters_2018_12.fa) query

I0804 20:20:40.947294 140258065744832 utils.py:40] Finished Jackhmmer (mgy_clusters_2018_12.fa) query in 416.603 seconds

I0804 20:21:01.298550 140258065744832 hhsearch.py:85] Launching subprocess "/usr/bin/hhsearch -i /tmp/tmpwk1zvn0h/query.a3m -o /tmp/tmpwk1zvn0h/output.hhr -maxseq 1000000 -d /mnt/pdb70_database_path/pdb70"

I0804 20:21:01.353056 140258065744832 utils.py:36] Started HHsearch query

I0804 20:21:01.404658 140258065744832 utils.py:40] Finished HHsearch query in 0.051 seconds

Traceback (most recent call last):

  File "/app/alphafold/run_alphafold.py", line 427, in <module>

    app.run(main)

  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 312, in run

    _run_main(main, args)

  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 258, in _run_main

    sys.exit(main(argv))

  File "/app/alphafold/run_alphafold.py", line 412, in main

    is_prokaryote=is_prokaryote)

  File "/app/alphafold/run_alphafold.py", line 164, in predict_structure

    msa_output_dir=msa_output_dir)

  File "/app/alphafold/alphafold/data/pipeline.py", line 179, in process

    pdb_templates_result = self.template_searcher.query(uniref90_msa_as_a3m)

  File "/app/alphafold/alphafold/data/tools/hhsearch.py", line 96, in query

    stdout.decode('utf-8'), stderr[:100_000].decode('utf-8')))

RuntimeError: HHSearch failed:

stdout:

stderr:

- 20:21:01.404 ERROR: In /tmp/hh-suite/src/ffindexdatabase.cpp:11: FFindexDatabase:

- 20:21:01.404 ERROR: could not open file '/mnt/pdb70_database_path/pdb70_cs219.ffdata'

 

R table return integer as chr

 

adf.test error

tryCatch fixed this running error. 

i = 2677

Error in if (k < 0) stop("k negative") : 

  missing value where TRUE/FALSE needed

Calls: adf.test

In addition: There were 50 or more warnings (use warnings() to see the first 50)

Execution halted

get_adf_pvalue (tb_local$era5_t2m)

Loving, Texas, There are zero cases in the time window. So, adf.test is meaningless. 

re-install R and Rstudio to remove segfaulty memory not mapped error.

tried reinstall R and packages

https://stackoverflow.com/questions/49190251/caught-segfault-memory-not-mapped-error-in-r

get_cajo_rank ()

There was a typo in a variable name, which took 2-3 hours to fix. 

# Function to get cajo test rank

get_cajo_rank = function( in_system, in_K= 12, in_cval_index=3, in_max_cajo_order=1  ) {

    tryCatch ( 

      {   test_tmp = ca.jo(in_system, type='trace', ecdet = 'const', K= in_K)

          ratio_test_cvalPer = test_tmp@teststat / test_tmp@cval[,in_cval_index]  #ratio of test statistics vs critical values

          number_of_positive_tests = table( ratio_test_cvalPer > 1)['TRUE'] #return the number of TRUE, which can > max_cajo_order

          output_rank = min( number_of_positive_tests,  in_max_cajo_order ) # make sure max_cajo_order is the up limit

          print(paste("get_cajo_rank():: output_rank = ", output_rank))

          return(output_rank)   

      },

      error=function(error_message) {

          output_rank = -999 # an error code

          return(output_rank)   

      })

}

 

UTC wireless loggin problem on mac laptop

 Tech asked whether VirualBox or VMWare on my Apple laptop caused the wireless connection issues. 

In log-in, I typed username but I cannot type in the password. 

As suggested by the tech, I typed in the password first, and then my username, and it worked. 

VirtualBox errors on MacOS BigSurf

 

Kernel driver not installed (rc=-1908) Getting Errors in macOS Big Sur 11.0.1

https://stackoverflow.com/questions/65149373/kernel-driver-not-installed-rc-1908-getting-errors-in-macos-big-sur-11-0-1

(base) hqin@CS313BQin ~ % sudo kextload -b org.virtualbox.kext.VBoxDrv

Password:

Executing: /usr/bin/kmutil load -b org.virtualbox.kext.VBoxDrv

Error Domain=KMErrorDomain Code=27 "Extension with identifiers org.virtualbox.kext.VBoxNetAdp,com.apple.nke.rvi,com.paloaltonetworks.kext.pangpd,org.virtualbox.kext.VBoxUSB,org.virtualbox.kext.VBoxDrv,org.virtualbox.kext.VBoxNetFlt not approved to load. Please approve using System Preferences." UserInfo={NSLocalizedDescription=Extension with identifiers org.virtualbox.kext.VBoxNetAdp,com.apple.nke.rvi,com.paloaltonetworks.kext.pangpd,org.virtualbox.kext.VBoxUSB,org.virtualbox.kext.VBoxDrv,org.virtualbox.kext.VBoxNetFlt not approved to load. Please approve using System Preferences.}

(base) hqin@CS313BQin ~ % 

Checked the "Security & Privacy", Restart the laptop to make the update function. 

This worked, and the error went over. 

auto cajo pbs on ts117

 job in EQW status? !

This does NOT worked for batch cointegration using Johansen test

-bash-4.2$ 

-bash-4.2$ cat cajo.pbs 

#!/bin/bash -l

#$ -S /bin/bash

#$ -N epinow_job

#$ -V

#$ -cwd

. /etc/profile.d/modules.sh

module load anaconda/5.2.0

source activate r2021nov

EpiNow2 runntime errors

This runntime error seems to be just a warning, because Rt was still reported.  

https://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup

Logging threshold set at INFO for the EpiNow2 logger

Writing EpiNow2 logs to the console and: /var/folders/bw/k6_tkc2142v1wh5r_1yhkqhc0000gp/T//Rtmpb4xcT4/regional-epinow/2020-10-27.log

Logging threshold set at INFO for the EpiNow2.epinow logger

Writing EpiNow2.epinow logs to the console and: /var/folders/bw/k6_tkc2142v1wh5r_1yhkqhc0000gp/T//Rtmpb4xcT4/epinow/2020-10-27.log

WARN [2021-11-09 19:42:32] epinow: There were 2 divergent transitions after warmup. See

http://mc-stan.org/misc/warnings.html#divergent-transitions-after-warmup

to find out why this is a problem and how to eliminate them. - 

WARN [2021-11-09 19:42:32] epinow: Examine the pairs() plot to diagnose sampling problems

RStudio freezing and QT Webengine eating CPU,

This bug was solved by re-install R. 

https://community.rstudio.com/t/rstudio-freezing-and-qt-webengine-eating-cpu/28019

R write.csv versus write.csv2

write.csv uses "." for the decimal point and a comma for the separator.

write.csv2 uses a comma for the decimal point and a semicolon for the separator, the Excel convention for CSV files in some Western European locales.

python and ipython has different path, modules not found

(base) CS313BQin:~ hqin$ python

Python 3.7.4 (default, Aug 13 2019, 15:17:50) 

[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import sys

>>> sys.path

['', '/opt/anaconda3/lib/python37.zip', '/opt/anaconda3/lib/python3.7', '/opt/anaconda3/lib/python3.7/lib-dynload', '/opt/anaconda3/lib/python3.7/site-packages', '/opt/anaconda3/lib/python3.7/site-packages/aeosa']

import sys

sys.path.append("/opt/anaconda3/lib/python3.7/site-packages")

This solved the seaborn module loading problem in jupyter-notebook for big macbookpro

However, there is still a loading problem for libriso

Reference:
https://stackoverflow.com/questions/15514593/importerror-no-module-named-when-trying-to-run-python-script/15622021#15622021

git push problem on ts117

git push, remote hung up unexpected, (after 1 hour, I realized I inadvertently added a 2.3 G grib file) 

I tried to increase HTTP post buffer
https://confluence.atlassian.com/stashkb/git-push-fails-fatal-the-remote-end-hung-up-unexpectedly-282988530.html

git config --global http.postBuffer 157286400 #did not work. same error. 

it did not work.

Workaround: tar gzip, transfer file from ts117 back to laptop through VPN. still not working.

(base) CS313BQin:parse_GRIB_JHUCSSE_COVID19 hqin$ du -sh

6.8G .

(base) CS313BQin:parse_GRIB_JHUCSSE_COVID19 hqin$ export GIT_HTTP_MAX_REQUEST_BUFFER=100M

(base) CS313BQin:parse_GRIB_JHUCSSE_COVID19 hqin$ git config --global http.postBuffer 100000000

(base) CS313BQin:parse_GRIB_JHUCSSE_COVID19 hqin$ git push

Enumerating objects: 165, done.

Counting objects: 100% (165/165), done.

Delta compression using up to 12 threads

Compressing objects: 100% (162/162), done.

Writing objects:   2% (4/163), 174.49 MiB | 10.24 MiB/s 

MacOS Jupyternote book pdf feature bug fix

On my new Macbook pro, I cannot generate PDF from JupyterNotebook.

This is old OS.

I added the following line to .bash_profile.

#Texlive 20200323

export PATH=$PATH:/Library/TeX/texbin

I then

source .bash_profile

I then restart JupyterNotebook, and pdf file can be generated.

macbook pro black screen problem

macbook pro, 15 inches, screen is black after failed wakeup from sleep.

https://macpaw.com/how-to/fix-mac-black-screen

it works when connects to an VGA monitor.

Run disk unility, fixed volume issue, restart, still black screen.

===============Tried this, still black screen===========

Reset your Mac’s NVRAM settings

NVRAM stands for non-volatile RAM and is basically a functionality that stores in memory settings for display, speakers, primary startup disk etc. Everything even remotely connected with boot process (like your MacBook won’t turning on) can be set back to factory settings during the NVRAM reset.

To flush NVRAM settings:

1. Shut your Mac down.
2. Press the Power key.
3. Wait for your Mac to start loading.
4. When you hear a startup sound, hold down Cmd + Option + P + R.
5. Keep pressing the keys until you hear a second startup sound

==================================

Zap your Mac’s SMC settings

SMC is System Management Controller. What it controls is temperature, lights, keyboard, fans, and many other side-processes. According to Mac support forums, resetting SMC helps in 90% of the cases when your Mac’s screen goes black. To perform this trick follow the tips below.

For a MacBook with a non-removable battery:

1. Shut your Mac down. 
2. Connect your Mac to a power outlet. 
3. While your Mac is still shut, Press Shift + Option + Control and Power key at the same time. 
4. Let go of the keys and boot your Mac again.

==================================

Update OS X system

https://support.apple.com/kb/DL1969?locale=en_US

upgrade from 10.12.6 Sierra to 10.13 High Sierra

Downloaded high Sierra dmg file. Installed. Restarted not responding after 3-5 minutes.

14:47, hard-shutdown, and restart again. Fan can be heard running.

ipad zoom meeting sharing bug fixed

Zoom cannot share screen on ipad.

Removed and reinstalled Zoom, fixed this problem.

ridgesize running error, sda1 space out

I run 10 jobs on ridgesize, with debuging output to text file.  These text files run up to 450G in size, and take all spaces in /tmp/ in /dev/sda1/.