tymichaelsen/covid19

Bioinformatic pipelines used to assemble SARS-CoV-2 genomes from DK

covid19's Issues

What happens to the fastq in /processing/ after livebasecalling is done?

Create an nwk file for microreact

• Assign a fake id for each case in the microreact.tsv output and save it in a dictionary
• Read the .nwk file whose path is in the config parameter named "clade_tree_path" and replace the IDs with the fake IDs
• output the tree to the output path given in the config parameter named: "out_react_nwk"

Assigned to Miro

Create wiki page for neo4j users

Basic neo4j use and a few sample analyses:
Table of the paper: https://docs.google.com/document/d/1hVjJQdUld0qeZG-OmzGyHZ_rJiHa_5V87544BHrVfVE/edit

Pregnant men

Active cases parish / municipality

Create next-strain csv file

• Update mysql conversion script to latest tsv file
• Add municipality code to mysql convert script
• Create MySQL query to extract data and group by week, municipality and strain
• Find a data source with Long. Lat. for each Danish municipality @miro
• Add fake ID for each group
• Create a script to extract result, filter out less than 3 rows and write to csv

Create a mockup of the website with Mads and Thomas

• Ask Mads and Thomas for a list of content they wish to have @tomersagi
• Ask Katja about AAU website designing services/templates @tomersagi
• Get access to the covid19 server and install website building system (vue js) @miro
• Design a mock in pptx of the website @MiroPakanec
• Alter mock files - show all cases and sequenced cases on a graph (by all, age groups, region)

Create conversion script for metadata to neo4j and mysql format

• Write script to convert data to cypher create statements
• Write script to convert metadata.tsv to SQL Create statements

Cannot connect to Neo4J instance created from sif file

To reproduce:

1. Run ./start_neo4j.sh
2. sudo singularity shell instance://ne4j1
3. Singularity> curl http://localhost:7474/
   You should get a welcome message from neo4j.
4. exit
5. curl http://localhost:7474/
6. You get a 'Connection refused' message.

Ideas:

1. Perhaps when connecting to the instance, neo4j is getting a connection request not from localhost - check neo4j connection logs

Figure out how to get adhoc markers from Excel such as the mink case in /srv/rbd/covid19/metadata/raw-ssi-metadata/2020-06-29_nordjylland-case.xlsx

Prepared event framework. Now try it on Excel.

Enable tranmission lines for local aupsice

It would be nice to have this information beside location plot on the map.

Review Table 1 from the paper and report the same thing.

• convert 12K xlsx file to csv/tsv in same format as metadata file
• prepare SQL queries for analyses
• run over coviddb and record results
• run mysql over 12K file and analyze again

Clade ingestion fails to create edges with people

e.g., clade 20C/C3743T should be matched with at least hCoV-19/DK/ALAB-SSI-160/2020 and hCoV-19/DK/ALAB-SSI-159/2020

covid19_workflow.sh needs full path to input folder to work - should be relative path

Create ingestion process for international clade file

Example is in /srv/rbd/covid19/global_data/GISAID_clade_assignment.tsv

Create mySQL instance singularity container

TODO

Publish the Covid-19 Data in Denmark as KG

• Make a list of relevant KG to copy/link to and update in the wiki
• Establish a SPARQL endpoint - ask Gabriela
• Start a script to convert the LL data to RDF
• Start a script to convert the clade data to RDF

Move wiki ssi schema to a secure google doc

Share with possible collaborators

Add global nextstrain data to neo4j

• obtain data from https://nextstrain.org/ncov/global?c=region
• map schema to our own
• modify neo4j loader script to use both files

Add bloom over neo4j

Install, etc.

Add PPHMyAdmin to the MySQL server container

It would simplify working with the database if we could use a GUI to look at the created table data and schema info.

https://www.phpmyadmin.net/

Reclassify GISAID data to match the Danish strain naming system

current GISAID data is using naming systems for strains that are endemic to the researcher that created the data.
We need to run it through the same pipeline as our data goes through in order to be able to integrate it with our data.

Creating cingularity sif file for pentaho fails

Thanks, but the build fails for some reason which I can't figure out.
Any ideas? It managed to install all I asked it and fails on the creation itself:

[[email protected]@a1024-gc1-e-03:/srv/rbd/covid19/git/covid19/bi_system] [base] master(+5/-4)+* ± singularity build --fakeroot pentaho.sif pentaho.def ...
INFO: Adding labels
INFO: Adding environment to container
INFO: Adding runscript
INFO: Creating SIF file...
FATAL: While performing build: while creating SIF: while creating container: container file creation failed: open /srv/rbd/covid19/git/covid19/bi_system/pentaho.sif: permission denied

@KasperSkytte @biocyberman ideas?

Add low/high SES regions to neo4j data

Ghettos etc.

map is not included in .svg auspice

Since the update to dropdown table in auspice the geographic map is not included when downloading a .svg of the current session

Count cases per region in the original Danish metadata and filter microreact output accordingly

• Find the Danish full-case-list @tomersagi
• Change the output query to output single cases @tomersagi
• Write a script to count the cases by week and region. @miro
• Use this count in our to_microreact script to filter out region-week cases with less than 3 cases (reduce to 0) @miro

add message for barcode-match

Would be helpful for debugging that the demultiplex.sh script outputs information on if barcode labels in sample_sheet.csv were found in the data and if not prints out which were not found.

Cleansing script is not outputting parish code for any case

The cleansing script is not outputting parish code for any case. See output file here: ../../../bisystem/staging/danish_metadata.tsv

Metadata contains supicous pregnancy data

There are 102 cases of a person having Sex='M' and Pregnancy=1. Seems unreasonable.
@TYMichaelsen maybe you can help track this down?

Create a script to run the full pipeline to microreact

0. Find the latest result folder and update the config file
1. Run cleansing
2. Run to_mysql
3. run to_microreact

• Clean up the scripts
• Scripts should write to a log file.
• Print out the log files when done.
• Write files to ftp folder using a date naming system with a folder named latest for the latest files.
• Attach to nextstrain script (Rasmus)
• Use replace uuid with gisaid_id`
• Generate website data files
• Send website data to the server
• Change NUTS3 to NUTS2
• Setup sftp for microreact
• Change config with an authorized user

Add a relation between clades

• Create hierarchical relation by Cypher statements
• Add code to metadata conversion script to do so

Add script that changes ulimit and then runs neo4j

you need to run ulimit -n 40000 before you start neo4j from the same shell that is running the neo4j script. Can't do it through an exec command.

Create data cleansing script

• Figure out where to trigger the script -- @KasperSkytte please advise on this issue
• Make list of boolean fields and look for non True/False values, convert known mistakes (0,1,Falsk, Sand), dump other non-conforming values
• Check for non-dates in date fields.
• Make a list of probable data inconsistencies and check for them, e.g. - pregnant and Male, pregnant and >45 years old / <17 years old,

Create singularity container for Pentaho BI

I want to create a user in the singularity image but do not want to store the password in the .def file for obvious reasons.
Should I perhaps create the user without the password and use a public-private keypair to login to it?
@KasperSkytte @biocyberman Ideas are welcome.