pepkit/geopephub

Automatic uploader of GEO metadata to PEPhub.

geopephub's Issues

conflict of sqlalchemy versions

Now pepdbagent is using new sqlalchemy (version >=0.2.0), sqlmodel is using old version of sqlaclhemy (<=1.4.0). It leads to conflicts, and uploader is not working.
Two solutions:

1. Change sqlmodel to sqlalchemy
2. Use old, modified version of pepdbagent. We already have it:)

Add link to gse in the description

Correct upload date and update date in geo and bedbase namespaces

When projects were uploaded to the pephub from geo, the upload and update date was set as date of upload to pephub, change it to geo dates

Using of Environment variables

Add .env file, which metageo_pephub will use to run commands.

Add feature that dumps all geo projects to one zip file

We need to add a new feature to do a routine database dump. So, I think this is doable, but it's not currently set up. The good thing about this is that since we're automatically indexing and parsing GEO daily as new records are added, you can always come back later and just get a new one to rerun on a more up-to-date version.

Next steps (increment of value)

Tasks before first stable version:

• Add to geofetch soft file size checker.
• Add time limit to the gse populator
• Make script fail resistant

Add upload function of gse by period of time

Now, there is no possibility to check or upload GSE projects using specified start and end date. Add function that will receive start period, end period and 1)check if we have this period in the queue 2) add to the queue 3) add project to the pephub

Exit Code 137 - Out of Memory

While downloading GSE178610 GitHub action failing:

Trying GSE178610 (not a file) as accession...
Skipped 0 accessions. Starting now.
Processing accession 1 of 1: 'GSE178610'
/home/runner/work/_temp/4dfe2662-0b6d-4b4d-84ff-aad23b8dfa34.sh: line 1:  1810 Killed                  python geo_pipeline_script.py --namespace geo_recent --host *** --db *** --user *** -***
Error: Process completed with exit code 137.

I am not sure why this error is happening. I guess, it's because geofetch is creating to huge annotation dict or list.
Do you have any ideas how to handle this error and if it occurs how to skip this accession? @nleroy917 @nsheff

Add statusbar to downloading project

Add to db/Create file with information about passed and failed uploads.

Now, a lot of files are failing to upload. We have to save this logs and upload that files later manually. We should add this pipeline to our script.

Error while uploading project, because connection already closed

Add functionallity that deals with closed connection
Errors:

Error whiled downloading: GSE155854
Error message: connection already closed

https://github.com/pepkit/geo_pephub/actions/runs/3427197775/jobs/5709905888#step:5:2228

Automated fetching experiment metadata from GEO that contain bed files

Fetch metadata from GEO with bed files and run this process every day, or week.

• Fetch new metadata with bed files every week for files
• Check every month if all metadata was uploaded

On some of the random days there was no projects uploaded

On some of the days there was no projects uploaded. Investigate this issue: