Clean/verify packages in another way

[LeoQuote]

Clean/verify packages in another way

[cooperlees]

verify was never tested for anything other than standard POSIX file systems. For non POSIX, I'm open to many ideas.

I'm happy to add checkpointing support to verify as a temporary workaround (to get around when it's interrupted).

Now, in regards to your idea:

• I don't like the guessing notion of your idea
• Python package metadata and naming is not super clean ... it's an old language

Python packages and their naming + metadata isn't as strictly enforced as one would like. So making guesses is going to land you in a bad place.

Now that we have the simple API in JSON, maybe pull that index file you've generated and then work on each project from that list and use the success/failed checkpoint feature I alluded to up above. Due to the site of the mirrors, it's hard to do this efficiently without getting more metadata + applying it to PyPI every time I've thought about it. But happy to be proven wrong.

[LeoQuote]

I have not found any clear MUST for project name and package name, I think you're probably right about the "guessing", we should not do that.

So about the checkpoint part, my thoughts are like follows:

Separate verify into stages

1. scan all JSON file to get project and designated package list
2. delete project that not exists anymore
3. scan all packages file to get actual package list
4. compare designated package list and actual package list, delete or download package on need

We can have persistent storage in both stage 1 and stage 3, storing package list and project list into text files, one line a project/package, if there are any properties (active, inactive, update time, etc...), separate them with comma, maybe just use CSV to make it simpler.

During stage 1,3, there's also a checkpoint to indicate which is the next package, my suggestion would be every 10000 project/package, store it in a new file, so an interruption would only lose about progress of 10000 project. The file structure would be like

verify_data/
├── desinated_package
│   ├── package_list.csv.1
│   └── package_list.csv.2
├── package
│   ├── package_list.csv.1
│   └── package_list.csv.2
└── project
    ├── project_list.csv.1
    └── project_list.csv.2

For stage 2 and stage 4, they are similar, read the files generated in the previous stage, load into memory (a dict to have O(1) complexity), and compare it.

restore from interruption

If verify is interrupted, first determine which stage were we? We can have a separate verify_status.json file to show which stage were we.

Read the last line of last package_list.csv or project_list.csv.2, continue from there, iterate file from beginning, continue process until we met the checkpoint.

If we were interrupted during stage 2 or stage 4, it's ok, these two stage are both very time-efficient, just read the designated list and actual list, compare it and schedule a deletion.

[cooperlees]

My other thought is using tables in a sqllite database maybe for the state. Comitting using something like aiosqlite as you go moving from a simple todo and done table. OR just had a runid you use until finished. But I'm open to ideas.

The process seems sane. How do you want to structure the code? Refactor verify.py? I've always wanted to merge more of t into other classes and use the storage plugins directly for POSIX filesystem vs. other storage. Let's get a similiar agreement there before you get stuck into writing it. I'm happy to chat on discord too if you want more real time chatting it out.

[LeoQuote]

I think all the things we do is to make the global info sharable in all stages of verify processes.

If the project accepts an external database or embed database in memory, I'd recommend those options:

1. sqlite3, this is rather simple
2. Rocksdb, similar to sqlite3, but just KV, no SQL knowledge required
3. external databases like MySQL, PgSQL (not recommended, make this project heavier)

The database file must be placed in POSIX filesystem as sqlite3/rocksdb on s3 would be inefficient.

About the code structure, I don't have many thoughts right now, I think maybe I can go create a demo first.

[LeoQuote]

😄 I tried to use verify in my host and failed, it took 7 days and 20G+ memory and finally OOM killed by kernel.

There are a few problems that I want to solve:

1. packages may have been changed during such long time, we should consider scan packages first to make sure no file would be deleted wrongly
2. memory usage is too high, maybe I can get a packages list at first, then scan JSON files, for each file that exists in JSON, deleted it from packages list, when the JSON iteration is over, what left in packages list would be the packages that need cleaning. That would make the memory usage less and less during the iteration process.