coreweave/dataset-downloader

dataset-downloader's Issues

Issue downloading datasets

Hi! I'm taking a look at CoreWeave's LLM documentation, and I'm trying to get GPT-J fine-tuned with the example dataset.

There seems to be an issue with the dataset-downloader -- my PVC does not get populated with the dataset.

I'm currently trying to understand if there's an issue with the path being passed to it, or if the internals of main.go aren't working correctly. Debugging this is a bit more involved because the dataset-downloader is running in a GitHub-produced distroless container, so it will take some extra effort to inspect the state during run time.

My K8s logs for the relevant pod only show this:

2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/140
2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/0
2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/20
2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/40
2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/60
2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/80
2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/100
2023/01/17 20:04:12 Getting book links from https://www.smashwords.com/books/category/1245/downloads/0/free/any/120

I'd expect the logs to also show something like the following (from this Printf statement):

Downloaded XYZ to /data/finetune-data/dataset/xyz

Anyway, I'm still debugging this, but just wanted to reach out and see if you have suggestions in the meantime. Thanks in advance!

Clean up and reify `dataset_downloader`

We should make it more useful and less hard-coded, as it's hard-coded to Western Romance, for example.

• Remove hardcoded assumptions and parameterize them.
• Parallelize the downloader using goroutines
• Potentially support other sites.