Comments (11)
- Created extra issue for the split of train/test data,
- Found solution for providing the ebird_codes in a seperate file
- Created a notebook to pack files in evenly sized .tar files (tar is recommanded by hf)
- Streaming the dataset is now also possible
HF repo: https://huggingface.co/datasets/DBD-research-group/gadme_v1_1/tree/main
from birdset.
from birdset.
- git cloning is not recommended: https://huggingface.co/docs/huggingface_hub/concepts/git_vs_http
- pushing with git lfs leads to errors
- HfApi Client should be used programmatically
from birdset.
Repo
https://huggingface.co/datasets/DBD-research-group/gadme_v1
from birdset.
Lessons Learned
- The
loading_script
file has to have the same name as the repo! Otherwise, it reverts back to some default loading which does not really work. - If you want to debug the loading script, I would strongly suggest cloning the huggingface repository. Then you can directly call the loading script within the
load_dataset
method and debug it accordingly:
ds = load_dataset("gadme_clone_repo/gadme_cloned_repo.py", "sapsucker_woods")
or:
ds = load_dataset("gadme_clone_repo", "sapsucker_woods")
- Cloning and pushing the repo (and the files) is not recommended. Use the HfAPI to push something to the repo: https://huggingface.co/docs/huggingface_hub/concepts/git_vs_http. There is an example notebook in the
dataset/data/huggingface_hub
directory in the GADME repo - The download_manager https://huggingface.co/docs/datasets/v2.14.4/en/package_reference/builder_classes#datasets.DownloadManager loads the files that are stored locally in cache (metadata + audio files in this case) .
- I zipped all the files (this has to be sharded in parquet files, I guess)
metadata.zip (consists of train.csv and test.csv)
files.zip
(all .ogg files)
- The files are downloaded and cached in the split generator and then called with the
_generate_examples
method - This method loads the cached metadata file and then iterates over every row with the names of the files from the metadata
Repo Structure
Example
├──data
│ ├── sapsucker woods
│ │ ├──meta.zip
│ │ ├──files.zip
│ ├── amazon basin
│ │ ├──meta.zip
│ │ ├──files.zip
├─gadme.py
TODOS
- Add all features from the main metadata.csv file (right now, only a fraction of the features are added as an example
- Add all subtasks and create train and test data (should be an issue by itself)
- Right now, the ebird_codes (the label names for the ClassLabel Features) are added manually in a list. Unfortunately, the .txt file with the codes are loaded before the
.info
method is called. Maybe there is a more sophisticated workaround. Otherwise, we would have to provide a list for every sub-task. I looked at some example codes - They always provided a list (to be fair, they did not have the same number of labels we have). - Use a better approach to save the data. Maybe the
.zip
files work, and we do not need to put in more effort (we should ask the hf people). Otherwise, we have to change to the shard and parquet format. - I did not yet test the working flow I suggested in notion (
.encode_column
->.map
->.set_transform
).
from birdset.
I assigned you now. Please read the lessons learned, extract some issues from this, and then close this issue, when you are done. Thanks! :)
from birdset.
@Moritz-Wirth What's the status here?
from birdset.
Thanks! Looks good :)
from birdset.
@Moritz-Wirth everything done, has to be pushed
from birdset.
- add "no_call" class to classes.py (@ stefan)
- add only zenodo task to dataset builder
- fix multiclass 5s
- add fsl dataset
from birdset.
@lurauch this is done, right?
from birdset.
Related Issues (20)
- Implement instance peak normalization
- remove no_call class
- Add EfficientNet, MobileNet, and ConvNext models
- clean up transform and make it faster
- transform is loaded multiple times
- Paper: add dataset table HOT 1
- How to save pretrained weights HOT 2
- Update XC data files HOT 1
- Auswertung von BirdNet, Perch [done] HOT 5
- check test_5s splits
- Tutorial: add Model training section HOT 1
- Default Dataloaders
- trainer devices cant be set to specific gpu
- Class limit for datamodules in the multi-label setting
- Implement ConvNext and EfficientNet as HF models
- Extend Perch logits for missing labels & BirdNet
- Clean up Code to (maybe) make it a dataset package
- Error with background noise augmentation during evaluation HOT 7
- Remove augmentations from evaluation HOT 1
- Remove Train Split for eval.py
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from birdset.