Question regarding train / val / test split

[allenai]

Question regarding train / val / test split

[armancohan]

We randomly split it. If you provide these ids to the create_training_files script it will generate the corresponding instances in each set for you. details in readme:

[armancohan]

1- The only metric that is computed during training is loss and since loss is not very useful for evaluating the effectiveness of the embeddings, we do not use a final test set for this. In initial version we planned to implement additional metrics in the main training script but as the evaluation tasks grew we decided to have a separate full benchmark for evaluating the final embeddings. We encourage using a dev set for tracking loss but do final evaluation using the full SciDocs benchmark: https://github.com/allenai/scidocs

2- If for your specific use case you need to pass in a separate test set, you need to modify the jsonnet config file: Uncomment lines 7 and 143 and add a "evaluate_on_test" : true under the trainer config in line 144

[victor714]

Thanks @armancohan and how the publication ids in corpus were assigned to train/val/test? Was it done randomly or based on some other metric?