Train baseline models for evaluation

[huu4ontocord]

Train baseline models for evaluation

[huu4ontocord]

We definately should try on eleuther eval harness. but just testing validity loss will tell us something too. regular finetuing vs. expert finetuning + merge

[mrcabbage972]

We have an issue for Eval Harness in the backlog.

[huu4ontocord]

so i am told that:
It seems they were trained on 1k batches
I think the batch size was 8 because of the number of GPUs
So that gives uhs 8k samples

So the above 1000 examples should be 8K examples.

[mrseeker]

Have you tried using the EleutherAI eval harness? It should give you a nice representation on how well the model performs, and can be used as an indicator?

[mrcabbage972]

I didn't understand the part about the 1000 training examples. Our datasets are much bigger than that!

[huu4ontocord]

didn't we just train our models on 1000 examples only? Or did i misunderstand that

[jordiclive]

@ontocord for 2. we want layer_9,10,11,12,13 ?

[mrcabbage972]

@jordiclive @ontocord
We had used layers 9-13 when we trained the experts. See: https://github.com/ontocord/MDEL/blob/main/src/mdel/train.sh#L4

[mrcabbage972]

@jordiclive Any updates on this issue?

[jordiclive]

@mrcabbage972 I trained 1. a model (all layers) on the exact splits...https://wandb.ai/ontocord/jordi_testing/runs/hu8j9ta1?workspace=user-jordanclive if you toggle the evaluation.

But I then thought we decided on automating the experiment again with more training data/less validation, maybe same amount of final testing data #47