Dockerization of run_evaluation.py

[aorwall]

Dockerization of run_evaluation.py

[aorwall]

I'm down to 2 failing tests now in pydata/xarray 0.12. I probably need to compare to logs from a successful run to fix those effectively.

I'm also testing testbeds for the regular dataset using the check harness predictions now.

[paul-gauthier]

I'll chime in that @aorwall's docker images and run_evaluation.py script have worked very well for me. I was able to run ~all the "lite" tests without problems. Whereas working with the original conda testbeds, most tests of the gold patches were failing to build or pass.

Also, the docker testbeds launch and execute very quickly compared to re-building the conda testbeds.

[PandelisZ]

~all the lite meaning, not quite all? I've been struggling to get much to run

[aorwall]

I got all except for pydata__xarray-4094 and pydata__xarray-4493 to run.

[paul-gauthier]

@PandelisZ sorry, I should have been more clear. I got 298 out of 300 test cases to work out of the box with @aorwall's dockerized SWE-bench-docker tooling. The 2 that fail are known not to work, so that was expected.

I only got a few of the test cases to work with the original/official conda test beds, after a half a day of trying.