ragnargrootkoerkamp/bapctools

Tools for developing ICPC-style programming contest problems.

bapctools's Issues

Do not require .ans files for interactive problems

Use user+kernel time instead of walltime.

This is actually rather annoying because it can only be obtained via os.wait4(pid, 0), which doesn't handle timeouts.

The subprocess module doesn't seem to have this functionality.

Also, rusage isn't supported on Windows.

Support WA annotations in TLE/RTE solutions

Some solutions can fail in multiple ways. DomJudge allows adding a list of expected outcomes. We should support this (with the same syntax) as well.

See also Kattis/problemtools#48.

Reuse temporary builddir for c++ binaries

We don't have to rebuild all sources when only tests are updated.

Generate/run/test should work as much as possible when files are missing

Generate should work fine when there are no validators

Test result cache

To speed up evals, we could cache (hash of submission, hash of testcase) pairs and do a lookup there instead of rerunning the code.

Support for interactive problem.

We should support interactive problems.

See Kattis/problemtools#113.

Prompt to move a solution to the appropiate directory if it fails.

When an AC solution gives WA/TLE, y/n prompt to move it to that directory.
Also allow --copy to copy AC solutions to the right directory. This allows fixing them in the AC dir, while keeping track of non-working version.

Generated testcases

Spec for generators/gen.yaml

See #50.

Fix submodule problem

Currently the BAPCtools repo must be a git submodule of the contest repo in order to make the validation.h symlinks work. It would be nice if we could circumvent this requirement.

Possible solutions:

• Make each validation.h a copy. This is problematic if bugs are found.
• Remove the symlink, and add the headers directory to the paths the compiler searches for header files. Small drawback is that it won't be possible to compile the IO validators without the BAPCtools, and the validators will need to be included when exporting to DJ/kattis.

Generate should take a --timeout flag

Instead of just having an infinite default timeout.

Add memory limit option

We should limit memory usage so that we can test such RTE solutions nicely without freezing the system.

Investigate twig

It's nicer to use a well defined library than using manual string replace commands in latex template files.

Implement limits

See https://clics.ecs.baylor.edu/index.php?title=Problem_format#limits

Fall back to lower priority languages when compiler not found

Would be nice to fall back to python[3] when pypy[3] is not found.

test tool does not print solution stderr

When using tools.py test, the stderr output of the solution under test is not displayed, which is really anoying for testing

New-problem should fill config files

The new-problem subcommand should (interactively) ask for required configuration values:

• Problem name
• Problem author
• Validation type (custom/default)

Furthermore, the source and source_url fields should be extracted from a contest-level configuration file.

add support for VIVA

Hi, my favorite input format validator is VIVA which for many problems allows very concise input format validators that meet Kattis/problemtools standards. In many cases, it entirely eliminates the need for additional input validation before the problem can be approved.
In Kattis problemtools, it recognizes the presence of .viva files in the input_format_validator directory, e.g. here

Consider using it/adding support for it.

Fuzz testing from manual generator

Given a custom generator for testcases, it would be cool if we could automatically could create a lot of testcases with random parameters within some bounds, and then only keep the ones that make AC solutions disagree/many WA solutions fail.

`bt problem` should preserve symlinks

Currently the /skel/problem/generators/input_validator symlink is copied as a directory.

Use Pathlib instead of os.path

IO validators should have a compile time flag to disable `source_location` dependency

Read languages.yaml

We should support a contest-local or git-repo-local languages.yaml like problemtools' global config

`data/bad` Validators tests

Files in data/bad can be used to test the validators.

The plan is to implement the following:

A single data/bad/test.in should fail at least one input validator.

A triple data/bad/test.{in,ans,out} should pass all input validators and should fail at least one output validator, where the .ans is OK, but .out is bad. If only one of .ans and .out is given, it will be assumed that a .ans is not needed to validate a .out and the single file will be used both as .ans and .out.

This can easily be changed when this is standardised in the problemarchive format. See also Kattis/problemtools#51 and Kattis/problemtools#86

Project settings

bt problem should only copy relevant things.
Project wide settings can opt out of:

• .ctd validator
• .viva validator
• custom c++ validator

Building contest pdf should not build all problem pdfs

By default only the contest itself should be build. Adding -a/--all can build it for all problems as well.

Windows support

Would be nice if at least run works on windows.

IO validators should not depend on stdc++.h

Better warning + error handling

Add visualizer support

Proposal:

Add {input,output}_visualizer/ directory in the root of the problem directory (next to generators/).

Different types of visualizers:

• input only visualizer: called as <visualizer> testcase.in or <visualizer> < testcase.in
• input+output visualizer: called as <visualizer> testcase.in testcase.ans or <visualizer> <testcase.in <testcase.ans.

There is at most one visualizer of each type, and the entire input_visualizer is treated as a single program (similar to a validator or generator program).

The latter form of piping both the input and output into stdin is useful for Asymptote.

update docs

Docs should be updated now that there are some new features.

Also put things in order of how often they're used, instead of usage order

Support directories for validators and submissions

Add `run --samples` flag to only run samples.

Useful to not spoil results on hidden testcases

Default template files should have `.template` extension and be ignored.

Especially validator.{ctd,viva,cpp}.template

And also submissions/accepted/ragnar.cpp.template

Add option to `validate` to remove broken testcases.

Clean up timeout vs timelimit flags and logic

`generate --add-manual` to automatically add untracked manual testcases to `generators.yaml`

In addition we could add generator --move-manual <dir> to also move all inline manual cases to the specified directory (e.g. generators/manual)

Add 'contest' mode

Teams should be able to run this is contest mode:

• start a contest and write config to a file: #problems, duration of contest
• run code on samples, print output -> wrapper around test
• run code on all data, recording AC/WA and submission time
• print contest stats: solve times, penalty, problems solved
• optionally: end contest and print final stats.

Don't warn bits/stdc++.h for submissions

but keep it for validators.

Check generators are deterministic

We should run generators twice to see if they produce the same result.

Add windows support in master instead of windows branch

Using sys.platform we can do platform dependent stuff, so porting the windows changes to master should be possible.

Print warning for .in files without .ans file

stats count input files, but we always require answer files, even for problems with custom validation that don't strictly need them.

Export to kattis by zipping directory 

Rewrite the current zip --kattis command to just zip the kattis directory that is created by the kattis subcommand.

Do not try to show progressbar on small terminals

If a line wrap happens the overwriting system doesn't work anymore and will only overwrite the wrapped part, not the first line. We should avoid going longer than the terminal width if we can.

Stats command should ignore unmodified files

Unmodified code examples (input/output validator, example submission) should not be counted.

Differentiate between normal timeout and aborted timeout

It's not always immediately clear whether a program finished by itself or was killed. Would be nice if we could show this disctinction.

Mutation test accepted submissions

We could find gaps in testdata by mutation testing the accepted c++ submissions.
Any (most?) mutated solutions should fail.

Fuzz test output validators

We could improve the stability of our output validators by fuzz testing them.

Add tag to submissions to prevent export to kattis.

Verifyproblem doesn't like it when submissions are close to each other in time; maybe we can just skip some submissions or at least mark them as unimportant for timing?

`generate` should also validate

Generated input/answers should be validated.