kanaricc/iron_tank

(WIP)A fast and reliable judge written in Rust.

Iron Tank

WIP. The usage and data format may change dramatically as development progresses.

中文

Iron Tank is a fast and reliable judge written in Rust.

Judge is a program used commonly in ICPC, CCPC, OI and other programming contests to check if the competitors' solution code is correct or not by supervising the whole running process, for example, time and memory usage, and comparing the final result it outputs with the predefined answer.

Install

Iron Tank relies on the features that Linux supports at present. I have only tested it on Linux. Not sure will it work on OS X, FreeBSD or not.

Does not support Windows.

Install From Pre-builded Version

1. Download the newest version from the release page, iron_tank and iron_cell.
2. Put the executable file you just downloaded in one same folder.
3. Done.

Install From Source

1. Install Rust toolchain using Rustup.
2. Clone the repo.
3. Run command cargo build --release in the root directory.
4. Look for path target/release to find iron_tank and iron_cell.
5. Jump to the step 2 in Install From Pre-builded Version.

Usage

You can use tank_cli help.

Server

By starting Iron Tank in server mode, you get a judge backend. (WIP)

Normal

• Input and answer are read from file.
• Program IO uses standard io stream.
• Program should only use limited memory and exit in limited time, or it will be killed.
• Program is granted ONLY basic permissions such as allocating memory, reading standard stream and some system-related operations.

This is the common and useful mode for most situation.

Command pattern:

$ tank_cli normal <src> -i <input> -a <answer> -t <time-limit> -m <memory-limit> -c <compare-mode>

• <src>, the path of source.
• <input>, the input file for program.
• <answer>, the answer file.
• <time-limit>, time limit(MS) for program.
• <memory-limit>, memory limit(MB) for program.
• <compare-mode>, define the approach to compare the output and answer.

Just for example. The command below will start a "cell", in which program can only use about 256 MB memory at most, run no longer than about 1 second, only read/write to standard io without permissions such as opening file, conencting network and forking new process. The output by ./user_code is compared with content of 1.ans line by line.

$ tank_cli normal ./user_code -i 1.in -a 1.ans -t 1 -m 256 -c line

Judge Result

(WIP)

10 kinds of result are provided for now.

pub enum JudgeStatus {
    Uncertain,
    Accept,
    WrongAnswer,
    PresentationError,
    MemoryLimitExceeded,
    TimeLimitExceeded,
    InteractionTimeLimitExceeded,
    ComplierError,
    ComplierLimitExceeded,
    RuntimeError,
}

Comparation Mode

• full. Output must be the absolutely same with Answer, including blank characters.
• line. Output and Answer are trimmed firstly to remove the blank chars at the beginning and ending position of them. Then comparison are held on each line of them, ignoring blank chars at the ending position. (Output are readed from left to right.)
• value. Output and Answer are compared without any blank chars.

Status PE may appear when comparison mode is set to the first or second one.

Speical (Speical Judge)

• Input is readed from file.
• A user-defined checker is used to check if program gives correct output.

Command pattern:

$ tank_cli special <checker> <src> -i <input> -t <time-limit> -m <memory-limit>

• <src>, the path of source.
• <input>, the input file for program.
• <checker>, the path of checker
• <time-limit>, time limit(MS) for program.
• <memory-limit>, memory limit(MB) for program.

Checker

A checker will receive input, output of the program, and give the result of comparison.

For example:

#include <iostream>
#include <fstream>
using namespace std;

int main(int argc,char* argv[]){
    ifstream input(argv[1]);
    ifstream output(argv[2]);

    string s1, s2;
    input >> s1;
    output >> s2;

    if(s1 == s2) {
        cout << "same" << endl << "" << endl;
    } else {
        cout << "different" << endl << "" << endl;
    }

    return 0;
}

Interactive

• Input is dynamically generated by a program called interactor on-the-fly.
• User program uses standard IO.
• Output is checked by interactor on-the-fly.

In short, interactor and user program are directly connected, they can interact in real-time.

Command pattern:

$ tank_cli special <interactor> <src> -i <input> -t <time-limit> -m <memory-limit>

Interactor

An interactor is a program, output and input of which will be connected to the input and output of user program.

For example,

#include <iostream>
using namespace std;

int main(){
    bool ok=true;
    for(int i=0;i<10;i++){
        cout<<i<<endl;
        int x;cin>>x;
        fout<<x<<endl;
        if(x!=(1<<i))ok=false;
    }

    if(ok){
        cerr<<"same"<<endl;
    }else{
        cerr<<"different"<<endl;
    }

    return 0;
}

Prefab

By using a YAML configuration file, you can edit a problem beforehand and quickly use that configuration to create tasks.

Command pattern:

$ tank_cli prefab <config> <src>

• <config>: config file.
• <src>: the path of source.

To make a prefab,

1. Create a folder, named by the title of problem (for example, A) or whatever you want.
2. Touch a new file in it named problem.yaml.

Content of a problem.yaml likes

name: A                       # the title of problem
limitConfig:
  time:imit: 1000             # time limit (ms)
  memory:imit: 256            # and memory limit (MB)
judgeMode:                    # judge mode
  Normal:                     # here we use normal mode
    comparisionMode: Line     # compare output using `Line` mode
inputLint:                    # add lint and check your data to avoid mistakes
  linters:
    - unexpected-bytes
    - consecutive-empty-lines
    - start-with-empty-line
    - extra-spaces-after-lines
    - consecutive-spaces
  customLints:
    - |-
      data.rint();
      data.eeof();
      0
answerLint:
  linters:
    - unexpected-bytes
    - consecutive-empty-lines
    - start-with-empty-line
    - extra-spaces-after-lines
    - consecutive-spaces
  customLints:
    - |-
      data.rint();
      data.eeof();
      0
cases:                        # you can add many cases for one problem
  - inputfilePath: 1.in       # the path is relative to this config file
    answerfilePath: 1.ans
  - inputfilePath: 2.in
    answerfilePath: 2.ans

Then, prepare and put your data in correct place according to this config file. I suggest you put them in the same folder.

judgeMode

judgeMode:
  Normal:
    comparisionMode: Full/Line/Value

judgeMode:
  Special:
    checker: path

judgeMode:
  Interactive:
    interactor: path
    has_input: true/false. input defined in test cases will be provided to *interactor* as argument.

In interactive mode, you also need to set test cases' inputs and outputs, even if the interactor does not care about them. If you set has_input as false, however, both inputs and outputs of the test cases are just placeholders which imply the number of cases.

Lint

By using a YAML configuration file, you get the benefit that the data can be checked by tank.

Command pattern:

$ tank_cli lint <config>

• <config>: config file.

Language Support

Compiler	command
g++	g++ <input> -o <output> -O2
Python3	python3 <src>

iron_tank's Issues

Run checker in cell

• place checkers into cell.

Add custom lint.

Rename the project

Since it only took 10 seconds to come up with the current name, I think it can be changed in due course.

Enhance the comparision methods

• add a custom comparision mode with options.
• refactor existing modes by using it.
• implement an universal special checker lib. (or using an existing one)

I'm not sure it's appropriate or not to replace normal mode with special mode and predefined checker. This will be discussed in future.