codewars/codewars-runner-cli

Old CodeRunner project. See https://github.com/codewars/runner instead.

codewars-runner-cli's Issues

Update documentation

At the moment, it's not really clear what has to be provided to add support for a new language or to upgrade to a newer version of a compiler/runtime. Apparently one needs a docker image (or change the full image?), and a test framework. But this doesn't include building/running strategies for the actual tests, which are essential.

I'm not sure whether those informations are considered internal by you or whether they're simply missing in this repository, but they're required to create support for something new.

Separate docker files

The single Docker file has gotten out of control. At this point we end up having to rebuild most if not all of the thing too often. This causes space issues as well as upload issues since all of the old stuff can't be re-used. I want to try to group common sets so that we don't have 20 images to manage. Here is what I am thinking:

Base Image:

• ENV config
• setup codewarrior user
• common services (redis, mongo, sqlite, elasticsearch, etc)
• any packages commonly used by installers (python libs, node, etc).

Node Image

• node
• coffeescript
• typescript
• npm packages

Ruby Image

• ruby
• gems
• run ruby tests

Python Image

• python 2
• python 3
• language packages
• run python tests

.NET Image

• c#
• f#
• dlls
• c#/f# language tests

JVM Image

• java
• groovy
• clojure
• scala
• language packages
• JVM runner
• run jvm language tests

Systems image

• c
• c++
• asm/arm
• language packages
• ?
• language specific tests

Lispy Image

• Lisp
• Racket
• OCaml
• Haskell
• language packages
• tests

Alt Languages Image (less commonly used)

• R
• Rust
• Erlang
• Elixir
• Go
• Julia
• Lua
• language packages
• tests

We would deploy the runner to all of the non-base images just as it is today. The only difference is that the server will need to pick the image based off of the language provided. We will also need to update the build, pull and push utilities to do all of the image magic for us.

Haskell build issue on "docker_files" branch

Run node build haskell to see error. If your first run ever on the new branch then you should run node build base haskell.

Issue is related to codewarrior user installing cabal packages.

Upgrade Erlang to R17+

the erlang vm used is R16B03. Please upgrade it to R17+, since the introduction of "Named Funs" makes it far easier to write recursive functions (Maps were also added)

http://joearms.github.io/2014/02/01/big-changes-to-erlang.html.

F# support

I'd love to help get F# katas running in codewars. AFAICT the .NET runner already supports F#, so I was wondering what else needs to be done to get to the point where we can start porting katas over to F#?

Scala support

Ruby file permissions

require 'yaml/store'
store = YAML::Store.new 'candy.yml'

is failing due to file access error.

Cheating by replacing the global object

After reading the source code here: https://github.com/entrefuse/cdr/blob/master/frameworks/javascript/cw-2.js

I have found a way of cheating that works for almost every javascript kata.
The key to it is hacking the global object, and the trick to it is that although the crucial keywords of global can't be modified because of defineProperty, but global can be REPLACED, and then console.log('<PASSED::>All tests passed')

Javascript Runner Broken

Not sure how this happened, I hope it wasn't one of my commits...

But when I run:

docker build -t codewars/cli-runner . ; docker run -i --entrypoint mocha codewars/cli-runner test/*

I get:

...
  2 failing

  1) javascript runner .run cw-2 error handling should gracefully handle reference errors:
     Uncaught AssertionError: expected '<DESCRIBE::>test\n<IT::>test1\n<ERROR::>ReferenceError: b is not defined<:LF:>        at Test.it<:LF:>        at Test.describe<:LF:>            <:LF:>            \n<COMPLETEDIN::>2\n<IT::>test2\n<PASSED::>Test Passed\n<COMPLETEDIN::>1\n<COMPLETEDIN::>9\n' to include '\\n'
      at /codewars/test/runners/javascript_spec.js:77:50
      at /codewars/lib/shovel.js:16:9
      at ChildProcess.<anonymous> (/codewars/lib/shovel.js:106:9)
      at ChildProcess.emit (events.js:98:17)
      at Process.ChildProcess._handle.onexit (child_process.js:809:12)

  2) javascript runner .run cw-2 error handling should gracefully top level handle reference errors:
     Uncaught AssertionError: expected '<ERROR::>ReferenceError: b is not defined<:LF:>        <:LF:>            \n' to include '\\n'
      at /codewars/test/runners/javascript_spec.js:91:50
      at /codewars/lib/shovel.js:16:9
      at ChildProcess.<anonymous> (/codewars/lib/shovel.js:106:9)
      at ChildProcess.emit (events.js:98:17)
      at Process.ChildProcess._handle.onexit (child_process.js:809:12)

Swift Support

Rust support

First stable version was released today.
I would love to help, btw.

Verilog

c# support

Check out this repo for a version of the CW test framework ported over to C#. This is an older version that writes HTML output directly, so that code needs to be updated.

Java 8?

Want to use Oracle's Java 8, rather than OpenJDK 6?

Coq, HOL-Light, Isabelle/HOL, or Isabelle/ZF

I've always dreamt that mathematics education, in particular formal proof, could largely be automated.

Codewars could be a platform for this... it could be crazy awesome.

Julia Fast Fail

Julia should fail fast

Python3 support

go support

Load runner files in run.js

Most languages don't actually support running with solutionfile.

IDL

The preferred data processing language for the working astronomer. It's okay if you haven't heard of it. It's proprietary, but the good news is that there's a free dialect.

Erlang Test Framework

This has not been written, so Erlang can't be said to be done...

Rspec support

JSON Intercommunication or Dependency Support

We have some other issues open regarding restructuring the containers, and I know that this is being planned for.

The idea of having small services that intercommunicate via JSON is cool. I'd like to get started thinking about how to support this.

Notes:

• One difficulty with this problem is: while it is straightforward to wrap a function in a dynamically typed language in something that calls it with arguments read from a JSON, this gets to be more challenging and ambiguous for typed languages and systems languages.
• What happens if two services call one another, and get into a loop? How will this be prevented?
• There's a lot of overhead waiting for a service. Something similar would be if we could let kata/kumite depend on other kata/kumite as libraries. In the case of GCC/LLVM languages, this would be a matter of linking object code together after compilation. In the case of python, this would be a matter of writing out modules into directory trees in a sane/predictable way. And so on for other languages.
  • If we ever decide to support Isabelle or Coq, then this would be a natural mechanism for stacking lemmas and theorems together to prove complex facts.

Python2 support

Hide Python's `reload` somehow

So a kata writer tried to block itertools.permutations for a little premutations kata in python.

This is really easy to sidestep:

def permutations(*args): 
  return (x for x in reload(__import__('itertools')).permutations(*args))

We need to figure out how to hide reload somehow...

what TDD test framework does codewars use for javascript?

Java is outputing <:LF:> tags, possibly HTML escaped

Within the Codewars UI, <:LF:> tags are being shown, which probably means they are being escaped before they are able to be translated back into new lines.

See http://www.codewars.com/kumite/53ffdfd2153f95440b0002d6?sel=53ffdfd2153f95440b0002d6

C++ support

Fortran

Don't be Root

It's really bad that the runner runs as root. It should probably just run as a set user, who doesn't have permission to do much at all other than write some stuff to /tmp, execute some executables and read some files.

Drinks?

I'm in LA for the week, wanna go get drinks?

(I recognize this is the wrong way to ask, sorry...)

ASM

I have some contacts in industry and aerospace that have expressed that they would like to see this. In particular, people want

• x86 ASM (with MMX?)
• ARM ASM
• PTX

I'm not sure the vagrant box supports MMX extensions...

Also, we need to figure out a test framework for this.

C Threads

Daemons / Slow Test Runners

When I was writing the clojure runner, I found its performance was pitiful. I expect scala to be even worse. It would be pretty awful, but not surprising, if hello world in scala took more than 5 seconds.

One solution to the performance problem would be to switch over to a queue, like RabbitMQ Reddis, or SMQ, and have persistent JVMs process jobs off the queue. I recognize this is painful to implement and test, but otherwise you are looking at 1 to 2 seconds of load time you want to do anything with the JVM.

Docker Build is failing

Step 51 : RUN echo"Package: racket-docPin: origin ''Pin-Priority: -1" >> /etc/apt/preferences
 ---> Running in 6362be0ef402
/bin/sh: 1: echoPackage: racket-docPin: origin ''Pin-Priority: -1: not found
2014/08/18 02:34:07 The command [/bin/sh -c echo"Package: racket-docPin: origin ''Pin-Priority: -1" >> /etc/apt/preferences] returned a non-zero code: 127

Tests Are Broken In Docker Image

When I run:

docker build -t codewars/cli-runner . ; docker run -i --entrypoint mocha codewars/cli-runner test/*

I get:

  1) go runner .run should handle basic code evaluation:
     Uncaught Error: spawn ENOENT
      at errnoException (child_process.js:1000:11)
      at Process.ChildProcess._handle.onexit (child_process.js:791:34)

  2) java runner junit should handle basic junit tests:

      Uncaught AssertionError: expected '' to equal '42\n'
      + expected - actual

      +"42\n"
      -""

      at /codewars/test/runners/go_spec.js:15:42
      at /codewars/lib/shovel.js:14:9
      at exit (/codewars/lib/shovel.js:76:9)
      at null._onTimeout (/codewars/lib/shovel.js:135:13)
      at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)

  3) java runner junit should handle junit tests failing:

      Uncaught AssertionError: expected '<DESCRIBE::>myTestFunction(TestFixture)\n<FAILED::>Failed Message expected:<5> but was:<3>\n' to equal '<DESCRIBE::>myTestFunction(TestFixture)\ntest out\n<PASSED::>Test Passed\n'
      + expected - actual

      +"<DESCRIBE::>myTestFunction(TestFixture)\ntest out\n<PASSED::>Test Passed\n"
      -"<DESCRIBE::>myTestFunction(TestFixture)\n<FAILED::>Failed Message expected:<5> but was:<3>\n"

      at /codewars/test/runners/java_spec.js:42:43
      at /codewars/lib/shovel.js:14:9
      at ChildProcess.<anonymous> (/codewars/lib/shovel.js:124:9)
      at ChildProcess.emit (events.js:98:17)
      at Process.ChildProcess._handle.onexit (child_process.js:809:12)

  4) javascript runner .run cw-2 error handling should gracefully handle custom errors:
     Error: timeout of 2000ms exceeded
      at done (/usr/local/lib/node_modules/mocha/lib/runnable.js:202:67)
      at /usr/local/lib/node_modules/mocha/lib/runnable.js:223:9
      at /codewars/test/runners/java_spec.js:68:17
      at /codewars/lib/shovel.js:14:9
      at ChildProcess.<anonymous> (/codewars/lib/shovel.js:124:9)
      at ChildProcess.emit (events.js:98:17)
      at Process.ChildProcess._handle.onexit (child_process.js:809:12)

  5) javascript runner .run cw-2 error handling should gracefully handle reference errors:
     Uncaught AssertionError: expected '<DESCRIBE::>test\n<IT::>test1\n<ERROR::>ReferenceError: b is not defined<:LF:>        at Test.it<:LF:>        at Test.describe<:LF:>            <:LF:>            \n<COMPLETEDIN::>1\n<IT::>test2\n<PASSED::>Test Passed\n<COMPLETEDIN::>0\n<COMPLETEDIN::>15\n' to include '\\n'
      at /codewars/test/runners/javascript_spec.js:85:50
      at /codewars/lib/shovel.js:14:9
      at ChildProcess.<anonymous> (/codewars/lib/shovel.js:124:9)
      at ChildProcess.emit (events.js:98:17)
      at Process.ChildProcess._handle.onexit (child_process.js:809:12)

  6) typescript runner .run should handle basic code evaluation:

      Uncaught AssertionError: expected '' to equal '42\n'
      + expected - actual

      +"42\n"
      -""

      at /codewars/test/runners/typescript_spec.js:9:42
      at /codewars/lib/shovel.js:14:9
      at ChildProcess.<anonymous> (/codewars/lib/shovel.js:124:9)
      at ChildProcess.emit (events.js:98:17)
      at Process.ChildProcess._handle.onexit (child_process.js:809:12)

Replace sync code with async code.

Much of the code has been implemented in their sync forms. The exec-sync package was recently added to the project as well. This package is not meant to be used in production code.

All code should be asynchronous to ensure that the server doesn't lock up when dealing with additional requests.

Elixir

No great reason not to support this if we are supporting Erlang

Full Embedded Environment for ARM Assembly

A little bird told me that the coolest thing to learn in ARM assembly, a black art that is almost impossible to learn online, is interrupt request handling.

Here are a couple of blog posts related on how to do this QEMU:

http://balau82.wordpress.com/2010/02/28/hello-world-for-bare-metal-arm-using-qemu/
http://balau82.wordpress.com/2012/04/15/arm926-interrupts-in-qemu/

VHDL

Expose the Disassembler in Node.JS

Right now we don't have disassembly enabled for node.js

To illustrate:

http://www.codewars.com/kumite/53f4d3a067dcf4c2a500197b?sel=53f4d3a067dcf4c2a500197b

This would be nice for arguing over micro-optimization...

Find a way to enforce disc limits on a per container basis

Batteries Included

For a lot of languages, there’s not much built into the core, with the exception of Python and maybe Racket.

Here's a list of core libraries/frameworks I can think of for various languages:

• Python
  • pandas
  • numpy
  • scipy
  • django
  • jinja2
  • requests
  • sh
  • matplotlib
  • twisted
  • Niche Stuff
    • scikit-learn
    • pymc
    • PIL
    • OpenCV
    • NLTK
    • PyQuery
• Java
  • Spring
  • h2
  • Apache Commons
  • Tomcat
  • Google Guava
• Javascript
  • JQuery
  • D3.js
  • React
  • 〈Your favorite MVC Framework〉(mine's Angular)
• Clojure
  • ^^^The JAVA Stuff^^^
  • java.jdbc
  • core.async
  • core.logic
  • core.matrix
  • core.match
  • core.unify
  • core.cache
  • prismatic/schema
  • prismatic/plumbing
  • compojure
  • hiccup
  • enlive

More later...

Fix <:LF:> for Line Feeds in Haskell, Clojure, Julia

I could do this, however:

1. I'm not 100% sure what I'm supposed to do exactly, since I don't know how the downstream code that parses the test framework output works
2. I'm siloed as the sole developer for these things. What's worse, I'm not an employee... so I really think someone else should touch this code (I'll review it)

CI For Github

Can we hook into Travis CI? Would be nice...

Erlang Needs a Killswitch Thread

From time to time when I am working on testing erlang, my test crashes and then erlang hangs and can't start another VM.

Erlang needs a thread that always starts at boot and kills the VM after some timeout.

Java support

Haskell support

Write All Temporary Files to Directories in /tmp, Delete them When Done

I'm using node-temp for my haskell runner, since I noticed that the runner was leaking files. Now all the files are deleted right after they are created and run, and they all live in unique directories so they can never collide.

I am of the opinion that all of the runners should be done this way.

Escape character's for user's console.log

If the user writes console.log('<PASSED::>All tests passed') this shouldn't be a new passed test.
By escaping < and >, this can be avoided.