twitter-archive/clockworkraven

Human-Powered Data Analysis with Mechanical Turk

clockworkraven's Issues

Add Markdown support to the Template Editor

I often want to use Markdown when writing instructions.

Thrift API needs tests

Make a separate html component, separate from the text component

The text component should html-escape all characters. People often run into HTML5 Parse errors when they have things like "< 21" in the text box.

Remove metadata functionality entirely

1. It's confusing and overly complicated.
2. It keeps on breaking.
3. The data slicing stuff looks cool, but I don't find it useful (especially not for layered slicing).

Supporting Postgres?

I've been testing Clockwork Raven locally, and I'd like to deploy it, but was hoping to be able to do so on Heroku — the main catch being that it's currently got some MySQL stuff baked in.

I was wondering if you had any plans to support PG, OR if you knew what would need to be addressed if someone were to try to add support for PG.

Thanks!
Adam

Bug: m_turk_users table uses id as a primary key...

...But the same user can work on both production and sandbox jobs (which currently means that you'd need two rows with the same id in the table, one with prod=1 and one with prod=0).

In the evaluations page, don't show the MC/FR questions header if there are no such questions.

Enable checkbox functionality

Currently, we can ask:

• Free response questions
• Multiple choice questions with radio buttons (where the judge can only select a single response)

We should enable multiple choice questions with checkboxes, so that judges can select multiple responses (or possibly none at all).

Display fields from the original data file in the All Responses table.

Unable to upload files

I've tried on several occasions to upload a csv file to clockwork, but I get an error.

NoMethodError in EvaluationsController#create

undefined method `force_encoding' for #String:0x7f26bf21e930

Rails.root: /home/jrubinstein/clockworkraven
Application Trace | Framework Trace | Full Trace

lib/input_parser.rb:26:in parse' app/controllers/evaluations_controller.rb:102:increate'

any idea what's going on?
I've got CR running in a VM running Ubuntu 12

Deal with malformed CSV/TSVs better

For example:

1. It's often fine if a few bad rows are skipped.
2. Print out offending lines.

Styling of the datatables is off

The search box and pagination buttons aren't being appropriately right-justified. (and oscillate between being centered and right-justified when the window gets resized)

Remove numeric value functionality from multiple choice questions

simplify

Add enhanced worker pages

We should have a page that allows us to see worker statistics and other information. For example:

• Number of answers by a judge on a particular evaluation, and overall
• How often the judge differs from the majority response (for questions where multiple judges are allowed to answer). If we want to get fancy, we could add an EM algorithm, but I think this is overkill because most judges are trusted.
• Judge speed. (I don't think this is that useful, because some judges simply work much faster, and Amazon doesn't seem to always correctly measure time. But sometimes it's helpful.) How much total money we've paid them (just an interesting fact).
• Add comments about workers (e.g., I interact a lot with our workers, so I know extra information about them). Give them a friendly name.
• etc.

Make it easier to support different sets of qualified workers

For example, trusted workers vs. workers who speak Japanese vs. workers who can draw, etc.

ruby-aws 1.5 vs. 1.6

When I was installing Clockwork Raven, I needed to manually install ruby-aws 1.5 by grabbing the gem from here. I had to do this because the bundle install couldn't find ruby-aws 1.5, and installing ruby-aws 1.6 gave me errors when I ran rake tasks:

uninitialized constant Amazon
clockworkraven/lib/m_turk_utils.rb:37:in `init_prod'

Everything works fine once I install 1.5, but I'm just warning you that this no longer seems to be grabbed by default via bundle/gem.

Allow multiple judges to answer a question

Currently, only a single judge is allowed to answer a question. However, Amazon supports multiple judgment functionality, where you can specify that N judges answer each question.

(After this basic functionality is done, we should also be able to do things like sort questions by the mean and standard deviation of the responses, etc.)

Stop requiring names to be unique

CR should display information about remaining funds

CR should show the funds remaining in the account, warn users who try to submit jobs that cost more than the remaining balance, and provide warnings when funds get below a certain amount.

Password protected redis

How would I go about adding a password for redis? I would like to be able to query redis remotely but do not want to leave authentication open.

Thanks
Chris

Drag-and-drop of templates is broken in 0.2.0-wip branch

(Broken by someone's commit. Haven't done a git bisect yet to figure out whose.)

Issue with Raven Secret Value

Hi,

I followed the setup steps and now I'm getting this error message:

ArgumentError (Secret should be something secure, like "e21a5b34167bf5ed9a770ee7439fd0a6". The value you provided, "secret", is shorter than the minimum length of 30 characters):

even though I've generated the secret value using the rake task as described in the README.

DataTables needs functional tests

The new controller methods that back the DataTable need functional tests.

Add an auto-collapsed component to the template builder

For example, we often have long instructions, but judges don't need to read them every time (and it slows them down and annoys them, because then they have to scroll). It'd be nice if it we could auto-collapse these.

0.2.0 branch

How stable is this branch?
Looks like lots of good stuff in there, but not much recently.
What's the plan?
Should I use that one if I want to fork and do some custom stuff?