idle job killer thinks job flows running Hive are idle

[yelp]

idle job killer thinks job flows running Hive are idle

[coyotemarin]

So, the step that runs hive reports that it's "done":

  "Steps": [
    {
      "ExecutionStatusDetail": {
        "StartDateTime": 1289265498.0,
        "EndDateTime": 1289265523.0,
        "LastStateChangeReason": null,
        "CreationDateTime": 1289265353.0,
        "State": "COMPLETED"
      },
      "StepConfig": {
        "Name": "Setup Hive",
        "HadoopJarStep": {
          "Jar": "s3:\/\/us-east-1.elasticmapreduce\/libs\/script-runner\/script-runner.jar",
          "Args": [
            "s3:\/\/us-east-1.elasticmapreduce\/libs\/hive\/0.4\/install-hive"
          ],
          "Properties": [],
          "MainClass": null
        },
        "ActionOnFailure": "TERMINATE_JOB_FLOW"
      }
    }
  ],

Probably the easiest fix is to leave alone job flows where the last step (and in most cases, the only step) isn't running the Hadoop streaming jar.

Not sure what happens if you try to run streaming jobs in a job flow where you're already running Hive. Going to try it!

[coyotemarin]

It looks like the streaming job works, until you run a Hive query, and that kills the streaming job. So they don't mix in any practical way.

So it looks like we should leave alone a job that has any non-streaming jar steps (since mrjob will only run streaming steps)

[coyotemarin]

Huh, so here's what a Hadoop Debugging step looks like (which is compatible with hadoop streaming):

    {
      "ExecutionStatusDetail": {
        "StartDateTime": null,
        "EndDateTime": null,
        "LastStateChangeReason": "Job terminated",
        "CreationDateTime": 1285049200.0,
        "State": "CANCELLED"
      },
      "StepConfig": {
        "Name": "Setup Hadoop Debugging",
        "HadoopJarStep": {
          "Jar": "s3n:\/\/us-east-1.elasticmapreduce\/libs\/script-runner\/script-runner.jar",
          "Args": [
            "s3n:\/\/us-east-1.elasticmapreduce\/libs\/state-pusher\/0.1\/fetch"
          ],
          "Properties": [],
          "MainClass": null
        },
        "ActionOnFailure": "TERMINATE_JOB_FLOW"
      }
    },

I think I'll only leave alone job flows that have no hadoop streaming steps, since they're clearly not the purview of mrjob.

[coyotemarin]

Fixed in dev, and heavily tested!

[coyotemarin]

Fixed in v0.2.0.