serihiro/simple_map_reduce

Distributed MapReduce implementation written in ruby.

simple_map_reduce's Issues

Manage tasks at JobTracker

Abstract

• The latest version of this does not store tasks at JobTracker.
• Task management is necessary so that the users can find their job is finished.

Inject context object into each models

Current

• All methods to control models (save, change transition, etc.) are implemented at out of Model class
• So there are many similar code snippets like this

          response = http_client(SimpleMapReduce.job_tracker_url).put do |request|
            request.url("/workers/#{map_worker_id}")
            request.body = { event: 'ready' }.to_json
          end

• I can also introduce API Client Class. But in JobTracker class, the model control is executed not via API access but via just calling transition method of each models (provided by aasm).
• I want to control models via same interface

I'd like to do this

• Create Context classes
• Inject a context into model object
• Context classes provide same interface to control models and they concealment the implementation to control the model.

Introduce status polling from JobTracker

Abstract

• Modify worker status check style.
• In 0.1.0, JobWorker tells its status to JobTracker
• But this style is complicated and uncertain.
  • e.g. If a JobWorker is down in processing job, JobTracker keeps wrong status of the JobWorker.
• So introduce polling style from JobTracker to JobWorker.