ignore unrecognized arguments

[dhuy237]

ignore unrecognized arguments

[dhuy237]

I found a workaround solution but I hope there will be a better way of doing this.

I have to define the argument again inside the mrjob class so it can recognize it:

from datetime import datetime
import json
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("-t", "--time", help = "Output file")
# Use parse_known_args() to ignore all the arguments for mrjob class
args, unknown = parser.parse_known_args()

class Calculate(MRJob):
    def configure_args(self):
        super(Calculate, self).configure_args()
        # Define the argument again to ignore the argparse when running the MR job
        self.add_passthru_arg("-t", "--time", help="output folder for time")

start_time = datetime.now()
Calculate.run()
execute_time = (datetime.now() - start_time).total_seconds()

data = {}
data["step1"] = execute_time
with open(args.time+'/time.json', 'w') as outfile:
    json.dump(data, outfile)

And run with this command:

python cal.py data/input/input.txt --output data/output --time data/output

My question on Stack Overflow