Help Required on semantic segmentation example of Potsdam

[azavea]

Help Required on semantic segmentation example of Potsdam

[lewfish]

From the log you provided it looks like the output is going to Users/gunjanthakuria/Desktop/projects/data_project/Satellite/postdam_contest/raster-vision-examples-master/output/

[lewfish]

That appears to be a relative path (because it doesn't start with a forward slash) so that directory should be located in the directory you ran Raster Vision from.

[gunjanthakuria]

Hey Thanks,that worked on the test run. However when I tried running the program using AWS batch having set up the required AMIs, I again face the problem of no outputs and no error message also
My input and output folder are in S3:
DATA_URI=s3://gunjanpotsdam/data
ROOT_URI= s3://postdamout/output
I ran the batch command : rastervision run aws_batch -e potsdam.semantic_segmentation -a root_uri ${ROOT_URI} -a data_uri ${DATA_URI}

However other than the command-config.json files I don't see any outputs in the S3 bucket "s3://postdamout/output" nor is there any error . Am I making any mistake here ? Please find below the logs:

root@4f4e529fb316:# ROOT_URI=s3://postdamout/output
root@4f4e529fb316:# DATA_URI=s3://gunjanpotsdam/data
root@4f4e529fb316:# rastervision run aws_batch -e potsdam.semantic_segmentation -a root_uri ${ROOT_URI} -a data_uri ${DATA_URI}
Ensuring input files exist [####################################] 100%
Checking for existing output [####################################] 100%
Saving command configuration to s3://postdamout/output/chip/potsdam-seg/command-config.json...
CHIP command submitted job with jobName=CHIP_potsdamseg_fc22008b and jobId=53135eb2-2577-4b37-b65e-c00ed61343f7
Saving command configuration to s3://postdamout/output/train/potsdam-seg/command-config.json...
TRAIN command submitted job with jobName=TRAIN_potsdamseg_c6421281 and jobId=18032ec5-eadb-49e2-ad92-8f06b5473d39
Saving command configuration to s3://postdamout/output/bundle/potsdam-seg/command-config.json...
BUNDLE command submitted job with jobName=BUNDLE_potsdamseg_4686b1a7 and jobId=0bb9a468-ed03-4733-bacd-d4a5a9422d11
Saving command configuration to s3://postdamout/output/predict/potsdam-seg/command-config.json...
PREDICT command submitted job with jobName=PREDICT_potsdamseg_5cfe81f9 and jobId=d6d367ff-77fb-4463-a3bb-6195e211fbb8
Saving command configuration to s3://postdamout/output/eval/potsdam-seg/command-config.json...
EVAL command submitted job with jobName=EVAL_potsdamseg_5f468c9d and jobId=cdaeb5dc-3b0d-4602-8202-d48e05e9662a
root@4f4e529fb316:#

[lewfish]

When you run a job on Batch, you can view the status of that job and any output in the Batch console. You will need to look at the output to see what the problem is.

[gunjanthakuria]

Thanks so much for your help and pointing me in the right direction. I managed to run the training and validation using AWS Batch job; it turns out the instances were not launching due to volume issues which I resolved and the jobs ran successfully and the outputs are in my aws bucket.

However I have run into another problem while using the predict_package to run predictions on other images. One of the outputs after training the model is the predict_package.zip file in the bundle folder which contains the "model" file and "bundle_config.json" file . When I run the predict function from the command line , I get an error saying it is not able to parse the JSON file. Can you please help me point me to the right direction to investigate what am I doing wrong here. Below is the log:

root@c9096928f740:~# rastervision predict https://s3.amazonaws.com/postdamout/output/bundle/potsdam-seg/predict_package.zip https://s3.ap-south-1.amazonaws.com/gunjanpotsdam/data/4_Ortho_RGBIR/top_potsdam_2_13_RGBIR.tif 2_13.tif
2019-04-04 18:12:05:rastervision.utils.files: INFO - Downloading https://s3.amazonaws.com/postdamout/output/bundle/potsdam-seg/predict_package.zip to /tmp/tmp4ms6rljf/tmpmlw_6rts/http/s3.amazonaws.com/postdamout/output/bundle/potsdam-seg/predict_package.zip
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 489, in _ConvertFieldValuePair
message_descriptor.fields))
google.protobuf.json_format.ParseError: Message type "rv.protos.CommandConfig" has no field named "splitId".
Available Fields(except extensions):

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/src/rastervision/utils/files.py", line 247, in load_json_config
return json_format.Parse(file_to_str(uri, fs=fs), message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 406, in Parse
return ParseDict(js, message, ignore_unknown_fields)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 421, in ParseDict
parser.ConvertMessage(js_dict, message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 452, in ConvertMessage
self._ConvertFieldValuePair(value, message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 554, in _ConvertFieldValuePair
raise ParseError(str(e))
google.protobuf.json_format.ParseError: Message type "rv.protos.CommandConfig" has no field named "splitId".
Available Fields(except extensions):

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/src/rastervision/main.py", line 17, in
rv.main()
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/opt/src/rastervision/cli/main.py", line 239, in predict
channel_order)
File "/opt/src/rastervision/predictor.py", line 48, in init
msg = load_json_config(bundle_config_path, CommandConfigMsg())
File "/opt/src/rastervision/utils/files.py", line 251, in load_json_config
raise ProtobufParseException(error_msg) from e
rastervision.filesystem.filesystem.ProtobufParseException: Problem parsing protobuf file /tmp/tmp4ms6rljf/tmpmlw_6rts/package/bundle_config.json. You might need to run scripts/compile

[lewfish]

The splitId was recently added to RV. So it looks like you generated a predict package with a recent version of RV that wrote a splitId value, and are running the predict command using an older version of RV that doesn't recognize that field.

[lewfish]

I just confirmed that the predict command you ran works with the master branch of RV (or -latest Docker image).

[gunjanthakuria]

Thanks; I believe now I am running the latest version of RV for both training and prediction from the latest docker image. However I am still facing problems with predict_package.The error that now I am getting is that DeeplabConfig is not able to find field names doEval. Can you please help me out here

google.protobuf.json_format.ParseError: Message type "rv.protos.BackendConfig.TFDeeplabConfig" has no field named "doEval".

Below is the complete log
root@e2601c724dc9:/opt/src# rastervision predict https://s3.amazonaws.com/postdamout/output/bundle/potsdam-seg/predict_package.zip https://s3.ap-south-1.amazonaws.com/gunjanpotsdam/data/4_Ortho_RGBIR/top_potsdam_2_13_RGBIR.tif /opt/data/2_13.tif
2019-04-05 07:37:20:rastervision.utils.files: INFO - Downloading https://s3.amazonaws.com/postdamout/output/bundle/potsdam-seg/predict_package.zip to /tmp/tmpsh8e20v3/tmp39mr3ake/http/s3.amazonaws.com/postdamout/output/bundle/potsdam-seg/predict_package.zip
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 489, in _ConvertFieldValuePair
message_descriptor.fields))
google.protobuf.json_format.ParseError: Message type "rv.protos.BackendConfig.TFDeeplabConfig" has no field named "doEval".
Available Fields(except extensions):

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 547, in _ConvertFieldValuePair
self.ConvertMessage(value, sub_message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 452, in ConvertMessage
self._ConvertFieldValuePair(value, message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 554, in _ConvertFieldValuePair
raise ParseError(str(e))
google.protobuf.json_format.ParseError: Message type "rv.protos.BackendConfig.TFDeeplabConfig" has no field named "doEval".
Available Fields(except extensions):

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 547, in _ConvertFieldValuePair
self.ConvertMessage(value, sub_message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 452, in ConvertMessage
self._ConvertFieldValuePair(value, message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 554, in _ConvertFieldValuePair
raise ParseError(str(e))
google.protobuf.json_format.ParseError: Message type "rv.protos.BackendConfig.TFDeeplabConfig" has no field named "doEval".
Available Fields(except extensions):

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 547, in _ConvertFieldValuePair
self.ConvertMessage(value, sub_message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 452, in ConvertMessage
self._ConvertFieldValuePair(value, message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 552, in _ConvertFieldValuePair
raise ParseError('Failed to parse {0} field: {1}'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse backend field: Message type "rv.protos.BackendConfig.TFDeeplabConfig" has no field named "doEval".
Available Fields(except extensions):

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/src/rastervision/utils/files.py", line 247, in load_json_config
return json_format.Parse(file_to_str(uri, fs=fs), message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 406, in Parse
return ParseDict(js, message, ignore_unknown_fields)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 421, in ParseDict
parser.ConvertMessage(js_dict, message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 452, in ConvertMessage
self._ConvertFieldValuePair(value, message)
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/json_format.py", line 554, in _ConvertFieldValuePair
raise ParseError(str(e))
google.protobuf.json_format.ParseError: Failed to parse backend field: Message type "rv.protos.BackendConfig.TFDeeplabConfig" has no field named "doEval".
Available Fields(except extensions):

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/src/rastervision/main.py", line 17, in
rv.main()
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/opt/src/rastervision/cli/main.py", line 239, in predict
channel_order)
File "/opt/src/rastervision/predictor.py", line 48, in init
msg = load_json_config(bundle_config_path, CommandConfigMsg())
File "/opt/src/rastervision/utils/files.py", line 251, in load_json_config
raise ProtobufParseException(error_msg) from e
rastervision.filesystem.filesystem.ProtobufParseException: Problem parsing protobuf file /tmp/tmpsh8e20v3/tmp39mr3ake/package/bundle_config.json. You might need to run scripts/compile
root@e2601c724dc9:/opt/src#

[lewfish]

I still think this is a versioning issue. I would verify that the Docker image is indeed the latest published one. There might be some special command you need to run to force it to pull the latest instead of using the cached one.

[gunjanthakuria]

I managed to run the predict function by building the Docker image again and running the container using docker/build and docker/run .

However when I install rastervision on my CPU machine using "pip install rastervision" it is still not able to parse the JSON file. The rastervision version that gets installed is 0.8.1. Is 0.8.1 the latest version of RV ?

[lewfish]

I'm assuming you are pip installing in order to use the QGIS plugin. The published version will not work with the master branch of RV. Using the published QGIS plugin at the moment will require using 0.8.1 of RV. BTW, we are having a new release in the next week.