create_assignments_from_csv xField label corruption issue

[esri]

create_assignments_from_csv xField label corruption issue

[apulverizer]

Okay let me see if I can reproduce with my own bash script and project

[jonathancgroves]

After Googling, I realise this is a UTF 8 Byte Order Mark (see here: https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8) but I have no idea how it got here.

Jonathan

[apulverizer]

It sounds like your issue is resolved, correct?

[jonathancgroves]

Thanks for the speed reply. No, it's not resolved. The BOM is clearly causing the Python script to fail with the keyerror - I don't know where the BOM is coming from though...

[apulverizer]

Okay so I can reproduce what you are seeing when I do it the way you are creating the bash variables and passing them to the script.

I made a sample script (omitted extra params for brevity) that works:

#!/bin/bash
source activate arcgis3.6
xField="xField test"
python create_assignments_from_csv.py -csvFile "../uc_data/assignments.csv" -xField "$xField" ...

Can you try this?

[jonathancgroves]

Thanks, Aaron.
I don't understand the source activate arcgis3.6 line, I don't do this...
With the other changes my complete command looked like this (redacted):
sudo python create_assignments_from_csv.py -csvFile "./inputfile.csv" -u "xxx" -p "xxx" -url "https://webgis.xxx.com/arcgis" -pid "xxx" -xField "xField test" -yField "yField" -assignmentTypeField "Type" -locationField "Location" -descriptionField "Description" -priorityField "Prior ity" -workOrderIdField "WorkOrderId" -dispatcherIdField "Dispatcher" -dueDateField "DueDate" -attachmentFileField "attachment" -logFile "./log.txt" -workerField " Worker" -timezone "US/Eastern"
The assignment created:
{'yField': '50.93831389', 'Description': 'Cut 2', 'Due Date': '08/21/2018 23:59:59', '\xef\xbb\xbfxField': '-0.033513889', 'Worker': 'ruralgrass1', 'Work Order Id': 'RUZ7', 'Priority': '3', 'Location': 'Rural Zone 7', 'Type': '2', 'Dispatcher': '1', 'attachment': ''}

So, no change, I'm afraid.

Appreciate your help.

[apulverizer]

The source activate arcgis3.6 is just activating a conda virtual environment that has the Python API for ArcGIS installed. It shouldn't affect anything.

I cannot reproduce the issue you see. I copied the commands above and substituted in my fields and works as expected.

Even in the above case when you hard-coded it you are seeing the issue with xField?

[jonathancgroves]

Oh, thanks. I don't use a virtualenv because I have a dedicated machine.
Yes - I see the same result with the value hard-coded.
But you saw it before, right?
This is going to cause us massive problems, I can't upload the hundreds of Assignments we need for this summer.
Any suggestions gratefully received.

[apulverizer]

I could reproduce the initial issue which I think was due to how you are setting the xField bash variable. But using the sample script here I cannot reproduce it anymore.

I have not seen the error outside of that.

What machine are you running it on? Are you using any other tools in the pipeline, like Jenkins or something (I have seen similar issues when integrating Jenkins/Bash/Python).

Have you tried deleting the entire xField string/parameter and retyping it (maybe there are hidden characters or something?

[jonathancgroves]

Thanks Aaron.
It's a dedicated headless Raspberry Pi Model 3 - running Debian 4.9.35. Python 2.7, running the ArcGIS Workforce standalone scripts downloaded from this GitHub repo. Nothing exotic in the pipeline. This worked perfectly last year when the Customer was using AGOL - they just moved to an Enterprise on-prem solution earlier this year and I've been working hard to jump through the hoops of making the old solution work again in the Enterprise context..
I did indeed try re-typing the whole line where the xField variable is initialised.

[jonathancgroves]

Aaron, problem solved (thanks so much).
I realised that the extra character was at the very start of my input file, where the column headings are shown. The first column heading was 'xField' and this had somehow picked up this hidden character at the start in editing on my Mac or Windows machine and was invisible on Linux. I did try retyping the xField string/parameter when you suggested it but only in my bash script - forgetting the other half of the story being the input csv.
Appreciate the time you spent on this.