ValueError exception running sub_balancing.geography=TRACT model: Length of values (2) does not match length of index (37)

[werdnabae]

ValueError exception running sub_balancing.geography=TRACT model: Length of values (2) does not match length of index (37)

[werdnabae]

I'm getting a similar error when I try to run without multiprocessing.

INFO - restoring 24 zero weight rows INFO - do_simul_integerizing succeeded for PUMA_101 status OPTIMAL. INFO - balancing 1/1 seed 103, PUMA 103 INFO - Time to execute all models until this error : 1918.814 seconds (32.0 minutes) ERROR - activitysim run encountered an unrecoverable error Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/activitysim/cli/run.py", line 284, in run pipeline.run(models=config.setting("models"), resume_after=resume_after) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/activitysim/core/pipeline.py", line 650, in run run_model(model) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/activitysim/core/pipeline.py", line 502, in run_model orca.run([step_name]) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/orca/orca.py", line 2177, in run step() File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/orca/orca.py", line 973, in __call__ return self._func(**kwargs) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/populationsim/steps/sub_balancing.py", line 277, in sub_balancing zone_weights_df = balance_and_integerize( File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/populationsim/steps/sub_balancing.py", line 160, in balance_and_integerize balanced_sub_zone_weights = balance( File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/populationsim/steps/sub_balancing.py", line 78, in balance controls[zone_name] = sub_controls_df.loc[zone].values File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/frame.py", line 3950, in __setitem__ self._set_item(key, value) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/frame.py", line 4143, in _set_item value = self._sanitize_column(value) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/frame.py", line 4870, in _sanitize_column com.require_length_match(value, self.index) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/common.py", line 576, in require_length_match raise ValueError( ValueError: Length of values (2) does not match length of index (37) Traceback (most recent call last): File "run_populationsim.py", line 39, in <module> sys.exit(run(args)) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/activitysim/cli/run.py", line 284, in run pipeline.run(models=config.setting("models"), resume_after=resume_after) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/activitysim/core/pipeline.py", line 650, in run run_model(model) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/activitysim/core/pipeline.py", line 502, in run_model orca.run([step_name]) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/orca/orca.py", line 2177, in run step() File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/orca/orca.py", line 973, in __call__ return self._func(**kwargs) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/populationsim/steps/sub_balancing.py", line 277, in sub_balancing zone_weights_df = balance_and_integerize( File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/populationsim/steps/sub_balancing.py", line 160, in balance_and_integerize balanced_sub_zone_weights = balance( File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/populationsim/steps/sub_balancing.py", line 78, in balance controls[zone_name] = sub_controls_df.loc[zone].values File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/frame.py", line 3950, in __setitem__ self._set_item(key, value) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/frame.py", line 4143, in _set_item value = self._sanitize_column(value) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/frame.py", line 4870, in _sanitize_column com.require_length_match(value, self.index) File "/home/ubuntu/anaconda3/envs/popsim/lib/python3.8/site-packages/pandas/core/common.py", line 576, in require_length_match raise ValueError( ValueError: Length of values (2) does not match length of index (37)

[werdnabae]

I wasn't able to find what the problem was with the error message above. However, I successfully generated a synthetic population for California by going county by county, instead of trying to synthesize it all at once. I think that problems arise when the input data is too large.

[bettinardi]

Interesting - I wonder how many controls you had specified, I can imagine running out of RAM or similar with a huge state like California and a large number of controls. If there weren't a lot of controls it would surprise me to learn that the software still blows up (fails).

[werdnabae]

I had 37 total controls. I could not figure out what the issue was, so I decided to synthesize county by county instead of trying to synthesize for the entire state of California at once.

[bettinardi]

Thanks again for sharing this issue. I wish we had a more definitive answer for you and any readers, but I'm glad that you a running solution.

[binnympaul]

Hi @werdnabae, from your error message, it doesn't seem like a memory issue but perhaps some sort of inconsistency in the input data. Are all the geographic IDs unique? Depending on the type of Census geographies you are using, the IDs may repeat if not using the long-form Census IDs.

[bettinardi]

Good thought Binny. It made me remember that we had an issue with our statewide population for 2020. We used 2020 block controls with a 2017-2021 ACS PUMS seed. The 2017-21 PUMS data still use 2010 PUMAs, while the 2020 block data maps to 2020 PUMAs. That might not be the issue here - but a good reminder from Binny to check and ensure that everything is crosswalked correctly with unique identifiers.

[bettinardi]

I believe this issue is resolved