Manual edits

[dt-woods]

Manual edits

[m-jamieson]

To be fair, the idea, whether it was correctly implemented or not, was the "Year" field would limit which years these replacement were applied to. If this is happening while generating 2020 results, I would expect things not to work. If this is happening using an existing configuration for 2016, then maybe the times when nothing is changed reflects that the source data was fixed.

[dt-woods]

Here's a potential fix:

generation.py:
    create_generation_process_df:
        entry_1:
          edit_type: "reassign"
          data_source: "yaml"
          column_to_reassign: "FuelCategory"
          incoming_value: "SOLAR"
          outgoing_value: "GAS"
          filters:
              eGRID_ID:
                - 56938
              Source:
                - "NEI"
              Year:
                - "2016"
        entry_2:
          edit_type: "reassign"
          data_source: "yaml"
          column_to_reassign: "FuelCategory"
          incoming_value: "SOLAR"
          outgoing_value: "GAS"
          filters:
              eGRID_ID:
                  - 58697
              Source:
                - "NEI"
                - "eGRID"
                - "RCRA"
                - "TRI"
              Year:
                  - "2016"
        entry_3:
          edit_type: "reassign"
          data_source: "yaml"
          column_to_reassign: "eGRID_ID"
          incoming_value: 56938
          outgoing_value: 58697
          filters:
              Source:
                - "NEI"
              Year:
                - "2016"
        entry_4:
          edit_type: "reassign"
          data_source: "yaml"
          column_to_reassign: "eGRID_ID"
          incoming_value: 56944
          outgoing_value: 55077
          filters:
              Source:
                - "NEI"
                - "eGRID"
                - "RCRA"
                - "TRI"
              Year:
                - "2016"
                - "2015"
        entry_5:
          edit_type: "reassign"
          data_source: "yaml"
          column_to_reassign: "eGRID_ID"
          incoming_value: 56938
          outgoing_value: 58697
          filters:
              Source:
                - "NEI"
              Year:
                - "2016"
        entry_6:
          edit_type: "reassign"
          data_source: "yaml"
          column_to_reassign: "eGRID_ID"
          incoming_value: 56944
          outgoing_value: 55077
          filters:
              Source:
                - "NEI"
                - "eGRID"
                - "RCRA"
                - "TRI"
              Year:
                - "2016"
                - "2015"
        entry_7:
          edit_type: "remove"
          data_source: "yaml"
          filters:
            eGRID_ID:
              - 60880
            Year:
              - "2016"

[dt-woods]

Even after the correction, only 4/7 manual edits are actually changing anything (see below where 0 rows are edited).

2023-11-29 16:30:10.930:INFO:manual_edits:check_for_edits:Edits found for generation.py.create_generation_process_df
2023-11-29 16:30:10.930:INFO:manual_edits:reassign:Re-assigning using data from yaml
2023-11-29 16:30:10.940:INFO:manual_edits:reassign:Reassigning 25 rows
2023-11-29 16:30:10.940:INFO:manual_edits:check_for_edits:Edits found for generation.py.create_generation_process_df
2023-11-29 16:30:10.940:INFO:manual_edits:reassign:Re-assigning using data from yaml
2023-11-29 16:30:10.950:INFO:manual_edits:reassign:Reassigning 0 rows
2023-11-29 16:30:10.950:INFO:manual_edits:check_for_edits:Edits found for generation.py.create_generation_process_df
2023-11-29 16:30:10.950:INFO:manual_edits:reassign:Re-assigning using data from yaml
2023-11-29 16:30:10.955:INFO:manual_edits:reassign:Reassigning 25 rows
2023-11-29 16:30:10.955:INFO:manual_edits:check_for_edits:Edits found for generation.py.create_generation_process_df
2023-11-29 16:30:10.955:INFO:manual_edits:reassign:Re-assigning using data from yaml
2023-11-29 16:30:10.961:INFO:manual_edits:reassign:Reassigning 15 rows
2023-11-29 16:30:10.961:INFO:manual_edits:check_for_edits:Edits found for generation.py.create_generation_process_df
2023-11-29 16:30:10.961:INFO:manual_edits:reassign:Re-assigning using data from yaml
2023-11-29 16:30:10.966:INFO:manual_edits:reassign:Reassigning 0 rows
2023-11-29 16:30:10.966:INFO:manual_edits:check_for_edits:Edits found for generation.py.create_generation_process_df
2023-11-29 16:30:10.966:INFO:manual_edits:reassign:Re-assigning using data from yaml
2023-11-29 16:30:10.971:INFO:manual_edits:reassign:Reassigning 0 rows
2023-11-29 16:30:10.971:INFO:manual_edits:check_for_edits:Edits found for generation.py.create_generation_process_df
2023-11-29 16:30:10.971:INFO:manual_edits:remove:Removing using data from yaml
2023-11-29 16:30:10.974:INFO:manual_edits:remove:Removing 1 rows