[JOSS] ViewFactors error: "dateutil.parser._parser.ParserError: Unknown string format: 11-01 00:00%"

[kandersolar]

[JOSS] ViewFactors error: "dateutil.parser._parser.ParserError: Unknown string format: 11-01 00:00%"

[EwaGomez]

Dear @kanderso-nrel,
Unfortunatly I was not able to reproduce your error.
Nevertheless, I changed the two lines, you mentioned and pushed it. In both cases the year is included in the dataframe.
Does that solve your error?

[kandersolar]

Does that solve your error?

Unfortunately not -- it seems 6223780 only removed the extra % in the tracking case, but it is still there in the fixed-tilt case.

But I wonder why you don't get the same errors I do? This doesn't seem like something where the OS would matter (unfortunately I cannot test on Windows at the moment). Do you not get errors from this snippet?

In [1]: import pandas as pd

In [2]: pd.__version__
Out[2]: '1.3.4'

In [3]: pd.to_datetime('11-01 00:00%')

  File "/tmp/ipykernel_236209/955505565.py", line 1, in <module>
    pd.to_datetime('11-01 00:00%')
...
ParserError: Unknown string format: 11-01 00:00%


In [4]: pd.to_datetime('11-01 00:00')

  File "/tmp/ipykernel_236209/3743771804.py", line 1, in <module>
    pd.to_datetime('11-01 00:00')
...
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1-11-01 00:00:00

[kandersolar]

When I remove that extra % sign locally, I get another error a little later on this line:

BifacialSimu/BifacialSimu_src/BifacialSimu/Handler/BifacialSimu_radiationHandler.py

Line 602 in 6223780

mask = (df.index >= dtStart) & (df.index <= dtEnd)

Click to expand!

Exception in thread Thread-11:
Traceback (most recent call last):
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 540, in _validate_comparison_value
    self._check_compatible_with(other)
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py", line 502, in _check_compatible_with
    self._assert_tzawareness_compat(other)
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py", line 691, in _assert_tzawareness_compat
    raise TypeError(
TypeError: Cannot compare tz-naive and tz-aware datetime-like objects.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 1008, in _cmp_method
    other = self._validate_comparison_value(other)
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 543, in _validate_comparison_value
    raise InvalidComparison(other) from err
pandas.core.arrays.datetimelike.InvalidComparison: 2019-11-01 00:00:00-07:00

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/kevin/projects/BifacialSimu/BifacialSimu_src/GUI.py", line 562, in StartSimulation
    Controller.startSimulation(SimulationDict, ModuleDict, resultsPath)
  File "/home/kevin/projects/BifacialSimu/BifacialSimu_src/BifacialSimu/Controller/BifacialSimu_simulationController.py", line 91, in startSimulation
    df_reportVF, df, test = BifacialSimu_radiationHandler.ViewFactors.simulateViewFactors(simulationDict, demo, metdata,  df, resultsPath, onlyFrontscan = False)
  File "/home/kevin/projects/BifacialSimu/BifacialSimu_src/BifacialSimu/Handler/BifacialSimu_radiationHandler.py", line 602, in simulateViewFactors
    mask = (df.index >= dtStart) & (df.index <= dtEnd) 
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/indexes/extension.py", line 156, in wrapper
    return op(other)
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/ops/common.py", line 69, in new_method
    return method(self, other)
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/arraylike.py", line 52, in __ge__
    return self._cmp_method(other, operator.ge)
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 1010, in _cmp_method
    return invalid_comparison(self, other, op)
  File "/home/kevin/miniconda3/envs/bifacialsimu-review/lib/python3.8/site-packages/pandas/core/ops/invalid.py", line 34, in invalid_comparison
    raise TypeError(f"Invalid comparison between dtype={left.dtype} and {typ}")
TypeError: Invalid comparison between dtype=datetime64[ns] and datetime

Using the debugger I see that the boundary timestamps are tz-localized but the dataframe index is not:

> /home/kevin/projects/BifacialSimu/BifacialSimu_src/BifacialSimu/Handler/BifacialSimu_radiationHandler.py(604)simulateViewFactors()
    602             import pdb
    603             pdb.set_trace()
--> 604             mask = (df.index >= dtStart) & (df.index <= dtEnd)
    605             df = df.loc[mask]
    606 


IPdb [1]: dtStart
2019-11-01 00:00:00-07:00

IPdb [2]: dtEnd
2019-11-16 00:00:00-07:00

IPdb [3]: df.index
DatetimeIndex(['2019-11-01 00:00:00', '2019-11-01 01:00:00',
               '2019-11-01 02:00:00', '2019-11-01 03:00:00',
               '2019-11-01 04:00:00', '2019-11-01 05:00:00',
               '2019-11-01 06:00:00', '2019-11-01 07:00:00',
               '2019-11-01 08:00:00', '2019-11-01 09:00:00',
               ...
               '2019-12-31 14:00:00', '2019-12-31 15:00:00',
               '2019-12-31 16:00:00', '2019-12-31 17:00:00',
               '2019-12-31 18:00:00', '2019-12-31 19:00:00',
               '2019-12-31 20:00:00', '2019-12-31 21:00:00',
               '2019-12-31 22:00:00', '2019-12-31 23:00:00'],
              dtype='datetime64[ns]', name='timestamp', length=1464, freq=None)

I am still following the steps I put in the first post:

1. Start GUI
2. Hit "set default!"
3. In "Simulation Control", select "front and back simulation with Viewfactors"
4. Hit "start simulation"

I think it is important that we understand why I am getting these errors but others are not. I will try to reproduce this on Windows but it may take be a few days.

[EwaGomez]

@kanderso-nrel
Unfortunately I am still not able to reproduce the error. I even tried a fresh installation of BifacialSimu on OS and it worked.
The only error I get on OS is "name 'rootPath' is not defined", but it works, when you run the code a second time. Currently, around 20 students are working with BifacialSimu and we didn't see this error before. Would it be possible for you to do a fresh new installation?

[EwaGomez]

@kanderso-nrel
Unfortunately I am still not able to reproduce the error. I even tried a fresh installation of BifacialSimu on OS and it worked.
The only error I get on OS is "name 'rootPath' is not defined", but it works, when you run the code a second time. Currently, around 20 students are working with BifacialSimu and we didn't see this error before. Would it be possible for you to do a fresh new installation?

[kandersolar]

Hi @EwaGomez, I was finally able to test things on a Windows computer (my previous tests were all on Linux). Thank you for your patience during my delays.

Indeed I cannot reproduce the error on Windows either. The difference is in that trailing % symbol again -- it gives different behavior on Linux than it does on Windows. Here is a simple example:

>>> pd.to_datetime(['2019-11-01 00:00:00-07:00']).strftime('%Y-%m-%d %H:%M%')
Index(['2019-11-01 00:00:00-07:00'], dtype='object')  # Windows
Index(['2019-11-01 00:00%'], dtype='object')  # Linux

So when I removed that % symbol on Linux it got rid of the timezone, leading to the errors in my last comment. But on Windows the extra % symbol ends up just returning the "full" timestamp string for some reason. I'm not seeing this behavior documented anywhere in the strftime format string specification: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes

I guess the difference is explained by this comment on that page:

The full set of format codes supported varies across platforms, because Python calls the platform C library’s strftime() function, and platform variations are common. To see the full set of format codes supported on your platform, consult the strftime(3) documentation. There are also differences between platforms in handling of unsupported format specifiers.

Between this issue and #18, maybe the easiest way forward is to say in the README that Linux is not supported? I can finish the JOSS review only using Windows if so. Let me know what you think.

[EwaGomez]

Hi @kanderso-nrel,
I agree! I just integrated a part in the README, that Linux is not supported at the moment. But I will leave the issue open, so I can work on it in the next weeks after the JOSS review. I see the importance of BifacialSimu beeing available on Linus, Mac and Windows.
Do you agree with that procedure?

[kandersolar]

Sounds good 👍

[EwaGomez]

We fiexed this issue in the newest version #30 . BifacialSimu is now compatible with Linux.