christiantremblay/pyhaystack

Pyhaystack is a module that allow python programs to connect to a haystack server project-haystack.org. Connection can be established with Niagara Platform running the nhaystack, Skyspark and Widesky. For this to work with Anaconda IPython Notebook in Windows, be sure to use "python setup.py install" using the Anaconda Command Prompt in Windows. If not, module will be installed for System path python but won't work in the environment of Anaconda IPython Notebook. You will need hszinc 1.3+ for this to work.

pyhaystack's Issues

Need support for older Python version (<= 2.6)

We've been asked about the possibility of using older Python version.

AttributeError for SkySpark

Hi,
I recently started using Pyhaystack for Skyspark connection by:

import hszinc
from pyhaystack.client.skyspark import SkysparkHaystackSession
session = SkysparkHaystackSession(uri='http://ip',
username='username',
password='password',
project='project',
grid_format=hszinc.MODE_JSON
)

As suggested on closed issue #14

I also tried what was on http://pyhaystack.readthedocs.io/en/latest/connect.html#skyspark:
However, I get the following error:

expect_format=hszinc.MODE_ZINC, multi_grid=False,

AttributeError: 'module' object has no attribute 'MODE_ZINC'

Any idea why this happens and how to resolve it ?

SkySpark throws error 405 on any GET request

Hi ,
We are trying to communicate with Skyspark using the pyhaystack module(python).
The version of of Skyspark and Python we are using are 3.0.22 and 3.7 respectively.
So far we have been successful in connecting with the server and retrieving the site information i.e if we run :

from pyhaystack.client.skyspark import SkysparkScramHaystackSession
session = SkysparkHaystackSession(uri='https://skyspark3.server.org',
                             username='user',
                             password='my_password',
                             project='demo')
session.sites

However, on using the 'his_read' function it is returning a 404 Client error :

 File "/home/pi/.local/lib/python3.7/site-packages/pyhaystack/client/http/sync.py", line 77, in _request
   response.raise_for_status()
 File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
   raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 405 Client Error: GET not allowed for op 'hisRead' for url: https://skyspark3.server.org/api/demo/hisRead?id=%40p%3Ademo%3Ar%3A255873a0-2366039f&range=yesterday

Also attaching the screenshot of the full error message.
Any help on this would be appreciated.
pyHaystackError.log

Skyspark SCRAM login

pyhaystack must be able to log using the new authentication mechanism
http://www.project-haystack.org/forum/topic/462#c4

Bracket search key error

The bracket search implementation presume that tags dis, navName and navNameFormat exist.

It is not always the case which lead to a keyError

Niagara4 Login doesn't work

Following the Skyspark SCRAM authentication issue, we will have to find our way through Niagara4 login mechanism

NaN value in reading history raises exception

When using the method his_read_series to read the history values of a point a "nan" (float) value raises the error "AttributeError: 'float' object has no attribute 'value'"

The error is caused at "pyhaystack/client/ops/his.py", line 127 (as time of writing) values = [each.value for each in data]

Screenshot of niagara history containing the nan value : https://files.gitter.im/ChristianTremblay/pyhaystack/j2I0/image.png

@ChristianTremblay , let me know if you want me to make a PR and how you want to correct it.
I would test for either the "value" attribute in each or check if it's a float. The advantage of the former is that we wouldn't raise an exception on other cases where "value" is not present and the disadvantage is that it may hide other issues. Personaly, I prefer the later, since it's more specific.

HaystackError: java.lang.ClassCastExceptio

Would there be anyway someone could give me a tip on if I am doing anything wrong? To make a long story short a customer is interested in reports above & beyond the Niagara reporting service capabilities but they are at an N4 revision too low to give them an example of pyhaystack/pandas/matplotlib... (4.2, non TLS)

So I am hoping to test this out/give them an example from a different job/server which is at revision 4.7 but also secure TLS.

Im experimenting in Jupyter lab notebooks on Python 3.7. I can print the NAV file of the N4 server:

But attempting this code Ill get a long error:

# Retrieve some points
op = session.find_entity(filter_expr='hwst')
op.wait()
hwst = op.result
# Read today's history for all found points
op = session.his_read_frame(hwst, rng= 'today')
op.wait()
b = op.result
b

Full trace back:

---------------------------------------------------------------------------
HaystackError                             Traceback (most recent call last)
<ipython-input-5-f1f36714fb3f> in <module>
----> 1 hwst = op.result

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pyhaystack\util\state.py in result(self)
     95 
     96         if self.is_failed:
---> 97             self._result.reraise()
     98 
     99         if not self._result_copy:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pyhaystack\util\asyncexc.py in reraise(self)
     26 
     27     def reraise(self):
---> 28         reraise(*self._exc_info)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pyhaystack\client\ops\entity.py in _on_read(self, operation, **kwargs)
     39         try:
     40             # See if the read succeeded.
---> 41             grid = operation.result
     42             self._log.debug('Received grid: %s', grid)
     43 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pyhaystack\util\state.py in result(self)
     95 
     96         if self.is_failed:
---> 97             self._result.reraise()
     98 
     99         if not self._result_copy:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pyhaystack\util\asyncexc.py in reraise(self)
     26 
     27     def reraise(self):
---> 28         reraise(*self._exc_info)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\six.py in reraise(tp, value, tb)
    691             if value.__traceback__ is not tb:
    692                 raise value.with_traceback(tb)
--> 693             raise value
    694         finally:
    695             value = None

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pyhaystack\client\ops\grid.py in _check_err(grid)
    296                         raise HaystackError(
    297                                 grid.metadata.get('dis', 'Unknown Error'),
--> 298                                 grid.metadata.get('traceback', None))
    299                     return grid
    300                 except:

HaystackError: java.lang.ClassCastException

Any tips greatly appreciated, as I mentioned this before its just to get sample of pyhaystack capabilities to use on a different site that is non TLS. (Get the customer to upgrade to 4.7). There is another group in house that does Skysparks that had this nHaystack service running already and I just added the hwst tag as a test trial.

Authentication problem with Niagara 4.6

I have been getting the following error when trying to connect to a JACE 8000:

File "C:\ProgramData\Anaconda3\lib\site-packages\pyhaystack\client\ops\grid.py", line 133, in _do_auth_failed
raise AuthenticationProblem()

AuthenticationProblem

I have set up the user as a basicHTTP user but I still can't connect. After watching the traffic it looks like the JACE sends back a Hello handshake but then pyhaystack just gives up and tries again.

HTTP client needs cookie support

This will be needed for Niagara4 support (issue #39)

lixs74
@lixs74
20:24
almost finish still have one things but it's more python related I have to go but in case
I got a response with the header
Set-Cookie: niagara_userid=pyhaystack;Expires=Fri, 06-Apr-2018 10:22:50 GMT
Set-Cookie: JSESSIONID=eb18a79a08f8d13e694fd76a23460fb4533265d10cceb06c9c;Path=/;HttpOnly
Content-Type: text/plain;charset=iso-8859-1
Connection: close
My issue two of them have the same name Set-Cookie
but I need to get the value
Once we have that we should have a working version

Skyspark : history range

Actually, it seems that Skyspark REST api need the quotes when sending range in a hisRead command :

sensors_his = session.his_read_series('MyPointId.ReturnTemp', rng='"2016-06-05,2016-06-06"').result

This makes use of slice impossible :

import datetime
rng=slice(datetime.date(2016,6,4),datetime.date(2016,6,5))
sensors_his = session.his_read_series('MyPointId.ReturnTemp', rng=rng).result

Gives :

HaystackError: sys::CastErr: java.lang.ClassCastException: fan.sys.Date cannot be cast to java.lang.String

We'll submit this in www.project-haystack.org and see what's going on.
Actually, only Skyspark is affected by this.

isConnected variable doesn't keep track of connectivity to server

If session has been defined for too long, it will be closed by server. A call of session.something will fail.
Maybe we should reconnect more often, or deal with that differently...

add Exceptions instead of print

Call SkySpark Functions

I wanted to ask if we can call developed functions in SkySpark with specific inputs through pyhaystack and get the result back ?

Authentication issue using domain name

Hi there,

Hope you are doing well. I have been trying to use pyhaystack to connect to a SkySpark instance running on a remote server. I am able to do so using the IP address as the URI, but am receiving an error while trying to use our domain name as the URI: pyhaystack.client.http.exceptions.HTTPStatusError: [Errno 401 Client Error: Unauthorized for url: https://tipify.brightpowerinc.com/ui] 401 despite using the same credentials and project as I did with the IP address.

I'm able to successfully authenticate using this node project with either the IP or domain name, which leads me to believe the issue may be with pyhaystack but please correct me if I'm wrong.

My company works with Intellastar (one of the SkyFoundary partners) to host our skyspark instance, they were able to provide some context from the SkySpark side, that I thought might be helpful to bring up:

• Authentication requests are being made to /ui instead of /api/projectname/about
• The requests are being made as GET requests instead of POST requests. I stumbled across this conversation So I realize you are already aware of this issue, but was just curious if you had any updates.

Either way, I'm not sure if either of those issues would impact the problem of using a domain name vs the IP but thought I would share.

Thanks in advance,

Trish

help with HisReadSeriesOperation

I've got a session open with skyspark and can pull data via session.about(), etc but HisReadSeriesOperation is giving me issues. My code:

op = pyhaystack.client.ops.his.HisReadSeriesOperation(session, '<point name>', 'today', None, 'dict')
op.go()
op.wait()

Always returns op.is_failed = True. The traceback looks like this:

Traceback (most recent call last):
  File "pyhay.py", line 64, in <module>
    if (op.result):
  File "C:\Users\USAP677924\api\lib\site-packages\pyhaystack\util\state.py", line 97, in result
    self._result.reraise()
  File "C:\Users\USAP677924\api\lib\site-packages\pyhaystack\util\asyncexc.py", line 28, in reraise
    reraise(*self._exc_info)
  File "C:\Users\USAP677924\api\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\USAP677924\api\lib\site-packages\pyhaystack\client\ops\his.py", line 104, in _on_read
    grid = operation.result
  File "C:\Users\USAP677924\api\lib\site-packages\pyhaystack\util\state.py", line 97, in result
    self._result.reraise()
  File "C:\Users\USAP677924\api\lib\site-packages\pyhaystack\util\asyncexc.py", line 28, in reraise
    reraise(*self._exc_info)
  File "C:\Users\USAP677924\api\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\Users\USAP677924\api\lib\site-packages\pyhaystack\client\ops\grid.py", line 298, in _check_err
    grid.metadata.get('traceback', None))
pyhaystack.exception.HaystackError: sys::CastErr: java.lang.ClassCastException

Also I assume that the output of HisReadSeriesOperation is similar to the data series to be written in HisWriteSeriesOperation but the documentation doesn't really elaborate on that...

SkySpark Authentication

Hi, I enabled the haystack connector on SkySpark. (As I understand it, I do not need to create a specific instance of the haystack connector on SkySpark since that's only needed for SkySpark to retrieve information from another server, although I have tried both ways.) Now, I am trying to connect to a session via pyhaystack by calling the following on JupyterLab:

session = SkysparkHaystackSession(uri='http://server:port/api',
                                  username='user',
                                  password='pw',
                                  project='proj_name',
                                  pint=True)
session.is_logged_in

I have tried all combinations of uri's I can think of, including our server's IP address, a localhost address for a local instance of SkySpark, default ports (80, 8080) and port numbers listed on our SkySpark server. However, session.is_logged_in always returns False, and on the SkySpark end there is also no record of a log-in.
Could you recommend how to fix this issue?
Thank you for your help!

Proposition to clean HaystackClient class with the use of Mixin

I'll commit a branch for comments

pyhaystack server implementation

Just in case I missed it. Is it possible to use this library create a server that listens for haystack calls?

If not is an implementation being considered?

Niagara 4.9 | Possible breaking change for login

Ref : Breaking change: HTTP SCRAM Authentication Session Cookie

I don't have 4.9 yet to make any test, here is what article tells... with Java code extract

DESCRIPTION

Summary

Prior to Niagara 4.9, the HTTP SCRAM authentication client reference implementation AuthClientExample.java made the assumption that the session ID cookie would be set in the first Set-Cookie response header. This assumption was unsafe, and due to changes to the web server in Niagara 4.9, the session ID cookie will no longer be set in the first Set-Cookie response header. Any HTTP SCRAM authentication client implementations based on the AuthClientExample reference implementation must be updated to account for differing ordering of Set-Cookie headers to properly capture the session ID cookie.

Remediation

HTTP SCRAM authentication client implementations based on the AuthClientExample reference implementation should have a section of code similar to this section from the reference implementation:

// Set the session Cookie we got from the server
// make sure you save the sessionId for subsequent requests for the same session
String cookie = connection.getHeaderField("Set-Cookie");
if (cookie != null && cookie.startsWith(niagaraParameters.getSessionCookieName()))
{
  sessionId = (cookie.split(";"))[0].trim();
  sessionId = sessionId.split("=")[1];
  System.out.println("*** sessionid: " + sessionId);
}

This section of code should be updated to match the new reference implementation to account for multiple Set-Cookie headers:

// Set the session Cookie we got from the server
// make sure you save the sessionId for subsequent requests for the same session
List<String> cookieHeaders = connection.getHeaderFields().get("Set-Cookie");
if (cookieHeaders != null)
{
  for (String cookie : cookieHeaders)
  {
    if (cookie != null && cookie.startsWith(niagaraParameters.getSessionCookieName()))
    {
      sessionId = (cookie.split(";"))[0].trim();
      sessionId = sessionId.split("=")[1];
      System.out.println("*** sessionid: " + sessionId);
      break;
    }
  }
}

@sjlongland For your info

pyhaystack authentication fail

Hello,

I followed the documentation for SkySpark authentication...

 session = SkysparkHaystackSession(uri='https://someDomain.haystack.someProvider.com',
    username='someUserName',
    password='somePassword',
    project='someProject',
    pint = True)

znt = session.find_entity(filter_expr='sensor and power')

print znt.result

I receive the following error message:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/jbackes/linkedin/infradev/haystack-test/__main__.py", line 14, in <module>
    print znt.result
  File "/usr/local/lib/python2.7/site-packages/pyhaystack/util/state.py", line 97, in result
    self._result.reraise()
  File "/usr/local/lib/python2.7/site-packages/pyhaystack/util/asyncexc.py", line 28, in reraise
    reraise(*self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/pyhaystack/client/ops/entity.py", line 41, in _on_read
    grid = operation.result
  File "/usr/local/lib/python2.7/site-packages/pyhaystack/util/state.py", line 97, in result
    self._result.reraise()
  File "/usr/local/lib/python2.7/site-packages/pyhaystack/util/asyncexc.py", line 28, in reraise
    reraise(*self._exc_info)
  File "/usr/local/lib/python2.7/site-packages/pyhaystack/client/ops/grid.py", line 247, in _do_auth_failed
    raise AuthenticationProblem()

Does not support Salt Auth for Skyspark v3

any plans to implement it?

thanks

his_read_series returning grids with same values in all rows

The code below returns different results to skyspark

points = session.read(filter_expr='sensor and id==<point_id>', limit=1)
history = session.his_read_series(points.result[0]['id'], rng="yesterday").result
for row in history.iteritems():
    print(row)

Gives the result below in the prompt:

2016-06-07 00:00:02+10:00    20.690001
2016-06-07 00:15:03+10:00    20.690001
2016-06-07 00:30:03+10:00    20.690001
2016-06-07 00:45:03+10:00    20.690001
2016-06-07 01:00:04+10:00    20.690001
2016-06-07 01:15:04+10:00    20.690001
2016-06-07 01:30:03+10:00    20.690001
2016-06-07 01:45:02+10:00    20.690001
2016-06-07 02:00:02+10:00    20.690001
2016-06-07 02:15:03+10:00    20.690001
2016-06-07 02:30:03+10:00    20.690001
2016-06-07 02:45:03+10:00    20.690001
2016-06-07 03:00:06+10:00    20.690001
2016-06-07 03:15:04+10:00    20.690001
2016-06-07 03:30:04+10:00    20.690001
2016-06-07 03:45:02+10:00    20.690001
2016-06-07 04:00:03+10:00    20.690001
2016-06-07 04:15:04+10:00    20.690001
2016-06-07 04:30:03+10:00    20.690001
2016-06-07 04:45:02+10:00    20.690001
2016-06-07 05:00:03+10:00    20.690001
2016-06-07 05:15:03+10:00    20.690001
2016-06-07 05:30:03+10:00    20.690001
2016-06-07 05:45:02+10:00    20.690001
2016-06-07 06:00:04+10:00    20.690001
2016-06-07 06:15:03+10:00    20.690001
2016-06-07 06:30:02+10:00    20.690001
2016-06-07 06:45:02+10:00    20.690001
2016-06-07 07:00:02+10:00    20.690001
2016-06-07 07:15:03+10:00    20.690001
                               ...
2016-06-07 16:30:03+10:00    20.690001
2016-06-07 16:45:02+10:00    20.690001
2016-06-07 17:00:04+10:00    20.690001
2016-06-07 17:15:03+10:00    20.690001
2016-06-07 17:30:04+10:00    20.690001
2016-06-07 17:45:05+10:00    20.690001
2016-06-07 18:00:12+10:00    20.690001
2016-06-07 18:15:14+10:00    20.690001
2016-06-07 18:30:20+10:00    20.690001
2016-06-07 18:45:02+10:00    20.690001
2016-06-07 19:00:03+10:00    20.690001
2016-06-07 19:15:03+10:00    20.690001
2016-06-07 19:30:04+10:00    20.690001
2016-06-07 19:45:03+10:00    20.690001
2016-06-07 20:00:03+10:00    20.690001
2016-06-07 20:15:05+10:00    20.690001
2016-06-07 20:30:07+10:00    20.690001
2016-06-07 20:45:03+10:00    20.690001
2016-06-07 21:00:02+10:00    20.690001
2016-06-07 21:15:04+10:00    20.690001
2016-06-07 21:30:03+10:00    20.690001
2016-06-07 21:45:03+10:00    20.690001
2016-06-07 22:00:02+10:00    20.690001
2016-06-07 22:15:04+10:00    20.690001
2016-06-07 22:30:02+10:00    20.690001
2016-06-07 22:45:03+10:00    20.690001
2016-06-07 23:00:03+10:00    20.690001
2016-06-07 23:15:03+10:00    20.690001
2016-06-07 23:30:03+10:00    20.690001
2016-06-07 23:45:03+10:00    20.690001

The skyspark grid:

    6-Jun-2016 Mon 23:45:03 AEST    20.73 °C
    7-Jun-2016 Tue 00:00:02 AEST    20.69 °C
    7-Jun-2016 Tue 00:15:03 AEST    20.64 °C
    7-Jun-2016 Tue 00:30:03 AEST    20.61 °C
    7-Jun-2016 Tue 00:45:03 AEST    20.59 °C
    7-Jun-2016 Tue 01:00:04 AEST    20.57 °C
    7-Jun-2016 Tue 01:15:04 AEST    20.54 °C
    7-Jun-2016 Tue 01:30:03 AEST    20.51 °C
    7-Jun-2016 Tue 01:45:02 AEST    20.47 °C
    7-Jun-2016 Tue 02:00:02 AEST    20.46 °C
    7-Jun-2016 Tue 02:15:03 AEST    20.44 °C
    7-Jun-2016 Tue 02:30:03 AEST    20.41 °C
    7-Jun-2016 Tue 02:45:03 AEST    20.39 °C
    7-Jun-2016 Tue 03:00:06 AEST    20.37 °C
    7-Jun-2016 Tue 03:15:04 AEST    20.37 °C
    7-Jun-2016 Tue 03:30:04 AEST    20.34 °C
    7-Jun-2016 Tue 03:45:02 AEST    20.29 °C
    7-Jun-2016 Tue 04:00:03 AEST    20.28 °C
    7-Jun-2016 Tue 04:15:04 AEST    20.25 °C
    7-Jun-2016 Tue 04:30:03 AEST    20.24 °C
    7-Jun-2016 Tue 04:45:02 AEST    20.23 °C
    7-Jun-2016 Tue 05:00:03 AEST    20.23 °C
    7-Jun-2016 Tue 05:15:03 AEST    20.2 °C
    7-Jun-2016 Tue 05:30:03 AEST    20.18 °C
    7-Jun-2016 Tue 05:45:02 AEST    20.18 °C
    7-Jun-2016 Tue 06:00:04 AEST    20.16 °C
    7-Jun-2016 Tue 06:15:03 AEST    20.16 °C
    7-Jun-2016 Tue 06:30:02 AEST    20.17 °C
    7-Jun-2016 Tue 06:45:02 AEST    20.2 °C
    7-Jun-2016 Tue 07:00:02 AEST    20.23 °C
    7-Jun-2016 Tue 07:15:03 AEST    20.24 °C
    7-Jun-2016 Tue 07:30:03 AEST    20.28 °C
    7-Jun-2016 Tue 07:45:04 AEST    20.31 °C
    7-Jun-2016 Tue 08:00:03 AEST    20.34 °C
    7-Jun-2016 Tue 08:15:03 AEST    20.28 °C
    7-Jun-2016 Tue 08:30:03 AEST    20.38 °C
    7-Jun-2016 Tue 08:45:03 AEST    20.49 °C
    7-Jun-2016 Tue 09:00:04 AEST    20.63 °C
    7-Jun-2016 Tue 09:15:02 AEST    20.78 °C
    7-Jun-2016 Tue 09:30:02 AEST    21.05 °C
    7-Jun-2016 Tue 09:45:03 AEST    20.94 °C
    7-Jun-2016 Tue 10:00:03 AEST    21.02 °C
    7-Jun-2016 Tue 10:15:05 AEST    21.12 °C
    7-Jun-2016 Tue 10:30:03 AEST    21.17 °C
    7-Jun-2016 Tue 10:45:02 AEST    21.25 °C
    7-Jun-2016 Tue 11:00:03 AEST    21.21 °C
    7-Jun-2016 Tue 11:15:03 AEST    21.24 °C
    7-Jun-2016 Tue 11:30:04 AEST    21.21 °C
    7-Jun-2016 Tue 11:45:02 AEST    21.22 °C
    7-Jun-2016 Tue 12:00:04 AEST    21.17 °C
    7-Jun-2016 Tue 12:15:03 AEST    21.14 °C
    7-Jun-2016 Tue 12:30:04 AEST    21.07 °C
    7-Jun-2016 Tue 12:45:03 AEST    21.04 °C
    7-Jun-2016 Tue 13:00:03 AEST    20.99 °C
    7-Jun-2016 Tue 13:15:03 AEST    20.98 °C
    7-Jun-2016 Tue 13:30:03 AEST    21.04 °C
    7-Jun-2016 Tue 13:45:03 AEST    21.09 °C
    7-Jun-2016 Tue 14:00:03 AEST    21.15 °C
    7-Jun-2016 Tue 14:15:03 AEST    20.59 °C
    7-Jun-2016 Tue 14:30:03 AEST    20.67 °C
    7-Jun-2016 Tue 14:45:02 AEST    20.69 °C
    7-Jun-2016 Tue 15:00:04 AEST    20.69 °C
    7-Jun-2016 Tue 15:15:03 AEST    20.66 °C
    7-Jun-2016 Tue 15:30:04 AEST    20.69 °C
    7-Jun-2016 Tue 15:45:02 AEST    20.69 °C
    7-Jun-2016 Tue 16:00:02 AEST    20.7 °C
    7-Jun-2016 Tue 16:15:02 AEST    20.66 °C
    7-Jun-2016 Tue 16:30:03 AEST    20.62 °C
    7-Jun-2016 Tue 16:45:02 AEST    20.62 °C
    7-Jun-2016 Tue 17:00:04 AEST    20.59 °C
    7-Jun-2016 Tue 17:15:03 AEST    20.54 °C
    7-Jun-2016 Tue 17:30:04 AEST    20.57 °C
    7-Jun-2016 Tue 17:45:05 AEST    20.54 °C
    7-Jun-2016 Tue 18:00:12 AEST    20.57 °C
    7-Jun-2016 Tue 18:15:14 AEST    20.66 °C
    7-Jun-2016 Tue 18:30:20 AEST    20.71 °C
    7-Jun-2016 Tue 18:45:02 AEST    20.78 °C
    7-Jun-2016 Tue 19:00:03 AEST    20.83 °C
    7-Jun-2016 Tue 19:15:03 AEST    20.88 °C
    7-Jun-2016 Tue 19:30:04 AEST    20.89 °C
    7-Jun-2016 Tue 19:45:03 AEST    20.9 °C
    7-Jun-2016 Tue 20:00:03 AEST    20.92 °C
    7-Jun-2016 Tue 20:15:05 AEST    20.91 °C
    7-Jun-2016 Tue 20:30:07 AEST    20.88 °C
    7-Jun-2016 Tue 20:45:03 AEST    20.93 °C
    7-Jun-2016 Tue 21:00:02 AEST    20.93 °C
    7-Jun-2016 Tue 21:15:04 AEST    20.93 °C
    7-Jun-2016 Tue 21:30:03 AEST    20.88 °C
    7-Jun-2016 Tue 21:45:03 AEST    20.93 °C
    7-Jun-2016 Tue 22:00:02 AEST    20.93 °C
    7-Jun-2016 Tue 22:15:04 AEST    20.73 °C
    7-Jun-2016 Tue 22:30:02 AEST    20.69 °C
    7-Jun-2016 Tue 22:45:03 AEST    20.68 °C
    7-Jun-2016 Tue 23:00:03 AEST    20.66 °C
    7-Jun-2016 Tue 23:15:03 AEST    20.65 °C
    7-Jun-2016 Tue 23:30:03 AEST    20.6 °C
    7-Jun-2016 Tue 23:45:03 AEST    20.54 °C
    8-Jun-2016 Wed 00:00:03 AEST    20.51 °C

Writing histories to SkySpark

Hi, I am having trouble writing histories to SkySpark via the his_write_series function. I saw this previous post that says the function can take in a dict with keys=datetime.datetime objects and values=data points. I have implemented this such that my hist_dict outputs:

hist_dict = {datetime.datetime(2018, 5, 10, 0, 0): 19.26300048828125,
 datetime.datetime(2018, 5, 10, 0, 1): 19.26300048828125, ...} 

The data points were read from another point from start_time to end_time, which are also datetime objects.
Next, I call:

op = session.his_write_series(samplePointToWrite, hist_dict )
op.wait()

However, when I read the history of the samplePointToWrite from start_time to end_time, an empty MetaSeries is returned. The his_read_series function has been working for other points, and there is also no data written on SkySpark.

Do you have any suggestions on how to proceed? Thank you!

History Read / Empty results handling

When empty results are given by the server (no data) a few bugs will come up in :

• HisReadFrameOperation._do_postprocess()
• HisReadSeriesOperation._on_read()

merging from widesky branch - broke sync connection

Problems with imports
Typos...
Need some correction

support for Kodaro haystack driver

Does pyhaystack support the Kodaro haystack driver? I don't see it mentioned in the docs, but would like to confirm.

Assuming this is a feature request, can the maintainers give me a sense of how hard this would be, what level of interest they have in the feature, and what support they can give if I decide to have a go?

I'm running into some Niagara devices running this plugin. It's not yet clear whether installing the N4 plugin is an option at these sites.

function HayStackSession::_get requests zinc when told to get raw json

If I do this:

from urrlib.parse import quote

session = SkysparkScramHaystackSession('http://address:port', username='someone', password=******, project='someproject', http_args={'tls_verify': False})

request = 'readAll(sometag)'
result = session._get_grid(f'eval?expr={quote(request)}',None,expect_format='json',raw_response=True)

Inspecting the request using a network sniffer I see:

GET /api/someproject/eval?expr=readAll%28sometag%29 HTTP/1.1
Host: address:port
User-Agent: python-requests/2.21.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: BEARER authToken=sXtUkHhuwVUF0BIZ7cR6xH7tXYRNp7riWfqWSDAmbvGg-2a8
Cookie: skyarc-auth-9080=sXtUkHhuwVUF0BIZ7cR6xH7tXYRNp7riWfqWSDAmbvGg-2a8`

The Accept header is wrong it should be: Accept: application/json
It follows that the server returns zinc and not json.

Without the raw_response argument the Accept header is correct. So it seems that the combination of raw_response=True and expect_format='json' does not work as expected.

dump function fills empty cells with `N`

Hello,

I noticed that empty cells are treated as null values when a dump is done.

Is it possible to have a null type for zinc to differentiate it from an empty cell?

ver:"2.0"
air,zone,temp,point,sp,fan,kind,his,id,writable,envizi,tz,dis,supply,defaultVal,sensor,return,nubeCur,cur,connRef,nubeConnRef,nubeWrite,nubeHis,pressure
M,M,M,M,M,M,"Number",M,@U02,M,"ZAT_SP","Sydney","ZAT Setpoint",N,N,N,N,N,N,N,N,N,N,N
M,N,M,M,M,M,"Number",M,@UO1,M,"SAT_SP","Sydney","SAT Setpoint",M,N,N,N,N,N,N,N,N,N,N
M,N,M,M,M,M,"Number",M,@AC1_SAT_SP,N,"AC1_SATSP","Sydney","SAT_SP",M,0,N,N,N,N,N,N,N,N,N
M,N,M,M,N,M,"Number",M,@UI2,N,"SAT!","Sydney","SAT",M,N,M,N,N,N,N,N,N,N,N
M,N,M,M,M,M,"Number",M,@AC1_SAT_SP1,N,"AC1_SATSP","Sydney","SAT_SP",M,0,N,N,N,N,N,N,N,N,N
M,N,M,M,N,M,"Number",M,@UI3,N,"RAT","Sydney","RAT",N,N,M,M,N,N,N,N,N,N,N
M,M,M,M,N,M,"Number",M,@UI4,N,"ZAT","Sydney","ZAT",N,N,M,N,N,N,N,N,N,N,N
M,N,M,M,M,M,"Number",M,@UO1_point,M,"SAT_SP","Sydney","SAT Setpoint",M,0,N,N,"@UO1",M,"@nubeConnId","@nubeConnId","@UO1","@UO1",N
M,N,N,M,N,M,"Number",M,@UI1,N,"SAP","Sydney","SASP",M,N,M,N,N,N,N,N,N,N,M

EDIT: Added example zinc output

Question: Retrieve files from SkySpark uri

Is there any built in solution that we can retrieve files from SkySpark uri through pyhaystack ?
For example, the following uri returns an image file sitting in SkySpark bin folder:
http://skysparkuri/api/projectA/rec/someid/file

ValueError in grid is symptomatic of a disconnection from server

Getting this error should trig a reconnection attempt when content_type is text/html

Gitter/Jar Files

Hi Christian,

Do you use gitter still for pyhaystack help? I cant find it thru googling...

Could you give me a tip on where I can find the Jar files online to implement nHaystack on an N4 server? Also would you know if the server running 4.2, can I use a 4.7 workbench to put nHaystack jar files on the server thru the software manager on a platform connection?

Thanks,
Ben

Documentation effort needed before release 0.92

Documentation must be reviewed to add :

• New authentication schemes
• usage of pyhaystack.connect() or get_instance
• usage of new mixins
• usage of session.site vs session.sites
• how to pass a certificate for https connections
• how to use a JSON or YAML file to log (pass it to get_instance)
• any other point forgot here

Skyspark with pyhaystack to create and modify points / tags

Hello,

I'm a new user of skyspark and we're interested in integrating our campus' sites and historical data through the REST API or by using the pyhaystack library if that supports what we're trying to accomplish.

So far I've been successful in connecting to my server with the pyhaystack client and been able to read tags with operations like op = session.find_entity(filter_expr='temp')

an example sensor looks like this:

t1.tags
Out[20]: {air, disMacro='$equipRef $navName', equipRef=Ref('p:ucdavis:r:246332ce-cd5328e0', 'GBSF VAV6512', True), his, kind='Number', locationRef=Ref('p:ucdavis:r:24638759-cc1a3b2a', 'Floor 5', True), mod=datetime.datetime(2019, 5, 7, 17, 4, 17, 282000, tzinfo=datetime.timezone.utc), navName='Zone Air Temp', point, sensor, siteRef=Ref('p:ucdavis:r:24632687-0e4bd94c', 'GBSF', True), temp, tz='Los_Angeles', unit='°F'}

All is fine and well, except when I want to modify the tag list:

t1.tags['faketag1'] = None

Traceback (most recent call last):
File "<ipython-input-27-4eb0de79124a>", line 1, in <module>
t1.tags['faketag1'] = None

TypeError: 'ReadOnlyEntityTags' object does not support item assignment

Is there a setting in my skyspark server or the pyhaystack client that is ensuring that all of my tags are of type "ReadOnlyEntityTags" and not "MutableEntityTags"

This is the section of the pyhaystack documentation I was following along with. https://pyhaystack.readthedocs.io/en/latest/tags.html#adding-changing-and-deleting-tags

Thanks!

Start making tests

Need a documentation update before merging develop to master

Must cover :
evalexpr for Skyspark
bql for Niagara

any other correction if needed.

pyhaystack through a proxy script

Documenting this from project-haystack website
@sjlongland

DANIEL VILLA jeu. 16 mai
Is there any way to connect to a haystack server that uses a proxy server script using pyhaystack? I have managed to do so on my own using a pypac session but I would rather leverage the broader functionality of pyhaystack if possible.

Stuart Longland ven. 17 mai
pyhaystack internally uses the python-requests module for its HTTP access.

https://2.python-requests.org/en/master/user/advanced/#proxies

That page suggests the library should respect the "industry standard" HTTP_PROXY and HTTPS_PROXY environment variables, so one way that may work is to do the following:

from sys import environ

environ['HTTP_PROXY'] = 'http://your.proxy.server:3128'
environ['HTTPS_PROXY'] = 'http://your.proxy.server:3128'

prior to establishing your session with pyhaystack.

Alternatively for more flexibility, all session objects take a keyword argument; http_args. You can pass a dict object to this parameter with any options for the HTTP client class:

https://pyhaystack.readthedocs.io/en/latest/pyhaystack.client.http.html#pyhaystack.client.http.base.HTTPClient

The value you pass here is passed direct to python-requests, so:

from pyhaystack.client import get_instance

session = get_instance(implementation='yourserver', arg1='val1', … etc,
                       http_args=dict(
                          proxies={
                              'http': 'http://your.proxy.server:3128',
                              'https': 'http://your.proxy.server:3128',
                              'http://example.org/haystack': 'http://special.proxy.server:3128',
                          }
                       ))

It is also possible to use SOCKS proxies, if that's what you require.

Reviewing workflow

Before issuing the newest version...

Cleanup of the repo
New branch workflow :

• Features added as feature branch -> ex. feature/my_new_idea
• Stage becomes develop
• Master keeps its importance as the stable version

We'll try to stick to this model
http://nvie.com/posts/a-successful-git-branching-model/

I apologize right now for every future mistakes I will make :-)

<FindEntityOperation failed>

I get the error "FindEntityOperation failed" when I run:
op = session.find_entity(filter_expr="zone and temp and sensor")
however it return correct values when I run:
op = session.find_entity(filter_expr="ahu and equip")

Also when I run:
op = session.find_entity(filter_expr= """id==@...""")
for some points it returns the right entity and for some points it returns:

I have checked both points exist.

This is through SkysparkHaystackSession
I really appreciate to know what I am missing.

Question: SkySpark 3 Authentication

Does pyhaystack authentication work on SkySpark 3 ?

pyReadLine error

Hi,

i'm getting an error when i use the grid_format=hszinc.MODE_JSON when setting up the session

Traceback (most recent call last):
File "C:\Users\makhdum shaikh\AppData\Local\Continuum\Anaconda3\lib\site-packages\pyreadline\console\console.py", line 768, in hook
rapper_23
res = ensure_str(readline_hook(prompt))
File "C:\Users\makhdum shaikh\AppData\Local\Continuum\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 571, in readline
self._readline_from_keyboard()
File "C:\Users\makhdum shaikh\AppData\Local\Continuum\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 536, in _readline_fro
keyboard
if self._readline_from_keyboard_poll():
File "C:\Users\makhdum shaikh\AppData\Local\Continuum\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 556, in readline_fro
keyboard_poll
result = self.mode.process_keyevent(event.keyinfo)
File "C:\Users\makhdum shaikh\AppData\Local\Continuum\Anaconda3\lib\site-packages\pyreadline\modes\emacs.py", line 243, in process
yeven

"refresh" not implemented on watch_poll method

Although the documentation of method def watch_poll(self, watch, refresh=False, callback=None): has following text:
If refresh is True, then all points on the watch will be updated, not just those that have changed since the last poll.
, the parameter refresh is unused and hence above functionality as stated on documentation isn't obtained.

Skyspark connection

Need upgrade to 0.9 to work

Skyspark needs post requests instead of get requests for his_read

Let see how we can figure this out

@justindarcy
@sjlongland I will need your input on that.

From what I remember, all implementations were using the same base (get requests). Question will be do we need to rewrite everything or create something else exclusive for Skyspark...

(Question?): Closing PyHaystack sessions on both client and server.

We are encountering a problem with how to execute PyHaystack frequently on a client machine in a way that does not burden our SkySpark server. Our client machine executes a python script every 10 minutes. This script is being used to pull data from our Skyspark server and push data to another data server we are working with. The framework of our script is as follows:

session = pyhaystack.connect('skyspark',
                            uri='https://serveraddress',
                            username='user',
                            password='password',
                            project = 'project',
                            pint=True,
                            grid_format=hszinc.MODE_JSON,
                            http_args={'debug':False},
                             )

data_pull = session.get_eval("apiDataTransfer()")
data_pull.wait()
data_json=remove_null_bool(json.loads(hszinc.dump(data_pull.result, mode=hszinc.MODE_JSON)))

do other things until end of file

This script runs every 10 minutes on the client machine (as a recurring cron job). The script takes about 30 seconds to run, in terms of connecting to the Skyspark server and pulling data from the apiDataTransfer() call.

We have come to learn that by running the script this way, the SkySpark server opens its own session each time the client connects to it (i.e., every 10 minutes). Each session is left open for a default period of 3 hours as the Python script does not provide any indication to the server that the session should close. As a result, our run-every-10-minutes script results in opening 180 sessions on the server and is unnecessarily overloading the server.

Whilst we can fix this problem by running our every-10-minutes process as a recurring loop within a single Python file (and thus only 1 session), I would like to know if there is a way to disconnect a session properly through the PyHaystack library.

In the same way that pyhaystack.connect establishes a connection, is there anything to the effect of a pyhaystack.disconnect that would ensure the connection is properly closed on both client and server?

Many thanks for your feedback.

pyhaystack.connect failed with Niagara 4.4.73.24

Results in Authentication failure

Using :

from pyhaystack.client.niagara import Niagara4HaystackSession
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
session = Niagara4HaystackSession(uri='https://192.168.1.2', 
                                  username='user', 
                                  password='password',
                                  http_args={'debug':True, 'tls_verify':False})
            
session.about().result

works

Failing to log in to SkySpark with correct credentials

Tried testing with some simple code with skyspark with the code at the bottom of this issue. The result is posted below:

RESULT

False
<GetGridOperation failed>

CODE

from pyhaystack.client.skyspark import SkysparkHaystackSession
session = SkysparkHaystackSession(uri='http://url_to_server',
                                username='myusername',
                                password='secretpassword',
                                project = 'project')

if(__name__ == '__main__'):
    about = session.about()
    print(session.is_logged_in)
    print(about)

Running pyhaystack on a Mac gives attribute error about hszinc.MODE_ZINC

Submitted by Ronald Snyder on gitter.im

Improvement: Asynchronous HTTP client

There's one already, it's a "dummy HTTP client" used for unit testing, but I'm thinking a real one would be a nice-to-have at some point in the future.

Possible asynchronous frameworks to target:

• asyncio (Python 3.3 and above)
• -Trollius (this is a Python 2.x backport of the tulip asynchronous framework, the forerunner to asyncio, or maybe not as it's now "depreciated")-
• TornadoWeb
• TwistedMatrix
• others?

No rush on this now, the API core is able to support an asynchronous client, this is why it returns a state machine instead of the value when calling methods: the state machine could be running in a separate thread or it could be waiting for control to return to the IO loop.

Supporting a new client is basically a case of writing an implementation in pyhaystack/client/http supporting the new HTTP client.

Needle and Pyhaystack

Hi Pyhastack team.

So glad to see this effort, connecting SkySpark and Niagara frameworks to harness the significant utility provided by the Python scientific stack is something needed for this building energy community. :)

Myself and a few others in the open source community (@christopher-hartley) wanted to understand better how pyhaystack compares with the Smart-Green/needle library? Browsing across the commit history shows that it is a fork of pyhaystack, and indeed it seems both projects goals are totally aligned.

Any feedback would be appreciated—thanks so much!