ke-works/pykechain

Open source Python API to KE-chain 2

KE-chain Python SDK

About pykechain

pykechain is a python library for advanced users of KE-chain. It will enable users to connect and fully interact to all features of KE-chain, the digital verification and high tech systems design platform of KE-works. With it you can interact with KE-chain, its parts, projects, forms, workflows, activities, scripts and all other aspects of KE-chain from within python scripts or iPython / Jupyter notebooks.

It requires a normal user access to a KE-chain (version 3) instance for it to work.

Note

This version of pykechain (> 4.0.0) is compatible with the latest release of KE-chain where we added the Forms feature. It is fully backward compatible with all KE-chain versions v2022 and higher. This version discontinues support for python version 3.6.

Note

This version of pykechain (> 3.0.0) is suited from KE-chain versions > 3 (or > v2021) running on python >= 3.7 exclusively. If you desire to connect to an older version of KE-chain or run on python 2.7, please use a pykechain v2 release. Put in the requirements pykechain~=2.7.

Basic usage

Ensure you have member access to a KE-chain instance and login:

from pykechain import Client
kec = Client(url='https://<domain>.ke-chain.com')
kec.login(username='demo_user', password='pastaplease')

Now interact with it:

project = kec.scope('Bike Project')
for part in project.parts():
    print(part.name)

Installation

pykechain is easily installed using pip. pykechain is Python 3.7, 3.8, 3.9, 3.10 and pypy3 compatible:

pip install pykechain

Or if you want to live on the edge, install the latest and greatest from the master branch:

pip install https://github.com/KE-works/pykechain/archive/main.zip

In scripts you can either use Pipenv or a pip requirements.txt file to install pykechain as a requirement

in a Pipfile:

[packages]
pykechain = "*"
# or when you want to install a certain branch
pykechain = {ref = "main", git = "https://github.com/KE-works/pykechain"}

in a pip requirements.txt:

pykechain
# or when you want to install a certain branch i.e. `main`
git+https://github.com/KE-works/pykechain.git@main#egg=pykechain

Changelog

A proper changelog is maintained in the Changelog

pykechain's Issues

use activity.children() to retrieve subtasks/processes of an activity, similar to parts

add parent filter to part requester

• add a parent= filter to the Client.parts method.
• add a children() method to the Part model

Get documentation up to date

I noticed something missing:

• autoclass:: pykechain.models.ReferenceProperty

And verify all else is ok.

Cannot edit property (model) name

I'd like something similar to "edit" command on activities, tasks, models etc.

provide a `Part.to_dict()` method that return the part properties as a dictionary

eg

>>> bike = project.part('Bike')
>>> print(bike.to_dict())
{ 'rear_wheel': True,
  'front_wheel': False,
  'frame': 'women'}

Introduce bulk deletion

Retrieve related objects (parts/properties/proxies etc)

Things I'd like to do:

• Retrieve the model of an instance
• Retrieve the instances of a model
• Retrieve the proxies of a catalog model
• Retrieve the catalog model of proxy
• Retrieve the (direct) properties of a part instance
• Retrieve the (direct) properties of a part model

This way, I can lookup sets of parts/properties and act on them (e.g. configure them). Currently, without the above, it's difficult to do more complex operations since retrieving the right parts is difficult.

validate all property values types

I don't think datetime and link property values have been tested yet

edit: also revisit enums

Missing 'six' package

When retrieving a part (kec.scope('Bike Project')), I got the following error:

Traceback (most recent call last): File "script.py", line 1, in <module> from pykechain import Client File "/home/bennie/Development/docker-test/_venv/local/lib/python2.7/site-packages/pykechain/__init__.py", line 3, in <module> from .client import Client File "/home/bennie/Development/docker-test/_venv/local/lib/python2.7/site-packages/pykechain/client.py", line 9, in <module> from .models import Scope, Activity, Part, PartSet, Property File "/home/bennie/Development/docker-test/_venv/local/lib/python2.7/site-packages/pykechain/models/__init__.py", line 5, in <module> from .activity import Activity File "/home/bennie/Development/docker-test/_venv/local/lib/python2.7/site-packages/pykechain/models/activity.py", line 7, in <module> from six import text_type ImportError: No module named six

After installing six everything works as expected.

Ability to get the configuration of a task

At times I want to get the configured properties (and parts) of a task, so that I can act on them one by one. It is currently possible to set the configuration, but not to get it

Ability to add/retrieve activities

For my work in AGILE

method to return the muliticiplity of a part (model

see this code...

str(part_model._json_data.get('multiplicity')) == "ZERO_MANY":

Add possibility to edit description of property models

Increase tests coverage for 1.9

Goal: test coverage > 95%

Docs error in add_property

https://pykechain.readthedocs.io/en/latest/api/models.html?highlight=add_property

It says: see create_property for parameters. That's not true, create property requires the model as input, add_property creates a property on a model - model.add_property( * args )

if pykechain is installed in the system site-packages it gives an no env file warning when importing

1. install pykechain in the system sitepackages (not a virtualenv)
2. start a python interpreter
3. write >>> import pykchain
4. see the warning:

Python 3.4.4 (default, Mar 15 2016, 14:38:45)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pykechain
/usr/lib/python3.4/site-packages/envparse.py:195: UserWarning: Could not any envfile.
  warnings.warn('Could not any envfile.')
>>> exit()

culprit is https://github.com/KE-works/pykechain/blob/master/pykechain/client.py#L21

add ability to upload a filedescripter directly as attachment

in: https://github.com/KE-works/pykechain/blob/master/pykechain/models/property_attachment.py#L53

we could add the ability to detect if we get injected a filedescripter (open file) and upload this directly.

Increase tests coverage

Goal: bring it to around 95%

add KE-chain user triggered execution of pykechain code

In order to fully support automation in KE-chain it should be possible to execute pykechain code with the push of a button.

I want to prototype the following setup:

• Create a function with a predefined signature
• Register the function to a task, by listening to a specific event on the websocket
• The event triggers execution of the function (i.e. user presses 'Run' in task)

Additional enhancements could include automated registrations of scripts, KE-node style maybe?

Add bulk_edit of a part (part name and property values)

This might help me a lot in Award, if the edit of a part would also include it's property values in one call.

upload of matplotlib figure is not correctly linked to upload

@wilcoschoneveld

It seems that the upload of a matplotlib plot to an attachment field is not correctly linked. I guess we need to use the _upload() method on an attachment property.

add dictionary update function for part and its properties

if you have a part p and its properties p1 and p2 you might want to update the properties p1 and p2 from a dict {p1: v1, p2: v2}.

example code:

p = project.part('a_part')
p.update({'p1': 5, 'p2': 'hello marry lou'})

in pykechain pseudo code

for k,v in update_dict:
    try:
        p.property(k).value = v
    except:
        pass

Get the (direct) children / siblings of an activity

parts has a "children" and siblings function, activities does not but would be handy to have all the same.

standardize (and create guidelines for) pykechain architecture

Since pykechain is expanding rapidly, we might want to establish some guidelines regarding the architecture. This would also involve refactoring the current code to follow these guidelines.

Remarks regarding the codebase:

• The Client has a private _request method and is responsible for building requests (e.g. Client.activites and Client.create_part), however sometimes this is violated; e.g. Property.delete and Activity.configure build a request themselves.
• Most of the APIErrors do not include additional information (e.g. why could a new activity not be created?)

add ability to edit part instance names and description after creation

add feature to configure the activity using pykechain (association api)

• analog the the 'configure' of a task
• use the api to write the inputs, outputs configuration to an association

see testcode snippet from KEC2::

association1 = Association.objects.get(activity=activity1)
response = self.SUPERclient.put("/api/associations/{}".format(str(association1.id)),
                                        dict(inputs=[self.prop_model1.id, self.prop_model1.id],
                                             outputs=[self.prop_model1.id]))

you might want to use the /api/associations?activity=<> to get the association of an activity. Other way is to see if it is listed on an activity response itself.

GET /api/activities/6ff02812-5dee-4960-9807

{
    "results": [
        {
            "id": "6ff02812-5dee-4960-9807",
            ...
            "configuration": "3fb65cbf-cefe-43dd-ba68",
            "status": "OPEN",
           ...
            "association_id": "3fb65cbf-cefe-43dd-ba68",
            "activity_class": "UserTask",
            "widget_config": null
        }
    ]
}

This is either the configuration or the association_id (the same)

detach jupyter specific renderings

I am thinking about moving all _repr_html stuff to kenotebook, and inject the renderers on load. This makes jupyter notebook specific stuff independent of base pykechain.

use pagination of requests with default limit (100) to populate local PartSet

I get back from SVE that a full retrieval of some parts based on a model is taking too long to retrieve, beyond the limit of 30s timeout.

I though the following:

• insert limit filter (defaults to 100?)
• the resulting response contains a 'next' json key
• populate the partset and follow the 'next' key for as long as it does not provide None
• it may take long time > 30s but it can overcome the retrieval of large partsets

initiate session using tokenauth or basic auth and then the corresponding use session cookie

ref: KEC-16217

Observation:

"PyKechain uses requests package to make requests to Kechain. In the current version, no session information, a.k.a. cookie, is used. This results in an authentication of the user for every request is made.
[Suggestion]
We can change and add cookie information on the Client object of PyKechain.
PyKechain needs to request a cookie by logging in Kechain via ../accounts/login/?next=/ .
After logging in successfully, a cookie is saved in the Session object of requests.
Logically, this session cookie will be used by any further request as authentication proof."

Response:

Pykechain is using either basic auth or token auth and indeed each request is 'reauthenticating' itself.

It does not login using the webform/url

I have to see how we can emit a session cookie from the server on token/basic auth and resupply that cookie (the authentication will become session auth) in each request. Will investigate this further.

upgrade pykechain to use the bulk edit property api of KE-chain

I can request associated (= configured) models of a task, but not the associated properties

See http://pykechain.readthedocs.io/en/latest/api/models.html?highlight=assoc#pykechain.models.Activity.associated_parts

This gives me the associated models, but it's the properties I'm after. Also I'd like to see for each property if it's configured as View or Edit.

cc: @jberends

ability to create models using pykechain

• investigate and implement the ability to create models using pykechain

Ability to edit activities (similar to edit parts)

Activities can not yet be edited (name, desc, due dates etc)

Use helper objects to speed up customization of the activities (inspector NG)

some inspiration

class SuperGrid(object):
    parent = None
    model = None
    
    def __init__(self, parent, model):
        self.model = model
        self.parent = parent
    
    def render(self):
        return {'xtype':'SuperGrid', 'parent': self.parent, 'model':self.model}
    
class Custimisation(object):
    components_list = []
    
    def add_component(self, component):
        self.components_list.append(component)
        
    def render(self):
        return json.dumps({'components': self.components_list}, indent=4)

provide way to make requests to KE-chain in a parallel fashion (Thread pool)

We have a case that there is code that loops over the following usecase many times

1. create a part from a model under a parent
2. get the part and update its properties

we know the model_id and know the parent_id and have a list of dicts to update each part.

As pykechain is synchronously calling KEC we can speed up bulk creation of the parts dramatically by usuing requests in e.g. threadpool.

solution ways:

1. bulk create and update in pyke, with a threadpool of parallel requests using part.add and part.update methods in sequence
2. create the desired KEC api endpoint to bulk update a single part (detail route) or bulk create parts with their properties (list route).

As we cannot control KEC code from this repo we could provide the parallel connection solution path.

sources:

• https://github.com/kennethreitz/grequests
• http://docs.python-requests.org/en/master/user/advanced/#blocking-or-non-blocking

get the part from its property model does not work as we inject category=INSTANCE in the part retriever

Out[2]: 
<pyke Property 'Werkvak' id 699a9b9f>
project.part(pk='87d64829-8534-4a05-bbc0-7f587626f3d3')
Traceback (most recent call last):
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-0104b0b28d1c>", line 1, in <module>
    project.part(pk='87d64829-8534-4a05-bbc0-7f587626f3d3')
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\models\scope.py", line 33, in part
    return self._client.part(*args, bucket=self.bucket.get('id'), **kwargs)
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\client.py", line 345, in part
    raise NotFoundError("No part fits criteria")
pykechain.exceptions.NotFoundError: No part fits criteria
project.model(pk='87d64829-8534-4a05-bbc0-7f587626f3d3')
Out[4]: 
<pyke Part 'Keuringsformulier dwarsliggers' id 7626f3d3>
prop.part_id
Traceback (most recent call last):
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-0be2809e44f9>", line 1, in <module>
    prop.part_id
AttributeError: 'Property' object has no attribute 'part_id'
prop._json_data['part']
Out[6]: 
'87d64829-8534-4a05-bbc0-7f587626f3d3'
project.model(prop._json_data['part'])
Traceback (most recent call last):
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-c53c86791b85>", line 1, in <module>
    project.model(prop._json_data['part'])
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\models\scope.py", line 47, in model
    return self._client.model(*args, bucket=self.bucket.get('id'), **kwargs)
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\client.py", line 364, in model
    raise NotFoundError("No model fits criteria")
pykechain.exceptions.NotFoundError: No model fits criteria
project.model(pk=prop._json_data['part'])
Out[8]: 
<pyke Part 'Keuringsformulier dwarsliggers' id 7626f3d3>
project.part(model=project.model(pk=prop._json_data['part']))
Out[9]: 
<pyke Part 'Keuringsformulier dwarsliggers' id dd324f4d>

Introduce the option to sort properties under a part model

Fix the failing tests

• some tests are now failing and it's time for two brave men to save the world @jberends

Customize a task using pykechain

Use pykechain to customize a given task by feeding it the customization json. If the pyke developer handles this well, he can configure & customize many tasks in a row using the same customization blueprint

Add documentation for the reference property

• seen an backward incompatibility issue where in the past you provide the ID (not the part), also the ref-prop.value procvides a part instance back immediate (not a pk).

improve documentation to include an example of updating a single select property

a single select property model has an 'options' field in the API which is perculiar on the data it want to receive.

The options field content should be:

{value_choices: ["item2", "item3", "item1", "item 4"]}

We might want to add an example for this in the documentation

datetime is incorrectly recorded when providing tzinfo

Thanks @joost.schut for reporting this

form_task.edit(due_date=datetime.datetime(2017,6,5, 5,0,0))

Traceback (most recent call last):
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-60-94037dfba8f6>", line 1, in <module>
    form_task.edit(due_date=datetime.datetime(2017,6,5, 5,0,0))
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\models\activity.py", line 121, in edit
    "This date is interpreted as UTC time.".format(start_date.isoformat(sep=' ')))
AttributeError: 'NoneType' object has no attribute 'isoformat'

and

orm_task.edit(start_date=datetime.datetime(2017,6,7, 5,0,0))
C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\models\activity.py:111: UserWarning: The startdate '2017-06-07 05:00:00' is naive and not timezone aware, use tzinfo. This date is interpreted as UTC time.
  "This date is interpreted as UTC time.".format(start_date.isoformat(sep=' ')))
form_task.edit(start_date=datetime.datetime(2017,6,5, tzinfo=pytz.timezone('Europe/Amsterdam')))
form_task.edit(start_date=datetime.datetime(2017,6,5,0,0,0, tzinfo=pytz.timezone('Europe/Amsterdam')))
form_task.edit(due_date=datetime.datetime(2017,6,5,0,0,0, tzinfo=pytz.timezone('Europe/Amsterdam')))
form_task.edit(due_date=datetime.datetime(2017,6,5,0,0,0)))
  File "<ipython-input-70-aca48fc4cd5a>", line 1
    form_task.edit(due_date=datetime.datetime(2017,6,5,0,0,0)))
                                                              ^
SyntaxError: invalid syntax
form_task.edit(due_date=datetime.datetime(2017,6,5,0,0,0))
Traceback (most recent call last):
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-71-14185e8cc051>", line 1, in <module>
    form_task.edit(due_date=datetime.datetime(2017,6,5,0,0,0))
  File "C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\models\activity.py", line 121, in edit
    "This date is interpreted as UTC time.".format(start_date.isoformat(sep=' ')))
AttributeError: 'NoneType' object has no attribute 'isoformat'
form_task.edit(start_date=datetime.datetime(2017,6,5,0,0,0))
C:\Users\joost.schut\AppData\Local\Programs\Python\Python35\lib\site-packages\pykechain\models\activity.py:111: UserWarning: The startdate '2017-06-05 00:00:00' is naive and not timezone aware, use tzinfo. This date is interpreted as UTC time.
  "This date is interpreted as UTC time.".format(start_date.isoformat(sep=' ')))
form_task.edit(due_date=datetime.datetime(2017,6,5,5,0,0, tzinfo=pytz.timezone('Europe/Amsterdam')))
form_task.edit(due_date=datetime.datetime(2017,6,5,5,40,0, tzinfo=pytz.timezone('Europe/Amsterdam')))
form_task.edit(due_date=datetime.datetime(2017,6,5,5,20,0, tzinfo=pytz.timezone('Europe/Amsterdam')))

retrieve the model of a part instance

in the Part class

def model():
   # add check if instance, otherwise nothing
   the_model = get(self._json_data, 'model')
   return # the part that is the model (so do a retrieve to KEC)

• make a new branch from master called #77-model-of-part
• write code in part
• write tests
• do intermediate commits and push to github
• ensure a merge request is made from your branch

retrieve all models and parts related to an activity (activity_id)

For future needs we want to restrict parts and model updated restricted to the scope of an activity_id. For this we need all the parts that are 'configured' in an activity and available to work on.

Get the instances of a given model

Similar to the some_part.instance.model() method but the other way around (result as a PartSet I guess)

Add ability to empty a reference part property (no id selected)

Test framework is unstable and not easily extended

Requirements for testing:

• Running tests should be possible with and without a KE-chain server
• (ofcourse pykechain actual usage is limited to having access to a KE-chain server)
• Tests should cover over 90% of the pykechain code base, including functionality that interacts with a KE-chain

Current problems:

• Not all tests run with the pycharm test suite runner
• Sometimes 'betamax requests can not be found' errors when running tests (even though request is recorded)
• Scope name of test project is not consistent
• ... [in progress]

add ability to set reference part/property value

update the inspector components/widget to the new format

Add pyke support for the "Add part model from catalog" functionality in the KEC Data Model