rwth-ebc/filip

FIWARE Library for Python (FiLiP) to work with FIWARE API

FiLiP

FiLiP (Fiware Library for Python) is a python software development kit (SDK) for accelerating the development of web services that use Fiware's Generic Enablers (GEs) as backend.

It is mainly based on the Pydantic package which is a sophisticated library for data validation and settings management using python type annotations. Pydantic enforces type hints at runtime, and provides user friendly errors when data is invalid. We mainly use the Pydantic model to build our own data model structure required for efficient data model parsing and validation and interaction with FIWARE services' RestAPIs.

For API interaction, FiLiP relies on the well-known requests package. It is important to understand that we do not in any way restrict any features of requests.

Furthermore, FiLiP is designed to help with the fast development of FIWARE-based applications and avoid hundreds of lines of boilerplate, but it cannot substitute learning the basic concepts behind the used FIWARE components.

General Motivation

Why implement a client library when clients can be auto-generated from openapi documentation? A general prerequisite to do so is that the documentation is in depth and of good quality. While FIWARE generally provides openapi documentation, here are some thoughts on the challenges of auto-generating client code from these documents:

• Auto-generated code tends to become rather bulky and its quality strongly depends on the provided input data.
• Manipulating generated code can result in a big hassle for maintenance if additional features need to be integrated.
• The underlying NGSI (Next Generation Service Interface) for FIWARE is a rather generic specification. Hence, generated models may also be of generic types as lists and dicts in Python. So there is no real benefit. Furthermore, there is no chance for reasonable validation and error handling.

Getting started

The following section shortly describes how to use the library.

Prerequisites

Since FiLiP is designed as a client library, it requires a server that provides the target Service-APIs. Hence, if you do not yet have a running instance of a FIWARE based platform, using docker is the most convenient way to set it up. Please check here for a tutorial on this. If this is not an option for you, FIWARE also provides a testing server. You can register for a testing account here.

Note: FiLiP is now compatible to Pydantic V2. If your program still require Pydantic V1.x for some reason, please use release v0.2.5 or earlier version of FiLiP. Besides, we recommended to use pydantic~=1.10 in the requirements.txt

Installation

The easiest way to install the library is via pip:

pip install -U filip

If you want to benefit from the latest changes, use the following command (This will install the current master branch from this repository):

pip install -U git+git://github.com/RWTH-EBC/filip

Install semantics module (optional)

If you want to use the optional semantics module, use the following command (This will install the libraries that only required for the semantics module):

pip install -U filip[semantics]

Introduction to FIWARE

The following section introduces FIWARE. If you are already familiar with FIWARE, you can skip this section and go straight to Getting Started.

What is FIWARE?

FIWARE is a framework of open-source cloud platform components, created to facilitate the development of smart solutions within various application domains. At the moment, the FIWARE catalogue contains over 30 interoperable software modules, so-called Generic Enablers (GE) for developing and providing customized IoT platform solutions.

To get familiar with the APIs of the different modules we highly recommend checking the step-by-step tutorial. It provides a good overview on FIWARE and its basic usage. Whereas the tutorial helps to understand most of the general concepts, for a deep dive, where you can learn about the components in more detail, FIWARE also offers extended lessons through their academy.

However, usually one only requires a small set of components. Hence, we recommend using the cited pages only as needed.

How to set up a FIWARE platform?

The easiest way to set up a FIWARE platform is by using docker as all GEs are open-source and distributed as docker containers on dockerhub.

However, as mentioned before, for most use cases only a subset of GEs is required. Hence, we wrote a small tutorial explaining how to set up a platform suited for most use cases within the energy domain.

FIWARE GEs covered by FiLiP

FiLiP is a library developed on demand. Hence, we do not aim to cover the APIs of all GEs that are included in the catalogue. This would mean an unnecessary development overhead. Therefore, FiLiP currently only covers the APIs of the following GEs:

• NGSIv2 Context Broker for managing context data. We use its reference implementation ORION for testing.

  • documentation
  • github
  • swagger
  • NGSI v2 specifications

• IoT-Agents for managing IoT Devices. IoT agents are implemented using the FIWARE IoT Agent Node Lib as a common framework.

  • documentation
  • github

• IoT-Agent-JSON for managing devices using a JSON message payload protocol format.

  • documentation
  • github
  • apiary (partly deprecated)

  Example payload:

    {
        "humidity": "45%",
        "temperature": "23",
        "luminosity": "1570"
    }  
  

• IoT-Agent-Ultralight for managing devices using an Ultralight 2.0 message payload protocol.

  • documentation
  • github
  • apiary (partly deprecated)

  Example payload:

    humidity|45%|temperature|23|luminosity|1570
  

• QuantumLeap for the management of time series data

  • documentation
  • github
  • swagger

Structure of FiLiP

Documentation

We are still working on the documentation. You can find our current documentation here.

Running examples

Once you have installed the library, you can check the examples to learn how to use the different components.

Currently, we provide basic examples for the usage of FiLiP for the FIWARE GEs mentioned above. We suggest to start in the right order to first understand the configuration of clients. Afterwards, you can start modelling context data and interacting with the context broker and use its functionalities before you learn how to connect IoT Devices and store historic data.

Testing

We use unittests to write our test cases. To test the source code of the library in our CI workflow, the CI executes all tests located in the tests-directory and prefixed with test_ .

How to contribute

Please see our contribution guide for more details on how you can contribute to this project.

Authors

• Thomas Storek
• Junsong Du (corresponding)
• Saira Bano
• Sebastian Blechmann

Alumni

• Jeff Reding
• Felix Rehmann
• Daniel Nikolay

References

We presented or applied the library in the following publications:

• S. Blechmann, I. Sowa, M. H. Schraven, R. Streblow, D. Müller & A. Monti. Open source platform application for smart building and smart grid controls. Automation in Construction 145 (2023), 104622. ISSN: 0926-5805. https://doi.org/10.1016/j.autcon.2022.104622

• Haghgoo, M., Dognini, A., Storek, T., Plamanescu, R, Rahe, U., Gheorghe, S, Albu, M., Monti, A., Müller, D. (2021) A cloud-based service-oriented architecture to unlock smart energy services https://www.doi.org/10.1186/s42162-021-00143-x

• Baranski, M., Storek, T. P. B., Kümpel, A., Blechmann, S., Streblow, R., Müller, D. et al., (2020). National 5G Energy Hub : Application of the Open-Source Cloud Platform FIWARE for Future Energy Management Systems. https://doi.org/10.18154/RWTH-2020-07876

• T. Storek, J. Lohmöller, A. Kümpel, M. Baranski & D. Müller (2019). Application of the open-source cloud platform FIWARE for future building energy management systems. Journal of Physics: Conference Series, 1343, 12063. https://doi.org/10.1088/1742-6596/1343/1/012063

License

This project is licensed under the BSD License - see the LICENSE file for details.

Copyright

2021-2024, RWTH Aachen University, E.ON Energy Research Center, Institute for Energy Efficient Buildings and Indoor Climate

Institute for Energy Efficient Buildings and Indoor Climate (EBC)
E.ON Energy Research Center (E.ON ERC)
RWTH University Aachen, Germany

Disclaimer

This project is part of the cooperation between the RWTH Aachen University and the Research Centre Jülich.

Related projects

Acknowledgments

We gratefully acknowledge the financial support of the Federal Ministry
for Economic Affairs and Climate Action (BMWK), promotional references 03ET1495A, 03ET1551A, 0350018A, 03ET1561B.

This project has received funding in the framework of the joint programming initiative ERA-Net Smart Grids Plus, with support from the European Union’s Horizon 2020 research and innovation programme.

filip's Issues

Python Docstring: Raises missing

Most methods are missing the Raises part in the Docstring commands.

Some are informing about possible Error raises in the description, but not all.

I would suggest adding the Raises tag and detail information to all methods that could raise an error. With this it is directly clear to the user if he needs to handle errors and which types to expect.

Complete device update

Is your feature request related to a problem? Please describe.
When using update_device to push made changes in a device to Fiware, the device in the iota client is correctly updated, but the corresponding entity is not. Here only new attributes are added, but removed attributes are not deleted and changed values are not updated. Further changed device_settings as transport are not taken over to Fiware.

Currently the only reliable way to take over all changes is to delete the device and the corresponding entity and to repost them.

Describe the solution you'd like
To make the interaction here much cleaner, I propose the method patch_device (linked to Issue 21), that takes a device model object and makes all the needed updates for the user to take over all made changes to Fiware. It will internaly delete and repost the device only if the main device settings (endpoint,..) were changed, else it will use the methods update_device and patch_entity to update the stats.

This issue will also provide the tests to ensure the correctness of the functions (and a test for update_device, that is still missing). Further a few simple quality of live methods as does_device_exist are introduced.

Add MQTT notifications to CB_client

Is your feature request related to a problem? Please describe.
With the new version of the Orion Context Broker notifications via MQTT are supported.
https://fiware-orion.readthedocs.io/en/3.2.0/user/mqtt_notifications/index.html

Describe the solution you'd like
We want to add this functionality in our notifications models.
Furthermore, we need to introduce the deprecated query parameter for skipping the initial notification for backward compatibility. If set the the client will check the version of the context broker first and then return a warning to the user.

Describe alternatives you've considered
None

Ontologies to Semantic

Describe the solution you'd like
Take the semantic logic that is currently situated in the SENSE fronted and transfer it over into Filip.
Further this logic needs to be transformed to better synergies with Filip entities.

Instead of the currently used instances that are only transformed to Filip entities when saving the state, a wrapper will be written to simply extend the entities with the needed semantic functionalities.

To make the storage and reparsing of the semantic vocabulary much easier the ontology files will be saved inside the pydantic vocabulary object, which can be stored as json.

If this feature is finished a user can provide a semantic structure to Filip by giving ontologies, and use this structure to create models, connect them and model a state.

QuantumLeap request pagination

Is your feature request related to a problem? Please describe.
QuantumLeap API implements a pagination mechanism in order to help clients to retrieve large sets of resources. (https://quantumleap.readthedocs.io/en/latest/user/pagination/)
However, this mechanism has not yet been implemented in the QuantumLeap client (see at https://github.com/RWTH-EBC/FiLiP/blob/development/filip/clients/ngsi_v2/quantumleap.py). Hence, the maximum number of results is limited to 10000 for a single request.

Describe the solution you'd like
The pagination mechanism is somewhat similar to the one already implemented for the context broker, compare https://github.com/RWTH-EBC/FiLiP/blob/development/filip/clients/ngsi_v2/cb.py (and also see https://fiware-orion.readthedocs.io/en/master/user/pagination/index.html). So, the solution should follow a similar approach.
However, the response is different, which results in a few changes, for instance, the json objects cannot be extended but rather the resulting data lists should be extended.
Please find a suggestion below:

def __pagination(*,
                method: PaginationMethod = PaginationMethod.GET,
                url: str,
                headers: Dict,
                limit: Union[PositiveInt, PositiveFloat] = None,
                offset: int = None,
                session: requests.Session = None,
                params: Dict = None,
                data: str = None) -> List[Dict]:
        """
        NGSIv2 implements a pagination mechanism in order to help clients to
        retrieve large sets of resources. This mechanism works for all listing
        operations in the API (e.g. GET /v2/entities, GET /v2/subscriptions,
        POST /v2/op/query, etc.). This function helps getting datasets that are
        larger than the limit for the different GET operations.
        https://fiware-orion.readthedocs.io/en/master/user/pagination/index.html
        Args:
            url: Information about the url, obtained from the original function
            headers: The headers from the original function
            params:
            limit:
        Returns:
            object:
        """
        if limit is None:
            limit = inf
        if limit > 10000:
            params['limit'] = 10000  # maximum items per request
        else:
            params['limit'] = limit
        if offset is None:
            params['offset'] = 0
        else:
            params['offset'] = offset
        additional_offset = 0
        if not session:
            session = requests.Session()
        with session:
            res = session.request(method=method,
                                  url=url,
                                  params=params,
                                  headers=headers,
                                  data=data)
            if res.ok:
                items = res.json()
                # do pagination
                count = int(res.headers['Content-Length'])
                attributes = params['attrs']

                # fresh start
                items['index'] = []
                items['attributes'][0]['values'] = []

                while len(items['index']) < limit and len(items['index']) < count:
                    try:
                        # Establishing the offset from where entities are retrieved
                        for attr in attributes:
                            params['attrs'] = attr
                            params['offset'] = len(items['index']) + additional_offset
                            params['limit'] = min(10000, (limit - len(items['index'])))
                            res = session.request(method=method,
                                                  url=url,
                                                  params=params,
                                                  headers=headers,
                                                  data=data)
                            if res.ok:
                                # items.extend(res.json()) --> this does not work for the given response structure
                                if len(items['attributes']) == len(attributes): # update entries
                                    i = find(lst=items['attributes'],key=attr)
                                    items['attributes'][i]['values'].extend(res.json()['attributes'][0]['values'])
                                elif len(items['attributes']) < len(attributes):
                                    items['attributes'].extend(res.json()['attributes'])
                                else:
                                    logger.error('Inconsistent number of attributes %i', len(items['attributes']))
                            else:
                                res.raise_for_status()
                        items['index'].extend(res.json()['index'])
                    except Exception as e:
                        if '404 Client Error' in e.args[0]:
                            # no records found, skip period
                            limit -= 10000
                            additional_offset += 10000
                        else:
                            print(e)
                    if not (pytz.utc.localize(dateutil.parser.parse(params['toDate']))>
                        dateutil.parser.parse(items['index'][-1])+datetime.timedelta(seconds=1)):
                        break
                logger.debug('Received: %s', items)
                items['entity_id'] = params['entity_id']
                items['entity_type'] = params['entity_type']
                ts = parse_obj_as(TimeSeries, items)
                return ts
            res.raise_for_status()

Describe alternatives you've considered
The obvious alternative is to individually evoke single requests and post-process the results to the desired format.

Additional context
Beside the different response structure, when there are no records within a period, the function is stuck since the request will fail and neither the items nor the limit will be updated. This exception has to be caught. Note that in the above example, the request of several attributes has been considered.

fix data package

Describe the bug
Apparently there are still errors when installing filip

To Reproduce
Steps to reproduce the behavior:

1. pip install .
2. pip uninstall filip
   The first does not copy the hdf file properly
   The second one would leave fragments

Expected behavior
Both commands should work

Issue saving complex information in StaticDeviceAttributes

There are some issues with adding complex information (with Enums) to value fields in DeviceAttributes. The here describe issue could also be happening for Context Attribute, this was not yet tested. Below are two approches that I took to save a complex Basemodel in a value field of a StaticDeviceAttribute, both with there own issues.

class DeviceAttribute(Basemodel):
	name: str
    attribute_type: DeviceAttributeType

if we have a DeviceAttribute da, we can export the model in 2 forms:

da.dict()
	=> {'name': 'd1', 'attribute_type': <DeviceAttributeType.lazy: 'lazy'>}
da.json()
	=> {"name": "d1", "attribute_type": "lazy"}

If we add the da export to a values array:

v1 = values.append(da.dict()) 
	=> [{'name': 'd1', 'attribute_type': <DeviceAttributeType.lazy: 'lazy'>}]
v2 = values.append(da.json()) 
	=> ['{"name": "d1", "attribute_type": "lazy"}'] 

If we add the values array to a StaticDeviceAttribute:

attr = iot.StaticDeviceAttribute(
		name="attributeName",
		type=DataType.STRUCTUREDVALUE,
		value= vN,
		entity_name=None,
		entity_type=None,
		reverse=None,
		expression=None,
		metadata=None
	))

With vN = v1:
attr => name='attributeName' ... value={'name': 'attribute_type'}
With vN = v2:
attr => name='attributeName' ... value=['{"name": "d1", "attribute_type": "lazy"}']

The dict() variant gets strangely transformed.
The json() variant does look correct and can be posted without error, but the device containing this attribute and its corresponding context entity will not appear in Fiware.

Currently I can not think of a reason for any of those two.

make loading units a function

Currently, units are automatically loaded on import.
This needs to change in order to make FiLiP PyPy ready.
Erase old code fragments.

Device BaseAttribute; entity_name and entity_type

Describe the bug
The description of the field states:

"entity_name: the presence of this attribute indicates "
"that the value will not be stored in the original device "
"entity but in a new entity with an ID given by this "
"attribute. The type of this additional entity can be "
"configured with the entity_type attribute. If no type is "
"configured, the device entity type is used instead. "

But if a value other than None is given (see other issue: we need to always state a value, even if we want None) the Attribute does not appear in the context broker.

If the entity_id and type are equal to the device info, it does not appear in the device (this is an edge case, and could be an expected behaviour)
But if some other value is given no entity is created with the given id and type, thus the attribute is not present in the context broker.

To Reproduce
Steps to reproduce the behavior:

1. Create a device object
2. Add some Attribute to it, with entity_name and entitiy_type no null
3. post_device

Expected behavior
A new entity with (entity_name and entity_type) is present in the ContextBroker and holds the attribute

Broken test for time series

Describe the bug
The tests for time series leave old subscriptions in orion.
Running multiple test over time will let them pile up.

To Reproduce
Steps to reproduce the behavior:

1. Run tests for time series several times
2. check for subscriptions in the context broker

Expected behavior
Subscriptions should be deleted after testing

Solutions
Use teardown at the end of the test case this will delete old subscriptions even if the test fails

Semantics: Better load function interface

This issue will make the search for existing semantic objects and their loading more flexible by including more of the query parameters that get_entity_list() offers in load_instances_from_fiware().

Missing Tests for DeviceModels

The current test landscape leaves a lot untested for the DeviceModels.

New tests need to be created to cover all details in the device model creation process and if the created models can be correctly transferred to Fiware.

paho-mqtt twice in requirements.txt

Describe the bug
double definition of paho-mqtt in requirements.txt

DeviceAttribute; @validator('metadata')

Describe the bug
The validator of the DeviceAttribute Model in iot.py, should accept:
Optional[Dict[str, Union[Dict, str]]]

But it throws an error on the values: None, {"test", "testing"}. Both values are correct and can be successfully treated by the post_device methode.

To Reproduce
Steps to reproduce the behavior:

1. Create a new DeviceAttribute object with one of the stated values in the metadata field.

Expected behavior
No Error

make contribution guid more visiable

currently the contribution.md is locating in docs we want to move it on top level

better cleanup

Is your feature request related to a problem? Please describe.
Currently, the clean up fuction do not check weather the data is realy deleted. Hence, this should be checked instead of a time.sleep()

Describe the solution you'd like
Substitute time.sleep() with get_requests on list level.

Describe alternatives you've considered
None

Running Tests in CI

Creating and testing a github action to run all tests automaticaly on a PUSH.

automatic issue branch creation

Is your feature request related to a problem? Please describe.
I'm always frustated when I have to create an issue/feature branch by myself

Describe the solution you'd like
automatic branch creation with: robvanderleek/create-issue-branch

simple_ql: QueryStatement.parse_str regex

Describe the bug
Parsing of queries containing special characters in attribute name returns None.
This behavior is caused by the used regular expression rf"^\w(\w*|\.(?=\w))*{op}\w*"

To Reproduce
QueryStatement.parse_str('diffuse-radiation!=\'123\'') returns None

Expected behavior
As described in the Specification, attribute names may include all special characters except control characters, whitespace, &, ?, / and #. Therefore the statement above should return a valid QueryStatement.

Cleanup

Removing the author and creation date from the module docstrings.
Also deleted files that are no longer needed.

StaticDeviceAttribute Metadata Type

Describe the bug
Currently the StaticDevice metadata field allows the type Optional[Dict[str, Dict]]. But Fiware only allows a very specific Dict structure. Giving any other structure results in a success in Filip, but an uncatched background error in Fiware.

To Reproduce
Steps to reproduce the behavior:

1. Create an ContextDevice containing a StaticDeviceAttribute with the metadata: {"meta": { "test": "1" }}
2. Filip allows this assignment, you can now post the device without error
3. But if you now look in Fiware, no device or corresponding ContextEntity appeared.

Expected behavior
FiLiP tells you, that the assignment of the metadata was not valid, and stops the execution

Fix
Change the type of the StaticDevice metadata field to a BaseModel modeling the allowed structure. The ContextMetadata and NamedContextMetadata modeld in context.py, can probably be reused for this task. The StaticDeviceAttribute metadata field would then have the same structure and logic as the ContextAttribute metadata field.

Requests-Bugfixes

The method post_request does not work, as the http field needs to be in json form:

Current body:
{"provider": {"http": "http://localhost:1234"}, "dataProvided": ...}

Required body:
{"provider": {"http": {"url": "http://localhost:1234" }, ...}

Further this issue will test if existing registrations are correctly removed (or edited) if the corresponding entity is deleted

Field "expressionLanguage" in Device Model is not correct

The Field "expressionLanguage" in Device Model is not correct

• It is states in the description that it is a boolean value, and that it has the possible values legacy or jexl. And is of type ExpressionLangauge.

• It is described as an Optional Field, with the default value of legacy. But it has no default value, as a result the user needs to explicitly state the value of this field for each Device that he creates

  expressionLanguage: Optional[ExpressionLanguage] = Field(
  description="optional boolean value, to set expression language used "
  "to compute expressions, possible values are: "
  "legacy or jexl. When not set or wrongly set, legacy "
  "is used as default value."
  )

Tests for subscriptions are failing

Describe the bug
Two tests are failing in the pipeline

To Reproduce
Run the test test_subscriptions and test_cb_subscriptions

IRI as Datatype

When using Filip for the Cloud Sense Semantic App, each Entity and each Entitiyattribute has an entry that states the IRI this object/attribute is symbolising. This can be done with Datatype.Text, but to better symbolize the meaning of these fields an new Datatype coulb be introduced. Due to the fact that it is not a random char sequence but an concrete identifier of the semantic web.

Refactor Examples

Refactoring all examples to match structure of semantics examples, achieving a coherent style and the ability to parse them all to Jupyter notebooks

Adding new QL endpoints

Add endpoints /v2/attrs and /v2/attrs/
(this issue is copied from https://git.rwth-aachen.de/EBC/Team_BA/projects/n5geh/tools/n5geh.tools.filip/-/issues/46)

Couldn't update subscription with default values

Describe the bug
Method update_subscription of filip.clients.ngsi_v2.ContextBrokerClient has following configuration for request serialization:

res = self.patch(
                url=url,
                headers=headers,
                data=subscription.json(exclude={'id'},
                                       exclude_unset=True,
                                       exclude_defaults=True,
                                       exclude_none=True))

If a subscription is created/updated with non default values you can never re-set the default ones because defaults are excluded from serialization.

To Reproduce
Steps to reproduce the behavior:

As an example try to update the status of a subscription:

        with ContextBrokerClient(url='http://localhost:1026',
                                 fiware_header=FiwareHeader(
                                     service='fw-service',
                                     service_path='/testing',
                                 )) \
                as cb_client:

            subscription = cb_client.get_subscription('xyz')
            subscription.status = Status.INACTIVE

            cb_client.update_subscription(subscription)

→ subscription updates status to "inactive"

            subscription.status = Status.ACTIVE
            cb_client.update_subscription(subscription)

→ subscription still has status "inactive"

Environment (please complete the following information):

• OS: Windows 10
• Version 20H2
• Python 3.8

remove unnecessary submodules

currently the repo contains the specifications as submodules. However, this is not necessary and only makes the repo unnessary large.

adding mqtt client

we like to add an mqtt client that can be directly used with the iotagent-ul or iotagent-json.
The client will be placed in the clients.mqtt

Here is the orginal discussion on the module:

https://git.rwth-aachen.de/EBC/EBC_all/fiware/filip/-/issues/45

The client should especially simplify the topic subscription patterns for the mqtt interface.
This is not trivial in FIWARE:

https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/api.md#relationship-between-service-groups-and-devices

And also part of current discussions:

https://github.com/telefonicaid/iotagent-json/issues/585

substitute fuzzywuzzy with rapidfuzz

Due to Licence issues, we should substitute the fuzzywuzzy package used for unit validation with RapidFuzz.

https://github.com/maxbachmann/RapidFuzz

The usage of RapidFuzz should work in very similar way.

PATCH request remains in PENDING state / handling of cmdexe

What is the issue?

When sending a post_command request to the cb and querying the entity state, the set point status may change to PENDING and will not respond to any further requests.

Proposal

The issue might be related to the cmdexe (see https://github.com/FIWARE/tutorials.IoT-Agent-JSON#southbound-traffic-commands) and the cb expecting to receive an acknowledgement from the device.
As many device might not offer an acknowledgement, this should be optional/settable by the user.
If applied, the cmdexe handling still requires implementation.

hints for unit validation

Currently, unit validation is very strict. Therefore, we want to add hints to the user in order to find the right unit.

• write a try and except block that catches the key error from the unit-tables
• use fuzzywuzzy to generate and provide appropriate error message and hints

Semantics: Flexible model generation output

Currently the generate_vocabulary_models always produces a python file with the given path and filename.

To be more flexible with its usage, it should only save the models if both path and filename are given, else it returns the file-content as string.

This opens the possibilities for easier transference of models to a backend server. It is an optional feature which does not impact any currently implemented logic.

client log does not output messages

Apparently, there is a minor bug in the client log function. It checks for the wrong existence of the response objekt.

if err.response is not None:
    ....output message

Missing "autoprovision" in "ServiceGroup" in models/ngsi_v2/iot.py

Fiware service returns path

This line returns the fiware service path but it needs to return the fiware service

FiLiP/filip/clients/base_http_client.py

Line 110 in 57d72eb

return self.fiware_headers.service_path

Support for python 3.9

Is your feature request related to a problem? Please describe.
Support python 3.9 by dropping support for PyTables.

Describe the solution you'd like
Make tables support optional.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
PyTables/PyTables#823

move back to simplier docs design

currently, pydantic models are not rendered very well in the material design. Therefore, we want to move back to the readthedocs design

Post_subscription prevent duplicates

Prevent that post_subscription (ngsi_v2.cb.py) creates a duplicate entry.

The methode shall now check, if the exact same subscription already exists.
If yes, it makes a log entry and returns the ID of the already created subscription

Dont set the basicConfig of the logging module

You are doing

logging.basicConfig(level=settings.LOGLEVEL,
                    format='%(asctime)s - FiLiP.%(name)s - %(levelname)s: %('
                           'message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

in config.py.
This way, you change the settings for all other modules as well.
the LOGLEVL should be set by the user, not you.

Deleting complet state of an object

Is your feature request related to a problem? Please describe.
If the user creates a device a context entity is automaticaly created. But when deleting a device the entity is not automaticaly removed. Same if the linked entity is removed. This can be quite confussing for the user, as it is behaviour that is not suspected, and can lead to issues, as remenants of objects that are thought to be deleted still contiune existing.

Describe the solution you'd like
A parameter option for the delete methods, that allows the user to clear the whole state of the object with one methode call.
Even if the user does not use this optional parameter, it makes aware that the methode does not normally cleans the whole state.

Describe alternatives you've considered
None

Refactor test environments

• Removing all the now unneeded .env.fillip
• Adding a new .env to be used for local development
• Adapting tests that rely on the additional config files

Regexes of Fiware_safe fields are allowing spaces

The used regex:

regex="^((?![?&#/])[\x00-\x7F])*$" # Make it FIWARE-Safe"

does not catch the use of " " in these strings, but FIWARE does not allow spaces in the protected fields.

improve documentaton

Is your feature request related to a problem? Please describe.
Currently the docs are far from perfect. we want to improve them for new users.

Describe the solution you'd like
Refactor the docs and add more examples

ContextEntity: Get Commands

Extend the ContextEntity model with methods to easily access the existing commands and their additional information (status, info)

Semantic Instances: Name and Comment

All semantic instance should hold two variables: name, comment that the user can fill out to keep a better overview of the instances.

This feature is mainly inspired by the needs of the SENSE app, but it can be a useful tool for each instance organistation.
And as it is optional, it does not really disturb if it is not used (only an unused additional field in the CB).

Patch_entity and Patch_device.

Sibling methods to post_entity and post_device.

These methods shall allow the user to readout an entity/device from Fiware, edit the object in any way they want and then publish the made changes (while keeping as much of the current Fiware state) to Fiware.

The PATCH Method introduces a simpler and more compressed interface for the user then the currently used UPDATE.
UPDATE has following issues:

• It does not add new metadata to the fields
• Does only append fields, does not deleted removed ones
• The user does need to know if the entity already exists
• Overwrites current state of fields, even if they were not changed locally in the entity

PATCH allows the user a simpler usage flow.

1. He downloads a few entities
2. Deletes/Adds a few fields
3. Changes values
4. Creates new entities

With Patch, he can now simply call the patch(entity) method for each entity that he has and does not need to differentiate between cases(new entity, field deleted, ...).

Additionally he can provide the entity_state before his editions to the method to only update the fields that he actively changed, thus he does not override the complete Fiware state of the entity, if there is concurrent access.

Test clean up, uses wrong method to clean entities

Describe the bug
Certain tests leave artifacts in the Fiware contextbroker after their clean up.
The clean uses the .update(action_type="delete") function which only deletes the attributes of the entity if it has some, leaving an entity entry with id an type in the broker.

Solution
Replace in clean up function:
client.update(entities=entities, action_type='delete')
with
for entity in entities: client.delete_entity(entity_id=entity.id, entity_type=entity.type)

Introduce concurrent testing

Is your feature request related to a problem? Please describe.
Currently, concurrent testing will automatically interfere with each other because the servicepaths are static. Therefore, the tests will all use the same tenants which will result in failing tests, if they try to access the same resources.

Describe the solution you'd like
The idea is to introduce a TestingSettings-class that will check for a local environment for its variables. If a *.env. is not available it will try to use environment variables. These variables should be set in the CI because the CI should not contain a file with information about the used servers. The servicepath should then be either autogenerated or set in the first or second way.