Set correct owner when creating entities

Improve BDD tests, should be faster and easier to write tests

Given there exist a package "TestData"
    And "TestData" contains blueprint "ItemType"
    And "ItemType" has attribute "list" of type "string"
    And "ItemType" is array

Scenario: Inherited properties should be added and overwrite parent properties
Given there exist a blueprint "Role" with attributes "id, name, description"
And there exists a blueprint "Employee" with attributes "id, name, roles"
And there exists a blueprint "Programmer" with attributes "id, name, skills"
When the domain expert extends the blueprint "Programmer" with the "Employee" blueprint
Then attributes of the "Employee" blueprint are added and overwrites attributes of the "Programmer" blueprint
And so attributes of the blueprint "Programmer" becomes "id, name, skills, roles:Role"

Update Search: Only return documents with 'read' privileges

Access control need to be enforced on the result of the search

No document found

Try to click on SverdrupAnchorReplace2021 operation.

Deploy to Azure

Se end for tips on how to deploy after a pull request is merged: https://github.community/t5/GitHub-Actions/How-to-make-PHPUnit-test-and-then-if-successfully-passed-deploy/td-p/32479

Support some storageAttributes on a StorageRecipe level

not contained

Feature: Enhance search to allow sorting

Implement a feature to enable users to sort search results by properties.

Single DataSource should support different Repositories

View an extended blueprint

Scenario: View a blueprint that inherit from another blueprint 
Given there exist a blueprint "Role" with properties "id, name, description"
And there exists a blueprint "Employee" with properties "id, name, roles:Role"´
And there exists a blueprint "Programmer" with properties "id, name, skills"
And the "Programmer" blueprint extends "Employee" blueprint 
When the domain expert views the "Programmer" blueprint 
Then the attribute values shown are "id, name, skills" # It should not shown the attributes of Employee blueprint
And extends information is shown, that "Programmer" extends "Employee"

View an extended entity

Scenario: View an instance with a blueprint type that inherit from another blueprint 
Given there exist a blueprint "Role" with properties "id, name, description"
And there exists a blueprint "Employee" with properties "id, name, roles:Role"
And there exists a blueprint "Programmer" with properties "id, name, skills"
And the "Programmer" blueprint extends "Employee" blueprint 
And there exist an instance of type "Programmer" with name "Dennis Ritchie"
When the domain expert views the instance with name "Dennis Ritchie"
Then the attribute values shown are "id, name, skills, roles: Role".

Enable DMT w/o Equinor account

Index Endpoint with filter ( get only Blueprints)

Cleanup backend project structure

Cleanup project structure to match the clean architecture, e.g. remove core folder
See if it makes sense to divide things under data, domain, and presentation @eoaksnes
Remove files not needed
Use dependency injection everywhere. Seems like this is not the case right now, where things are instantiated in the wrong layers. GetDocumentUseCase seems to instantiate DocumentService instead of passing in.

Blobs 2.0

Uploading(Updating) and Deleting of blobs does not work too well now.

We should come up with a better design for this.

Needs to function properly if we'r going to use it for STasks.

Access Control: How to restrict "datasource root level access" e.g. create a rootPackage?

Authentication in DMT/DMSS

Identity Providers

Option 1: One per deployment

Config in WebApp and API (plugin? Or only oauth2?)

Option 2: One per dataSource

No config in webApp
Login to dataSource (each datasource has own oauth2 config)
Store tokens per dataSource client side

Access Control

Requirements

Read/Write access level per root package (idealy per file)
Different institutions can host their own data sources and DMSS instance
DMSS instances can share their data sources, and "hook up" to data sources owned by other institutions

Option 1: Repositories handle JWT

Can be fleshed out to fully support Oauth2 on-behalf-of flow

+ Can use users credential on the repository (Openworks etc.)
+ No need to keep storage credentials in DMSS
+ Arguably safer. Less privilege, and uses well-known-auth-schemes

- On-behalf-of will be unusably slow withouth a token cache
- Complicated, many moving parts
- Heavily dependent on repository support, or we will need to create wrappers
- Many different technologies (read only mongo vs azure blobs)
- Entities split over multiple repositories
- Hard to maintain and setup (lots of manual work)
- Hard to implement

Option 2: Data source handle auth per file in lookup table

Set a default "umask" for each datasource
Unix like (owner, groups, other --> ReadWrite)
Possible to support root-package level access control for additional cost (rather inherit when created)

+ Minimal performance cost (we already have to lookup each file in the lookup table)
+ Confined and managed within the DMSS system
+ Not reliant on other technologies

- Repository credentials are shared across users (possible to setup separate DS cred per user)
- Repository credentials stored in DMSS

Access	Level
Write	2
Read	1
None	0

DataSource select repository based on StorageRecipe

Avoid run github workflows - pypi and docker registry

These worksflows should only run after new releases, need to adjust regexp in workflow settings.

Move DMT export to DMSS

DMT should not duplicate complicated document_service.save() function

Add authentication to api

https://github.com/Azure-Samples/ms-identity-python-webapp

uses b2c so external user can get access.

the api is public, available here:
https://api-dmss-dev.playground.radix.equinor.com/api/v1/data-sources

bug: blobs does not get deleted

"gridfs.errors.FileExists: file with _id '4ec0d646-907c-4eba-9e65-24106236d61b' already exists"

Recreate:
Import an entity with a blob with a set _id.
Delete the package and import the same entity.

On import change default ACL policy to be more restrictive.

Feat: "Type inheritance"

Deleting documents should not fail on missing blueprints or missing references

bug: Shared global user_context does not work well with multiple concurrent API calls

With concurrent API calls, the context gets set and unset with conflicts.
Need to rewrite to keep user context in the request object.

Bug: Search does not return all matches when multiple entities with the same name exists

As reported by @babakommani , when multiple entities (from the same Blueprint) with an identical name exist, the search function only returns one match in the result.

Support nested uncontained

The save function does not pick up on a second level uncontained entity.

bug: data source not found in default-form UI plugin (edit). Perhaps this is connected to data source alias?

how to produce:

right click and create new UI recipe for a blueprint
open the newly created UI recipe and go to edit tab

Add search

Move logic from DMT repository to here for search.

Feature: No _id should be visible to user

This is a meta issue, encapsulating several issues.
The goal is to have all pointers and identifiers to documents/entities use a pathlike string.
This is necessary for the API to be friendlier to work with, have functional cross-data-source-references, be able to dump/import a large package with both contained and uncontained entities with ease. And the general "human-friendlyness" of the system.

These should be completed in the given order;

Remove the "package.content" abstraction where we hide the "content" object
TreeNode should be able to generate a path to itself
DMT Index should set objects paths
DMSS API should work with (also handle) paths
Entity references should be a path

Support attribute in update document

This parameter is missing

Switch to FastAPI

Change to FastAPI with minimal changes
Request validation
OpenAPI refactoring and cleanup

update ACL recursively

Bug: Overriding type when creating new child does not check for valid types.

Should be a check on frontend and backend
To validate on frontend, we must expose the "valid_subtype?()" function in an endpoint

Authorize on every DataSource operation

Remove "description" as a required value?

Does it make sense to have "description" as a core attribute in our code? I think not, and that it should be removed from any places where it's hard coded, and kept only as a normal attribute (often inherited from NamedEntity)

bug: "insert reference" does not fail when the referenced type is wrong, it should

DMSS: Set and read ACL

Pre-commit hooks

Cleanup and check that all pre-commit works as expected after the move from old repository.

New Endpoint: Set ACL

Improve api documentation

The api documentation in the current state does give an overview of all endpoints and code snippets of how to communicate, which is pretty neat.

Actual usage of the api does require special knowledge of most of the endpoints.

We should add:

description of what the endpoint do
what are valid input?
what is the output? this one is less important, but "returns entity of an blueprint" would be sufficent in some cases.

Add extends to core Blueprint.json schema
Add extends to the Blueprint class. self.extended_attributes: List[BlueprintAttribute] =
Update get_blueprint_cached function to add extended attributes (recursive function) to the returned blueprint. Alternative adds parent link recursively until the parent is found. self.parent : Blueprint = get_blueprint_cached(self.extends)
Update get_none_primitive_types, get_primitive_types inside Blueprint class to return extended attributes

Scenario: Inherited properties should be added and overwrite parent properties
Given there exist a blueprint "Role" with attributes "id, name, description"
And there exists a blueprint "Employee" with attributes "id, name, roles"
And there exists a blueprint "Programmer" with attributes "id, name, skills"
When the domain expert extends the blueprint "Programmer" with the "Employee" blueprint
Then attributes of the "Employee" blueprint are added and overwrites attributes of the "Programmer" blueprint
And so attributes of the blueprint "Programmer" becomes "id, name, skills, roles:Role"

Delete schema.py and code generator code

Delete schema.py/code generator code

equinor / data-modelling-storage-service Goto Github PK

data-modelling-storage-service's People

Contributors

Stargazers

Watchers

data-modelling-storage-service's Issues