netsach/concrete-datastore

A Highly Versatile REST Datastore built on top of Django

README

Qualité du code

Description

concrete-datastore est un Datastore HTTP REST très polyvalent basé sur le framework Web Django.

Il est principalement utilisé comme serveur de base de données HTTP pour une application Web monopage (AWM).

Contrairement à un serveur de base de données classique tel que PostgreSQL ou MySQL où les requêtes sont effectuées à l'aide du langage SQL, chaque opération est effectuée à l'aide de requêtes HTTP simples.

concrete-datastore peut être considéré comme un serveur NoSQL ou comme une alternative à Firebase.

Démarrage rapide

git clone https://github.com/Netsach/concrete-datastore.git
cd concrete-datastore
docker run --name postgres-concrete-datastore -e POSTGRES_DB=db-concrete-datastore -e POSTGRES_USER=user-concrete-datastore -e POSTGRES_PASSWORD=pwd-concrete-datastore -d -p XXXX:5432 postgis/postgis:12-master
export POSTGRES_PORT=XXXX
export DATAMODEL_FILE=./docs/assets/sample-datamodel.yml
python3 -m venv env
source env/bin/activate
pip install -e ".[full]"
concrete-datastore makemigrations
concrete-datastore migrate
concrete-datastore createsuperuser
concrete-datastore runserver

Naviguez maintenant jusqu'à http://127.0.0.1:8000/concrete-datastore-admin/

Vous pouvez maintenant créer un jeton pour utiliser l'API (ou utiliser le Endpoint d'authentification).

Fonctionnalités

concrete-datastore est livré avec de nombreuses fonctionnalités intégrées telles que :

• Gestion des utilisateurs et des autorisations
• Backoffice généré automatiquement pour les administrateurs
• API entièrement REST utilisant JSON comme format de sérialisation
• Génération de statistiques simples
• Capacités d'envoi d'e-mails à partir de l'API
• ...

Comment ça marche ?

Afin de décrire le schéma de la base de données, le développeur doit écrire un fichier datamodel en YAML ou JSON. Ce fichier datamodel permet à concrete-datastore de gérer la base de données sous-jacente à l'aide de PostgreSQL.

Chaque demande d'API est contrôlée par ce fichier datamodel car il agit comme une spécification de ce qui se trouve dans la base de données et de ce qui devrait être autorisé par chaque utilisateur.

Vous pouvez créer manuellement le fichier datamodel en suivant les exemples et la documentation ou utiliser l'éditeur en ligne platform.concrete-datastore

F.A.Q

Si vous avez des questions, elles ont peut-être déjà été répondues dans le FAQS.md

Documentation officielle

Voir la documentation officielle

Version Anglaise

Vous pouvez retrouver la version Anglaise ici

concrete-datastore's Issues

Fix UnsupportedMediaType

We are getting an error while dispatch on the API. The error received is an error 500 with a sentry.
When the error UnsupportedMediaType is raised, we have to return a response 400.

UnsupportedMediaType: Unsupported media type "" in request.
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "concrete_datastore/api/v1/views.py", line 1908, in dispatch
    data = self.request.data
  File "rest_framework/request.py", line 216, in data
    self._load_data_and_files()
  File "rest_framework/request.py", line 279, in _load_data_and_files
    self._data, self._files = self._parse()
  File "rest_framework/request.py", line 351, in _parse
    raise exceptions.UnsupportedMediaType(media_type)

Provide endpoint to download calendar events

An endpoint should be created to be able to transform an instance in a calendar event. The calendar event should be in the iCalendar (.ics) format to be directly imported into different calendars, e.g. Outlook. The format looks as follows:

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Example Corp.//CalDAV Client//EN
BEGIN:VEVENT
UID:[email protected]
DTSTAMP:20200516T060000Z
DTSTART:20200517T060000Z
DTEND:20200517T230000Z
RRULE:FREQ=YEARLY
SUMMARY:Do the needful
END:VEVENT
END:VCALENDAR

The endpoint can either be called for all active instances like that, where for each instance one event will be returned (all summarized in one .ical file):
/<model>/caldav/?c_event_datetime_fields=start_dateend_date&c_event_description=description&c_event_subject=name

Or for one specific instance:
/<model>/<instance_id>/caldav/?c_event_datetime_fields=start_date,end_date&c_event_description=description&c_event_subject=name

While this start_date should be provided for the event to be displayed in the calendar, the end_date is optional. As a deefault for the event subject, the instance name or id can be used.

To implement this feature, the iCalendar python library can be used.
The following code snippet creates one event:

from icalendar import Calendar, Event
import uuid
from datetime import datetime

def format_timestamp(ts):
    return ts.strftime('%Y%m%dT%H%M%SZ')

def create_ical_event():
    event = Event()
    cal = Calendar()

    # Add event information
    event.add('summary', 'I am a summary')
    event.add('description', 'I am a description')
    event.add('uid', str(uuid.uuid1()))
    event['dtstart'] = format_timestamp(datetime.now())
    # event['dtend'] = format_timestamp(datetime.now() + timedelta(hours=9))
    # event['created'] = format_timestamp(current_time)
    # event['last-modified'] = format_timestamp(current_time)
    # event['dtstamp'] = format_timestamp(current_time)
    
    # Add one or more events to the calendar
    cal.add_component(event)

    # Write the file 
    f = open('event.ics', 'wb')
    f.write(cal.to_ical())
    f.close()

    return f

Enhance official documentation

The following parts are missing in the official documentation:

• API error codes for the 400 BAD REQUEST responses
• stats endpoint with the use of timestamp_start and timestamp_end

Fix create instance with specify uuid from the API

When we create an instance of any model. We must be able to specify a uuid

Delete old files when updating a file field

Currently, (v 1.52.0) When an instance of a model containing a file field is deleted, the latest file is also deleted from the storage.

However, when an instance is updated with a new file, the old file still exist on the storage.

We need to delete the old file locally, when updating an instance with a new one.

Add horizontal filtering for every m2m

It would be good if every many to many field could be filtered horizontally like in the user:

It is easier to manage content with it than use the ctrl and select when we have many items.

Feature : mechanism to avoid mid-air edit collisions

In order to avoid mid-air edit collisions, GET responses on an instance should include an ETag header with as value the last_modification_date of this instance.

When attempting a PATCH request on an instance, the request should include the If-Match header with as value the ETag value retrieved from the API (i.e the last_modification_date of the instance):

• If a PATCH request does not include a If-Match header: update the instance anyways.
• Otherwise:
  • if the If-Match matches with the last_modification_date of the instance to update, allow the request to update the instance.
  • if it does not match, return a HTTP 409 CONFLICT response.

Useful sources:

• ETags: https://devdocs.io/http/headers/etag
• Django Middelware: https://docs.djangoproject.com/fr/2.2/ref/middleware/#django.middleware.http.ConditionalGetMiddleware

Fix date and datetime filters

Description of the problem

Concrete range / comparaison filters use a method convert_type that takes an argument close_period (defaults to True):

For DateTime fields, if close_period is true, we consider the end of the day, otherwise we consider the start of the day.

Example:

convert_type("2022-02-24T15:50:00Z", "DateTimeField", close_period=True)  # 2022-02-24T00:00:00Z
convert_type("2022-02-24T15:50:00Z", "DateTimeField", close_period=False)  # 2022-02-24T23:59:59Z

If we have a model MyModel with field datetime, and two instances of this model:

• instance1: datetime = 2022-02-24T18:45:12.1653Z
• instance2: datetime = 2022-02-24T20:02:33.233Z

If we apply the API filter: ?datetime__range=2022-02-24T19:00:00Z,2022-02-24T22:00:00Z, we expect to get the instance 2 in the results of the request, but we have both results, because what is actually done is a datetime__range=2022-02-24T00:00:00Z,2022-02-24T23:59:59Z

Fix

⚠️ This issue should be handeled first #124

Supported formats for DateTime filters

• YYYY-MM-DD: (regex: r'^\d{4}-\d{2}-\d{2}$')
• YYYY-MM-DDTHH:mm:ssZ: (regex: r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$')
• YYYY-MM-DDTHH:mm:ss.xxxxxxZ: (regex: r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d{1,6})?Z$')

Possible filters

Comparaison filters

lookup \ date format	YYYY-MM-DD	YYYY-MM-DDTHH:mm:ssZ	YYYY-MM-DDTHH:mm:ss.xxxxxxZ
__gte	dt.start_of('day')	dt.start_of('second')	dt
__gt	dt.end_of('day')	dt.end_of('second')	dt
__lte	dt.end_of('day')	dt.end_of('second')	dt
__lt	dt.start_of('day')	dt.start_of('second')	dt

Range filters

For the range filter: __range=datetime_min,datetime_max:

value \ date format	YYYY-MM-DD	YYYY-MM-DDTHH:mm:ssZ	YYYY-MM-DDTHH:mm:ss.xxxxxxZ
datetime_min	dt.start_of('day')	dt.start_of('second')	dt
datetime_max	dt.end_of('day')	dt.end_of('second')	dt

[BUG] When an object is created and patched, does not appear in the deleted_uids

Description

When using the timestamp_start with a filter, we expect to find the recently changed objects that do not match the filters in the deleted_uids.

This does not include the recently created instances, because of performance reasons.

Bug case

We have a model MyModel with a boolean archived, we fetch all instances with a timestamp_start=0.0 and we keep the timestamp_end as the new timestamp start (ts_start).

We create an instance of this object with archived=False and we store this object in a local storage. We patch this object afterwards with archived=True

We fetch all the instances with timestamp_start=ts_start and with the filter archived=False.

Expected

We expect this object to be in the deleted uids in order to remove it from the local storage

Actual result

The object was created before the timestamp_start, so it does not appear in the deleted uids. The object is always in the local storage.

Fix

Do not exclude the objects created between the timestamp_start and the timestamp_end

authentication by email

Work

The user creates an account with his e-mail. He will receive a password by email, he will use it to connect.

New route

api/v1.1/auth/register_by_email

Field

email

Function

• The user give his email
• Check if the email is valid
• If is valid: Generate password
• Send password to user's email

Ordering sur FK

Implémenter l'ordering sur une fk en query params.

Exemple: un model avec un champ category

On souhaite pouvoir faire un ordering sur une liste de ce model par category name

...&ordering=category__name

Le nom du model en fk suivi de dunder le nom du champ sur lequel on souhaite faire l'ordering.

Upgrade pendulum version to 2.0

ConcreteDatastore uses Pendulum to handle dates.

Currently pendulum version in this project is pendulum>=0.8,<1(see setup.cfg).

pendulum version should be upgraded to 2.0

Rearrange Admin view

The admin view should contain the following admin configs:

• Models : contains the registered models from datamodel definition (except User and Group)
• Auth : contains the models User, Group, Token, Permission and Role
• Extra : contains the extra models as Email

Add a setting to configure the path to the swagger UI

Minimum level for update field

Method Update/Retrieve

We have to check the level user, before update or retrieve one field.
If the minimum level of the user is not high enough, we decline the request.

Create entrypoint to get Datamodel

A new endpoint /datamodel should be added to load the datamodel of the application in JSON format.

Amelioration de performances sur les permission

Permission

Première amélioration

A TESTER:

• On a un queryset avec values_list qui est fait deux fois, on peut le faire qu'une fois
• Ajouter un paramètre a distinct() => dictinct(pk) pour gagner en performance (à voir si le order_by n'est pas derangeant pour exploiter le queryset

Suggestion de modification

Model InstancePermission

Ce model servira à verifier les droit d'un utilisateur sur une instance.

Field name	Field type	Required	Default value	Description
model_name	char	True	-	le nom du model des instances
instance_uids	list	True	-	liste des instances sur lesquelles l'utilisateur a des permissions
user	fk	True	-	No description
can_edit	bool	False	False	Autorisation pour patch et post

Fonctionnement:

Ce model servira uniquement en back, on ne va pas le serializer.
La création et la modification des instances de ce model se fera par des taches celery activées par des signaux.
Les taches seront exécutées en ASYNCRONE.

Cas d'utilisation: RETRIEVE/GET sur une instance

Un utilisateur souhaite récupérer une instance du model Projet. Le nom du projet est Projet-1

On regarde l'intance d'InstancePermission du user avec le model_name égale à Project.

project_1 = Project.objects.get(name="Projet-1")
inst_perm = nuser.InstancePermission.filter(model_name='"Projet", instance_uids__icontains=str(project_1.uid))
if inst_perm.exists():
    if inst_perm.can_view:
         # Edit
    else:
        # view    

1er cas:

• L' utilisateur ne possède aucun droit sur le Projet-1
• L'utilisateur ne possède pas d'instance d'InstancePermission qui contient l'uid de Projet-1 dans le champ instance_uids

2eme cas:

• L' utilisateur possède le droit de vue sur le Projet-1
• L'utilisateur possède une instance d'InstancePermission qui contient l'uid de Projet-1 dans le champ instance_uids
• Le champ can_edit est False

3eme cas:

• L' utilisateur possède le droit d' edit sur le Projet-1
• L'utilisateur possède une instance d'InstancePermission qui contient l'uid de Projet-1 dans le champ instance_uids
• Le champ can_edit est True

Cas d'utilisation: POST/PATCH sur une instance

Un utilisateur à créé une instance du model Projet. Le nom est Projet-1

Quand un user crée le Projet avec can_view=True et can_admin=True , le signal est déclenché

1er cas:

• [POST] Création de L'instance Projet-1
• La tache Celerie verifie si InstancePermission existe deja avec user=user, model_name=Projet-1, can_admin=True .
• Si oui, elle update le Projet-1, en récuperant la liste istance_uid . Ajoute l'uid de Projet-1 et update InstancePermission
• Sinon elle crée un InstancePermission avec user=user, model_name=Projet-1, can_admin=True instance_uids[project.uid,]

2eme cas:

• [PATCH] le champcan_view pass à True de L'instance Projet-1
• La tache Celerie verifie si InstancePermission existe deja avec user=user, model_name=Projet-1, can_admin=False .
• Si oui, elle update le Projet-1, en récuperant la liste istance_uid . Ajoute l'uid de Projet-1 et update InstancePermission
• Sinon elle crée un InstancePermission avec user=user, model_name=Projet-1, can_admin=False instance_uids[project.uid,]

3eme cas:

• [PATCH] le champcan_admin pass à True de L'instance Projet-1
• La tache Celerie verifie si InstancePermission existe deja avec user=user, model_name=Projet-1, can_admin=True .
• Si oui, elle update le Projet-1, en récuperant la liste istance_uid . Ajoute l'uid de Projet-1 et update InstancePermission
• Sinon elle crée un InstancePermission avec user=user, model_name=Projet-1, can_admin=True instance_uids[project.uid,]

Pour les cas ou, il y a plusieuers instances du model InstancePermission à créer ou modifier, on peut utliser les bulk-create/update

Problématique

DEPLOIEMENT

Si les droits d'un utilisateur changent au moment du déploiement, Comment créer ou modifier les InstancePermission associés?

• [IDEE] Créer une management command, qu'on va executer manuellement

Modfication level_user/Group/

Quand les droits d'un utilisateur sont modifiés, comment peut-on procéder?

• [IDEE] Tache celery
  Quand les droits un utilisateur est ajouté à un groupe, qu'il à des nouveaux droits grâce à ce groupe. Comment peut-on procéder?
• [IDEE] Tache celery

Admin manager

Quand on modifie les droits d'un utilisateur/groupe sur la vue admin, comment peut-on procéder?

• [IDEE] Créer une action django admin

Autres questions

• Est-ce dérangeant d'avoir Celery dans tout les projets?
• Est-ce que les taches Celery vont s'executer rapidement?
• P-A à soumit l'idée d'avoir un main group qui ne change jamais pour simplifier les permissions sur les groupes
  (dans le cas des projets avec un ou peu de groupe)
• Est ce qu' on peut utiliser les permissions django? Ex: has_perm

Change on delete rule for some User related models

The on_delete for user's relations PasswordChangeToken and Email should be CASCADE

Create command management for reset password

Create a management command to reset a password users. The password will be generated, then sent by email

Name command:

• reset_password
  Parameter:
• email

Using supplied datamodel causes error on 'makemigrations'

Thanks for making concrete-datastore available, it looks like an interesting project.

Issue

In following the README.md instructions I've come across an error and I'm not sure if it's something I've done or a defect.

When I run

concrete-datastore makemigrations

the process stops with the error

concrete_datastore.parsers.exceptions.DuplicatedRelationForModel: [DUPLICATED_RELATION]: Relation duplicated in one_to_many_relations from "Project" to "User" for field "manager" Please remove the duplicated relations to make sure only one exists.

Environment

1. I'm running concrete-datastore 1.5.0 under python 3.7 .
2. I'm using the datamodel found at ./concrete-datastore/docs/assets/sample-datamodel.yml (I couldn't find the one mentioned in README.md)
3. I have an instance of postgres running on the machine I'm using so I ignore the part of the instructions which refer to launching a docker container with postgres in it.

Attempted Resolution 0

Without being too sure of the significance of what I was doing I attempted to follow the error message direction and so removed one of the two one-to-many relations from "Project" to "User" for field "Manager". I then tried running

concrete-datastore makemigrations

again. This resulted in a different error message

concrete_datastore.parsers.exceptions.MissingRelationForModel: [MISSING_RELATION]: Missing many_to_many_relations from Group to User for field "members". Please add it to many_to_many_relations section of your datamodel.

Attempted Resolution 1

Now even less sure of the significance of what I was doing I followed the directions of the error message and added a many to many relation from Group to User for field "members" as shown below :

many_to_many_relations:
  - source_field: groups
    source_model:
      uid: 9a820c34-d618-48cb-b08d-0dc37a3ca26a
      name: Project
    target_model:
      uid: 87a72328-6efc-49d6-9941-d46042c080fd
      name: Group
  - source_field: members
    source_model:
      uid: 87a72328-6efc-49d6-9941-d46042c080fd
      name: Group
    target_model:
      uid: 8dde7da5-3a0a-42a7-b579-390cd686b3fd
      name: User

I then tried running

 concrete-datastore makemigrations

again. This resulted in an error but not with a structured error message as before but just a plain old traceback as shown below

Traceback (most recent call last):
  File "/home/rshea/Envs/py37conc/bin/concrete-datastore", line 11, in <module>
    load_entry_point('concrete-datastore', 'console_scripts', 'concrete-datastore')()
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/development/cli.py", line 18, in django
    execute_from_command_line(sys.argv)
  File "/home/rshea/Envs/py37conc/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/rshea/Envs/py37conc/lib/python3.7/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/home/rshea/Envs/py37conc/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/rshea/Envs/py37conc/lib/python3.7/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/home/rshea/Envs/py37conc/lib/python3.7/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/concrete/models.py", line 30, in <module>
    from concrete_datastore.concrete.meta import (
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/concrete/meta.py", line 89, in <module>
    meta_models = loads_meta(settings.META_MODEL_DEFINITIONS) + loads_meta(
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/loaders.py", line 40, in loads_meta
    return load_func(model_definitions)
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/loaders.py", line 25, in loads_models_v1
    return modelisation.get_meta_models()  # pylint: disable=no-member
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/meta.py", line 271, in get_meta_models
    model_resource_queries,
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/meta.py", line 284, in make_model_cls
    model_uid=spec['uid'],
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/meta.py", line 318, in make_field_cls
    model_uid=model_uid,
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/meta.py", line 436, in make_cls
    resource_queries=resource_queries,
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/meta.py", line 371, in update_specifier_data
    target_model_uid=attributes['to'][self.element_id],
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/meta.py", line 167, in get_relation_for_field
    relations[f_type],
  File "/home/rshea/src/concrete-sandbox/concrete-datastore/concrete_datastore/parsers/meta.py", line 166, in <lambda>
    and x['target_model'][self.element_id] == target_model_uid,

Conclusion

I've now decided that it would be better to see if anyone could offer some advice before going any further.

I will attach the most recent version of the datamodel I've been using to this ticket as an update in just a few moments

Feature: Add more information on the /stats/ endpoint

Currently, a GET request on the endpoint /stats/ of a concrete-datastore model returns a JSON response with the data objects_count, timestamp_startand timestamp_end.

This view would be enhanced with other information such as num_total_pages, max_allowed_objects_per_page and a list/mapping of the urls of all the available pages.

the reponse data could look like:

{
  "objects_count": 34,
  "timestamp_start": 0.0,
  "timestamp_end": 123456789,
  "max_allowed_objects_per_page": 10,
  "num_total_pages": 4,
  "pages_urls": {
    "page1": "http://domain.ext/api/v1.1/model-name/",
    "page2": "http://domain.ext/api/v1.1/model-name/?page=2",
    "page3": "http://domain.ext/api/v1.1/model-name/?page=3",
    "page4": "http://domain.ext/api/v1.1/model-name/?page=4"
  }
}

Feature: Add error codes to concrete API 400_BAD_REQUEST responses

When Concrete API responds with a 400_BAD_REQUEST, the response should be:

return Response(
    data={
        "message": error_data,
        "_errors": [error_code],
    },
    status=HTTP_400_BAD_REQUEST,

Here is a list of the missing error codes with the corresponding cases:

• "_errors": ["INVALID_DATA"]

  • serializer errors
  • change password: user not found

• "_errors": ["WRONG_EMAIL_ADDRESS"]

  • retrieve secure token: user not found
  • generate secure token: user not found

• "_errors": ["INVALID_TOKEN"]

  • secure login: token does not exist
  • change password: token does not exist
  • change password: token is not the token's user
  • change password: token expired

• "_errors": ["TOKEN_HAS_EXPIRED"]

  • secure login: token has expired

• "_errors": ["CANNOT_USE_SAME_PASSWORD"]

  • change password: same password as the old one

• '_errors': ['INVALID_PARAMETER']

  • register: anonymous user passes an email_format
  • register: email_format has onvalid format

• '_errors': ['MISMATCH_PASSWORDS']

  • register: password1 != password2
  • change password: password1 != password2

• "_errors": ["EMAIL_NOT_AUTHORIZED_TO_REGISTER"]

  • register: email not allowed to register

• "_errors": [e.code] : this contains the PasswordSecurityException message

  • password security

• '_errors': ['INVALID_QUERY']

  • CRUD queries: if timestamp_start < 0
  • CRUD queries: if c_resp_page_size is not or is < 1
  • CRUD queries: if c_resp_nested is not in [true, false`]
  • CRUD queries: if wrong date format
  • CRUD queries: if filter against not allowed field

• '_errors': ['INVALID_SCOPES_HEADERS']

  • CRUD scope: invalid entity UID

The settings DISABLED_MODELS is obsolete

Current version: v1.44.0

The settings DISABLED_MODELS was used before to ignore some models defined in the datamodel.

It is now obsolete and should be removed.

Featuring group stats

The stats endpoint provides global informations about the requested model

{
  "objects_count": 125,
  "num_total_pages": 2,
  "max_allowed_objects_per_page": 100,
  "timestamp_start": 0.0,
  "timestamp_end": 701184650.0,
  "page_urls": {
    "page1": "https://<webapp>/api/v1.1/my-model/",
    "page2": "https://<webapp>/api/v1.1/my-model/?page=2"
  }
}

It would be usefull to group the objects count by a field name, to have a counter of the each value.

For example, we could get the count of teh objects by status:

{
  "objects_count": 125,
  "num_total_pages": 2,
  "max_allowed_objects_per_page": 100,
  "timestamp_start": 0.0,
  "timestamp_end": 701184650.0,
  "page_urls": {
    "page1": "https://<webapp>/api/v1.1/my-model/",
    "page2": "https://<webapp>/api/v1.1/my-model/?page=2"
  },
  "results":{
    "status":{
      "COMPLETED": 113,
      "FAILED": 12
    }
  }
}

We could also specify more than one field name, and get the counter of the values of these fields (separated or combined, depending on the user's needs):

Separated:

{
  "objects_count": 125,
  "num_total_pages": 2,
  "max_allowed_objects_per_page": 100,
  "timestamp_start": 0.0,
  "timestamp_end": 701184650.0,
  "page_urls": {
    "page1": "https://<webapp>/api/v1.1/my-model/",
    "page2": "https://<webapp>/api/v1.1/my-model/?page=2"
  },
  "results":{
    "status":{
      "COMPLETED": 113,
      "FAILED": 12
    },
    "archived":{
      "true": 53,
      "false": 72
    }
  }
}

combined:

{
  "objects_count": 125,
  "num_total_pages": 2,
  "max_allowed_objects_per_page": 100,
  "timestamp_start": 0.0,
  "timestamp_end": 701184650.0,
  "page_urls": {
    "page1": "https://<webapp>/api/v1.1/my-model/",
    "page2": "https://<webapp>/api/v1.1/my-model/?page=2"
  },
  "results":{
    "status,archived":{
      "COMPLETED,true": 45,
      "COMPLETED,false": 68,
      "FAILED,true": 8,
      "FAILED,false": 4
    }
  }
}

Date filters on creation and modification date fields do not check if the given value is a valid date

When using concrete date filters (comparaison and range) on the fields creation_date and modification_date, there is no checks on the the format of the given value.

Concrete uses the method concrete_datastore.api.v1.datetime.ensure_pendulum attempts to convert the given value to a pendulum instance without checking the format of this value, so if the value is an invalid date, the server raises a 500 Internal Error.

Example:

from concrete_datastore.api.v1.datetime import ensure_pendulum

ensure_pendulum('INVALID_FORMAT')  # ParserError: Unable to parse string [INVALID_FORMAT]

For the datamodel fields, this checks are performed in the views.py by the method check_date_format.

This method is applied only on the fields of self.fields which does not contain the custom Concrete Datastore fields such as creation_date and modification_date.

AnonymousUser self register

Currently any user can be self registered to concrete datastore.

In some cases, we don't want that any user can be self-registered, so this must be enabled/disabled by a settings ENABLE_USERS_SELF_REGISTER default to True in order to keep the existant:

If ENABLE_USERS_SELF_REGISTER is False, we must fail any regiter request that have been performed by an anonymous user (not authenticated) => 400_BAD_REQUEST with the data

{
    "message": "Self register is not allowed",
    "_errors": ["NOT_ALLOWED_TO_SELF_REGISTER"]
}

Enable datamodel distribution for smart clients

An optional endpoint may be created to distribute the datamodel in order to allow the development of smart clients.

Similar to the /version/ endpoint we may create the /datamodel/ url.

In a first approach we can restrict the datamodel access to user > admin

Add exclude filter

It would be useful to have an exclude filters to exclude certain values from the results.

Example:

/api/v1.1/<model-name>/?status__exclude=FAILED

This request will return every instance that does not have FAILED as status field

Add SVG file management to concrete

any .svg image is currently not managed by concrete-datastore .
.svg is an xml image format, and should be considered as an image field, currently, it is only recongized as a type file, so we have to overload image field in concrete in order to add .svgfiles

in the admin form for mfa, the field password has autocompletion

In some browsers like chrome, when attemting to login with the MFA enabled, chrome autocompletes the field password with the password stored. this field should disable the autocompletion

Cannot delete an instance of SecureConnectToken

When deleting an object from a concrete datastore DB, an instance of DeletedModel is created with the uid of this object.

SecureConnectToken does not have an uid field, so this model should be ignored when deleting instances

Allow micoseconds for datetime filters

Currently, the datetime filters only accept these two formats:

• YYYY-MM-DD
• YYYY-MM-DDTHH:mm:ssZ

We should be able to also accept the format with the microseconds YYYY-MM-DDTHH:mm:ss.xxxxxxZ with at most 6 digits for microseconds (regex: r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d{1,6})?Z$')

Authentication using c_auth_with_token should support uuid (ProcessRegisterView)

Python 3.10 support

Currently Python 3.10 is not supported.
This is due to the package psycopg2-binary that can not be installed using wheels using pip.

Filtering against FK field

We should be able to filter against a foreign key field.
Example:

We have the datamodel

Book

• author (fk)
• published_date (date)
• editor (txt)
• ...

Author

• first_name (txt)
• last_name (txt)
• city (txt)

We should be able to do these kind of filters:

• /api/v1.1/book/?author__first_name=John
• /api/v1.1/book/?author__city__in=paris,london
• ...

Fix README instructtions to init a ConcreteDatastore project

missing parts in README:

• Running a postgres Docker instance should specify the batabase name, the user name and the user password (the default values are in /development/settings.py in DATABASES):

docker run --name postgres-concrete -e POSTGRES_DB=db-concrete-datastore -e POSTGRES_USER=user-concrete-datastore -e POSTGRES_PASSWORD=pwd-concrete-datastore -d -p 5432:5432 postgres

• Before installing the package, make sure setuptools and pip are up-to-date:

pip install -U pip
pip install -U setuptools

Add JSONField key filtering

It should be possible to filter against JSONField data.
For example with this data for a field called data and a model MyModel:

{
  "data": {
    "name": "Concrete",
    "type": "python",
    "owner": "Netsach"
  }
}

We should be able to filter like so:

• /api/v1.1/my-model/?data__name=Concrete
• /api/v1.1/my-model/?data__type=python
• /api/v1.1/my-model/?data__owner=Netsach

Bug retrieving the divider model for an anonymous user

Even if the minimum_retrieve_level on the divider model is set to anonymous, ConcreteDatastore does not allow this method.

Any retrieve on the divider model with an anonymous user returns zero results.

This should be fixed by returning the public objects of the divider model

Validate uid tokens for unsubscription views

For unsubscription views here:
https://github.com/Netsach/concrete-datastore/blob/master/concrete_datastore/concrete/views.py#L50

A get_object_or_404 is done without checking the type of the token. If it's not an UUID, an error is raised and it should only return a 400.

Suggestions:

• Check in the urls directly that we have an UUID in the path
• Add an UUID validation before the get_object_or_404

Authorize anonymous creation with throttling

An anonymous user should be able to create instances to the datastore with some limitations.
See https://www.django-rest-framework.org/api-guide/throttling/
After X requests in a minute for example, the anonymous user should receive a 429.

Ability to add a language as queryparams for `export`

export endpoint should be able to accept the queryparam ?lang= with the value of the language desired for the export (for example fr).

If this queryparam exists, the following should be transalted to the selected language:

• Booleans: for example if ?lang=fr, True becomes VRAI and False becomes FAUX
• datetime and date fields: the value of pendulum with the right locale set should be returned in the format: YYYY/MM/DD HH:mm:ss
• csv headers: the name of the files should be translated depending on a setting in concrete datatsore: this setting is a dict and looks like:

{
  "fr": {
    "filed_name_1": "nom_du_champ_1",
    "filed_name_2": "nom_du_champ_2"
  }
}

Ordering on fields creation_date and modification_date is not enabled

ConcreteDatastore ordering is enabled on all the fields of the display_fields of the datamodel. These fields are automatically generated, so they won't appear in the display_fields of the models of the datamodel.

When parsing the yaml file, these fields should be added to the m_list_display property of all the models, and removed from the list_display of the admin models to prevent redundancy in the admin site