inseefr/arc

Workbench for acquisition and normalization of data sets

arc's Issues

Translate to english

There is a lot of french in the application (documentation or fields/variables/methods/classes name) which need to be translate in English.

Security alert

Dependency to Insee internal configuration component

It is written as an outstanding warning in the documentation : there is a dependency to an INSEE internal java component for ARC. This link make the whole application not usable.

When a solution will be provided ?

Reorganising the documentation

The main documentation (provided through the README.md) is useful but maybe a bit dense for a first grip on what ARC is about.

Identically to what is generally provided by open-source software (1), we may provide a first README that would :

• provide a quick context paragraph
• how to run the application
• a link to more documentation (the existing README file for example)
• how to interact with the maintainers and contribute

(1) see for example the tensorflow github repo or the one for airflow

Error adding Control rule

hi,
when I try to add Control Rule, I get this error:

org.postgresql.util.PSQLException: ERROR: date/time field value out of range: "31/12/2020"
Hint: Perhaps you need a different "datestyle" setting.
Position: 254

Francesco

End of support for java8 by Sonarcloud

Sonarcloud will stop supporting Java 8 in October 2020.

This is related to the jdk version declared in .travis.yml (openjdk8).

Enhancing the Dockerfile for the webapp

The existing Docker images for ARC is only handling the deployment of a pre-built WAR archive into a Tomcat application server.

Good practice nowadays is to include the build step into the image (a.k.a a build container) in order to have a unique and overarching build tool.

Ref : https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

Example : https://github.com/I3S-ESSnet/is2/blob/master/app.Dockerfile

Redesign tests that use the database

Some unit tests currently need a real connexion to a running database (see for example this one), running database that doesn't exit in a CI context.

For the moment, we disable those tests but we want in a near future to be able to run them. We need for that a DB test strategy :

• using a mock DB
• or an in-memory DB

How to generate test report on github ?

We are asked to generate test reports. Any idea of what solution to implement Romain or HealerMikado ?
Also, is it possible to implement the integration gitlab-CI on Github the same way we does in our internal repository ?

BiUntokenizer interface not used

My IDE tell me that BiUntokenizer is never used.

Is it dead code or work-in-progress ? In the first case, we should remove the interface, shouldn't we ?

scale version of dataretrieval webservice

Make ARC be able to retrieve data from several executor nods

Database user and rights on docker script

Check if the users is a non admin user and set the database initialization script to handle a non superuser

Execution of docker-compose up on Windows 10

Dear ARC Team,

I am trying to install ARC on a Windows machine. I execute the 'docker-compose up' command, but I get the following error message:
Step 8/15 : ARG LOG_PATH=logs/log-arc.log ---> Running in d175228b6d89 Removing intermediate container d175228b6d89 ---> 3e420a404f9b Step 9/15 : ARG LOG_LEVEL=ERROR ---> Running in 67e1109380ac Removing intermediate container 67e1109380ac ---> fa0955e40c84 Step 10/15 : ARG LOG_SETTINGS=fr/insee/config/log4j.xml ---> Running in 1911904ebd8f Removing intermediate container 1911904ebd8f ---> ed1df1c30c6a Step 11/15 : COPY . /usr/src/app/ ---> c27f84c77339 Step 12/15 : RUN chmod +x usr/src/app/script.sh && usr/src/app/script.sh ---> Running in 1f9f3cc346c4 /bin/sh: usr/src/app/script.sh: not found ERROR: Service 'app' failed to build: The command '/bin/sh -c chmod +x usr/src/app/script.sh && usr/src/app/script.sh' returned a non-zero code: 127

We need tests !

I'm currently translating the classes in utils.sqlengine and i sadly noticed the lack of tests, for example tests related to the table naming schema.

It's nearly mandatory nowadays to provide a code base with an extensive test suite, at least for building confidence. Test also helps understand the inner workings of the software.

Could you please take care of the follwing module in the package ARC/arc-utils/src/main/java/fr/insee/arc/utils/ ?

Romain, could you please take care of the follwing module in the package ARC/arc-utils/src/main/java/fr/insee/arc/utils/ ?
ressourceUtils
sqlengine
structure
textUtils
utils
webutils
Thank you

Originally posted by @Nolife999 in #7 (comment)

Logger used is deprecated

Switching to Log4j2 as it was already declared in project dependencies

could you please take care of the following module in the package ARC/arc-utils/src/main/java/fr/insee/arc/utils/ ?

Franck, could you please take care of the follwing module in the package ARC/arc-utils/src/main/java/fr/insee/arc/utils/ ?
batch
bdd
dao
files
format
Thank you

Originally posted by @Nolife999 in #7 (comment)

[Features ?] Loader for flat file with positional records ?

I have just read your very interesting documentation. But as there is a warning about its incompleteness, I allow myself to open this issue because I have a doubt about the features offered.

Are flat files with positional records supported as loader?

Like this one ?

123456789012345678901234567890123456789012345678901234567890123456789012345  
US        Alice Smith         123 Maple Street    Mill Valley    CA 90952  

• An attribute named country/region that is left-aligned, 10 characters in length, with a zero character offset.
• An element named name that is left-aligned, 20 characters in length, with a zero character offset.
• ...

I don't think so. I could only read information about key files, values, xml and csv.

ARC REST API

Simply suggesting here the use of a dedicated REST API tool (Jersey is a good start) instead of implementing it from scratch with the use of Servlets.

This way we will benefit from related technology, like JSON parsing and ser/deser.

Cannot create model table with numbers in name

In model family managment, numbers cannot be used in the table name (0,1,2...,9) as it should be

Naming conventions for packages

The package naming is a bit curious here : underscores ( _ ) could be replaced by dashes ( - ), and i also think that the base name should be "arc" instead of "arc_essnet".

Generic authentication method

The initial version of the application was relying on an Insee internal solution for authentication.

We should now provide a generic solution:

1. which will provide a simple auth implementation for testing purpose
2. and will allow to plug into a provided authentication method

Translate the user guide

A french user guide exists (50 pages of great quality), we should translate it to english. 🇬🇧

Continuous integration tooling

To expand the transparency on the project, i suggest to add at least :

• a build service (like Travis CI and such),
• a code quality tool (thinking here about Code climate for example).

Sonar CI fails on PR

As experienced in the last few PR (see https://travis-ci.org/InseeFr/ARC/pull_requests), CI is broken for pull requests due to sonar trying to send it's results to localhost instead of sonarcloud. This seems to be caused by GH-32.

Nonexistent profile 'coverage' in travis.yml

The sonar call in travis.yml use the profile "coverage" :
mvn clean verify sonar:sonar -Pcoverage
This profile is absent, resulting in the following warning :
[WARNING] The requested profile "coverage" could not be activated because it does not exist.

This might or might no be related to Sonar not picking up test coverage reports.

Here is a working example of a a pom with a coverage profile for reference : https://github.com/SonarSource/sq-com_example_java-maven-travis/blob/master/pom.xml

Errror on WAR deploy

Hi,
I'm trying to deploy the arc-web.war on Tomcat 8, but I get this error:

01-Oct-2020 08:51:13.287 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'gererFamilleNormeAction': Unsatisfied dependency expressed through field 'properties'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'properties' defined in class path resource [applicationContext.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'logConfiguration' threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130)

Regards,
Francesco

Suggestion : uses stricter version range or no range at all in pom

I just had a few issues setting up the project, mostly linked to Maven fetching an RC version of Spring 5. This is coherent with the currently defined range in the pom of [4.3.17,) (any version equal or greater to 4.3.17).
My take on this is that, at the very least, version range should have an upper bound to prevent any major version change (as in here from 4 to 5). Major version changes, by definition, do not guaranty retro-compatibility.
That would be for instance [4.3.17,4.9999) (cf https://medium.com/@MichaKutz/legit-but-useless-maven-version-ranges-explained-d4ba66ac654)

More generally, i feel like any version range still has the major setback of reducing reproducibility and portability.

Use classes or enums not interfaces for storing constants

In the textUtils package, three interfaces hold various constants (see for example IConstanteCaractere).

Why don't we simply use classes for that ? Or best, enums (see that post for reference) ! ✨