inseefr/pogues-model

Classes and converters for the Pogues data model

pogues-model's Issues

add missing details in pogues questionnaire model

When translate JSON Pogues to Pogues model, some details are missing:

• type from VariableType
• type from DataType
• owner

Also, deserializing to json objects can only be done with files. One should be allowed to do it with inputstream

Redesign loop modeling: DynamicIterationType

Redesign

<xs:complexType name="DynamicIterationType">
    <xs:complexContent>
      <xs:extension base="IterationType">
        <xs:sequence>
          <xs:element name="IterableReference" type="xs:token">
            <xs:annotation>
              <xs:documentation>Could be a Roster (dynamic table) or another iteration</xs:documentation>
            </xs:annotation>
		  </xs:element>
          <xs:element name="Filter" type="ExpressionType" minOccurs="0"/>
        </xs:sequence>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>

into:

<xs:complexType name="DynamicIterationType">
    <xs:complexContent>
      <xs:extension base="IterationType">
        <xs:sequence>
           < xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
          <xs:element name="Maximum" type="xs:nonNegativeInteger" minOccurs="0"/>
          <xs:element name="Step" type="xs:nonNegativeInteger" minOccurs="0"/>
          <xs:element name="IterableReference" type="xs:token">
            <xs:annotation>
              <xs:documentation>Could be a Roster (dynamic table) or another iteration</xs:documentation>
            </xs:annotation>
                          </xs:element>
          <xs:element name="Filter" type="ExpressionType" minOccurs="0"/>
        </xs:sequence>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>

Change the Minimum and Maximum type for a NumericDatatypeType

About the NumericDatatypeType type.
We have to define a Minimum and a Maximumas as Decimal.
Possible to change it in decimal instead of integer?

Iteration : Minimum could be a formula

For some questionnaires like the one for VQS study or the TCM, the minimum of an iteration could be a formula.
Is it please possible to modify:
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Maximum" type="ExpressionType" minOccurs="0"/>
<xs:element name="Step" type="xs:nonNegativeInteger" minOccurs="0"/>
...
into
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="ExpressionType" minOccurs="0"/>
<xs:element name="Maximum" type="ExpressionType" minOccurs="0"/>
<xs:element name="Step" type="xs:nonNegativeInteger" minOccurs="0"/>

"Please, specify" option

Use case: add the "Please, specify" option.

"Please, specify" is a request for additional information from a list of choices defined a priori (single or multiple choice question).
This additional information is related to a response domain and triggered when a specific response value is selected or typed.

It's characterised by:

a label (can be something else than "Please, specify")
a response to collect the specified value
a link to a response domain
a trigger value to display the "Please, specify" request

Side note:
a single choice question contain only one response (a code list)
a multiple choice question contain one response (code list or boolean) by choice

Iteration : Maximum could be a formula

Maximum could be a formula. Example : for VQS Survey, the iteration maximum will be a collected variable corresponding to the number of people living in the household.

Thank you to redesign:
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Maximum" type="xs:nonNegativeInteger" minOccurs="0"/>
...
into:
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Maximum" type="ExpressionType" minOccurs="0"/>

Add the concept "variable" in the model

We have to add the concept of variable in the model.

A variable can a be collected, calculated or external.
A collected variable is a statistical variable collected within a questionnaire for the survey need.
An external variable is a variable
A calculated variable is a variable calculated from others variables including the calculated variables.

An external variable refers to a variable not collected in the questionnaire, but useful for personalization.
For example, it may be a collection wave number for filtering questions, a date to be displayed in the wording of a question, etc.

We also need to reference these three kind of variable to write the conditions (or more generally the controls) in all the questionnaire. That's why it should be interesting to wrap the variables in an element "Variables" as we did whith "CodeLists".

To explain more, an modelization example below....but would it be better to use an "abstract"?, typed the variables? others ???

  <xs:element name="Variables">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="CollectedVariable" minOccurs="0" maxOccurs="unbounded"/>
        <xs:element ref="ExternalVariable" minOccurs="0" maxOccurs="unbounded"/>        
        <xs:element ref="CalculatedVariable" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>


  <xs:element name="CollectedVariable">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Name" type="xs:token"/>
        <xs:element name="Label" type="xs:token"/>
      </xs:sequence>
      <xs:attribute name="id" type="xs:ID" use="required"/>
    </xs:complexType>
  </xs:element>

  <xs:element name="CalculatedVariable">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Name" type="xs:token"/>
        <xs:element name="Label" type="xs:token"/>
        <xs:element name="Formula" type="xs:token"/>
      </xs:sequence>
      <xs:attribute name="id" type="xs:ID" use="required"/>
    </xs:complexType>
  </xs:element>


  <xs:element name="ExternalVariable">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Name" type="xs:token"/>
        <xs:element name="Label" type="xs:token"/>
      </xs:sequence>
      <xs:attribute name="id" type="xs:ID" use="required"/>
    </xs:complexType>
  </xs:element>

Consequence, we add the element to the ResponseType:

  <xs:complexType name="ResponseType">
    <xs:sequence>
      <xs:element name="CodeListReference" type="xs:token" minOccurs="0"/>
      <xs:element name="Datatype" type="DatatypeType"/>
      <xs:element name="Value" type="xs:anyType" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element name="NonResponseModality" type="NonResponseModalityType" minOccurs="0"/>
      <xs:element ref="CollectedVariable"/>
    </xs:sequence>
    <xs:attribute name="simple" type="xs:boolean"/>
    <xs:attribute name="mandatory" type="xs:boolean"/>
  </xs:complexType>

Change 'invite' term to 'probe' in NonResponseModalityType

Our questionnaire methodologist asks for changing 'invite' term to 'probe' in NonResponseModalityType. This term seems to be more appropriate in the methodological documentation.
Probing is a common technique that researchers use in interviewer-administered surveys when respondents initially refuse to answer a question or say they "don't know." Interviewers are trained to use neutral probing techniques -- such as "Would you lean more toward [answer] or [answer]?" or "Just your best guess is fine" -- to encourage valid responses. Probing can be particularly effective when respondents initially hesitate to provide an answer, and it can increase the number of valid responses.

As a reminder, the NonResponseModalityType below:

<xs:complexType name="NonResponseModalityType">
    <xs:sequence>
      <xs:element name="Label" type="xs:token"/>
      <xs:element name="Value" type="xs:anyType"/>
      <xs:element name="Invite" type="xs:string" minOccurs="0"/>
    </xs:sequence>
    <xs:attribute name="firstIntentionDisplay" type="xs:boolean" use="required"/>
 </xs:complexType>

Document the model

Add the loop concept

A loop is an iteration (according to one criterion) on a group of questions.

• The group of questions is equivalent to a sequence (= Module) or a sub-sequence (= sub-module).
  We could imagine taking a subset of questions within a module (or sub-module) but it was chosen to make simple.
• an iteration criterion is a condition defining the number of times the question group is repeated. It can simply be (considering that by default we start at 1 and that the step is 1):

-- explicitly define a maximum,
-- define the maximum from a reference to a variable (calculated or external)
-- indicates the table on which to measure the maximum (maximum number of individuals declared in the table). Its seems to be a calculated variable.

Add variableRepresentation to characterize a variable

We need to know how a variable is represented to transform the Pogues control syntax in XPath syntax.

Could we have an enumeration as:

• code
• numeric
• text
• date

Add MaxOccurs in ClarificationQuestion

Hello,

We have to define multiple ClarificationQuestion for a unique Question. For instance in one of our business surveys about very small enterprises (less than 9 people), we can find a table with a line for each employee and a column about the type of contract and if other contract is chosen, a clarificationQuestion to clarify this other contract.
It would be therefore necessary to add a: maxOccurs="unbounded" to the element below included in the QuestionType:

<xs:element name="ClarificationQuestion" type="QuestionType" minOccurs="0"/>

Thanks a lot !

Improve Log4j 2 configuration for the rest service

See https://logging.apache.org/log4j/2.x/manual/webapp.html

bug: cannot serialize variables types with pogues model imported in other project

When using pogues-model from another app, the serialization of variables types fails when retrieving variables from pogues endpoint (serialization of global questionnaire works though)

Organize transition from GoTo to FlowControl

GoToType and FlowControlType are both defined in the model for smooth transition. Next steps are:

• rename GoTo element in ComponentType (this is invalidating) and retype it
• remove GoTo type

Updating the schema for the CodeListSpecification

Two things in fact must be done :

1. the uri element should be renamed retrievalURI or something like this because it holds the request that returns the values of a classification level,
2. we should also add a label element for naming purpose.

GoTo or IfThenElse logic to specif the dynamic of a questionnaire

Survey designer should have the possibility to choose if they prefer to describe the dynamic of their questionnaire (jump directly to a question, a sequence or a subsequence according to a condition on some variables) with a GoTo or an IfThenElse logic. Therefore, PoguesModel should enable this choice thanks to a dedicated parameter. However, DDI can not handle this information.
Thanks for your help !

Remove SequentialIterationType

Remove SequentialIterationType.
First step is to verify that the new modeling for DynamicIterationType is sufficient (see #48).

Date and duration format

PoguesModel is currently managing only one date format : ddmmyyyy.

In order to manage other date and duration questions, we need to change Pogues model in order to allow :
-2 new date format : mmyyyy and yyyy
-duration format composed of years, months, days, hours, minutes (for exemple survey can ask for a duration in years-months or months-days or hours-minutes and so one)

We can also notice that there will be 4 new "numeric" duration : hours, days, months, years. But, as they will be managed with 4 corresponding new measurement units for numeric format, no consequence for PoguesModel.

The new PoguesModel should be ideally compatible with old/current questionnaires in which a DateDatatype only means the date format ddmmyyyy. If not, we should remember to correct json file of these questionnaires before we reuse them.

Thanks a lot !

Add QuestionType (or ResponseFormat) ?

We want to differentiate several Response format or question type. It's a characteristic of the question.
The question type can be : SIMPLE, SINGLE, MULTIPLE and TABLE

This classification is maybe not the final release but it could help to better manage the serialization.

Can we add this information in the Model ?

Pairwise modelisation (missing SymLink)

To implement : InseeFr/Bowie#2 in link with : InseeFr/Pogues#601

Remove GoTo constructs

Remove GoToType and GoTo element in ComponentType.
GoTo constructs are superseded by flow controls.

First step is to verify that GoToType is not used in existing questionnaires.

Update DeclarationTypeEnum

At the moment, we can find in the model:
<xs:simpleType name="DeclarationTypeEnum">
<xs:restriction base="xs:token">
<xs:enumeration value="INSTRUCTION"/>
<xs:enumeration value="COMMENT"/>
<xs:enumeration value="HELP"/>
<xs:enumeration value="WARNING"/>
</xs:restriction>
</xs:simpleType>

But we don't need WARNING and COMMENT. However, we need also a CODECARD modality.
So, the model could be updated like that:

<xs:simpleType name="DeclarationTypeEnum">
<xs:restriction base="xs:token">
<xs:enumeration value="INSTRUCTION"/>
<xs:enumeration value="HELP"/>
<xs:enumeration value="CODECARD"/>
</xs:restriction>
</xs:simpleType>

Consistency in naming: can we stop mixin upper/lower case ?

Pogues naming convention tells that elements are uppercase while attributes are lowercase.

This makes us end with a JSON where keys are inconsistently distributed between lower/uppercase, making the model hard to predict.

One solution would be to use lower camel case everywhere in the model.

Add a specific modality attached to the Response

We want to specify a specific modality (the most significant case is the modality Don't know or later Refusal) attached to the Response
For the moment, a specific modality can be attached to a response only for a question type Single

A SpecificModality is defined by:

• a Label (the label of the modality)
• a Value (the Value of the code)
• a DynamicBehaviour: an enumeration with the values FIRST_INTENTION and SECOND_INTENTION (the dynamic behaviour in the generated questionnaire)
• a UrgingMessage (the message for the respondant)

ComponentGroup contains Components

Do we really want a ComponentGroup to be as sequence of ComponentType elements ? That would mean that we need to serialize every component in multiple places : in the sequence it belongs to, and in the component groups where it is referenced, wouldn't it ?

Might we use an xs:token element like we do for codeListReference or ifTrue element, juste to keep track of the id of the component ?

<xs:element name="ComponentGroup">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="Name" type="xs:token"/>
      <xs:element name="Label" type="xs:token" maxOccurs="unbounded"/>
      <xs:element name="Declaration" type="DeclarationType" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element name="Member" type="ComponentType" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute name="id" type="xs:ID" use="required"/>
  </xs:complexType>
</xs:element>

Remove AFTER__RESPONSE modality

About the position of a Declaration.
AFTER_RESPONSE position is equivalent to DETACHABLE (i.e. footnote). We don't really have use cases for using both modalities.
We prefer removing AFTER_RESPONSE.

The Question element as a global element

Use case : reuse a question
To reuse a question from the repository, we have to get back only one question (and not the whole questionnaire) in the XML Pogues format.
Can we add a question element as a global element?

Manage hierarchical code lists

We have to manage a hierarchical code lists. Its means that a code may contain a code that may contain a code, etc. (as a recursive description).

One idea: create a type: "CodeType" containing a code typed "CodeType" and refers it in the CodeList:

  <xs:element name="CodeType">
   <xs:complexType>
     <xs:sequence>
       <xs:element name="Value" type="xs:token"/>
        <xs:element name="Label" type="xs:string"/>
        <xs:element name="Code" type="CodeType"/>
       </xs:sequence>
     </xs:complexType>
    </xs:element>      

The CodeList description could become:

  <xs:element name="CodeList">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Name" type="xs:token"/>
        <xs:element name="Label" type="xs:token"/>
        <xs:element name="Code" type="CodeType" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="id" type="xs:ID" use="required"/>
    </xs:complexType>
  </xs:element>

Maybe should we add a Code directly in the CodeList without creating the CodeType?

Add a FlowControl element within a Questionnaire

We want to use the FlowControl element in two ways:

• as a GoTo only used in a question
• as a Filtrer only used in the questionnaire

We prefer describing a Filter at the same level of the other components, i.e in the Questionnaire.

It could mean adding something like that in the Questionnaire:
<xs:element name="FlowControl" type="FlowControlType" minOccurs="0" maxOccurs="unbounded"/>

Redesign the type of question "Table"

We have to identify each data cell. Could we use a cell coordinates system instead of the mappingType element?

Use cases:

• bind a collected variable to a response...and well managed the table structure changes (for exemple if a table with 4 lines and 3 columns become a table with 3 lines and 4 columns).
• define a specific cell as "no data by definition" (a no meaning intersection)

Create a new version of an object

Use case: Create a new version of an object in the repository
To do it, we need to refers a "colectica id" (or something like an URI) in Pogues to indicate on which object create the new version. In brief, we neek **a key between an object in Pogues and the same object in the repository.
If there is no key, it means: create a new object.

In addition, we need another information to say if we want to:

• create a new object
• create a new version
• reuse the same version of the object

At this time, the list of versioning objects is:
Sequence, Question, Declaration (we'll need an Id, see #21), Variable, CodeList
Maybe later: a Response, GotTo ?, Control?

Add maxoccurs in NonResponseModalityType

We have to definie multiple NonResponseModality for a unique Response. For instance for the interviewer-administered surveys, we often define these two NonResponseModality:

• Don't know
• Refusal

It would be necessary to add a: maxOccurs="unbounded" to the element below included in the ResponseType:

<xs:element name="NonResponseModality" type="NonResponseModalityType" minOccurs="0"/>

Add the statistical context to a questionnaire

A series pools a set of statistical operations sharing a homogeneous mthodology and a main business characteristic.
It's characterised by:

• a short name (or ID ?)
• a label

A statistical operation can be defined as a set of collection, capture, process and data dissemination unique and consistent. Each statistical operation can contain one or several data collection.
It's characterised by:

• a series
• a year

A data collection (or a data collection campaign) is a survey data collection realization period. Each data collection can contain one ore several questionnaire.
It's characterised by:

• a statistical operation
• a period (M01 - January,...,T02 - Second quarter,etc.)

Starting hypothesis: all these concepts are created outside Pogues.

We need to add these concepts in the model.

Measurement unit

Subject to discuss.

We need to define a measurement unit attached to a NumericDatatypeType.
One is to mangage an enumeration.

It would be a first version to manage a measurement unit because few special cases must be studied (manage a measurement unit depending on a variable, etc.).

Example of enumeration:
€
K€ (or thousands of euros or both?)
%
m²
tonnes
m³
litres
10³Nm³
MWh
kW
hours (maybe we should create a DurationDatatypeType?)
hours / week
days worked / week (jours travaillés/ semaine)
hours / year
days / year
working days
calendar days

Finally, maybe it's better to have a xs:token?

Link between Variables and CollectedVariable

There is no link between the variable in the wrap and the variable created in the response element.

For exemple: if we delete a collected variable in a response, we can't delete it in the wrap, because we don't know what is the variable to delete.

One idea is to have a CollectedVariableReference in the ResponseType (instead of the CollectedVariable) as we did for the code list.

Add a status to the questionnaire

We have to know if a questionnaire is "in progress" or "published"....maybe other status we will define all the process.

Idea: add an attribut "status" to the questionnaire.

enhancement: ability to validate model

If pogues-model could validate the questionnaire model, it could be a great addition, we could detect validation errors at the beginning of the questionnaire workflow.

Project does not build

Project does not compile any more since fields has been renamed in the schema.

See commits:

453f034
7390a49
c88a4f2
dd42888
9f50429

More over, it looks like there's a mapping issue as just replacing Survey by DataCollection as a type in QuestionnaireFactory does not solve this issue: No setter is added to Questionnaire for DataCollection when sources are generated from schema.

Change the type of the Next element

The Next element within the FlowControleType is defined as a xs:token:
<xs:element name="Next" type="xs:token"/>
This element is used to model nested filters.
It means that we have to reference a FlowControl. It would be easier to type this element as a FlowControlType (and so defining the FlowControlType as recursive element):

<xs:element name="Next" type="FlowControlType"/>

Declaration ID

Add an ID to the Declaration element

The Sequence element as a global element

The same idea than #17
Use case : reuse a sequence
To reuse a sequence from the repository, we have to get back only one sequence (and not the whole questionnaire) in the XML Pogues format.
Can we add a sequence element as a global element?

enchancement: DateFormatEnum only include patterns for JS

DateFormatEnum returns patterns for JS (ex: YYYY-MM-DD). The corresponding pattern for java: yyyy-MM-dd

This causes problems when trying to validate data type like DateDatatypeType

Add documentation about transformations between json and xml

See TODO in https://github.com/InseeFr/Pogues/blob/38cb8ea2221cc6d1a6561112d9cf32288993f7d5/doc/remote-apis/questionnaire-json.md

Add the questionnaire last update date

We have to know the questionnaire last update date when we save or publish a questionnaire.

FlowControl : There can be several Next for a same FlowControl

We need sometimes several Next for a same FlowControl.
For example, in the TCM (Tronc commun des ménages) (set of questions common to every household surveys), there's a filter on living areas.
The sequence about living areas is asked only to people having another living area.
And within this sequence, we have for example a question only asked for students and another one only for workers.
That's why we shoud redesign:
<xs:complexType name="FlowControlType">
xs:sequence
<xs:element name="Description" type="xs:string" minOccurs="0"/>
<xs:element name="Expression" type="ExpressionType"/>
<xs:element name="IfTrue" type="xs:token"/>
<xs:choice minOccurs="0">
<xs:element name="IfFalse" type="xs:token"/>
<xs:element name="Next" type="FlowControlType"/>
</xs:choice>
</xs:sequence>
into:
<xs:complexType name="FlowControlType">
xs:sequence
<xs:element name="Description" type="xs:string" minOccurs="0"/>
<xs:element name="Expression" type="ExpressionType"/>
<xs:element name="IfTrue" type="xs:token"/>
<xs:choice minOccurs="0">
<xs:element name="IfFalse" type="xs:token"/>
<xs:element name="Next" type="FlowControlType" maxOccurs="unbounded"/>
</xs:choice>
</xs:sequence>

Add declarationMode element

The declarationMode is defined as attribute. We have to define several mode for each declaration.
We should define it as element as we did for the ComponentType (and not as attribute).

<xs:complexType name="DeclarationType">
    <xs:sequence>
      <xs:element name="Text" type="xs:string"/>
    </xs:sequence>
    <xs:attribute name="id" type="xs:ID" use="required"/>
    <xs:attribute name="declarationType" type="DeclarationTypeEnum"/>
    <xs:attribute name="declarationMode" type="SurveyModeEnum"/>
    <xs:attribute name="position" type="DeclarationPositionEnum"/>
  </xs:complexType>

would become:

<xs:complexType name="DeclarationType">
    <xs:sequence>
      <xs:element name="Text" type="xs:string"/>
     <xs:element name="DeclarationMode" type="SurveyModeEnum" minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute name="id" type="xs:ID" use="required"/>
    <xs:attribute name="declarationType" type="DeclarationTypeEnum"/>
    <xs:attribute name="position" type="DeclarationPositionEnum"/>
  </xs:complexType>

enhancement: problem serializing/deserializing with jackson

As beans in pogues-model are generated with jaxb, annotations are xml based. This causes problems when using jackson for deserializing/serializing for enums for example. Could be great to have non empty constructors too .

Reuse a declaration

Use case: reuse a declaration within the Pogues questionnaire.

Idea: only refer the declaration in the Component and create it in a wrap

Choose the language (VTL or xpath) for formulas

Current tools need formulas written in pseudo-xpath (cf. xforms/Orbeon technologies).
The new tools based on javascript and Lunatic need formulas written in VTL. Depending on the context of the integration of their questionnaire (xforms/Orbeon called V1 or Javascript/Lunatic called V2), the user should be able to specify in Pogues HMI whether they want to specify the formulas of their questionnaire in xpath or in VTL.