inseefr / pogues-model Goto Github PK
View Code? Open in Web Editor NEWClasses and converters for the Pogues data model
License: MIT License
Classes and converters for the Pogues data model
License: MIT License
When translate JSON Pogues to Pogues model, some details are missing:
Also, deserializing to json objects can only be done with files. One should be allowed to do it with inputstream
Redesign
<xs:complexType name="DynamicIterationType">
<xs:complexContent>
<xs:extension base="IterationType">
<xs:sequence>
<xs:element name="IterableReference" type="xs:token">
<xs:annotation>
<xs:documentation>Could be a Roster (dynamic table) or another iteration</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="Filter" type="ExpressionType" minOccurs="0"/>
</xs:sequence>
</xs:extension>
</xs:complexContent>
</xs:complexType>
into:
<xs:complexType name="DynamicIterationType">
<xs:complexContent>
<xs:extension base="IterationType">
<xs:sequence>
< xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Maximum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Step" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="IterableReference" type="xs:token">
<xs:annotation>
<xs:documentation>Could be a Roster (dynamic table) or another iteration</xs:documentation>
</xs:annotation>
</xs:element>
<xs:element name="Filter" type="ExpressionType" minOccurs="0"/>
</xs:sequence>
</xs:extension>
</xs:complexContent>
</xs:complexType>
About the NumericDatatypeType type.
We have to define a Minimum and a Maximumas as Decimal.
Possible to change it in decimal instead of integer?
For some questionnaires like the one for VQS study or the TCM, the minimum of an iteration could be a formula.
Is it please possible to modify:
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Maximum" type="ExpressionType" minOccurs="0"/>
<xs:element name="Step" type="xs:nonNegativeInteger" minOccurs="0"/>
...
into
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="ExpressionType" minOccurs="0"/>
<xs:element name="Maximum" type="ExpressionType" minOccurs="0"/>
<xs:element name="Step" type="xs:nonNegativeInteger" minOccurs="0"/>
Use case: add the "Please, specify" option.
"Please, specify" is a request for additional information from a list of choices defined a priori (single or multiple choice question).
This additional information is related to a response domain and triggered when a specific response value is selected or typed.
It's characterised by:
a label (can be something else than "Please, specify")
a response to collect the specified value
a link to a response domain
a trigger value to display the "Please, specify" request
Side note:
a single choice question contain only one response (a code list)
a multiple choice question contain one response (code list or boolean) by choice
Maximum could be a formula. Example : for VQS Survey, the iteration maximum will be a collected variable corresponding to the number of people living in the household.
Thank you to redesign:
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Maximum" type="xs:nonNegativeInteger" minOccurs="0"/>
...
into:
<xs:complexType name="DynamicIterationType">
xs:complexContent
<xs:extension base="IterationType">
xs:sequence
<xs:element name="Minimum" type="xs:nonNegativeInteger" minOccurs="0"/>
<xs:element name="Maximum" type="ExpressionType" minOccurs="0"/>
We have to add the concept of variable in the model.
A variable can a be collected, calculated or external.
A collected variable is a statistical variable collected within a questionnaire for the survey need.
An external variable is a variable
A calculated variable is a variable calculated from others variables including the calculated variables.
An external variable refers to a variable not collected in the questionnaire, but useful for personalization.
For example, it may be a collection wave number for filtering questions, a date to be displayed in the wording of a question, etc.
We also need to reference these three kind of variable to write the conditions (or more generally the controls) in all the questionnaire. That's why it should be interesting to wrap the variables in an element "Variables" as we did whith "CodeLists".
To explain more, an modelization example below....but would it be better to use an "abstract"?, typed the variables? others ???
<xs:element name="Variables">
<xs:complexType>
<xs:sequence>
<xs:element ref="CollectedVariable" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="ExternalVariable" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="CalculatedVariable" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="CollectedVariable">
<xs:complexType>
<xs:sequence>
<xs:element name="Name" type="xs:token"/>
<xs:element name="Label" type="xs:token"/>
</xs:sequence>
<xs:attribute name="id" type="xs:ID" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="CalculatedVariable">
<xs:complexType>
<xs:sequence>
<xs:element name="Name" type="xs:token"/>
<xs:element name="Label" type="xs:token"/>
<xs:element name="Formula" type="xs:token"/>
</xs:sequence>
<xs:attribute name="id" type="xs:ID" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="ExternalVariable">
<xs:complexType>
<xs:sequence>
<xs:element name="Name" type="xs:token"/>
<xs:element name="Label" type="xs:token"/>
</xs:sequence>
<xs:attribute name="id" type="xs:ID" use="required"/>
</xs:complexType>
</xs:element>
Consequence, we add the element to the ResponseType:
<xs:complexType name="ResponseType">
<xs:sequence>
<xs:element name="CodeListReference" type="xs:token" minOccurs="0"/>
<xs:element name="Datatype" type="DatatypeType"/>
<xs:element name="Value" type="xs:anyType" minOccurs="0" maxOccurs="unbounded"/>
<xs:element name="NonResponseModality" type="NonResponseModalityType" minOccurs="0"/>
<xs:element ref="CollectedVariable"/>
</xs:sequence>
<xs:attribute name="simple" type="xs:boolean"/>
<xs:attribute name="mandatory" type="xs:boolean"/>
</xs:complexType>
Our questionnaire methodologist asks for changing 'invite' term to 'probe' in NonResponseModalityType. This term seems to be more appropriate in the methodological documentation.
Probing is a common technique that researchers use in interviewer-administered surveys when respondents initially refuse to answer a question or say they "don't know." Interviewers are trained to use neutral probing techniques -- such as "Would you lean more toward [answer] or [answer]?" or "Just your best guess is fine" -- to encourage valid responses. Probing can be particularly effective when respondents initially hesitate to provide an answer, and it can increase the number of valid responses.
As a reminder, the NonResponseModalityType below:
<xs:complexType name="NonResponseModalityType">
<xs:sequence>
<xs:element name="Label" type="xs:token"/>
<xs:element name="Value" type="xs:anyType"/>
<xs:element name="Invite" type="xs:string" minOccurs="0"/>
</xs:sequence>
<xs:attribute name="firstIntentionDisplay" type="xs:boolean" use="required"/>
</xs:complexType>
A loop is an iteration (according to one criterion) on a group of questions.
-- explicitly define a maximum,
-- define the maximum from a reference to a variable (calculated or external)
-- indicates the table on which to measure the maximum (maximum number of individuals declared in the table). Its seems to be a calculated variable.
We need to know how a variable is represented to transform the Pogues control syntax in XPath syntax.
Could we have an enumeration as:
Hello,
We have to define multiple ClarificationQuestion for a unique Question. For instance in one of our business surveys about very small enterprises (less than 9 people), we can find a table with a line for each employee and a column about the type of contract and if other contract is chosen, a clarificationQuestion to clarify this other contract.
It would be therefore necessary to add a: maxOccurs="unbounded" to the element below included in the QuestionType:
<xs:element name="ClarificationQuestion" type="QuestionType" minOccurs="0"/>
Thanks a lot !
When using pogues-model from another app, the serialization of variables types fails when retrieving variables from pogues endpoint (serialization of global questionnaire works though)
GoToType and FlowControlType are both defined in the model for smooth transition. Next steps are:
Two things in fact must be done :
Survey designer should have the possibility to choose if they prefer to describe the dynamic of their questionnaire (jump directly to a question, a sequence or a subsequence according to a condition on some variables) with a GoTo or an IfThenElse logic. Therefore, PoguesModel should enable this choice thanks to a dedicated parameter. However, DDI can not handle this information.
Thanks for your help !
Remove SequentialIterationType.
First step is to verify that the new modeling for DynamicIterationType
is sufficient (see #48).
PoguesModel is currently managing only one date format : ddmmyyyy.
In order to manage other date and duration questions, we need to change Pogues model in order to allow :
-2 new date format : mmyyyy and yyyy
-duration format composed of years, months, days, hours, minutes (for exemple survey can ask for a duration in years-months or months-days or hours-minutes and so one)
We can also notice that there will be 4 new "numeric" duration : hours, days, months, years. But, as they will be managed with 4 corresponding new measurement units for numeric format, no consequence for PoguesModel.
The new PoguesModel should be ideally compatible with old/current questionnaires in which a DateDatatype only means the date format ddmmyyyy. If not, we should remember to correct json file of these questionnaires before we reuse them.
Thanks a lot !
We want to differentiate several Response format or question type. It's a characteristic of the question.
The question type can be : SIMPLE, SINGLE, MULTIPLE and TABLE
This classification is maybe not the final release but it could help to better manage the serialization.
Can we add this information in the Model ?
To implement : InseeFr/Bowie#2 in link with : InseeFr/Pogues#601
Remove GoToType and GoTo element in ComponentType.
GoTo constructs are superseded by flow controls.
First step is to verify that GoToType is not used in existing questionnaires.
At the moment, we can find in the model:
<xs:simpleType name="DeclarationTypeEnum">
<xs:restriction base="xs:token">
<xs:enumeration value="INSTRUCTION"/>
<xs:enumeration value="COMMENT"/>
<xs:enumeration value="HELP"/>
<xs:enumeration value="WARNING"/>
</xs:restriction>
</xs:simpleType>
But we don't need WARNING and COMMENT. However, we need also a CODECARD modality.
So, the model could be updated like that:
<xs:simpleType name="DeclarationTypeEnum">
<xs:restriction base="xs:token">
<xs:enumeration value="INSTRUCTION"/>
<xs:enumeration value="HELP"/>
<xs:enumeration value="CODECARD"/>
</xs:restriction>
</xs:simpleType>
Pogues naming convention tells that elements are uppercase while attributes are lowercase.
This makes us end with a JSON where keys are inconsistently distributed between lower/uppercase, making the model hard to predict.
One solution would be to use lower camel case everywhere in the model.
We want to specify a specific modality (the most significant case is the modality Don't know
or later Refusal
) attached to the Response
For the moment, a specific modality can be attached to a response only for a question type Single
A SpecificModality
is defined by:
FIRST_INTENTION
and SECOND_INTENTION
(the dynamic behaviour in the generated questionnaire)Do we really want a ComponentGroup
to be as sequence of ComponentType
elements ? That would mean that we need to serialize every component in multiple places : in the sequence it belongs to, and in the component groups where it is referenced, wouldn't it ?
Might we use an xs:token
element like we do for codeListReference
or ifTrue
element, juste to keep track of the id
of the component ?
<xs:element name="ComponentGroup">
<xs:complexType>
<xs:sequence>
<xs:element name="Name" type="xs:token"/>
<xs:element name="Label" type="xs:token" maxOccurs="unbounded"/>
<xs:element name="Declaration" type="DeclarationType" minOccurs="0" maxOccurs="unbounded"/>
<xs:element name="Member" type="ComponentType" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="id" type="xs:ID" use="required"/>
</xs:complexType>
</xs:element>
About the position of a Declaration.
AFTER_RESPONSE position is equivalent to DETACHABLE (i.e. footnote). We don't really have use cases for using both modalities.
We prefer removing AFTER_RESPONSE.
Use case : reuse a question
To reuse a question from the repository, we have to get back only one question (and not the whole questionnaire) in the XML Pogues format.
Can we add a question element as a global element?
We have to manage a hierarchical code lists. Its means that a code may contain a code that may contain a code, etc. (as a recursive description).
One idea: create a type: "CodeType" containing a code typed "CodeType" and refers it in the CodeList:
<xs:element name="CodeType">
<xs:complexType>
<xs:sequence>
<xs:element name="Value" type="xs:token"/>
<xs:element name="Label" type="xs:string"/>
<xs:element name="Code" type="CodeType"/>
</xs:sequence>
</xs:complexType>
</xs:element>
The CodeList description could become:
<xs:element name="CodeList">
<xs:complexType>
<xs:sequence>
<xs:element name="Name" type="xs:token"/>
<xs:element name="Label" type="xs:token"/>
<xs:element name="Code" type="CodeType" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="id" type="xs:ID" use="required"/>
</xs:complexType>
</xs:element>
Maybe should we add a Code directly in the CodeList without creating the CodeType?
We want to use the FlowControl element in two ways:
We prefer describing a Filter at the same level of the other components, i.e in the Questionnaire.
It could mean adding something like that in the Questionnaire:
<xs:element
name="FlowControl" type="FlowControlType" minOccurs="0" maxOccurs="unbounded"/>
We have to identify each data cell. Could we use a cell coordinates system instead of the mappingType element?
Use cases:
Use case: Create a new version of an object in the repository
To do it, we need to refers a "colectica id" (or something like an URI) in Pogues to indicate on which object create the new version. In brief, we neek **a key between an object in Pogues and the same object in the repository.
If there is no key, it means: create a new object.
In addition, we need another information to say if we want to:
At this time, the list of versioning objects is:
Sequence, Question, Declaration (we'll need an Id, see #21), Variable, CodeList
Maybe later: a Response, GotTo ?, Control?
We have to definie multiple NonResponseModality for a unique Response. For instance for the interviewer-administered surveys, we often define these two NonResponseModality:
It would be necessary to add a: maxOccurs="unbounded" to the element below included in the ResponseType:
<xs:element name="NonResponseModality" type="NonResponseModalityType" minOccurs="0"/>
A series pools a set of statistical operations sharing a homogeneous mthodology and a main business characteristic.
It's characterised by:
A statistical operation can be defined as a set of collection, capture, process and data dissemination unique and consistent. Each statistical operation can contain one or several data collection.
It's characterised by:
A data collection (or a data collection campaign) is a survey data collection realization period. Each data collection can contain one ore several questionnaire.
It's characterised by:
Starting hypothesis: all these concepts are created outside Pogues.
We need to add these concepts in the model.
Subject to discuss.
We need to define a measurement unit attached to a NumericDatatypeType.
One is to mangage an enumeration.
It would be a first version to manage a measurement unit because few special cases must be studied (manage a measurement unit depending on a variable, etc.).
Example of enumeration:
€
K€ (or thousands of euros or both?)
%
m²
tonnes
m³
litres
10³Nm³
MWh
kW
hours (maybe we should create a DurationDatatypeType?)
hours / week
days worked / week (jours travaillés/ semaine)
hours / year
days / year
working days
calendar days
Finally, maybe it's better to have a xs:token?
There is no link between the variable in the wrap and the variable created in the response element.
For exemple: if we delete a collected variable in a response, we can't delete it in the wrap, because we don't know what is the variable to delete.
One idea is to have a CollectedVariableReference in the ResponseType (instead of the CollectedVariable) as we did for the code list.
We have to know if a questionnaire is "in progress" or "published"....maybe other status we will define all the process.
Idea: add an attribut "status" to the questionnaire.
If pogues-model could validate the questionnaire model, it could be a great addition, we could detect validation errors at the beginning of the questionnaire workflow.
Project does not compile any more since fields has been renamed in the schema.
See commits:
453f034
7390a49
c88a4f2
dd42888
9f50429
More over, it looks like there's a mapping issue as just replacing Survey by DataCollection as a type in QuestionnaireFactory does not solve this issue: No setter is added to Questionnaire for DataCollection when sources are generated from schema.
The Next
element within the FlowControleType is defined as a xs:token
:
<xs:element name="Next" type="xs:token"/>
This element is used to model nested filters.
It means that we have to reference a FlowControl. It would be easier to type this element as a FlowControlType (and so defining the FlowControlType as recursive element):
<xs:element name="Next" type="FlowControlType"/>
Add an ID to the Declaration element
The same idea than #17
Use case : reuse a sequence
To reuse a sequence from the repository, we have to get back only one sequence (and not the whole questionnaire) in the XML Pogues format.
Can we add a sequence element as a global element?
DateFormatEnum returns patterns for JS (ex: YYYY-MM-DD). The corresponding pattern for java: yyyy-MM-dd
This causes problems when trying to validate data type like DateDatatypeType
We have to know the questionnaire last update date when we save or publish a questionnaire.
We need sometimes several Next for a same FlowControl.
For example, in the TCM (Tronc commun des ménages) (set of questions common to every household surveys), there's a filter on living areas.
The sequence about living areas is asked only to people having another living area.
And within this sequence, we have for example a question only asked for students and another one only for workers.
That's why we shoud redesign:
<xs:complexType name="FlowControlType">
xs:sequence
<xs:element name="Description" type="xs:string" minOccurs="0"/>
<xs:element name="Expression" type="ExpressionType"/>
<xs:element name="IfTrue" type="xs:token"/>
<xs:choice minOccurs="0">
<xs:element name="IfFalse" type="xs:token"/>
<xs:element name="Next" type="FlowControlType"/>
</xs:choice>
</xs:sequence>
into:
<xs:complexType name="FlowControlType">
xs:sequence
<xs:element name="Description" type="xs:string" minOccurs="0"/>
<xs:element name="Expression" type="ExpressionType"/>
<xs:element name="IfTrue" type="xs:token"/>
<xs:choice minOccurs="0">
<xs:element name="IfFalse" type="xs:token"/>
<xs:element name="Next" type="FlowControlType" maxOccurs="unbounded"/>
</xs:choice>
</xs:sequence>
The declarationMode is defined as attribute. We have to define several mode for each declaration.
We should define it as element as we did for the ComponentType (and not as attribute).
<xs:complexType name="DeclarationType">
<xs:sequence>
<xs:element name="Text" type="xs:string"/>
</xs:sequence>
<xs:attribute name="id" type="xs:ID" use="required"/>
<xs:attribute name="declarationType" type="DeclarationTypeEnum"/>
<xs:attribute name="declarationMode" type="SurveyModeEnum"/>
<xs:attribute name="position" type="DeclarationPositionEnum"/>
</xs:complexType>
would become:
<xs:complexType name="DeclarationType">
<xs:sequence>
<xs:element name="Text" type="xs:string"/>
<xs:element name="DeclarationMode" type="SurveyModeEnum" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="id" type="xs:ID" use="required"/>
<xs:attribute name="declarationType" type="DeclarationTypeEnum"/>
<xs:attribute name="position" type="DeclarationPositionEnum"/>
</xs:complexType>
As beans in pogues-model are generated with jaxb, annotations are xml based. This causes problems when using jackson for deserializing/serializing for enums for example. Could be great to have non empty constructors too .
Use case: reuse a declaration within the Pogues questionnaire.
Idea: only refer the declaration in the Component and create it in a wrap
Current tools need formulas written in pseudo-xpath (cf. xforms/Orbeon technologies).
The new tools based on javascript and Lunatic need formulas written in VTL. Depending on the context of the integration of their questionnaire (xforms/Orbeon called V1 or Javascript/Lunatic called V2), the user should be able to specify in Pogues HMI whether they want to specify the formulas of their questionnaire in xpath or in VTL.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.