tonyfujs/mdlibtoddh

mdlibtoddh's Issues

Fix Africa mapping for geographical coverage

No proper mapping for Africa (currently mapping to Sub-Saharan Africa). To be fixed

map_chr() error in get_ddh_datasets_list()

map_chr() throws an error for columns with NULL values (Ex: data_class) in get_ddh_datasets_list().

Currently doesn't handle populating field_wbddh_source

Suggest treating 'studyType' as free text vs controlled vocab, with no validation.

I recommend treating studyType as free text that's not validated.
(1) We import metadata created by other institutions and NSOs that use the same template, but in different languages. They can also adjust the vocabulary to suit their requirements. If we have a limited list of terms that we check for we'll keep running into problems and constantly need to update.
(2) The list is out of date, it hasn't been updated to match all and recent study types. As is, any modern/recent survey types are mapped to 'Other' such that original study type information is lost and unavailable to users. Rather, let's take the metadata as is.

update_existing_dataset() doesn't handle version_date

field_wbddh_version_date is currently removed because multiple input formats are not handled correctly
(Add to type of problem encountered presentation)

get_ddh_datasets_list() - UUIDs

get_ddh_datasets_list():

• uuids to be replaced by harvest_sys_id
• check that harvest_sys_id is not empty

Multiple data_collection not supported in DDH

Script fails because it tries to push multiple values to DDH.
Example: Microdatalib ID4252

Update add_data_classification()

Data classification is currently hard coded in the function. Needs to pull from master lookup once master lookup is stable

Contacts details exclude descriptive information when exported to DDH

The contacts field in DDH excludes the descriptive information about the contact provided (see image below),that may be helpful to users. Wondering whether there a way this information might be captured as well?

In DDH the contacts field displays only the email addresses:

Add a function to check for duplicate when adding a new dataset.

clean md_placeholder

Currently contains unused machine_names. They should be removed

get_ddh_records_status

update to use harvest_src_id instead of UUID for the merge

Convert version date into the correct date format

add_data_classification: Fix hard coded values

359, 358, and 1194

Mapping fails for field_wbddh_source

ID# 7181
Value failing to map:
this series of savvy mortality surveillance system manuals, guides, and other documents is available at the measure evaluation\\nweb site at: http://www.cpc.unc.edu/measure/leadership/savvy.html

flatten_md_metadata

hard coded variable names. to be fixed

add trimming to input values from DDH

Emails with extra spaces are flagged as invalid. Implement automatic trimming.

Error in adding a new dataset

License has now become a required field, whereas Microdata uses terms of use.

add_internal_data.R - machine_names lookup

Modify script to pull only from DDH API, not local files: This will only be possible once the API is fixed (currently displays inconsistent machine names)

Refactor create_json_dataset() and create_json_resource() using ddhconnect::create_json_body()

Remove line of code form add_internal_data.R

Remove once API is fixed:
fields$ddh_machine_name[fields$ddh_machine_name == "field__wbddh_depositor_notes"] <- "field_wbddh_depositor_notes"

flatten_md_metadata

review code carefully. Currently not full proof

Fields to change to free text

Version description field does not accept greater than/less than <> symbols although field is open text.

format_md_metadata

hard coded variable name. to be fixed

Hard coded resource title

title present in two functions create_json_resource and create_json_attach. ready for a bug! TO BE FIXED

Change license mapping business rules

data classification logic is flawed

Data classification is common to both dataset and resource while, there should be two different fields following their own logic. This is fine for now given how microdata currently works, but might be the cause of future problems.

Changing reference IDs

reference IDs might change sometimes... There is nothing currently in the package that is designed to catch this. Needs to be build eventually.

field_wbddh_other_acknowledgement not handle by flatten function

handling of Study type is broken

Field has been changed to free text. Still handled like a controlled vocab and values are not being passed to DDH

Change date format

Review collection dates

Review how collection dates (start / end) are pulled. This is a nested field in Microdatalib, not in DDH. Potential bug in extract_md_metadata()

fix indexing from get_metadata output

test_created_dataset.R relies on the previous version of get_metadata(), change indexing to reflect updated version of the function

example

# previously
node_metadata$field_contact_email$und$value

# required change
node_metadata$field_contact_email$und[[1]]$value

add function that checks for duplicates

Add check_duplicates() functions in the get_ddh_records_status() that prints the duplicated dataset nodes if duplicates are present in DDH.

Error: object 'deto_add' not found

This occurs when I try to add datasets, I'm running the code from the master branch

Cases of invalid Microdata input

Build a check_input() function that takes a Microdata response + a master table of expected response as input, and check the validity of the response.
Ideally that would be done using a json schema from Microdata, but this is not currently available.

Check for dataset created without a resource after each dataset is created

Remove the dataset if it is created this way.

add_constant_metadata: Fix hard coded values

Ampersand regarded as invalid character in DDH Title field

3: error: Client error: (406) Not Acceptable - error: Title Field contains invalid content.
Title: Time Preference & Cognition Survey 2009 - 2010

Incomplete dates in microdata

Some dates have just the year or just the year and the month in microdata. Add a function to complete it with the first date of the year or month.

handle break line characters in text variables

Should break line characters be removed? Check with tech team.

get_ddh_records_status(): Hard coded URL

Root URL is hard coded: Update so it depends only from ddhconnect

create_json_attach doesn't handle multiple resources

Update function to handle multiple resoruces

create_json_resource fix hard coded values

resource_type is hard coded

Multiple field concatenated with no spaces

Multiple fields are concatenated without spaces. Makes it hard to read in DDH (for instance contact emails)

add workflow_status to the create_json_resource()

Add workflow_status=published by default

get_ddh_datasets_list

Re-write to leverage ddhconnect

How to handle syncing of multiple resources + add new ones without creating duplicates

Would need to store the source resource ID in DDH too...

populating contact emails

The current default is [email protected]
Should it stay that way, or should it be populated with the original contact email(s) from Microdata?
Talk to Matthew, Mehmood. See Cathrine's email June 22, 2017

map_md_metadata() handling of multiple values input

Implement feature to handle microdata fields that return multiple values