tonyfujs / mdlibtoddh Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
No proper mapping for Africa (currently mapping to Sub-Saharan Africa). To be fixed
map_chr() throws an error for columns with NULL values (Ex: data_class) in get_ddh_datasets_list().
I recommend treating studyType as free text that's not validated.
(1) We import metadata created by other institutions and NSOs that use the same template, but in different languages. They can also adjust the vocabulary to suit their requirements. If we have a limited list of terms that we check for we'll keep running into problems and constantly need to update.
(2) The list is out of date, it hasn't been updated to match all and recent study types. As is, any modern/recent survey types are mapped to 'Other' such that original study type information is lost and unavailable to users. Rather, let's take the metadata as is.
field_wbddh_version_date is currently removed because multiple input formats are not handled correctly
(Add to type of problem encountered presentation)
get_ddh_datasets_list():
Script fails because it tries to push multiple values to DDH.
Example: Microdatalib ID4252
Data classification is currently hard coded in the function. Needs to pull from master lookup once master lookup is stable
Currently contains unused machine_names. They should be removed
update to use harvest_src_id instead of UUID for the merge
359, 358, and 1194
ID# 7181
Value failing to map:
this series of savvy mortality surveillance system manuals, guides, and other documents is available at the measure evaluation\\nweb site at: http://www.cpc.unc.edu/measure/leadership/savvy.html
hard coded variable names. to be fixed
Emails with extra spaces are flagged as invalid. Implement automatic trimming.
License has now become a required field, whereas Microdata uses terms of use.
Modify script to pull only from DDH API, not local files: This will only be possible once the API is fixed (currently displays inconsistent machine names)
Remove once API is fixed:
fields$ddh_machine_name[fields$ddh_machine_name == "field__wbddh_depositor_notes"] <- "field_wbddh_depositor_notes"
review code carefully. Currently not full proof
Version description field does not accept greater than/less than <> symbols although field is open text.
hard coded variable name. to be fixed
title present in two functions create_json_resource and create_json_attach. ready for a bug! TO BE FIXED
Data classification is common to both dataset and resource while, there should be two different fields following their own logic. This is fine for now given how microdata currently works, but might be the cause of future problems.
reference IDs might change sometimes... There is nothing currently in the package that is designed to catch this. Needs to be build eventually.
Field has been changed to free text. Still handled like a controlled vocab and values are not being passed to DDH
Review how collection dates (start / end) are pulled. This is a nested field in Microdatalib, not in DDH. Potential bug in extract_md_metadata()
test_created_dataset.R
relies on the previous version of get_metadata()
, change indexing to reflect updated version of the function
example
# previously
node_metadata$field_contact_email$und$value
# required change
node_metadata$field_contact_email$und[[1]]$value
Add check_duplicates() functions in the get_ddh_records_status() that prints the duplicated dataset nodes if duplicates are present in DDH.
This occurs when I try to add datasets, I'm running the code from the master branch
Build a check_input() function that takes a Microdata response + a master table of expected response as input, and check the validity of the response.
Ideally that would be done using a json schema from Microdata, but this is not currently available.
Remove the dataset if it is created this way.
3: error: Client error: (406) Not Acceptable - error: Title Field contains invalid content.
Title: Time Preference & Cognition Survey 2009 - 2010
Some dates have just the year or just the year and the month in microdata. Add a function to complete it with the first date of the year or month.
Should break line characters be removed? Check with tech team.
Root URL is hard coded: Update so it depends only from ddhconnect
Update function to handle multiple resoruces
resource_type is hard coded
Multiple fields are concatenated without spaces. Makes it hard to read in DDH (for instance contact emails)
Add workflow_status=published by default
Re-write to leverage ddhconnect
Would need to store the source resource ID in DDH too...
The current default is [email protected]
Should it stay that way, or should it be populated with the original contact email(s) from Microdata?
Talk to Matthew, Mehmood. See Cathrine's email June 22, 2017
Implement feature to handle microdata fields that return multiple values
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.