tonyfujs / ddhconnect Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Might be helpful for non curation team users but only expose users to UI names rather than machine names?
download_resource uses the URL from dkanr_setup() and not the URL passed as the parameter. This is because it makes a call to get_resource_url() and fix_download_url() which do not take the URL as a parameter in the CRAN version.
get_resource_nids() exists in dkanr. Delete existing get_resource_nid() from ddhconnect, and export the dkanr function to ddhconnect
Use of grep() to ignore formatting differences also introduces incorrect mappings due to partial matches. Ex: 'Albania' maps to the tid of 'Albanian' in map_tids().
Add parameter in create_json_body() to make the default status as "published".
Order function parameters so that credentials and root url are always at a consistent position.
Add ready to use json data to use in the request's body
from Alp
id = 118383
updated_data = c(
"title" = "Togo - Firm Surveys for Comparing Personal Initiative Training to Traditional Business Training 2013-2016 TEST",
"workflow_status" = "published"
)
updated_data_json = create_json_body(values = updated_data, node_type = "dataset")
update_dataset(nid = id, body = updated_data_json)
error
Error: parse error: trailing garbage
3{ "microdata": { "d
(right here) ------^
happens in:
Line 23 in fc3ee71
(Check the ckanr package for one approach of writing tests for APIs)
create_dataset() vs create_resource()
update_dataset() vs update_resource()
Code is currently the same.
Need a place for users adding data to reference machine names to DDH form names. Also need explanation for machine names that you can use in search_catalog
(such as nid
).
With the move to the cloud, these functions will break without authentication
New function for downloadable resources which will save files. Otherwise, users will have to located the
url file string field_link_api$und$url
and declare their own file name. Also, might be good to print the citation (since we want to encourage use).
example
current
library(Rcurl)
resource_metadata = get_metadata(nid = 94974)
download.file(url = resource_metadata$field_link_api$und[[1]]$url, destfile = "WDI.zip")
suggested
# extract the existing file name?
resource_download(resource_nid)
Currently: Doesn't take credentials argument. Should be updated.
Is there a machine name to indicate whether the resource is actually the data. Might help to have a function that filters the resource results by download/query tool to make the actual data more obvious. Identifying the data file/query tool/link is not clear when there are several resources (ex: WDI is ~70).
example
current
indicators_resources = get_resource_nid(nid = wdi_nid)
# need to locate the tid for dataset resource_type
tid_download = 986
for (resource_id in indicators_resources){
resource_metadata = get_metadata(nid = resource_id)
if (resource_metadata$field_wbddh_resource_type$und$tid == tid_download){
print(resource_metadata$title)
}
}
library(rcurl)
resource_metadata = get_metadata(nid = 94974)
download.file(url = resource_metadata$field_link_api$und$url, destfile = "WDI")
suggested
indicators_resources = get_resource_nid(nid = wdi_nid)
get_data(resource_ids, access_type = c("download, "query tool"))
Does this function need to stand independently? If so, it needs clarification for users about what can pass in datatype
parameter (i.e. single strings or combination of filters, etc.).
ddhconnect/R/get_datasets_list.R
Line 36 in fc3ee71
The resource_json_format_lookup is missing some machine_names and the corresponding json_template values. Need to update the lookup to match the current form fields and values.
missing
Function that takes a list of named arguments, check whether the values are valid, and build a valid JSON string.
Doesn't take credentials argument. To implement.
Badly conceived. should rely on lower level search_dhh
The default value for the node_type parameter is 'dataset'. If the node type is 'resource' and if you updating the node without passing a parameter for node_type, then the node type is changed to dataset.
include if-statement which accounts for 0 returned results along with corresponding error message
Need to add information about which fields take lovs and which do not. Cumbersome for user to locate all lov machine names in "get_lovs()"
add readxl
to imports in DESCRIPTION because it's used in map_metadata_excel()
ddhconnect/R/map_metadata_excel.R
Line 15 in 71ec88f
dkanr::attach_file_to_node()
current test files are not updated with latest json format
might need to consider adding a check?
The creation of datasets and resources can still use tid values.
The map_tids() function breaks for some values. For instance:
metadata = c("field_topic" = "Poverty", "field_wbddh_data_class" = "Public" ) map_tids(metadata) field_topic field_wbddh_data_class "376" "378"
field_wbddh_data_class should have value "358"
It may be safer to build this function using inner_join()
Currently, the output is formatted as list of lists, including the "und"
formatting. We might want to consider the output being a cleaner, more standardized format especially since multiple results can be returned at once.
example
current
[[1]]$field_contact_email
[[1]]$field_contact_email$und
[[1]]$field_contact_email$und[[1]]
[[1]]$field_contact_email$und[[1]]$value
[1] "[email protected]"
[[1]]$field_contact_email$und[[1]]$format
NULL
recommended
named list format or dataframe?
$star
[1] "wars"
create_dataset()
and create_resource()
both do not display informative error/warnings when a field is populated incorrectly.
example
current
Error: Client error: (406) Not Acceptable - error: An illegal choice has been detected. Please contact the site administrator.
suggested
{"form_errors":{"field_topic][und":"An illegal choice has been detected. Please contact the site administrator.","field_wbddh_data_type][und":"An illegal choice has been detected. Please contact the site administrator."]}
Cache it as a data file after the first call, with an option to update. Can also do this for get_fields(), get_required_fields() and get_lov_fields()?
Examples of unused credentials parameter so far: get_metadata(), get_resource_nid(), search-catalog(), search_ddh(), get_datasets_list().
Using search_catalog()
with lov field names has a confusing work flow. Currently, the user must map their search string to the corresponding tid with get_lovs()
. Might be simpler for user if we automated looking up the string and returning the tid.
example:
current input
filters = c("field_wbddh_data_type" = "295")
suggested input
filters = c("field_wbddh_data_type" = "geospatial")
In the error message for a wrong field name or value, show the most similar field_name/value in create_json_body() and map_tids(). Right now, the error message displays all the valid values.
Once this is done: remove err_handler() code from ddhconnect
Returns invalid cookie and token. Don't understand why...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.