openreferral / hsds-transformer Goto Github PK
View Code? Open in Web Editor NEWConvert data into valid Human Service Data Specification (HSDS) datapackages. (Produced by the Open Referral Initiative.)
License: MIT License
Convert data into valid Human Service Data Specification (HSDS) datapackages. (Produced by the Open Referral Initiative.)
License: MIT License
Need to add unit tests for the custom transformation code in the Open211MiamiTransformer
class. ๐
It would be useful to have this packaged and available as a gem so that it can be used in other projects as a dependency rather than solely as a microservice API or command line tool.
We need to create the service_at_locations CSV in HSDS format: https://openreferral.readthedocs.io/en/latest/hsds/reference/#service_at_location
You will need to add this to the locations
section in the mapping.yaml file:
Services:
model: service_at_locations
field: service_id
foreign_key_name: location_id
foreign_key_value: Location_id
We want to use multiple input fields in one output field. For example, in the services
model in HSDS, there is an application_process
text field. In our fixture input data, the services.csv
contains multiple fields that are relevant for the application process: Application form link
, Process for applying by email
, Process for applying by phone``Process for applying online
, Process for walk-ins
. We should have a way to append all of these fields into a single output column.
This is what I'm thinking for an addition to services
in mapping.yaml
:
Process for applying by email
model: services
field: application_process
append: true
Process for applying by phone
model: services
field: application_process
append: true
Process for applying online
model: services
field: application_process
append: true
Process for walk-ins
model: services
field: application_process
append: true
and the append: true
field indicates that these should be appended to the output field, so that all the above are added into the same text string in application_process
.
This will likely involve some regex parsing for dates and times in a string column.
https://openreferral.readthedocs.io/en/latest/hsds/reference/#regular_schedule
Add a Dockerfile
with the current ruby version and build steps (bundle install
primarily).
The other repos have a dash rather than an underscore in their name. Can i change this to hsds-transformer without breaking anything?
Would be super useful to validate incoming data and transformed datapackages against https://github.com/spilio/openreferral-validator
Github pages site that gives overview of the tool, documentation, and relevant links. See this page for the Open Referral Validator for inspiration.
When we collect phone data from organizations, services, and locations, we're taking the whole field and putting it in the number
column in the output phones.csv
. This is a problem because the phone field might actually include extensions or other text. It would be great to at least parse out extensions in the number
column and move them to the extension
column.
See examples in this fixture: https://github.com/switzersc/open_referral_transformer/blob/master/spec/fixtures/output/resources/phones.csv
Right now everything gets piped to tmp
after being transformed, including the results of the tests. This isn't great because files will get overwritten when tests or new transformations are run. We should give the user the ability to tell the transformer to what path their files should be sent.
We recently had a set of sample data that came without a datapackage (and, also, I think some of the relationships between fields may not be carefully defined). So you can test with some of Florida's legal aid providers: https://drive.google.com/drive/folders/1UghRs1TnyrhgoVj-s6P0XqtA0Ic8erEl
(FYI, please only use this data for a transformation test, it's already out of date and incomplete, and wasn't intended for like publishing on the internet.)
Right now we do not dynamically generate the datapackage.json file based on the contents of the transformed data
directory, but we should. This is because transformed data may not include all HSDS models or fields, and may include additional custom fields, so using a static datapackage.json file would not be capture these idiosyncrasies. We want to use the datapackage-rb gem to do this, but this feature is not supported yet so is dependent on a PR to that gem to add the functionality. Once that's done, we can utilize the gem here to create the JSON file.
This should be a POST request that kicks off the OpenReferralTransformer
class to process the provided files. The request can either be a multipart form request that streams that file contents to the API, or the request could reference the web URIs where the files live. If the latter, then the API needs to pass those URIs to the OpenReferralTransformer
which should be able to get them during the transform process.
Does the input data need to have some basic structure in order for us to map? Should there be at least an organizations and locations file?
We need to generate eligibilities from input data. This is usually one text field in a CSV.
Headers: id, service_id, eligibility
See the HSDS reference: https://openreferral.readthedocs.io/en/latest/hsds/reference/#eligibility
For the test data mapping, you can add this to the locations
segment:
Income standard for eligibility:
model: eligibilities
field: eligibility
foreign_key_name: service_id
foreign_key_value: Services
This spec is failing because the services csv in spec/fixtures/input
has an invalid CSV line at the end. We need to add some layer of validation during transformation that removes invalid lines.
An address may be part of an incoming organization row or location row. We should save this as its own row in an addresses.csv with the correct foreign key linking it to the original organization or location.
Using the API with Docker
Hosted the project on Docker & trying API "/transform" method POST from the postman, Getting an error "A mapping file is required".
I have added the
input_path : /spec/fixtures/base_transformer/input
mapping : /spec/fixtures/base_transformer/mapping.yaml
But still getting the error "A mapping file is required".
Hi, it seems the airtable link https://airtable.com/universe/expTMdQFD5r9G6V9Y/open-referral-human-services-template doesn't resolve and brings up a 404 for me? I have tried this while logged in as well.
For context, I am looking at how we align the Leeds LOOP (repository) to openreferral, and adopt it with contributions to support the city's services, information and local 'language'
Right now there are phone numbers present in the locations, organizations, and services data. In HSDS, phone numbers are stored in their own table and linked to their parent record via a foreign key (e.g. service_id
).
We want to pull out the phone numbers from these 3 input files and save them as phone records in a new phones.csv
and maintain the correct foreign key links.
https://openreferral.readthedocs.io/en/latest/hsds/reference/#phone
license this boi
Build is failing on travis CI (though not locally), due to file reference issues. Should move to using the Fake File System gem for tests anyway, which should resolve the problem.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.