openreferral/hsds-transformer

Convert data into valid Human Service Data Specification (HSDS) datapackages. (Produced by the Open Referral Initiative.)

Open Referral

hsds-transformer's Issues

Add Miami custom transformer tests

Need to add unit tests for the custom transformation code in the Open211MiamiTransformer class. 🙀

Turn into a gem

It would be useful to have this packaged and available as a gem so that it can be used in other projects as a dependency rather than solely as a microservice API or command line tool.

Add service_at_locations csv

We need to create the service_at_locations CSV in HSDS format: https://openreferral.readthedocs.io/en/latest/hsds/reference/#service_at_location

You will need to add this to the locations section in the mapping.yaml file:

Services:
  model: service_at_locations
  field: service_id
  foreign_key_name: location_id
  foreign_key_value: Location_id

Add ability to append multiple input fields to one output column

We want to use multiple input fields in one output field. For example, in the services model in HSDS, there is an application_process text field. In our fixture input data, the services.csv contains multiple fields that are relevant for the application process: Application form link, Process for applying by email, Process for applying by phone``Process for applying online, Process for walk-ins. We should have a way to append all of these fields into a single output column.

This is what I'm thinking for an addition to services in mapping.yaml:

Process for applying by email
  model: services
  field: application_process
  append: true
Process for applying by phone
  model: services
  field: application_process
  append: true
Process for applying online
  model: services
  field: application_process
  append: true
Process for walk-ins
  model: services
  field: application_process
  append: true

and the append: true field indicates that these should be appended to the output field, so that all the above are added into the same text string in application_process.

Save regular schedule in HSDS format

This will likely involve some regex parsing for dates and times in a string column.

https://openreferral.readthedocs.io/en/latest/hsds/reference/#regular_schedule

Dockerize

Add a Dockerfile with the current ruby version and build steps (bundle install primarily).

Naming convention: hsds-transformer?

The other repos have a dash rather than an underscore in their name. Can i change this to hsds-transformer without breaking anything?

Automate validation with OpenReferralValidator

Would be super useful to validate incoming data and transformed datapackages against https://github.com/spilio/openreferral-validator

Create project page with docs

Github pages site that gives overview of the tool, documentation, and relevant links. See this page for the Open Referral Validator for inspiration.

Smart phone number creation

When we collect phone data from organizations, services, and locations, we're taking the whole field and putting it in the number column in the output phones.csv. This is a problem because the phone field might actually include extensions or other text. It would be great to at least parse out extensions in the number column and move them to the extension column.

See examples in this fixture: https://github.com/switzersc/open_referral_transformer/blob/master/spec/fixtures/output/resources/phones.csv

Add ability to specific output path

Right now everything gets piped to tmp after being transformed, including the results of the tests. This isn't great because files will get overwritten when tests or new transformations are run. We should give the user the ability to tell the transformer to what path their files should be sent.

Other sample data to test

We recently had a set of sample data that came without a datapackage (and, also, I think some of the relationships between fields may not be carefully defined). So you can test with some of Florida's legal aid providers: https://drive.google.com/drive/folders/1UghRs1TnyrhgoVj-s6P0XqtA0Ic8erEl

(FYI, please only use this data for a transformation test, it's already out of date and incomplete, and wasn't intended for like publishing on the internet.)

Use datapackage-rb gem to infer and generate datapackage.json from transformed data

Right now we do not dynamically generate the datapackage.json file based on the contents of the transformed data directory, but we should. This is because transformed data may not include all HSDS models or fields, and may include additional custom fields, so using a static datapackage.json file would not be capture these idiosyncrasies. We want to use the datapackage-rb gem to do this, but this feature is not supported yet so is dependent on a PR to that gem to add the functionality. Once that's done, we can utilize the gem here to create the JSON file.

Add HTTP API to expose services

This should be a POST request that kicks off the OpenReferralTransformer class to process the provided files. The request can either be a multipart form request that streams that file contents to the API, or the request could reference the web URIs where the files live. If the latter, then the API needs to pass those URIs to the OpenReferralTransformer which should be able to get them during the transform process.

What are the basic data structure constraints for this to work?

Does the input data need to have some basic structure in order for us to map? Should there be at least an organizations and locations file?

Add eligibilities.csv output

We need to generate eligibilities from input data. This is usually one text field in a CSV.

Headers: id, service_id, eligibility

See the HSDS reference: https://openreferral.readthedocs.io/en/latest/hsds/reference/#eligibility

For the test data mapping, you can add this to the locations segment:

Income standard for eligibility:
  model: eligibilities
  field: eligibility
  foreign_key_name: service_id
  foreign_key_value: Services

Failing spec: It "converts a services file into valid HSDS services data"

This spec is failing because the services csv in spec/fixtures/input has an invalid CSV line at the end. We need to add some layer of validation during transformation that removes invalid lines.

Save addresses from Org and Locations tables as Address records

An address may be part of an incoming organization row or location row. We should save this as its own row in an addresses.csv with the correct foreign key linking it to the original organization or location.

API Not Working (Using the API with Docker)

Using the API with Docker

Hosted the project on Docker & trying API "/transform" method POST from the postman, Getting an error "A mapping file is required".

I have added the
input_path : /spec/fixtures/base_transformer/input
mapping : /spec/fixtures/base_transformer/mapping.yaml

But still getting the error "A mapping file is required".

Airtable link

Hi, it seems the airtable link https://airtable.com/universe/expTMdQFD5r9G6V9Y/open-referral-human-services-template doesn't resolve and brings up a 404 for me? I have tried this while logged in as well.
For context, I am looking at how we align the Leeds LOOP (repository) to openreferral, and adopt it with contributions to support the city's services, information and local 'language'

Save phone numbers as individual phone records

Right now there are phone numbers present in the locations, organizations, and services data. In HSDS, phone numbers are stored in their own table and linked to their parent record via a foreign key (e.g. service_id).

We want to pull out the phone numbers from these 3 input files and save them as phone records in a new phones.csv and maintain the correct foreign key links.

https://openreferral.readthedocs.io/en/latest/hsds/reference/#phone

Make license

license this boi

Fix the build

Build is failing on travis CI (though not locally), due to file reference issues. Should move to using the Fake File System gem for tests anyway, which should resolve the problem.