resostandards / uli-service Goto Github PK
View Code? Open in Web Editor NEWUnique Licensee Matching and Resolution Service
Home Page: https://uli.reso.org
License: Other
Unique Licensee Matching and Resolution Service
Home Page: https://uli.reso.org
License: Other
The Batch Ingest API uses records in JSON format, which are represented by an array of ULI payload items. For example:
POST http://localhost/uli-service/v1/ingest/M0000033
[
{
"MemberFullName": "ohai2 ohai",
"MemberLastName": "ohai",
"MemberFirstName": "ohai2",
"MemberMiddleInitial": "ohai",
"MemberNickname": "ohai",
"MemberType": "ohai",
"MemberNationalAssociationId": "ohai",
"MemberStateLicense": "ohai",
"MemberStateLicenseType": "ohai",
"MemberStateLicenseState": "ohai",
"MemberMlsId": "ohai",
"OfficeName": "ohai",
"OfficeMlsId": "ohai",
"SourceSystemID": "ohai",
"SourceSystemName": "ohai",
"OriginatingSystemID": "ohai",
"OriginatingSystemName": "ohai"
}
]
In order to make it easier for people to test the system, the UI should allow the user to upload a CSV file with membership data matching the ULI Template Schema.
This means that data will need to be converted from CSV to JSON either on the client side, before submitting it to the Batch Ingest API, or the Ingest API will need to be extended to support CSV.
There are two possibilities if the functionality is implemented on the API: a) either the endpoint could be the same (in which case the function would have to check the content type and convert if CSV, otherwise use JSON), or b) a different URL could be used. Using the same URL is probably preferable if this is to be done on the ULI API rather than UI.
The goal in this issue is to set up a Docker environment for a NodeJS and Nginx server to host a React app.
In order to make the project so that any developer can test it locally, we'll need to be able to generate realistic testing data.
With the backend Elastic instance running, we'll want to have a Node/Express service that can connect to the instance and insert the simulated data. We'll want to make sure we simulate this with auth (bearer tokens):
POST /uli-service/v1/<providerUoi>/generate-test-data/[num-records]
Authorization: Bearer <token>
We want the num-records parameter so we can load test the system with varying sizes of member data sets, up to 100,000.
We'd probably have spinner or progress indicator on the frontend to show where the generation and matching process is at.
Let's assume this will go into an index called uli-import
. In order to do that, you can use the Elastic Node Client and POST data into the Elastic index, once running. See documentation on connecting to an Elastic instance.
Add the following fields to the white paper so we can link to it from other information:
uli-service/uli-pilot-ingest.json
Lines 6 to 22 in f71787a
For the initial onboarding process, the ULI System needs to do bulk matching against the data set provided and find the following:
When a new system is being onboarded, we want them to de-duplicate their own data first before trying to match existing bad records against the larger set (in the long run we probably want the same sanity check for live, inbound streaming records).
We probably want to add a "Upload Roster" button in CSV to the page to make it easier for them to work with the system during initial testing. We'll want to check that the CSV they provided matches the schema of the template file.
This page would show all Assigned ULIs, Suggested ULIs, and ones that were still processing in a simple view.
Existing comps can be found here: https://drive.google.com/drive/folders/18nFqTI1ySaWf5i0kaS5QkaI8pvsH4Tik
Due Date: Demo on Jan 7 2022.
In order to onboard ULI Pilot participants and test the matching algorithm, we need to be able to onboard each org one at a time and return the possible duplicates and their confidence scores in a payload.
The matching algorithm that this service uses will also be queried from an interactive search form down the road, and should share the same search functions.
Create an endpoint that can ingest new data from a given Provider UOI.
A request to this endpoint would look something like the following:
POST /uli-service/v1/<providerUoi>
{
"value": [
{
"MemberFullName": "",
"MemberLastName": "",
"MemberFirstName": "",
"MemberMiddleInitial": "",
"MemberNickname": "",
"MemberType": "",
"MemberNationalAssociationId": "",
"MemberStateLicense": "",
"MemberLicenseType": "",
"MemberStateLicenseState": "",
"MemberAddress1": "",
"MemberCity": "",
"MemberStateOrProvince": "",
"MemberPostalCode": "",
"OfficeName": "",
"OfficeMlsId": "",
"OfficeAddress1": "",
"OfficeCity": "",
"OfficeStateOrProvince": "",
"OfficePostalCode": "",
"OrganizationUniqueId": "",
"OrganizationName": ""
}
]
}
Response:
HTTP/2 200 OK
{
"status": "Import Queued",
"numRecords": 134,
"eventQueueUrl": "/uli-service/v1/<providerUoi>/processing",
"method": "GET"
}
Where each item in the value array is a ULI data structure matching the above schema.
As such, schema validation should be performed to ensure that some proper subset of the above dictionary fields is present, and that they are passed in the value array. If true, then respond with a 200, and if not then 400 and tell them which field(s) caused the error. Use ajv or Yup for schema validation.
We should also validate UOIs against the reference sheet. This will mean we'll have a cache of orgs running locally on the API server.
Later on, we'll need the ability to refresh from the UOI sheet. We could take this from the Cert API - just the Sync service and UI. We don't need to do this by the Jan 7 demo though. For now, have a singleton service that populates the cache the first time it's used, and then returns what's there until restarted. If we reuse what's in Cert then we can deal with it that way.
ULI Assigned
, ULI Suggested
, Processing
Once the ingest job has been started, then if the user makes a request to the queue there won't be records in it right away. It will take some time before there are results. We'll need some kind of notification in the future.
Once each record gets pushed, then we'll run a scavenging job on them against what's already in the system.
In order to do this, we'll dynamically form a query based on the information each record contains according to the matching formula, and be able to support a variable set of weights through a separate index that would eventually have its own UI in production.
In the case where a new ULI can be created or an existing match is suggested, we'll use a format similar to the following:
urn:reso:uli:f81d4fae-7dec-11d0-a765-00a0c91e6bf6
which is a uuid that uses the RESO URN namespace (3.4.3).
The API will classify the matching events into event types such as ULI Assigned
, ULIs Suggested
.
ULI Assigned
- in this case, a ULI was assigned since there was no matching record within the configurable confidence threshold. A confidence score will be shown for the item including the fields that were matched on. A ULI can also be assigned through the review process.ULI Suggested
- for each inbound record there may be one or more suggested ULIs pertaining to that record. They will also be shown with their confidence scores and which fields they match on.Processing
- we will also likely want access to the records currently being processed as well, so there should be a third event type of Processing
, but it shouldn't be returned unless asked for.There may be other events added in the future, but this is a good start.
As such, we'll want the root path to also take an optional eventType
parameter:
GET /uli-service/v1/<providerUoi>[/<uli-assigned|possible-match|processing>]
{
"value": [
{
"EventKey": "uniqueKey1",
"EventType": "ULI Assigned",
"ULI": "urn:reso:uli:f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
"OriginatingSystemUsi": "<providerUsi>",
"PathToOriginalRecord": "/uli-service/v1/<providerUoi>/ingest/<key>",
"Score": 0.89,
"ScoringFactors": {
"MemberFullName": 0.55,
"MemberNationalAssociationId": 0.34
},
"ModificationTimestamp": "2022-01-04T00:39:56Z"
}, {
"EventKey": "uniqueKey2",
"EventType": "Possible Match",
"ULI": "urn:reso:uli:f81d4fae-7dec-11d0-a765-00a0c91e6b00",
"OriginatingSystemUsi": "<providerUsi>",
"PathToOriginalRecord": "/uli-service/v1/<providerUoi>/ingest/<key>",
"Score": 0.99,
"ScoringFactors": {
"MemberFullName": 0.85,
"MemberNationalAssociationId": 0.14
},
"ModificationTimestamp": "2022-01-03T00:39:56Z"
}, {
"EventKey": "uniqueKey3",
"EventType": "Possible Match",
"ULI": "urn:reso:uli:f81d4fae-7dec-11d0-a765-00a0c91e6b00",
"OriginatingSystemUsi": "<providerUsi>",
"PathToOriginalRecord": "/uli-service/v1/<providerUoi>/ingest/<key>",
"Score": 0.82,
"ScoringFactors": {
"MemberStateLicenseState": 0.40,
"MemberStateLicense": 0.33,
"FullName": 0.09
},
"ModificationTimestamp": "2022-01-03T00:39:56Z"
}
}
]
}
Where not specifying a type returns all items except Processing
.
The ScoringFactors
data should be accessible by having Elastic explain each query. It's not required for MVP though, just a nice to have.
The ULI Pilot R&D process will produce a white paper containing the following information:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.