boshrin/or-match

Postgres based identity match engine

or-match's Issues

Implement required Attribute Keyword

If a required attribute is missing, the request would be considered invalid.

Move Documentation to a Wiki

The one gigantic README.md is hard to navigate and maintain. Move the documentation to a wiki somewhere, probably depending on where the project ends up.

Also review second person vs third person style. (The README switches about halfway through from "you" to "one".)

Forced Reconciliation Request

Rename Project Again

Current proposal: Reference ID Match (Service)

Allow substr And token Searches in Canonical Matches

It might make sense to allow substr (#16) and token (#23) matches to be canonical. One way to do this would be to have a hierarchy of exact/token/substr, and whichever (if any) is first defined for an attribute would be used in determining canonical matches. Another way would be to allow canonical matches to specify search types (like potential matches), though it's not clear that it makes sense to have different canonical rules for the same attribute with these search types.

Automatic matchgrid Generation

Add a utility to generate matchgrid based on the current configuration.

Require Configured SOR Only

Currently, the match engine will allow any value for SOR, assuming the client is authorized for it. So if a client is authorized for '*', it might send 'sis', 'SIS', or 'student' all meaning the same thing. Perhaps require SORs to be defined in the [sors] section of config.in. This could then allow 'sis' while rejecting the other 2 as undefined.

Installation Instructions

Sequence For reference_id

Currently, reference_id is generated as a UUID. Offer a configurable option to generate it from a sequence (which can be cast to a varchar so the matchgrid column doesn't need to change).

Request Pending/Resolved Match(es)

Utility to Manage matchauth

Request Inventory of Requests

Use SELECT FOR UPDATE?

It may make sense to use SELECT FOR UPDATE to handle a read/write request to avoid concurrency issues.

Match on sor+sorid throws error

Review CIFER ID Match API

Review API documentation for missing features/requirements and implement (or create tickets) as necessary.

Utility to Define Attributes

Create a utility of some form to define attributes as specified in config.ini.

Dictionary Search type

This would handle (eg) John vs Jack.

See postgres documentation on Dictionaries (§12.6).

• http://www.censusdiggins.com/nicknames.htm
• http://deron.meranda.us/data/nicknames.txt
• http://usgenweb.org/research/nicknames.shtml
• Eric Pierce re perl module USF found

Reassign Reference Identifier

casesensitive Flag Ignored

HR record in Mixed Case does not match STU record in ALL CAPS. (See notes for details.)

Put Against Existing Record Throws Duplicate Key Violation

This should already be handled by conversion to an update attributes request, but apparently isn't.

Update Match Attributes Request

Implement Update Match Attributes Request.

Pending Matches Not Skipped

searchDatabase() does not ignore records where reference_id is null (ie: pending resolution).

Token Search Type

Implement a tokenization search type, that would (eg) treat "anna marie" and "anna" as the same by ignoring everything from the space onwards. This might be doable with SQL's substring(string from pattern for escape).

Search Only Request via GET

Check For Existing Reference ID

Especially when using a sequence, check for an existing record with a newly assigned reference ID, and if one exists try again (up until some reasonable max). This will simplify bulk loading existing data that may exist in a conflicting range.

Soundex Search Type

This may or may not be a good/useful idea, but is supported by more databases.

Reference License In File Headers

Individual files should reference the BSD license (as mentioned in the README and LICENSE).

Request/Delete Current Values

Auto-Define sor and sorid

These two attributes are mandatory, and so should always be defined. (Perhaps inserted into config during parsing?)

Obtain SOR Records

Should SOR Be Case Insensitive?

Should "hr" and "HR" be considered equivalent?

Notification on Fuzzy Match

Support notification to an email address (per SOR?) when a new potential match is recorded (202, not 300).

Substring Search type

Like the exact search type, but only operates on a substring (eg: first 5 characters, last 5, second through fifth, etc).

Skip Attributes Consisting of All Blanks/Zeroes

If an attribute consists of only blanks or zeroes (and possibly non-alphanumeric characters according to the configuration), skip it for matching purposes. eg: lastname="", dob="0000-00-00", ssn="000000000".

Perhaps have a configuration option to disable this on a per-attribute basis.

Add Support For EmailAddress

Should invalidates Support Modifiers?

Currently invalidates operates on a simple string comparison. Should it honor modifiers such as casesensitive and alphanum?

Return Reference ID as Identifier

Offer an option to return the reference ID as an identifier in the API response, with the identifier type configurable.

Join Reference Identifiers

Log Levels

Add support for none / connections / logic / trace (sql)

Support Externalized Authentication

Allow authentication via $REMOTE_USER. ie: ignore matchauth:apikey, but still use matchauth for authz.

Link Attribute Columns to SOR+SORID

Add the ability for an attribute to match against SORID. Something like

[attribute:employeeid]
; Match this attribute against SORID where SOR is "HR"
search['sor'] = 'HR'

Move Configuration Into Database

Almost all of config.ini should be moved to the database (ie: database driven configuration).