mabiesen/rails_universal_api

attempting a dry-er, more developer-controlled approach to cross-project api usage.

Rails Universal Api

About

Features of Project

Intended primarily to be run in a docker local to the project at hand.

• be DRY in your code efforts across projects. No longer are the days of including 30 gems, no longer should you be concerned that the gem will not update as rapidly as the api, or require dependencies you do not have
• use intuitively named urls to acommplish your goals. No more concerns regarding interpolation: once the endpoint is discovered and submitted to the code base, you can strictly use traditional parameters
• there is no longer a need to worry about nested hashes either. Data submitted as a 'flat' hash can be converted to a nested object, provided that a body_template(json template) has been supplied to the endpoint
• benefit from community contribution! As endpoints are discovered the clients and endpoints will be added to the codebase. Filters-creation options for retrieved endpoint data coming soon.
• utilize a fork of this project for your company, you can add all internal clients and identify endpoints. Could assist in testing, troubleshooting, basic_understanding of a given API client, etc.
• use the database of endpoints to generate quick scripts of endpoint calls via curl, httpie, faraday, etc. (todo, this may be tricky due to varied permissioning)
• use the UI to hit endpoints and receive real time feedback on validity (at least as regards class type expections)

Ruby version

2.6.0

Usage

Credentials Setup

Pending a better methodology (coming very, very soon):

Check out /config/initializers/endpoint_clients. This directory contains initializers for our endpoint clients (github, your-internal-server, etc). Take a look at what environment variables are referenced and set them up and/or send them to the docker image at runtime.

Local

Set up in traditional rails fashion

bundle install
bundle exec rake db:create
bundle exec rake db:migrate

Depends on a running postgres instance, tested successfully with 10 and higher.

Docker

To run the service in docker

docker-compose up --build

From there you should be able to hit localhost:3003/

Available Routes

The following JSON endpoints exist for this project:

/list_endpoints - returns a list of endpoints /call/:client_tag/:request_name - call a given endpoint /validate_params/:client_tag/:request_name - validate parameters for a given endpoint /validate_param/:client_tag/:request_name - validate a single param for a given endpoint, intended for use in live form validation.

Example - /list_endpoints

Basic example:

http://localhost:3000/list_endpoints

^^ fetches all endpoints

Example filtering for endpoints with a client_tag 'like':

http://localhost:3000/list_endpoints client_tag:git

^^ would return all endpoints with a client_tag containing the word 'git'

Example filtering for endpoints with a name 'like':

http://localhost:3000/list_endpoints request_name:pull

Example - /call/:client_tag/:request_name

This route is the bread and butter of the application, it is how we reach out to external endpoints (github, facebook, etc).

Basic example:

http://localhost:3000/call/github/get_pull_requests

^^ get all pull requests of any status from github

Example filtering for status:

http://localhost:3000/call/github/get_pull_requests state:closed

Example - /validate_params/:client_tag/:request_name

This route exists primarily as a development convenience, it provides a way to validate your parameters will work as expected. It will also provide whollistic reivew of UI entries prior to form submission.

Basic example:

http://localhost:3000/validate_params/github/get_pull_requests owner:mabiesen
# would return error because repo is not supplied, but repo is not optional

http://localhost:3000/validate_params/github/get_pull_requests owner:mabiesen repo:<this_repo> stant:closed
# would return error message indicating stant is not a parameter

http://localhost:3000/validate_params/some_api_provider/some_reqeust  owner:mabiesen repo:<this_repo> date:marcus
# would return error message indicating marcus is not a date

Example(singular!) - /validate_param/:client_tag/:request_nam

Same validations as /validate_params, but only one param is supplied at a time.

Primarily intended for UI live form validation.

Testing

from circleci cli while in home directory

cirlceci local execute

Contributing

Steps for adding clients

1. add a /config/<client_name>.yml to store url information
2. add a /config/initializers/<client_name>.rb to initialize your client.
3. add the client to the client map in /app/models/endpoint.rb
4. environment variables consumed by the initializer should be placed in /.env.local

Steps for adding endpoints

1. rails g migration adding_some_endpoint_name
2. fill out the migration
3. run rails db:migrate RAILS_ENV=development

Versioning a new release should be done as follows:

• Patch Update - Fix application related concerns
• Minor Update - Change to an existing endpoint or client (Example: 0.7.122 -> 0.8.0)
• Major Update - New/Delete endpoint or client, route addtions (0.7.122 -> 1.0.0)

NOTE: endpoints may not be added until a client has been added

TODOS

DRY Data filtering

TODO

Often when API responses are received they are filtered for specific logic required for business operations. No one needs alllllllll of the information an API like the github pull request api provides, so often we filter for what we need at the business layer. But because this is occuring at the business layer within an application, these common filters are not usually granted cross-project applicability.

Permissioning

TODO

Should have proper authentication. API credentials should be stored better, presently just a .env.local file.

Skinny Project Templating

TODO

Some projects do not require the robustness that this service offers, and many developers prefer not to add an extra networking layer to their concerns. A templating mechanism will be created to allow one to generate http request scripts for use in other projects.

UI Endpoint Execution

TODO

This UI would template out parameters in table format and offer AJAX data validation prior to data send.

rails_universal_api's Issues

For internal Endpoints: Alert when route or controller file has been modified in git

This would help maintain the integrity of the endpoints in the database.

Would just be a cautionary popup and/or return from api.

Popup would include ability to view most recent file diff to help determine whether the endpoint would be affected

Add yammllinter

good practice, even if yamls are not frequently utilized

Call to action in the Readme - Let people request

The Readme should immediately be update with a form for endpoint submission requests.

Requests should be in the form of asking users to submit issues with the following information:

• what client is it? Provide url to domain and/or api documentation url (preferred)
• what endpoints would you like to see?
• ultimately, what is the information you are trying to get? This is necessary for the future creation of filters.

Add coverband

Add coverband and redis to project, coverband requires redis

https://github.com/danmayer/coverband

coverband is how many times a line i the code is run, whereas simplecov(codecov) is related to spec coverage

BUG - extra params returning zero values when it should be nil

It looks like we are not removing nils early enough in our process to prevent the conversion from nil to zero.

irb(main):007:0> extra_params = erb.extra_params({q: 'chicago'})
=> {"q"=>"chicago", "days"=>0, "unixdt"=>0, "unixend_dt"=>0, "hour"=>0}

Need to modify endpoint request builder extra params method

mvp should include spec testing of all value types in the endpoint request builderclass. good opportunity to practice looped rspec pattern

Add translation to curl requests

Ability to translate an endpoint to a curl request.

Create ability to apply Endpoint Filters

This is a critical enhancement that will allow project performance to be more comparable to gems

EndpointFilter would be a new activerecord object

There would be 1 filter to an endpoint, an endpoint can have many filters.

The filter would be used to parse received json for specific variables.

The filter would be accessible from the endpoint submission page

There would be a route to obtain a list of filters for an endpoint

An option would be added onto the endpoint controller action to apply a filter

Proper methodology for storing API keys

maybe use ruby encrypted secrets yaml strategy

maybe some other upload-once-and-never-see strategy

ultimately this is not hooooooooooribly important
but at present new developers are encouraged to bad practices, our methodology should change for this reason.

Add social media endpoints

Everyone wants to play with social media, all the time, good to have

Add Translation For Postman Requests

Ability to translate to the json format used in postman collections.

Postman is highly utilized, being able to output endpoints as scripts for Postman would greatly enhance the value of this project.

Create script with live user IO to assist with endpoint addition

Given:

1. The necessary headers are known
2. The means of authentication is known
3. The name of the api is known
4. Any required default param is known
5. The hostname is known

We should be able to create a template for client Faraday initializers and config, additionally we should be able to tell the user how to set credentials.

In the event that a portion of the params are not known, or if a specific means of implementation has not been developed for an auth, the script may still be able to assist with creation and inform the user there is still work to be done.

API requests should return report of url, params, etc utilized. Both to the application and to External APIs.

Like benchmark, this great to have. Great for troubleshooting or assistance with reconstructing the endpoint call in another language.

Regarding External Apis:

• Add a method to the faraday client to ONLY expose the details of the request, not execute it.
• Add a route to ONLY obtain request details
• In the controller, capture relevant detail from the response body

Regarding Data Sent to rails_universal_api from the user:

• im certain theres a method in the 'request' object that can give me everything I want here. Create independent controller method and a route to expose request data

Add Views for Listing Endpoints and Using

Add a UI to view all endpoints and to call endpoints.

JSON AJAX requests will be used for form validation, no higher-than-dirt logic is to be included in javascript usage.

datatables js package will be used to make tables pretty
will likely use bootstrap css

I envision the endpoint submission page containing many things:

• Endpoint detail - not everything must be displayed, but all must be capable of viewing
• Form - live validations for inputs, prior to submit validates all inputs as a collective, submits. All ajax
• Returned-Data Capture Window - view returned data, I picture this below the form
• Endpoint Building Window - While filling in the form, this window would show you how the path and arguments are being constructed.
• Option to view endpoint initializer and config - when issues occur, have the means of diagnosis at hand.
• Optional output returned data to file

Eventually: Option to apply a filter

Evenutally: Page to list clients, page to view clients and related endpoints (with option to generate zip for faraday, since we have all config).

Possibly: Option to view the full, unfiltered faraday response(not just the body and status), good for troubleshooting

---

MVP for a first PR:

• listing page is made and uses datatables. A methodology is used to keep parameters from getting squished.
• listing page has links to endpoint pages
• endpoint page has form with validations and in-page showing of returned data