beapi-io/spring-boot-starter-beapi

Springboot 'convention over config' starter for API Automation

spring-boot-starter-beapi's Issues

CRITICAL: 0.6.0 notr showing in Nexus anymore

Nexus repo must have reverted to old revision in last day or two as now artifact is missing. Have to reload.

TASK : Avoid processing params and use SpringcontextholderAwareRequestWrapper

Since we are using BOTH RequestMappingHandlerMapping and SimpleUrlHandlerMapping, we can get parameters from the SpringcontextholderAwareRequestWrapper via request.getParameterMap()

This can greatly reduce our processing

BUG : BatchFunctionalTest not properly rolling back

BatchFunctionalTest can install ROLES that do not get properly rolled back causing errors upon next run.

This is an easy fix if you know to just uninstall the authorities from the database but you shouldn't have to do this.

We need to have transaction and rollback in our services

TASK : automate PKEY/FKEY hash

Need to test if request/response variable TYPE is PKEY/FKEY and HASH (do not hash an 'INDEX' - if this isn't in there may need to add)

That way we can return PKEY/FKEY in response and have them sent in a way we can compare to original.

Also need to create a way to change/rotate SALT every 24 hrs
NOTE : randSalt should be implemented as webHook that pushes value to all services which in turn writes it to local properties.

TASK : calls with no 'action' will auto return apidocs for that controller

so while RESTful advocates like /controller/? to map to CRUD rest calls, it is very limiting.

And to date I have just thrown an error with behaviour like this.

But what if I were to return the apidoc data for the entire controller if no action is sent?

Useful and better than nothing. Plus can help with create your api calls and api chaining.

BUG/TASK : Cors registry needs to be handled in implementing app (not starter)

So after much head banging against desk, realizing I have to create a map of cached values that can be passed in properties/cache(more likely cache) to implementing app so they can build a configurationSource for cors

@Override
protected void configure(HttpSecurity http) throws Exception {
    http.csrf().disable().cors().configurationSource(configurationSource()).anyRequest().requiresSecure();   
    ...
}


private CorsConfigurationSource configurationSource() {
      UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
      CorsConfiguration config = new CorsConfiguration();
      config.addAllowedOrigin("*");
      config.setAllowCredentials(true);
      config.addAllowedHeader("X-Requested-With");
      config.addAllowedHeader("Content-Type");
      config.addAllowedMethod(HttpMethod.POST);
      config.addAllowedMethod(HttpMethod.GET);
      config.addAllowedMethod(HttpMethod.PUT);
      config.addAllowedMethod(HttpMethod.DELETE);
      source.registerCorsConfiguration("/**", config);
      return source;
    }
 }

Just have to send the map of networkgrp allowed origins per 'controller' and create the rules via antMatcher. This right here just became a WHOLE SEPARATE project... UGHG!

Documentation: Server endpoints

Need to have list of available SDK server endpoints and what they are for.

• ApiDocs
• Properties
• Hooks
• User
• Connector
• JwtAuthenticationController

TASK : Params Functional Test

need a functional test to test params via:

• get params (uri)
• form/post params
• and combination of all

SDK : Hook Exchange Service

Need to add a HookExchangeService for handling CRUD functionality for all endpoints.

Have already added routing so just have to make sure an exchange is there to process it.

TASK: Apidocs not sending mockdata

apidocs need to send mockdata so we can build out the 'api call example' and potentially add back in sandbox.

Right now the 'api call' example is incorrect

TASK : automatic issue generation

Since I am not seeing feedback for the plugin, I'm going to integrate an automatic issue creation tool in case anyone has problems with build. This will mitigate 'shyness' and give me the reports I want

Need to investigate a few options and see best way to do this but since most of GITHUB is api driven, should be very simple.

SDK : create beapi_cache.yaml

right now have default values for caches but as people scale, these need to be overwritten.

Can come up with an 'autocalc' but who are we kidding... these are Java devs, we need a separate properties file. :)

BUG : Demo project needs to have YamlPropertySourceFactory to allow for ENV properties

Hi,

I'm not sure if this is the correct place to post issues regarding documentation, but it is currently deployed on gh-pages from this repo so I thought I'd post it here.

The instructions for configuring the demo project reference a state of the config repo that contained beapi_api.yaml in the root of the repo, but this no longer seems to be the case. I assumed I needed to copy the config folder for the dev enviornment.

Similarly, I noticed that boot.sql was deleted recently (in this commit, but the current installation/setup instructions don't reflect this change.

TASK: Simplify Bootstrapping Users

Right now we are creating config for ADMIN/TEST users and then manually bootstrapping from properties

Proper way to do this would be with a service.

May be able to create separate UserBootstrap class that can be added to Application. Need to investigate this.

TASK : encrypt/decrypt keys for RESPONSE/REQUEST so DB doesn't have to

the more I look at encrypting/hashing the PKEY/FKEY/IDX, the more I ponder encryption as an option and replacing hashing altogether.

Streaming cyphers are faster and perform similarly (at least for our purposes) and using encryption would make it easier for testing and rotating hash (or encryption in this case).

The excuse previosuly has been that it would be easier for testing but didn't want to go down this route because it opens privacy concerns.

This is why we would need to make this a VERY OPEN and WELL DOCUMENTED set of processes to maintain trust if we go down this route.

Providing security means providing TRUST.

TASK : Use custom Servlet rather than Filter

Look into using a custom servlet rather than using filter; should be faster and more functional and reduce collisions with othger filters.

All in all just better solution if I can do it this way.
https://www.baeldung.com/context-servlet-initialization-param

TASK : Build 'receivesList'/'responseList' using getIOSet as part of cache

we are using getIOSet in RequestInitializationFolter to BUILD the requestList/responseList with each endpoint call

This should be built at runtime as part of ApiDescriptor as a finished list able to be reference by:

• ROLE (authority)
• or if role not found 'permitAll' (assuming ROLE passes checkNetworkGrp() for endpoint)

TASK : Scaffolding Command Script/Service

Creation of consolidated script for all scaffolding

beapi scaffold-controller full.class.name
beapi scaffold-domain full.class.name
beapi scaffold-connector full.class.name

FEATURE: X-LINK-RELATIONS

Unlike HATEOAS that AUTOMATICALLY sends back a ton of additional data that you may not want, I wanted to make this optional upon REQUEST.

You can send a HEADER ('X-LINK-REQUEST') for the related links and it wouyld provide a 'concatenated'response:

        "response": {
         ...
        },
        "links": {
            "/accounts/12345/deposit":{
                    "POST",
                   "REQUEST":{"id"},
                   "RESPONSE":{"id","amount","acctNo"}
            }
        }

Naturally like everything else, this would be related to the requesting users ROLES in their token.

This makes us more feature compliant with HATEOS while not being as bloated as HATEOS

BUG : duplicate call due to bad filter handling

filter is calling interceptor twice but second call is ignored. Still this is major issue and needs to be addressed

TASK : Build tests for APIDocs per different roles

APIdocs are autogenerated based on ROLE so USER should NOT see ADMIN endpoint, etc

Need to build tests to make sure this is working properly

IMPROVEMENT (SDK) : Detect device associated with token and keep longterm record

To better secure users, we need to record 'device' used with token and to compare upon in initial REQUEST in order to see if token was 'hijacked'

https://github.com/ahand/mobileesp/blob/master/Java/UAgentInfo.java

TASK : Rename IO State files to 'connectors'

most everyone is calling these 'connectors' now so we should probably rename these.

Everything should work the same; just need to rename directory and change where directory points (plus change documentation reference).

TASK: DAILY Quartz/CRON to clean old tokens

Need DAILY quartz on MASTER that will clean EXPIRED tokens in DB

IMPROVEMENT(SDK) : Use CDN for loading/storing yaml IO State files

Just realized I can simplify a few things by loading properties files via a CDN. This should be added to the SDK

All will download same file from CDN at runtime but PARENT can reload and push (via webhook) to CHILD servers

TASK : long term cache for webhooks

since I can't (and don't want to) use a DB in starter, we can store in cache and backup to disk the webhooks. This reduces need for developers to implement controllers/services and everything can be internalized without DB.

Just have to implement into existing cache functionality.

Bug (minor) : IO State files for static controllers are not being read properly

Only reading in one file right now when we need to read in everything from dir.

see 'parseResource' in IoStateService

BUG : missing functionality for apidocs

need to have a boolean toggle for BATCH/HOOKS in apidocs that tells whether this functionality is available to this user based on their roles in this networkGrp

TASK : beapi_security.yml

Going to need to separate out all properties based on role:

• ROLE
  • throttleLimit

etc

TASK : associate request detailt with JWT token

Need to check request details to make sure token hijacking isn't occurring (yes, this CAN be spoofed but this provides an additional layer that they have to spoof as well when doing a MITM; this allows us to log out user and pre-emptively warn at the very least)

String ip = request. getRemoteAddr()
String userAgent = request.getHeader("User-Agent");

NOTE: will have to have associated table ('a given User has one /many RequestDetail and RequestDetail has one User')

Found this on Stackoverflow. This is sloppy and should be converted to compiled regex...

       String  browserDetails  =   request.getHeader("User-Agent");
        String  userAgent       =   browserDetails;
        String  user            =   userAgent.toLowerCase();
    
        String os = "";
        String browser = "";

        log.info("User Agent for the request is===>"+browserDetails);
        //=================OS=======================
         if (userAgent.toLowerCase().indexOf("windows") >= 0 )
         {
             os = "Windows";
         } else if(userAgent.toLowerCase().indexOf("mac") >= 0)
         {
             os = "Mac";
         } else if(userAgent.toLowerCase().indexOf("x11") >= 0)
         {
             os = "Unix";
         } else if(userAgent.toLowerCase().indexOf("android") >= 0)
         {
             os = "Android";
         } else if(userAgent.toLowerCase().indexOf("iphone") >= 0)
         {
             os = "IPhone";
         }else{
             os = "UnKnown, More-Info: "+userAgent;
         }
         //===============Browser===========================
        if (user.contains("msie"))
        {
            String substring=userAgent.substring(userAgent.indexOf("MSIE")).split(";")[0];
            browser=substring.split(" ")[0].replace("MSIE", "IE")+"-"+substring.split(" ")[1];
        } else if (user.contains("safari") && user.contains("version"))
        {
            browser=(userAgent.substring(userAgent.indexOf("Safari")).split(" ")[0]).split("/")[0]+"-"+(userAgent.substring(userAgent.indexOf("Version")).split(" ")[0]).split("/")[1];
        } else if ( user.contains("opr") || user.contains("opera"))
        {
            if(user.contains("opera"))
                browser=(userAgent.substring(userAgent.indexOf("Opera")).split(" ")[0]).split("/")[0]+"-"+(userAgent.substring(userAgent.indexOf("Version")).split(" ")[0]).split("/")[1];
            else if(user.contains("opr"))
                browser=((userAgent.substring(userAgent.indexOf("OPR")).split(" ")[0]).replace("/", "-")).replace("OPR", "Opera");
        } else if (user.contains("chrome"))
        {
            browser=(userAgent.substring(userAgent.indexOf("Chrome")).split(" ")[0]).replace("/", "-");
        } else if ((user.indexOf("mozilla/7.0") > -1) || (user.indexOf("netscape6") != -1)  || (user.indexOf("mozilla/4.7") != -1) || (user.indexOf("mozilla/4.78") != -1) || (user.indexOf("mozilla/4.08") != -1) || (user.indexOf("mozilla/3") != -1) )
        {
            //browser=(userAgent.substring(userAgent.indexOf("MSIE")).split(" ")[0]).replace("/", "-");
            browser = "Netscape-?";
                  
        } else if (user.contains("firefox"))
        {
            browser=(userAgent.substring(userAgent.indexOf("Firefox")).split(" ")[0]).replace("/", "-");
        } else if(user.contains("rv"))
        {
            browser="IE-" + user.substring(user.indexOf("rv") + 3, user.indexOf(")"));
        } else
        {
            browser = "UnKnown, More-Info: "+userAgent;
        }
        log.info("Operating System======>"+os);
        log.info("Browser Name==========>"+browser);

BUG : Spring CorsFilter using RequestMappingHandlerMapping; cannot change

For all API's, we are automating and autopopulating with SimpleUrlHandlerMapping but Spring is defaulting to RequestMappingHandlerMapping with hardcodes values with annotations rather than DYNAMICALLY POPULATING FROM A PROPERTIES FILE.

We need to write our own CORS filter and inject in Security Filter Chjain in place of their CorsFilter and allow it to :

• use RequestMappingHandlerMapping for public Apis
• use SimpleUrlHandlerMapping for secure Apis

TASK : BootstrapConfig ???

Need to have a config that checks that all values have been changed in :

• application.properties
• beapi_db.yaml

and any other settings files

This will throw appropriate error (prior to hitting any tests) and exit build.

Literally, this SHOULD only need to be checked after a new install but its good to have something like this handy when you screw up :)

NOTE : Going to hate having to test this because this means having to screw up my environment... ALOT!

BUG : CachedResult not properly working with cache.returns data

CachedResult resides in Request initialization filter and does not properly take into consideration results for ROLES. Needs to be adjusted to consider requesting ROLE.

Also should consider automating Controller automation for cache an return data off a common object and role:

LinkedHashMap data = generateReturnData(Object object, String Role)
return data

TASK : IO State prop for cacheOn

Need to have a property in each IO State URI to allow a toggle for 'CacheOn'

This would allow developers/maintainers to set whether we 'cache' return sets or not.

The reasoning is that some endpoints may ALWAYS want LIVE DATA.

TASK: Add ROLE_TEST for bootstrapped user

Bootstrapped user should have a limited role mainly for testing. This role can also be used for automated testing.

Bootstrap user with ROLE_TEST instead of ROLE_USER and edit IOSTATE/connectors to have this role as a limited baseline (or default to permitALL)

TASK : business logic can be services as well as controllers

adding in functionality to allow services to be declared for business logic as well as controllers.

Literally would just have to add a TYPE to IO state [CONTROLLER,SERVICE] and dynamically call service and route in BeapiController (much the same way I do with controller)

This provides alternative way of building, flexibility and way to build internal static calls (which I need) for the tool.

This way I can also add localizd IO state files to the project without adding controllers.

TASK : public uri's handled by client

so rather than having the endUser have to put their public uri's in 'apiProperties.reservedUris', lets just assume that ALL uri's implemented with '@RequestMapping' (or variations thereof) are public... or at least not handled by the starter (thus we assume they are public).

We can check this by checking:

• if they are not cached with the api metadata (from IO state file)
• if the were properly requested (/v1.0/,/b1.0/,/c1.0/,etc)

Localized public apis will have no reason to create IO state files.

PLUS... if they are annotated with @controller, we can create a 'publicApis' field in the cache when we are creating the mappings as they will not map to a state file.

This will have to be tested.

IMPROVEMENT : method to generate return object/cachedResult

should consider automating Controller automation for cache and return data off a common object and role:

LinkedHashMap data = ApiObjectService.generateReturnData(Object object, String Role)
return data

This simplifies ALL controller generation and aids in automation.

We can more easily template off this and autoGernerate controller as well

TASK : Rate Limiting needs to be NETWORKGRP/role/user based

networkGrp has a rateLimit which concatenates with ROLE rateLimit and even USER rateLimit

In beapi_api.yml

    rateLimit: {'ROLE_USER':1000}
    dataLimit: {'ROLE_USER':1000000}

if ROLE rate limit does not exist, defaults to rateLimit for networkGrp. If networkGrp rate limit does not exist, no rateLimit is applied (UNLIMITED)

NOTE: would need to add two new fields to user table to accomodate for this.

the open apis (ones that do not require sign-in or token) will not have these checks applied and therefore will be unlimited.

This promotes the ability to SELL additional access to api both for ROLE(company wide) and for user(individual)

TASK : Post 0.7, create separate branches for config files

in spring-boot-starter-beapi-config repo, we need the config stored in separate branches for version like we do all code.

Lets start this startying in 0.7

As part of this process, lets go through config and determine what is deprecated and start cleaning things up and consolidating.

TEST : Build tests for ThrottleCacheService

Need to build tests for ThrottleCacheSrvice to make sure rate limiting is working properly

MINOR ISSUE : status code should be 204 on empty/null responses

empty/null responses should set statusCode to 204 (not 200)

BUG : ACCEPT mimetype '*/*' needs to default to content-type else JSON

I spit on */* as an ACCEPT type!!! If you don't know WHAT THE HELL you are sending me, then I don't know if I'm accepting it! Standards need to lock this shit down and stop creating engineering docs for 'elbows that turn 360 degrees'!

DEFAULT THIS SHIT to the content-type (if thats what we are sending back, then thats what they should be sending in 90% of cases) else JSON. UGH!

TASK : logger.devnotes

BRILLIANT FEATURE!!!

give dev ability to run with logging value of 'devnotes' (much like INFO/WARN/DEBUG) so you can get 'developer notes on why a HTTP STATUS code may have occurred'

Better yet... rather than logging, can be emailed letting them know of error and possible solutions

Needs to be internationalized but this can avoid TONS of question and further automate platform and make easier to use.

MINOR ISSUE: add error cacthing for fallback RequestMappingHandlerMapping

badly formatted URI's are falling to RequestMappingHandlerMapping where (if not found) throw a long error:
java.lang.IllegalStateException: Cannot call sendError() after the response has been committed

we need to build functionality to at LEAST catch this error so logs don't fill due to badly formatted URI's

Task : automated apidocs

adding in api docs as a service to autogenerate api docs based on role in token.

Thus (unlike SWAGGER), docs will only show endpoints token has access to.

This is in compliance with OWASP API security

BUG : PRECHAIN method not being changed to GET for continuing calls in chain

ChainExchangeService is not properly changing the PRECHAIN request method to GET for followup API calls in chain and thus it throws ERROR for all PRECHAIN calls (as it should).

This is easily fixed by doing a detect and changing method to GET for all following calls in a prechain call

TASK : convert ulist to util/UriObject

implementors may want to use uList and thus we need to convert to the UriObject we already have built; have no clue why we used uList (probably just a placeholder to test prior to using object)

The following classes are affected:

• RequestInitializationFilter
• ApiInterceptor
• BeapiRequestHandler
• ExchangeService
• ChainExchangeService
• BatchExchangeService

Fix in THAT ORDER!

TEST : Build tests for TraceCacheService

Trace needs to have tests built to see if it is working properly

TASK : implement caches through class so we can change properties on the fly

need to test changing of cache variables on the fly (ie diskExpiryThreadIntervalSeconds)

Since these variables can be changed on the fly without a restart, would be invaluable to make this part of caching functionality (and also very hazardous if people get security incorrect)

These will never be an issue for caching because only GET calls are cached.