rohannagar/thunder

REST API application that manages user databases

A fully customizable user management REST API.

Read the Documentation

Features • Running Locally • Running on Kubernetes • Client Libraries • Changelog

Features

• Connects to AWS DynamoDB or MongoDB
• REST API for CRUD (Create/Retrieve/Update/Delete) operations
• Built-in email verification with AWS Simple Email Service
• Server-side password hashing
• Support for both Basic Auth and OAuth 2.0
• Customizable user properties with validation
• Customizable email message contents
• Generated OpenAPI (Swagger) specification
• Metrics and healthchecks
• Official Docker Image and Helm Chart
  • Docker Hub
  • GitHub Container Registry
• Multiple native client libraries

Running Locally

Fork this repo on GitHub. Then, clone your forked repo onto your machine and navigate to the created directory.

$ git clone YOUR-FORK-URL
$ cd thunder

Compile and package the source code with Maven.

$ mvn package

Start up local dependencies (DynamoDB and SES) in the background so that Thunder can perform all functionality.

$ node scripts/tools/run-local-dependencies.js &

Run the packaged jar.

$ java -jar application/target/application-*.jar server config/local-dev-config.yaml

Thunder should now be running on localhost port 8080!

Running on Kubernetes

The official Thunder Docker image is published on Docker Hub.

Thunder is deployed through a Helm chart. See the scripts/deploy/helm/thunder directory for steps on deploying through Helm.

Further Documentation

Full documentation can be found on ReadTheDocs. For Thunder development documentation, refer to the wiki for information on how to build and write tests.

thunder's Issues

Add Email Field to PilotUser

Users should register with an email.

Return response when DynamoDB is down instead of throwing exception

If Thunder is unable to access DynamoDB for any reason, it throws a bunch of exceptions and eventually the server returns a 500 response code instead of something more appropriate.

INFO  [2016-01-24 01:23:29,715] com.amazonaws.http.AmazonHttpClient: Unable to execute HTTP request: Connection refused
! java.net.ConnectException: Connection refused
! at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_51]
! at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) ~[na:1.8.0_51]
! at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_51]
! at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_51]
! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_51]

Upgrade Java Version to JDK 11

Now that Dropwizard 1.3.0 is out and officially supports JDK 1.9, we can migrate Thunder to use Java 9. This would be a big step and we should probably move to using modules.

Java 10 is also now out, and moving to Java 9 would mean the Java 10 upgrade is a lot easier. In Java 10, we can do cool stuff like use var to declare local variables.

Marking this as low priority, but will keep this on the radar.

https://dzone.com/articles/a-practical-guide-to-java-9-migration
dropwizard/dropwizard#2318

Python SDK

Describe the problem that this feature would solve

Would be great to have a Python SDK for interacting with Thunder.

Verify email should return an HTML page to the user

When a user clicks on the verification link through their email thunder should return a success HTML page for the user. Currently a PilotUser is returned providing no indication that the link worked as far as a user is concerned.

Class Specific JavaDocs

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

Currently we have JavaDocs in place for methods throughout the project, however it would be beneficial when generating JavaDocs for potential contributors to have descriptions of each class.

Additional Information:

Class commenting practice can be seen in the source for Java's included packages. This would also be a place to include class specific information like Authors or License information.

http://www.docjar.net/html/api/java/util/Collections.java.html
http://www.dummies.com/programming/java/how-to-use-javadoc-to-document-your-classes/

Swift SDK

Would be nice to have a thunder SDK that handles requests and user account interaction

Use HTML Templates

Description:

Instead of hard-coding the HTML verification success page, we should allow for templates and for the template to be specified in the configuration file.

Additional Information:

Consider moving from GitLab to Codefresh for Docker builds

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

Right now we have an entire mirror on GitLab for Docker builds. Codefresh is a popular CI/CD tool for building Docker images that we should consider.

Additional Information:

https://codefresh.io/docker-tutorial/java_docker_pipeline/

Publish Client and Javadoc to Maven Central

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

We should publish the Java client on Maven Central instead of JitPack for v2. This is more standard and will let people use the Java Client without including an additional repository in their POM.

Additional Information:

https://maven.apache.org/repository/guide-central-repository-upload.html
https://blog.idrsolutions.com/2015/06/how-to-upload-your-java-artifact-to-maven-central/

Follow these guides:
https://central.sonatype.org/pages/ossrh-guide.html
https://central.sonatype.org/pages/requirements.html
https://central.sonatype.org/pages/apache-maven.html

Publish JavaDocs

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

We should publish JavaDocs using Github pages. We can build javadocs and put them in a docs/ folder on master. Then we just need to turn on Github pages in the repo settings.

Additional Information:

Upgrade Retrofit from v1.9 to v2.x

Retrofit v2 has been out for some time, and will require a lot of changes in the client.

Customize Email & Verification Success

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

The email message sent on POST /verify should be configured through the configuration. At the very least, it needs to contain an option on what the domain name is so that the link in the email is correct.

Additional Information:

This issue was combined with #122. The verification and success pages will be pulled in as raw HTML instead of configurable properties to prevent restricting the structure of the page itself.

Remove Dropbox Properties in PilotUser

Due to the direction Pilot is going in, Dropbox is no longer necessary.

JUnit 5 Migration

Now that JUnit 5 has been out for some time, we can start thinking about upgrading.

Some resources:

https://junit.org/junit5/docs/current/user-guide/
http://www.baeldung.com/junit-5-migration
https://www.petrikainulainen.net/junit-5-the-ultimate-resource/#migrating-from-junit4
http://www.codingrevolution.com/junit-5-migration/

Improve Logging

There needs to be much more detailed logging everywhere.

Extensive v2 Documentation

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

Documentation needs a full overhaul just prior to v2.0.0 release. v2 is intended to be used by anyone in the open source community, so documentation in the README and the Wiki should be very good and cover all usage scenarios.

Additional Information:

Set up Travis to deploy tagged commits to Github Releases

Step 1 in the containerization and deployment process.

Travis should continue to build and report status as normal, but additionally on tagged commits (commits that are intended to be a new version of Thunder), the application jar should be added to the release.

This is more of a consolidation into Github as a platform, and not relying on AWS S3 to hold our releases. We should use JitPack going forward for including the api and client modules in other projects, instead of relying on S3. This will improve the development process for Lightning, as AWS keys will no longer be needed to build.

Use correct logic with custom URL placeholder

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

Continuation of #150. The user should be able to (optionally) customize the string that is used as a placeholder for the URL. Here are the following scenarios and what should happen:

1. No urlPlaceholderString defined and no custom email pages defined

   Use the default CODEGEN-URL and the default HTML/Text files

2. Custom urlPlaceholderString (not equal to CODEGEN-URL) and no custom email pages

   Log warning and use CODEGEN-URL anyway with the default files

3. No urlPlaceholderString defined and one or both custom email pages defined

   Use default CODEGEN-URL and the custom pages or one custom page and one default

4. Custom urlPlaceholderString (not equal to CODEGEN-URL) and one custom email page

   Use the custom placeholder for the one page that is custom, use CODEGEN-URL for the other

5. Custom urlPlaceholderString (not equal to CODEGEN-URL) and both custom email pages

   Use the custom placeholder for both custom pages

Additional Information:

Fix deprecated Mockito methods

With the Mockito upgrade from 1.x to 2.x, there are a lot of deprecated methods that need to be replaced.

JitPack build broken

I can work around this by downgrading the maven check style plugin.

v1.1.0 is currently broken and can't be pulled from JitPack.

Convert Guava Optional to Java 8 Optional

In ThunderAuthenticator we are currently using the Guava Optional class. This can be converted to the Java 8 Optional class.

HTTPS Support

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

At some point it would be nice to sport HTTPS for communication between the server and client. We could allow users to utilize certificates purchased from a certificate authority or their own self signed certificate.

HTTPS allows encrypted communication to the server. This would be necessary in a production environments where users are sending credentials to the server. In many cases these password would be hashed but this still opens us up to dictionary/rainbow table attack on our users password hashes.

HTTP also allows users to verify the identity of Thunder servers. (see trust chaining below)

I would also like to point out that if a future goal is to have Thunder as a Service (TaaS) then a lack of HTTPS support would certainly scare aways a large number of potential users.

Additional Information:

https://en.wikipedia.org/wiki/HTTPS
https://en.wikipedia.org/wiki/Chain_of_trust
https://blog.instantssl.com/https/seo-advantages-switching-https/

Move DynamoDB table name to config file

Currently the dynamo table name is hard-coded in PilotUsersDao.java. This should be moved to config.yaml and injected into the DAO to allow for easier table name change.

Support Updating User Email

It is not possible to change the user email right now due to restrictions in Dynamo, plus the code does not support it.

The PUT method in the resource will need to take the existing email as a QueryParam, and the update method in the DAO will need to determine if the email needs to be updated, and call a delete then a put in Dynamo.

Add Failure Cases to Integration Tests

Currently, the integration tests only test the successful path. We should also run tests to ensure that malformed requests or bad requests return what we expect.

Add option to request CAPTCHA on user creation

Describe the problem that this feature would solve

One of the standard ways to prevent bot signups is to require the end user to answer a CAPTCHA sent from the server. Thunder does not currently have this functionality.

Describe the solution you'd like

There should be a configurable option to enable CAPTCHA on POST /users. Ideally, when enabled, the flow would look like:

1. Client application sends a POST request with the user data
2. The server responds with an appropriate error code asking the client to re-try with a solution to the given CAPTCHA challenge
3. The client displays the CAPTCHA challenge to the user, they answer
4. The client retries the request with the answer to the CAPTCHA

Additional context

I have never worked with CAPTCHA before, so I may not know all of the limitations or how the protocol should actually work.

Make Javadoc Conform to Format

Description

Continuation of #164. We need to agree on a format for all Javadoc and make the current documentation match the format.

Additional context

@NickEckert can you come up with a format and then I can review and we can agree on one?

Links:
http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html
https://docs.oracle.com/javase/1.5.0/docs/api/java/util/Formatter.html

Bootstrap Script Improvements

This issue relates to:

• Bug
• Suggestion
• Feature Proposal

Description:

Update scripts/tools/bootstrap.sh to manually install dependencies if apt-get or brew fail.

Additional Information:

Github Release Deploy doesn't work

The application jar is not published to Github releases on a tagged commit because Travis only does a test, not a package.

Travis needs to package on a tagged commit.

Also, for easier releases we need to change the maven-release-plugin to only install and not attempt to publish the artifacts.

Verify User Email Account

Thunder needs to validate an email by sending an email to a newly registered user or newly registered email for an existing user. Once Thunder can validate that a user has access to the registered email the registration process can finish.

The process will go something like this:

1. Start the validation process by creating a new user and denoting the associated email as unverified. If the user already exists then update the email and denote it as unverified.
2. Generate a unique hash and store it alongside the unconfirmed field in the DB.
3. Append the hash as a query param to a url that points to a validation endpoint on Thunder and send the url to the address you want to validate.
4. When Thunder receives an email with a hash, parse the hash and confirm the user is associated with the given hash.
5. Mark the email as valid in thunder.

Note that a user account can be viewed as unverified if the email is not verified. This can be useful for tracking down accounts that are registered but never verified.

The end goal is to allow account recovery and general contact with our user base.

• Create Email class that contains: String address, boolean verified, String verificationToken
• PilotUser is updated to use Email email instead of String email
• PilotUser still has the String email address as the Dynamo primary key
• Ability to generate a hash to store in the DB
• A class to handle sending of emails. This should generate the URL of the form thunder.sanctionco.com/verify?email=XXX&token=XXX
• Create a new resource class to handle verification requests at the endpoint /verify. Perhaps called VerificationResource. Create a POST method that takes query params email and token
• The new endpoint verifies the token matches the one stored for the user, and marks the email as now verified

AWS Deployment Templates

We need a set of AWS deployment templates that will:

On first deployment:

• Create ASG (AutoscaleGroup) that deploys the Thunder jar on boot-up
• Create DyanmoDB with autoscaling

On update:

• Do a rolling upgrade of the EC2 instances with a new jar
• Does not modify the DynamoDB instance

Validate Email

When creating a new user, the email of that user should be validated to ensure it is a proper email address.

Use wildcard type parameter for Class

Description

Quick fix for some compiler warnings. Should use Class<?> instead of Class in PropertyValidationRule and PropertyValidator.

Additional context

https://stackoverflow.com/questions/4337313/why-is-class-preferred-to-class

Unsupported dynamic URLs in custom HTML pages

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

PR #135 added custom html pages but introduced a problem by preventing the inclusion of dynamic URL's in the html page. This makes it impossible for a user receiving a verification email to connect back to Thunder.

Additional Information:

Cut release v1.1.1 & v1.1.2

This release should fix the JitPack build, and also should test and make sure that Travis will deploy the jar to Github Releases.

Before the release is cut, issue #49 needs to be resolved.

Integration Test Script Enhancements

• Include script in Travis build
• Convert script to Node.js
• Use DynamoDB and SES Local on Box
• Cleanup if failure happens by calling DELETE /users
• Make each test case a function and call in list

Related Work:

• #30 Email configuration and script improvements
• #31 DynamoDB configuration
• #50 Replace Python Scripts with Node.js Scripts

Enable (Optional) User Property Validation

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

Something we can enable is property validation on User creates and updates. What this will include is:

• Allow user to specify if they want property validation in the config file.
• If they want validation, they also have to list the property names and types in the config file.
• On a User create/update, Thunder will validate that the User object coming in the POST or PUT has the same number of properties as listed in the config, as well as they have the same property names and the value is the correct type.

This would be a huge feature that would allow users of Thunder to be more confident that the data they are creating in the database looks consistent. It will be optional in case some users want the flexibility to put any number of properties in the User object.

Additional Information:

The config section may look like:

propertyValidation: true
properties:
  - name: myFirstProperty
    type: string
  - name: mySecondProperty
    type: list

New Logo

Description

We need a new logo for Thunder.

Additional context

Add More User Deserialization Tests

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

We should add more tests in the UserTest to test and a make sure that different types of properties can be deserialized. Included in this issue is making the test objects generic (i.e. don't reference facebookAccessToken).

Additional Information:

Marking this as high priority so that we can ensure future changes don't break this.

Better DynamoDB Error Handling

We need to also be catching AmazonServiceExceptions and logging these exceptions more appropriately.

Deployment Documentation

We need documentation around deploying Thunder to Kubernetes. This should be done just before releasing v1.2.0

Feature: DB Integration: Azure Cosmos DB

Description:

We can allow for different database providers in Thunder. There should be an ability to specify which DB provider is desired in the config, and then Thunder will use the correct implementation. We need to also implement Cosmos DB.

Additional Information:

https://docs.microsoft.com/en-us/azure/cosmos-db/

https://github.com/zeit/cosmosdb-server

https://docs.microsoft.com/en-us/java/api/overview/azure/cosmos-readme?view=azure-java-stable

Move Validation Logic into PropertyValidator Class

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

The validation logic particularly in the UserResource is repeated across the methods. We can move the validation logic into its own class and make the code easier to read. This would also make the tests easier to understand and more logical.

Additional Information:

Generalize User Object

Looking forward to v2.0:

1. The PilotUser object needs to be re-named to simply User or ThunderUser.
2. All properties other than email and password will move into a map or list. This way it will be easy to have any number of additional properties on a User object. The email and password will always be required and are part of the logic of Thunder, so these should remain.
3. It may be possible to use polymorphic configuration to achieve dynamic User object configuration? However, it may not apply. In that case, we can make a map of <String, Object>, where the key is the property name and the value is the value. This should work in theory, but will require experimentation.
4. We should not need to change the DAO.

Turn on/off email verification

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

Allow email verification to be enabled or disabled in the configuration.

What this means:

• New configuration option for disabling email
• If disabled, do not require all other email configuration
• If disabled, do not register the VerificationResource to Jersey

Open questions:

1. Do we need a different User object that contains just a String email instead of the Email object with the verified and verificationToken fields?

Additional Information:

Rename Test Docker Image to Edge

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

Currently, we build a new Docker image on every push to master and tag it with the tag test on Docker Hub. We should rename the tag to edge to signify that it is an "edge" build, and if you use that image you will be getting the latest features, but they may not be fully ready since it is not an official release.

Additional Information:

Change Configuration YAML to Camel Case

This issue relates to a:

• Bug
• Suggestion
• Feature Proposal

Description:

To follow correct convention, we need to use camelCase instead of hypen-style in the config.yaml. This means updating the *Configuration.java classes by changing the @JsonProperty annotations to use camel case.

Additional Information:

Use Dropwizard Validation

Right now there is the potential for exceptions to occur because not all parameters to the resource methods are being validated.

We should use Dropwizard validation to ensure that all User objects are @Valid, which means that emails and passwords are not null and not empty.

In particular, a NullPointerException can occur when getting the email address if the Email field in PilotUser is null. Also, it appears that we can post a new user without a password, which should not be allowed.

Containerize Thunder with Docker

If we can build a docker image that runs thunder, we can open ourselves up to much easier deployment, testing, and autoscaling.

We can then run Thunder on Amazon ECS and eventually Amazon EKS (Managed Kubernetes).