usdot-jpo-ode / jpo-s3-deposit Goto Github PK
View Code? Open in Web Editor NEWSUBMODULE: Generic Kafka stream to S3 depositing modules. Packages JSON kafka streams into files for depositing into research data environments.
SUBMODULE: Generic Kafka stream to S3 depositing modules. Packages JSON kafka streams into files for depositing into research data environments.
The Dockerfile should allow for the environment value of group to be passed through. Currently this is set at a constant "group1" as seen here
Update all constant variables values to application arguments. Need to keep all variables in the .env (hard coding in the Java code feels harder for implementers to work with)
The USDOT SDC environment is moving from the public AWS cloud to the private USDOT ECS cloud. A new restriction (based on ECS) requires moving from static credentials (stored in the .env file) for Firehose/Kinesis connections to dynamic (time limited) credentials. To support this the SDC is providing a Python script that needs to be run (or could be used for sample code if it works better to port to Java) that generates the dynamic credentials. This script is available from Tony English. The request is to update the jpo-s3-deposit code (https://github.com/usdot-jpo-ode/jpo-s3-deposit/blob/010e221cf96cbf0ebfe9bea9c1bcf5d9d24549f1/src/main/java/us/dot/its/jpo/ode/aws/depositor/AwsDepositor.java) to support dynamic credentials.
The current logging settings are proving problematic - I'm seeing growing log file sizes in excess of 37GB on a single container. To help combat this issue, the AWS Depositor should be configured with a default logging level of warn rather than debug. Additionally, the logback.xml file should include a maxFileSize so the file cannot grow indefinitely.
It can also be a bit complex to work out the differences between the logback settings and docker logs. I recommend that the documentation is updated to reflect these differences and how to adjust the logback settings to be in line with the docker logging settings.
The latest changes to make the API_ENDPOINT an optional field included some unintended consequences and now the system will not read in the value. As can be seen in the apache documentation the Option constructor includes a field to determine whether the Option object should take an argument or not. There are several values in the AwsDepositor.java file that should be taking in an argument and are currently set to false for this value. As a result, no data can be passed in for those parameters.
On occasion the s3 deposit fails to subscribe to the Kafka topic. This seems to occur after a reboot of the machine or a restart of the ODE and the logs show a message of "LEADER_NOT_AVAILABLE" for the FilteredOdeBsmJson topic. I've attached an associated log to help debug.
bsm_error.log
Issue to test zap notification. Ignore.
While sending data to Firehose, an error occurred but doesn’t seem to have been thrown. Nothing in the logs indicated that the system was stuck, frozen, in a failed state. The container was online, but the logs show that no messages were processed for the past few days even though the corresponding topic has received thousands of messages in that timeframe. Kafka no longer tracked the corresponding consumer group because it had been > 24 hours since the consumer checked in.
This type of error should instead be handled rather than allow the system to become stuck in a failed state without indication of the failure in the logs.
While attempting to run the latest from Dev we ran into an issue with the Dockerfile. The CMD line is failing to execute due to missing '' at the end of line 26 and 27. Once added in, the module ran.
In AwsDepositor.java line 144 the auto commit is set to false. There is also no manual commit of the Kafka offset, which means if we lose connectivity the module doesn't know where to start up from again. This also makes it impossible to trace lag with this module which adds to difficulties when debugging issues.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.