hdsdi3g/prodlib

Some utilities libs and tools, and parent (pom) projects for small java projects.

ProdLib

Some utilities libs and tools, and parent (pom) projects for small java projects.

All these projects are consistent with each other. Some use Spring Boot when necessary.

Please use Maven and Java 21 for build, test and run.

All is tested on Windows 10, Linux and GitHub Actions (Ubuntu). Should be ok on macOS.

Relases are sent to Maven central.

Projects / modules

Starter

Commons Maven starter project for bootstrap App, Web app and libs. Pom project, Spring Boot optional, stable.

Parent

Base project for lib. Pom project, Spring Boot, stable.

Parent-web

Base project for web lib. Pom project, Spring Boot, stable.

App

Base project for standalone application. Pom project, Spring Boot, stable.

App-web

Base project for standalone web application. Pom project, Spring Boot, stable.

Interfaces

Java Interfaces and DTOs for connect projects without direct dependencies, for some shared cross-projects objects and definitions.

Java lib, Spring Boot, stable.

Transfertfiles

Manipulate files across protocols: a Java library for upload and download files, with protocol abstraction (local File, FTP/FTPS/FTPES, SFTP), and transfer progression events.

Java lib, standalone, stable.

Jobkit

Base project for all jobkit modules. Pom project, Spring Boot optional, stable.

Jobkit/engine

Execute onetime and scheduled jobs with dynamic queues and events.

Engine embed a Supervisable tool, used in conjonction with JobKitEngine to track, and collect events during the task execution.

Java lib, standalone, stable.

Jobkit/springboot-service

Execute as SpringBoot module the JobKit lib. Java lib, Spring Boot, stable.

Jobkit/watchfolder

Regulary scan folders (local or distant) and throws events on activity.

It can found recently added files and directories in local filesystem, and start events (run Java code) on it.

Java lib, standalone, stable.

Jobkit/watchfolder-jpa

Watchfolder lib with persistence. Java lib, Spring Boot, stable.

Mailkit

Mail engine as notification abstraction.

MailKit can manage Supervisable produced by JobKit, and can transform it in mail: Notification.

Java lib, Spring Boot, stable.

Env-version

Tools for provide project and deps version. Java lib, Spring Boot, stable.

Jsconfig

Let setup JS files as configuration and expose it as a Spring Boot service. Java lib, Spring Boot, Alpha.

Java lib, Spring Boot, Alpha.

Contributing / debugging

Versioning: just use SemVer.

Author and License

This project is writer by hdsdi3g and licensed under the LGPL License; see the LICENCE.TXT file for details.

prodlib's Issues

Notification tool logic in MailKit

NotificationRouterMail log message should display the reason to send a mail (group / enduser), and the used template

About NotificationRouterMail ‹main› Send a mail to [xxx]

Close Supervisable/Notifications mails are not sended during the App shutdown

Seeams related to SpoolExecutors / Shutdown all / SpringApplicationShutdownHook...

Create a new SpringApplicationShutdownHook for Supervisables/Notifications ?

OR maybe it never send the Watchfolder close event ?

Better log verbosity for jobkit

Resolve warns after sonatype-lift review

Prefer instanceof to getClass when implementing Object#equals.

For:

• BufferVault
• FTPFileSystem

NULL_DEREFERENCE: object protocol last assigned on line 74 could be null and is dereferenced at line 90.

For: AbstractFileSystemURL

ImmutableEnumChecker: enums should be immutable: 'DigestFilterHashExtraction' has field 'digestProvider' of type 'tv.hd3g.transfertfiles.filters.DigestFilterHashExtraction.ExtractionInstanceProvider', the declaration of type 'tv.hd3g.transfertfiles.filters.DigestFilterHashExtraction.ExtractionInstanceProvider' is not annotated with @com.google.errorprone.annotations.Immutable

For: DigestFilterHashExtraction

Instance SupervisableKit (its Executor) via JobKit

Translate (MailKit) missing keys log messages should not displays string in uppercase

Conf collision with Sonar and Parent project

Add explicit hamcrest dependency in starter pom

Add:

  <groupId>org.hamcrest</groupId>
  <artifactId>hamcrest</artifactId>

Normally provided by spring-boot-starter-test

Add SonarQube code coverage

https://stackoverflow.com/questions/42450322/jacocoreport-aggregate-either-gets-shallow-coverage-or-none-at-all

Remove applifecycle from MailKit

Add Exceptions for JobKit Runnables

Update Apache Commons Text to 1.10.0 (security)

Experiment GraalVM Native Image

Remove unnecessary CWD before start download file for Transfertfiles/FTPFile

Refactor sendLocalToAbstract and copyAbstractToLocal with downloadAbstract / uploadAbstract.

Strange bug with CWD operations.

Create CSV utility

Upgrade Authkit API.md / check Spring REST Docs

https://spring.io/projects/spring-restdocs

Create a Notification send mail retention policy and methods

Add a Database, in a separate project ?

Create a Method to Compare 2 Events => same/not same/urgent (not wait).
Create an API to filter urgent/not urgents events.

Create a Policy logic:

• day/night hours with offline days: collect, release
• online days and hours: wait n hours, collect, release

Release will aggregate and collect all Events in a same mail.

Never reduce, only collect+release, risk of loose some important datas.
If same: just list the iterations with dates.

SFTP has trouble with "/"

WatchFolder can’t detect updated files after a first detection

Create a new Watchfolder event, on detected and after updated file.

Add Sonarcloud checks via GH Actions

https://stackoverflow.com/questions/62584227/building-multi-module-maven-project-for-sonarcloud

Create a mkdirs function on AbstractFile/Transfertfiles project

Run all waited JobKit BackgroundServices after Spring Boot application is ready run

If a watchfolder is setup for 1/hour, don't wait one hour for run the first start.

• Add a JobKitEngine function for that
• Add an automatic trigger for that on a Spring Boot app
• Disabled it on FlatJobKitEngine

Create a Supervisable logic on JobKit

Trace before run:

• run context vars
• End-users list (external) and admin users list (internal) not here, see #36
• Callbacks for Job lifecycle

During run:

• status (waiting, run, ended, error)

After run:

• trigger for some events, like mail send, logs, stats… with context vars and lists. Only after end will be implemented now.
• archive solutions for errors and detect new/same errors (flat files?). Not now, not here

Supervisable to Notification glue

Add a max execution time for a Jobkit Job

Add a Stopper callback and an onStop job event callbacks. Add this events on main Event callback manager.

Add cache to GH Actions

https://github.com/actions/setup-java#caching-packages-dependencies

AuthKit resolve Hibernate warn with Spring Boot 3

From UserDao:

WARN  HHH90000021: Encountered deprecated setting [javax.persistence.sharedCache.mode], use [jakarta.persistence.sharedCache.mode] instead

Run SpringBoot methods via a JobKit Spooler

With the help of Spring AOP

Correct MailKit Notification templates

• Remove ul/li
• Add explicit "Consolas" and "Monaco" fonts as monospace fonts.

Resolve Duration serializable with jackson-datatype-jsr310 on Supervisable

    <dependency>
        <groupId>com.fasterxml.jackson.datatype</groupId>
        <artifactId>jackson-datatype-jsr310</artifactId>
    </dependency>

Warn for Watchfolder if grace time detection is very lower than watch period

Merge Authkit README with project README

Buggy templates on debug/dev Notification mails

• Missing monospace font for caller
• Notification env should be grey

Set default Messages for MailKit/Notification

Better behavior to protect bad Supervisable messages to Translate

Support Inotify WatchFolders

Via

https://stackoverflow.com/questions/43141838/which-operating-systems-support-native-inotify-like-file-watching-in-java
https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/

For "File" WF. But only if Inotify/like is really used, not via a simple poller (need POC).

Remove JobKit web API

Race condition with Supervisable/Notification and external code

When Spring Boot load on this order:

• JobKit
• MailKit
• external code, like load WatchFolder

=> On boot Notification are sended

BUT

• JobKit
• external code, like load WatchFolder
• MailKit

=> On boot Notification are lost.

Add a retention mechanism for onEndEvents, and liberate and free them when MailKit plug NotificationManager on SupervisableManager.

Switch from Java 11 (LTE) to Java 17 (LTE)

Create Thread-context for SupervisableKit

https://logging.apache.org/log4j/2.x/manual/thread-context.html

And Notification

Upgrade to Spring Boot 3

ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

Create a Watchfolder Supervisable tool

Notification to MailKit glue

Add commons-collections4 dependency and property version

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-collections4</artifactId>
            <version>${commons.collections4.version}</version>
        </dependency>

https://mvnrepository.com/artifact/org.apache.commons/commons-collections4

Implements a MailKit Notification tool read-to use

Setup nexus-staging-maven-plugin

Resolve warning with

[INFO] Installing Nexus Staging features:
[INFO]   ... total of 15 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin

Manage extended extensions for Watchfolder

Like .csv.gz, in ObservedFolder.setAllowedExtentions

Resolve THREAD_SAFETY_VIOLATION on transfertfiles/BufferVault

sonatype-lift says:

L306

Read/Write race. Non-private method BufferVault.equals(...) reads without synchronization from this.datas. Potentially races with write in method BufferVault.write(...).
Reporting because another access to the same memory occurs on a background thread, although this access may not.

And L288/289

Read/Write race. Non-private method BufferVault.hashCode() reads without synchronization from this.size. Potentially races with write in method BufferVault.write(...).
Reporting because another access to the same memory occurs on a background thread, although this access may not.

Missing translate keys for Watchfolder on MailKit

Like mailkit.notification.(ObservedFolders).startscans