cognitivej/cognitivej

Cognitive Image Analysis for Java

Looking for new collaborators for this repo - open an issue if you would like to be considered

CognitiveJ - Image Analysis in Java

CognitiveJ is an open-source fluent Java (8) API that manages and orchestrates the interaction between Java applications and Microsofts’ Cognitive (Project Oxford) Machine Learning & Image Processing libraries and allows you to query and analyze images.

Face service access is limited based on eligibility and usage criteria. You will need a valid subscription id to Azure Face service

Faces

• Facial Detection – Capture faces, gender, age and associated facial features and landmarks from an image
• Emotion Detection – Derive emotional state from faces within an image
• Verification – Verify, with a confidence scale on whether 2 different faces are of the same person
• Identification – Identify a person from a set of known people.
• Find Similar – detect, group and rank similar faces
• Grouping – group people based on facial characteristics
• Person Group/Person/Face Lists; Create, manage and train groups, face lists and persons to interact with the identification/grouping/find similar face features.

Vision

• Image Describe - Describe visual content of an image and return real world caption to what the Image is of.
• Image Analysis – extract key details from an image and if the image is of an adult/racy nature.
• OCR – detect and extract text from an image.
• Thumbnail – Create thumbnail images based on key points of interest from the image.

Overlay (Experimental)

• Apply image layers onto images to visually represent found features.
• Apply captions onto faces and images
• Graphically illustrate the Faces/Vision feature sets.
• Pixelate faces in an image.

Other Features

• Works with local or remote images
• validation of parameters

Getting Started

• Java 8 or above
• Subscription keys for the MS Cognitive libraries (free registration here)
• Add the dependency from JCenter (Since JCenter is no longer available, build from source until artifacts are published elsewhere)

Gradle

dependencies {
    implementation 'cognitivej:cognitivej:0.6.2'
    ...
}

Maven

    <dependency>
      <groupId>cognitivej</groupId>
      <artifactId>cognitivej</artifactId>
      <version>0.6.2</version>
      <type>pom</type>
    </dependency>

Chained Builders - The builders are simple lightweight wrappers over the MS Cognitive REST calls that manages the marshalling of parameters/responses, the HTTP communications and retry strategies. The builders are chained to allow for follow up manipulation on resources that have been created or retrieved & where applicable.

Scenarios - Scenarios are real world use case classes that greatly simplifies the interaction between the builders and the wrapper classes. While there is no reason you can’t interact directly with the builders, scenarios have much of the boilerplate logic in place to reduce burden.

Overlay - Allows for creating and writing new images based on the results from the queries. Note: work is ongoing around collision detection and observing boundaries

Wrappers Simple domain wrappers around request/response/parameter objects (e.g. Face, FaceAttributes,Person etc)

Face – Detect can detect faces from within an image and return the results as a collection of ‘face’ results.

Example

public static void main(String[] args) {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.face.subscriptionKey"),
            getProperty("azure.cognitive.emotion.subscriptionKey"));
    ImageOverlayBuilder imageOverlayBuilder = ImageOverlayBuilder.builder(IMAGE_URL);
    imageOverlayBuilder.outlineFacesOnImage(faceScenarios.findFaces(IMAGE_URL), RectangleType.FULL,
            CognitiveJColourPalette.STRAWBERRY).launchViewer();
}

Face – Landmarks can detect faces from within an image and apply facial landmarks

Example

public static void main(String[] args) throws IOException {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.face.subscriptionKey"),
            getProperty("azure.cognitive.emotion.subscriptionKey"));
    Face faces = faceScenarios.findSingleFace(IMAGE_URL);
    ImageOverlayBuilder.builder(IMAGE_URL).outFaceLandmarksOnImage(faces).launchViewer();
}

Face – Detect with Attributes displays associated attributes for detected faces

Example

public static void main(String[] args) {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.face.subscriptionKey"),
            getProperty("azure.cognitive.emotion.subscriptionKey"));
    List<Face> faces = faceScenarios.findFaces(IMAGE_URL);
    ImageOverlayBuilder.builder(IMAGE_URL).outlineFacesOnImage(faces, RectangleType.CORNERED,
            CognitiveJColourPalette.MEADOW).writeFaceAttributesToTheSide(faces, CognitiveJColourPalette.MEADOW).launchViewer();
}

Face – Verify will validate (with a confidence ratio) if 2 different faces are of the same persons.

Example

public static void main(String[] args) {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.face.subscriptionKey"),
            getProperty("azure.cognitive.emotion.subscriptionKey"));
    ImageOverlayBuilder imageOverlayBuilder = ImageOverlayBuilder.builder(CANDIDATE_1);
    imageOverlayBuilder.verify(CANDIDATE_2, faceScenarios.verifyFaces(CANDIDATE_1, CANDIDATE_2)).launchViewer();
}

Face – Identify will identify a person (or people) within an image. Before the library can identify, we need to provide the the Cognitive libraries with the samples set of candidates. Currently supports 1000 candidates.

Example

public static void main(String[] args) {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.face.subscriptionKey"),
            getProperty("azure.cognitive.emotion.subscriptionKey"));
    ImageOverlayBuilder imageOverlayBuilder = ImageOverlayBuilder.builder(IMAGE);
    List<ImageHolder> candidates = candidates();
    People people = ScenarioHelper.createPeopleFromHoldingImages(candidates, ImageNamingStrategy.DEFAULT);
    String groupId = faceScenarios.createGroupWithPeople(randomAlphabetic(6).toLowerCase(), people);
}

Face – Pixelate will identify all faces within an image and pixelate them.

public static void main(String[] args) {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.face.subscriptionKey"),
            getProperty("azure.cognitive.emotion.subscriptionKey"));
    ImageOverlayBuilder imageOverlayBuilder = ImageOverlayBuilder.builder(IMAGE);
    faceScenarios.findFaces(IMAGE).stream().forEach(imageOverlayBuilder:: pixelateFaceOnImage);
    imageOverlayBuilder.launchViewer();
}

Emotion – Detect will detect what emotion a face(s) is showing within an image.

public static void main(String[] args) {
    FaceScenarios faceScenarios = new FaceScenarios(getProperty("azure.cognitive.face.subscriptionKey"),
            getProperty("azure.cognitive.emotion.subscriptionKey"));
    ImageOverlayBuilder.builder(IMAGE_URL).outlineEmotionsOnImage(faceScenarios.findEmotionFaces(IMAGE_URL)).launchViewer();
}

Vision – Describe will analyse and describe the contents of an image in a human readable caption.

public static void main(String[] args) {
    ComputerVisionScenario computerVisionScenario = new ComputerVisionScenario(getProperty("azure.cognitive.vision.subscriptionKey"));
    ImageDescription imageDescription = computerVisionScenario.describeImage(IMAGE_URL);
    ImageOverlayBuilder.builder(IMAGE_URL).describeImage(imageDescription).launchViewer();

}

Vision – OCR will analyse and extract text from within an image into a computer understandable stream.

public static void main(String[] args) {
    ComputerVisionScenario computerVisionScenario = new ComputerVisionScenario(getProperty("azure.cognitive.vision.subscriptionKey"));
    OCRResult ocrResult = computerVisionScenario.ocrImage(IMAGE_URL);
    ImageOverlayBuilder.builder(IMAGE_URL).ocrImage(ocrResult).launchViewer();
}

cognitivej's Issues

Typo in SURPRISE emotion

fixed in PR: #9

can't search the maven jar on maven mvnrepository

http://search.maven.org/#search%7Cga%7C1%7Ccognitivej

http://www.mvnrepository.com/search?q=cognitivej

Tests should run during CI

Hi,I want to use the cognitivej in an android app.

When building the project,I find that "BufferedImage" can't be imported in android.
Is there any other methods to load the image?

missing artifact

artifact id missing "cognitivej:cognitivej:pom:0.6.2", eclipse yelling.

findEmotionFaces(InputStream image) returns List<Face> instead of List<Emotion>

I suppose the whole function is duplicate of another one above

Relevant part:

cognitivej/src/main/java/cognitivej/vision/face/scenario/FaceScenarios.java

Line 405 in 9ebcd5e

return faceTaskBuilder.detectFace(true, true, FaceAttributes.ALL, image).withResult();

Publish build artifacts

Now that bintray is no more an alternative is needed. Potentially GitHub Packages could be used but publishing to maven central is better.

building on recent Java versions

the code in this repo won't build as-is with recent JDK releases such as Java 17. It will at least require a newer version of Gradle. It would be great a github action can be setup to build using multiple JDK versions to ensure maximum compatibility. Personally I would like to see JPMS support which would also mean dropping Java 1.8 support.

Are there any plans for development to continue on this repo? I'm happy to make a PR for a working CI that builds on JDK 8, 11, 17, and 18 if so.

face and emotion not working for some images

Sometimes the single face detection fails also emotion detection fails, I am attaching 2 images where the face detection and emotion detection failed.
no emotion was detected.

failed to detect the single face