radar-base/radar-commons

RADAR-base platform's common Java utilities library containing basic schemas, streaming features, testing bridges and utils.

RADAR-Commons

Common utilities library containing basic schemas, streaming features, testing bridges and utils.

Usage

Add the RADAR-Commons library to your project with Gradle by updating your build.gradle.kts file with:

repositories {
    mavenCentral()
}

dependencies {
    implementation("org.radarbase:radar-commons:1.1.2")
}

Example use, after adding radar-schemas to classpath:

// Set URLs for RADAR-base installation
val baseUrl = "https://..."
val oauthToken = ...

val kafkaSender = restKafkaSender {
    baseUrl = "$baseUrl/kafka/"
    headers.append("Authorization", "Bearer $oauthToken")
    httpClient {
        timeout(10.seconds)
    }
    schemaRetriever ("$baseUrl/schema/")
}

// Configure topic to send data over
val topic = AvroTopic(
    "linux_raspberry_temperature",
    ObservationKey.getClassSchema(),
    RaspberryTemperature.getClassSchema(),
    ObservationKey::class.java,
    RaspberryTemperature::class.java
)

val topicSender = kafkaSender.sender(topic)

val key = ObservationKey("myProject", "myUser", "mySource")

// Send data to topic.
runBlocking {
    val values: List<RaspberryTemperature> = readValuesFromSystem()
    topicSender.send(key, values)
}

Note that this code above does not include any flows for registering a source with the ManagementPortal.

For server utilities, include radar-commons-server:

repositories {
    mavenCentral()
    maven(url = "https://packages.confluent.io/maven/")
}

dependencies {
    implementation("org.radarbase:radar-commons-server:1.1.2")
}

For mocking clients of the RADAR-base infrastructure, use that 'radar-commons-testing' repository:

repositories {
    mavenCentral()
    maven(url = "https://packages.confluent.io/maven/")
}

dependencies {
    testImplementation("org.radarbase:radar-commons-testing:1.1.2")
}

To test your backend with a MockProducer, copy testing/mock.yml.template to testing/mock.yml and edit its parameters. Then run

./gradlew :testing:run

to send data to your backend.

To use the RADAR Gradle plugins, see the README of the radar-commons-gradle directory.

Contributing

For latest code use dev branch. This is released on JFrog's OSS Artifactory. To use that release, add the following fragment to your build.gradle file.

repositories {
    maven(url = "https://oss.sonatype.org/content/repositories/snapshots")
}

configurations.all {
    // Check for updates every build
    resolutionStrategy.cacheChangingModulesFor(0, "SECONDS")
}

dependencies {
    implementation("org.radarbase:radar-commons:1.1.3-SNAPSHOT")
}

Code should be formatted using the Kotlin official style guide, in addition to ktlint rules. If you want to contribute a feature or fix browse our issues, and please make a pull request.

radar-commons's Issues

Creation of Temporary File in Directory with Insecure Permissions SNYK-JAVA-COMGOOGLEGUAVA-5710356

Overview

com.google.guava:guava is a set of core libraries that includes new collection types (such as multimap and multiset,immutable collections, a graph library, functional types, an in-memory cache and more.

Affected versions of this package are vulnerable to Creation of Temporary File in Directory with Insecure Permissions due to the use of Java's default temporary directory for file creation in FileBackedOutputStream. Other users and apps on the machine with access to the default Java temporary directory can access the files created by this class. This more fully addresses the underlying issue described in CVE-2020-8908, by deprecating the permissive temp file creation behavior.

NOTE: Even though the security vulnerability is fixed in version 32.0.0, the maintainers recommend using version 32.0.1, as version 32.0.0 breaks some functionality under Windows.

Remediation

Upgrade com.google.guava:guava to version 32.0.0-android, 32.0.0-jre or higher.

References

• GitHub Commit
• GitHub Issue
• GitHub Issue
• GitHub Release

Add custom classes required by kafka components

Add custom classes like AbstractKafkaAvroDesrializer,etc required by kafka components like connector, streams,etc as a separate module so they can be shared easily.

Integer Overflow or Wraparound SNYK-JAVA-ORGXERIALSNAPPY-5710961

Overview

Affected versions of this package are vulnerable to Integer Overflow or Wraparound via the function compress(char[] input) in Snappy.java due to improper validation of the array length.

Exploiting this vulnerability is possible when the “buf” array compiled by the maxCompressedLength function is successfully allocated but its size might be too small to use for the compression, causing a fatal Access Violation error.

Note:
The issue most likely won’t occur when using a byte array since creating a byte array of size 0x80000000 (or any other negative value) is impossible in the first place.

PoC

package org.example;
import org.xerial.snappy.Snappy;

import java.io.*;

public class Main {

    public static void main(String[] args) throws IOException {
        char[] uncompressed = new char[0x40000000];
        byte[] compressed = Snappy.compress(uncompressed);
    }
}

Remediation

Upgrade org.xerial.snappy:snappy-java to version 1.1.10.1 or higher.

References

• GitHub Commit
• Vulnerable Code
• Vulnerable Code

Remove Jackson dependancy

On Android, Jackson contains a lot of functions that need to be added to Dex. In addition, it uses reflection which is slow on android. RestSender and SchemaRegistry could use org.json instead, which API is already part of Android, so the dependency can be removed there.

Denial of Service (DoS) SNYK-JAVA-COMFASTERXMLJACKSONCORE-3038426

Overview

com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.
Affected versions of this package are vulnerable to Denial of Service (DoS) in the _deserializeWrappedValue() function in StdDeserializer.java, due to resource exhaustion when processing deeply nested arrays.

NOTE: This vulnerability is only exploitable when the non-default UNWRAP_SINGLE_VALUE_ARRAYS feature is enabled.

Details

Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.

Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.

One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.

When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.

Two common types of DoS vulnerabilities:

• High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.

• Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws package

Remediation

Upgrade com.fasterxml.jackson.core:jackson-databind to version 2.12.7.1, 2.13.4.1 or higher.

References

• Chromium Bugs
• Documentation
• GitHub Commit
• GitHub Commit
• GitHub Commit
• GitHub Issue

Denial of Service (DoS) SNYK-JAVA-COMFASTERXMLJACKSONCORE-2326698

Overview

com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.

Affected versions of this package are vulnerable to Denial of Service (DoS) when using JDK serialization to serialize and deserialize JsonNode values.
It is possible for the attacker to send a 4-byte length payload, with a value of Integer.MAX_VALUE , that will eventually cause large buffer allocation and out of heap memory.

Details

Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.

Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.

One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.

When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.

Two common types of DoS vulnerabilities:

• High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.

• Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws package

Remediation

Upgrade com.fasterxml.jackson.core:jackson-databind to version 2.13.1, 2.12.6 or higher.

References

• GitHub Commit
• GitHub Issue

Improper Locking SNYK-JAVA-ORGJETBRAINSKOTLIN-2628385

Overview

org.jetbrains.kotlin:kotlin-stdlib is a Kotlin Standard Library for JVM.

Affected versions of this package are vulnerable to Improper Locking due to inability to lock dependencies for Multiplatform Gradle Projects.

Remediation

Upgrade org.jetbrains.kotlin:kotlin-stdlib to version 1.6.0 or higher.

References

• JetBrains Issue
• JetBrains Security Announcment
• RedHat Bugzilla Bug

Improper Input Validation SNYK-JAVA-ORGAPACHECOMMONS-5901530

Overview

org.apache.commons:commons-compress is an API for working with compression and archive formats.

Affected versions of this package are vulnerable to Improper Input Validation when parsing TAR files. An attacker can manipulate file modification times headers, leading to a denial of service issue via CPU consumption.

Note: This is only exploitable if applications are using the CompressorStreamFactory class (with auto-detection of file types), TarArchiveInputStream, and TarFile classes to parse TAR files.

Remediation

Upgrade org.apache.commons:commons-compress to version 1.24.0 or higher.

References

• Apache Advisory
• GitHub Commit

Integer Overflow or Wraparound SNYK-JAVA-ORGXERIALSNAPPY-5710959

Overview

Affected versions of this package are vulnerable to Integer Overflow or Wraparound via the shuffle(int[] input) function due to improper validation of the multiplications done on the input length.
Exploiting this vulnerability is possible by passing negative, zero, float, very small, or very long values to the shuffle functions, which later on are multiplicated by four.
A successful exploration results in “java.lang.ArrayIndexOutOfBoundsException" or “java.lang.NegativeArraySizeException” exceptions which can crash the program.

PoC

package org.example;
import org.xerial.snappy.BitShuffle;

import java.io.*;


public class Main {

    public static void main(String[] args) throws IOException {
        int[] original = new int[0x40000000];
        byte[] shuffled = BitShuffle.shuffle(original);
        System.out.println(shuffled[0]);
    }
}

The program will crash, showing the following error (or similar):

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.example.Main.main(Main.java:12)

Process finished with exit code 1

Remediation

Upgrade org.xerial.snappy:snappy-java to version 1.1.10.1 or higher.

References

• GitHub Commit
• Vulnerable Code

Cordova Plugin

Developing a Cordova plug-in for sending data into RADAR reusing the RADAR-Common could be a valuable effort.

Dependency should be documented

When using the module radar-commons-server, the kafka-clients dependency requires maven { url 'http://packages.confluent.io/maven/' } repository. This should be added in the README.

Generalisation of AvroTopic

The constructor of AvroTopic class verifies if the valueSchema contains fields time and timeReceived. After the introduction of Questionnaire, this constrain is no longer valid.

We may generalise the constructor adding a dummy validation that a subclass of AvroTopic can override.

public AvroTopic(String name, Schema keySchema, Schema valueSchema,  Class<K> keyClass, Class<V> valueClass) {
        super(name);
        this.keySchema = keySchema;
        this.valueSchema = valueSchema;
        validateKey(this.keySchema)
        validateSchema(this.valueSchema)
        this.valueClass = valueClass;
        this.keyClass = keyClass;
        List<Schema.Field> fields = valueSchema.getFields();
        this.valueFieldTypes = new Schema.Type[fields.size()];
        for (int i = 0; i < fields.size(); i++) {
            valueFieldTypes[i] = fields.get(i).schema().getType();
        }
}

public void validateKey(Schema schema) {}
public void validateSchema(Schema schema) {}

And then for example the subclass SensorTopic can do

public class SensorTopic <K, V> extends AvroTopic<K, V> {
    ...
    public void validateKey(Schema schema) {}
    public void validateSchema(Schema schema) {
        if (schema.getField("time") == null) {
            throw new IllegalArgumentException("Schema must have time as its first field");
        }
        if (schema a.getField("timeReceived") == null) {
            throw new IllegalArgumentException("Schema must have timeReceived as a field");
        } 
   }
    ...
}

I do not have a clear idea about possible implications of this change.

@blootsvoets @MaximMoinat what do you think?

Information Exposure SNYK-JAVA-ORGJETBRAINSKOTLIN-2393744

Overview

org.jetbrains.kotlin:kotlin-stdlib is a Kotlin Standard Library for JVM.

Affected versions of this package are vulnerable to Information Exposure. A Kotlin application using createTempDir or createTempFile and placing sensitive information within either of these locations would be leaking this information in a read-only way to other users also on this system.

Note: As of version 1.4.21, the vulnerable functions have been marked as deprecated. Due to still being useable, this advisory is kept as "unfixed".

PoC by JLLeitschuh

package org.jlleitschuh.sandbox

import org.junit.jupiter.api.Test
import java.io.BufferedReader
import java.io.File
import java.io.IOException
import java.io.InputStreamReader
import java.nio.file.Files

class KotlinTempDirectoryPermissionCheck {
    @Test
    fun `kotlin check default directory permissions`() {
        val dir = createTempDir()
        runLS(dir.parentFile, dir) // Prints drwxr-xr-x
    }

    @Test
    fun `Files check default directory permissions`() {
        val dir = Files.createTempDirectory("random-directory")
        runLS(dir.toFile().parentFile, dir.toFile()) // Prints drwx------
    }

    @Test
    fun `kotlin check default file permissions`() {
        val file = createTempFile()
        runLS(file.parentFile, file) // Prints -rw-r--r--
    }

    @Test
    fun `Files check default file permissions`() {
        val file = Files.createTempFile("random-file", ".txt")
        runLS(file.toFile().parentFile, file.toFile()) // Prints -rw-------
    }

    private fun runLS(file: File, lookingFor: File) {
        val processBuilder = ProcessBuilder()
        processBuilder.command("ls", "-l", file.absolutePath)
        try {
            val process = processBuilder.start()
            val output = StringBuilder()
            val reader = BufferedReader(
                InputStreamReader(process.inputStream)
            )
            reader.lines().forEach { line ->
                if (line.contains("total")) {
                    output.append(line).append('\n')
                }
                if (line.contains(lookingFor.name)) {
                    output.append(line).append('\n')
                }
            }
            val exitVal = process.waitFor()
            if (exitVal == 0) {
                println("Success!")
                println(output)
            } else {
                //abnormal...
            }
        } catch (e: IOException) {
            e.printStackTrace()
        } catch (e: InterruptedException) {
            e.printStackTrace()
        }
    }
}

Remediation

There is no fixed version for org.jetbrains.kotlin:kotlin-stdlib.

References

• GitHub Issue
• Vulnerable Code #1
• Vulnerable Code #2

Distribution: publish the package to JCenter

Right now, the project can only be used as a git submodule/gradle subproject. It would be nicer to publish a version to JCenter each time we do a release. That would solve dependency management automatically. See the JCenter Gradle plugin for more details.

Denial of Service (DoS) SNYK-JAVA-ORGXERIALSNAPPY-5710960

Overview

Affected versions of this package are vulnerable to Denial of Service (DoS) via the hasNextChunk function due to improper validation of the chunkSize variable value.

Exploiting this vulnerability is possible by passing a negative number (such as 0xFFFFFFFF, which is -1), which will cause the code to raise a java.lang.NegativeArraySizeException exception.
A worse case would happen when passing a huge positive value (such as 0x7FFFFFFF), raising the fatal java.lang.OutOfMemoryError error.

PoC

package org.example;
import org.xerial.snappy.SnappyInputStream;

import java.io.*;

public class Main {

    public static void main(String[] args) throws IOException {
        byte[] data = {-126, 'S', 'N', 'A', 'P', 'P', 'Y', 0, 0, 0, 0, 0, 0, 0, 0, 0,(byte) 0x7f, (byte) 0xff, (byte) 0xff, (byte) 0xff};
        SnappyInputStream in = new SnappyInputStream(new ByteArrayInputStream(data));
        byte[] out = new byte[50];
        try {
            in.read(out);
        }
        catch (Exception ignored) {

        }
    }
}

Details

Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.

Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.

One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.

When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.

Two common types of DoS vulnerabilities:

• High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.

• Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws package

Remediation

Upgrade org.xerial.snappy:snappy-java to version 1.1.10.1 or higher.

References

• GitHub Commit
• Vulnerable Code

HTTP status code 415

The new unsafe sender returns HTTP 415 Unsupported Media Type while requesting https://localhost/kafka/topics/schemaless-value.

WARN - Schema for android_empatica_e4_acceleration value was not yet added to the schema registry. [org.radarcns.producer.SchemaRetriever:185]
WARN - Schema for android_empatica_e4_acceleration value was not yet added to the schema registry. [org.radarcns.producer.SchemaRetriever:185]
INFO - Accept: application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json
[org.radarcns.producer.rest.RestSender:245]
INFO - Request{method=POST, url=https://localhost/kafka/topics/schemaless-value, tag=null} [org.radarcns.producer.rest.RestSender:246]
ERROR - FAILED to transmit message: {"error_code":415,"message":"HTTP 415 Unsupported Media Type"} -> {"key_schema":"{"type":"record","name":"MeasurementKey","namespace":"org.radarcns.key","doc":"Measurement key in the RADAR-CNS project","fields":[{"name":"userId","type":{"type":"string","avro.java.string":"String"},"doc"... [org.radarcns.producer.rest.RestSender:258]
ERROR - FAILED to transmit message: {"key_schema":"{"type":"record","name":"MeasurementKey","namespace":"org.radarcns.key","doc":"Measurement key in the RADAR-CNS project","fields":[{"name":"userId","type":{"type":"string","avro.java.string":"String"},"doc"... [org.radarcns.producer.rest.RestSender:264]

java.io.IOException: Failed to submit (HTTP status code 415): {"error_code":415,"message":"HTTP 415 Unsupported Media Type"} at org.radarcns.producer.rest.RestSender$RestTopicSender.send(RestSender.java:260) at org.radarcns.producer.rest.BatchedKafkaSender$BatchedKafkaTopicSender.send(BatchedKafkaSender.java:88) at org.radarcns.mock.MockFile.send(MockFile.java:107) at org.radarcns.mock.MockProducer.start(MockProducer.java:136) at org.radarcns.pipeline.EndToEndTest.streamToKafka(EndToEndTest.java:283) at org.radarcns.pipeline.EndToEndTest.endToEnd(EndToEndTest.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Suppressed: java.io.IOException: Failed to submit (HTTP status code 415): {"error_code":415,"message":"HTTP 415 Unsupported Media Type"} at org.radarcns.producer.rest.RestSender$RestTopicSender.send(RestSender.java:260) at org.radarcns.producer.rest.BatchedKafkaSender$BatchedKafkaTopicSender.flush(BatchedKafkaSender.java:128) at org.radarcns.producer.rest.BatchedKafkaSender$BatchedKafkaTopicSender.close(BatchedKafkaSender.java:137) at org.radarcns.mock.MockFile.send(MockFile.java:113) ... 25 more

Fail build if release cycle is not successfully processed.

Travis build status should indicate FAILED if releases are performed and github or bintrayUpload are not successful.

Merge some more REST-API and Backend tests

As per comment @nivemaham in #11:

There are still few more classes we could add to commons-test that can be reusable in rest-api and mongo-db-connector e2e tests.

No data landing into topics

Data sent by MockDevice with RestSender does not reach topics. Used MockDevice from 0b17335 against RADAR-base/RADAR-Docker@0abdd14 adding topics schema less-value and schema less-key.

Rest-Proxy log does not report any error

....
[2017-05-11 21:24:46,946] INFO 172.23.0.6 - - [11/May/2017:21:24:46 +0000] "HEAD / HTTP/1.0" 200 0  1 (io.confluent.rest-utils.requests)
[2017-05-11 21:24:47,303] INFO 172.23.0.6 - - [11/May/2017:21:24:47 +0000] "GET /topics/schemaless-value HTTP/1.0" 200 559  17 (io.confluent.rest-utils.requests)
[2017-05-11 21:24:47,362] INFO 172.23.0.6 - - [11/May/2017:21:24:47 +0000] "HEAD / HTTP/1.0" 200 0  0 (io.confluent.rest-utils.requests)
[2017-05-11 21:24:47,406] INFO 172.23.0.6 - - [11/May/2017:21:24:47 +0000] "HEAD / HTTP/1.0" 200 0  0 (io.confluent.rest-utils.requests)
...

Likewise MongoDb Connector

....
[2017-05-11 21:24:29,617] INFO 0 have been written in MongoDB 0 records need to be processed. (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:24:59,372] INFO 0 have been processed (org.radarcns.mongodb.MongoDbSinkTask)
[2017-05-11 21:24:59,617] INFO 0 have been written in MongoDB 0 records need to be processed. (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:25:29,132] INFO [FLUSH-WRITER] Time-elapsed: 3.402E-6 s (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:25:29,132] INFO [FLUSH] Time elapsed: 1.37946E-4 s (org.radarcns.mongodb.MongoDbSinkTask)
[2017-05-11 21:25:29,132] INFO WorkerSinkTask{id=radar-connector-mongodb-sink-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask)
[2017-05-11 21:25:29,372] INFO 0 have been processed (org.radarcns.mongodb.MongoDbSinkTask)
[2017-05-11 21:25:29,618] INFO 0 have been written in MongoDB 0 records need to be processed. (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:25:59,372] INFO 0 have been processed (org.radarcns.mongodb.MongoDbSinkTask)
...

Kafka Log files

root@ceca758ee7b8:/var/lib/kafka/data/android_empatica_e4_acceleration-0# /usr/bin/kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files 00000000000000000000.log
Dumping 00000000000000000000.log
Starting offset: 0

Denial of Service (DoS) SNYK-JAVA-COMFASTERXMLJACKSONCORE-3038424

Overview

com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.
Affected versions of this package are vulnerable to Denial of Service (DoS) in the _deserializeFromArray() function in BeanDeserializer, due to resource exhaustion when processing a deeply nested array.

NOTE:
For this vulnerability to be exploitable the non-default DeserializationFeature must be enabled.

Details

Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.

Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.

One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.

When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.

Two common types of DoS vulnerabilities:

• High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.

• Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws package

Remediation

Upgrade com.fasterxml.jackson.core:jackson-databind to version 2.12.7.1, 2.13.4 or higher.

References

• Chromium Bugs
• GitHub Commit
• GitHub Commit
• GitHub Issue

Reduce number of HEAD requests

With concurrent senders, a lot of head requests are made to the server. This number should be reduced.

Denial of Service (DoS) SNYK-JAVA-COMFASTERXMLJACKSONCORE-2421244

Overview

com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.

Affected versions of this package are vulnerable to Denial of Service (DoS) via a large depth of nested objects.

Details

Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.

Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.

One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.

When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.

Two common types of DoS vulnerabilities:

• High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.

• Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws package

Remediation

Upgrade com.fasterxml.jackson.core:jackson-databind to version 2.12.6.1, 2.13.2.1 or higher.

References

• GitHub Commit
• GitHub Issue
• GitHub PR