radar-base / radar-commons Goto Github PK
View Code? Open in Web Editor NEWRADAR-base platform's common Java utilities library containing basic schemas, streaming features, testing bridges and utils.
License: Apache License 2.0
RADAR-base platform's common Java utilities library containing basic schemas, streaming features, testing bridges and utils.
License: Apache License 2.0
Affected versions of this package are vulnerable to Integer Overflow or Wraparound via the function compress(char[] input)
in Snappy.java
due to improper validation of the array length.
Exploiting this vulnerability is possible when the “buf” array compiled by the maxCompressedLength
function is successfully allocated but its size might be too small to use for the compression, causing a fatal Access Violation error.
Note:
The issue most likely won’t occur when using a byte array since creating a byte array of size 0x80000000 (or any other negative value) is impossible in the first place.
package org.example;
import org.xerial.snappy.Snappy;
import java.io.*;
public class Main {
public static void main(String[] args) throws IOException {
char[] uncompressed = new char[0x40000000];
byte[] compressed = Snappy.compress(uncompressed);
}
}
Upgrade org.xerial.snappy:snappy-java
to version 1.1.10.1 or higher.
Developing a Cordova plug-in for sending data into RADAR reusing the RADAR-Common could be a valuable effort.
Data sent by MockDevice
with RestSender
does not reach topics. Used MockDevice from 0b17335 against RADAR-base/RADAR-Docker@0abdd14 adding topics schema less-value
and schema less-key
.
Rest-Proxy log does not report any error
....
[2017-05-11 21:24:46,946] INFO 172.23.0.6 - - [11/May/2017:21:24:46 +0000] "HEAD / HTTP/1.0" 200 0 1 (io.confluent.rest-utils.requests)
[2017-05-11 21:24:47,303] INFO 172.23.0.6 - - [11/May/2017:21:24:47 +0000] "GET /topics/schemaless-value HTTP/1.0" 200 559 17 (io.confluent.rest-utils.requests)
[2017-05-11 21:24:47,362] INFO 172.23.0.6 - - [11/May/2017:21:24:47 +0000] "HEAD / HTTP/1.0" 200 0 0 (io.confluent.rest-utils.requests)
[2017-05-11 21:24:47,406] INFO 172.23.0.6 - - [11/May/2017:21:24:47 +0000] "HEAD / HTTP/1.0" 200 0 0 (io.confluent.rest-utils.requests)
...
Likewise MongoDb Connector
....
[2017-05-11 21:24:29,617] INFO 0 have been written in MongoDB 0 records need to be processed. (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:24:59,372] INFO 0 have been processed (org.radarcns.mongodb.MongoDbSinkTask)
[2017-05-11 21:24:59,617] INFO 0 have been written in MongoDB 0 records need to be processed. (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:25:29,132] INFO [FLUSH-WRITER] Time-elapsed: 3.402E-6 s (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:25:29,132] INFO [FLUSH] Time elapsed: 1.37946E-4 s (org.radarcns.mongodb.MongoDbSinkTask)
[2017-05-11 21:25:29,132] INFO WorkerSinkTask{id=radar-connector-mongodb-sink-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSinkTask)
[2017-05-11 21:25:29,372] INFO 0 have been processed (org.radarcns.mongodb.MongoDbSinkTask)
[2017-05-11 21:25:29,618] INFO 0 have been written in MongoDB 0 records need to be processed. (org.radarcns.mongodb.MongoDbWriter)
[2017-05-11 21:25:59,372] INFO 0 have been processed (org.radarcns.mongodb.MongoDbSinkTask)
...
Kafka Log files
root@ceca758ee7b8:/var/lib/kafka/data/android_empatica_e4_acceleration-0# /usr/bin/kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files 00000000000000000000.log
Dumping 00000000000000000000.log
Starting offset: 0
As per comment @nivemaham in #11:
There are still few more classes we could add to commons-test that can be reusable in rest-api and mongo-db-connector e2e tests.
org.jetbrains.kotlin:kotlin-stdlib is a Kotlin Standard Library for JVM.
Affected versions of this package are vulnerable to Information Exposure. A Kotlin application using createTempDir
or createTempFile
and placing sensitive information within either of these locations would be leaking this information in a read-only way to other users also on this system.
Note: As of version 1.4.21, the vulnerable functions have been marked as deprecated. Due to still being useable, this advisory is kept as "unfixed".
package org.jlleitschuh.sandbox
import org.junit.jupiter.api.Test
import java.io.BufferedReader
import java.io.File
import java.io.IOException
import java.io.InputStreamReader
import java.nio.file.Files
class KotlinTempDirectoryPermissionCheck {
@Test
fun `kotlin check default directory permissions`() {
val dir = createTempDir()
runLS(dir.parentFile, dir) // Prints drwxr-xr-x
}
@Test
fun `Files check default directory permissions`() {
val dir = Files.createTempDirectory("random-directory")
runLS(dir.toFile().parentFile, dir.toFile()) // Prints drwx------
}
@Test
fun `kotlin check default file permissions`() {
val file = createTempFile()
runLS(file.parentFile, file) // Prints -rw-r--r--
}
@Test
fun `Files check default file permissions`() {
val file = Files.createTempFile("random-file", ".txt")
runLS(file.toFile().parentFile, file.toFile()) // Prints -rw-------
}
private fun runLS(file: File, lookingFor: File) {
val processBuilder = ProcessBuilder()
processBuilder.command("ls", "-l", file.absolutePath)
try {
val process = processBuilder.start()
val output = StringBuilder()
val reader = BufferedReader(
InputStreamReader(process.inputStream)
)
reader.lines().forEach { line ->
if (line.contains("total")) {
output.append(line).append('\n')
}
if (line.contains(lookingFor.name)) {
output.append(line).append('\n')
}
}
val exitVal = process.waitFor()
if (exitVal == 0) {
println("Success!")
println(output)
} else {
//abnormal...
}
} catch (e: IOException) {
e.printStackTrace()
} catch (e: InterruptedException) {
e.printStackTrace()
}
}
}
There is no fixed version for org.jetbrains.kotlin:kotlin-stdlib
.
The new unsafe sender
returns HTTP 415 Unsupported Media Type
while requesting https://localhost/kafka/topics/schemaless-value
.
WARN - Schema for android_empatica_e4_acceleration value was not yet added to the schema registry. [org.radarcns.producer.SchemaRetriever:185]
WARN - Schema for android_empatica_e4_acceleration value was not yet added to the schema registry. [org.radarcns.producer.SchemaRetriever:185]
INFO - Accept: application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json
[org.radarcns.producer.rest.RestSender:245]
INFO - Request{method=POST, url=https://localhost/kafka/topics/schemaless-value, tag=null} [org.radarcns.producer.rest.RestSender:246]
ERROR - FAILED to transmit message: {"error_code":415,"message":"HTTP 415 Unsupported Media Type"} -> {"key_schema":"{"type":"record","name":"MeasurementKey","namespace":"org.radarcns.key","doc":"Measurement key in the RADAR-CNS project","fields":[{"name":"userId","type":{"type":"string","avro.java.string":"String"},"doc"... [org.radarcns.producer.rest.RestSender:258]
ERROR - FAILED to transmit message: {"key_schema":"{"type":"record","name":"MeasurementKey","namespace":"org.radarcns.key","doc":"Measurement key in the RADAR-CNS project","fields":[{"name":"userId","type":{"type":"string","avro.java.string":"String"},"doc"... [org.radarcns.producer.rest.RestSender:264]
java.io.IOException: Failed to submit (HTTP status code 415): {"error_code":415,"message":"HTTP 415 Unsupported Media Type"} at org.radarcns.producer.rest.RestSender$RestTopicSender.send(RestSender.java:260) at org.radarcns.producer.rest.BatchedKafkaSender$BatchedKafkaTopicSender.send(BatchedKafkaSender.java:88) at org.radarcns.mock.MockFile.send(MockFile.java:107) at org.radarcns.mock.MockProducer.start(MockProducer.java:136) at org.radarcns.pipeline.EndToEndTest.streamToKafka(EndToEndTest.java:283) at org.radarcns.pipeline.EndToEndTest.endToEnd(EndToEndTest.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Suppressed: java.io.IOException: Failed to submit (HTTP status code 415): {"error_code":415,"message":"HTTP 415 Unsupported Media Type"} at org.radarcns.producer.rest.RestSender$RestTopicSender.send(RestSender.java:260) at org.radarcns.producer.rest.BatchedKafkaSender$BatchedKafkaTopicSender.flush(BatchedKafkaSender.java:128) at org.radarcns.producer.rest.BatchedKafkaSender$BatchedKafkaTopicSender.close(BatchedKafkaSender.java:137) at org.radarcns.mock.MockFile.send(MockFile.java:113) ... 25 more
com.google.guava:guava is a set of core libraries that includes new collection types (such as multimap and multiset,immutable collections, a graph library, functional types, an in-memory cache and more.
Affected versions of this package are vulnerable to Creation of Temporary File in Directory with Insecure Permissions due to the use of Java's default temporary directory for file creation in FileBackedOutputStream
. Other users and apps on the machine with access to the default Java temporary directory can access the files created by this class. This more fully addresses the underlying issue described in CVE-2020-8908, by deprecating the permissive temp file creation behavior.
NOTE: Even though the security vulnerability is fixed in version 32.0.0, the maintainers recommend using version 32.0.1, as version 32.0.0 breaks some functionality under Windows.
Upgrade com.google.guava:guava
to version 32.0.0-android, 32.0.0-jre or higher.
com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.
Affected versions of this package are vulnerable to Denial of Service (DoS) in the _deserializeFromArray()
function in BeanDeserializer
, due to resource exhaustion when processing a deeply nested array.
NOTE:
For this vulnerability to be exploitable the non-default DeserializationFeature
must be enabled.
Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.
Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.
One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.
When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.
Two common types of DoS vulnerabilities:
High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.
Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws
package
Upgrade com.fasterxml.jackson.core:jackson-databind
to version 2.12.7.1, 2.13.4 or higher.
Right now, the project can only be used as a git submodule/gradle subproject. It would be nicer to publish a version to JCenter each time we do a release. That would solve dependency management automatically. See the JCenter Gradle plugin for more details.
Affected versions of this package are vulnerable to Integer Overflow or Wraparound via the shuffle(int[] input)
function due to improper validation of the multiplications done on the input length.
Exploiting this vulnerability is possible by passing negative, zero, float, very small, or very long values to the shuffle
functions, which later on are multiplicated by four.
A successful exploration results in “java.lang.ArrayIndexOutOfBoundsException" or “java.lang.NegativeArraySizeException” exceptions which can crash the program.
package org.example;
import org.xerial.snappy.BitShuffle;
import java.io.*;
public class Main {
public static void main(String[] args) throws IOException {
int[] original = new int[0x40000000];
byte[] shuffled = BitShuffle.shuffle(original);
System.out.println(shuffled[0]);
}
}
The program will crash, showing the following error (or similar):
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at org.example.Main.main(Main.java:12)
Process finished with exit code 1
Upgrade org.xerial.snappy:snappy-java
to version 1.1.10.1 or higher.
org.apache.commons:commons-compress is an API for working with compression and archive formats.
Affected versions of this package are vulnerable to Improper Input Validation when parsing TAR files. An attacker can manipulate file modification times headers, leading to a denial of service issue via CPU consumption.
Note: This is only exploitable if applications are using the CompressorStreamFactory
class (with auto-detection of file types), TarArchiveInputStream
, and TarFile
classes to parse TAR files.
Upgrade org.apache.commons:commons-compress
to version 1.24.0 or higher.
On Android, Jackson contains a lot of functions that need to be added to Dex. In addition, it uses reflection which is slow on android. RestSender
and SchemaRegistry
could use org.json
instead, which API is already part of Android, so the dependency can be removed there.
org.jetbrains.kotlin:kotlin-stdlib is a Kotlin Standard Library for JVM.
Affected versions of this package are vulnerable to Improper Locking due to inability to lock dependencies for Multiplatform Gradle Projects.
Upgrade org.jetbrains.kotlin:kotlin-stdlib
to version 1.6.0 or higher.
With concurrent senders, a lot of head requests are made to the server. This number should be reduced.
Add custom classes like AbstractKafkaAvroDesrializer,etc required by kafka components like connector, streams,etc as a separate module so they can be shared easily.
Travis build status should indicate FAILED if releases are performed and github or bintrayUpload are not successful.
When using the module radar-commons-server
, the kafka-clients dependency requires maven { url 'http://packages.confluent.io/maven/' }
repository. This should be added in the README.
Affected versions of this package are vulnerable to Denial of Service (DoS) via the hasNextChunk
function due to improper validation of the chunkSize
variable value.
Exploiting this vulnerability is possible by passing a negative number (such as 0xFFFFFFFF
, which is -1), which will cause the code to raise a java.lang.NegativeArraySizeException
exception.
A worse case would happen when passing a huge positive value (such as 0x7FFFFFFF
), raising the fatal java.lang.OutOfMemoryError
error.
package org.example;
import org.xerial.snappy.SnappyInputStream;
import java.io.*;
public class Main {
public static void main(String[] args) throws IOException {
byte[] data = {-126, 'S', 'N', 'A', 'P', 'P', 'Y', 0, 0, 0, 0, 0, 0, 0, 0, 0,(byte) 0x7f, (byte) 0xff, (byte) 0xff, (byte) 0xff};
SnappyInputStream in = new SnappyInputStream(new ByteArrayInputStream(data));
byte[] out = new byte[50];
try {
in.read(out);
}
catch (Exception ignored) {
}
}
}
Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.
Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.
One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.
When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.
Two common types of DoS vulnerabilities:
High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.
Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws
package
Upgrade org.xerial.snappy:snappy-java
to version 1.1.10.1 or higher.
The constructor of AvroTopic class verifies if the valueSchema
contains fields time
and timeReceived
. After the introduction of Questionnaire, this constrain is no longer valid.
We may generalise the constructor adding a dummy validation that a subclass of AvroTopic
can override.
public AvroTopic(String name, Schema keySchema, Schema valueSchema, Class<K> keyClass, Class<V> valueClass) {
super(name);
this.keySchema = keySchema;
this.valueSchema = valueSchema;
validateKey(this.keySchema)
validateSchema(this.valueSchema)
this.valueClass = valueClass;
this.keyClass = keyClass;
List<Schema.Field> fields = valueSchema.getFields();
this.valueFieldTypes = new Schema.Type[fields.size()];
for (int i = 0; i < fields.size(); i++) {
valueFieldTypes[i] = fields.get(i).schema().getType();
}
}
public void validateKey(Schema schema) {}
public void validateSchema(Schema schema) {}
And then for example the subclass SensorTopic
can do
public class SensorTopic <K, V> extends AvroTopic<K, V> {
...
public void validateKey(Schema schema) {}
public void validateSchema(Schema schema) {
if (schema.getField("time") == null) {
throw new IllegalArgumentException("Schema must have time as its first field");
}
if (schema a.getField("timeReceived") == null) {
throw new IllegalArgumentException("Schema must have timeReceived as a field");
}
}
...
}
I do not have a clear idea about possible implications of this change.
@blootsvoets @MaximMoinat what do you think?
com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.
Affected versions of this package are vulnerable to Denial of Service (DoS) in the _deserializeWrappedValue()
function in StdDeserializer.java
, due to resource exhaustion when processing deeply nested arrays.
NOTE: This vulnerability is only exploitable when the non-default UNWRAP_SINGLE_VALUE_ARRAYS
feature is enabled.
Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.
Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.
One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.
When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.
Two common types of DoS vulnerabilities:
High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.
Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws
package
Upgrade com.fasterxml.jackson.core:jackson-databind
to version 2.12.7.1, 2.13.4.1 or higher.
com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.
Affected versions of this package are vulnerable to Denial of Service (DoS) when using JDK serialization to serialize and deserialize JsonNode
values.
It is possible for the attacker to send a 4-byte length payload, with a value of Integer.MAX_VALUE
, that will eventually cause large buffer allocation and out of heap memory.
Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.
Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.
One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.
When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.
Two common types of DoS vulnerabilities:
High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.
Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws
package
Upgrade com.fasterxml.jackson.core:jackson-databind
to version 2.13.1, 2.12.6 or higher.
com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.
Affected versions of this package are vulnerable to Denial of Service (DoS) via a large depth of nested objects.
Denial of Service (DoS) describes a family of attacks, all aimed at making a system inaccessible to its intended and legitimate users.
Unlike other vulnerabilities, DoS attacks usually do not aim at breaching security. Rather, they are focused on making websites and services unavailable to genuine users resulting in downtime.
One popular Denial of Service vulnerability is DDoS (a Distributed Denial of Service), an attack that attempts to clog network pipes to the system by generating a large volume of traffic from many machines.
When it comes to open source libraries, DoS vulnerabilities allow attackers to trigger such a crash or crippling of the service by using a flaw either in the application code or from the use of open source libraries.
Two common types of DoS vulnerabilities:
High CPU/Memory Consumption- An attacker sending crafted requests that could cause the system to take a disproportionate amount of time to process. For example, commons-fileupload:commons-fileupload.
Crash - An attacker sending crafted requests that could cause the system to crash. For Example, npm ws
package
Upgrade com.fasterxml.jackson.core:jackson-databind
to version 2.12.6.1, 2.13.2.1 or higher.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.