confluentinc/rest-utils

Utilities and a small framework for building REST services with Jersey, Jackson, and Jetty.

Confluent REST Utils

Confluent REST Utils provides a small framework and utilities for writing Java REST APIs using Jersey, Jackson, Jetty, and Hibernate Validator.

See the examples/ directory for a simple demo service.

Security

The REST Application can support both http and https. https is disabled by default. Use the listeners configuration parameter to control which protocol is used.

The https implementation is similar to Kafka, where the server specifies a keystore and trust store. When SSL client auth is configured, the client must authenticate -- the key in the client's keystore must be trusted by the server's trust store.

Metrics

One metrics group exists that spans all listeners. Meaning, when using multiple listeners and observing metrics, the values of each metric are a combination of all listeners. For example, if the maximum latency for a http listener is 50ms and the maximum latency for a https listener is 100ms, the request-latency-max metric will be 100ms, the max across all listeners.

Contribute

• Source Code: https://github.com/confluentinc/rest-utils
• Issue Tracker: https://github.com/confluentinc/rest-utils/issues

License

The project is licensed under the Apache 2 license.

rest-utils's Issues

Upgrade jetty to fix CVE-2020-27223

I found this issue while working on KAFKA-12400: Upgrade jetty to fix CVE-2020-27223.

In Eclipse Jetty 9.4.6.v20170531 to 9.4.36.v20210114 (inclusive), 10.0.0, and 11.0.0 when Jetty handles a request containing multiple Accept headers with a large number of quality (i.e. q) parameters, the server may enter a denial of service (DoS) state due to high CPU usage processing those quality values, resulting in minutes of CPU time exhausted processing those quality values. (source: CVE-2020-27223)

As of present, confluentinc/rest-utils is using 9.4.36.v20210114, exposing schema registry and ksqldb to this vulnerability. We should fix it.

Allow specifying the network interface the server listens on

See confluentinc/schema-registry#159 for some discussion about the two ways you could interpret the setting host.name. To control which interface the server is listening on, we'd need to add support to rest-utils, which is what sets up the socket. The fix would be in MetricsSelectChannelConnector.

rest-utils uses vulnerable version of hibernate-validator

Description
I checked confluent kafka 5.5.1 distribution with WhiteSource and find out that rest-utils uses vulnerable library:

• hibernate-validator-6.0.17.Final.jar has CVE-2019-10219 and CVE-2020-10693 vulnerabilities. The way to fix it is to upgrade to org.hibernate.validator:hibernate-validator:6.0.20.Final

To Reproduce
Download Confluent Kafka 5.5.1 distribution (for example curl -O http://packages.confluent.io/archive/5.5/confluent-community-5.5.1-2.12.tar.gz)
Open share/java/rest-utils folder in it and find hibernate-validator-6.0.17.Final.jar

Expected behavior

• hibernate-validator upgraded to 6.0.20.Final or higher

Actual behaviour

• hibernate-validator is 6.0.17.Final

Unable to build project from master branch

Hi,
I have cloned the project from master branch and tried running mvn clean install. But unable to do so as getting the below error.

Could not transfer artifact io.confluent:common:pom:7.4.0-886 from/to confluent (${confluent.maven.repo}): Cannot access ${confluent.maven.repo} with type default using the available connector factories: BasicRepositoryConnectorFactory

Please suggest if there is any configuration needed.

Jetty-level errors can return non-JSON data

For example, if you have an invalidly formatted HTTP request, that gets caught before it sees any of our code and generates a 400 with HTML output. I am assuming this gets caught at the Jetty level, which might mean we need to hook into their error system better and ensure we override the resulting error message.

The way this was first encountered was using python and the requests library where a header was set with an incorrect extra colon, i.e.

headers = { 'Content-Type:' : 'application/vnd.kafka.binary.v1+json' }

Custom jetty thread pool?

I read the source code for starting the jetty server, and there is no configuration for the jetty connection pool, but the default value is used.

server = new Server() {
  @Override
  protected void doStop() throws Exception {
	super.doStop();
	Application.this.metrics.close();
	Application.this.doShutdown();
	Application.this.shutdownLatch.countDown();
  }
};

The default configuration of the jetty thread pool is max=200 min=8.
In the case of too much concurrency, jetty performance is not guaranteed

// jetty source code
public QueuedThreadPool()
{
	this(200);
}
public QueuedThreadPool(@Name("maxThreads") int maxThreads)
{
	this(maxThreads, 8);
}

If I increase the thread pool size properly, will it affect?

int maxThreads = 1000; // value from config file
int minThreads = 50;  // value from config file
QueuedThreadPool queuedThreadPool = new QueuedThreadPool(maxThreads, minThreads);
server = new Server(queuedThreadPool) {
  @Override
  protected void doStop() throws Exception {
	super.doStop();
	Application.this.metrics.close();
	Application.this.doShutdown();
	Application.this.shutdownLatch.countDown();
  }
};

Excessive logging despite debug=false is set

Confluent schema-registry produces excessive number of log messages, if it wants to access a schema for a topic key/value which is not defined to have a schema, which should be ignored.

Example (real topic name replaced by a placeholder):
2021-02-04T05:51:16.226 level=ERROR logger=io.confluent.rest.exceptions.DebuggableExceptionMapper thread="qtp1105628551-27" msg="Request Failed with exception " io.confluent.rest.exceptions.RestNotFoundException: Subject '{topicName}-key' not found. at io.confluent.kafka.schemaregistry.rest.exceptions.Errors.subjectNotFoundException(Errors.java:69) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.validateAndGetSchema(KafkaSchemaRegistry.java:913) at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.getSchemaByVersion(SubjectVersionsResource.java:116)

JacksonMessageBodyProvider should only throw exceptions that will generate < 500 HTTP statuses

When testing kafka-rest, I saw I got a 500 response caused by a parsing exception:

    at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:148)
    at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:749)
    at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:59)
    at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:12)
    at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1056)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:124)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:232)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206)
    at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
    at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:238)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:118)
    at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1232)
    at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:676)
    at com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:808)
    at io.confluent.rest.validation.JacksonMessageBodyProvider.readFrom(JacksonMessageBodyProvider.java:65)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:251)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:229)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:149)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:72)
    at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:149)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1124)
    at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:853)
    at org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:262)
    at org.glassfish.jersey.server.internal.inject.EntityParamValueFactoryProvider$EntityValueFactory.provide(EntityParamValueFactoryProvider.java:96)
    at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:136)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:406)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:350)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:106)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:259)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:319)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:219)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:695)

This is caused by rethrowing a JsonMappingException which ends up getting converted into a 500. Instead, it should throw an exception that will get converted into something in the 400 range, probably 400 Bad Request.

"ConfigException: Found a listener without a port" even though port is set

Issue

Getting ConfigException: Found a listener without a port even though a listener with a port is clearly set.

Steps to reproduce

Set a listener with a port, for example confluent-registry-0.confluent-registry-listeners.default.svc.cluster.local:8081

Error message

[2017-06-12 14:20:40,890] ERROR Server died unexpectedly:  (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
io.confluent.common.config.ConfigException: Found a listener without a port. All listeners must have a port. The listener without a port is: confluent-registry-0.confluent-registry-listeners.default.svc.cluster.local:8081
	at io.confluent.rest.Application.parseListeners(Application.java:350)
	at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.getPortForIdentity(KafkaSchemaRegistry.java:196)
	at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:129)
	at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:53)
	at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:37)
	at io.confluent.rest.Application.createServer(Application.java:149)
	at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

Caused by

No sanity checking of wether a URI scheme has been provided before trying to get the URI port, see Application.java#L382

Quick and dirty REPL proof:

JMX metrics: connections-active counter decreases.

When monitoring schema registry through JMX, I get decreasing "connections-active" counter.

I found by debugging that the closed() method of MetricsListener is called two times for every connections, which explains the decrease, but I could not spot the source of the problem.

Create and update kafka connect in standalone mode over REST API

Hi,
I’m trying to create and update kafka connect in standalone mode over REST API.
For create new connector, I use the request:
curl -X POST -H "Content-Type: application/json" --data '{"name2": "local-console-source4", "config": {"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector", "tasks.max":"1", "topic":"connect-test123456789", "file":"/etc/kafka/test8.txt"}}' http://localhost:8083/connectors

The new connector created without any problems.

But, what I should change in the request if I need to update only a file config parameter without recreated the connector?

I’m trying to update with the request:
curl -X POST -H "Content-Type: application/json" --data '{"config": {"file":"/etc/kafka/test8.txt"}}' http://localhost:8083/connectors/local-file-source/config
but:

[root@pnode1 ~]# curl -X POST -H "Content-Type: application/json" --data '{"config": {"file":"/etc/kafka/test8.txt"}}' http://localhost:8083/connectors/local-file-source/config

Error 405
HTTP ERROR: 405
Problem accessing /connectors/local-file-source/config. Reason:
Method Not Allowed
Powered by Jetty://

rest-utils uses some vulnerable dependencies

Description
I checked confluent kafka 6.1.1 distribution with WhiteSource and find out that some libraries have vulnerabilities.
Here they are:

• jetty-io-9.4.38.v20210224.jar has CVE-2021-28165 vulnerability. The way to fix it is to upgrade to org.eclipse.jetty:jetty-io:9.4.39 or org.eclipse.jetty:jetty-io:10.0.2 or org.eclipse.jetty:jetty-io:11.0.2
• jersey-common-2.31.jar has CVE-2021-28168 vulnerability. The way to fix it is to upgrade to org.glassfish.jersey.core:jersey-common:2.34
• jetty-webapp-9.4.38.v20210224.jar has CVE-2021-28164 vulnerability. The way to fix it is to upgrade to org.eclipse.jetty:jetty-webapp:9.4.39
• jetty-server-9.4.38.v20210224.jar has CVE-2021-28164 vulnerability. The way to fix it is to upgrade to org.eclipse.jetty:jetty-webapp:9.4.39
• guava-28.1-jre.jar has CVE-2020-8908 vulnerability. The way to fix it is to upgrade to version 30

To Reproduce
Download Confluent Kafka 6.1.1 distribution (for example curl -O http://packages.confluent.io/archive/6.1/confluent-community-6.1.1.tar.gz)
Open share/java/rest-utils folder in it and find jars, listed above.
Check that these jars with corresponding versions are mentioned in corresponding vulnerability description (for example, on https://nvd.nist.gov/vuln/detail/CVE-2021-28168 jersey-common 2.31 is mentioned in "Known Affected Software Configurations" list)

Expected behavior

• jetty-io upgraded to 9.4.39 or higher
• jersey-common upgraded to 2.34 or higher
• jetty-webapp upgraded to 9.4.39 or higher
• jetty-server upgraded to jetty-webapp:9.4.39 or higher
• guava upgraded to version 30 or higher

Actual behaviour

• jetty-io is 9.4.38
• jersey-common is 2.31
• jetty-webapp is 9.4.38
• jetty-server is 9.4.38
• guava is 28.1-jre

not able to build

on executin mvn clean install ,getting
Failed to execute goal on project rest-utils: Could not resolve dependencies for project io.confluent:rest-utils:jar:3.0.0-SNAPSHOT: The following artifacts could not be resolved: io.confluent:common-config:jar:3.0.0-SNAPSHOT, io.confluent:common-metrics:jar:3.0.0-SNAPSHOT: Failure to find io.confluent:common-config:jar:3.0.0-SNAPSHOT in http://server.development.com:8080/libs-snapshot was cached in the local repository, resolution will not be reattempted until the update interval of snapshots has elapsed or updates are forced -> [Help 1]
[ERROR]

Allow configuring excluded ciphers and SSL protocols

Protocol and ciphers come from JDK. JDK can add new protocols/ciphers at any time.
If we use only inclusion lists we block support of new protocols/ciphers because they will not be in inclusion list and application will ignore theirs.
We need to use exclusion lists so that we don’t block out future support.

newbie issue: building 3.3.0

hi,

i'm trying to build 3.3.0 and i get the following error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check (validate) on project rest-utils-parent: Execution validate of goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check failed: Plugin org.apache.maven.plugins:maven-checkstyle-plugin:2.17 or one of its dependencies could not be resolved: Could not find artifact io.confluent:build-tools:jar:3.3.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]

i'm using the following dockerfile, any suggestions?

FROM maven:latest

ENV CONFLUENT_VERSION v3.3.0

RUN mkdir -p /usr/src/ && \
    cd /usr/src/ && \
    git clone --branch ${CONFLUENT_VERSION} https://github.com/confluentinc/common && \
    cd common && \
    mvn -Dmaven.test.skip=true clean install

RUN cd /usr/src/common/parent && \
    git clone --branch ${CONFLUENT_VERSION} https://github.com/confluentinc/rest-utils && \
    cd rest-utils && \
    mvn -Dmaven.test.skip=true clean install

Unable to build 5.1.x version

I am trying to build Kafka-util from source and it is throwing an exception saying org.apache.kafka:kafka_2.11:jar:test:2.1.1-SNAPSHOT is missing.

[INFO] Installing /Users/anandha_ranganathan/github/confluent/rest-utils/target/rest-utils-parent-5.1.1-SNAPSHOT-tests.jar to /Users/anandha_ranganathan/.m2/repository/io/confluent/rest-utils-parent/5.1.1-SNAPSHOT/rest-utils-parent-5.1.1-SNAPSHOT-tests.jar
[INFO]
[INFO] ----------------------< io.confluent:rest-utils >-----------------------
[INFO] Building rest-utils 5.1.1-SNAPSHOT                                 [2/5]
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for org.apache.kafka:kafka_2.11:jar:test:2.1.1-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.apache.kafka:kafka-clients:jar:test:2.1.1-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] rest-utils-parent 5.1.1-SNAPSHOT ................... SUCCESS [  3.549 s]
[INFO] rest-utils ......................................... FAILURE [  0.691 s]
[INFO] rest-utils-test .................................... SKIPPED
[INFO] rest-utils-example ................................. SKIPPED
[INFO] rest-utils-package 5.1.1-SNAPSHOT .................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.439 s
[INFO] Finished at: 2019-01-16T12:07:16-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project rest-utils: Could not resolve dependencies for project io.confluent:rest-utils:jar:5.1.1-SNAPSHOT: The following artifacts could not be resolved: org.apache.kafka:kafka_2.11:jar:test:2.1.1-SNAPSHOT, org.apache.kafka:kafka-clients:jar:test:2.1.1-SNAPSHOT: Failure to find org.apache.kafka:kafka_2.11:jar:test:2.1.1-SNAPSHOT in http://packages.confluent.io/maven/ was cached in the local repository, resolution will not be reattempted until the update interval of confluent has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

I already built Kafka snapshot and it doesn't have a version of 2.1.1 but it does have 2.1.0 Version.

/.m2/repository/org/apache/kafka/kafka_2.11/2.1.0-SNAPSHOT/kafka_2.11-2.1.0-SNAPSHOT-test.jar

Git doesn't show 2.1 version on the remote.
M28285442GTDY:kafka anandha_ranganathan$ git branch -a
1.0

• trunk
  remotes/origin/0.10.0
  remotes/origin/0.10.1
  remotes/origin/0.10.2
  remotes/origin/0.11.0
  remotes/origin/0.7
  remotes/origin/0.7.0
  remotes/origin/0.7.1
  remotes/origin/0.7.2
  remotes/origin/0.8
  remotes/origin/0.8.0-beta1-candidate1
  remotes/origin/0.8.1
  remotes/origin/0.8.2
  remotes/origin/0.9.0
  remotes/origin/1.0
  remotes/origin/1.1
  remotes/origin/2.0
  remotes/origin/HEAD -> origin/trunk
  remotes/origin/consumer_redesign
  remotes/origin/legacy_client_libraries
  remotes/origin/transactional_messaging
  remotes/origin/trunk
  remotes/origin/update-tests-not-to-use-scala-consumers

How do I resolve this issue?

Add TLS Channel

See confluentinc/kafka-rest#75

rest-utils uses vulnerable version of jakarta.el

Describe the bug
I checked confluent kafka 7.0.0 distribution with WhiteSource and find out that jakarta.el-3.0.3.jar has CVE-2021-28170 vulnerability. The way to fix it is to upgrade to org.glassfish:jakarta.el:3.0.4.

To Reproduce
Download Confluent Kafka 7.0.0 distribution (for example curl -O http://packages.confluent.io/archive/7.0/confluent-7.0.0.tar.gz)
Open share/java/rest-utils folder in it and find jakarta.el-3.0.3.jar.

Expected behavior

• jakarta.el upgraded to 3.0.4 or higher

Actual behaviour

• jakarta.el is 3.0.3

Jetty/Jersey version conflicts

Hi
We are using schema registry client in our rest service and we are facing version conflict with Jetty+Jersey packages.

rest-utils : Jetty (version : 8.1.16.v20140903) and Jersey ( version : 2.6)
our rest service : Jetty (version : 9.2.0.v20140526) and Jersey ( version : 2.9)
But we have to use Jersey 2.9 version for binders in our project.

As rest-utils brings Jetty 8.x version and it depends 2.6 version which breaks our rest service.

I see in latest rest-utils 2.0 snapshot jetty version is upgraded to 9.x. May I know when this would be released ? So we can use 2.x version of rest-utils.

Thanks
Raji

Check if a file transfer has been completed to kafka

Hi,
I need to understand when a file transfer has been completed to kafka with kafka-connect.
It seems the only way to do it is (without check some data in a topic) check an offset-file.
When a file transfer has been completed a new string added to an offset-file.
But an offset-file has binary file format. How can I read some data from an offset-file?
Perhaps, can I use a REST interface?
Are there some other way to check if a file transfer has been completed to kafka?

SaslTest and StaticResourceTest fail on Windows

The following tests do not pass on trunk when building rest-utils on windows:

• SaslTest.testAuthorizedAttempt:143
• SaslTest.testUnauthorizedAttempt:168
• StaticResourceTest.testDefaultServletMapsToIndex:77
• StaticResourceTest.testStaticContent:75

If you skip tests it still builds without issue, however, the above-mentioned tests fail. All tests pass on Mac and based on Linux.

Update to Jetty 9.2 and latest version of Jersey

Once #19 is merged, we can upgrade to Jetty 9.2 (9.2 requires Java 7 while 9.3 requires Java 8) and the latest version of Jersey (seems to be 2.19).

We should also consider removing JettyTestContainer as its documentation implies that it was only needed due to the Java 6 requirement.

rest-utils 6.2.0 uses vulnerable dependencies

Describe the bug
I checked confluent kafka 6.2.0 distribution with WhiteSource and find out that some libraries have vulnerabilities.
Here they are:

• jetty-http-9.4.40.v20210413.jar has CVE-2021-28169 vulnerability. The way to fix it is to upgrade to org.eclipse.jetty:jetty-http:9.4.41.v20210516
• jetty-server-9.4.40.v20210413.jar has CVE-2021-28169 and CVE-2021-34428 vulnerabilities. The way to fix it is to upgrade to org.eclipse.jetty:jetty-server:9.4.41.v20210516
• jetty-servlets-9.4.40.v20210413.jar has CVE-2021-28169 vulnerability. The way to fix it is to upgrade to org.eclipse.jetty:jetty-servlets:9.4.41.v20210516

To Reproduce
Download Confluent Kafka 6.2.0 distribution (for example curl -O http://packages.confluent.io/archive/6.2/confluent-community-6.2.0.tar.gz)
Open share/java/rest-utils folder in it and find jars, listed above.
Check that these jars with corresponding versions are mentioned in corresponding vulnerability description (for example, on https://nvd.nist.gov/vuln/detail/CVE-2021-28169 jetty is mentioned in "Known Affected Software Configurations" list)

Expected behavior

• jetty-http upgraded to 9.4.41.v20210516 or higher
• jetty-server upgraded to 9.4.41.v20210516 or higher
• jetty-servlets upgraded to 9.4.41.v20210516 or higher

Actual behaviour

• jetty-http is 9.4.40.v20210413
• jetty-server is 9.4.40.v20210413
• jetty-servlets is 9.4.40.v20210413

Provide exception that is automatically converted into standard ErrorMessage response with custom, application-specific error code

We already setup WebApplicationExceptionMapper to convert WebApplicationExceptions to ErrorMessages and set the correct response code. However, the error code reported in the JSON response is just the HTTP status code. It should be easy to customize the error code to provide more machine readable information (to complement the human readable error message) while maintaining the HTTP status code.

This could either work with the existing exception mapper (i.e. subclass WebApplicationException and detect the subtype when converting to ErrorMessage) or use it's own independent type. Either way, it would also be helpful to provide all the convenience subclasses like NotFoundException as well.

Expose Jetty Metrics via JMX

We are using the confluent rest proxy and are looking for more detailed metrics about request wait time, thread usage etc.

Jetty has a wealth of JMX metrics but they need to be manually enabled. Can we enable these metrics?
http://www.eclipse.org/jetty/documentation/current/jmx-chapter.html

Java 11 dependency resolution failure on artifact "javafx.base"

Hi. I recently upgraded to Confluent Platform 5.4.0 and pull in kafka-avro-serializer (which has a dependency on rest-utils), and I noticed that our Java 11 builds were failing with a dependency resolution failure:

[error] (tests / update) sbt.librarymanagement.ResolveException: Error downloading org.openjfx:javafx.base:11.0.0-SNAPSHOT

We were able to track this down to a dependency rest-utils declares here

rest-utils/pom.xml

Line 53 in f4b871c

<jersey.version>2.28</jersey.version>

The artifact org.glassfish.jersey.ext:jersey-bean-validation:jar:2.28 brings in a version of hibernate-validator that references the non-published javafx.base artifact.

The issue is described in more detail in this upstream hibernate ticket, and why it's only applicable in Java 11 builds. There's a StackOverflow post that discusses the problem as well. I think upgrading your jersey version may address the issue, but in the meantime I've added an exclusion rule for javafx.base.

Is WindowedCount the right stat type to use in MetricsResourceMethodApplicationListener?

By default, an application is configured to keep two samples in a SampledStat, e.g., WindowedCount, each sample maintains a 30-second window. Typically, metrics are exposed by JmxReporter. When the value of an MBean is being retrieved, it calls the stat's measure method. In WindowedCount, measure simply combines all data points in all (two) samples to get the sum. Unless the MBean retrievals are well aligned with the start of a window with the intervals being (window size * # of samples), I don't see how it could report the right count.

For instance:
T: 0----5----[10]----15----[20]----25----[30]
W: |-----1-----|------2------|------3-------|

Let's assume the window size is 10, the metrics get collected every 10 seconds.
Collection 1 at 15: it returns the sum of (full window 1 and half of window 2), i.e., 0-15
Collection 2 at 25: it returns the sum of (full window 2 and half of window 3), i.e., 10-25
In this example, the values between 10-15 are repeatedly reported.

Can anyone please verify if this is the right understanding?

NPE in MetricsRequestEventListener when @PreMatching ContainerRequestFilter throws exception.

I registered a @PreMatching ContainerRequestFilter via kafka.rest.resource.extension.class and sometimes need to throw exceptions that are mapped to responses. This works fine (client gets the correct response), but the MetricsRequestEventListener logs a NPE.

Root cause: The event RequestEvent.Type.MATCHING_START is never invoked in this case (exception is thrown in PreMatching filter), thus never initializing the wrappedRequestStream field. Later, handling the FINISHED event assumes wrappedRequestStream is set, throwin the NPE.

I believe MetricsRequestEventListener must be patched to initialize the wrappedRequestStream and started without depending on MATCHING_START event, because in this case it is never triggered.

I'm thinking about the following alternative to fix the issue: Replace the condition if (event.getType() == RequestEvent.Type.MATCHING_START) by if (started == 0L) or if (wrappedRequestStream == null) (at https://github.com/confluentinc/rest-utils/blob/5.3.1-post/core/src/main/java/io/confluent/rest/metrics/MetricsResourceMethodApplicationListener.java#L301) to initialize variables the first time any event is triggered.

I'll open a PR.

Cheers,

Fabio

Add support for gzip compression

Embedded jetty should be able to configure gzip compression.

We might also want to figure out what other common features might need to be enabled as long as we are only embedding jetty and not running jetty directly as a servlet container with an xml config it can load to adjust things like this.

build problem

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.confluent:rest-utils-parent:[unknown-version]: Could not transfer artifact io.confluent:common:pom:4.0.0 from/to eapnexus (https://eapnexus.jnj.com/nexus/content/groups/public/): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target and 'parent.relativePath' points at wrong local POM @ line 8, column 13
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project io.confluent:rest-utils-parent:[unknown-version] (C:\Work\Hub\CodeHub\kafka\rest-utils\pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for io.confluent:rest-utils-parent:[unknown-version]: Could not transfer artifact io.confluent:common:pom:4.0.0 from/to eapnexus (https://eapnexus.jnj.com/nexus/content/groups/public/): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target and 'parent.relativePath' points at wrong local POM @ line 8, column 13 -> [Help 2]

org.eclipse.jett.io.EofException: Early EOF Error

When using the Kafka Rest Proxy, we run into the 'org.eclipse.jett.io.EofException: Early EOF' Error on line 282 in the MetrixResoureMethodApplicationListener.java. This only occurs when we have a field in an AVRO message with more than 20,330 characters. It causes a 500 internal server error to return. Any idea why this is occurring? Is there a size limit for the rest proxy somewhere for producers? We have the consumer size set to 64 MB.

Thanks

SSL/HTTPS setup?

Hi Folks,

What is the best way to use rest-util (schema service, kafka-rest, etc) with HTTPS?
Is it better to make it native with Jetty? Or better to use a proxy that terminates HTTPS? Is there any example solution documentation?

Thanks a lot!
-Yushu