Comments (6)
Unfortunately the info in the log doesn't seem to help much.
A couple of things come mind.
Did you create the topic "wimt_collaudo" before running the connector?
Is "http://webservice:8080/count" running and reachabble in your environment?
I'd need more information about how you run it to find the issue?
Thanks,
- lenny
from kafka-connect-rest.
HI @llofberg, yes the topic was created before running the connector.
Even the webservice is reacheable:
root@eb4631b752d0:/# curl -X POST "http://webservice:8080/count"
{"timestamp":"2018-06-14T13:33:55.004+0000","status":400,"error":"Bad Request","message":"Required request body is missing: public void hello.GreetingController.count(java.lang.Object)","path":"/count"}
This is the error log:
echo "===> Launching ... "
+ echo '===> Launching ... '
exec /etc/confluent/docker/launch
+ exec /etc/confluent/docker/launch
[2018-06-14 14:21:28,430] DEBUG expanded subtype org.apache.kafka.connect.connector.Task -> org.apache.kafka.connect.sink.SinkTask (org.reflections.Reflections)
[2018-06-14 14:21:28,430] DEBUG expanded subtype org.apache.kafka.connect.connector.Connector -> org.apache.kafka.connect.sink.SinkConnector (org.reflections.Reflections)
[2018-06-14 14:21:28,475] DEBUG expanded subtype org.apache.kafka.connect.connector.Task -> org.apache.kafka.connect.sink.SinkTask (org.reflections.Reflections)
[2018-06-14 14:21:28,475] DEBUG expanded subtype org.apache.kafka.connect.connector.Connector -> org.apache.kafka.connect.sink.SinkConnector (org.reflections.Reflections)
[2018-06-14 14:21:28,494] DEBUG expanded subtype org.apache.kafka.connect.connector.Task -> org.apache.kafka.connect.sink.SinkTask (org.reflections.Reflections)
[2018-06-14 14:21:28,494] DEBUG expanded subtype org.apache.kafka.connect.connector.Connector -> org.apache.kafka.connect.sink.SinkConnector (org.reflections.Reflections)
Jun 14, 2018 2:21:31 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.
[2018-06-14 14:21:31,347] DEBUG Updating configuration for connector RestSinkConnector configuration: {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.storage.KafkaConfigBackingStore)
[2018-06-14 14:21:31,347] DEBUG Storing new config for task RestSinkConnector-0 this will wait for a commit message before the new config will take effect. New config: {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, task.class=com.tm.kafka.connect.rest.RestSinkTask, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.storage.KafkaConfigBackingStore)
[2018-06-14 14:21:34,862] DEBUG WorkerConnector{id=RestSinkConnector} Initializing connector RestSinkConnector with config {connector.class=com.tm.kafka.connect.rest.RestSinkConnector, rest.sink.method=POST, rest.sink.payload.converter.schema=true, tasks.max=1, topics=wimt_collaudo, rest.sink.properties=Content-Type:application/json, name=RestSinkConnector, rest.sink.url=http://webservice:8080/count, rest.sink.payload.converter.class=com.tm.kafka.connect.rest.converter.JsonPayloadConverter} (org.apache.kafka.connect.runtime.WorkerConnector)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-read (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-send (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-active-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;partition-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-seq-number (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion-skip (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:34,868] DEBUG Added sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;put-batch-time (org.apache.kafka.common.metrics.Metrics)
rest.sink.method = POST
rest.sink.payload.converter.class = class com.tm.kafka.connect.rest.converter.JsonPayloadConverter
rest.sink.payload.converter.schema = true
rest.sink.properties = [Content-Type:application/json]
rest.sink.retry.backoff.ms = 5000
rest.sink.url = http://webservice:8080/count
rest.sink.velocity.template = rest.vm
rest.sink.method = POST
rest.sink.payload.converter.class = class com.tm.kafka.connect.rest.converter.JsonPayloadConverter
rest.sink.payload.converter.schema = true
rest.sink.properties = [Content-Type:application/json]
rest.sink.retry.backoff.ms = 5000
rest.sink.url = http://webservice:8080/count
rest.sink.velocity.template = rest.vm
[2018-06-14 14:21:37,961] DEBUG Stopping sink task, setting client to null (com.tm.kafka.connect.rest.RestSinkTask)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;partition-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion-skip (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-send (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-active-count (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-commit-completion (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;offset-seq-number (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;put-batch-time (org.apache.kafka.common.metrics.Metrics)
[2018-06-14 14:21:37,998] DEBUG Removed sensor with name connect-sensor-group: sink-task-metrics;connector=RestSinkConnector;task=0;sink-record-read (org.apache.kafka.common.metrics.Metrics)
from kafka-connect-rest.
This is how I launch the cointainer:
docker run -d \
--name=kafka-connect \
--net=confluent \
-e CONNECT_PRODUCER_INTERCEPTOR_CLASSES=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor \
-e CONNECT_CONSUMER_INTERCEPTOR_CLASSES=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor \
-e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
-e CONNECT_REST_PORT=8083 \
-e CONNECT_GROUP_ID="docker-connect-group" \
-e CONNECT_CONFIG_STORAGE_TOPIC="docker-connect-configs" \
-e CONNECT_OFFSET_STORAGE_TOPIC="docker-connect-offsets" \
-e CONNECT_STATUS_STORAGE_TOPIC="docker-connect-status" \
-e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
-e CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" \
-e CONNECT_REST_ADVERTISED_HOST_NAME="kafka-connect" \
-e CONNECT_LOG4J_ROOT_LOGLEVEL=DEBUG \
-e CONNECT_LOG4J_LOGGERS=org.reflections=DEBUG \
-e CONNECT_PLUGIN_PATH=/jars \
-e CONNECT_REST_HOST_NAME="kafka-connect" \
-v /home/U370780/kafka-connect-rest/target:/jars \
-v /home/U370780/kafka-connect-rest/examples/spring/config:/config \
confluentinc/cp-kafka-connect:4.1.0
from kafka-connect-rest.
Did you have matching Kafka version (4.1.0) in pom.xml when you built the connector?
from kafka-connect-rest.
Hi @llofberg , in the default pom.xml the kafka version is 1.1.0 and confluent.version is 4.1.1, which property should I change?
<kafka.version>1.1.0</kafka.version>
<confluent.version>4.1.1</confluent.version>
from kafka-connect-rest.
I found the error in the task of the connector:
root@03a30e469d6d:/# curl 10.0.13.86:8083/connectors/RestSinkConnector/status {"name":"RestSinkConnector","connector":{"state":"RUNNING","worker_id":"kafka-connect:8083"},"tasks":[{"state":"FAILED","trace":"org.apache.kafka.connect.errors.DataException: JsonConverter with schemas.enable requires \"schema\" and \"payload\" fields and may not contain additional fields. If you are trying to deserialize plain JSON data, set schemas.enable=false in your converter configuration.\n\tat org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:338)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:467)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:301)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:205)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:173)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\n","id":0,"worker_id":"kafka-connect:8083"}],"type":"sink"}
from kafka-connect-rest.
Related Issues (15)
- compilation error
- kafka-connect-rest and confluent-5.0.0 HOT 1
- support for dynamic request payload HOT 3
- Json sink HOT 2
- URL parameters in source connector don't seem to work HOT 4
- Error in AvroData.java while creating Source Connector HOT 3
- Question about pulling data from https endpoint
- Package and release
- Is this pull new data or pulls all data every time?
- Quick Question regarding GET request with header
- What's the license of this tool? HOT 1
- Sending data from kafka topic to multiple endpoint URLs HOT 2
- `\x0A` in JSON from source HOT 2
- Running in distributed mode HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-rest.