Comments (18)
I think this feature might be hiding a bit more complexity than anticipated.
Regarding the token you get back from your authentication API:
- For how long is it valid?
- Is it for a number of requests of a period of time?
- How is it renewed? just by using it, or do you need to refresh it over time?
Regarding the implementation details, as of now, there are 2 ways of sharing state with the HTTP request:
- Offset: you can initially setup some offset properties that are later updated with the information of every record processed, so the last record processed determines the offset used in the next query. This is managed by Kafka Connect and persisted in Kafka, so it allows the connector to continue where it left in case of a restart. The kind of properties we would use here are timestamps or sequence numbers.
- Partition: just today I merged a feature that allows for the connector to partition requests, this allows to hit the same API with entirely different configurations, and route their responses to different topics (this part is not implemented yet). The point is, every partition has its own static state, which would be the set of properties that makes them different from each other.
What I initially thought about the implementation of this use case was to define a global/connector state resolved on startup, and accessible from where offset and partition state are currently being accessed, which is mainly HttpRequestFactory, where the HTTP Request is formed, usually with templated url, body, headers, etc.
However, and thinking about the different authentication scenarios resulting from the different answers to the questions above, we might want to deal with authentication in a specific way, instead of trying to generalize it into the above.
from kafka-connect-http.
Hi @aggarwalb ,
Unfortunately, that usage pattern is not currently supported as of today.
I think it should be simple enough to implement though. feel free to take a look at the code and see if you could implement it yourself, I'll be happy to help any way I can.
Otherwise, it might not be implemented soon, as I'm a bit busy these days. Although I understand the value of this feature, so I think it will eventually be implemented.
Thanks,
Best regards.
from kafka-connect-http.
Hi @castorm ,
I am ready to contribute to this code if you are ok with this. Just one point if you can guide me how you want proceed with this further. This is very important feature as most the REST API world needs security access token to access the API's. If we are able to make this connector support this. then I will use this connector in our production solution else I have to write separate code in streams to fulfill my requirement.
from kafka-connect-http.
The alternative to two calls per Kafka record (if not caching the API key) would be wrap the Kafka Connect API calls and embed the API key in the connector config
- Hit https:///security-api/api (This gives back access token)
- POST to kafka-connect/connectors with
{"rest.api.key": "---key---", "rest.url": "https://customers"}
from kafka-connect-http.
@OneCricketeer When we hit the first API we will get record back in the kafka topic. What you are saying is reading the value from that topic and embed that value in connector config and then hit the second API with connector. Right?
If this is the case I would rather right one stream app in which I can wrap the two calls and process the message from source processor to sink processor.
from kafka-connect-http.
record back in the kafka topic
No. You'd get the key back in an HTTP client, which you then forward as part of the connector config
from kafka-connect-http.
This means extending the existing code ?
from kafka-connect-http.
Assuming your url doesn't support full URI like https://user:[email protected]/
, then yes
from kafka-connect-http.
That is what I have already communicated in the above comments to @castorm that I am ready to contribute to code base for this requirement (as he agreed that this is good to have feature). The only thing I want is how he want things to be
from kafka-connect-http.
@castorm the case I am talking about needs to refresh token after a specified time interval. However, I agree with you that we have to think of all the authentication scenario's available for REST API's. I had seen some implementation being done in kafka http sink connector which we can include in this code base also. If you agree then we can plan implement authentication pattern's in it. By default we can configure it to No Auth.
from kafka-connect-http.
Hi all,
As of 0.8.0 an HttpAuthenticator
has been included to facilitate this extension. For now the available authentication types are just None
and Basic
, but implementing new types should be fairly straightforward as all it takes is implementing the method preparing the Authorization header:
@FunctionalInterface
public interface HttpAuthenticator extends Configurable {
Optional<String> getAuthorizationHeader();
}
Authorization header is requested before executing the request, and after receiving a 401 response (so tokens could be refreshed at that point).
Oath2 or any other contribution for authentication mechanisms are welcome. At some point I might implement it myself if no contribution is received.
Thanks,
Best regards.
from kafka-connect-http.
Hello @castorm, Are there any good examples of a working preauth config? I think the documentation is good when paired with the examples, but none seem to exist for the http.auth stuff yet. I'm new at this and trying to configure preauth.
from kafka-connect-http.
Hi @ShawnUCD,
I'm afraid the reason there are no examples is because it's not just a matter of configuration. You have to implement your own authentication mechanism based on the extension point provided here (the interface shared in my last comment HttpAuthenticator
).
The only authentication mechanism provided out-of-the-box is Basic auth, which, as documented, only requires for you to provide the following configuration properties:
"http.auth.type": "Basic",
"http.auth.user": "your-username",
"http.auth.pass": "your-password",
I hope that clears it up.
Best regards.
from kafka-connect-http.
Yes, thank you for getting back to me. I'm trying to do a two-step auth where a bearer token is obtained (and renewed if expired) after POSTing user/pass (pub/priv API keys in this case) to an endpoint. If this kind of auth is not supported without more code (something that would take me a very long time) then I think I have my answer. Thank you for being so accessible, it's really nice to get such immediate feedback.
Shawn
from kafka-connect-http.
May extend on our fork to cover getting a token from an auth endpoint now, but not sure how generalizable it is. The flow would be
- POST request to configured endpoint, ie.
/auth
, with payload like{"uid": "foo", "pwd": "bar"}
- retrieve token from response at configured path, ie.
access_token
if the response was{"access_token": "abcd1234567"}
Willing to contribute a feature for the main repository too, but some sparing would be welcome to cover as many use-cases as possible.
from kafka-connect-http.
I've given this a shot at #142
from kafka-connect-http.
Just to post an update on this as well. The PR #142 works well in it's current form and I've added a patch to cache the token for a fixed time interval.
from kafka-connect-http.
I might use this. Maybe we can push #142 over the line?
@dingobar or @castorm, can you give guidance on how I might help? In the PR say you want integration tests as well as adaptation to work with OkHttp's Authenticator.
I'm not sure I can get my environment set up to build or run integration tests. I don't think I can really push actual development forward but I'd be happy to test. Or is this connector too dead to have hope? I'm beginning to wonder if Kafka Connect is dead, as it seems silly that there is no popular HTTP source connector anywhere.
Also, for what it's worth, I think fetching a token perhaps should be done preemptively, not having to get a 401 before doing so. If Token_Endpoint is the mode but we have no token, my vote is we should go get one before attempting the API call.
from kafka-connect-http.
Related Issues (20)
- Offset isn't committed directly HOT 2
- Maven package build fails on JDK 11.0.11 HOT 2
- Parse response of the following format {"items":{"k1": {"k1":"v1"} ,"k2":{"k2":"v2"}}} HOT 1
- URL encoding of query parameters HOT 1
- Http source or sink not taking HTTP PROXY define in system level in REDHAT 8 HOT 1
- Handling multi-line JSON response HOT 1
- Make timestamp parser able to use optional property? HOT 1
- [DepShield] (CVSS 7.5) Vulnerability due to usage of org.apache.commons:commons-compress:1.20
- How to transform with REST API
- Does the connector support SSL certification verification
- java.lang.NoClassDefFoundError: freemarker/template/Configuration HOT 1
- Broken link: The "See examples" link on https://castorm.github.io/kafka-connect-http/ is broken
- Connector vulnerabilities HOT 1
- DepShield Deprecation Notice
- Add support for unordered results
- Duplicated records with low `http.timer.interval.millis` HOT 1
- AWS Signv4 authentication
- Maven build failed HOT 2
- Add Bearer Token support for HTTP authentication HOT 1
- Node not found - parse error - flat arry
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-http.