puppetlabs/clj-http-client

HTTP client library wrapping Apache HttpAsyncClient

puppetlabs/http-client

This is a wrapper around the Apache HttpAsyncClient library providing some extra functionality for configuring SSL in a way compatible with Puppet.

Installation

Add the following dependency to your project.clj file:

Details

Async versions of the http methods are exposed in puppetlabs.http.client.async, and synchronous versions are in puppetlabs.http.client.sync. For information on using these namespaces, see the page on making requests with clojure clients.

Additionally, this library allows you to make requests using Java clients. For information on how to do this, see the page on making requests with java clients.

Testing

The tests require pki files in the dev-resources/ssl/ directory of:

• ca.pem: a CA cert with the CN of "puppet"
• key.pem: a node private key
• cert.pem: a cert signed by ca.pem for the private key at key.pem with a CN of "localhost"
• alternate-ca.pem: a valid but untrusted CA cert

The repo contains these files needed for testing, though if needed you may want to read dev-resources/gen-pki.sh for the commands to generate additional sets of files.

Upgrading dependencies

The APIs provided by httpclient change significantly in 5.0 and some of the internal classes that we've extended change within minor releases of the 4.5 series. Whenever upgrading the apache/httpcomponents dependencies an audit of the Java classes should be undertaken, especially the classes pertaining to SafeRedirectedRequest and RedirectStrategy.

Support

We use the Trapperkeeper project on JIRA for tickets on clj-http-client, although Github issues are welcome too.

clj-http-client's Issues

Charset is overwritten with "; charset=UTF-8" when manually set.

For the Clojure Sync Client:

I'm trying to pass a header:

"content-type": "application/json; version=5"

Which is a requirement of an API I am using... this gets changed, silently, into:

"content-type": "application/json; charset=UTF-8"

Causing the call to fail. There doesn't seem to be an option for this behavior or easy workaround. At the very least, perhaps this should be indicated in the returned :opts for the response, as library currently indicates the header was sent correctly.

I'm currently looking through the patch for TK-145 which seems to touch similar code / issues, however in the mean time would appreciate any guidance or patch.

Thanks!

Update README with clojar distribution

i'm keen on trying out clj-http-client's persistent connection functionality.

is a clojars distribution in the works any time soon?

~B

TrustAnchor with subject "CN=XXX" is not a CA certificate - self signed certificates

Error

Execution error (ValidatorException) at sun.security.validator.PKIXValidator/verifyTrustAnchor (PKIXValidator.java:404).
TrustAnchor with subject "CN=myhost-CA" is not a CA certificate

My code is

(let [response (sync/get "https://myhost:4242/qrs/about"
                           {:ssl-cert "./client.pem"
                            :ssl-key "./client_key.pem"
                            :ssl-ca-cert "./root.pem"
                            :insecure? true
                            ;;:validate-hostnames false
                            :accept :json
                            :headers {"x-qlik-xrfkey" "0123456789abcdef" "X-Qlik-User" "UserDirectory=internal;UserId=sa_repository"}
                            :query-params {"xrfkey" "0123456789abcdef"}}
                            )]
    (println (slurp (:body response)))))

Works with curl

it works if I run

curl --cert ./client.pem --insecure --key ./client_key.pem https://myhost:4242/qrs/about?xrfkey=0123456789abcdef --header "x-qlik-xrfkey: 0123456789abcdef" --header "X-Qlik-User: UserDirectory=internal;UserId=sa_repository" -v
https://help.qlik.com/en-US/sense-developer/February2020/Subsystems/RepositoryServiceAPI/Content/Sense_RepositoryServiceAPI/RepositoryServiceAPI-Example-Connect-cURL-Certificates.htm

Environment

• Clojure 1.10.1
• Platform Debian testing x64

Fails to build with JDK 17

Describe the Bug

Fails to build when attempting to run the build with JDK 17. Based on the error message it looks like this is a check in project.clj but I don't know if this is simply a matter of enabling support for version 17 or if code changes will be required. I tried to see if there was an overall plan for support for newer JDKs but didn't find anything.

The error message is as follows:

$ lein
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
java.lang.Exception: Error loading /home/debian/dev/clj-http-client/project.clj
 at leiningen.core.project$read_raw$fn__7638.invoke (project.clj:1046)
    leiningen.core.project$read_raw.invokeStatic (project.clj:1040)
    leiningen.core.project$read_raw.invoke (project.clj:1036)
    leiningen.core.project$read.invokeStatic (project.clj:1057)
    leiningen.core.project$read.invoke (project.clj:1054)
    leiningen.core.project$read.invokeStatic (project.clj:1058)
    leiningen.core.project$read.invoke (project.clj:1054)
    leiningen.core.main$_main$fn__7024.invoke (main.clj:447)
    leiningen.core.main$_main.invokeStatic (main.clj:442)
    leiningen.core.main$_main.doInvoke (main.clj:439)
    clojure.lang.RestFn.invoke (RestFn.java:397)
    clojure.lang.AFn.applyToHelper (AFn.java:152)
    clojure.lang.RestFn.applyTo (RestFn.java:132)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.main$main_opt.invokeStatic (main.clj:514)
    clojure.main$main_opt.invoke (main.clj:510)
    clojure.main$main.invokeStatic (main.clj:664)
    clojure.main$main.doInvoke (main.clj:616)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.main.main (main.java:40)
Caused by: clojure.lang.Compiler$CompilerException: Syntax error compiling at (/home/debian/dev/clj-http-client/project.clj:1:1).
#:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source "/home/debian/dev/clj-http-client/project.clj"}
 at clojure.lang.Compiler.load (Compiler.java:7652)
    clojure.lang.Compiler.loadFile (Compiler.java:7578)
    clojure.lang.RT$3.invoke (RT.java:327)
    leiningen.core.project$read_raw$fn__7638.invoke (project.clj:1044)
    leiningen.core.project$read_raw.invokeStatic (project.clj:1040)
    leiningen.core.project$read_raw.invoke (project.clj:1036)
    leiningen.core.project$read.invokeStatic (project.clj:1057)
    leiningen.core.project$read.invoke (project.clj:1054)
    leiningen.core.project$read.invokeStatic (project.clj:1058)
    leiningen.core.project$read.invoke (project.clj:1054)
    leiningen.core.main$_main$fn__7024.invoke (main.clj:447)
    leiningen.core.main$_main.invokeStatic (main.clj:442)
    leiningen.core.main$_main.doInvoke (main.clj:439)
    clojure.lang.RestFn.invoke (RestFn.java:397)
    clojure.lang.AFn.applyToHelper (AFn.java:152)
    clojure.lang.RestFn.applyTo (RestFn.java:132)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.main$main_opt.invokeStatic (main.clj:514)
    clojure.main$main_opt.invoke (main.clj:510)
    clojure.main$main.invokeStatic (main.clj:664)
    clojure.main$main.doInvoke (main.clj:616)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.main.main (main.java:40)
Caused by: clojure.lang.ExceptionInfo: Unsupported major Java version. Expects 8 or 11.
{:major "17", :minor "0"}
 at leiningen.core.project$eval657.invokeStatic (project.clj:58)
    leiningen.core.project$eval657.invoke (project.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:7181)
    clojure.lang.Compiler.load (Compiler.java:7640)
    clojure.lang.Compiler.loadFile (Compiler.java:7578)
    clojure.lang.RT$3.invoke (RT.java:327)
    leiningen.core.project$read_raw$fn__7638.invoke (project.clj:1044)
    leiningen.core.project$read_raw.invokeStatic (project.clj:1040)
    leiningen.core.project$read_raw.invoke (project.clj:1036)
    leiningen.core.project$read.invokeStatic (project.clj:1057)
    leiningen.core.project$read.invoke (project.clj:1054)
    leiningen.core.project$read.invokeStatic (project.clj:1058)
    leiningen.core.project$read.invoke (project.clj:1054)
    leiningen.core.main$_main$fn__7024.invoke (main.clj:447)
    leiningen.core.main$_main.invokeStatic (main.clj:442)
    leiningen.core.main$_main.doInvoke (main.clj:439)
    clojure.lang.RestFn.invoke (RestFn.java:397)
    clojure.lang.AFn.applyToHelper (AFn.java:152)
    clojure.lang.RestFn.applyTo (RestFn.java:132)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.main$main_opt.invokeStatic (main.clj:514)
    clojure.main$main_opt.invoke (main.clj:510)
    clojure.main$main.invokeStatic (main.clj:664)
    clojure.main$main.doInvoke (main.clj:616)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.main.main (main.java:40)

Expected Behavior

Since JDK 17 now is the current LTS version of Java, I would expect to be able to build and run this and other repositories with it.

Steps to Reproduce

Steps to reproduce the behavior:

1. Install and set JDK 17 as the default Java version on your system
2. Run lein in the root directory.

Environment

• Version: latest main
• Platform: Debian Sid
• JDK: OpenJDK 17.0.5

Additional Context

I've seen similar errors on at least
https://github.com/puppetlabs/trapperkeeper-metrics
https://github.com/puppetlabs/trapperkeeper-webserver-jetty9
Since those repos contain the same version number check, I would assume other clojure repos may also be affected.

Testsuite failure: sync-client-test-ssl-protocols (HttpAsyncRequestExecutor.java:356) should not connect to a server when protocols don't overlap clojure sync client

Describe the Bug

The testsuite currently fails with this error:

ERROR in (sync-client-test-ssl-protocols) (HttpAsyncRequestExecutor.java:350)
should not connect to a server when protocols don't overlap clojure sync client
expected: (thrown? SSLException (clj-https-get-with-protocols ["TLSv1.2"] nil))
  actual: org.apache.http.ConnectionClosedException: Connection closed
 at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput (HttpAsyncRequestExecutor.java:350)
    org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput (DefaultNHttpClientConnection.java:261)
    org.apache.http.impl.nio.client.InternalIODispatch.onInputReady (InternalIODispatch.java:81)
    org.apache.http.impl.nio.client.InternalIODispatch.onInputReady (InternalIODispatch.java:39)
    org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady (AbstractIODispatch.java:121)
    org.apache.http.impl.nio.reactor.BaseIOReactor.readable (BaseIOReactor.java:162)
    org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent (AbstractIOReactor.java:337)
    org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents (AbstractIOReactor.java:315)
    org.apache.http.impl.nio.reactor.AbstractIOReactor.execute (AbstractIOReactor.java:276)
    org.apache.http.impl.nio.reactor.BaseIOReactor.execute (BaseIOReactor.java:104)
    org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run (AbstractMultiworkerIOReactor.java:588)
    java.lang.Thread.run (Thread.java:829)

Expected Behavior

The testsuite should not fail :)

Steps to Reproduce

Steps to reproduce the behavior:

1. Spin a new Debian unstable VM
2. git clone https://github.com/puppetlabs/clj-http-client
3. git checkout 1.2.0
4. lein test :only puppetlabs.http.client.sync-ssl-test/sync-client-test-ssl-protocols

Environment

• Version: 1.2.0
• Platform: Debian unstable

Additional Context

I have been able to reproduce this failure:

1. On a Debian unstable machine using libraries downloaded from Maven Central (probably what you do in your dev environment)
2. On a Debian unstable machine in the puppetlabs-http-client-clojure Debian package (thus using Debian libraries only): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986513

Happy to provide more info or run tests if you need me to.