budjb/http-requests

An HTTP client abstraction that provides a common interface to several different client implementations.

http-requests's Issues

SL4J Logging Filter does not log Query Parameters

The SLF4J Logging Filter, when applied to requests, fails to output the query parameters in any fashion (either as a separate section, or appended to the URL).

NullPointerException When Using an Incomplete URI

The error is very easy to reproduce.
The URI used needs to NOT throw a URISyntaxException but to also not have all the pieces used in HttpRequest.parseUri(URI uri).
Simple example: httpClient.get('foobar')

Adding Query Params, Headers, and Form Data with Null Values throws NullPointerException

Example with addHeader():

def foo() {
    HttpRequest httpRequest = new HttpRequest()
    httpRequest.addHeader('foo', null)
}

Example with addQueryParameter():

def foo() {
    HttpRequest httpRequest = new HttpRequest()
    httpRequest.addQueryParameter('foo', null)
}

Example with addField():

def foo() {
    FormData formData = new FormData()
    ormData.addField('foo', null)
}

The reason for this is due to the add... methods having a (String, String) and (String, List) signature. Groovy is currently just picking one (unless the user expressly casts the type of null), which picks the List signature, and then the attempt to add null to the MultiValuedMap results in the NullPointerException.

One remedy may be to perform a null-check in both methdos and either return without action, or insert '' (blank String).

A current work-around for Clients is to cast the value or to do like so:

def foo(String bar) {
    HttpRequest httpRequest = new HttpRequest()
    if (bar) {
        httpRequest.addHeader('bar', bar)
    }
}

Or to use the set* methods, which execute different code in MultiValuedMap that does not call addAll and so does not throw a NullPointerException (note: Not available for FormData). However, these cases may not be as desirable, as the value may still be presented as the String 'null', which may not be valid for the intent, which would be a blank ('') value.

Example stacktrace:

Caused by: java.lang.NullPointerException: null
	at java.util.ArrayList.addAll(ArrayList.java:581)
	at com.budjb.httprequests.MultiValuedMap.add(MultiValuedMap.java:42)
	at com.budjb.httprequests.HttpRequest.addHeader(HttpRequest.java:279)

contentType.substring error

there is an error caused by 'contentType.substring(0, contentType.indexOf(';'))'
when clientResponse content type is 'text/plain' in JerseyHttpClient.groovy line 146.
what should I do or you will fix this as a bug?

Support for posting XML content

I'm working with a couple legacy APIs that only support XML for POST Content-Type. Looking through the code I see that http-requests only supports the XmlSlurperEntityReader, but not its ...Writer compatriot.

Are you planning to support posting XML?

can not get response json

when I post an inputStream to a remote server with contentType being set to "multipart/form-data", the response status is 200 but the entity is null. I found the response entity is string when I use apche http.

grails 3.1.6 can not use this plugin

when I compile 'com.budjb:http-requests-grails:0.1.7.BETA' in my project with grails 3.1.6, I can't run my project and throw exception java.lang.IllegalStateException: no provider with type 'com.budjb.httprequests.HttpClientFactory' found on the classpath, can you help me?

Implicitly Decodes Encoded URIs

Not sure if this is to be expected but the HttpRequest.parseUri method is using the getPath() method of java.net.URI instead of getRawPath(). This presents a problem in two places:

1. In AbstractHttpClient.run the request is cloned which triggers another call to HttpRequest.parseUri from HttpRequest.setUri on the new Cloned Object but this will receive the decoded path. This will throw an exception if the original uri required encoding to prevent a URISyntaxException from occuring
2. Also within JerseyHttpClient.doExecute the client.resource call is passed the decoded URI which will throw an exception as well if the decoded URI required encoding.

Maybe I am missing something but I can't see how else to get around the issue given the API provided. I ended up having to extend the HttpRequest class and override the calls to clone and parseUri while using reflection to access the uri field.

EntityConverterManager limitations and solutions

First of all: I have a need for an abstraction for http-clients and was about to start my own, but then I found http-requests - wow, exactly what I had in mind, very nice job 👍

However, the EntityConverterManager is a bit unaccommodating. More specifically with the EntityReader and EntityWriter interfaces: they don't expose the type to be marshaled to/from. This prevents you from fully delegating this to another library, like for example Jackson.

When I implement the EntityReader interface, it looks like I have to hard-code what the marshaling type is ?

I should be able to deserialize json into a custom POJO, without writing a specialized EntityReader for this POJO, etc ...

EntityConverterManager converter = new EntityConverterManager(Arrays.asList(
                new JacksonEntityReader(new ObjectMapper())
        ));

        HttpEntity httpEntity = new HttpEntity(someInputStreamWithJson);
        MyCustomPojo response = converters.read(MyCustomPojo.class, httpEntity);

I have this actually working locally, by subclassing EntityConverterManager, overriding some methods and introducing new -Reader/-Writer interfaces that actually expose the type in the read/write methods. That is of course not ideal, it would be a lot nicer to have this in http-requests-core.

Are you interested in a PR that fixes all of that ?

It is fully possible to do this without losing backwards compatibility, but it makes the current EntityConverterManager, EntityWriter and EntityReader eventually @Deprecated.

How to post MultipartFormEntity ?

How to post MultipartFormEntity ?

AuthenticationTokenHeaderFilter example

Hi, @budjb, thanks so much for the excellent library. I'm using it quite a bit.

I read the docs, source code and tests, but I'm still having trouble understanding how to implement an AuthenticationTokenHeaderFilter. What would the authenticate() method look like if my auth endpoint is http://my.api/auth? Does it need to do a full request/response to get the auth token inside the method?

<hostname>:-1 in Logs

Please note that in some cases when using the http-requests library, you may encounter something like this:

https://example.com:-1/path/to/resource

When you expect something like this:

https://example.com:80/path/to/resource

Or, in other words, the port is shown as :-1.

You may try to investigate why this value is :-1 for a long time. Stop. It is a red herring. Double-check your URI / URL and ensure it is correct. In 100% of cases where this occurred, so far, it has been an issue with either the hostname, or the path being incorrect, and it was determined that the appearance of the :-1 in the port is only a logging anomaly and is just a misdirecting symptom of a different problem.

Redirect from http to https not implemented

By design the underlying Java libraries do not redirects that traverse from HTTP to HTTPS and visa versa because it views the two as different protocols.

Original Java bug
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4620571

Library call that need to be augmented
https://github.com/budjb/http-requests/blob/1.0.2/http-requests-core/src/main/groovy/com/budjb/httprequests/AbstractHttpClient.groovy#L588

Some Basic URL Encoding?

Would it be possible to provide some level of basic URL Encoding for things like spaces?

The following code-snippet of a Grails Controller action (using http-requests-grails:1.0.2) will demonstrate the error:

def index() {
        Slf4jLoggingFilter slf4jLoggingFilter = new Slf4jLoggingFilter()
        slf4jLoggingFilter.setLogger(log)

        HttpClient httpClient = httpClientFactory.createHttpClient()
        httpClient.clearFilters()
        httpClient.addFilter(slf4jLoggingFilter)
        httpClient.addFilter(new HttpStatusExceptionFilter())

        HttpRequest httpRequest = new HttpRequest('http://127.0.0.1:8080/foo bar')

        httpClient.get(httpRequest)
}

For convenience, I would expect the URL to be encoded as http://127.0.0.1:8080/foo+bar for me by the library, but the following error occurs:

java.net.URISyntaxException: Illegal character in path at index 25: http://127.0.0.1:8080/foo bar
	at java.net.URI$Parser.fail(URI.java:2848)
	at java.net.URI$Parser.checkChars(URI.java:3021)
	at java.net.URI$Parser.parseHierarchical(URI.java:3105)
	at java.net.URI$Parser.parse(URI.java:3053)
	at java.net.URI.<init>(URI.java:588)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springsource.loaded.ri.ReflectiveInterceptor.jlrConstructorNewInstance(ReflectiveInterceptor.java:1075)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
	at com.budjb.httprequests.HttpRequest.parseUri(HttpRequest.groovy:420)
	at com.budjb.httprequests.HttpRequest.<init>(HttpRequest.groovy:114)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springsource.loaded.ri.ReflectiveInterceptor.jlrConstructorNewInstance(ReflectiveInterceptor.java:1075)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
	at com.rackspace.vdo.sandbox.SandboxController.index(SandboxController.groovy:41)
	... 59 common frames omitted

While this use-case is demonstrative just of spaces, general encoding may be helpful / convenient.

NullPointerException When Trying to Send Null Payload

How to reproduce:

HttpRequest request = new HttpRequest()
// set request fields

httpClient.post(request, null)

Stacktrace:

java.lang.NullPointerException: null
	at com.budjb.httprequests.converter.EntityConverterManager.write(EntityConverterManager.java:137)
	at com.budjb.httprequests.AbstractHttpClient.execute(AbstractHttpClient.java:104)
	at com.budjb.httprequests.AbstractHttpClient.post(AbstractHttpClient.java:216)
	at com.budjb.httprequests.HttpClient$post.call(Unknown Source)

Whether it is good practice to call httpClient.post() with a null payload is debatable, but I don't believe it should throw a NullPointerException.

HttpResponse.getEntity(Class<T> type) Throws NullPointerException When Response Has no Body

HttpResponse response = httpClient.post(...)

// response had empty body, entity is null
assert response.getEntity() == null

// this will throw a NullPointerException
String entityString = response.getEntity(String.class)

And the stacktrace:

java.lang.NullPointerException: null
        at com.budjb.httprequests.converter.EntityConverterManager.read(EntityConverterManager.java:175)
        at com.budjb.httprequests.HttpResponse.getEntity(HttpResponse.java:162)
        at com.budjb.httprequests.HttpResponse$getEntity.call(Unknown Source)
        ...

encoding error when post

if I post a input stream which contains Chinese characters will response correct, but I get an error like this ""errmsg" -> "invalid charset. please check your request, if include \uxxxx will create fail! hint: [3VS4na0952vr18]"" when I post the same thing with json request. I try to set charset to utf-8, but it didn't work.