lojewalo/khttp

Kotlin HTTP requests library. Similar to Python requests.

khttp's Issues

Websockets

Hi, thanks for this library.

Do you have any plans to add support for websockets?
Or is there a websocket library you'd recommend in combination with khttp?

Thanks.

Redirected Requests apply Params Again

When I make a GET request to an endpoint that reroutes the protocol, say http://example.com gives a 301 > https://example.com, parameters in the parameter map are applied to the uri twice.

get("http://example.com/", params = mapOf("foo" to "bar"))

Instead of requesting https://example.com/?foo=bar it tries https://example.com/?foo=bar?foo=bar.

This is using the 0.1.0 version on jcenter per the readme.

Use on gradle / Android Project Installation

It would be nice if you see as part of the installation something like:

repositories { ... maven { url "https://jitpack.io" } }

and then you can add the dependencie like:

dependencies { ... implementation "com.github.jkcclemens:khttp:master-SNAPSHOT" }

It's very easy but it will save people that want to use it on Android projects some time.

Regards
I really like the project

Async mode

Hi,

is there an way to do a request asynchronously?

thanks
andre

Http Sessions?

Is there a sample of creating a http browsing session? Python requests does this, and the rabbit page for khttp also says it's possible (http://khttp.readthedocs.io/en/latest/index.html#feature-support)

Unable to post SOAP messages

It seems like it is impossible to post any sort of XML content via khttp.post.

On doing the following -

val response = post(
                    url = soapEndpoint,
                    headers = mapOf(
                            "Accept-Encoding" to "gzip,deflate",
                            "Content-Type" to "text/xml;charset=UTF-8"
                    ),
                    data = getSoapRequest()
            )

I receive an error from the server saying Content-Type cannot be text/plain. But, if you look at my POST invocation above, I explicitly set the Content-Type to be text/xml.

I suspected that my Content-Type header was being overwritten somewhere by the khttp library. I investigated into this and I landed up at the following lines of code within the khttp.requests.GenericRequest class on line 45 -

val DEFAULT_DATA_HEADERS = mapOf(
            "Content-Type" to "text/plain"
        )

And on line 134

if (json == null) {
            this.data = data
            if (data != null) {
                mutableHeaders += GenericRequest.DEFAULT_DATA_HEADERS
            }
        }

So it does seem like your Content-Type header always ends up being over-written in case you're posting XML content.

ignore certificate validation (like curl --insecure) for self-signed or invalid httpS ?

Hi

Thank you for this library.
I was trying to use it for crawling internal company site and since the site uses invalid ssl certificate - my get() requests failed with the below exception.
QUESTION: is there a way to ignore ssl certificate validation when using khttp?

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching   xxxxxx.yyy.zzz.com found.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1944)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1939)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1938)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1508)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
	at khttp.responses.GenericResponse.openRedirectingConnection$khttp(GenericResponse.kt:127)
	at khttp.responses.GenericResponse.getConnection(GenericResponse.kt:163)
	at khttp.responses.GenericResponse.getRaw(GenericResponse.kt:207)
	at khttp.responses.GenericResponse.getContent(GenericResponse.kt:216)
	at khttp.responses.GenericResponse.init$khttp(GenericResponse.kt:350)
	at khttp.KHttp.request(KHttp.kt:59)
	at khttp.KHttp.get(KHttp.kt:28)
	at khttp.KHttp.get$default(KHttp.kt:27)

OpenJDK 11 yields "Illegal reflective access operation has occurred" in GenericRequest

When using a simple khttp.get("...") access, on Ubuntu with OpenJDK 11, you immediately get a massive warning, telling you that:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by khttp.requests.GenericRequest (file:/home/zordid/.gradle/caches/modules-2/files-2.1/khttp/khttp/0.1.0/810c5e89d44b032c2d079aa1c05230e5e7cfcc81/khttp-0.1.0.jar) to field java.net.URL.host
WARNING: Please consider reporting this to the maintainers of khttp.requests.GenericRequest
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Can this be avoided and/or fixed? Thanks!

Test dependencies transitively visible to production

Hi,

When depending on khttp from a Gradle project, the artifacts spek-api & spek-junit-platform-engine get pulled transitively as dependencies to production. Maybe Gradle can't handle test-compile & test-runtime tasks.
These dependencies can be manually excluded when depending on khttp, but still it shouldn't happen.

Kotlin 1.0.0

Had trouble using khttp in a kotlin 1.0.0 environment in Intellij.

Now that Kotlin 1.0.0 is released, it would make sense to migrate.

Here is a patch (without the pom) for khttp. Perhaps you might want to use it...
Upgrade_khttp_to_kotlin_1_0_0.patch.txt

Provide .jar artifacts

I had trouble using JitPack, but those are my problems

The issue with khttp is that there is nowhere one can download actual compiled .jar for the library.

For example JCenter provides such an option, so I can use libs hosted there. I believe it is much easier to setup than Maven Central, you should look into it anyway.

Providing jars for some snapshots right on github is not a bad idea either.

EOFException on empty POST response body

Trying to post to an EventStore stream (https://eventstore.org/docs/http-api/creating-writing-a-stream/index.html?tabs=tabid-1%2Ctabid-3%2Ctabid-5%2Ctabid-7%2Ctabid-17%2Ctabid-11%2Ctabid-13%2Ctabid-15). The expected response is a 201 Created status and an empty response body. The event posts successfully but khttp errors out with the following exception.

java.io.EOFException: null
data-api_1    | 	at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268) ~[na:1.8.0_171]
data-api_1    | 	at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258) ~[na:1.8.0_171]
data-api_1    | 	at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164) ~[na:1.8.0_171]
data-api_1    | 	at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79) ~[na:1.8.0_171]
data-api_1    | 	at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91) ~[na:1.8.0_171]
data-api_1    | 	at khttp.responses.GenericResponse.getRealInputStream(GenericResponse.kt:197)

Tried the same request with http4k and it worked fine.

IllegalStateException: Cannot access request header fields after connection is set

I've stumbled upon what I think is a bug. If I do something like this:

khttp.get(
	Info.RootUrl,
	params = mapOf(
		"action" to "status",
		"minified" to "true"
	),
	headers = mapOf(
		"Accept" to "application/json"
	),
	allowRedirects = false,
	cookies = mapOf( ..... )
)

...then I'm getting the following exception:

Process: com.sbrl.peppermint, PID: 10133
java.lang.IllegalStateException: Cannot access request header fields after connection is set
 at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getRequestProperties(HttpURLConnectionImpl.java:232)
 at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getRequestProperties(DelegatingHttpsURLConnection.java:182)
 at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getRequestProperties(Unknown Source:0)
 at com.android.tools.profiler.support.network.httpurl.TrackedHttpURLConnection.getRequestProperties(TrackedHttpURLConnection.java:204)
 at com.android.tools.profiler.support.network.httpurl.TrackedHttpURLConnection.trackPreConnect(TrackedHttpURLConnection.java:63)
 at com.android.tools.profiler.support.network.httpurl.TrackedHttpURLConnection.connect(TrackedHttpURLConnection.java:126)
 at com.android.tools.profiler.support.network.httpurl.HttpsURLConnection$.connect(HttpsURLConnection$.java:366)
 at khttp.responses.GenericResponse.openRedirectingConnection$khttp(GenericResponse.kt:125)
 at khttp.responses.GenericResponse.getConnection(GenericResponse.kt:163)
 at khttp.responses.GenericResponse.getRaw(GenericResponse.kt:207)
 at khttp.responses.GenericResponse.getContent(GenericResponse.kt:216)
 at khttp.responses.GenericResponse.init$khttp(GenericResponse.kt:377)
 at khttp.KHttp.request(KHttp.kt:61)
 at khttp.KHttp.get(KHttp.kt:30)
 at khttp.KHttp.get$default(KHttp.kt:29)

I really don't understand what's going on here. Sometimes it works once but not twice in a row, and others it doesn't :-( It always comes down to the khttp.get() call above.

Thoughts?

async.get should handle Transfer-Encoding: chunked ?

i am trying ot use a endpoint that sends a json chunk at a time with random intervals.. its connected to a chat server

can i use a get method to process each chunk i receive ?

possibly like this ?

async.get(url, timeout=10.0 ) { chunk -> println(chunk) }

the timeout would here just specify how long it receives chunks and then stop blocking

Proposal: Provide a way to send data in multipart/form-data format

This proposal is about adding a data class FormData(val value: Any, val type: String) to khttp, and accept it in data as Map<String, FormData>. When GenericRequest sees this combination, it sets the Content-Type header to multipart/form-data; boundary=%s, and adds the right Content-Type to every part in that multipart request body. If there are some files to be added to the requests, they can be added too, but this will be optional.

Motivation:

On a previous project of mine, we needed to integrate with a system that accepted JSON as long as we send it encoded with multipart/form-data setting the respected headers.

Looking at GenericRequest.kt, there is almost a way to hack it with khttp by providing data as Map and a files handle that is empty - the only thing missing is an ability to provide the Content-Type for the parts in the Map of the data.

Currently, I have created a workaround as described here: https://kerestey.net/writing/2019-10-26-post-multipart-form-data-using-khttp.html but I'd like to see this resolved in the library itself.

I am certain this needs a little bikeshedding, and maybe there is a better idea of how to handle this out there, therefore the proposal. If there is a consensus on how to resolve this and needs some changes in the library, I would also be happy to try providing a pull request...

incorrectly encoding + in urls

i am running into a prooblem where khttp double encodes data where the webserver returns already encoded data

i solved that temporarily by handling redirects myself and decoding the location header,
but it fails for '+' because khttp does not encode it properly, but double encodes it if i leave it encoded as ' %2B' , the result is then %25B2

(to be fair i wish there was no filenames contianing literal + buit nothing i can do there..)

not sure if this a fault at khttp or the webserver in question.. or both..

the following four urls are bebehaving strangely.. i can downlaod them fine using chrome.. but using khttp the redirects lead to invalid locations

sample:

    var r = get(entry.url, allowRedirects = false)
    while(r.statusCode == 302) {
        val url = URLDecoder.decode(r.headers["Location"]!!, "UTF-8")
        logger.info("following to {}", url)
        r = get(url, allowRedirects = false)
    }
    if(r.statusCode == 200)
    {
       cacheFile.writeBytes(r.content)
    } else {
         logger.error("invalid statusCode {} from {}", r.statusCode, entry.url)
         logger.error("connection url: {}",r.connection.url)
         logger.error("content: {}", r.text)
         return
}

comparison of some apparently wrong r.connection.url

thats what khttp does.. fails

get https://files.forgecdn.net/files/2443/194/BetterBuildersWands-1.12-0.11.1.245+69d0d70.jar
 -> https://media.forgecdn.net/files/2443/194/BetterBuildersWands-1.12-0.11.1.245%2B69d0d70.jar
 -> https://media.forgecdn.net/files/2443/194/BetterBuildersWands-1.12-0.11.1.245%252B69d0d70.jar
-> 403

and now lets try to guide it a little.. fail

get https://media.forgecdn.net/files/2443/194/BetterBuildersWands-1.12-0.11.1.245+69d0d70.jar
-> https://media.forgecdn.net/files/2443/194/BetterBuildersWands-1.12-0.11.1.245+69d0d70.jar
-> 403

this is what it should do

get https://media.forgecdn.net/files/2443/194/BetterBuildersWands-1.12-0.11.1.245+69d0d70.jar
-> https://media.forgecdn.net/files/2443/194/BetterBuildersWands-1.12-0.11.1.245%2B69d0d70.jar
-> 200

it seems like the urlencoded % is getting encoded again, but replacing it with a + does not work either because that does not get properly urlencoded into %2B

it seems to happen both when following redirects automatically and manually

JCenter maven information is not in the docs.

JCenter maven information is not in the docs, only the JitPack information is.
#61 Solves this issue.

NonSdkApiUsedViolation

Hi @ascclemens (I want to first say that I love this library... thanks).
But I am having a huge problem with it. I can no longer deploy my code to Google Play!

Their Pre-launch report says that this library is using blacklisted APIs.

I cannot find any reports of this issue elsewhere, but cannot imagine that we are the first to encounter this problem.

Please let me know how to resolve it.

I tried using the latest version but still have the same problem:
implementation 'io.karn:khttp-android:0.1.2'

Thank you in advance.

Arturo.

Fully restricted
1 error identified
The following APIs are blacklisted and will cause your app to break on all versions of Android

StrictMode policy violation: android.os.strictmode.NonSdkApiUsedViolation: Lcom/android/okhttp/internal/huc/DelegatingHttpsURLConnection;->delegate:Ljava/net/HttpURLConnection;
at android.os.StrictMode.lambda$static$1(StrictMode.java:407)
at android.os.-$$Lambda$StrictMode$lu9ekkHJ2HMz0jd3F8K8MnhenxQ.accept(Unknown Source:2)
at java.lang.Class.getDeclaredField(Native Method)
at khttp.responses.GenericResponse.getField(GenericResponse.kt:351)
at khttp.responses.GenericResponse.updateRequestHeaders(GenericResponse.kt:365)
at khttp.responses.GenericResponse.init$library_release(GenericResponse.kt:380)
at khttp.KHttp.request(KHttp.kt:72)
at khttp.KHttp.get(KHttp.kt:41)
at khttp.KHttp.get$default(KHttp.kt:40)
at ai.mnemo.lib.communication.AccessPointProxy.executeWithDecoded(AccessPointProxy.kt:25)
at ai.mnemo.whoo.SetupActivity.startPairing(SetupActivity.kt:92)
at ai.mnemo.whoo.SetupActivity.access$startPairing(SetupActivity.kt:18)
at ai.mnemo.whoo.SetupActivity$onCreate$1.onClick(SetupActivity.kt:45)
at android.view.View.performClick(View.java:7259)
at android.view.View.performClickInternal(View.java:7236)
at android.view.View.access$3600(View.java:801)
at android.view.View$PerformClick.run(View.java:27892)
at android.os.Handler.handleCallback(Handler.java:883)
at android.os.Handler.dispatchMessage(Handler.java:100)
at androidx.test.espresso.base.Interrogator.a(Interrogator.java:31)
at androidx.test.espresso.base.UiControllerImpl.a(UiControllerImpl.java:132)
at androidx.test.espresso.base.UiControllerImpl.a(UiControllerImpl.java:126)
at androidx.test.espresso.base.UiControllerImpl.a(UiControllerImpl.java:42)
at androidx.test.espresso.action.MotionEvents.a(MotionEvents.java:75)
at androidx.test.espresso.action.Tap.b(Tap.java:16)
at androidx.test.espresso.action.Tap$1.a(Tap.java:2)
at androidx.test.espresso.action.GeneralClickAction.perform(GeneralClickAction.java:11)
at androidx.test.espresso.ViewInteraction$SingleExecutionViewAction.perform(ViewInteraction.java:8)
at androidx.test.espresso.ViewInteraction.a(ViewInteraction.java:33)
at androidx.test.espresso.ViewInteraction$1.call(ViewInteraction.java:2)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at android.os.Handler.handleCallback(Handler.java:883)
at android.os.Handler.dispatchMessage(Handler.java:100)
at android.os.Looper.loop(Looper.java:214)
at android.app.ActivityThread.main(ActivityThread.java:7356)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)

Illegal access with Java 9

When running with java 9 you get the following warning:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by khttp.requests.GenericRequest (file:/C:/Users/Torben/.m2/repository/khttp/khttp/0.1.0/khttp-0.1.0.jar) to field java.net.URL.host
WARNING: Please consider reporting this to the maintainers of khttp.requests.GenericRequest
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Android compatibility

Hi,

Glad to see this library is maintained again, it looks very cool!

I'm opening this issue to let you know about a major Android incompatibility starting Pie, you can read more in this thread: https://stackoverflow.com/questions/53106378/no-field-host-in-class-ljava-net-url/55481683

Also, referring to #54, I want to point out that I had to include the dependency as following due to an error with duplicated classes (more of it here):

dependencies {
    implementation ('khttp:khttp:1.0.0') {
        exclude group: 'org.jetbrains.kotlin', module: 'kotlin-reflect'
    }
}

not possible to POST JSONArray ?

I'm always getting an error 415 media unsupported when trying to post a JSONArray. Is it not possible to do this in khttp ?

When use GET method with body it will always request as POST method instead of GET

When use GET with body it will alway request as POST

Always getting an error when trying to parse response to Json

Whether I try to .jsonArray or .jsonObject, I always get an org.json.JSONException: End of input at character 0 of error.

Warnings: Dependency org.json:json:20150729

Warning:WARNING: Dependency org.json:json:20150729 is ignored for debug as it may be conflicting with the internal version provided by Android.

Warning:WARNING: Dependency org.json:json:20150729 is ignored for release as it may be conflicting with the internal version provided by Android.

Information:BUILD SUCCESSFUL

Content-Type header is overwritten.

It makes sense to set this for the user if they haven't provided it, but the fact that someone has provided data as a string does not make the content type text/plain. I was attempting to use this library to test some Graphql endpoints and unfortunately can't do it because of this.

Feature Support for SSLContext

Hi,

Thank you for this awesome http client. I noticed that there is no option to pass a sslcontext and hostname verifier for ssl configuration. It would be great to have that option as well for https requests. Will there also be a support for SSLContext to configure the client?

Kinds regards,
Hakan

incapsula on coinexchange api fails

Accept=application/json was sufficient, or perhaps upstream configs out of my control.

Can't send a raw binary file without a base64 header

Apparently (I am not an expert) it is possible to send a binary file to a web server without a base64 header. Khttp does not appear to support this.

If you try to set the Content-Type on a binary file to enable this, you get an ArrayIndexOutOfBoundsException at line 98 of GenericRequest. That code assumes that the header includes the String boundary= which it does not do if the user set their own Content-Type.

New stable release?

Hi,

I would like to use khttp for a small project. Unfortunately, the latest stable release 0.1.0 is quite old and doesn't even have the async API. Maybe you could think about releasing a tag with the new functionality?

Convert async implementation to coroutines

Upon updating the Kotlin version for this project, the existing implementation for async functionality should be refactored to coroutines.

Case insensitive headers

I found an issue with cookie extraction on our test environment:

Upon further investigation, we realised the issue has to do with the set-cookie header (and potentially other headers?) not being treated in a case insensitive manner.

I presume the culprit is here https://github.com/jkcclemens/khttp/blob/master/src/main/kotlin/khttp/responses/GenericResponse.kt#L35 and it should be easy to patch (happy to provide a PR if that's acceptable).

failed to resolve org.json:json:20150729 in Android3.0 with kotlin

allprojects { repositories { google() // jcenter() jcenter({url "http://jcenter.bintray.com/"}) maven { url 'https://jitpack.io' } } }
and

compile 'com.github.jkcclemens:khttp:0.1.0'

but once I sync gradle it will tell me the error:
Error:Failed to resolve: org.json:json:20150729

Can't resolve jitpack from Groovy

Unsure if the problem is on my end, or if you didn't pay the monthly JitPack bill, but I can't get this to work:

@GrabResolver(name='jitpack.io', root='https://jitpack.io')
@Grab('com.github.jkcclemens:khttp:-SNAPSHOT')
import khttp

Error:

unable to resolve class khttp
   @GrabResolver(name='jitpack.io', root='https://jitpack.io/')
   ^

authentication header in android

This looked like a nice library for use in an Android app. But using get("https://&c", auth=BasicAuthorization("user", "pass")) as documented gave an error:

W/System.err: java.lang.NoClassDefFoundError: Failed resolution of: Ljava/util/Base64;
W/System.err: at khttp.structures.authorization.BasicAuthorization.getHeader(BasicAuthorization.kt:13)
W/System.err: at khttp.requests.GenericRequest.(GenericRequest.kt:160)
W/System.err: at khttp.KHttp.request(KHttp.kt:60)
W/System.err: at khttp.KHttp.get(KHttp.kt:30)
W/System.err: at khttp.KHttp.get$default(KHttp.kt:29)

Some googling suggests this is because in android, we'd need to use android/util/Base64 instead of java/util/Base64.

If I deal with encoding the user/password myself and inserting them using the generic headers option, it does work fine.

Streaming a file doesn't work (with solution)

Hi,

First of all - great library! Really enjoying the simple API.
I encountered an issue where trying to stream files would most of the time not work if the inputStream didn't report .available() bytes immediately.

My use case was that I was trying to download an image in chunks and update the download progress to a UI. Using Response.contentIterator would return immediately without reading anything because the raw inputStream wasn't reporting any .available() bytes immediately, but if we just waited a bit, it would.
.available() is a non-blocking operation, which is probably not what we want in this case, anyway. We want to block until data is available, and then read all of it until EOF is encountered.

Also, for whatever reason, the site I was downloading the image from didn't always send Content-Length. Can be seen in test output below.

Test code:

for (i in 1..10) {
    val bytes = download("http://www.giantbomb.com/api/image/scale_avatar/2739447-assassins-creed-unity-china-chronicles-1.jpg") { downloaded, total ->
        println("Download progress: $downloaded/$total")
    }
    assert(bytes.size == 11731) { "Invalid size: ${bytes.size}, expected = 11731"}
    println("*******************************")
}

Implementation using Response.contentIterator, will not pass test:

fun download(url: String,
             progress: (downloaded: Int, total: Int) -> Unit = { ignored, ignored2 -> }): ByteArray {
    val response = get(url, stream = true)
    println("available = ${response.raw.available()}")
    
    val contentLength = response.headers["Content-Length"]?.toInt() ?: 32768
    val os = ByteArrayOutputStream(contentLength)
    for (chunk in response.contentIterator(DEFAULT_BUFFER_SIZE)) {
        os.write(chunk)
        progress(os.size(), contentLength)
    }
    return os.toByteArray()
}

Output:

available = 3695
Download progress: 3695/32768

java.lang.AssertionError: Invalid size: 3695, expected = 11731

In the above case, some data was immediately available, but after that none. Since we don't wait for more data to be available, but use the non-blocking poll .availble() which returns 0, the stream is closed. Continue reading below and you will see that the data available varies constantly, sometimes some of it being immediately available, sometimes not.

Blocking implementation that works, I will admit that it's ugly, but it's just meant to get the point across:

fun download(url: String,
             progress: (downloaded: Int, total: Int) -> Unit = { ignored, ignored2 -> }): ByteArray {
    val response = get(url, stream = true)
    println("available = ${response.raw.available()}")

    val start = System.currentTimeMillis()
    val contentLength = response.headers["Content-Length"]?.toInt() ?: 32768
    println("contentLength = $contentLength")
    val os = ByteArrayOutputStream(contentLength)
    val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
    var bytes = response.raw.read(buffer)   // Block until we actually read.
    if (bytes >= 0) println("Waited ${System.currentTimeMillis() - start} ms for data to be available.")
    while (bytes >= 0) {
        os.write(buffer, 0, bytes)
        progress(os.size(), contentLength)
        bytes = response.raw.read(buffer)    // Block until we actually read.
    }
    return os.toByteArray()
}

Output:

available = 0
contentLength = 11731
Waited 187 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************
available = 7775
contentLength = 32768
Waited 0 ms for data to be available.
Download progress: 7775/32768
Download progress: 9135/32768
Download progress: 11731/32768
*******************************
available = 0
contentLength = 11731
Waited 195 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************
available = 0
contentLength = 11731
Waited 187 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************
available = 0
contentLength = 11731
Waited 194 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************
available = 0
contentLength = 11731
Waited 195 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************
available = 7775
contentLength = 32768
Waited 0 ms for data to be available.
Download progress: 7775/32768
Download progress: 9135/32768
Download progress: 11731/32768
*******************************
available = 0
contentLength = 11731
Waited 203 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************
available = 0
contentLength = 11731
Waited 201 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************
available = 0
contentLength = 11731
Waited 203 ms for data to be available.
Download progress: 8192/11731
Download progress: 11731/11731
*******************************

The solution above works whether the site returns content-length or not - just need to prettify it.

Can't use URLs with spaces

The following snippet does not work:

            val request = khttp.get("http://example.com/test test")

The following error is thrown:

java.net.URISyntaxException: Illegal character in path at index 23: http://example.com/test test
    at libcore.net.UriCodec.validate(UriCodec.java:63)
    at java.net.URI.parseURI(URI.java:402)
    at java.net.URI.<init>(URI.java:204)
    at java.net.URL.toURI(URL.java:505)
    at khttp.requests.GenericRequest.toIDN(GenericRequest.kt:195)
    at khttp.requests.GenericRequest.makeRoute(GenericRequest.kt:198)
    at khttp.requests.GenericRequest.<init>(GenericRequest.kt:128)
    at khttp.KHttp.request(KHttp.kt:58)
    at khttp.KHttp.get(KHttp.kt:28)
    at khttp.KHttp.get$default(KHttp.kt:27)

The same line works with python requests.

Accepting Invalid http certificate

Hello

Is there a way for this lib to accept invalid/self-signed SSL certificates?

How to persist cookie

Do you have a documentation of how to configure Sessions with cookie persistence?

HTTP/2 ?

Hi! great project - question, I see that there's a possibility to perform async requests, my question is, is there any plans for this library to support HTTP/2 ? Or, what is the current HTTP/2 status with this library?

https://khttp.readthedocs.io/en/latest/user/advanced.html

thanks!

Outdated readthedocs

Read the Docs needs to be updated for the FileLike API changes in PR #20

json与androidSDK 冲突(Json conflict with the Android SDK)

Maven central

Please publish the package to Maven central.

http://central.sonatype.org/pages/ossrh-guide.html

GenericRequest initialization overwrite customized headers

Hi, pretty new to kotlin, and just started using khttp.

Looks like i found a bug,

In GenericRequest.kt, if there is a customized headers map provided, especially, for header fields that have default values, for example content-type, guess the issue is over here
if (json == null) { this.data = data if (data != null) { mutableHeaders += GenericRequest.DEFAULT_DATA_HEADERS } }

will overwrite the customized value, so even through my customized header indicates content-type: application/json, it got overwritten back to content-type: text/plain

please! how to use send a request to a website with proxys

I want to crawler somethings from other website,and I want to use khttp to do this job,but I do not know how to send a request to a website with proxys

support multiple headers for authorization

It would be nicer if authorization had support for multiple headers.

alternately support a different mechanism where the user could manipulate the request object. here is the doc from the python requests version, http://docs.python-requests.org/en/master/user/advanced/#custom-authentication

please could you add a downloadFile api

add a downloadFile api will better to use id

"" is not a cookie.

When a response's Set-Cookie is empty:

Set-Cookie: 

khttp throws:

Exception in thread "main" java.lang.IllegalArgumentException: "" is not a cookie.
	at khttp.structures.cookie.Cookie$Companion.toCookie(Cookie.kt:13)
	at khttp.structures.cookie.Cookie$Companion.access$toCookie(Cookie.kt:10)
	at khttp.structures.cookie.Cookie.<init>(Cookie.kt:24)
	at khttp.responses.GenericResponse$Companion.getCookieJar$khttp(GenericResponse.kt:35)
	at khttp.responses.GenericResponse$Companion$defaultEndInitializers$3.invoke(GenericResponse.kt:116)
	at khttp.responses.GenericResponse$Companion$defaultEndInitializers$3.invoke(GenericResponse.kt:32)
	at khttp.responses.GenericResponse$connection$2.invoke(GenericResponse.kt:164)
	at khttp.responses.GenericResponse$connection$2.invoke(GenericResponse.kt:30)
	at khttp.responses.GenericResponse.openRedirectingConnection$khttp(GenericResponse.kt:124)
	at khttp.responses.GenericResponse.getConnection(GenericResponse.kt:163)
	at khttp.responses.GenericResponse.getRaw(GenericResponse.kt:207)
	at khttp.responses.GenericResponse.getContent(GenericResponse.kt:216)
	at khttp.responses.GenericResponse.init$khttp(GenericResponse.kt:350)
	at khttp.KHttp.request(KHttp.kt:59)
	at khttp.KHttp.post(KHttp.kt:48)
	at khttp.KHttp.post$default(KHttp.kt:47)

Content-Type is not overridable

We're writing a client for an API that uses "Content-Type: application/json; charset=utf-8". When we make a khttp.put request with this header, the request headers do not include the charset and instead uses the default header "Content-Type: application/json;" for a json parameter and "Content-Type: text/plain" for data.

Cannot connect to url with underscore "_".

issue

try {
    khttp.post("http://example_path:50000/test", json = mapOf("key" to "value"))
} catch (e: Exception) {
    println(e) // => java.net.URISyntaxException: Illegal character in hostname at index 14: 
}

cause

stackoverflow uri,url

hope

post successful. (like a Fuel)

test-env:

sample code

303 Redirects should use the GET method

Sending a POST-request to a site which is then redirecting back to itself results in reposting the request. This can result in an endless loop which eventually throws a StackOverflowException.

At least 303 Redirects should overwrite the request method to GET. Please read here:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections#Temporary_redirections