jonathonmcmurray/req

HTTP requests library in kdb+/q

req's Issues

Make default behaviours more configurable

Make it possible to disable parsing with a flag e.g. .req.PARSE (similar to .req.VERBOSE flag).

Also .req.SIGNAL for controlling signalling if HTTP error response received.

Possibly also allow defining a "custom parse" function that receives headers & body and can do whatever necessary.

Also, for .req.VERBOSE allow defining as an int i.e. a handle to write to, so it's possible to configure this to write to stderr instead of stdout etc. - ensure backwards compatibility, cast booleans to int

Cookie tailmatching

Currently, the reQ cookie functionality will perform "tailmatching" for any cookie received with a Host attribute. For example, if a cookie is received with Host=example.com, it will (and should) match e.g. good.example.com subdomain. However, it will also match badexample.com, which is incorrect.

cURL suffered from a similar bug a few years back: https://daniel.haxx.se/blog/2013/04/15/tailmatching-them-cookies/

Add support for DNS-over-HTTPS

DNS-over-HTTPS provides the functionality to perform DNS lookups over HTTPS rather than in plaintext, giving increased security.

APIs for this are provided by Google and CloudFlare, support should be added for these (disabled by default)

Google: https://developers.google.com/speed/public-dns/docs/dns-over-https#api_specification
CloudFlare: https://developers.cloudflare.com/1.1.1.1/dns-over-https/json-format/

CloudFlare also provide a version accepting & returning DNS wireformat rather than JSON, support for that could also be added: https://developers.cloudflare.com/1.1.1.1/dns-over-https/wireformat/

• Google JSON
• CloudFlare JSON
• CloudFlare wireformat
• TTL/cache invalidation
• Fallback to system DNS

RFCs

https://tools.ietf.org/html/rfc7858
https://tools.ietf.org/html/rfc8310
https://tools.ietf.org/html/rfc8484

Move decompression to before display for verbose mode

Currently, decompression (i.e. gzip decompression in kdb 4.0+) happens in .req.parseresp, meaning that verbose mode shows compressed response.

Should be done in .req.send as soon as response is received.

Should also make it simple to plug in additional handlers for different Content-Encoding types, probably via a dictionary of handlers that can be extended by users.

Cache basic authentication credentials

As of 7750c99 reQ supports prompting for Basic auth credentials where server requests them.

In order to prevent re-prompting, reQ should cache the entered credentials for automatic use in future requests

Add caching support

HTTP clients should implement caching as per RFC 7234

The goal of caching in HTTP/1.1 is to significantly improve
performance by reusing a prior response message to satisfy a current
request.

Caching results makes our "requests" faster & reduces load on the server. Should handle Cache-Control & Expires response headers & cache if these are present.

Should also support If-Modified-Since & ETag on HTTP requests, handling the subsequent 304 HTTP status code for unchanged content

• Define cache in-memory object
• Cache responses with Cache-Control & Expires headers
• Detect requests with a cached response & use it
• If-Modified-Since & ETag support
• Handle 304 status code

Restructure library into component parts

Separate out separate components (e.g. cookies) into separate q scripts

Create init.q for loading via qutil (.utl.pkg each component) & req.q for loading in vanilla q (\l each component)

• Separate scripts for each component
• init.q script
• req.q script
• build script - not everyone uses qutil, make an easy to load "monoscript" compiled from others

Add GitHub issue example

GitHub API allows addition of issues, should add this to the GitHub example script.

Ignore case in HTTP response headers

In several places in code (e.g. setting cookies, redirections, content parsing) we rely on response headers having Title-Case - if headers are instead sent lower-case, code doesn't work correctly.

Simple fix is to convert all headers to lower case when received & reference them in this way throughout

./

"Protocol not available" error over https

First off, thanks so much for the reQ package!

I tried to follow your blog article and get this error when trying https protocol. http works fine.

q).utl.require"req"
q).req.g"https://httpbin.org/get"
'conn. OS reports: Protocol not available
  [2]  /opt/miniconda3/envs/build/q/pkg/req/req.q:164: .req.send:
  d:headers[us;pr;hd;p];                                                            //get dictionary of HTTP headers for request
  r:hs d:buildquery[m;pr;nu;h;d;p];                                                 //build query and execute
    ^
  if[v;-1"-- REQUEST --\n",string[hs],"\n",d];                                      //if verbose, log request
q.req))
q).req.g"http://httpbin.org/get"
args   | (`symbol$())!()
headers| `Accept`Connection`Host`User-Agent!("*/*";"close";"httpbin.org";"kdb..
origin | "208.110.70.170"
url    | "http://httpbin.org/get"

I think I installed all dependencies:

(build) [dk@s110519 /home/dk/conda-pkgs]$ conda list
# packages in environment at /opt/miniconda3/envs/build:
#
# Name                    Version                   Build  Channel
ca-certificates           2018.03.07                    0  
json                      0.1.1                         0    file://opt/channel
q                         3.6                           0    file://opt/channel
qhttps                    0.0.1                         0    local
qlic                      2018.11.25                    0    file://opt/channel
qutil                     3.2.1                         0    file://opt/channel
req                       0.1.2                         0    file://opt/channel

Add support for timeouts

Most HTTP clients implement timeout functionality for slow responding servers. In q, the -T cmd line arg or \T command can set a timeout for remotely executed commands, which we can use to timeout a local command by sending it to ourselves over handle 0 e.g.

q)\T 1
q)0(`.req.g;"http://httpbin.org/delay/10")
'stop
  [2]  /home/jmcmurray/miniconda3/q/packages/req/req.q:164: .req.send:
  d:headers[us;pr;hd;p];                                                            //get dictionary of HTTP headers for request
  r:hs d:buildquery[m;pr;nu;h;d;p];                                                 //build query and execute
    ^
  if[v;-1"-- REQUEST --\n",string[hs],"\n",d];                                      //if verbose, log request
q.req))

As an initial version, I'll implement a wrapper that

(i) sets timeout
(ii) performs request (with error trap) over handle 0
(iii) resets timeout
(iv) signals if timeout was hit

Future rewrites may allow a more sophisticated approach.

Add DELETE functions

Add a set of functions to issue DELETE requests, with projections for no body and no custom header scenarios

.req.hap fails if there is an @ in URL path

For example:

.req.hap"wss://stream.binance.com:9443/ws/btcusdt@depth"
"wss://"
"stream.binance.com:9443/ws/btcusdt"
"depth"
,"/"

host should be stream.binance.com:9443 and path /ws/btcusdt@depth

Should use .req.ty when checking type of response

https://github.com/jonathonmcmurray/reQ/blob/master/req.q#L179

:$[(`j in key`)&r[0][`$"Content-Type"]like .h.ty[`json],"*";.j.k;] r[1];

In this line we use .h.ty to check type of response, but we have .req.ty created by reQ which should be used instead (in case of other modules modifying .h.ty etc. e.g. serve from https://github.com/jonathonmcmurray/qutil_packages )

Record permanent redirects

When 301 status code is received, we need to log that this is a permanent URL change, as opposed to temporary with 302.

So we should keep a record when we receive 301s and automatically substitute replaced URLs in future HTTP requests

• Record permanent redirects
• Substitute URLs as necessary in future requests

.req.auth broken on 3.3 and below

With 01ed1d6 backwards compatibility was added for older versions of q (.Q.k>=2.8 tested). However, .req.auth (i.e. base64 encode) was not properly tested and this is broken on .Q.k<=3.3 e.g. on 3.2

λ q req.q
KDB+ 3.2 2015.04.07 Copyright (C) 1993-2015 Kx Systems
w32/ 4()core 4095MB jonat laptop-o8a8co1o 169.254.46.92 NONEXPIRE

q).req.auth "user:pass"
k){x\:y}
'nyi
@
64\:
7697253
q.q))

.req.b64decode is also broken:

q).req.b64decode "aGVsbG8gd29ybGQ="
k){x\:y}
'nyi
@
256\:
6841708
q.q))

Improve usability of .req.post

Currently it's a little cumbersome to pass e.g. a JSON body to a request. We should accept something like

.req.post[url;`json;`a`b`c!1 2 3]

to auto-convert to JSON & put correct Content-Type

Need to consider how it should work with custom headers required as well

Parse URLs without a forward slash before query

Currently, URL parsing splits hostname from path by searching for /

e.g. foo://example.com:8042/over/there?name=ferret

• here, the path (including query) is /over/there?name=ferret

In rare circumstances, it could be possible that a URL has no "path" but does have a query e.g.

foo://example.com:8042?name=ferret

In current implementation, a / is required before the ? to properly parse the path as /?name=ferret.

Should handle this automatically & insert leading / to path if necessary

Update Advent of Code example to use cookiejar

The current Advent of Code example manually adds the Cookie header to each request, this should be changed to add the cookie to .req.cookiejar and let it be sent automatically with requests

Add .req.params function for adding parameters to URL

Add function to more easily construct a URL with parameters

e.g.

q).req.params[url;`param1`param2!(123;"abc")]
"{url}?param1=123&param2=abc"

Add support for multipart formposts

Add support for creating & sending multipart formpost requests (required for, for example, uploading files to Slack: https://api.slack.com/methods/files.upload )

Relevant links:

https://tools.ietf.org/html/rfc7578
https://ec.haxx.se/http-multipart.html

Read login credentials from .netrc file

Read ~/.netrc for login credentials if not passed in URL or cached in q session

https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html
https://httpie.org/doc#netrc

Should fall back to ~/.netrc in .auth.getcache if no value found in cache. Should also be a variable to disable e.g. .auth.usenetrc:1b by default

Improve verbose output

In verbose mode, check if body ends with \n & add one if not

Add qutil package

Create a qutil package for simple use of library

.req.enchd does not accept non-string values

Currently, .req.enchd is broken for non-string values or keys:

q).req.enchd`abc`def!("abc";"def")
'type
  [1]  /home/jmcmurray/git/reQ/req.q:113: .req.enchd:
  /* convert KDB dictionary to HTTP headers */
  :("\r\n" sv ": " sv/:flip (key;value)@\:d),"\r\n\r\n";                            //encode headers dict to HTTP headers
                     ^
 }
q.req))\
q).req.enchd("abc";"def")!"ab"
'type
  [1]  /home/jmcmurray/git/reQ/req.q:113: .req.enchd:
  /* convert KDB dictionary to HTTP headers */
  :("\r\n" sv ": " sv/:flip (key;value)@\:d),"\r\n\r\n";                            //encode headers dict to HTTP headers
                     ^
 }
q.req))\
q).req.enchd("abc";"def")!1 2
'type
  [1]  /home/jmcmurray/git/reQ/req.q:113: .req.enchd:
  /* convert KDB dictionary to HTTP headers */
  :("\r\n" sv ": " sv/:flip (key;value)@\:d),"\r\n\r\n";                            //encode headers dict to HTTP headers
                     ^
 }
q.req))

All of these should work, with keys/values being converted to strings if necessary

Currently keys will be converted in .req.headers if all keys are symbols (which should never happen given default headers are specified in .req.def with string keys)