zmthy/http-media

Processing HTTP Content-Type and Accept headers

http-media

This library is intended to be a comprehensive solution to parsing and selecting quality-indexed values in HTTP headers. It is capable of parsing both media types and language parameters from the Accept and Content header families, and can be extended to match against other accept headers as well. Selecting the appropriate header value is achieved by comparing a list of server options against the quality-indexed values supplied by the client.

In the following example, the Accept header is parsed and then matched against a list of server options to serve the appropriate media using mapAcceptMedia:

send = getHeader >>= maybe send406Error sendResourceWith . mapAcceptMedia
    [ ("text/html",        asHtml)
    , ("application/json", asJson)
    ]

Similarly, the Content-Type header can be used to produce a parser for request bodies based on the given content type with mapContentMedia:

recv = getContentType >>= maybe send415Error readRequestBodyWith . mapContentMedia
    [ ("application/json", parseJson)
    , ("text/plain",       parseText)
    ]

The API is agnostic to your choice of server.

http-media's Issues

bytestring-0.12 not accepted

This line constrains it:

• http-media/http-media.cabal

  Line 83 in e4b2ac2

  bytestring >= 0.10 && < 0.12,

Comparison of `charset` parameter's should be case-insensitive

As specified in [1], names for character encodings are case-insensitive. So comparison of charset values should be special-cased to not take case into account. In the current version of the library, there is no special handling of charset, so the following fails:

λ> matchAccept ["application/json;charset=utf-8"] "application/json;charset=UTF-8" :: Maybe MediaType
Nothing

[1] http://www.w3.org/TR/html4/charset.html#h-5.2.1

Release master?

It would help with making servant-* packages work with ghc8.10.

Thanks! :)

Missing Haddock in hackage.

There's no haddocks for the latest version of this package on http://hackage.haskell.org/package/http-media

Pretty-printing ByteStrings for Accept header

I like the general setup of your package, but it is focused mostly on the parsing side and not so much on the pretty-printing side. Just as you include ByteString inputs for each type, you may also want to include functionality for outputting ByteStrings from each type. That would make the package even more comprehensive and useful.

You do have Network.HTTP.Media.MediaType.toByteString, which is nice for the Content-Type header.

One aspect that seems to be missing is pretty-printing a list of media types with optional quality, necessary for the Accept header. A simpler, but also useful, solution would be pretty-printing a [MediaType] without quality.

Support GHC-9.6

base-4.18 seems to be unnecessarily excluded in the cabal file. Allow-newer seems to work, so possibly a revision would be sufficient.

Allow deconstructing Quality values

I've got a use case in which I want to see what the priorities are for the different content types. Unfortunately, there doesn't seem to be a way to get at the information through the Quality type... looking at the datatype, it doesn't seem too risky to provide a way to enable accessing the data after the fact via some sort of function that reconstructs the priority.

Simultaneous match and map accept

In the interests of self describing media, I find myself often implementing these:

mapDescribe :: Accept a => [(a, b)] -> ByteString -> Maybe (a, b)
mapDescribe options header = mapAccept (map (\(a, b) -> (a, (a, b))) options) header

mapQualityDescribe :: Accept a => [(Quality a, b)] -> [Quality a] -> Maybe (a, b)
mapQualityDescribe options header = mapQuality (map (\(a, b) -> (a, (a, b))) options) header

It seems like this sort of function belongs to this library rather than users' code since a) it's quite useful for self-describing media, and b) an internal implementation can do it with fewer thunks. I'm not really sure about the names I've chosen here, though.

Inconsistent content matching in the presence of parameters

Does anyone have an idea what's going on here?

ghci> M.mapContentMedia [("text" M.// "plain" M./: ("charset","utf-8"), 1)] "text/plain"
Nothing

ghci> M.mapContentMedia [("text" M.// "plain" M./: ("charset","utf-8"), 1), ("text" M.// "plain", 2)] "text/plain"
Just 1

For context: servant's PlainText corresponds to text/plain;charset=utf-8 and will not accept text/plain. Upon adding a Latin1 content type defined to handle both text/plain and text/plain;charset=iso-8859-1 servant suddenly tries to parse text/plain using PlainText.

Exception building wildcard media types with `(//)`

> "application" // "*"
*** Exception: Invalid character in "*"

MediaType parameters' values shouldn't be case-insensitive

Just yesterday I was reminded of subj. Instead, said parameters defined as (CI ByteString) as well as parameter names at

http-media/src/Network/HTTP/Media/MediaType/Internal.hs

Line 87 in 5db8b78

type Parameters = Map (CI ByteString) (CI ByteString)

Is this intentional? Because https://datatracker.ietf.org/doc/html/rfc7231#page-9 says otherwise IMO

The type, subtype, and parameter name tokens are case-insensitive. Parameter values might or might not be case-sensitive, depending on the semantics of the parameter name.

`matchAccept` doesn't properly process `*/*` (media-range with asterisks/stars/wildcards) when the `Accept` instance is `ByteString`

Minimum code to reproduce (and its result)

-- lts-18.6 contains http-media-0.8.0.0:
$ stack --resolver=lts-18.6 exec --package=http-media --package=bytestring  ghci
> import Network.HTTP.Media
> import Data.ByteString (ByteString)
> :set -XOverloadedStrings
> matchAccept ["text/plain" :: ByteString] "*/*"
Nothing
> matchAccept ["text" // "plain"] "*/*"
Just text/plain

Expected result

matchAccept ["text/plain" :: ByteString] "*/* should also return a Just as matchAccept ["text" // "plain"] "*/*" does.

Test suite compiles with "Stub" postfix

When nix tries to build http-media, we see the following error

In-place registering test-http-media-0.4.0...
[1 of 1] Compiling Main             ( dist/build/test-http-mediaStub/test-http-mediaStub-tmp/test-http-mediaStub.hs, dist/build/test-http-mediaStub/test-http-mediaStub-tmp/Main.dyn_o )
Linking dist/build/test-http-mediaStub/test-http-mediaStub ...
ghc: panic! (the 'impossible' happened)
  (GHC version 7.8.4 for x86_64-unknown-linux):
    Don't understand library name test-http-media

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

builder for ‘/nix/store/sw1kfp4ryf8zv8bqq58c40fhxkgyh9fd-haskell-http-media-0.4.0.drv’ failed with exit code 1

Any idea why the tests get compiled as test-http-mediaStub instead of test-http-media?

hackage release doesn't allow ghc-9.2 and newer libs

unlike git master

setup: Encountered missing or private dependencies:
QuickCheck >=2.8 && <2.14,
base >=4.8 && <4.13,
test-framework >=0.8 && <0.9,
test-framework-quickcheck2 >=0.3 && <0.4

Property Test Failure

I got a test failure when building 0.7.1.2. Here's the relevant part of the output

Media:
  parseQuality:
    Without quality: [OK, passed 100 tests]
    With quality: [OK, passed 100 tests]
    With extensions: [OK, passed 100 tests]
  matchAccept:
    Highest quality: [OK, passed 100 tests]
    Most specific: [OK, passed 100 tests]
    Nothing: [OK, passed 100 tests]
    Never chooses q=0: [OK, passed 100 tests]
    Left biased: [OK, passed 100 tests]
    Against */*: [OK, passed 100 tests]
    Against type/*: [Failed]
*** Failed! Falsifiable (after 6 tests):
Just R/c;s=d;o=WyKaS;iH=m;I=ls;0=yf9qR /= Just R/C
(used seed 6662628523875800959)

It passed on the next run.

Fails to build with GHC 7.10.1 – ambiguous type variables

Here's the output from an attempted build of http-media with GHC 7.10.1:

src/Network/HTTP/Media/MediaType.hs:97:27:
    No instance for (Foldable t0) arising from a use of ‘notElem’
    The type variable ‘t0’ is ambiguous
    Note: there are several potential instances:
      instance Foldable (Control.Applicative.Const m)
        -- Defined in ‘Control.Applicative’
      instance Foldable (Either a) -- Defined in ‘Data.Foldable’
      instance Foldable Data.Proxy.Proxy -- Defined in ‘Data.Foldable’
      ...plus six others
    In the first argument of ‘ensure’, namely ‘(`notElem` ",;")’
    In the second argument of ‘(.)’, namely ‘ensure (`notElem` ",;")’
    In the expression: CI.mk . ensure (`notElem` ",;")

src/Network/HTTP/Media/MediaType.hs:97:37:
    No instance for (Data.String.IsString (t0 Char))
      arising from the literal ‘",;"’
    The type variable ‘t0’ is ambiguous
    Note: there are several potential instances:
      instance (Data.String.IsString s, CI.FoldCase s) =>
               Data.String.IsString (CI s)
        -- Defined in ‘case-insensitive-1.2.0.4:Data.CaseInsensitive.Internal’
      instance Data.String.IsString [Char] -- Defined in ‘Data.String’
    In the second argument of ‘notElem’, namely ‘",;"’
    In the first argument of ‘ensure’, namely ‘(`notElem` ",;")’
    In the second argument of ‘(.)’, namely ‘ensure (`notElem` ",;")’

Accept-Charset and Accept-Encoding support

Support for Accept-Charset and Accept-Encoding parsing and choosing would be really useful to me.

q=0 can be overridden by a less specific match

For instance, text/html;q=0,* will still match text/html resources despite the fact that they are explicitly unacceptable because the * is considered sufficient to match it.

Identical media types with whitespace in parameters fail to match

The following code should succeed, but returns Nothing:

λ> matchAccept ["application/json; charset=utf-8"] "application/json; charset=utf-8" :: Maybe MediaType
Nothing

However, if you remove the space after the semicolon, it succeeds:

λ> matchAccept ["application/json;charset=utf-8"] "application/json;charset=utf-8" :: Maybe MediaType
Just application/json;charset=utf-8

It looks like this might be related to the library not ignoring whitespace in the parameters:

λ> parameters "application/json; charset=utf-8"
fromList [(" charset","utf-8")]

compile error with http-media-0.7 and base-4.6

Configuring component lib from http-media-0.7.0...
Preprocessing library http-media-0.7.0...

src/Network/HTTP/Media.hs:61:18:
    Could not find module `Data.Proxy'

I've already revised the meta-data at https://hackage.haskell.org/package/http-media-0.7.0/revisions/ so there's no immediate need to upload a new release (unless for restoring base-4.6 support).

See also https://matrix.hackage.haskell.org/package/http-media for an overview

PS: While at it, I also corrected the meta-data for

• https://hackage.haskell.org/package/http-media-0.6.0/revisions
• https://hackage.haskell.org/package/http-media-0.5.1/revisions
• https://hackage.haskell.org/package/http-media-0.5.0/revisions

which required a tighter upper bound on base due to the Foldable/Traversable incompat

one test failed with stackage nightly ghc-7.10.2

Not sure if this is due to the change to ghc-7.10.2
but one of the tests for http-media-0.6.2 failed like this:

Media:
  parseQuality:
    Without quality: [OK, passed 100 tests]
    With quality: [OK, passed 100 tests]
    With extensions: [OK, passed 100 tests]
  matchAccept:
    Highest quality: [OK, passed 100 tests]
    Most specific: [OK, passed 100 tests]
    Nothing: [Failed]
*** Failed! Falsifiable (after 2 tests):
(used seed 6944558893301243320)
    Never chooses q=0: [OK, passed 100 tests]
    Left biased: [OK, passed 100 tests]
    Against */*: [OK, passed 100 tests]
    Against type/*: [OK, passed 100 tests]

while trying to build today Stackage nightly

Derive Generic for MediaType

Is it possible to derive the Generic typeclass for the MediaType type located in
Network.HTTP.Media.MediaType ?

I'm using the type to represent something in a Dhall config and need to have an Interpret instance, so that's why I would need it to derive Generic.

Problem with `Accept` Parsing

We're using servant, and we're running into two issue with a client sending us the following Accept header:

Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2

Both of these issues causes servant to return a 404, but I wanted to check to see if this should be fixed in http-media instead.

Issue 1: Not parsing when the leading 0 is missing

parseQuality returns Nothing when the q-value doesn't have a leading 0:

> parseQuality "*/*; q=0.2" :: Maybe [Quality MediaType]
Just [*/*;q=0.2]
> parseQuality "*/*; q=.2" :: Maybe [Quality MediaType]
Nothing

Issue 2: Not parsing a single *

parseAccept/parseQuality returns Nothing when * is sent instead of */*:

> parseQuality "*/*" :: Maybe [Quality MediaType]
Just [*/*;q=1]
> parseQuality "*/*; q=0.2" :: Maybe [Quality MediaType]
Just [*/*;q=0.2]
> parseQuality "*" :: Maybe [Quality MediaType]
Nothing
> parseAccept "*/*" :: Maybe MediaType
Just */*
> parseAccept "*" :: Maybe MediaType
Nothing

containers-0.7 not accepted

GHC-9.10 ships containers-0.7, which means that http-media isn't accepting GHC-9.10 right now. It would be nice if this bound was bumped, for example with a Hackage revision.

Quoted string parameters

According to RFC 7231, parameter values can be quoted, and the quoted form should be considered equivalent to the unquoted form.

A parameter value that matches the token production can be transmitted either as a token or within a quoted-string. The quoted and unquoted values are equivalent. For example, the following examples are all equivalent, but the first is preferred for consistency:

text/html;charset=utf-8
text/html;charset=UTF-8
Text/HTML;Charset="utf-8"
text/html; charset="utf-8"

RFC 7230 defines the syntax of quoted-string.

Test suite fails to compile

Resolving dependencies...
Configuring http-media-0.5.1...
Building http-media-0.5.1...
Preprocessing library http-media-0.5.1...
In-place registering http-media-0.5.1...
Preprocessing test suite 'test-http-media' for http-media-0.5.1...
[ 1 of 17] Compiling Network.HTTP.Media.Utils ( src/Network/HTTP/Media/Utils.hs, dist/build/Network/HTTP/Media/Utils.o )
[ 2 of 17] Compiling Network.HTTP.Media.RenderHeader ( src/Network/HTTP/Media/RenderHeader.hs, dist/build/Network/HTTP/Media/RenderHeader.o )
[ 3 of 17] Compiling Network.HTTP.Media.Quality ( src/Network/HTTP/Media/Quality.hs, dist/build/Network/HTTP/Media/Quality.o )
[ 4 of 17] Compiling Network.HTTP.Media.Accept ( src/Network/HTTP/Media/Accept.hs, dist/build/Network/HTTP/Media/Accept.o )
[ 5 of 17] Compiling Network.HTTP.Media.Language.Internal ( src/Network/HTTP/Media/Language/Internal.hs, dist/build/Network/HTTP/Media/Language/Internal.o )
[ 6 of 17] Compiling Network.HTTP.Media.Language ( src/Network/HTTP/Media/Language.hs, dist/build/Network/HTTP/Media/Language.o )
[ 7 of 17] Compiling Network.HTTP.Media.MediaType.Internal ( src/Network/HTTP/Media/MediaType/Internal.hs, dist/build/Network/HTTP/Media/MediaType/Internal.o )
[ 8 of 17] Compiling Network.HTTP.Media.MediaType ( src/Network/HTTP/Media/MediaType.hs, dist/build/Network/HTTP/Media/MediaType.o )
[ 9 of 17] Compiling Network.HTTP.Media ( src/Network/HTTP/Media.hs, dist/build/Network/HTTP/Media.o )
[10 of 17] Compiling Network.HTTP.Media.Gen ( test/Network/HTTP/Media/Gen.hs, dist/build/Network/HTTP/Media/Gen.o )
[11 of 17] Compiling Network.HTTP.Media.Accept.Tests ( test/Network/HTTP/Media/Accept/Tests.hs, dist/build/Network/HTTP/Media/Accept/Tests.o )
[12 of 17] Compiling Network.HTTP.Media.Language.Gen ( test/Network/HTTP/Media/Language/Gen.hs, dist/build/Network/HTTP/Media/Language/Gen.o )
[13 of 17] Compiling Network.HTTP.Media.Language.Tests ( test/Network/HTTP/Media/Language/Tests.hs, dist/build/Network/HTTP/Media/Language/Tests.o )
[14 of 17] Compiling Network.HTTP.Media.MediaType.Gen ( test/Network/HTTP/Media/MediaType/Gen.hs, dist/build/Network/HTTP/Media/MediaType/Gen.o )
[15 of 17] Compiling Network.HTTP.Media.MediaType.Tests ( test/Network/HTTP/Media/MediaType/Tests.hs, dist/build/Network/HTTP/Media/MediaType/Tests.o )
[16 of 17] Compiling Network.HTTP.Media.Tests ( test/Network/HTTP/Media/Tests.hs, dist/build/Network/HTTP/Media/Tests.o )
[17 of 17] Compiling Tests            ( test/Tests.hs, dist/build/Tests.o )
In-place registering test-http-media-0.5.1...

dist/build/test-http-mediaStub/test-http-mediaStub-tmp/test-http-mediaStub.hs:2:8:
    Could not find module ‘Distribution.Simple.Test.LibV09’
    Perhaps you meant Distribution.Simple.Test (from Cabal-1.18.1.5)
    Use -v to see a list of the files searched for.

Please include Changelog.

Especially, what are the changes between 0.6 and 0.7?

Difficult to introduce fallbacks when parseQuality fails

In #1, I suggested that some sort of a -> Float -> Quality a function was unnecessary, but now that I'm using this library more, I think that was a poor idea, mainly for this reason:

fromMaybe (fromJust $ parseQuality "*/*") $
    parseQuality =<< lookup "Accept" (Wai.requestHeaders req)

Or in English, when there is no Accept header (or it's invalid), then I want to behave as if the client will accept anything. The code above is a workaround, but while I know the fromJust will work, I've developed a special enmity towards this particular partial function. It would be much better to replace (fromJust $ ...) with something like ("*/*" `q` 1) (though probably q isn't the best name... but I'll let you sort that out).

And, to answer the question from #1 about a fractional vs. integral argument, I think the fractional is the clearest. After all, if it's written 0.8 in the header, it may as well be written 0.8 in Haskell.

No API to create `Quality a` values

I was hoping to use your library to handle negotiation on Accept-Language headers, but once I got to parsing those headers, I realized that there's no way to attach a Quality value to my data type. Exposing Network.HTTP.Media.Quality would be plenty sufficient, as well as save me from re-writing readQ.

While I'm at it, are you interested in a Langauge type and parsers to integrate into this library, or would that be feature creep?

Structured Syntax Suffix support?

Are there any plans to support the Structured Syntax Suffix RFC: http://trac.tools.ietf.org/html/draft-ietf-appsawg-media-type-suffix-regs-02

Specifically I would want to be able to use the +json suffix as defined in http://trac.tools.ietf.org/html/draft-ietf-appsawg-media-type-suffix-regs-02#section-3.1 with a wildcard like the following:

application/*+json

It would also be nice to be able to even be able to specify:

*/*+json

Matching should be case insensitive

application/json should match APPLICATION/JSON, but it doesn't.

λ> matchAccept ["application"//"json"] "APPLICATION/JSON"
Nothing
λ> matchAccept ["application"//"json"] "application/json"
Just application/json

Parse error on empty param

> parseAccept "application/json;q" :: Maybe MediaType 
Just *** Exception: Data.ByteString.tail: empty ByteString

parseAccept should return Nothing in this situation