aristidb/aws

Amazon Web Services for Haskell

Introduction

The aws package attempts to provide support for using Amazon Web Services like S3 (storage), SQS (queuing) and others to Haskell programmers. The ultimate goal is to support all Amazon Web Services.

Installation

Make sure you have a recent GHC installed, as well as cabal-install, and installation should be as easy as:

$ cabal install aws

If you prefer to install from source yourself, you should first get a clone of the aws repository, and install it from inside the source directory:

$ git clone https://github.com/aristidb/aws.git
$ cd aws
$ cabal install

Using aws

Concepts and organisation

The aws package is organised into the general Aws module namespace, and subnamespaces like Aws.S3 for each Amazon Web Service. Under each service namespace in turn, there are general support modules and and Aws.<Service>.Commands.<Command> module for each command. For easier usage, there are the "bundling" modules Aws (general support), and Aws.<Service>.

The primary concept in aws is the Transaction, which corresponds to a single HTTP request to the Amazon Web Services. A transaction consists of a request and a response, which are associated together via the Transaction typeclass. Requests and responses are simple Haskell records, but for some requests there are convenience functions to fill in default values for many parameters.

Example usage

To be able to access AWS resources, you should put your into a configuration file. (You don't have to store it in a file, but that's how we do it in this example.) Save the following in $HOME/.aws-keys.

default AccessKeyID SecretKey

You do have to replace AccessKeyID and SecretKey with the Access Key ID and the Secret Key respectively, of course.

Then, copy this example into a Haskell file, and run it with runghc (after installing aws):

{-# LANGUAGE OverloadedStrings #-}

import qualified Aws
import qualified Aws.S3 as S3
import           Control.Monad.Trans.Resource
import           Data.Conduit ((.|), runConduit)
import           Data.Conduit.Binary (sinkFile)
import           Network.HTTP.Conduit (newManager, tlsManagerSettings, responseBody)

main :: IO ()
main = do
  {- Set up AWS credentials and the default configuration. -}
  cfg <- Aws.baseConfiguration
  let s3cfg = Aws.defServiceConfig :: S3.S3Configuration Aws.NormalQuery

  {- Set up a ResourceT region with an available HTTP manager. -}
  mgr <- newManager tlsManagerSettings
  runResourceT $ do
    {- Create a request object with S3.getObject and run the request with pureAws. -}
    S3.GetObjectResponse { S3.gorResponse = rsp } <-
      Aws.pureAws cfg s3cfg mgr $
        S3.getObject "haskell-aws" "cloud-remote.pdf"

    {- Save the response to a file. -}
    runConduit $ responseBody rsp .| sinkFile "cloud-remote.pdf"

You can also find this example in the source distribution in the Examples/ folder.

Frequently Asked Questions

S3 questions

• I get an error when I try to access my bucket with upper-case characters / a very long name.

  Those names are not compliant with DNS. You need to use path-style requests, by setting s3RequestStyle in the configuration to PathStyle. Note that such bucket names are only allowed in the US standard region, so your endpoint needs to be US standard.

Release Notes

See CHANGELOG

Resources

• aws on Github
• aws on Hackage (includes reference documentation)
• Official Amazon Web Services website

Contributors

Name	Github	E-Mail	Company	Components
Abhinav Gupta	abhinav	[email protected]	-	IAM, SES
Aristid Breitkreuz	aristidb	[email protected]	-	Co-Maintainer
Bas van Dijk	basvandijk	[email protected]	Erudify AG	S3
David Vollbracht	qxjit
Felipe Lessa	meteficha	[email protected]	currently secret	Core, S3, SES
Nathan Howell	NathanHowell	[email protected]	Alpha Heavy Industries	S3
Ozgun Ataman	ozataman	[email protected]	Soostone Inc	Core, S3, DynamoDb
Steve Severance	sseveran	[email protected]	Alpha Heavy Industries	S3, SQS
John Wiegley	jwiegley	[email protected]	FP Complete	S3
Chris Dornan	cdornan	[email protected]	Iris Connect	Core
John Lenz	wuzzeb			DynamoDB, Core
Joey Hess	joeyh	[email protected]	-	Co-Maintainer, S3

aws's Issues

SES is broken in last release (0.7.6.3)

Just started running into an issue after upgrading to latest with xml-conduit-1.1.0.3:

Name {nameLocalName = "SendRawEmailResponse", nameNamespace = Just "http://ses.amazonaws.com/doc/2010-12-01/", namePrefix = Nothing}, but received: EventEndDocument.

This functionality was working without any issues right before the update.

Check SSL certificates

We need certificate checking for SSL requests.

Add an option to disable TLS certificate checks

Uploading an object to S3 fails about half the time (with TlsException (HandshakeFailed (Error_Protocol ("certificate has unknown CA",True,UnknownCa)))). I suspect a bug in the haskell tls package. It wouldn't be the first one that I came across.

Create and Terminate EC2 instances

As far as EC2 api support is concerned, does this package only retrieve metadata for running EC2 instances? I'd like to be able to create, start, stop and terminate EC2 instances, and add tags to them. Wanted to be sure these things weren't supported before I attempt to submit a patch.

DynamoDB Item API support

It looks like the current DynamoDB code only supports the parts of the API dealing with the data definition (creating, updating, deleting tables) but not data retrieval and modification (get, put, update, query, and scan items).

This is something I'm willing to add, but wanted to check first whether anyone else is already working on it.

(pinging @wuzzeb)

Conduit dependency out of date

With the release of conduit 0.3., it looks like the conduit dependency is out of date and requires more than a trivial version bump.

Any reason why there isn't a SNS module?

I can see we have a SNS submodule to handle SES specific events, but not a general one. Is there any blocker for that or just it haven't been thought yet?

I might suspect SNS by nature is fairly domain-specific and might be unfeasible to come up with a general module, is this the reason?

Thanks!
Alfredo

Examples

Some examples of how to use the package would be very useful.

S3 PutObject should take a strict bytestring and provide a content length to http conduit

I fused a s3 source to s3 and it worked, but as soon as I did some processing per line, the socket would vanish. S3 also should be given a content length, which a lazy source can't provide.

For these reasons, I believe put object should only take a strict byte string, and encode the content length from it.

delete S3 bucket

The API seems to be missing a way to delete a S3 bucket (once it's already been emptied of all objects).

Once that API is available, a good example program could be added, that did a mass-removal of all the contents of a bucket and then deleted it. Particularly handy when you've been learning the API, and so creating various junk in a bucket. ;)

Use "Security Token Service" for access (request side)

Allow making requests using STS (http://docs.amazonwebservices.com/IAM/latest/UsingSTS/Welcome.html / http://docs.amazonwebservices.com/STS/latest/APIReference/Welcome.html).

This issue is NOT about full support for STS, just making requests using these temporary credentials. There are two caveats:

1. These tokens expire, so there needs to be a way to specify refresh methods - before the token has expired.
2. There needs to be an additional request parameter it seems, the "SessionToken" which needs to be passed in addition to the supplied temporary access key ID and secret access key.

This is especially interesting in conjunction with #29 and #37, so temporary credentials can be periodically obtained with the EC2 instance metadata.

Add support for multipart uploads

You can break a 5 GB upload (the current limit on the size of an S3 object) into as many as 1024 separate parts and upload each one independently, as long as each part has a size of 5 megabytes (MB) or more. If an upload of a part fails it can be restarted without affecting any of the other parts. Once you have uploaded all of the parts you ask S3 to assemble the full object with another call to S3.

This would greatly improve upload times for large objects.

Build of 0.4.* fails on OS X (dependency hell)

Building of 0.4.* fails on OS X with HP 2011.4.0.0. 0.3.2 builds just fine. The build error is:

cabal: cannot configure void-0.5.5.1. It requires semigroups >=0.8.2 && <0.9
For the dependency on semigroups >=0.8.2 && <0.9 there are these packages:
semigroups-0.8.2, semigroups-0.8.3, semigroups-0.8.3.1 and semigroups-0.8.3.2.
However none of them are available.
semigroups-0.8.2 was excluded because semigroups-0.8 was selected instead
semigroups-0.8.3 was excluded because semigroups-0.8 was selected instead
semigroups-0.8.3.1 was excluded because semigroups-0.8 was selected instead
semigroups-0.8.3.2 was excluded because semigroups-0.8 was selected instead

Can't seem to find the issues after creation for this repo

When I make an issue I can't seem to find it anymore. Are issues being filtered upon creation?
Doesn't show up unless I manually go to the URL of the issue. #91

aeson-0.8.0.0

Is it possible to make it work with aeson-0.8?

Add PutObjectCopy command

I would like to have support for the PUT Object (Copy) command.

I can write the patch if you want but I don't know when I have time for it.

Build Failure: Compiling Aws.Core fails with "No instance for (Data.Default.Default HTTP.Request) arising from a use of `def'"

Here's the output:

$ cabal install aws
...
...
...
Building aws-0.9...
Preprocessing library aws-0.9...
[ 1 of 64] Compiling Aws.Ec2.InstanceMetadata ( Aws/Ec2/InstanceMetadata.hs, dist/build/Aws/Ec2/InstanceMetadata.o )
[ 2 of 64] Compiling Aws.Core         ( Aws/Core.hs, dist/build/Aws/Core.o )

Aws/Core.hs:379:14:
    No instance for (Data.Default.Default HTTP.Request)
      arising from a use of `def'
    Possible fix:
      add an instance declaration for (Data.Default.Default HTTP.Request)
    In the expression: def
    In the second argument of `($)', namely
      `def
         {HTTP.method = httpMethod sqMethod,
          HTTP.secure = case sqProtocol of {
                          HTTP -> False
                          HTTPS -> True },
          HTTP.host = sqHost, HTTP.port = sqPort, HTTP.path = sqPath,
          HTTP.queryString = HTTP.renderQuery False sqQuery,
          HTTP.requestHeaders = catMaybes
                                  [checkDate (\ d -> ("Date", fmtRfc822Time d)) sqDate,
                                   fmap (\ c -> ("Content-Type", c)) contentType,
                                   fmap
                                     (\ md5 -> ("Content-MD5", Base64.encode $ toBytes md5))
                                     sqContentMd5,
                                   ....]
                                ++ sqAmzHeaders ++ sqOtherHeaders,
          HTTP.requestBody = case sqMethod of {
                               PostQuery
                                 -> HTTP.RequestBodyLBS . Blaze.toLazyByteString
                                    $ HTTP.renderQueryBuilder False sqQuery
                               _ -> case sqBody of {
                                      Nothing -> HTTP.RequestBodyBuilder 0 mempty
                                      Just x -> x } },
          HTTP.decompress = HTTP.alwaysDecompress,
          HTTP.checkStatus = \ _ _ _ -> Nothing}'
    In a stmt of a 'do' block:
      return
      $ def
          {HTTP.method = httpMethod sqMethod,
           HTTP.secure = case sqProtocol of {
                           HTTP -> False
                           HTTPS -> True },
           HTTP.host = sqHost, HTTP.port = sqPort, HTTP.path = sqPath,
           HTTP.queryString = HTTP.renderQuery False sqQuery,
           HTTP.requestHeaders = catMaybes
                                   [checkDate (\ d -> ("Date", fmtRfc822Time d)) sqDate,
                                    fmap (\ c -> ("Content-Type", c)) contentType,
                                    fmap
                                      (\ md5 -> ("Content-MD5", Base64.encode $ toBytes md5))
                                      sqContentMd5,
                                    ....]
                                 ++ sqAmzHeaders ++ sqOtherHeaders,
           HTTP.requestBody = case sqMethod of {
                                PostQuery
                                  -> HTTP.RequestBodyLBS . Blaze.toLazyByteString
                                     $ HTTP.renderQueryBuilder False sqQuery
                                _ -> case sqBody of {
                                       Nothing -> HTTP.RequestBodyBuilder 0 mempty
                                       Just x -> x } },
           HTTP.decompress = HTTP.alwaysDecompress,
           HTTP.checkStatus = \ _ _ _ -> Nothing}
Failed to install aws-0.9
cabal: Error: some packages failed to install:
aws-0.9 failed during the building phase. The exception was:
ExitFailure 1

Requests using HTTPS just hang

OS X Lion, conduit 0.4.1.1, http-conduit 1.4.1.2

Will try on other systems, too, and with the stable version of aws.

@meteficha Can you reproduce this?

formatTime and parseTime are slow

Although it is tedious, I would recommend manually rendering the timestamps. formatTime is extremely inefficient. The function is responsible for about 10% of my CPU utilization when benchmarking an application that uploads to S3.

We also have a timestamp parsing library that may be faster than the parseTime function: http://hackage.haskell.org/package/timeparsers. The library is still pretty new and probably needs some work before being used in something like 'aws'.

Debugging text

Every request prints out debug information into standard output - there is no way to disable this. Debug output has the following format:

AWS Debug: String to sign - ...

Support Object Expiration Rules

http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUTlifecycle.html

Would be really nice to have. I can try to take a stab at it myself.

Retry requests on error

This should be configurable, too.

PutObject Error

Unsure if this is s3 or the aws pkg. It says this operation isn't Implemented. Was wondering if it's because I'm using a NormalQuery for my S3Configuration when doing a PUT. Any ideas are appreciated.

uploadToCompletedBucket :: Worker ()
uploadToCompletedBucket = do
  prnt "uploading"
  (acid, cfg, QMsg{..}) <- ask
  let s3cfg = Aws.defServiceConfig :: S3.S3Configuration Aws.NormalQuery
  zipSize <- io $ sz (T.unpack qId ++ ".zip")
  io $ withManager $ \mgr ->
      do S3.PutObjectResponse { porVersionId = result } <-
             Aws.pureAws cfg s3cfg mgr $ S3.putObject stcompleted qId $
             RequestBodySource (fromIntegral zipSize) (source qId)
         io $ print result
  prnt "Put object onto completed bucket"
  where
    source :: MonadResource m => T.Text -> Source m Builder
    source qId = CB.sourceFile (output ++ "/" ++ T.unpack qId <> ".zip") $= CL.map fromByteString
    sz file = fileSize <$> getFileStatus file

And the corresponding error.

 S3Error {s3StatusCode = Status {statusCode = 501,statusMessage = "Not Implemented"}, 
             s3ErrorCode = "NotImplemented", 
             s3ErrorMessage = "A header you provided implies functionality that is not implemented", 
             s3ErrorResource = Nothing, 
             s3ErrorHostId = Just "KF8FwxnEalEL4sgPQhFcoxmu3q2rdOhz1mgG0+S7E/yz1kWvZPX5Hy2S0ILwwufc", 
             s3ErrorAccessKeyId = Nothing, 
             s3ErrorStringToSign = Nothing
             }

AWS Monad Transformer

When I started working on aws a while ago, I wanted to put everything in a monad. This plan was scrapped fairly quickly, however now I have plans to have another go, albeit with different motivations. Two things first:

1. This will be optional: I have no plans at all to make people put their programs in an AWS monad. The plan is that complex functionality that spans multiple REST request yet is logically one AWS "call" uses this, for the most part.
2. This will be based on FreeT (http://hackage.haskell.org/package/free or http://hackage.haskell.org/package/transformers-free) or something similar, in order to allow different "run" functions.

So why do I want this?

Originally, I just wanted to avoid giving the configuration / service configuration / HTTP manager parameters every time. But that isn't really a good reason IMHO.

I have recently started adding the first "complex" functionality to aws, inspired by @larskuhtz, specifically the IteratedTransaction functions awsIteratedSource and awsIteratedList. They just hammer out the HTTP requests and that's that. But what if we want to do things like:

• Rate limiting (#31)
• Retry failed requests (#5)

Or perhaps fancier things that I haven't thought of yet.

I think there might be two components to this:

1. A reader-like part where the configuration / HTTP Manager is stored. This part could be omitted, if somebody thinks it's a bad idea.
2. A FreeT-like part for the actions.

It would also have to be a transformer so there can be actions in some underlying monad.

And of course it would have pluggable interpreters, which can then implement request retry / rate limiting / ....

Compilation failure on GHC 7.4

Aws/DynamoDb/Commands/Table.hs:42:18:
    Could not find module `GHC.Generics'
    It is a member of the hidden package `ghc-prim'.
    Perhaps you need to add `ghc-prim' to the build-depends in your .cabal file.
    Use -v to see a list of the files searched for.

Elastic Transcoder

AWS have announce Elastic Transcoder.

I am expecting to build out an interface to this service fairly soon for a project, and I thought it would be a good idea to check in here to see if you are already doing this or would be interested in extending the AWS package to cover this. (I will of course be very happy to feed the code back into the AWS package.)

Unauthenticated requests

Some requests don't require authentification and no credentials. 'aws' currently has no real support for that.

Use EC2 metadata to get access credentials automatically

See this http://aws.typepad.com/aws/2012/06/iam-roles-for-ec2-instances-simplified-secure-access-to-aws-service-apis-from-ec2.html

That's pretty cool, I want that!

S3 - GetObject Range header

On GetObject, is it possible to support giving the HTTP Range header? That way we can access partial data.

See http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html Range headers for more information

Rate limiting

To prevent scripts from hammering Amazon's servers and getting rate limited server-side, add rate limiting here.

If there is an existing Haskell package for rate limiting, consider using it.

convenience functions (with more than one message)

hi,

i am writing a mws (marketplace web service) extension to aws and noticed, that there don't seem to be any convenience functions that, say, send one specific message, process the result in one way or another and then send another specific message.

one example on where i want to offer such a function is the mws' subscriptions api, so that one only has to call (name and type to change)

setupQueueForMwsSubscription cfg sqsCfg mwsCfg mgr

and it will add the right permissions to the queue and register the queue as a destination in MWS' subscription api.

is something like that not desirable or just not important for the other parts of aws?

if so, should it be in Aws.Mws or in an extra module as Aws.Mws.Convenience?

i'd really like to follow the spirit of aws.

Test system

We need a test system.

CC @meteficha

generating Credentials from 2 ByteStrings

My program already stores AWS credentials in its own way, so I need a way to get from ByteString -> ByteString -> Credentials

It's certainly possible to write that in my program, but then I have to worry about the v4SigningKeys and iamToken fields, which seem like implementation details that could change. And one of them, being an IORef, I'm wary of constructing on my own, since making an IORef that's empty when it should be full can lead to problems, for example.

So, I suggest providing in aws:

genCredentials :: ByteString -> ByteString -> IO Credentials

Using that might also simplify the implementation of loadCredentials*

SimpleDB Post request to PutAttributes with umlauts does not work

*Aws> runAws (aws $ putAttributes "ßß" [addAttribute "ö" "ü"] "MyDomain") bc
AWS Debug: String to sign - "POST\nsdb.amazonaws.com\n/\nAWSAccessKeyId=AKIAIOJF6QL7WAZUCBBA&Action=PutAttributes&Attribute.1.Name=%C3%B6&Attribute.1.Value=%C3%BC&DomainName=MyDomain&ItemName=%C3%9F%C3%9F&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-01-31T14%3A19%3A06&Version=2009-04-15"
*** Exception: SdbError {sdbStatusCode = 403, sdbErrorCode = SignatureDoesNotMatch, sdbErrorMessage = "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.", sdbErrorMetadata = FromSdbMetadata (SdbMetadata {requestId = "4ab63c9c-5d2e-0bfd-f6eb-a544555e63f4", boxUsage = Nothing})}

where bc uses a POST configuration

*Aws> runAws (aws $ putAttributes "ßß" [addAttribute "ö" "ü"] "MyDomain") c
AWS Debug: String to sign - "GET\nsdb.amazonaws.com\n/\nAWSAccessKeyId=AKIAIOJF6QL7WAZUCBBA&Action=PutAttributes&Attribute.1.Name=%C3%B6&Attribute.1.Value=%C3%BC&DomainName=MyDomain&ItemName=%C3%9F%C3%9F&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-01-31T14%3A18%3A57&Version=2009-04-15"
SdbResponse {fromSdbResponse = PutAttributesResponse, sdbResponseMetadata = SdbMetadata {requestId = "80dcad41-8e89-c3f9-db94-09190487ff87", boxUsage = Just "0.0000219909"}}

here with GET configuration

s3SignQuery Issue

If you look at s3SignQuery in Aws/S3/Core.hs

s3SignQuery :: S3Query -> S3Configuration qt -> SignatureData -> SignedQuery            
s3SignQuery S3Query{..} S3Configuration{..} SignatureData{..}                           
    = SignedQuery {                                                                     
        sqMethod = s3QMethod                                                         
      , sqProtocol = s3Protocol                                                         
      , sqHost = B.intercalate "." $ catMaybes host                                     
      , sqPort = s3Port                                                                 
      , sqPath = mconcat $ catMaybes path                                                                                            
      , sqQuery = sortedSubresources ++ s3QQuery ++ authQuery :: HTTP.Query          
      , sqDate = Just signatureTime                                                     
      , sqAuthorization = authorization                                                 
      , sqContentType = s3QContentType                                                  
      , sqContentMd5 = s3QContentMd5                                                    
      , sqAmzHeaders = amzHeaders                                                       
      , sqOtherHeaders = s3QOtherHeaders                                                
      , sqBody = s3QRequestBody                                                         
      , sqStringToSign = stringToSign                                                
      }

You'll see that

sqQuery = sortedSubresources ++ s3QQuery ++ authQuery :: HTTP.Query 

differs from its form in the stringToSign

HTTP.renderQueryBuilder True sortedSubresources

This hasn't been an issue because all of the commands that can set the s3QQuery field
to something other than an empty list do not do so in practice.

Example: in GetObject

  s3QQuery = HTTP.toQuery [                         
         ("response-content-type" :: B8.ByteString,) <$> goResponseContentType
       , ("response-content-language",) <$> goResponseContentLanguage
       , ("response-expires",) <$> goResponseExpires
       , ("response-cache-control",) <$> goResponseCacheControl
       , ("response-content-disposition",) <$> goResponseContentDisposition
       , ("response-content-encoding",) <$> goResponseContentEncoding
       ] 

In practice, people must use getObject

getObject :: Bucket -> T.Text -> GetObject                                           
getObject b o = GetObject b o Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing

I ran into this issue writing DeleteObjects. Putting "delete" in the s3QQuery doesn't work. Making the obvious fix to the stringToSign process broke another command, (GetBucket?), so I just put it in s3QSubresources as a hack.

Test case that should generate a 403 (slightly modified version of the GetObject example):

{-# LANGUAGE OverloadedStrings #-}                                                                                                   

import qualified Aws                                                                                                                 
import qualified Aws.S3 as S3                                                                                                        
import qualified Aws.S3.Commands.GetObject as G                                                                                      
import           Data.Conduit (($$+-))                                                                                               
import           Data.Conduit.Binary (sinkFile)                                                                                      
import           Network.HTTP.Conduit (withManager, responseBody)                                                                    

main :: IO ()                                                                                                                        
main = do                                                                                                                            
  {- Set up AWS credentials and the default configuration. -}                                                                        
  cfg <- Aws.baseConfiguration                                                                                                       
  let s3cfg = Aws.defServiceConfig :: S3.S3Configuration Aws.NormalQuery                                                             

  {- Set up a ResourceT region with an available HTTP manager. -}                                                                    
  withManager $ \mgr -> do                                                                                                           
    {- Create a request object with S3.getObject and run the request with pureAws. -}                                                
    S3.GetObjectResponse { S3.gorResponse = rsp } <-                                                                                 
      Aws.pureAws cfg s3cfg mgr $                                                                                                    
        (S3.getObject "haskell-aws" "cloud-remote.pdf") { G.goResponseCacheControl = Just "300" }                                    

    {- Save the response to a file. -}                                                                                               
    responseBody rsp $$+- sinkFile "cloud-remote.pdf"

Lensify

Use classy lenses for the record fields of requests and responses, that should also make "subtyping" of requests more convenient.

We might also be able to use it for the Response type.

This is a big change, I guess.

Release current aws

#60 and #61 were merged but not yet released.

Thanks!

cannot parse Last-Modified header

Whenever I try to fetch an object from S3, I get the exception from parseObjectMetadata that the Last-Modified timestamp is wrong.

In the web interface, the AWS console reports that the last modified value is "Fri Aug 03 17:41:20 GMT-700 2012". I do not know for sure if this is actually the string it yields when you make the HTTP request. If it is, then it certainly does not conform to the rfc822Time string in the aws package (which is "%a, %_d %b %Y %H:%M:%S GMT").

Currently there is no way to create a HEAD request for objects

This is an oft-cited trick for checking whether a specific object exists, rather than listing the objects in the bucket that match a prefix.

Provide a way to retag S3Configuration values

S3Configuration carries a phantom type of the request type. I am storing my S3Configuration in a bigger Env data type, and it's fine if I always do NormalQuerys. However, I need to do both types of queries, so I end up having to do:

    let s3' = case s3 of
                S3.S3Configuration a b c d e f -> S3.S3Configuration a b c d e f

It would be nice if aws provided this inside the library itself.

Glacier support

It would be nice if aws would support Amazon Glacier. Possibly, this would make it possible for git-annex to support Glacier without external tools.

transformers-0.4.1

Is it possible to make this library work with transformers 0.4.1 ?

S3 - Versioned Objects

I am uncertain how to work with versioned objects using this library. Shall I use the s3QQuery field of S3Query? It seems not possible to do it through the interface provided by GetBucket.

api signing is broken?

We have been getting an HTTP 403 error back from AWS because the API signature does not match. We have discovered that the problem happens on ghc-7.6.1 but not on ghc-7.0.4, and that it happens for getObject when any of the GetObject Maybe fields (goVersionId, goResponseContentType, goResponseContentLanguage, goResponseExpires, goResponseCacheControl, goResponseContentDisposition, goResponseContentEncoding) are set to Just.

I'm not sure, but it seems like s3SignQuery is not taking into account all the parameters that the signing should be based on. I have no idea why this is working on ghc-7.0.4 and not ghc-7.6.1.

Some S3 PutObject headers don't work

Hey!

There are some headers that I'm setting on PutObject that simply don't work: poContentType, poCacheControl and poExpires. If I'm reading the code correctly, that's becuase these headers are actually being sent in the query, not as headers. So AWS simply ignore them. Is that correct?

Cheers!

SignatureDoesNotMatch

Hello,

i get this error in response back on S3 Put Object command.

"The request signature we calculated does not match the signature you provided. Check your key and signing method".

I checked that credentials are valid.

Add DevPay support

It would be great to have DevPay support in this library. It changes a bit REST requests adding product key and user token. Have you considered adding it i near future?

EC2 Instance metadata

EC2 instances have access to a rich source of meta-data

http://169.254.169.254//

It contains directories (names ending with "/") and files. No authentication is required, but access is local to the machine. Some EC2 images lock it down, so it isn't always accessible.

To do it properly, probably depends on #36.

No instance for (exceptions-0.5:Control.Monad.Catch.MonadThrow (ResourceT IO))

I got an error while compiling your library. It does not work with exceptions >= 0.4. Please update library to use newest exceptions.

Preprocessing library aws-0.8.5...
[ 1 of 64] Compiling Aws.Ec2.InstanceMetadata ( Aws/Ec2/InstanceMetadata.hs, dist/dist-sandbox-acf35a42/build/Aws/Ec2/InstanceMetadata.o )

Aws/Ec2/InstanceMetadata.hs:20:41:
    No instance for (exceptions-0.5:Control.Monad.Catch.MonadThrow
                       (ResourceT IO))
      arising from a use of `HTTP.parseUrl'
    Possible fix:
      add an instance declaration for
      (exceptions-0.5:Control.Monad.Catch.MonadThrow (ResourceT IO))
In a stmt of a 'do' block:
      req <- HTTP.parseUrl ("http://169.254.169.254/" ++ p ++ '/' : x)
    In the expression:
      do { req <- HTTP.parseUrl
                    ("http://169.254.169.254/" ++ p ++ '/' : x);
           HTTP.responseBody <$> HTTP.httpLbs req mgr }
    In an equation for `getInstanceMetadata':
        getInstanceMetadata mgr p x
          = do { req <- HTTP.parseUrl
                          ("http://169.254.169.254/" ++ p ++ '/' : x);
                 HTTP.responseBody <$> HTTP.httpLbs req mgr }
Failed to install aws-0.8.5

aws-sign4, aws-elastic-transcoder & cloudfront-signer released

Folks, today I realised that although I have been using my elastic-transcoder aws-extensions in a production context, I hadn't done anything with the code since mid-March, when it was pretty-much ready to go.

So today I re-spun the code as two extension packages:

• aws-sign4 for authenticating an AWS request with the Signature V4 protocol in the aws framework and
• aws-elastic-transcoder, which uses aws-sign4 to build an elastic transcoder interface.

AWS have since built out the elastic transcoder api but I thought it was important to just get it out. I was very pleasantly pleased by how easy it was to just make up these two packages which just hook into the current aws -- a huge validation of its design IMHO.

As this arrangement seems to be so natural and productive I am inclined to stick with it. I consider it a part of the aws.

Have you considered moving a more module package structure? The design seems clearly to support it.

I have also published cloudfront-signer for signing CloudFront URLs. This is a fairly simple (if dead useful) standalone component without any dependencies on the aws package. It does live in module that fits with the aws scheme -- Aws.CloudFront.Signer -- and could be consider part of the aws family.

The current packages have been put out there on version 0.0.0.1 to get things moving. I expect to be revising them fairly quickly in light of feedback. They are all hosted on my personal github account (cdornan) but I expect to be re-housing them on the iconnect account soon.