steventen/grape-vs-martini

Compare Ruby's Grape to Golang's Martini for Web API Server

grape-vs-martini

API server example to compare Ruby's Grape Web Framwork to Martini Web Framwork in Go

See blog post for details

Respond to the following routes:

GET /projects(.json)
GET /project/:id(.json)

Response format would be like:

{"status": "Success", "data": [...]}
{"status": "Fail", "error_message": "Bad api key"}

An example curl command would be like:

curl "http://127.0.0.1:8080/projects?key=61c2339c1bc92bc48120b55513cd568b"

For Grape, rack app uses port 9292, rails app uses port 3000 as default.

For Martini, port 8080 is used.

How to run the server

Grape Example

Under grape-example, there are two folders, in which Grape API is mounted on Rack or Rails, using puma

On Rack

1. Modify database settings in config.ru

2. Run bundle

3. Run bundle exec rackup -o 127.0.0.1

On Rails

1. Modify database settings in config/database.yml

2. Run bundle

3. Run rails s puma -b 127.0.0.1

Martini Example

Under martini-example folder

1. Modify database settings in server.go file, change variables in sql.Open()

2. Run go run server.go entry.go

How to create sample data

Example uses MySQL as default database.

Inside mysql, run:

mysql> create database grape_vs_martini_api;

Dump sample data into database:

mysql -u root -p grape_vs_martini_api < grape_vs_martini_api.sql

Sample data includes 10 companies, each with 50 projects, so there are 500 projects in total.

Sample companies' api keys:

61c2339c1bc92bc48120b55513cd568b
bb27bc6b7330aaac63ac809df83311b8
e35402fa00728faf372abaf71623b7f4
bca4bf9d131ab311760c5b790c8568a5
49886a1c018496abb9ba4bf592c08d36
850eedd94a4d9a01962628faac7ace91
d036682c6ac91a41ac9eac064c02a43b
77a767c39adc71fb241285309ff37ee5
75e82d812630bf6d9371339f3e634801
b5125eeabadcd5b3e55a76694d2b62b8

Benchmark

A simple ab test is made, with command ab -c 10 -n 1000 http://127.0.0.1:8080/projects?key=61c2339c1bc92bc48120b55513cd568b

Test environment: Macbook Air CPU 1.7GHz Core i5, 8GB DDR3, OSX 10.9.1

ruby -v 2.0.0p247, go version go1.2 darwin/amd64, rails 3.2.16

Grape

On Rack

Concurrency Level:      10
Time taken for tests:   16.277 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2303000 bytes
HTML transferred:       2211000 bytes
Requests per second:    61.44 [#/sec] (mean)
Time per request:       162.769 [ms] (mean)
Time per request:       16.277 [ms] (mean, across all concurrent requests)
Transfer rate:          138.17 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    23  162  33.9    172     289
Waiting:       17  156  33.2    169     278
Total:         23  162  33.9    173     290

On Rails

Concurrency Level:      10
Time taken for tests:   15.902 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2492000 bytes
HTML transferred:       2211000 bytes
Requests per second:    62.88 [#/sec] (mean)
Time per request:       159.024 [ms] (mean)
Time per request:       15.902 [ms] (mean, across all concurrent requests)
Transfer rate:          153.03 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    15  158  23.5    170     186
Waiting:       15  158  23.5    170     185
Total:         16  158  23.5    171     186

Martini

Concurrency Level:      10
Time taken for tests:   0.900 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2314000 bytes
HTML transferred:       2211000 bytes
Requests per second:    1110.80 [#/sec] (mean)
Time per request:       9.003 [ms] (mean)
Time per request:       0.900 [ms] (mean, across all concurrent requests)
Transfer rate:          2510.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       3
Processing:     3    9   2.9      8      22
Waiting:        3    8   2.9      8      22
Total:          3    9   2.9      8      23

grape-vs-martini's Issues

Did you run Grape in Production

Can you please rerun grape in production !? It should make a large difference

Rack and Rails Grape numbers

I read your analysis, which is very nice, but I found it odd that the Rails performance numbers were so high relative to standalone Rack. So I ran the benchmarks myself and I think the Rails numbers you list are incorrect.

Here's my Rack run:

ab -c 10 -n 1000 http://localhost:9292/projects\?key\=61c2339c1bc92bc48120b55513cd568b
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:
Server Hostname:        localhost
Server Port:            9292

Document Path:          /projects?key=61c2339c1bc92bc48120b55513cd568b
Document Length:        2211 bytes

Concurrency Level:      10
Time taken for tests:   14.675 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2303000 bytes
HTML transferred:       2211000 bytes
Requests per second:    68.14 [#/sec] (mean)
Time per request:       146.754 [ms] (mean)
Time per request:       14.675 [ms] (mean, across all concurrent requests)
Transfer rate:          153.25 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    46  141 196.1    131    5051
Waiting:       39  134 195.8    121    5042
Total:         47  141 196.1    131    5051

Percentage of the requests served within a certain time (ms)
  50%    131
  66%    148
  75%    153
  80%    157
  90%    168
  95%    177
  98%    192
  99%    279
 100%   5051 (longest request)

Here's my first Rails run, which indeed shows very good performance:

ab -c 10 -n 1000 http://localhost:9292/projects\?key\=61c2339c1bc92bc48120b55513cd568b  
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:
Server Hostname:        localhost
Server Port:            9292

Document Path:          /projects?key=61c2339c1bc92bc48120b55513cd568b
Document Length:        9 bytes

Concurrency Level:      10
Time taken for tests:   1.886 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      199000 bytes
HTML transferred:       9000 bytes
Requests per second:    530.35 [#/sec] (mean)
Time per request:       18.855 [ms] (mean)
Time per request:       1.886 [ms] (mean, across all concurrent requests)
Transfer rate:          103.07 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       6
Processing:     9   19  42.1     12     437
Waiting:        9   18  42.1     11     437
Total:          9   19  42.1     12     437

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     13
  75%     13
  80%     15
  90%     29
  95%     31
  98%     40
  99%    426
 100%    437 (longest request)

However, I think these are all 404s. There is a setting in Grape to prefix your API URLs, and it's set to 'api', so here's another run with that prefix in place:

ab -c 10 -n 1000 http://localhost:9292/api/projects\?key\=61c2339c1bc92bc48120b55513cd568b
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:
Server Hostname:        localhost
Server Port:            9292

Document Path:          /api/projects?key=61c2339c1bc92bc48120b55513cd568b
Document Length:        2211 bytes

Concurrency Level:      10
Time taken for tests:   17.681 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2492000 bytes
HTML transferred:       2211000 bytes
Requests per second:    56.56 [#/sec] (mean)
Time per request:       176.812 [ms] (mean)
Time per request:       17.681 [ms] (mean, across all concurrent requests)
Transfer rate:          137.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:    41  176  12.4    172     217
Waiting:       41  176  12.4    172     217
Total:         41  176  12.3    172     218

Percentage of the requests served within a certain time (ms)
  50%    172
  66%    183
  75%    185
  80%    186
  90%    189
  95%    192
  98%    197
  99%    204
 100%    218 (longest request)

The numbers are a little more down-to-earth here, so I'm guessing you encountered the same problem in your figures.

This doesn't really affect your overall findings, just thought I'd point it out as I found it confusing based on my Rack/Rails experience.