Running on a public cloud ?

[the-benchmarker]

Running on a public cloud ?

[tbrand]

That's good idea but there are some issues for it.

• This project includes a Objective-C project that can be compiled only in Mac(?).
• It costs some. :money: (According to previous discussion, we decided not to run this on CI. #27)

As a founder of this project, I want(have) to take all results of this benchmark.
But you can fork this and choice some language as targets.

Following tips might help you.

# Choice ruby, crystal and go as targets
make benchmarker ruby crystal go

# Running benchmark
bin/benchmarker ruby crystal go

# If you want to record the results onto README.md
bin/benchmarker ruby crystal go --record

Thanks!

[waghanza]

Hi @tbrand,

You underline multiples arguments here.

• Some codes are compiled on Mac OS X, since Objective C
  • Travis could compile that ;-)
• It cost some :money:
  • Perhaps a good solution could be to create a blog to share the results as a community (I'd happy to support cost, hosting platform ... and some hosting services COULD be free for OSS)

[waghanza]

@tbrand @OvermindDL1 what do you think ?

@tbrand Objective-C could compile on docker => https://store.docker.com/community/images/dexec/objc
it is to take count on #123

[OvermindDL1]

I'm NOT a fan of performing benchmarks on a cloud/shared anything as then you will get inconsistent results (well you would get inconsistent results on a mac desktop too). An 'empty' dedicated server is what should be used (see my other issue with some results from that among other tests, which really should be built in to this too for statistical testing). #101

[waghanza]

@OvermindDL1 I understand you concern, I'm not a fan either. For me running, running on shared environments COULD break result consistency.

However, cloud IS NOT shared. Of course, technically an hypervisor is used

• XEN
• KVM
• Hyper V
• ...

However, running on cloud mean isolation (all instances has their own CPU / memory), at least cloud providers I have tested

• AWS
• DigitalOcean

[OvermindDL1]

I know cloud is not shared, but cloud does not always imply a 'set' instance of cpu/memory. Those can be migrated in surprising ways and though you may keep a speed/amount of cpu and memory there can be sudden and surprising latencies that will not occur on a dedicated system due to switching behind the scenes.

[waghanza]

I understand. However :

• a majority of (enterprise) application adopt cloud
  • why running elsewhere ?
  • is our goal to be closest to production-ready application usage ?
• bar-metal / full isolation are expensive
  • could bar-metal automatizable ?

[OvermindDL1]

is our goal to be closest to production-ready application usage ?

This is expressely not the goal of this repository. ;-)

We want to know the response time (routing time), not a usability.

And the latency randomness and other issues that cloud would induce would just induce random noise into the benchmarks, which is NOT what you want in a benchmark. Just because in some situations in real-life may induce noise, you really really do not want those in a benchmark. (Which is also why tbrand really needs to stop running the benchmarks on a noisy desktop... >.>)

bar-metal / full isolation are expensive

Not even remotely? I have a 16 core with 16gigs ram dedicated server that I test on here, see: #101

[waghanza]

This is expressely not the goal of this repository. ;-)
So what is the main goal ?

@OvermindDL1 I do not understand your last point

[OvermindDL1]

So what is the main goal ?

Pure speed testing with as few other variables as possible.

@OvermindDL1 I do not understand your last point

I.E. I can perform dedicated server tests with ease. It's a build server I own and it's trivial to suspend the CI for a period as I run tests, as I did in that post. In the post I show the statistical output of the tests and if run on a cloud or shared system then these would vary far more wildly that a statistical system could show but would otherwise corrupt the output of the current setup (as tbrand is currently doing by running it on a desktop, why do you think his results wildly differ from mine).

[waghanza]

Pure speed testing with as few other variables as possible.
yes, but it will be accurate to use a particular usage, production usage.
actually, django run on gunicorn (no built-in server) but flask use buit-in one.
none of enterprise project, using flask, run on built-in

@OvermindDL1 I understand that you have different result than @tbrand, not the same hardware ... so why not running on a standard (and usable by others) hosting, aka cloud ?

[OvermindDL1]

Because free server, and a lot more powerful than what most cloud things offer especially taking into account the prices... ^.^;

/me is still amazed at how slow crystal was on the tests here, wonder if related to them not running in production mode, should test again (ping me on that one issue if anyone wants the statistical tests to be re-run!)...

[waghanza]

free ? it is amazing, it don't known about free hardware

anyway, my personal purpose IS to test raw-performance in each frameworks, but in the closest way as enterprise does use those frameworks

[OvermindDL1]

free ? it is amazing, it don't known about free hardware

It's quite common! Once it's paid for you get free unlimited use of it! Excepting cost of electricity (and maybe internet if it's not shared out to something else, which mine is). ;-)

IS to test raw-performance in each frameworks,

Ah so you definitely don't want weird latencies popping up!

[waghanza]

It's quite common! Once it's paid for you get free unlimited use of it! Excepting cost of electricity (and maybe internet if it's not shared out to something else, which mine is). ;-)

Sure, it not free but the major cost part is initial ;-) but still not the same logic as the majoirty of businesses

Ah so you definitely don't want weird latencies popping up!

I don't understand

[OvermindDL1]

I don't understand

If you are wanting to test raw throughput without uncontrollable internet and system conditions interfering, then you want as stable a test-bed as possible. :-)

[waghanza]

ah, ok
you say that the internet connection stability could case result inconsistency

[OvermindDL1]

you say that the internet connection stability could case result inconsistency

That is one possible condition, hence why you'd prefer to test between 2 machines on a dedicated local wired network (which I also have, though this repo's built-for single server right now, which is not entirely reliable either since you should never test from the same system that the server is running on), with no other processes running on them, dedicated hardware with no chance of migration or anything else of the sort, etc...

Running it under any other conditions (like on a desktop machine) introduces in changes that be be quite significant (like the latest update that puts Phoenix faster than Plug, which would never ever happen as Phoenix is just 'more' plugs on top of Plug).

[waghanza]

interesting point of view
you must do a lot of automation, to provide a clean and isolated environment to run each framework, thus

even in cloud we can run on a dedicated connection (or as closest as a local one)

[OvermindDL1]

you must do a lot of automation

Ungodly huge amounts. CI build systems for everything from C++ to Elixir and more (Rust recently growing). Server deployment automation. Etc... I have no clue how I've even reached this point but even just me personal non-work stuff is massive... >.>

[waghanza]

I see that it is a point to run on cloud, though, at least to decrease your time automating everything

[OvermindDL1]

I see that it is a point to run on cloud, though, at least to decrease your time automating everything

Except that would cause me 'more' work, as well as substantially more cost. ^.^;