Multiple stacks #35

motin · 2015-12-09T17:56:17Z

Fixes #33 and includes fix for #34
Simplifies the fix for #8

Update: This PR also includes more robust results parsing, a php-based framework list, as well as fixes for #36 and #38 - all required to properly support multiple stacks.

…ents and easily benchmark against php 5.6, hhvm and php 7

…ut data

…ross the stacks

kenjis · 2015-12-09T21:12:07Z

cake-3.1/src/Controller/HelloController.php

@@ -8,7 +8,7 @@ class HelloController extends AppController

    public function index()
    {
-        $this->response->body('Hello World!');
+        $this->response->body('Hello World!' . require dirname(__FILE__).'/../../../libs/output_data.php');


I can't accept this change. It exclude shutdown process of the framework.

motin · 2015-12-10T08:13:01Z

And why do you have to remove $_SERVER['DOCUMENT_ROOT']? I hope the code exactly the same for all frameworks.

The docker stacks are not served in the sub-directory php-framework-benchmark (since it does not make sense to do so when running fully dedicated stacks for the benchmarks), thus relying on the document root + a sub-directory is not working in both cases. Dirname + __FILE__ always works, but I can refactor to use an environment variable instead.

Regarding putting require libs/output_data.php in the actual controller method. I can't accept this change. It exclude shutdown process of the framework. And why do you need to echo?

Many frameworks required the response to be returned in the actual controller method and not echoed after the framework has been shutdown. To be able to return the value without using output buffering, I changed the printf to sprintf. This also makes it ready to benchmark event-driven frameworks such as http://reactphp.org/

I traced this back to #38
In the process, I also came across #36

…sues as described in kenjis#38)

motin · 2015-12-10T09:35:36Z

The latest commits also fixes #36 and #38

…share the same colors per framework)

kenjis · 2015-12-10T10:56:05Z

The docker stacks are not served in the sub-directory php-framework-benchmark (since it does not make sense to do so when running fully dedicated stacks for the benchmarks)

The docker stacks should have the same sub-directory. So you don't have to change the code only for the docker stacks.

kenjis · 2015-12-10T11:00:10Z

list.php

+                'phalcon-1.3', // Not compiled for PHP7 - is it even supported?
+                'phalcon-2.0', // Not compiled for PHP7 - is it even supported?
+                'ice-1.0', // Not compiled for PHP7 - is it even supported?
+                'bear-1.0', // apc_fetch()-error on PHP7


BEAR.Sunday should work fine on php7.

…ramework-benchmark sub-directory

motin · 2015-12-10T11:48:11Z

The docker stacks should have the same sub-directory. So you don't have to change the code only for the docker stacks.

Fixed and reversed the changed code.

…frameworks to affect the relative results

motin · 2015-12-10T13:10:25Z

Btw, interesting results already especially regarding HHVM: facebook/hhvm#6645

kenjis · 2015-12-10T22:34:55Z

README.md

+By sharing underlying software stacks, the benchmark results vary only according to the host machine's hardware specs and differing code implementations.
+
+See [docker/README.md](docker/README.md)
+


Please remove this. I don't need it. I don't maintain the results of Docker.

This is not primarily for you :) It is for everyone that easily wants to run their own benchmarks. Instead of having to set-up several server environments, compile php extensions and configure their own server software to run all the different framework, they can instead simply install docker toolbox and be running within minutes.

I will remove the change from this PR in hope that you will consider this approach in the future: #41

Benchmark results in Docker do not matter unless you or someone deploy using it.

Is your production environment Docker on Mac OS X? If not, the benchmark results might mislead you.

My production environment is Docker in AWS instances and it is trivial to run the benchmarks as well when dealing with Docker stacks. Let's discuss in #41

kenjis · 2015-12-10T23:43:59Z

Why don't you move docker-results.sh and docker-urls.sh to docker/bin/?

kenjis · 2015-12-10T23:44:52Z

ab command is missing.

$ docker-compose run shell
root@889a4a34285c:/repo# sh setup.sh
ab command not found.

motin · 2015-12-11T07:01:06Z

Why don't you move docker-results.sh and docker-urls.sh to docker/bin/?

Sure, now when there are three scripts it makes more sense. When there was only a single script, it felt odd to create a new bin-folder instead of re-using the existing one.

motin · 2015-12-11T07:03:52Z

ab command is missing.

Weird, it is installed at https://github.com/motin/php-framework-benchmark/blob/e720dd142933fed0ffe3247bce6874fa5d900668/docker/stack/php/shell/Dockerfile#L3

I'll add docker-compose build shell to the readme so that there is no chance of it not being included.

kenjis · 2015-12-11T07:13:32Z

Weird, it is installed at https://github.com/motin/php-framework-benchmark/blob/e720dd142933fed0ffe3247bce6874fa5d900668/docker/stack/php/shell/Dockerfile#L3

You're right. I don't know why. I did according to your README.
I'll try it again later.

…dencies are being installed

motin added 10 commits December 9, 2015 16:26

Added initial docker stacks to be able to share benchmarking environm…

88ce5eb

…ents and easily benchmark against php 5.6, hhvm and php 7

Storing results separately for each stack

f069974

Framework changes necessary for all listed frameworks to run and outp…

93028ee

…ut data

Ability to check results stack by stack

a988715

Include error log on results page

8d10688

Do not bail if error log happens to be empty after a run

72fece1

Results for the frameworks that actually run in the various stacks

be9c180

Re-enabled "no-framework" so that relative results can be compared ac…

df036ff

…ross the stacks

Synced $output_dir usage

79c2e76

Graphs across stacks are now comparable

10178e5

kenjis reviewed Dec 9, 2015
View reviewed changes

motin added 6 commits December 10, 2015 10:23

Removed comment proven to be wrong + non-altering code

0b93772

Reversed output_data require-refactoring

8316f27

Requiring libs/output_data based on env var

51c2493

A single framework list to maintain

9907bbf

Using headers to output benchmark data (solved Content-Length curl is…

eb5ecbf

…sues as described in kenjis#38)

Only calculate output data if specifically requested (fixes kenjis#36)

4a64560

motin force-pushed the multiple-stacks branch from 3ae0eea to 4a64560 Compare December 10, 2015 09:34

motin added 2 commits December 10, 2015 11:41

Restored list as it was before this pr

b3b63ff

Only exit on failure during development

57dcaa7

motin added 2 commits December 10, 2015 12:48

Activated opcode cache

8f75cbc

Including all frameworks despite stack (so that the resulting graphs …

54f5768

…share the same colors per framework)

motin added 3 commits December 10, 2015 12:58

Updated results, including also in readme

9720d47

Added script to run all benchmarks against all docker stacks

d8abdcf

Added note about source files for the used docker images

c342017

kenjis reviewed Dec 10, 2015
View reviewed changes

motin added 3 commits December 10, 2015 13:06

Adjusted working directories for docker stack services

0ddaf5a

Docker stacks use /php-framework-benchmark/ sub-directory as well

19d8bd7

Reversed changes that were made to support requests without the php-f…

b4d0e02

…ramework-benchmark sub-directory

motin added 4 commits December 10, 2015 13:54

Added docker results export script

0f210a6

Zero out incomplete results and only allow valid results for enabled …

c41f14c

…frameworks to affect the relative results

Updated docker stack results

22a0569

Moved parameters for cross-stack graph consistency to query string

84842c3

Including graphs to current results (including "no-framework")

e720dd1

kenjis reviewed Dec 10, 2015
View reviewed changes

motin added 2 commits December 11, 2015 09:10

Moved docker-related shell scripts to docker/bin/

e171242

Reversed usage of php_framework_benchmark_path

4e137a5

motin added 3 commits December 11, 2015 09:15

Added docker-compose build command to readme to ensure that all depen…

9861949

…dencies are being installed

Removed docker stack info from main readme

d9af4ae

Updated show_fw_array.sh to use list.php directly

1ebe534

o5 mentioned this pull request Sep 29, 2017

Simple docker environment #82

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple stacks #35

Multiple stacks #35

motin commented Dec 9, 2015

kenjis Dec 9, 2015

motin commented Dec 10, 2015

motin commented Dec 10, 2015

kenjis commented Dec 10, 2015

kenjis Dec 10, 2015

motin commented Dec 10, 2015

motin commented Dec 10, 2015

kenjis Dec 10, 2015

motin Dec 11, 2015

kenjis Dec 11, 2015

motin Dec 11, 2015

kenjis commented Dec 10, 2015

kenjis commented Dec 10, 2015

motin commented Dec 11, 2015

motin commented Dec 11, 2015

kenjis commented Dec 11, 2015

		By sharing underlying software stacks, the benchmark results vary only according to the host machine's hardware specs and differing code implementations.

		See [docker/README.md](docker/README.md)

Multiple stacks #35

Are you sure you want to change the base?

Multiple stacks #35

Conversation

motin commented Dec 9, 2015

kenjis Dec 9, 2015

Choose a reason for hiding this comment

motin commented Dec 10, 2015

motin commented Dec 10, 2015

kenjis commented Dec 10, 2015

kenjis Dec 10, 2015

Choose a reason for hiding this comment

motin commented Dec 10, 2015

motin commented Dec 10, 2015

kenjis Dec 10, 2015

Choose a reason for hiding this comment

motin Dec 11, 2015

Choose a reason for hiding this comment

kenjis Dec 11, 2015

Choose a reason for hiding this comment

motin Dec 11, 2015

Choose a reason for hiding this comment

kenjis commented Dec 10, 2015

kenjis commented Dec 10, 2015

motin commented Dec 11, 2015

motin commented Dec 11, 2015

kenjis commented Dec 11, 2015