{
    "href": "/post/2010/10/27/comparing-benchmark-tools/",
    "relId": "2010/10/27/comparing-benchmark-tools",
    "title": "Comparing Benchmark Tools",
    "author": "pmjones",
    "markup": "html",
    "tags": [
        {
            "href": "/tag/benchmarks/",
            "relId": "benchmarks",
            "title": "Benchmarks",
            "author": null,
            "created": null,
            "updated": [],
            "markup": "markdown"
        },
        {
            "href": "/tag/php/",
            "relId": "php",
            "title": "PHP",
            "author": null,
            "created": null,
            "updated": [],
            "markup": "markdown"
        },
        {
            "href": "/tag/programming/",
            "relId": "programming",
            "title": "Programming",
            "author": null,
            "created": null,
            "updated": [],
            "markup": "markdown"
        }
    ],
    "created": "2010-10-27 15:43:47 UTC",
    "updated": [
        "2010-10-27 15:43:47 UTC"
    ],
    "html": "<p>As I noted last week, I have moved my framework benchmarking project to <a href=\"http://github.com/pmjones/php-framework-benchmarks\">GitHub</a>. As part of the move, I updated the project to allow benchmarking using any of three tools:  Acme <a href=\"http://www.acme.com/software/http_load/\">http_load</a>, Apache <a href=\"http://httpd.apache.org/docs/2.0/programs/ab.html\">ab</a>, or Joedog <a href=\"http://www.joedog.org/index/siege-home\">siege</a>. (For reference, the old project will remain at <a href=\"http://code.google.com/p/web-framework-benchmarks/\">GoogleCode</a>.)</p>\n<p>I thought it might be interesting to see what each of them reports for the baseline \u201cindex.html\u201d and \u201cindex.php\u201d cases on the new Amazon EC2 setup (using a 64-bit OS on an <code>m1.large</code> instance).  The results follow (all are at 10 concurrent users, averaged over 5 one-minute runs):</p>\n<pre><code>ab                       |      rel |      avg |\n------------------------ | -------- | -------- |\nbaseline-html            |   1.2660 |  3581.54 |\nbaseline-php             |   1.0000 |  2829.11 |\n\nhttp_load                |      rel |      avg |\n------------------------ | -------- | -------- |\nbaseline-html            |   1.2718 |  4036.24 |\nbaseline-php             |   1.0000 |  3173.56 |\n\nsiege                    |      rel |      avg |\n------------------------ | -------- | -------- |\nbaseline-html            |   1.2139 |  5060.25 |\nbaseline-php             |   1.0000 |  4168.76 |\n</code></pre>\n<p>They all show very different \u201cabsolute\u201d numbers of requests/second: <code>ab</code> thinks the server delivers about 3600 req/sec, <code>http_load</code> reports about 4000, and <code>siege</code> says about 5000.</p>\n<p>Note that the <code>ab</code> and <code>http_load</code> <em>relative</em> scores are in line with each other, reporting about a 26-27% slowdown for invoking PHP.  Siege thinks PHP is more responsive than that, with only a 21% slowdown.</p>\n<p>Which of these is the most accurate?  I don\u2019t know.  I ran the benchmarking tool on the same server as was being benchmarked, so the differences may result from how much processing power was being consumed by the benchmarking tools themselves.</p>\n<p>One interesting point is that <code>ab</code> no longer appears to be over-reporting the baseline cases, as I noted in an <a href=\"http://paul-m-jones.com/archives/421\">earlier benchmark posting</a>. There are two major changes between then and now: (1) the updated project uses Ubuntu 10.10 instead of 8.10, which means the packaged <code>ab</code> binary might have been flawed earlier, or that the new OS otherwise corrects some other issue; (2) the updated project uses an <code>m1.large</code> 64-bit instance instead of an <code>m1.small</code> 32-bit instance.  Either of those differences might be sufficient to account for the disparity in <code>ab</code> reporting previously.</p>\n"
}