{
    "href": "/post/2009/03/05/a-siege-on-benchmarks/",
    "relId": "2009/03/05/a-siege-on-benchmarks",
    "title": "A Siege On Benchmarks",
    "author": "pmjones",
    "markup": "html",
    "tags": [
        {
            "href": "/tag/benchmarks/",
            "relId": "benchmarks",
            "title": "Benchmarks",
            "author": null,
            "created": null,
            "updated": [],
            "markup": "markdown"
        },
        {
            "href": "/tag/php/",
            "relId": "php",
            "title": "PHP",
            "author": null,
            "created": null,
            "updated": [],
            "markup": "markdown"
        }
    ],
    "created": "2009-03-05 18:43:46 UTC",
    "updated": [
        "2009-03-05 18:43:46 UTC"
    ],
    "html": "<p>My regular readers (and perhaps the irregular ones as well ;-) know that I have been obsessed with baseline-responsiveness benchmarking of frameworks for years now. \u00c2\u00a0The idea has always been that, in order to know how far you can optimize your framework-based applications, you need to know the limits imposed by the framework itself. \u00c2\u00a0Only then can you have an idea of where to spend your limited resources on improvement. \u00c2\u00a0For example, if you need 200 <strong>dynamic</strong> requests/second, but the framework itself (with no application code in use) is capable only of 100, then you know that no amount of application or database optimization will help you -- it's time to start scaling, either horizontally or vertically.</p>\n<p>To perform these benchmarks, I have only employed the <code>ab</code> tool provided by the Apache web server. \u00c2\u00a0It was easy to use, and relatively easy to parse the output to automate reporting. \u00c2\u00a0However, it turns out that <code>ab</code> over-reports responsiveness of Apache when serving static HTML files, and when serving minimal PHP scripts such as <code>&lt;?php echo \"hello world\"; ?&gt;</code>. \u00c2\u00a0I discovered this just recently when attempting to find out <a href=\"http://paul-m-jones.com/?p=413\">why PHP appeared to be faster than HTML</a>, and then only with the assistance of <a href=\"http://blog.preinheimer.com/\">Paul Reinheimer</a>, whom I now owe a bottle of vodka for his trouble. \u00c2\u00a0;-)</p>\n<p>It turns out that the <code>siege</code> tool from <a href=\"http://www.joedog.org/index/siege-home\">JoeDog Software</a> is more accurate in reporting static HTML and PHP responsiveness. \u00c2\u00a0This is confirmed through Paul Reinheimer as well, who reported the expected responsiveness on other systems.</p>\n<p>The over-reporting from <code>ab</code> means that all my previous reporting on benchmarks is skewed too low when comparing framework responsiveness to PHP's maximum responsiveness. \u00c2\u00a0As such, I have re-run all the previously published benchmarks using <code>siege</code> instead of <code>ab</code>. \u00c2\u00a0Previous runs with <code>ab</code> are here ...</p>\n<ul>\n<li><a href=\"http://paul-m-jones.com/blog/?p=236\">http://paul-m-jones.com/blog/?p=236</a></li>\n<li><a href=\"http://paul-m-jones.com/blog/?p=238\">http://paul-m-jones.com/blog/?p=238</a></li>\n<li><a href=\"http://paul-m-jones.com/blog/?p=315\">http://paul-m-jones.com/blog/?p=315</a></li>\n</ul>\n<p>... and below are the updated <code>siege</code> versions. \u00c2\u00a0As with previous attempts, these benchmarks are performed on an Amazon EC2 \"small\" instance. \u00c2\u00a0There is one difference to note: previous runs used Xcache for bytecode caching, but these use APC; I don't suspect this change in caching engines has a significant effect, but I have not tested that assertion.</p>\n<table border=\"2\" cellspacing=\"0\" cellpadding=\"4\">\n<tr>\n<th>framework</th>\n<th>rel</th>\n<th>avg</th>\n</tr>\n<tr>\n<td>baseline-html</td>\n<td>1.1878</td>\n<td>985.69</td>\n</tr>\n<tr>\n<td>baseline-php</td>\n<td>1.0000</td>\n<td>829.82</td>\n</tr>\n<tr>\n<td>cake-1.1.10</td>\n<td>0.0938</td>\n<td>77.84</td>\n</tr>\n<tr>\n<td>cake-1.1.11</td>\n<td>0.1277</td>\n<td>105.96</td>\n</tr>\n<tr>\n<td>cake-1.1.12</td>\n<td>0.1288</td>\n<td>106.84</td>\n</tr>\n<tr>\n<td>cake-1.1.16</td>\n<td>0.1166</td>\n<td>96.77</td>\n</tr>\n<tr>\n<td>cake-1.1.17</td>\n<td>0.1165</td>\n<td>96.70</td>\n</tr>\n<tr>\n<td>cake-1.1.19</td>\n<td>0.1298</td>\n<td>107.69</td>\n</tr>\n<tr>\n<td>cake-1.2.0-rc2</td>\n<td>0.0516</td>\n<td>42.79</td>\n</tr>\n<tr>\n<td>solar-0.25.0</td>\n<td>0.1852</td>\n<td>153.66</td>\n</tr>\n<tr>\n<td>solar-0.26.0</td>\n<td>0.1789</td>\n<td>148.43</td>\n</tr>\n<tr>\n<td>solar-0.27.0</td>\n<td>0.1734</td>\n<td>143.93</td>\n</tr>\n<tr>\n<td>solar-0.28.0</td>\n<td>0.1671</td>\n<td>138.64</td>\n</tr>\n<tr>\n<td>solar-1.0.0alpha1</td>\n<td>0.1706</td>\n<td>141.58</td>\n</tr>\n<tr>\n<td>symfony-0.6.3</td>\n<td>0.0629</td>\n<td>52.22</td>\n</tr>\n<tr>\n<td>symfony-1.0.0beta2</td>\n<td>0.0758</td>\n<td>62.91</td>\n</tr>\n<tr>\n<td>symfony-1.0.6</td>\n<td>0.0746</td>\n<td>61.91</td>\n</tr>\n<tr>\n<td>symfony-1.0.6-dw</td>\n<td>0.0820</td>\n<td>68.03</td>\n</tr>\n<tr>\n<td>symfony-1.0.6-fp</td>\n<td>0.0853</td>\n<td>70.78</td>\n</tr>\n<tr>\n<td>symfony-1.0.17</td>\n<td>0.0744</td>\n<td>61.73</td>\n</tr>\n<tr>\n<td>symfony-1.1.0</td>\n<td>0.0745</td>\n<td>61.84</td>\n</tr>\n<tr>\n<td>zend-0.2.0</td>\n<td>0.2176</td>\n<td>180.56</td>\n</tr>\n<tr>\n<td>zend-0.6.0</td>\n<td>0.1998</td>\n<td>165.78</td>\n</tr>\n<tr>\n<td>zend-1.0.0</td>\n<td>0.1268</td>\n<td>105.25</td>\n</tr>\n<tr>\n<td>zend-1.0.1</td>\n<td>0.1263</td>\n<td>104.80</td>\n</tr>\n<tr>\n<td>zend-1.5.2</td>\n<td>0.0951</td>\n<td>78.93</td>\n</tr>\n</table>\n<p>Note the baseline-html and baseline-php numbers. \u00c2\u00a0Using <code>ab</code> previously, these were reported as 2100-2400 requests/second and 1100-1400 requests/second, respectively. \u00c2\u00a0The <code>siege</code> tool reports a much lower number for both, but the dropoff between static HTML and dynamic PHP is much smaller; with <code>ab</code> it looked like about 40-50%, but now with <code>siege</code> it looks like only about 15-18%. \u00c2\u00a0This behavior is much more like what we would expect from a memory-based PHP script.</p>\n<p>Note also the separate framework requests/second; they are very similar between <code>ab</code> and <code>siege</code>. \u00c2\u00a0This means that the framework responsiveness numbers are almost unchanged.</p>\n<p>Because the nearly-identical framework numbers are compared to a much smaller baseline PHP number, the frameworks now appear to be doing much better in relation to PHP's maximum responsiveness. \u00c2\u00a0For example, Solar-1.0.0alpha1 with <code>ab</code> appeared to run at about 11% of PHP's max, but with <code>siege</code> it looks close to 17%. \u00c2\u00a0All of the frameworks tested see this kind of comparative gain in their reporting.</p>\n<p>However, when compared to each other, the framework rankings are the same as before: \u00c2\u00a0Solar has the highest baseline responsiveness, followed by Cake and Zend (their respective releases are very close to each other in responsiveness), and Symfony trails with the lowest baseline responsiveness.</p>\n<p>In summary, using <code>ab</code> skewed the \"percentage of PHP\" comparisons because it over-reported PHP's maximum responsiveness, but the framework requests/second numbers and the framework comparative rankings are unchanged from previous reporting. \u00c2\u00a0The <a href=\"http://code.google.com/p/web-framework-benchmarks/\">Google project for the benchmarking system</a> has been updated to use <code>siege</code>, so all future reporting will reflect its results, not those of <code>ab</code>.</p>\n"
}