Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Benchmark Suite »

Benchmark Suite

php.internals

Sebastian Bergmann

19 years ago
Hi, I invested about half an hour this morning to refactor bench.php. It now starts a new PHP process for each benchmark and uses microtime(true) and memory_get_peak_usage() to measure benchmark data. Have a look at - http://phpfi.com/208939 This is the main script. - http://phpfi.com/208937 These are the iterators used by the main script. - http://phpfi.com/208935 This code is shared between the benchmark scripts. - http://phpfi.com/208941 This is an example of a benchmark script. Here is an example output: sb@vmware benchmark % php run.php Benchmark Time Peak Memory Usage ackermann 3.067452 715080 mandel 2.242472 127840 mandel2 2.185040 123448 simple 1.031363 117184 simplecall 3.849651 115528 simpleucall 4.226199 116968 simpleudcall 4.394062 117176 Note that I have not yet ported all benchmarks from bench.php. At this point I would like to know whether or not people are interested in this. No point in working on this if nobody cares, right? :-) Best, Sebastian
-- Sebastian Bergmann http://sebastian-bergmann.de/ GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Antony Dovgal

19 years ago
On 02/27/2007 01:23 PM, Sebastian Bergmann wrote:
> Here is an example output: > > sb@vmware benchmark % php run.php > Benchmark Time Peak Memory Usage > ackermann 3.067452 715080 > mandel 2.242472 127840 > mandel2 2.185040 123448 > simple 1.031363 117184 > simplecall 3.849651 115528 > simpleucall 4.226199 116968 > simpleudcall 4.394062 117176 > > Note that I have not yet ported all benchmarks from bench.php. > > At this point I would like to know whether or not people are interested > in this. No point in working on this if nobody cares, right? :-)
I like it. Please continue.
-- Wbr, Antony Dovgal

Sebastian Bergmann

19 years ago
Antony Dovgal wrote:
> I like it. Please continue.
Okay, great. Now: where should I put it? I think the current approach of having bench.php duplicated in each branch of the Zend Engine is sub- optimal. Maybe we could create a new top-level CVS module for this?
-- Sebastian Bergmann http://sebastian-bergmann.de/ GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Antony Dovgal

19 years ago
On 02/27/2007 02:42 PM, Sebastian Bergmann wrote:
> Antony Dovgal wrote: >> I like it. Please continue. > > Okay, great. Now: where should I put it? I think the current approach of > having bench.php duplicated in each branch of the Zend Engine is sub- > optimal. Maybe we could create a new top-level CVS module for this?
Maybe. But separate CVS module would make sense only if we're going to release it separately. I guess qa.php.net would be a good place for such releases. Opinions?
-- Wbr, Antony Dovgal

Marcus Börger

19 years ago
Hello Antony, QA sounds fine. Top-level is also fine as we have no real mathing place. When we offer it through QA we might aswell use qw-web as the cvs module but a seperate one sounds better. best regards marcus Tuesday, February 27, 2007, 5:58:41 PM, you wrote:
> On 02/27/2007 02:42 PM, Sebastian Bergmann wrote: >> Antony Dovgal wrote: >>> I like it. Please continue. >> >> Okay, great. Now: where should I put it? I think the current approach of >> having bench.php duplicated in each branch of the Zend Engine is sub- >> optimal. Maybe we could create a new top-level CVS module for this?
> Maybe. > But separate CVS module would make sense only if we're going to release it separately. > I guess qa.php.net would be a good place for such releases. > Opinions?
> -- > Wbr, > Antony Dovgal
Best regards, Marcus

Antony Dovgal

19 years ago
On 02/27/2007 11:48 PM, Marcus Boerger wrote:
> Hello Antony, > > QA sounds fine. Top-level is also fine as we have no real mathing place.
I meant using qa.php.net for releases, I don't care where exactly it lives in our CVS.
> When we offer it through QA we might aswell use qw-web as the cvs module > but a seperate one sounds better.
-- Wbr, Antony Dovgal

Sebastian Bergmann

19 years ago
Sebastian Bergmann wrote:
> Have a look at
I uploaded a snapshot of what I have now (I added fallbacks for missing functionality in PHP 4.4 and PHP 5.0 and the rest of the benchmarks) to http://static.phpunit.de/benchmark.tbz2.
-- Sebastian Bergmann http://sebastian-bergmann.de/ GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Marcus Börger

19 years ago
Hello Sebastian, looks pretty fine. Only you don't need fallbacks right now. Simply add a check for PHP >= 5.2.0 since you are using Iterators and I don't see any sense in testing 5.0 or 5.1. You could further more make the run script operate a bit like run-tests.php and have the PHP executable be configurable and get version tested. So you would first check that the PHP runiing the run script is new enough and that second the tests are executed by an acceptable PHP version. When you want to spend more work, ever thought of making the bench files have a layout similar to *.phpt? So that we could split up in sections like name, actual bench, clean-up, etc? Or do you prefer to do all from within one file and call bench funcs manually? best regards marcus Tuesday, February 27, 2007, 5:53:53 PM, you wrote:
> Sebastian Bergmann wrote: >> Have a look at
> I uploaded a snapshot of what I have now (I added fallbacks for missing > functionality in PHP 4.4 and PHP 5.0 and the rest of the benchmarks) to > http://static.phpunit.de/benchmark.tbz2.
> -- > Sebastian Bergmann http://sebastian-bergmann.de/ > GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Best regards, Marcus

Sebastian Bergmann

19 years ago
Marcus Boerger wrote:
> Only you don't need fallbacks right now. Simply add a check for > PHP >= 5.2.0 since you are using Iterators and I don't see any sense > in testing 5.0 or 5.1.
Already did that yesterday.
> You could further more make the run script operate a bit like > run-tests.php and have the PHP executable be configurable and get > version tested.
I am taking baby steps at the moment :) Of course the PHP binary will be configurable.
> When you want to spend more work, ever thought of making the bench > files have a layout similar to *.phpt? So that we could split up in > sections like name, actual bench, clean-up, etc?
The question is: is a separate system such as mine needed or could we extend run-tests.php? If we choose to extend run-tests.php, suddenly every *.phpt test would be a potential benchmark.
-- Sebastian Bergmann http://sebastian-bergmann.de/ GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Sebastian Bergmann

19 years ago
Sebastian Bergmann wrote:
> I uploaded a snapshot of what I have now (I added fallbacks for missing > functionality in PHP 4.4 and PHP 5.0 and the rest of the benchmarks) to > http://static.phpunit.de/benchmark.tbz2.
I updated the snapshot. This is what the output looks like now: Benchmark Time Peak Memory Peak Memory (Real) ------------------------------------------------------------------- ackermann 3.050 718336 786432 array 0.104 11949880 12058624 array2 0.101 11961848 12058624 array3 1.285 648016 786432 fibonacci 14.293 121936 262144 hash 0.653 7347400 7602176 hash2 0.115 247728 262144 heapsort 0.753 2941128 3145728 mandel 3.534 131920 262144 mandel2 2.806 124616 262144 matrix 0.703 636512 786432 nestedloop 2.844 125568 262144 sieve 0.796 2529288 2883584 simple 1.607 121320 262144 simplecall 5.828 119608 262144 simpleucall 6.526 121048 262144 simpleudcall 6.760 121256 262144 strcat 0.117 898008 1310720
-- Sebastian Bergmann http://sebastian-bergmann.de/ GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Antony Dovgal

19 years ago
Hello Sebastian. What's the status of this? Do you have anything working? On 27.02.2007 13:23, Sebastian Bergmann wrote:
> Hi, > > I invested about half an hour this morning to refactor bench.php. It now > starts a new PHP process for each benchmark and uses microtime(true) and > memory_get_peak_usage() to measure benchmark data. > > Have a look at > > - http://phpfi.com/208939 > > This is the main script. > > - http://phpfi.com/208937 > > These are the iterators used by the main script. > > - http://phpfi.com/208935 > > This code is shared between the benchmark scripts. > > - http://phpfi.com/208941 > > This is an example of a benchmark script. > > Here is an example output: > > sb@vmware benchmark % php run.php > Benchmark Time Peak Memory Usage > ackermann 3.067452 715080 > mandel 2.242472 127840 > mandel2 2.185040 123448 > simple 1.031363 117184 > simplecall 3.849651 115528 > simpleucall 4.226199 116968 > simpleudcall 4.394062 117176 > > Note that I have not yet ported all benchmarks from bench.php. > > At this point I would like to know whether or not people are interested > in this. No point in working on this if nobody cares, right? :-) > > Best, > Sebastian >
-- Wbr, Antony Dovgal

Sebastian Bergmann

19 years ago
Antony Dovgal wrote:
> Hello Sebastian.
Hello Antony,
> What's the status of this? > Do you have anything working?
back in February/March when I first posted about this, I discussed this with Marcus. The conclusion was that we wanted to rewrite run-tests.php based on the Iterator approach I used for the benchmark system's prototype. Benchmarks would then be .phpb in addition to .phpt for tests. I did some work on this but stopped when I got negative feedback (on IRC) towards the idea of rewriting "something that works" (run-tests.php). I see several options to continue: - Add .phpb style support for benchmarks to the current run-tests.php. - Improve run-tests.php to also collect Peak Memory Usage, store the data in SQLite, and have a test (optionally) fail when it takes more memory than it did in the past (one of the ideas that I had for the rewrite). - Do the rewrite. Opinions?
-- Sebastian Bergmann http://sebastian-bergmann.de/ GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Antony Dovgal

19 years ago
On 05.06.2007 11:25, Sebastian Bergmann wrote:
> Antony Dovgal wrote: >> Hello Sebastian. > > Hello Antony, > >> What's the status of this? >> Do you have anything working? > > back in February/March when I first posted about this, I discussed this > with Marcus. The conclusion was that we wanted to rewrite run-tests.php > based on the Iterator approach I used for the benchmark system's > prototype.
Require SPL for a test suite? Doesn't seem to be good idea..
> Benchmarks would then be .phpb in addition to .phpt for > tests. > > I did some work on this but stopped when I got negative feedback (on > IRC) towards the idea of rewriting "something that works" > (run-tests.php).
Well, I personally think we'll have to do it sooner or later. The current version run-tests.php is really hard to maintain and improve. We can have both for some time and drop the old one when we're sure that the new one is stable. It's a developers tool anyway..
> I see several options to continue: > > - Add .phpb style support for benchmarks to the current run-tests.php. > > - Improve run-tests.php to also collect Peak Memory Usage, store the > data in SQLite, and have a test (optionally) fail when it takes more > memory than it did in the past (one of the ideas that I had for the > rewrite).
Oh, that would be too much IMO..
> - Do the rewrite. > > Opinions?
I'm for the rewrite and I'm ready to help.
-- Wbr, Antony Dovgal

Lukas Kahwe Smith

19 years ago
Antony Dovgal wrote:
> On 05.06.2007 11:25, Sebastian Bergmann wrote: >> Antony Dovgal wrote: >>> Hello Sebastian. >> >> Hello Antony, >> >>> What's the status of this? >>> Do you have anything working? >> >> back in February/March when I first posted about this, I discussed this >> with Marcus. The conclusion was that we wanted to rewrite run-tests.php >> based on the Iterator approach I used for the benchmark system's >> prototype.
I did not catch the start of this. This is about performance regression testing?
> Require SPL for a test suite? > Doesn't seem to be good idea..
Well SPL should not be allowed to be disabled anyways, imho. regards, Lukas

Antony Dovgal

19 years ago
On 05.06.2007 12:22, Lukas Kahwe Smith wrote:
> Antony Dovgal wrote: >> On 05.06.2007 11:25, Sebastian Bergmann wrote: >>> Antony Dovgal wrote: >>>> Hello Sebastian. >>> >>> Hello Antony, >>> >>>> What's the status of this? >>>> Do you have anything working? >>> >>> back in February/March when I first posted about this, I discussed this >>> with Marcus. The conclusion was that we wanted to rewrite run-tests.php >>> based on the Iterator approach I used for the benchmark system's >>> prototype. > > I did not catch the start of this. This is about performance regression > testing? > >> Require SPL for a test suite? >> Doesn't seem to be good idea.. > > Well SPL should not be allowed to be disabled anyways, imho.
I agree, but that's out of the topic.
-- Wbr, Antony Dovgal

Pierre Joye

19 years ago
On 6/5/07, Antony Dovgal <antony@zend.com> wrote:
> > Well SPL should not be allowed to be disabled anyways, imho. > > I agree, but that's out of the topic.
Well, as we may require for the tests suite, and it looks like a non sense to be able to disable part of it, it is the right time to fix solve this issue :) It should not be possible to disable it. Or even better, we should move what we consider as core from SPL to Zend. I don't think the "hundreds" of php script and numerous iterators should be moved but the required ones. A couple of functions or features (I don't understand what they do in SPL) should be moved as well: class_implements, class_parents, spl_object_hash --Pierre

Lukas Kahwe Smith

19 years ago
Pierre wrote:
> It should not be possible to disable it. Or even better, we should > move what we consider as core from SPL to Zend.
Amen. regards, Lukas

Jani Taskinen

19 years ago
I'd like to say something involving smoke and carcass but that'll bite me later so.. :) I'm all for moving the useful stuff to Zend where it belongs. But I'm totally against making SPL another exception to the rule. Along with ext/date and ext/standard that is. Extension is an extension. It's not core, it's not "standard". If there are parts in the extension that should always be available -> move to core! --Jani Pierre kirjoitti:

Derick Rethans

19 years ago
On Tue, 5 Jun 2007, Antony Dovgal wrote:
> On 05.06.2007 12:22, Lukas Kahwe Smith wrote: > > Antony Dovgal wrote: > > > On 05.06.2007 11:25, Sebastian Bergmann wrote: > > > > Antony Dovgal wrote: > > > > > > > > > What's the status of this? > > > > > Do you have anything working? > > > > > > > > back in February/March when I first posted about this, I > > > > discussed this with Marcus. The conclusion was that we wanted to > > > > rewrite run-tests.php based on the Iterator approach I used for > > > > the benchmark system's prototype. > > > > I did not catch the start of this. This is about performance > > regression testing? > > > > > Require SPL for a test suite? > > > Doesn't seem to be good idea.. > > > > Well SPL should not be allowed to be disabled anyways, imho. > > I agree, but that's out of the topic.
Then what is the issue with using this in the tests suite? Derick

Antony Dovgal

19 years ago
On 05.06.2007 12:43, Derick Rethans wrote:
>> > > Require SPL for a test suite? >> > > Doesn't seem to be good idea.. >> > >> > Well SPL should not be allowed to be disabled anyways, imho. >> >> I agree, but that's out of the topic. > > Then what is the issue with using this in the tests suite?
<@tony2001> I know that ext/pcre can be disabled, but it's good idea to keep the number of required extensions as low as we can
-- Wbr, Antony Dovgal