Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Tests - and a question »

Tests - and a question

php.internals

Zoe Slattery

19 years ago
Hi all We added a few tests for array, string and variable handling functions at the weekend. As always - any feedback will be welcome. One thing we had some difficulty with was keeping the test output small enough to be easily read. We chose to divide up tests like this: test_name_b.phpt b=basic function test_name_v.phpt v=variation test_name_e.phpt e=error cases test_name_o.phpt o=object specific This may not be the right scheme and I'm happy to change it if anyone has better suggestions, although it is useful to seperate out the error case tests because quite a lot of the differences between php5 and php6 seem to be in the exact form of the error message. Using this doesn't completely solve the problem anyway - I think some of the "v" tests will will end up being test_name_v1, test_name_v2 etc. Thanks, Zoe IBM UK

Antony Dovgal

19 years ago
On 05/14/2007 08:43 PM, Zoe Slattery wrote:
> Hi all > > We added a few tests for array, string and variable handling functions > at the weekend. As always - any feedback will be welcome. > > One thing we had some difficulty with was keeping the test output small > enough to be easily read. We chose to divide up tests like this: > > test_name_b.phpt b=basic function > test_name_v.phpt v=variation > test_name_e.phpt e=error cases > test_name_o.phpt o=object specific
That would work for me, though I don't really see much sense in this. 001.phpt, test_name_001.phpt, func_1.phpt, test_name_v.phpt - they're all cryptic enough and are not supposed to carry any information. I believe they're just filenames and the _real_ test name is in the --TEST-- section.
> This may not be the right scheme and I'm happy to change it if anyone > has better suggestions, although it is useful to seperate out the error > case tests because quite a lot of the differences between php5 and php6 > seem to be in the exact form of the error message. Using this doesn't > completely solve the problem anyway - I think some of the "v" tests will > will end up being test_name_v1, test_name_v2 etc.
I would personally prefer having more small tests with names like "001.phpt" than four 50kb "func_name_001_[b|v|e|0].phpt"..
-- Wbr, Antony Dovgal

Marcus Börger

19 years ago
Hello Zoe, that's pretty fine. Instead of _[bveo][0-9]?.phpt you could do _(basic|variation|error|object)[0-9]{3}.phpt. Which is a tad bit better. However those are just filenames helping you to find the correct test file while developing. For that reason we didn not do so much about generating those files. We Only made it briefly and suggested a three digit numbering scheme. After all th ename in --TEST.. is the important thing. Thanks for the effort into test writing!!! best regards marcus Monday, May 14, 2007, 6:43:51 PM, you wrote:
> Hi all
> We added a few tests for array, string and variable handling functions > at the weekend. As always - any feedback will be welcome.
> One thing we had some difficulty with was keeping the test output small > enough to be easily read. We chose to divide up tests like this:
> test_name_b.phpt b=basic function > test_name_v.phpt v=variation > test_name_e.phpt e=error cases > test_name_o.phpt o=object specific
> This may not be the right scheme and I'm happy to change it if anyone > has better suggestions, although it is useful to seperate out the error > case tests because quite a lot of the differences between php5 and php6 > seem to be in the exact form of the error message. Using this doesn't > completely solve the problem anyway - I think some of the "v" tests will > will end up being test_name_v1, test_name_v2 etc.
> Thanks, Zoe > IBM UK
Best regards, Marcus

Zoe Slattery

19 years ago
Marcus Boerger wrote: Hi Marcus & Tony
> Hello Zoe, > > that's pretty fine. Instead of _[bveo][0-9]?.phpt you could do > _(basic|variation|error|object)[0-9]{3}.phpt. Which is a tad bit better. > However those are just filenames helping you to find the correct test > file while developing. For that reason we didn not do so much about > generating those files. We Only made it briefly and suggested a three > digit numbering scheme. After all th ename in --TEST.. is the important > thing. >
Thanks for the responses - you have both made the same point that it's the --TEST-- section that should really give a descriptive title. Marcus - I like your suggestion of basic|variation etc better than our current b|v etc scheme. Raghu - are you OK to switch to longer test case names? I think the test names are important to some degree - if you are trying to look for a testcase for a particular function it is easier to look at file names than to have to grep for the function name in a test title. Is there a mapping between the functions listed in the functional ref and their test cases? This kind of thing would be useful to us so I was thinking I'd write one. It's probably lazy of me :-) but I'd just like to click on a function name and go straight to a list of existing test cases. You also both made the same point that shorter tests are better and I think we can improve here - although we have been making an effort to make the output readable.

SoftVirus

19 years ago
Hello Zoe, Marcus & Tony, Files names having _(basic|variation|error|object).phpt looks much better. File names _[0-9]*.phpt does not give much detail about what are the functions being testing by this testcase until one looks at the --TEST-- section of the file. It would be nice to have the name of the testcase accomodate name of the functions that are being tested. For example if 009.phpt is written to test basic functionality of current() and next() functions, the better file name could be current_next_basic.phpt. Regards, Raghubansh, IBM India On 5/15/07, Zoe Slattery <zoe.slattery@googlemail.com> wrote:
> > Marcus Boerger wrote: > > Hi Marcus & Tony > > Hello Zoe, > > > > that's pretty fine. Instead of _[bveo][0-9]?.phpt you could do > > _(basic|variation|error|object)[0-9]{3}.phpt. Which is a tad bit better. > > However those are just filenames helping you to find the correct test > > file while developing. For that reason we didn not do so much about > > generating those files. We Only made it briefly and suggested a three > > digit numbering scheme. After all th ename in --TEST.. is the important > > thing. > > > Thanks for the responses - you have both made the same point that it's > the --TEST-- section that should really give a descriptive title. Marcus > - I like your suggestion of basic|variation etc better than our > current b|v etc scheme. Raghu - are you OK to switch to longer test case > names? > > I think the test names are important to some degree - if you are trying > to look for a testcase for a particular function it is easier to look at > file names than to have to grep for the function name in a test title. > Is there a mapping between the functions listed in the functional ref > and their test cases? This kind of thing would be useful to us so I was > thinking I'd write one. It's probably lazy of me :-) but I'd just like > to click on a function name and go straight to a list of existing test > cases. > > You also both made the same point that shorter tests are better and I > think we can improve here - although we have been making an effort to > make the output readable. > > > Thanks for the effort into test writing!!! > > > > best regards > > marcus > > > > Monday, May 14, 2007, 6:43:51 PM, you wrote: > > > > > >> Hi all > >> > > > > > >> We added a few tests for array, string and variable handling functions > >> at the weekend. As always - any feedback will be welcome. > >> > > > > > >> One thing we had some difficulty with was keeping the test output small > >> enough to be easily read. We chose to divide up tests like this: > >> > > > > > >> test_name_b.phpt b=basic function > >> test_name_v.phpt v=variation > >> test_name_e.phpt e=error cases > >> test_name_o.phpt o=object specific > >> > > > > > >> This may not be the right scheme and I'm happy to change it if anyone > >> has better suggestions, although it is useful to seperate out the error > >> case tests because quite a lot of the differences between php5 and php6 > >> seem to be in the exact form of the error message. Using this doesn't > >> completely solve the problem anyway - I think some of the "v" tests > will > >> will end up being test_name_v1, test_name_v2 etc. > >> > > > > > > > >> Thanks, Zoe > >> IBM UK > >> > > > > > > > > > > Best regards, > > Marcus > > > > > > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >
-- with Regards, Raghubansh

Antony Dovgal

19 years ago
On 05/15/2007 11:52 AM, SoftVirus wrote:
> Hello Zoe, Marcus & Tony, > > Files names having _(basic|variation|error|object).phpt looks much better. > File names _[0-9]*.phpt does not give much detail about what are the > functions being testing by this testcase until one looks at the --TEST-- > section of the file. > It would be nice to have the name of the testcase > accomodate name of the functions that are being tested. For example if > 009.phpt is written to test basic functionality of current() and next() > functions, the better file name could be current_next_basic.phpt.
Well, aesthetically - yes, "current_next_basic" might look better to those who don't like numbers. But you still have to open the test and read its --TEST-- section contents to learn what exactly it's testing. That said, I'm happy with both func_b.phpt and func_basic.phpt (and even 009.phpt) =) We just need more tests, doesn't matter what their names look like.
-- Wbr, Antony Dovgal

Zoe Slattery

19 years ago
Antony Dovgal wrote:
> On 05/15/2007 11:52 AM, SoftVirus wrote: >> Hello Zoe, Marcus & Tony, >> >> Files names having _(basic|variation|error|object).phpt looks much >> better. >> File names _[0-9]*.phpt does not give much detail about what are the >> functions being testing by this testcase until one looks at the --TEST-- >> section of the file. It would be nice to have the name of the testcase >> accomodate name of the functions that are being tested. For example if >> 009.phpt is written to test basic functionality of current() and next() >> functions, the better file name could be current_next_basic.phpt. > > Well, aesthetically - yes, "current_next_basic" might look better to > those who don't like numbers. > But you still have to open the test and read its --TEST-- section > contents to learn what exactly it's testing. > > That said, I'm happy with both func_b.phpt and func_basic.phpt (and > even 009.phpt) =) > We just need more tests, doesn't matter what their names look like. >
We're on it :-) Will probably have a few more questions on Unicode testing later, noticed that you are putting UEXPECT section in tests- but need to understand the implemenentation plan first...

Antony Dovgal

19 years ago
On 05/15/2007 12:44 PM, Zoe Slattery wrote:
> We're on it :-) Will probably have a few more questions on Unicode > testing later, noticed that you are putting UEXPECT section in tests- > but need to understand the implemenentation plan first...
We have two operation modes in PHP6 - "native" and Unicode. In these modes expected output of the same test may differ very much. That's why in some cases we have to use two expected output sections - EXPECT(F) for native and UEXPECT(F) for Unicode mode. * There is also UEXPECTREGEX - similar to EXPECTREGEX, but used in Unicode mode.
-- Wbr, Antony Dovgal

Zoe Slattery

19 years ago
Antony Dovgal wrote:
> On 05/15/2007 12:44 PM, Zoe Slattery wrote: >> We're on it :-) Will probably have a few more questions on Unicode >> testing later, noticed that you are putting UEXPECT section in tests- >> but need to understand the implemenentation plan first... > > We have two operation modes in PHP6 - "native" and Unicode. In these > modes expected output of the same test may differ very much. > That's why in some cases we have to use two expected output sections - > EXPECT(F) for native and UEXPECT(F) for Unicode mode.
Thanks - I've just been looking at this and would like to confirm that I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)-- in a PHPT file. In PHP5 run-tests.php will just ignore the --U.*-- sections. In PHP6 run-tests.php will use the --U.*-- sections if unicode_semantics=on and ignore the corresponding --EXPECT(F)-- sections. If unicode_semantics=off PHP6 will behave the same way as PHP5. Correct? Or is it more complicated that that?

Zoe Slattery

19 years ago
Zoe Slattery wrote:
> Antony Dovgal wrote: >> On 05/15/2007 12:44 PM, Zoe Slattery wrote: >>> We're on it :-) Will probably have a few more questions on Unicode >>> testing later, noticed that you are putting UEXPECT section in >>> tests- but need to understand the implemenentation plan first... >> >> We have two operation modes in PHP6 - "native" and Unicode. In these >> modes expected output of the same test may differ very much. >> That's why in some cases we have to use two expected output sections >> - EXPECT(F) for native and UEXPECT(F) for Unicode mode. > Thanks - I've just been looking at this and would like to confirm that > I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)-- > in a PHPT file. In PHP5 run-tests.php will just ignore the --U.*-- > sections. In PHP6 run-tests.php will use the --U.*-- sections if > unicode_semantics=on and ignore the corresponding --EXPECT(F)-- > sections. If unicode_semantics=off PHP6 will behave the same way as PHP5. > > Correct? Or is it more complicated that that?
Hmm - this isn't quite right. I get the test to behave differently with an --INI-- section which uses unicode_semantics=on/off, but not when I use a php.ini file...

Marcus Börger

19 years ago
Hello Zoe, it ignores the php.ini more or less. You should be using run-tests.php switches -u, -U and -N. Try: php run-tests.php -h best regards marcus Tuesday, May 15, 2007, 1:33:28 PM, you wrote:
> Zoe Slattery wrote: >> Antony Dovgal wrote: >>> On 05/15/2007 12:44 PM, Zoe Slattery wrote: >>>> We're on it :-) Will probably have a few more questions on Unicode >>>> testing later, noticed that you are putting UEXPECT section in >>>> tests- but need to understand the implemenentation plan first... >>> >>> We have two operation modes in PHP6 - "native" and Unicode. In these >>> modes expected output of the same test may differ very much. >>> That's why in some cases we have to use two expected output sections >>> - EXPECT(F) for native and UEXPECT(F) for Unicode mode. >> Thanks - I've just been looking at this and would like to confirm that >> I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)-- >> in a PHPT file. In PHP5 run-tests.php will just ignore the --U.*-- >> sections. In PHP6 run-tests.php will use the --U.*-- sections if >> unicode_semantics=on and ignore the corresponding --EXPECT(F)-- >> sections. If unicode_semantics=off PHP6 will behave the same way as PHP5. >> >> Correct? Or is it more complicated that that? > Hmm - this isn't quite right. I get the test to behave differently with > an --INI-- section which uses unicode_semantics=on/off, but not when I > use a php.ini file...
>>> >>> * There is also UEXPECTREGEX - similar to EXPECTREGEX, but used in >>> Unicode mode. >>> >> >>
Best regards, Marcus

Antony Dovgal

19 years ago
On 05/15/2007 02:46 PM, Zoe Slattery wrote:
> Antony Dovgal wrote: >> On 05/15/2007 12:44 PM, Zoe Slattery wrote: >>> We're on it :-) Will probably have a few more questions on Unicode >>> testing later, noticed that you are putting UEXPECT section in tests- >>> but need to understand the implemenentation plan first... >> >> We have two operation modes in PHP6 - "native" and Unicode. In these >> modes expected output of the same test may differ very much. >> That's why in some cases we have to use two expected output sections - >> EXPECT(F) for native and UEXPECT(F) for Unicode mode. > Thanks - I've just been looking at this and would like to confirm that > I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)-- > in a PHPT file. In PHP5 run-tests.php will just ignore the --U.*-- > sections. In PHP6 run-tests.php will use the --U.*-- sections if > unicode_semantics=on and ignore the corresponding --EXPECT(F)-- > sections. If unicode_semantics=off PHP6 will behave the same way as PHP5. > > Correct? Or is it more complicated that that?
Yup, everything is correct.
-- Wbr, Antony Dovgal

Stanislav Malyshev

19 years ago
> That said, I'm happy with both func_b.phpt and func_basic.phpt (and even > 009.phpt) =) > We just need more tests, doesn't matter what their names look like.
I think it does. If one needs to see which tests there are or are there any tests that test particular thing or inspect certain aspects of a functionality, navigating a maze of func_01234.phpt, all alike, is a nightmare. Of course, I could open all of them one by one and remember which one does what - that is if I were a being with unlimited lifespan, infallible memory and never felt bored - but for regular human it's a painful experience. So when writing a new tests it would be very good to give them names that make sense, it doesn't cost anything and helps people. Renaming old ones might be somewhat boring task, so not sure anybody ever would do that...
-- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/

Antony Dovgal

19 years ago
On 05/15/2007 08:52 PM, Stanislav Malyshev wrote:
>> That said, I'm happy with both func_b.phpt and func_basic.phpt (and even >> 009.phpt) =) >> We just need more tests, doesn't matter what their names look like. > > I think it does. If one needs to see which tests there are or are there > any tests that test particular thing or inspect certain aspects of a > functionality, navigating a maze of func_01234.phpt, all alike, is a > nightmare. Of course, I could open all of them one by one and remember > which one does what - that is if I were a being with unlimited lifespan, > infallible memory and never felt bored - but for regular human it's a > painful experience.
Well, I'm pretty sure regular humans know how to run `find . -name "*.phpt" | xargs grep "func`.
> So when writing a new tests it would be very good to > give them names that make sense, it doesn't cost anything and helps > people. Renaming old ones might be somewhat boring task, so not sure > anybody ever would do that...
-- Wbr, Antony Dovgal

Richard Lynch

19 years ago
On Tue, May 15, 2007 12:38 pm, Antony Dovgal wrote:
> On 05/15/2007 08:52 PM, Stanislav Malyshev wrote: > Well, I'm pretty sure regular humans know how to run `find . -name > "*.phpt" | xargs grep "func`.
I dunno... It took me years to figure out that xargs thing and how to make it go... I'd read the man page and see that it was what I wanted, and read the tutorials and all that, and end up trying a few zillion different things that failed before ever getting to the | and xargs bit... Not that I'm claiming to be a normal human. :-) [back on topic] I'm okay with any of the naming conventions, or possibly even a mixture that makes sense (e.g., bug report format could be unique from more generalized test names)
-- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So?

Zoe Slattery

19 years ago
Stanislav Malyshev wrote:
> I think it does. If one needs to see which tests there are or are > there any tests that test particular thing or inspect certain aspects > of a functionality, navigating a maze of func_01234.phpt, all alike, > is a nightmare. Of course, I could open all of them one by one and > remember which one does what - that is if I were a being with > unlimited lifespan, infallible memory and never felt bored - but for > regular human it's a painful experience. So when writing a new tests > it would be very good to give them names that make sense, it doesn't > cost anything and helps people. Renaming old ones might be somewhat > boring task, so not sure anybody ever would do that...
I'm happy to rename as we work through - if we can see an obvious and more descriptive name. Also, what about making the test directory structure match the functional reference for core functions? Some of it does at the moment (strings, array) but it would be good to put all the variable handling tests in ext/standard/tests/variable_handling for example. It's another thing we could do as we go along... Zoe Slattery IBM UK

SoftVirus

19 years ago
Hello Zoe, It would be nice thing to have mapping between categorization of the functions in the php doc and the source/test tree structure but keeping the tree structure as per the source code is better thing to do. If i have got it right as of now the testcases are places based on the location of the source file. For example testcases testing functions in the ext/standard/array.c are placed in ext/standard/tests/array dir, ext/standard/tests/reg.c is placed in ext/standar/tests/reg/ dir and so on. Testcases for functions found in the Zend dir are placed in Zend/tests dir. It would be nice to have the following addition to ext/standard/tests dir with respect to variable handling functions ( varible handling funnctions are implemented by different source file,e.g : type.c, var.c, Zend/zend_builtin_functions.c, etc) : A sub dir "type" (ext/standard/tests/type) where all the testcases testing functions in type.c can be placed, currently its placed in general_functions(ext/standard/tests/general_funtions) dir A sub dir "var" (ext/standard/tests/var) where all the testcases testing functions in var.c can be placed, currently they are placed in general_functions(ext/standard/tests/general_functions) dir, and so on. Doing this way its makes is very easy for locating a testcase and give better dir structure for test organization. It would be nice if these details were documented on php.net, May be right place is qa.php.net under the doc "how to write test". Marcus, Tony and others, Any comments from you ? Also notice that we have done commit of some string functions(strcasecmp, strcmp, strncmp) in string dir, which should have been committed in Zend/tests dir because those functions are implemented by source file placed in Zend dir. Best Regards, Raghubansh, IBM India On 5/16/07, Zoe Slattery <zoe.slattery@googlemail.com> wrote:

Marcus Börger

19 years ago
Hello Raghubansh, if such a mapping becomes more important i'd say we go with a xml file mapping maintained in either php-src or phpdoc cvs module. That way we do not have to rename all files and loose history. best regards marcus Wednesday, May 16, 2007, 1:58:42 PM, you wrote:
> Hello Zoe, > It would be nice thing to have mapping between categorization of the > functions in the php doc and the source/test tree structure but keeping the > tree structure as per the source code is better thing to do. If i have got > it right as of now the testcases are places based on the location of the > source file. For example testcases testing functions in the > ext/standard/array.c are placed in ext/standard/tests/array dir, > ext/standard/tests/reg.c is placed in ext/standar/tests/reg/ dir and so on. > Testcases for functions found in the Zend dir are placed in Zend/tests dir. > It would be nice to have the following addition to ext/standard/tests dir > with respect to variable handling functions ( varible handling funnctions > are implemented by different source file,e.g : type.c, var.c, > Zend/zend_builtin_functions.c, etc) : > A sub dir "type" (ext/standard/tests/type) where all the testcases testing > functions in type.c can be placed, currently its placed in > general_functions(ext/standard/tests/general_funtions) dir > A sub dir "var" (ext/standard/tests/var) where all the testcases testing > functions in var.c can be placed, currently they are placed in > general_functions(ext/standard/tests/general_functions) dir, and so on.
> Doing this way its makes is very easy for locating a testcase and give > better dir structure for test organization. It would be nice if these > details were documented on php.net, May be right place is qa.php.net under > the doc "how to write test".
> Marcus, Tony and others, Any comments from you ?
> Also notice that we have done commit of some string functions(strcasecmp, > strcmp, strncmp) in string dir, which should have been committed in > Zend/tests dir because those functions are implemented by source file placed > in Zend dir.
> Best Regards, > Raghubansh, IBM India
> On 5/16/07, Zoe Slattery <zoe.slattery@googlemail.com> wrote: >> >> Stanislav Malyshev wrote: >> > I think it does. If one needs to see which tests there are or are >> > there any tests that test particular thing or inspect certain aspects >> > of a functionality, navigating a maze of func_01234.phpt, all alike, >> > is a nightmare. Of course, I could open all of them one by one and >> > remember which one does what - that is if I were a being with >> > unlimited lifespan, infallible memory and never felt bored - but for >> > regular human it's a painful experience. So when writing a new tests >> > it would be very good to give them names that make sense, it doesn't >> > cost anything and helps people. Renaming old ones might be somewhat >> > boring task, so not sure anybody ever would do that... >> I'm happy to rename as we work through - if we can see an obvious and >> more descriptive name. Also, what about making the test directory >> structure match the functional reference for core functions? Some of it >> does at the moment (strings, array) but it would be good to put all the >> variable handling tests in ext/standard/tests/variable_handling for >> example. It's another thing we could do as we go along... >> >> Zoe Slattery >> IBM UK >> >> -- >> PHP Internals - PHP Runtime Development Mailing List >> To unsubscribe, visit: http://www.php.net/unsub.php >> >>
Best regards, Marcus