Everything

php

Home » Project » Newsgroups » php.internals » PHP Release Process Sucks »

PHP Release Process Sucks

php.internals

Stefan Esser

20 years ago

Hello,

okay, mistakes happen everyday but it really sucks that PHP.net
continues trying to hide mistakes.

1) PHP 5.1.4 was released with a nonsense announcement claiming that
there was only a problem with POST arrays or POST fileuploads.
-> In reality a paid Zend developer had destroyed the handling of
arrays in any kind of user input in PHP 5.1.3 completely. Ironically
after that incident another Zend man came forward and dares to say "I
don't trust our core testers anymore"
2) PHP 5.1.4 was lacking the PEAR installer which resulted in make
install downloading the file from the web.
a) this part should be removed from the make file completlely
because 'make install' is usually executed as root and under no
circumstances should download a file from an insecure HTTP source.
b) this fact was again hidden by silently replacing the PHP 5.1.4
tarball with a new one, after the other one was out for more than a week.

PHP.net is more and more turning into Microsoft (more than 3 months to
resolve critical security problems). I guess that comes with the
involvement of Enterprise companies.

Yours,
Stefan Esser

Ilia A.

20 years ago

On 15-May-06, at 9:39 AM, Stefan Esser wrote:

> Hello,
>
> okay, mistakes happen everyday but it really sucks that PHP.net
> continues trying to hide mistakes.
>
> 1) PHP 5.1.4 was released with a nonsense announcement claiming that
> there was only a problem with POST arrays or POST fileuploads.
>     -> In reality a paid Zend developer had destroyed the handling of
> arrays in any kind of user input in PHP 5.1.3 completely. Ironically
> after that incident another Zend man came forward and dares to say "I
> don't trust our core testers anymore"

That is what the bug reports that appeared in relation to the problem
were complaining about, the announcement was tailored to address
issues raised by those people.

> 2) PHP 5.1.4 was lacking the PEAR installer which resulted in make
> install downloading the file from the web.
>     a) this part should be removed from the make file completlely
> because 'make install' is usually executed as root and under no
> circumstances should download a file from an insecure HTTP source.

We didn't have an automated process for including PEAR's phar file
in, it is now part of the release generation script. As far as
automated download, for full releases this will not be necessary in
the future as it will be included by default. As far as the phar
download, I think it either needs to require a confirmation prompt or
simply not done automatically, either way is fine by me.

>     b) this fact was again hidden by silently replacing the PHP 5.1.4
> tarball with a new one, after the other one was out for more than a
> week.
>

The code in the release did not change on bit, the only change was
the inclusion of the missing phar file, this hardly warrants 5.1.5 or
even 5.1.4pl1. This will have no impact of people who have already
downloaded and installed PHP, nor will this impact people who have
yet to download PHP.

> PHP.net is more and more turning into Microsoft (more than 3 months to
> resolve critical security problems). I guess that comes with the
> involvement of Enterprise companies.
>

The patches for security holes are usually in within a week, if you
want to fetch them you can do so either in a form of a PHP snapshot
of a specific patch from CVS. To make releases every-time we get
security fault is impractical. Like most other projects a number of
security fixes is combined into group and when a group is
sufficiently large a release is made. None of the bugs resolved were
super-critical issues such as remote code execution, which would
indeed warrant something like an emergency release just for that
particular patch.

Ilia

Stefan Esser

20 years ago

Hey,

>
> The code in the release did not change on bit, the only change was the
> inclusion of the missing phar file, this hardly warrants 5.1.5 or even
> 5.1.4pl1. This will have no impact of people who have already
> downloaded and installed PHP, nor will this impact people who have yet
> to download PHP.
>

It will have an effect on everyone using f.e. gentoo linux or BSD port
system, because mysteriously the hash of the tarball changed and people
will get warnings about modified tarballs. It also has the effect that I
am getting emails from people asking me if PHP.net was backdoored,
because the MD5 hash changed....
And if you want to change tarballs and don't change the version number
(which is considered very bad by many people) then atleast WARN people
about the modified tarball. A simple message: tarball was missing PEAR
and was therefore rerolled is not so bad...

> The patches for security holes are usually in within a week, if you
> want to fetch them you can do so either in a form of a PHP snapshot of
> a specific patch from CVS. To make releases every-time we get security
> fault is impractical.

First the zend_hash_del() bug caused remote code execution in a bunch of
popular PHP scripts. Secondly most open source projects release security
bugfix releases. PHP.net on the other hand doesn't do this anymore.
There are no security only fixes anymore. Instead we release not
properly tested new versions of PHP that break tons of servers. (fastcgi
....)

And well... I still see no PHP 4.4.3 on PHP.net... However we still
offer the PHP 4.4.2 tarball (knowing that it has critical security
holes). So either we release a security FIX release or we kick the
tarball and declare PHP4 unsupported from now on.

Yours,
Stefan Esser

Ilia A.

20 years ago

On 15-May-06, at 11:50 AM, Stefan Esser wrote:

> Hey,
>>
>> The code in the release did not change on bit, the only change was
>> the
>> inclusion of the missing phar file, this hardly warrants 5.1.5 or
>> even
>> 5.1.4pl1. This will have no impact of people who have already
>> downloaded and installed PHP, nor will this impact people who have
>> yet
>> to download PHP.
>>
> It will have an effect on everyone using f.e. gentoo linux or BSD port
> system, because mysteriously the hash of the tarball changed and
> people
> will get warnings about modified tarballs. It also has the effect
> that I
> am getting emails from people asking me if PHP.net was backdoored,
> because the MD5 hash changed....
> And if you want to change tarballs and don't change the version number
> (which is considered very bad by many people) then atleast WARN people
> about the modified tarball. A simple message: tarball was missing PEAR
> and was therefore rerolled is not so bad...

I'll add that to the 5.1.4 release message on the front page, that
was an oversight on my part.

>
>> The patches for security holes are usually in within a week, if you
>> want to fetch them you can do so either in a form of a PHP
>> snapshot of
>> a specific patch from CVS. To make releases every-time we get
>> security
>> fault is impractical.
> First the zend_hash_del() bug caused remote code execution in a
> bunch of
> popular PHP scripts. Secondly most open source projects release
> security
> bugfix releases. PHP.net on the other hand doesn't do this anymore.
> There are no security only fixes anymore. Instead we release not
> properly tested new versions of PHP that break tons of servers.
> (fastcgi
> ....)
>

"Tons" is a very quantitate number ;-), while fastcgi is definitely a
used SAPI, it is no where near  the usage of the Apache sapi or even
plain cgi. According to a basic Google Search mod_php is about 8
times more popular then CGI/FastCGI and of the 24,000 found phpinfo()
for the latter I'd wager no more then 1/2 actually use FastCGI.

> And well... I still see no PHP 4.4.3 on PHP.net... However we still
> offer the PHP 4.4.2 tarball (knowing that it has critical security
> holes). So either we release a security FIX release or we kick the
> tarball and declare PHP4 unsupported from now on.

PHP 4 is still supported, no one is suggesting that we discontinue
it. Derick can better comment on when he plans on making the release,
but it will definitely happen.

Ilia

Sebastian Bergmann

20 years ago

Ilia Alshanetsky wrote:

> The code in the release did not change on bit, the only change was the
> inclusion of the missing phar file, this hardly warrants 5.1.5 or even
> 5.1.4pl1. This will have no impact of people who have already downloaded
> and installed PHP, nor will this impact people who have yet to download
> PHP.

The checksum(s) changed, which is a problem for Gentoo Linux, for
instance.

--
Sebastian Bergmann                      http://www.sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Steph

20 years ago

Stefan,

Ironically

> after that incident another Zend man came forward and dares to say "I
> don't trust our core testers anymore"

He dared to say it because there's a QA mechanism in place that isn't
working - AKA a bunch of application developers testing Release Candidates
on their real-world applications rather than testing only against PHP's own
test suite. If any of them had done so, they would have picked up the
problem with user input arrays straight away. They evidently didn't, and
didn't.

I don't see what's so 'ironic' about mentioning that, and I certainly don't
think it's any kind of bad thing that Tony went on to request extra time
prior to 5.1.4's being tagged so that he could download a bunch of PHP
applications out there and run his own real-world tests.


- Steph

Andi Gutmans

20 years ago

Stefan,

I don't see why this attack is directed at Zend people working on
PHP, where the release process is completely a community driven
effort (and last time I checked, no enterprise was involved in that
process either).
I agree the release process isn't perfect yet, and it becomes
increasingly hard as PHP grows, but your points would be better made
if they were not directed against individual contributors but as an
email to raise general awareness and discussion.
Ilia for one works hard and does his best (probably better than any
release manager before him) to juggle between the various issues and
priorities of each release.
Andi


At 06:39 AM 5/15/2006, Stefan Esser wrote:

Stefan Esser

20 years ago

Hello Andi,

> I don't see why this attack is directed at Zend people working on PHP,
> where the release process is completely a community driven effort (and
> last time I checked, no enterprise was involved in that process either).

Well I don't see why Zend people commit code that obviously broke a key
functionality without testing it. And then Zend people come out under
the trees and speak up that our testers are doing bad jobs. It is true
that obviously NOONE has tested the code, because otherwise it would
have been seen instantly.

> Ilia for one works hard and does his best (probably better than any
> release manager before him) to juggle between the various issues and
> priorities of each release.

Andi, I was not speaking against Ilia. I know that he does a job that
actually noone really likes todo. The problems with 5.1.x releases have
to be blamed on other parties. Maybe you can say it was his fault that
this stupid .phar was not in the archive, but the bigger fault is the
lack of a sense of basic security principles. From my point of view it
is unbelievable how someone can add a 'wget pear.php.net/bla...' to the
'make install' process. Of course this is not triggered if the .phar
file is packaged, but this fallback is insane.

And Ilia has also nothing todo with the lack of a PHP 4.4.3 release.

Yours,
Stefan Esser

Andi Gutmans

20 years ago

At 09:32 AM 5/15/2006, Stefan Esser wrote:

>Hello Andi,
> > I don't see why this attack is directed at Zend people working on PHP,
> > where the release process is completely a community driven effort (and
> > last time I checked, no enterprise was involved in that process either).
>Well I don't see why Zend people commit code that obviously broke a key
>functionality without testing it. And then Zend people come out under
>the trees and speak up that our testers are doing bad jobs. It is true
>that obviously NOONE has tested the code, because otherwise it would
>have been seen instantly.

My point was that this has nothing to do with Zend or not Zend. It's
happened to all of us in the past (except for maybe you) where we
have commited code that had problems. It's counter-productive to
attack individuals and/or entities about this especially when
everyone's working towards the same goal of contributing to PHP.

So people screw up. I prefer having the occasional screw up then less
people helping out.

Andi

Stefan Esser

20 years ago

Hey Andi

> My point was that this has nothing to do with Zend or not Zend.

My point is not that someone from Zend broke it, but that someone from
Zend blamed the community that THEY failed to find the problem. I
thought Zend is enough into PHP to test their own products against RC's,
too. It makes me angry that in the last months again and again people
who earn their money with PHP come and insult those who work on PHP for
free and do all the hard work.

> It's happened to all of us in the past (except for maybe you) where we
> have commited code that had problems.

Hehe... I also commited code that cause problems in a release... Like a
very slow unserialize() in 4.3.??. And yes I tested it back then but
unfortunately that bug only was causing problems were large quantities
of data had to be unserialized.

> So people screw up. I prefer having the occasional screw up then less
> people helping out.

Yeah. But maybe we should setup some rules what exactly we should do
when there is a broken tarball or missing files. While it seems nice to
just repackage and silently change it (or change it with a warning), it
is considered bad practice and is especially bad for people using
distributions like Gentoo or the BSD ports system. Even if we don't
change the version number we could rename the tarball to
php-5.1.4-repack-1.tar.bz2 or something alike.

Antony Dovgal

20 years ago

On 15.05.2006 21:10, Stefan Esser wrote:

> Hey Andi
>> My point was that this has nothing to do with Zend or not Zend.
> My point is not that someone from Zend broke it, but that someone from
> Zend blamed the community that THEY failed to find the problem. I
> thought Zend is enough into PHP to test their own products against RC's,
> too. It makes me angry that in the last months again and again people
> who earn their money with PHP come and insult those who work on PHP for
> free and do all the hard work.

Oh, no.. You said it.

I get NO MONEY for testing PHP, fixing bugs, writing tests, helping users to solve their own problems @ bugs.php.net, maintaining several my own PECL modules etc. etc.
This is all my own "just for fun" job that I do in my spare time FOR FREE.

So it makes me a bit angry when someone who did nothing (except for a couple of mails to internals@) for PHP since December starts treating me and Dmitry (who's one of the most active PHP contributors) like a millionaires who earned their millions from poor PHP community.
Whether you want it or not, but we are the part of this community and yes, it was partly MY own fault that the bug has slipped in.

So please stop your trolling session and start writing new tests or at least doing something more productive than you're doing now.
Thank you.

--
Wbr,
Antony Dovgal

Sebastian Bergmann

20 years ago

Antony Dovgal wrote:

> So it makes me a bit angry when someone who did nothing (except for a
> couple of mails to internals@) for PHP since December starts treating me
> and Dmitry (who's one of the most active PHP contributors) like a
> millionaires who earned their millions from poor PHP community.

Now this is an unfair argument as Stefan cannot (for whatever reasons)
commit his work to cvs.php.net.

--
Sebastian Bergmann                      http://www.sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Sebastian Bergmann

20 years ago

Sebastian Bergmann wrote:

> Now this is an unfair argument as Stefan cannot (for whatever reasons)
> commit his work to cvs.php.net.

Strike that, I was educated about this on IRC just now. My initial
point is still valid to some degree, IMHO: just because Stefan's work
does not go into cvs.php.net does not mean that he is not helping to
make PHP better (more secure in his case).

--
Sebastian Bergmann                      http://www.sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Steph

20 years ago

> Sebastian Bergmann wrote:
>> Now this is an unfair argument as Stefan cannot (for whatever reasons)
>> commit his work to cvs.php.net.
>
> Strike that, I was educated about this on IRC just now. My initial
> point is still valid to some degree, IMHO: just because Stefan's work
> does not go into cvs.php.net does not mean that he is not helping to
> make PHP better (more secure in his case).

True, but it'd be far more useful to have Stefan's feedback at RC stage
rather than at post-release stage. We all know he's good at what he does.

- Steph

Stefan Esser

20 years ago

> So it makes me a bit angry when someone who did nothing (except for a
> couple of mails to internals@) for PHP since December starts treating
> me and Dmitry (who's one of the most active PHP contributors) like a
> millionaires who earned their millions from poor PHP community.

Tony, you have obviously no clue about my contributions to the PHP
community. Do you have any clue how many security holes I have reported
in the last months? Do you have any clue how many mails I had to write
to angry security researchers that got ignored by the rest of the PHP
Security Response Team.

Yours,
Stefan Esser

Zeev Suraski

20 years ago

Stefan,

If anything, that was a good drill on why it's not a good idea to
write sarcastic, negative emails to people.  Unless of course, your
aim is to annoy them into starting a heated threads.  You've raised
some good points in your original email, and it's a shame you diluted
your message by implying all sorts of things about Dmitry and
Tony.  Instead of discussing the points, we're discussing these
pointless topics of who contributes more.  I suggest we stop here, I
think it's absurd to question the level of contribution from any of you three.

Zeev

At 21:08 15/05/2006, Stefan Esser wrote:

Stefan Esser

20 years ago

Zeev,

> Instead of discussing the points, we're discussing these pointless
> topics of who contributes more.  I suggest we stop here, I think it's
> absurd to question the level of contribution from any of you three.

You are right. The discussion can stop here. Antony once again proved
every single word I wrote about him with his reaction.

Stefan

Zeev Suraski

20 years ago

At 21:29 15/05/2006, Stefan Esser wrote:

>Zeev,
> > Instead of discussing the points, we're discussing these pointless
> > topics of who contributes more.  I suggest we stop here, I think it's
> > absurd to question the level of contribution from any of you three.
>You are right. The discussion can stop here. Antony once again proved
>every single word I wrote about him with his reaction.

Stefan,

You've got to be kidding.  Just stop.

Zeev

Dmitry Stogov

20 years ago

> -----Original Message-----
> From: Stefan Esser [mailto:sesser@php.net]
> Sent: Monday, May 15, 2006 8:32 PM
> To: Andi Gutmans
> Cc: PHP internals
> Subject: Re: [PHP-DEV] PHP Release Process Sucks
>
>
> Hello Andi,
> > I don't see why this attack is directed at Zend people
> working on PHP,
> > where the release process is completely a community driven
> effort (and
> > last time I checked, no enterprise was involved in that process
> > either).
> Well I don't see why Zend people commit code that obviously
> broke a key functionality without testing it. And then Zend
> people come out under the trees and speak up that our testers
> are doing bad jobs. It is true that obviously NOONE has
> tested the code, because otherwise it would have been seen instantly.

Hi Stefan,

It was my bug.
I am writing a lot of code for PHP and as result do some bugs.
I don't know a man who never does bugs.

But this bug wouldn't go into CVS, if PHP had a test case for POST arrays.

Sorry for the bug.

Thanks. Dmitry.

Stefan Esser

20 years ago

Hello Dmitry,

> It was my bug.
> I am writing a lot of code for PHP and as result do some bugs.
> I don't know a man who never does bugs.
>

Exactly. I (we) appreciate your work and the point was not that you
broke it. Like I replied to Andi, I also broke unserialize() in one of
the 4.3 releases in a way that it still worked but was ultra-slow. What
made me angry was just the fact that Tony was acting like: I am big Tony
from Zend, let me test this, the community is obviously too dumb to test.

Stefan

> But this bug wouldn't go into CVS, if PHP had a test case for POST arrays.
>

So true ;)

Antony Dovgal

20 years ago

On 15.05.2006 21:17, Stefan Esser wrote:

> Hello Dmitry,
>> It was my bug.
>> I am writing a lot of code for PHP and as result do some bugs.
>> I don't know a man who never does bugs.
>>
> Exactly. I (we) appreciate your work and the point was not that you
> broke it. Like I replied to Andi, I also broke unserialize() in one of
> the 4.3 releases in a way that it still worked but was ultra-slow. What
> made me angry was just the fact that Tony was acting like: I am big Tony
> from Zend, let me test this, the community is obviously too dumb to test.

True.
Obviously nobody from core testers did it, so I [am-the-big-tony-from-zend] decided to spent a
night on tests and asked Ilia to wait with the release for at least several hours.

Do you think I got a $0.01 for this?
No, I got only your "I-hate-those-zenders" email.

Your help is priceless, thank you.

--
Wbr,
Antony Dovgal

Marcus Börger

20 years ago

Hello Stefan,

Monday, May 15, 2006, 6:32:25 PM, you wrote:

> Hello Andi,
>> I don't see why this attack is directed at Zend people working on PHP,
>> where the release process is completely a community driven effort (and
>> last time I checked, no enterprise was involved in that process either).
> Well I don't see why Zend people commit code that obviously broke a key
> functionality without testing it. And then Zend people come out under
> the trees and speak up that our testers are doing bad jobs. It is true
> that obviously NOONE has tested the code, because otherwise it would
> have been seen instantly.

To rephrase Andi "So people screw up. I prefer having the occasional
screw up then less people helping out." We are a community [...]
What we need is more helping hands and less comlaining notes. Actually
we are constantly working on increasing or QA efforts. And from my
point Ilia so far has done a very good job. A few minor mistakes so
far maybe.

marcus

Lukas Smith

20 years ago

Marcus Boerger wrote:

> To rephrase Andi "So people screw up. I prefer having the occasional
> screw up then less people helping out." We are a community [...]
> What we need is more helping hands and less comlaining notes. Actually
> we are constantly working on increasing or QA efforts. And from my
> point Ilia so far has done a very good job. A few minor mistakes so
> far maybe.

That being said .. if there are any additions people have to give better
guidelines for RM's please drop a comment at the following address:
http://oss.backendmedia.com/ReleaseChecklist

regards,
Lukas

Phil Driscoll

20 years ago

On Tuesday 16 May 2006 01:04, Lukas Smith wrote:

> That being said .. if there are any additions people have to give better
> guidelines for RM's please drop a comment at the following address:
> http://oss.backendmedia.com/ReleaseChecklist

I've just added the following comment, which addresses part of the problem:
--------------------
Somewhere between the RC and the release, the release manager should
receive email confirmation from a number of volunteers that the RC has
been tested not only with 'make test', but on real world applications such
as the volunteers' own code and high profile applications such as
phpMyAdmin, SquirrelMail and so on.

The RC should not become a release until such confirmation is received.
--------------------

I hereby volunteer to do my code, phpMyAdmin and SquirrelMail for php5 RCs.

Clearly, for this to have maximum impact we need to cover as many
applications, platforms and SAPIs as possible.

--
Phil Driscoll

Jasper Bryant-Greene

20 years ago

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Phil Driscoll wrote:

> On Tuesday 16 May 2006 01:04, Lukas Smith wrote:
>> That being said .. if there are any additions people have to give better
>> guidelines for RM's please drop a comment at the following address:
>> http://oss.backendmedia.com/ReleaseChecklist
>
> I've just added the following comment, which addresses part of the problem:
> --------------------
> Somewhere between the RC and the release, the release manager should
> receive email confirmation from a number of volunteers that the RC has
> been tested not only with 'make test', but on real world applications such
> as the volunteers' own code and high profile applications such as
> phpMyAdmin, SquirrelMail and so on.
>
> The RC should not become a release until such confirmation is received.
> --------------------
>
> I hereby volunteer to do my code, phpMyAdmin and SquirrelMail for php5 RCs.
>
> Clearly, for this to have maximum impact we need to cover as many
> applications, platforms and SAPIs as possible.

I like this idea. I can volunteer to do reasonably thorough testing
(both a 'make test' and real-world app testing) of 5.x releases on a
very large OO application running on Linux / Apache 2.2. It's not an
open-source app, but that doesn't mean I can't help with testing, does it?

- --
Jasper Bryant-Greene
General Manager
Album Limited

http://www.album.co.nz/     0800 4 ALBUM
jasper@album.co.nz          021 708 334
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (MingW32)

iD8DBQFEaXtmFfAeHhDzT4gRA55iAKCVTuNMLR4l+aCmp1D86FoNQX12qwCffAgW
HaAcyfx6XSZlISShJVveg8o=
=9zMN
-----END PGP SIGNATURE-----

Richard Lynch

20 years ago

On Tue, May 16, 2006 2:12 am, Jasper Bryant-Greene wrote:

> I like this idea. I can volunteer to do reasonably thorough testing
> (both a 'make test' and real-world app testing) of 5.x releases on a
> very large OO application running on Linux / Apache 2.2. It's not an
> open-source app, but that doesn't mean I can't help with testing, does
> it?

Certainly, running the tests and reporting any problems is VERY VERY
GOOD.

But...

Now don't take this wrong...

But I really don't think it would be fair/right of PHP Devs to commit
to NOT releasing a version until after a closed-source application
"works" with it.

If the problems you have are wide-spread and likely to affect many
users, you're all set anyway, right?

But if it's some weird thing that only affects your application...
Well, what should happen?

The rest of the world waits for your problem to get resolved?

Just don't sit right in my gut, personally.

I trust you see my viewpoint and don't take this as an affront to the
offer to test -- which *IS* needed and will benefit you.

--
Like Music?
http://l-i-e.com/artists.htm

Jasper Bryant-Greene

20 years ago

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Richard Lynch wrote:

> On Tue, May 16, 2006 2:12 am, Jasper Bryant-Greene wrote:
>> I like this idea. I can volunteer to do reasonably thorough testing
>> (both a 'make test' and real-world app testing) of 5.x releases on a
>> very large OO application running on Linux / Apache 2.2. It's not an
>> open-source app, but that doesn't mean I can't help with testing, does
>> it?

[snip]

> But I really don't think it would be fair/right of PHP Devs to commit
> to NOT releasing a version until after a closed-source application
> "works" with it.
>
> If the problems you have are wide-spread and likely to affect many
> users, you're all set anyway, right?
>
> But if it's some weird thing that only affects your application...
> Well, what should happen?

Well, in that situation, if the problem lies with PHP, then probably
something should be done about it...

The point is that the more applications PHP is being tested against, the
higher the chance of catching (possibly obscure) bugs/regressions.

I'm not necessarily suggesting that a release be delayed because *my*
application doesn't work, I'm just offering to test RCs against it and
report any bugs that it turns up. If the bug is found to be with PHP and
is sufficiently serious, then perhaps the release will be delayed. That
would be up to the devs.

I would also suggest that our application is a good candidate for RC
testing as uses most of PHP5's OO features, is E_ALL|E_STRICT clean (so
it's easier to spot problems in the error log :), and has virtually 100%
test-case coverage (so we can test quickly and efficiently).

- --
Jasper Bryant-Greene
General Manager
Album Limited

http://www.album.co.nz/     0800 4 ALBUM
jasper@album.co.nz          021 708 334
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (MingW32)

iD8DBQFEak75FfAeHhDzT4gRA8qUAJsFsZLPbAU+VH7HaEO6/BnXdnF4ZgCglxhl
Bu+EJ0kTs6+H36Y1GMKPBJs=
=JfcT
-----END PGP SIGNATURE-----

Lukas Smith

20 years ago

Jasper Bryant-Greene wrote:

> I'm not necessarily suggesting that a release be delayed because *my*
> application doesn't work, I'm just offering to test RCs against it and
> report any bugs that it turns up. If the bug is found to be with PHP and
> is sufficiently serious, then perhaps the release will be delayed. That
> would be up to the devs.
>
> I would also suggest that our application is a good candidate for RC
> testing as uses most of PHP5's OO features, is E_ALL|E_STRICT clean (so
> it's easier to spot problems in the error log :), and has virtually 100%
> test-case coverage (so we can test quickly and efficiently).

Maybe some background as to why this primary testers idea was brought
up. As Marcus points out we still needs tons more tests. However even
then we are probably never able to test every possible permutation of
code and data flowing through it so thats where this primary testers
thing came up.

We actually went out and mailed tons of projects and got feedback from a
few. These have already helped in a few cases, but not as often as we
would have hoped in spotting show stopper bugs in a release.

Here are the current lists:
http://oss.backendmedia.com/PhP4yz
http://oss.backendmedia.com/PhP5yz
http://oss.backendmedia.com/PhP6yz

This will now obviously be moved to the moderated read only primary
testers mailing list which Wez just setup:
primary-qa-tester@lists.php.net

Note: someone still needs to figure out how to set the reply-to so it
points to php-qa@lists.php.net

So the conclusion:
1) Yes, it would be very important that as many of the big time projects
try out the RC's and report back both success/failure. In the best case
along with an analysis of what PHP feature broke. We do not want to food
the RM with application specific bugs.

2) We still need to continuously expand the number of tests we have. You
can see the numbers of on the gcov.php.net site. This might be an
opportunity for bored students to step up. Along with that it would be
nice to increase the number of hardware/OS platforms we test on. This
might be an opportunity for some of the bigger corporate users to step up ;)

regards,
Lukas

Marcus Börger

20 years ago

Hello Phil,

Tuesday, May 16, 2006, 9:07:13 AM, you wrote:

> On Tuesday 16 May 2006 01:04, Lukas Smith wrote:
>> That being said .. if there are any additions people have to give better
>> guidelines for RM's please drop a comment at the following address:
>> http://oss.backendmedia.com/ReleaseChecklist

> I've just added the following comment, which addresses part of the problem:
> --------------------
> Somewhere between the RC and the release, the release manager should
> receive email confirmation from a number of volunteers that the RC has
> been tested not only with 'make test', but on real world applications such
> as the volunteers' own code and high profile applications such as
> phpMyAdmin, SquirrelMail and so on.

> The RC should not become a release until such confirmation is received.

Maybe we should even just repack the RC with a single change allowed, the
vrsion bump.


Best regards,
Marcus

Ilia A.

20 years ago

For every RC we already send an e-mail to about 12 projects asking
them to test their code against the release and let us know of any
issues that come up. If you want to be part of this group ask Lukas
to add you to the wiki page that can be found here: http://
oss.backendmedia.com/PhP5yz


On 16-May-06, at 3:07 AM, Phil Driscoll wrote:

> On Tuesday 16 May 2006 01:04, Lukas Smith wrote:
>> That being said .. if there are any additions people have to give
>> better
>> guidelines for RM's please drop a comment at the following address:
>> http://oss.backendmedia.com/ReleaseChecklist
>
> I've just added the following comment, which addresses part of the
> problem:
> --------------------
> Somewhere between the RC and the release, the release manager should
> receive email confirmation from a number of volunteers that the RC has
> been tested not only with 'make test', but on real world
> applications such
> as the volunteers' own code and high profile applications such as
> phpMyAdmin, SquirrelMail and so on.
>
> The RC should not become a release until such confirmation is
> received.
> --------------------
>
> I hereby volunteer to do my code, phpMyAdmin and SquirrelMail for
> php5 RCs.
>
> Clearly, for this to have maximum impact we need to cover as many
> applications, platforms and SAPIs as possible.
> --
> Phil Driscoll
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

Ilia Alshanetsky
Advanced Internet Designs Inc.
ilia@prohost.org

Phil Driscoll

20 years ago

On Tuesday 16 May 2006 13:51, Ilia Alshanetsky wrote:

> For every RC we already send an e-mail to about 12 projects asking
> them to test their code against the release and let us know of any
> issues that come up. If you want to be part of this group ask Lukas
> to add you to the wiki page that can be found here: http://
> oss.backendmedia.com/PhP5yz

I already get these via the php-qa list, however the improvement I want to
make to the process is to stop an RC becoming a release until positive
responses have been received from at least a reasonable proportion of a list
of people who have committed to do some testing.

--
Phil Driscoll

Sebastian Bergmann

20 years ago

Ilia Alshanetsky wrote:

> For every RC we already send an e-mail to about 12 projects asking them
> to test their code against the release and let us know of any issues
> that come up.

Sadly, this does not seem to work. Maybe because nobody from the
projects feels responsible?

--
Sebastian Bergmann                      http://www.sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Richard Lynch

20 years ago

On Tue, May 16, 2006 8:04 am, Sebastian Bergmann wrote:

> Ilia Alshanetsky wrote:
>> For every RC we already send an e-mail to about 12 projects asking
>> them
>> to test their code against the release and let us know of any issues
>> that come up.
>
>  Sadly, this does not seem to work. Maybe because nobody from the
>  projects feels responsible?

I would submit that:

They assume they only need to tell you if something is WRONG because
that is what they are told to do.  (See first sentence above)

That's quite different from:
We're waiting for a "green light" from your project so we know it's a
good solid release.

If you only ask to hear about problems, you don't hear a "200 OK" when
you need it. :-)

It's also possible that the PHP RC release cycle is too quick for them
to test, depending on their current availability, so it never gets
done on time...

Though obviously it's gotta be pretty high priority for them to be
sure it's all good.

Development of automated testing for these projects (by the projects
themselves, not PHP Devs) would go a long way towards the ideal of
catching bugs before release...

But it seems like PHP itself could use some work in this area to lead
by example. :-)

Just some random thoughts when I should be working...

--
Like Music?
http://l-i-e.com/artists.htm

Greg Beaver

20 years ago

Sebastian Bergmann wrote:

> Ilia Alshanetsky wrote:
>
>>For every RC we already send an e-mail to about 12 projects asking them
>>to test their code against the release and let us know of any issues
>>that come up.
>
>
>  Sadly, this does not seem to work. Maybe because nobody from the
>  projects feels responsible?

Geez tensions are running high.

I am the tester for PEAR, and I tested the PEAR Installer (I do not test
the gazillion packages that I don't maintain) and it worked great with
PHP 5.1.4.  PEAR doesn't use $_POST and I did not detect any errors.

In the past I have found at least 4 PHP bugs when testing PEAR with PHP.
The fact is, most of the bugs I find are found the instant they are
introduced as I use the CVS of PHP_5_X for my testing.  In fact the
system is working quite well from my end, but I can't vouch for other
projects.

Greg

bertrand Gugger

20 years ago

Bonjour,
Greg Beaver wrote:

> Sebastian Bergmann wrote:
>
>>Ilia Alshanetsky wrote:
>>
>>
>>>For every RC we already send an e-mail to about 12 projects asking them
>>>to test their code against the release and let us know of any issues
>>>that come up.
>>
>>
>> Sadly, this does not seem to work. Maybe because nobody from the
>> projects feels responsible?
>
>
> Geez tensions are running high.
>
> I am the tester for PEAR, and I tested the PEAR Installer (I do not test
> the gazillion packages that I don't maintain) and it worked great with
> PHP 5.1.4.  PEAR doesn't use $_POST and I did not detect any errors.
>
> In the past I have found at least 4 PHP bugs when testing PEAR with PHP.
>  The fact is, most of the bugs I find are found the instant they are
> introduced as I use the CVS of PHP_5_X for my testing.  In fact the
> system is working quite well from my end, but I can't vouch for other
> projects.
>
> Greg

I've also been working in the same area.

I would like to add that:
PEAR code is often not php5 E_STRICT compatible for the obvious reason
that we belongs to the real user area , meaning not running only on some
dream installations but real production machines with all the no-choices
that induces and furnishing some layers to practical cases.

But that is not the point here.

Even if we use some kind of fork of run-tests , we have a bunch of tests
(.phpt and others).
So by release process, it could be a good idea to run them as a whole on
different installations.
Yes, we would have to shut off E_STRICT for some of them :)
Outer the fact that some packages need some external means , I believe
it would not be so hard to implement.
History shows we would have trap some bloody stuff if not bugs.

No, no , Greg , it's cool discussion :)

--
toggg

Brian Moon

20 years ago

Ilia Alshanetsky wrote:

> For every RC we already send an e-mail to about 12 projects asking them
> to test their code against the release and let us know of any issues
> that come up. If you want to be part of this group ask Lukas to add you
> to the wiki page that can be found here: http://oss.backendmedia.com/PhP5yz

Could someone add php-testing@phorum.org to that list?  We will do our
best to test Phorum against any releases.

--

Brian Moon
-------------
http://dealnews.com/
Its good to be cheap =)

Richard Lynch

20 years ago

On Tue, May 16, 2006 2:07 am, Phil Driscoll wrote:

> On Tuesday 16 May 2006 01:04, Lukas Smith wrote:
>> That being said .. if there are any additions people have to give
>> better
>> guidelines for RM's please drop a comment at the following address:
>> http://oss.backendmedia.com/ReleaseChecklist
>
> I've just added the following comment, which addresses part of the
> problem:
> --------------------
> Somewhere between the RC and the release, the release manager should
> receive email confirmation from a number of volunteers that the RC has
> been tested not only with 'make test', but on real world applications
> such
> as the volunteers' own code and high profile applications such as
> phpMyAdmin, SquirrelMail and so on.
>
> The RC should not become a release until such confirmation is
> received.
> --------------------
>
> I hereby volunteer to do my code, phpMyAdmin and SquirrelMail for php5
> RCs.
>
> Clearly, for this to have maximum impact we need to cover as many
> applications, platforms and SAPIs as possible.

Would it be worth having a checklist of "must-have" software?

It's easy to see something big getting missed just because "Joe" is on
vacation or too busy to test, and everybody knows "Joe" is gonna test
it...

--
Like Music?
http://l-i-e.com/artists.htm

Phil Driscoll

20 years ago

On Tuesday 16 May 2006 21:35, Richard Lynch wrote:

> Would it be worth having a checklist of "must-have" software?
>
> It's easy to see something big getting missed just because "Joe" is on
> vacation or too busy to test, and everybody knows "Joe" is gonna test
> it...

Yes, so we need a list plus at least two volunteers for everything on the list
for php4 and php5 testing.

So, to start the list (with apologies to authors of important software I've
missed out!):

make test
phpMyAdmin
Phorum
phpBB
SquirrelMail
FUDForum

I guess as well as a software list, we need an important platform list so we
know that an RC has been exercised on at least:
Linux X86 Apache module
Linux X86_64 Apache module
BSD Apache module
Solaris Apache module
Windows Apache CGI
Windows IIS CGI (and perhaps ISAPI - is that OK these days?)
Something with Fast CGI

Please add and/subtract to/from the above lists. We don't want either list to
get too big, otherwise we'll never get enough volunteers to cover the
spectrum.

--
Phil Driscoll

Brian Moon

20 years ago

Phil Driscoll wrote:

> Please add and/subtract to/from the above lists.

MediaWiki
Mambo/Joombla/Nuke

Those are the big in their space.  We use MediaWiki daily in our company.

--

Brian Moon
-------------
http://dealnews.com/
Its good to be cheap =)

Brion Vibber

20 years ago

Brian Moon wrote:

> Phil Driscoll wrote:
>> Please add and/subtract to/from the above lists.
>
> MediaWiki

[snip]

>
> Those are the big in their space.  We use MediaWiki daily in our company.

We're already testers.

Note that MediaWiki uses few (if any) $_POST arrays, so we encountered no
problems with 5.1.3 release candidates.

-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

Markus Fischer

20 years ago

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Phil Driscoll wrote:

> On Tuesday 16 May 2006 21:35, Richard Lynch wrote:
>> Would it be worth having a checklist of "must-have" software?
>>
>> It's easy to see something big getting missed just because "Joe" is on
>> vacation or too busy to test, and everybody knows "Joe" is gonna test
>> it...
>
> Yes, so we need a list plus at least two volunteers for everything on the list
> for php4 and php5 testing.
>
> So, to start the list (with apologies to authors of important software I've
> missed out!):
>
> make test
> phpMyAdmin
> Phorum
> phpBB
> SquirrelMail
> FUDForum

Is there any software out there which uses PHPUnit(2)? So those
automatic suites could be used as well.

- - Markus
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEauE/1nS0RcInK9ARAnjBAJ9A0J0HtnNO+40TjRxjmyQn1o0iXACfc8eo
DIcgr2idhDImje+GpOa0o0Y=
=aAti
-----END PGP SIGNATURE-----

Sebastian Bergmann

20 years ago

Markus Fischer wrote:

> Is there any software out there which uses PHPUnit(2)? So those
> automatic suites could be used as well.

I have brought this up before. Sebastian Nohn has taken up this idea and
set up a box that checks out PHP from CVS, builds it, and then runs
"make test" as well as the unit tests of projects like PHPUnit and the
Zend Framework.

--
Sebastian Bergmann                      http://www.sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

Martin Jansen

20 years ago

On Tue May 16, 2006 at 09:5439PM +0100, Phil Driscoll wrote:

> I guess as well as a software list, we need an important platform list so we
> know that an RC has been exercised on at least:
> Linux X86 Apache module
> Linux X86_64 Apache module
> BSD Apache module
> Solaris Apache module
> Windows Apache CGI
> Windows IIS CGI (and perhaps ISAPI - is that OK these days?)
> Something with Fast CGI

OS X, preferably both on the Intel- and the PowerPC-platform.

- Martin

Brian Moon

20 years ago

> OS X, preferably both on the Intel- and the PowerPC-platform.

Oh yes.  I need to be able to develop on my MacBook Pro.

--

Brian Moon
-------------
http://dealnews.com/
Its good to be cheap =)

Phil Driscoll

20 years ago

On Wednesday 17 May 2006 16:47, Martin Jansen wrote:

> OS X, preferably both on the Intel- and the PowerPC-platform.

added to
http://www.dialsolutions.com/phil/php/primarytesters.html
:)

--
Phil Driscoll

Phil Driscoll

20 years ago

Lukas

To move this process forward, is there any chance that you could make a few
changes to http://oss.backendmedia.com/ReleaseChecklist?

First add an extra item to 'Getting the Release Candidate out there' saying:
Email primary-qa-tester@lists.php.net to notify the primary testers to get
testing. An example email is here:
-----------------------
Hello!

You are receiving this email because you have volunteered to be a php primary
tester. Can you please test the latest PHP release candidate on the platform
and with the applications you have volunteered to be responsible for (and
also if possible do a make test and allow it to post the results to the
php-qa list). For a reminder of what you volunteered to test, check the table
at http://www.dialsolutions.com/phil/php/primarytesters.html. Please report
back by <some date - maybe a week in the future> to
<whoever@is.realease.manager> and cc internals@lists.php.net, letting us
know what applications you tested, on what platforms and informing us of
success or failure.

Until we have had sufficient responses from our primary testers to be sure
that this RC performs well on a range of applications and platforms, we will
not go ahead with the release. Consequently, your input to this process is
very important. If you are unable for any reason to test this RC, please
notify whoever@is.realease.manager as soon as possible so that we can make
alternative arrangements this time.

If you do not want to receive these heads-up emails, please reply to me
personally and I will remove you from the list; but, we hope that you
want to actively help us making PHP a better and more stable tool.

The first release candidate of PHP N can be found at
http://downloads.php.net/somewhere/ .

If you know anyone else who would like to be a PHP primary tester, please let
me know.

regards,

<Whoever>
PHP <N> Release Manager
-----------------------

Then add a new item between 'Getting the Release Candidate out there' and
'Rolling a release' entitled 'Making sure that the Release Candidate is fit to
become a release':

1. Await responses from the primary testers, and issue a reminder email to
primary-qa-tester@lists.php.net if responses are slow to come in.

2. Keep an eye on the make test results for the RC posted to the php-qa list
looking out for any significant problems.

3. Check that problem-free responses have been received in respect of all the
critical applications and platforms listed (with their volunteers) at
http://www.dialsolutions.com/phil/php/primarytesters.html (which I am happy
to look after, but another wiki page may be more sensible), and if necessary
chase up individual volunteers for their results.



Please feel free to add/delete/amend as appropriate. Suggestions and
volunteers welcome :)

--
Phil Driscoll

Lukas Smith

20 years ago

Phil Driscoll wrote:

> To move this process forward, is there any chance that you could make a few
> changes to http://oss.backendmedia.com/ReleaseChecklist?

I have added a link to your post as a comment in the page. Since I am
not an RM I will leave it to Ilia and Derick to incorporate these ideas
into the document.

Aside from that this document could probably be moved to php-qa website
once its mature enough (and therefore less volatile).

I feel the todo lists are so dynamic and have a tendency to require some
simple markup that its better to leave them in a wiki. Obviously there
is still history maintained there. The only drawback is that for people
to get access I need to create a new local user instead of being able to
reuse the cvs.php.net authentication scheme (which as an outsider I
should of course not be allowed to interface with).

regards,
Lukas