ref fix revisited

php.internals

Rasmus Lerdorf

20 years ago
Guys, could we take a look at making the ref to temp var fix a bit narrower? Currently we try to catch it at call-time. This means that something like: current(explode(' ','a b')) as per bug #34468 doesn't work. Now, I think there is a secondary bug here. I see no reason for current() to take a by-ref, so this particular one could be easily fixed. But there are many other cases where a function legitimately takes a by-ref and doesn't necessarily write to it or the write is a secondary action not required for the code to work. Could we not catch this on the write instead of on the call? The memory problem happens on the write. Or perhaps better, an E_NOTICE or E_STRICT on the call and an E_FATAL on the write. The current E_FATAL on the call seems out of whack. Gallery, for example, broke in a rather subtle way in their gallery_remote2.php script which meant the various client-side tools like iphototogallery and others got a cryptic "no album at URL" error message. I had to break out ethereal to track it down to a couple of functions where read-only function args were marked as by-ref. So they didn't actually have a memory corruption problem yet the E_FATAL killed them. SquirrelMail has code like this all over the place: $value = strtolower(array_shift(split('/\w/',trim($value)))); Here array_shift() does of course change the arg, so that is a potential problem. And yes, that's a dumb way to do this, but people write code like this. In some of these array manipulation calls, which seems to account for a number of the BC problems we are having, we could check for a non-ref and behave slightly differently. In the case of array_shift() we could return the first arg and throw a notice. Same would go for reset(), end(), next(), prev() and probably a few others. -Rasmus

Zeev Suraski

20 years ago
At 10:57 12/09/2005, Rasmus Lerdorf wrote:
>Guys, could we take a look at making the ref to temp var fix a bit >narrower? Currently we try to catch it at call-time. This means that >something like: > > current(explode(' ','a b')) > >as per bug #34468 doesn't work. Now, I think there is a secondary bug >here. I see no reason for current() to take a by-ref, so this >particular one could be easily fixed.
Yep, we need to fix that one.
> But there are many other cases >where a function legitimately takes a by-ref and doesn't necessarily >write to it or the write is a secondary action not required for the code >to work. Could we not catch this on the write instead of on the call?
The problem is that there's no way to tell that element apart at that time. It's too late. As soon as we treat a read-only zval as if it's read/write (take a ** instead of a *), it's too late, since we can't really detect later on where it came from.
>The memory problem happens on the write. Or perhaps better, an E_NOTICE >or E_STRICT on the call and an E_FATAL on the write. The current >E_FATAL on the call seems out of whack.
I don't really agree that it's out of whack, since you are passing a piece of data by reference, which is an undefined behavior. I agree that it would have been nice if we could allow for this and only complain if the data is written to in the function (in the PHP spirit of 'just work!'), but I don't see how that would be possible.
>Gallery, for example, broke in a rather subtle way in their >gallery_remote2.php script which meant the various client-side tools >like iphototogallery and others got a cryptic "no album at URL" error >message. I had to break out ethereal to track it down to a couple of >functions where read-only function args were marked as by-ref. So they >didn't actually have a memory corruption problem yet the E_FATAL killed >them.
You'd have to agree that it is a bug on Gallery's part, though, right? If they're read only, they shouldn't have been marked as by-ref (yes, we screwed up by only introducing this error now after it worked for years, but it's still problematic to let it go on working and create possible corruption).
>SquirrelMail has code like this all over the place: > > $value = strtolower(array_shift(split('/\w/',trim($value)))); > >Here array_shift() does of course change the arg, so that is a potential >problem. And yes, that's a dumb way to do this, but people write code >like this. In some of these array manipulation calls, which seems to >account for a number of the BC problems we are having, we could check >for a non-ref and behave slightly differently. In the case of >array_shift() we could return the first arg and throw a notice. Same >would go for reset(), end(), next(), prev() and probably a few others.
We could probably provide a way for internal functions to denote that they're handling by-ref 'wannabe' arguments, so that we won't generate a fatal error for them. We'd still have to do it for userland functions (and any other internal function). I'm not sure if it's worth it..? Zeev

Derick Rethans

20 years ago
On Mon, 12 Sep 2005, Zeev Suraski wrote:
> I don't really agree that it's out of whack, since you are passing a piece of > data by reference, which is an undefined behavior. I agree that it would have > been nice if we could allow for this and only complain if the data is written > to in the function (in the PHP spirit of 'just work!'), but I don't see how > that would be possible.
Works for PHP 4.4 just fine with a notice... Derick
-- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org

Rasmus Lerdorf

20 years ago
Zeev Suraski wrote:
>> But there are many other cases >> where a function legitimately takes a by-ref and doesn't necessarily >> write to it or the write is a secondary action not required for the code >> to work. Could we not catch this on the write instead of on the call? > > The problem is that there's no way to tell that element apart at that > time. It's too late. As soon as we treat a read-only zval as if it's > read/write (take a ** instead of a *), it's too late, since we can't > really detect later on where it came from.
Since we can detect a ref to a temp var at call time now, how about making a local copy in the function and effectively treat it as a by-value arg and also issue an E_NOTICE or an E_STRICT? I think that would solve the corruption problem without breaking BC. -Rasmus

Derick Rethans

20 years ago
On Tue, 13 Sep 2005, Rasmus Lerdorf wrote:
> Zeev Suraski wrote: > >> But there are many other cases > >> where a function legitimately takes a by-ref and doesn't necessarily > >> write to it or the write is a secondary action not required for the code > >> to work. Could we not catch this on the write instead of on the call? > > > > The problem is that there's no way to tell that element apart at that > > time. It's too late. As soon as we treat a read-only zval as if it's > > read/write (take a ** instead of a *), it's too late, since we can't > > really detect later on where it came from. > > Since we can detect a ref to a temp var at call time now, how about > making a local copy in the function and effectively treat it as a > by-value arg and also issue an E_NOTICE or an E_STRICT? I think that > would solve the corruption problem without breaking BC.
That's exactly what we do in 4.4. Derick
-- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org

Andreas Ahlenstorf

20 years ago
Am 14.09.2005 um 14:36 schrieb Derick Rethans:
>> Since we can detect a ref to a temp var at call time now, how about >> making a local copy in the function and effectively treat it as a >> by-value arg and also issue an E_NOTICE or an E_STRICT? I think that >> would solve the corruption problem without breaking BC. >> > > That's exactly what we do in 4.4.
array_pop(explode('.', $name))); raises Fatal Error in PHP 4.4.0 on FreeBSD => App broken. A.

Derick Rethans

20 years ago
On Wed, 14 Sep 2005, Andreas Ahlenstorf wrote:
> > Am 14.09.2005 um 14:36 schrieb Derick Rethans: > > > > Since we can detect a ref to a temp var at call time now, how about > > > making a local copy in the function and effectively treat it as a > > > by-value arg and also issue an E_NOTICE or an E_STRICT? I think that > > > would solve the corruption problem without breaking BC. > > > > > > > That's exactly what we do in 4.4. > > array_pop(explode('.', $name))); raises Fatal Error in PHP 4.4.0 on FreeBSD => > App broken.
Of course it does, you have a ) too much. Besides this, the following script does not make a fatal error: <?php $name = "derick"; array_pop(explode('.', $name)); ?> Derick
-- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org

Andreas Ahlenstorf

20 years ago
Am 14.09.2005 um 15:10 schrieb Derick Rethans:
> Of course it does, you have a ) too much. Besides this, the following
I should drink more coffee...
> script does not make a fatal error: > > <?php > $name = "derick"; > array_pop(explode('.', $name)); > ?>
Fatal error: Only variables can be passed by reference in /home/www/ core/media.class.php on line 221 Line 221 is this one: $filename = sprintf('%s.%s', md5(uniqid(rand(), true)), array_pop (explode('.', $name))); This replacement fixes the error: $_explode_tmp_result = explode('.', $name); $filename = sprintf('%s.%s', md5(uniqid(rand(), true)), array_pop ($_explode_tmp_result)); A.

Derick Rethans

20 years ago
On Wed, 14 Sep 2005, Andreas Ahlenstorf wrote:
> Fatal error: Only variables can be passed by reference in /home/www/ > core/media.class.php on line 221
The following script works fine too: <?php $filename = sprintf('%s.%s', md5(uniqid(rand(), true)), array_pop(explode('.', $name))); echo $filename; ?> echos: derick@kossu:~$ php-4.4dev -derror_reporting=2047 /tmp/test.php Notice: Undefined variable: name in /tmp/test.php on line 2 fb34401012d0b75111d002206cb8d349. Derick
-- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org

Andreas Ahlenstorf

20 years ago
Am 14.09.2005 um 15:40 schrieb Derick Rethans:
> <?php > $filename = sprintf('%s.%s', md5(uniqid(rand(), true)), array_pop > (explode('.', $name))); > echo $filename;
D'oh - I see it. My ISP changed the PHP version once again. It's now PHP 5.0.5. I'm very very sorry for the confusion. A.

Zeev Suraski

20 years ago
At 15:36 14/09/2005, Derick Rethans wrote:
>On Tue, 13 Sep 2005, Rasmus Lerdorf wrote: > > > Zeev Suraski wrote: > > >> But there are many other cases > > >> where a function legitimately takes a by-ref and doesn't necessarily > > >> write to it or the write is a secondary action not required for the code > > >> to work. Could we not catch this on the write instead of on the call? > > > > > > The problem is that there's no way to tell that element apart at that > > > time. It's too late. As soon as we treat a read-only zval as if it's > > > read/write (take a ** instead of a *), it's too late, since we can't > > > really detect later on where it came from. > > > > Since we can detect a ref to a temp var at call time now, how about > > making a local copy in the function and effectively treat it as a > > by-value arg and also issue an E_NOTICE or an E_STRICT? I think that > > would solve the corruption problem without breaking BC. > >That's exactly what we do in 4.4.
Except it is reported to break BC still.. (beyond the notices)? Zeev

Edin Kadribasic

20 years ago
Zeev Suraski wrote:
> At 15:36 14/09/2005, Derick Rethans wrote: > >> On Tue, 13 Sep 2005, Rasmus Lerdorf wrote: >> >> > Zeev Suraski wrote: >> > >> But there are many other cases >> > >> where a function legitimately takes a by-ref and doesn't necessarily >> > >> write to it or the write is a secondary action not required for >> the code >> > >> to work. Could we not catch this on the write instead of on the >> call? >> > > >> > > The problem is that there's no way to tell that element apart at that >> > > time. It's too late. As soon as we treat a read-only zval as if >> it's >> > > read/write (take a ** instead of a *), it's too late, since we can't >> > > really detect later on where it came from. >> > >> > Since we can detect a ref to a temp var at call time now, how about >> > making a local copy in the function and effectively treat it as a >> > by-value arg and also issue an E_NOTICE or an E_STRICT? I think that >> > would solve the corruption problem without breaking BC. >> >> That's exactly what we do in 4.4. > > > Except it is reported to break BC still.. (beyond the notices)?
The biggest BC break occured in 5.0.4->5.0.5 where some of these referrence issues cause a fatal error, not a mere notice. Edin

Zeev Suraski

20 years ago
At 15:54 14/09/2005, Edin Kadribasic wrote:
>Zeev Suraski wrote: > > At 15:36 14/09/2005, Derick Rethans wrote: > > > >> On Tue, 13 Sep 2005, Rasmus Lerdorf wrote: > >> > >> > Zeev Suraski wrote: > >> > >> But there are many other cases > >> > >> where a function legitimately takes a by-ref and doesn't necessarily > >> > >> write to it or the write is a secondary action not required for > >> the code > >> > >> to work. Could we not catch this on the write instead of on the > >> call? > >> > > > >> > > The problem is that there's no way to tell that element apart at that > >> > > time. It's too late. As soon as we treat a read-only zval as if > >> it's > >> > > read/write (take a ** instead of a *), it's too late, since we can't > >> > > really detect later on where it came from. > >> > > >> > Since we can detect a ref to a temp var at call time now, how about > >> > making a local copy in the function and effectively treat it as a > >> > by-value arg and also issue an E_NOTICE or an E_STRICT? I think that > >> > would solve the corruption problem without breaking BC. > >> > >> That's exactly what we do in 4.4. > > > > > > Except it is reported to break BC still.. (beyond the notices)? > >The biggest BC break occured in 5.0.4->5.0.5 where some of these >referrence issues cause a fatal error, not a mere notice.
I think it's probably equally big, but it's besides the point. I'm trying to understand what we can do (in 5.0.6 or whatever) that will really solve the problem. The 4.4 approach at this point doesn't appear to be any better since regardless of error messages, it makes PHP behave differently than it did before. Zeev

Pierre Joye

20 years ago
On 9/14/05, Zeev Suraski <zeev@zend.com> wrote:
> I think it's probably equally big, but it's besides the point. I'm trying > to understand what we can do (in 5.0.6 or whatever) that will really solve > the problem. The 4.4 approach at this point doesn't appear to be any > better since regardless of error messages, it makes PHP behave differently > than it did before.
I'm not sure what we should do, a 5.0.6 without the fix at all is not acceptable for many people, I can live without though. A 5.0.6 with the fixes (ref and sec) but only notices. For 5.1.0, if we really communicate about the breaks, I see no problem to leave it as it is now. There is many new things and some other fixes can introduce bugs as well (is_a, but well fixed by instanceof). And I do not want to wait one more week to get 5.1.0 :) --Pierre

Rasmus Lerdorf

20 years ago
Zeev Suraski wrote:
> At 15:36 14/09/2005, Derick Rethans wrote: > >> On Tue, 13 Sep 2005, Rasmus Lerdorf wrote: >> >> > Zeev Suraski wrote: >> > >> But there are many other cases >> > >> where a function legitimately takes a by-ref and doesn't necessarily >> > >> write to it or the write is a secondary action not required for >> the code >> > >> to work. Could we not catch this on the write instead of on the >> call? >> > > >> > > The problem is that there's no way to tell that element apart at that >> > > time. It's too late. As soon as we treat a read-only zval as if >> it's >> > > read/write (take a ** instead of a *), it's too late, since we can't >> > > really detect later on where it came from. >> > >> > Since we can detect a ref to a temp var at call time now, how about >> > making a local copy in the function and effectively treat it as a >> > by-value arg and also issue an E_NOTICE or an E_STRICT? I think that >> > would solve the corruption problem without breaking BC. >> >> That's exactly what we do in 4.4. > > Except it is reported to break BC still.. (beyond the notices)?
But do we have an example of this breakage? If there is real breakage, we need to address that. -Rasmus

Zeev Suraski

20 years ago
At 18:07 12/09/2005, Derick Rethans wrote:
>On Mon, 12 Sep 2005, Zeev Suraski wrote: > > > I don't really agree that it's out of whack, since you are passing a > piece of > > data by reference, which is an undefined behavior. I agree that it > would have > > been nice if we could allow for this and only complain if the data is > written > > to in the function (in the PHP spirit of 'just work!'), but I don't see how > > that would be possible. > >Works for PHP 4.4 just fine with a notice...
Of course we could switch to a NOTICE, but then we're still risking memory corruption. I was saying that I don't see how it's possible to warn 'JIT', i.e. accept the pass-by-ref, and complain only if the function actually tries to write to it. Zeev

Zeev Suraski

20 years ago
At 21:22 13/09/2005, Rasmus Lerdorf wrote:
>Zeev Suraski wrote: > >> But there are many other cases > >> where a function legitimately takes a by-ref and doesn't necessarily > >> write to it or the write is a secondary action not required for the code > >> to work. Could we not catch this on the write instead of on the call? > > > > The problem is that there's no way to tell that element apart at that > > time. It's too late. As soon as we treat a read-only zval as if it's > > read/write (take a ** instead of a *), it's too late, since we can't > > really detect later on where it came from. > >Since we can detect a ref to a temp var at call time now, how about >making a local copy in the function and effectively treat it as a >by-value arg and also issue an E_NOTICE or an E_STRICT? I think that >would solve the corruption problem without breaking BC.
Worth thinking about, but if it's possible it's going to be very messy. But still worth thinking about. Zeev

Pierre Joye

20 years ago
Hi Rasmus, You start a new thread for my question, maybe you will get more answers or at least better ones. If it is a reference or not, I do not care, it should not "act" differently. Understand that a quiet code in 5.0.4 must be quiet in 5.0.5. If we add notices in 5.1.0 and then make them fatal in 5.2.0, I'm fine. People will have time to migrate and "fix" their code. Regards, --Pierre

Rasmus Lerdorf

20 years ago
Just got back from Sri Lanka and am catching up on email. But in general I don't agree with your statement that things that are quiet in 5.0.4 must be quiet in 5.0.5. Quietly corrupting memory doesn't really work. Sometimes we need to break things slightly to fix things. My message was more about if there was a better way to fix it. I looked at the code and couldn't tell if it was somehow possible to catch this on the write instead of on the call. It sounds like that isn't possible. The second approach to try to regain some backwards compatibility is to make some of our internal functions aware of the fact that they are being passed a ref to a temp var and if so pretend it wasn't passed by reference. -Rasmus Pierre Joye wrote:

Pierre Joye

20 years ago
On 9/13/05, Rasmus Lerdorf <rasmus@lerdorf.com> wrote:
> Just got back from Sri Lanka and am catching up on email. But in > general I don't agree with your statement that things that are quiet in > 5.0.4 must be quiet in 5.0.5. Quietly corrupting memory doesn't really > work. Sometimes we need to break things slightly to fix things.
I agree about the fix, not about the fatal error in 5.0.5. See my answer in my post.
> My message was more about if there was a better way to fix it. I looked at > the code and couldn't tell if it was somehow possible to catch this on > the write instead of on the call. It sounds like that isn't possible. > The second approach to try to regain some backwards compatibility is to > make some of our internal functions aware of the fact that they are > being passed a ref to a temp var and if so pretend it wasn't passed by > reference.
As it is too late now anyway, regain BC makes little sense. Unless we restore it all related BC breaks and fix the internal temp vars management (which is equal to a rewrite of this part). From a user point of view, it will only end to the same confusion as in arguments order, or other little annoyances ;) Regards, --Pierre

Rasmus Lerdorf

20 years ago
Pierre Joye wrote:
> As it is too late now anyway, regain BC makes little sense.
I don't agree. This is one of the main things stopping people from migrating to PHP5 right now. If we can remove a bit of the migration pain with some of clever temp var handling, I think it is worthwhile. -Rasmus

Sönke Ruempler

20 years ago
Hi, Rasmus Lerdorf <mailto:rasmus@lerdorf.com> wrote on Tuesday, September 13, 2005 3:08 PM:
>> As it is too late now anyway, regain BC makes little sense. > > I don't agree. This is one of the main things stopping people from > migrating to PHP5 right now. If we can remove a bit of the migration > pain with some of clever temp var handling, I think it is worthwhile.
As Pierre, I can't upgrade my servers to 5.0.5. And as I noted - before the rollout - there are many widely spread applications like mediawiki that now have fatal errors :-/ An 5.0.5p1 with warnings/notices and pass-by-value would be nice. So everyone can upgrade with no/less fear that his application breaks - and gets advantage of the security fixe(s). Best Regards.

Andreas Korthaus

20 years ago
Rasmus Lerdorf wrote:
> SquirrelMail has code like this all over the place: > > $value = strtolower(array_shift(split('/\w/',trim($value)))); > > Here array_shift() does of course change the arg, so that is a potential > problem. And yes, that's a dumb way to do this, but people write code > like this. In some of these array manipulation calls, which seems to > account for a number of the BC problems we are having, we could check > for a non-ref and behave slightly differently. In the case of > array_shift() we could return the first arg and throw a notice. Same > would go for reset(), end(), next(), prev() and probably a few others.
I noticed the same problem with array_pop(). I'm happy with the new fatal error for user-space functions. But nobody - at least non-devs - knows that functions like array_pop() may not be used the way you described here anymore. It's not documented somewhere and it's hard to understand why. Many PHP programmers put more than one function in a row (I know it's bad style), so there a lot of such code out there. Would be great if that behaviour could be fixed. What I don't understand - upgrading from 5.0.4 to 5.0.5 broke by far more applications on my servers, than upgrading from 4.x to 5.x! And there was no information in the News/ChangeLog. For an admin it looked like a small, non-BC-braking security update which should be installed as soon as possible, without any risk to break scripts working with PHP 5.0.4. And in 5.0.5 it's fatal error, not only notice. best regards Andreas

Marcus Börger

20 years ago
Hello Rasmus, my solution was (and i proposed a patch for it here) that we have a way to allow function signatures that pass variables as const just like c++ allows. This i did because first it is faster and second it applies to most pass by ref signatures. Or in other words we could get rid of most pass by ref sigs. And just to note, it was declined because it would increase complexity a tight bit. best regards marcus Monday, September 12, 2005, 9:57:24 AM, you wrote:
> Guys, could we take a look at making the ref to temp var fix a bit > narrower? Currently we try to catch it at call-time. This means that > something like:
> current(explode(' ','a b'))
> as per bug #34468 doesn't work. Now, I think there is a secondary bug > here. I see no reason for current() to take a by-ref, so this > particular one could be easily fixed. But there are many other cases > where a function legitimately takes a by-ref and doesn't necessarily > write to it or the write is a secondary action not required for the code > to work. Could we not catch this on the write instead of on the call? > The memory problem happens on the write. Or perhaps better, an E_NOTICE > or E_STRICT on the call and an E_FATAL on the write. The current > E_FATAL on the call seems out of whack.
> Gallery, for example, broke in a rather subtle way in their > gallery_remote2.php script which meant the various client-side tools > like iphototogallery and others got a cryptic "no album at URL" error > message. I had to break out ethereal to track it down to a couple of > functions where read-only function args were marked as by-ref. So they > didn't actually have a memory corruption problem yet the E_FATAL killed > them.
> SquirrelMail has code like this all over the place:
> $value = strtolower(array_shift(split('/\w/',trim($value))));
> Here array_shift() does of course change the arg, so that is a potential > problem. And yes, that's a dumb way to do this, but people write code > like this. In some of these array manipulation calls, which seems to > account for a number of the BC problems we are having, we could check > for a non-ref and behave slightly differently. In the case of > array_shift() we could return the first arg and throw a notice. Same > would go for reset(), end(), next(), prev() and probably a few others.
> -Rasmus
Best regards, Marcus