Everything

php

Trapping "memory exhausted" error

php.internals

David Sklar

19 years ago

I am interested in being able to trap the (currently) fatal error that
results when memory usage exceeds the defined memory limit.

I was thinking it could work as follows:

- in addition to a memory_limit configuration directive, there could
be a "memory_limit_grace" configuration directive. This gets stored in
the struct _zend_mm_heap, along with the limit.

- Also added to struct _zend_mm_heap is a "initial limit reached" flag

- When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the
current conditions (memory_limit exceeded), it sets the "initial limit
reached" flag, adjusts the heap limit to the "memory_limit_grace"
value and throws some non-fatal error (or an exception if it is
feasible from here.)

- When zend_mm_safe_error() is invoked and the "initial limit reached"
flag is already set, it throws the fatal error exactly as it does now.

This "grace period" would provide a way to gracefully exit when a
memory limit is reached, but also has the hard limit still enforced so
that code which is supposed to be gracefully exiting doesn't chew up
too much additional memory.

Is it feasible to adjust the heap limit and throw a non-fatal error
from within zend_mm_safe_error()?

David

Stanislav Malyshev

19 years ago

> - in addition to a memory_limit configuration directive, there could
> be a "memory_limit_grace" configuration directive. This gets stored in
> the struct _zend_mm_heap, along with the limit.

That could be a problem because it's very hard to know exact memory
requirements for PHP code for most people not intimately knowing the
engine internals, and also recovering from failed allocation usually is
not simple since management structures could be in half-done state. So
writing useful handler for such an error would be rather hard to do.

Having said that, what exactly would you plan to do in this error handler?

--
Stanislav Malyshev, Zend Products Engineer
stas@zend.com  http://www.zend.com/

David Sklar

19 years ago

On 4/16/07, Stanislav Malyshev <stas@zend.com> wrote:

> > - in addition to a memory_limit configuration directive, there could
> > be a "memory_limit_grace" configuration directive. This gets stored in
> > the struct _zend_mm_heap, along with the limit.
>
> That could be a problem because it's very hard to know exact memory
> requirements for PHP code for most people not intimately knowing the
> engine internals, and also recovering from failed allocation usually is
> not simple since management structures could be in half-done state. So
> writing useful handler for such an error would be rather hard to do.
>
> Having said that, what exactly would you plan to do in this error handler?

In my current situation, bail out, but gracefully -- perhaps just send
a 302 to an error page and exit; maybe instead throw away the output
buffer and then directly output some friendly error text.

The idea would be not to recover from the failure and proceed with a
separate script execution path that requires a lot less memory, but to
have things fail more gracefully than just splatting a Fatal Error. So
being able to calculate with great accuracy the memory requirements of
particular bits of PHP code shouldn't be a prerequisite to making this
sort of thing useful.

That said, given the history of just about every other feature in PHP,
it wouldn't be surprising for people to attempt to use this feature to
do all sorts of crazy things after the soft memory limit has been
reached, and in that case they'd want to be able to calculate the
appropriate size of their grace period, but if they're going to go to
such contortions, then that's what they're stuck with.

David


Richard Lynch <ceo@l-i-e.com> wrote:

> You might also come at it from the other direction and detect/notify
> at some number smaller than the current hard limit, configurable in
> php.ini...
>
> This might play better with anything relying on the current behaviour

This is interesting and might actually take care of a lot of cases. I
don't know enough about the existing memory manager to know if it
would handle the case where a single new allocation would blow through
both the (lower) soft limit and the hard limit at the same time.

David

Stanislav Malyshev

19 years ago

> In my current situation, bail out, but gracefully -- perhaps just send
> a 302 to an error page and exit; maybe instead throw away the output
> buffer and then directly output some friendly error text.

I think you can do it with shutdown handler, shouldn't it work? I.e. you
can do it this way:
1. Install shutdown handler
2. Do something that takes a lot of memory
3. Memory is exhausted, script is terminated
4. The memory allocated by "something" is cleaned up, part by the
engine, part manually
5. Shutdown handler does something like output "sorry, better luck next
time" requiring minimal amount of memory.

--
Stanislav Malyshev, Zend Products Engineer
stas@zend.com  http://www.zend.com/

David Sklar

19 years ago

On 4/17/07, Stanislav Malyshev <stas@zend.com> wrote:

> > In my current situation, bail out, but gracefully -- perhaps just send
> > a 302 to an error page and exit; maybe instead throw away the output
> > buffer and then directly output some friendly error text.
>
> I think you can do it with shutdown handler, shouldn't it work? I.e. you
>   can do it this way:
> 1. Install shutdown handler
> 2. Do something that takes a lot of memory
> 3. Memory is exhausted, script is terminated
> 4. The memory allocated by "something" is cleaned up, part by the
> engine, part manually
> 5. Shutdown handler does something like output "sorry, better luck next
> time" requiring minimal amount of memory.

Ah, excellent, thanks. I'll take a look. This looks like what
http://ez.no/doc/components/view/latest/(file)/Execution/ezcExecution.html
does (Derick mentions this in the comments at
http://php100.wordpress.com/2007/04/16/graceful-recovery/)

David

Stanislav Malyshev

19 years ago

> Ah, excellent, thanks. I'll take a look. This looks like what
> http://ez.no/doc/components/view/latest/(file)/Execution/ezcExecution.html
> does (Derick mentions this in the comments at
> http://php100.wordpress.com/2007/04/16/graceful-recovery/)

Yes, that's basically where I've got the idea. Looks like that's the
best one could do in pure PHP.

--
Stanislav Malyshev, Zend Products Engineer
stas@zend.com  http://www.zend.com/

Richard Lynch

19 years ago

On Tue, April 17, 2007 3:16 am, David Sklar wrote:

> Richard Lynch <ceo@l-i-e.com> wrote:
>> You might also come at it from the other direction and detect/notify
>> at some number smaller than the current hard limit, configurable in
>> php.ini...
>>
>> This might play better with anything relying on the current
>> behaviour
>
> This is interesting and might actually take care of a lot of cases. I
> don't know enough about the existing memory manager to know if it
> would handle the case where a single new allocation would blow through
> both the (lower) soft limit and the hard limit at the same time.

If an allocation blows through the hard limit, it should behave
exactly as it does now, for BC, imho.

If you let the developer decide on a "soft limit" or a percentage of
the "hard limit" where some callback function happens, it's on their
heads if they do something silly that rips right past their selected
soft/hard limits.

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

David Sklar

19 years ago

On 4/17/07, Richard Lynch <ceo@l-i-e.com> wrote:

> On Tue, April 17, 2007 3:16 am, David Sklar wrote:
> > Richard Lynch <ceo@l-i-e.com> wrote:
> >> You might also come at it from the other direction and detect/notify
> >> at some number smaller than the current hard limit, configurable in
> >> php.ini...
> >>
> >> This might play better with anything relying on the current
> >> behaviour
> >
> > This is interesting and might actually take care of a lot of cases. I
> > don't know enough about the existing memory manager to know if it
> > would handle the case where a single new allocation would blow through
> > both the (lower) soft limit and the hard limit at the same time.
>
> If an allocation blows through the hard limit, it should behave
> exactly as it does now, for BC, imho.
>
> If you let the developer decide on a "soft limit" or a percentage of
> the "hard limit" where some callback function happens, it's on their
> heads if they do something silly that rips right past their selected
> soft/hard limits.

Well, the BC case could be handled by doing what you suggest if the
"grace limit" config option isn't defined.

I think it's important, in the case that the grace limit is set up, to
trigger its behavior even if a single allocation blows away both so
that the recovery can be graceful. For example, if you're retrieving a
feed to parse or the contents of a file and that feed or file is
unexpectedly large so it blows through both memory limits.

David

Richard Lynch

19 years ago

You might also come at it from the other direction and detect/notify
at some number smaller than the current hard limit, configurable in
php.ini...

This might play better with anything relying on the current behaviour.

On Mon, April 16, 2007 5:19 am, David Sklar wrote:

> I am interested in being able to trap the (currently) fatal error that
> results when memory usage exceeds the defined memory limit.
>
> I was thinking it could work as follows:
>
> - in addition to a memory_limit configuration directive, there could
> be a "memory_limit_grace" configuration directive. This gets stored in
> the struct _zend_mm_heap, along with the limit.
>
> - Also added to struct _zend_mm_heap is a "initial limit reached" flag
>
> - When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the
> current conditions (memory_limit exceeded), it sets the "initial limit
> reached" flag, adjusts the heap limit to the "memory_limit_grace"
> value and throws some non-fatal error (or an exception if it is
> feasible from here.)
>
> - When zend_mm_safe_error() is invoked and the "initial limit reached"
> flag is already set, it throws the fatal error exactly as it does now.
>
> This "grace period" would provide a way to gracefully exit when a
> memory limit is reached, but also has the hard limit still enforced so
> that code which is supposed to be gracefully exiting doesn't chew up
> too much additional memory.
>
> Is it feasible to adjust the heap limit and throw a non-fatal error
> from within zend_mm_safe_error()?
>
> David
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

Peter Hodge

19 years ago

Hello,

A soft memory limit could be very useful; it doesn't necessarily need to throw
an error, but it would provide an opportunity to free some memory and prevent
the script from crashing.  Perhaps something like:

// function to call when memory is running out
register_memory_limit_handler('freeSomeMemory');
function freeSomeMemory() {
$GLOBALS['someBigCache'] = null;
}

regards,
Peter



--- Richard Lynch <ceo@l-i-e.com> wrote:

> You might also come at it from the other direction and detect/notify
> at some number smaller than the current hard limit, configurable in
> php.ini...
>
> This might play better with anything relying on the current behaviour.
>
> On Mon, April 16, 2007 5:19 am, David Sklar wrote:
> > I am interested in being able to trap the (currently) fatal error that
> > results when memory usage exceeds the defined memory limit.
> >
> > I was thinking it could work as follows:
> >
> > - in addition to a memory_limit configuration directive, there could
> > be a "memory_limit_grace" configuration directive. This gets stored in
> > the struct _zend_mm_heap, along with the limit.
> >
> > - Also added to struct _zend_mm_heap is a "initial limit reached" flag
> >
> > - When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the
> > current conditions (memory_limit exceeded), it sets the "initial limit
> > reached" flag, adjusts the heap limit to the "memory_limit_grace"
> > value and throws some non-fatal error (or an exception if it is
> > feasible from here.)
> >
> > - When zend_mm_safe_error() is invoked and the "initial limit reached"
> > flag is already set, it throws the fatal error exactly as it does now.
> >
> > This "grace period" would provide a way to gracefully exit when a
> > memory limit is reached, but also has the hard limit still enforced so
> > that code which is supposed to be gracefully exiting doesn't chew up
> > too much additional memory.
> >
> > Is it feasible to adjust the heap limit and throw a non-fatal error
> > from within zend_mm_safe_error()?
> >
> > David
> >
> > --
> > PHP Internals - PHP Runtime Development Mailing List
> > To unsubscribe, visit: http://www.php.net/unsub.php
> >
> >
>
>
> --
> Some people have a "gift" link here.
> Know what I want?
> I want you to buy a CD from some indie artist.
> http://cdbaby.com/browse/from/lynch
> Yeah, I get a buck. So?
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

Send instant messages to your online friends http://au.messenger.yahoo.com

David Sklar

19 years ago

On 4/17/07, Peter Hodge <toomuchphp-phpdev@yahoo.com> wrote:

> Hello,
>
> A soft memory limit could be very useful; it doesn't necessarily need to throw
> an error, but it would provide an opportunity to free some memory and prevent
> the script from crashing.  Perhaps something like:
>
>   // function to call when memory is running out
>   register_memory_limit_handler('freeSomeMemory');
>   function freeSomeMemory() {
>     $GLOBALS['someBigCache'] = null;
>   }

My intention with the grace limit was to provide a graceful way of
failing, not to continue processing in a reduced state -- as Stanislav
said above, it could be difficult to do a lot after the allocation has
failed.

David

Adam Banko

19 years ago

2007. 04. 17, kedd keltezéssel 10.18-kor David Sklar ezt írta:

> My intention with the grace limit was to provide a graceful way of
> failing, not to continue processing in a reduced state -- as Stanislav
> said above, it could be difficult to do a lot after the allocation has
> failed.

What if the allocation didn't fail on the grace limit, return a new
memory as it should, but throw (set) an exception?

I'm not sure, but I think it very unlikely that throwing an exception
inside an emalloc would cause an inconsistent state. Maybe when the
emalloc was called from throw itself, I think but this case can be
handled.

Are there any other places where setting an exception would cause an
inconsistent state? What do you think?

Adam

Richard Lynch

19 years ago

On Tue, April 17, 2007 3:18 am, David Sklar wrote:

> On 4/17/07, Peter Hodge <toomuchphp-phpdev@yahoo.com> wrote:
>> Hello,
>>
>> A soft memory limit could be very useful; it doesn't necessarily
>> need to throw
>> an error, but it would provide an opportunity to free some memory
>> and prevent
>> the script from crashing.  Perhaps something like:
>>
>>   // function to call when memory is running out
>>   register_memory_limit_handler('freeSomeMemory');
>>   function freeSomeMemory() {
>>     $GLOBALS['someBigCache'] = null;
>>   }
>
> My intention with the grace limit was to provide a graceful way of
> failing, not to continue processing in a reduced state -- as Stanislav
> said above, it could be difficult to do a lot after the allocation has
> failed.

And we are suggesting that PHP Developers be provided with hooks to
anticipate failure BEFORE it happens, and to take pro-active measures
to see that there is no failure, rather than have some kind of
post-failure effort to do something.

For example, perhaps one could register a hook function to be called
when we are within 80% of the memory_limit setting, and another hook
if we reach 90%.

I presume one could just hack that with some kind of 'ticks' function
and memory_get_usage, but it probably wouldn't be nearly as efficient
as something more generic than the current hard failure in the engine.

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

Brian Shire

19 years ago

I have code at Facebook that currently does something similar by
catching fatal errors but there's a lot of gotcha's obviously that
need to be handled properly when the engine is in this state.  When
possible we enable the printing of backtraces and other debug
information, however you run the risk that instead of a nice
backtrace you get a nice "Segmentation fault" error instead.  This
can be wrapped up in an extension and is all C code functionality,
not exactly a soft memory limit so it's a little different than
what's being discussed, but it might be another option to what you're
looking for.  I'm happy to share with a few initially who would find
it useful/give feedback.

-shire

On Apr 16, 2007, at 3:19 AM, David Sklar wrote: