Everything

php

Home » Project » Newsgroups » php.internals » drastic memory consumption with a sequence of exceptions »

drastic memory consumption with a sequence of exceptions

php.internals

Ron Korving

21 years ago

I've noticed that in a script of mine, memory consumption can explode quite
drastically when Exceptions are thrown, opposed to very normal memory
consumption when exceptions are not thrown.

This is the idea:

<?
class Obj
{
function process($i)
{
// do a lot of stuff, occupy a lot of memory within the scope of this
function
throw new Exception("error");
}
}

$obj = new Obj();
$errors = array();

for ($i=0; $i < 60; $i++)
{
try
{
$obj->process();
}
catch (Exception $e)
{
$errors[] = $e->getMessage();
}
}
?>

When exceptions are not thrown, memory consumption is normal. The method's
local vars are freed and not much more memory is used in the 50th pass
compared to the 1st. But when exceptions are thrown, memory consumption
increases quite a bit every pass. In the end (within the 60 passes of my
loop) over 8 MB's are used.

Of course, I can unset a lot of data before throwing exceptions to free up
memory and the problem will be solved. But it would be nicer if the cleanup
would happen as an exception is thrown.

Can this be considered a bug or is this behavior known and accepted?

Ron

Ron Korving

21 years ago

It is very likely that this has something to do with DOM. It's a huge piece
of code though, making it hard to test where the problem really lies.

""Ron Korving"" <r.korving@xit.nl> schreef in bericht
news:BD.4F.04646.AA6C8F24@pb1.pair.com...

> I've noticed that in a script of mine, memory consumption can explode

quite

> drastically when Exceptions are thrown, opposed to very normal memory
> consumption when exceptions are not thrown.
>
> This is the idea:
>
> <?
> class Obj
> {
>   function process($i)
>   {
>     // do a lot of stuff, occupy a lot of memory within the scope of this
> function
>     throw new Exception("error");
>   }
> }
>
> $obj = new Obj();
> $errors = array();
>
> for ($i=0; $i < 60; $i++)
> {
>   try
>   {
>     $obj->process();
>   }
>   catch (Exception $e)
>   {
>     $errors[] = $e->getMessage();
>   }
> }
> ?>
>
> When exceptions are not thrown, memory consumption is normal. The method's
> local vars are freed and not much more memory is used in the 50th pass
> compared to the 1st. But when exceptions are thrown, memory consumption
> increases quite a bit every pass. In the end (within the 60 passes of my
> loop) over 8 MB's are used.
>
> Of course, I can unset a lot of data before throwing exceptions to free up
> memory and the problem will be solved. But it would be nicer if the

cleanup

Ron Korving

21 years ago

The problem occurs in this particular situation:

public function process($data)
{
$largeResultSet = $this->xmlParser->parseString($data);

foreach ($largeResultSet as &$item)
{
throw new Exception("error");
}
}

It only happens if the exception is thrown from within the foreach(). It
seems the $largeResultSet is not freed in this case. Using the foreach by
reference or not (&$item or $item) does not make a difference.

Ron


""Ron Korving"" <r.korving@xit.nl> schreef in bericht
news:21.AF.04646.20AC8F24@pb1.pair.com...

> It is very likely that this has something to do with DOM. It's a huge

piece

> of code though, making it hard to test where the problem really lies.
>
> ""Ron Korving"" <r.korving@xit.nl> schreef in bericht
> news:BD.4F.04646.AA6C8F24@pb1.pair.com...
> > I've noticed that in a script of mine, memory consumption can explode
> quite
> > drastically when Exceptions are thrown, opposed to very normal memory
> > consumption when exceptions are not thrown.
> >
> > This is the idea:
> >
> > <?
> > class Obj
> > {
> >   function process($i)
> >   {
> >     // do a lot of stuff, occupy a lot of memory within the scope of

this

> > function
> >     throw new Exception("error");
> >   }
> > }
> >
> > $obj = new Obj();
> > $errors = array();
> >
> > for ($i=0; $i < 60; $i++)
> > {
> >   try
> >   {
> >     $obj->process();
> >   }
> >   catch (Exception $e)
> >   {
> >     $errors[] = $e->getMessage();
> >   }
> > }
> > ?>
> >
> > When exceptions are not thrown, memory consumption is normal. The

method's

> > local vars are freed and not much more memory is used in the 50th pass
> > compared to the 1st. But when exceptions are thrown, memory consumption
> > increases quite a bit every pass. In the end (within the 60 passes of my
> > loop) over 8 MB's are used.
> >
> > Of course, I can unset a lot of data before throwing exceptions to free

up

Andi Gutmans

21 years ago

Can you send a complete script which also passes $data that reproduces the
problem? Thanks.

At 05:27 PM 8/9/2005 +0200, Ron Korving wrote:

Ron Korving

21 years ago

Okay Andi, the script in this message is as simple as it gets. I used a
syslog file to create a load of data in this case, but of course you can use
any (text) file for this.

#!/usr/bin/php5
<?
function process()
{
$data = file("/var/log/syslog.0");

foreach ($data as $line)
throw new Exception("error");
}

for ($i=0; $i < 30; $i++)
{
echo memory_get_usage()."\n";

try { process(); } catch (Exception $e) { }
}
?>

This is the output it gives me (PHP 5.0.4 btw):

40232
309528
578880
846720
1114560
1382400
1650240
1918080
2185920
2453760
2721600
2989440
3257280
...
7810560


I consider this a bug or a flaw anyway. I hope you agree.

Ron


"Andi Gutmans" <andi@zend.com> schreef in bericht
news:5.1.0.14.2.20050809123954.03491eb0@localhost...

> Can you send a complete script which also passes $data that reproduces the
> problem? Thanks.
>
> At 05:27 PM 8/9/2005 +0200, Ron Korving wrote:
> >The problem occurs in this particular situation:
> >
> >public function process($data)
> >{
> >   $largeResultSet = $this->xmlParser->parseString($data);
> >
> >   foreach ($largeResultSet as &$item)
> >   {
> >     throw new Exception("error");
> >   }
> >}
> >
> >It only happens if the exception is thrown from within the foreach(). It
> >seems the $largeResultSet is not freed in this case. Using the foreach by
> >reference or not (&$item or $item) does not make a difference.
> >
> >Ron
> >
> >
> >""Ron Korving"" <r.korving@xit.nl> schreef in bericht
> >news:21.AF.04646.20AC8F24@pb1.pair.com...
> > > It is very likely that this has something to do with DOM. It's a huge
> >piece
> > > of code though, making it hard to test where the problem really lies.
> > >
> > > ""Ron Korving"" <r.korving@xit.nl> schreef in bericht
> > > news:BD.4F.04646.AA6C8F24@pb1.pair.com...
> > > > I've noticed that in a script of mine, memory consumption can

explode

> > > quite
> > > > drastically when Exceptions are thrown, opposed to very normal

memory

> > > > consumption when exceptions are not thrown.
> > > >
> > > > This is the idea:
> > > >
> > > > <?
> > > > class Obj
> > > > {
> > > >   function process($i)
> > > >   {
> > > >     // do a lot of stuff, occupy a lot of memory within the scope of
> >this
> > > > function
> > > >     throw new Exception("error");
> > > >   }
> > > > }
> > > >
> > > > $obj = new Obj();
> > > > $errors = array();
> > > >
> > > > for ($i=0; $i < 60; $i++)
> > > > {
> > > >   try
> > > >   {
> > > >     $obj->process();
> > > >   }
> > > >   catch (Exception $e)
> > > >   {
> > > >     $errors[] = $e->getMessage();
> > > >   }
> > > > }
> > > > ?>
> > > >
> > > > When exceptions are not thrown, memory consumption is normal. The
> >method's
> > > > local vars are freed and not much more memory is used in the 50th

pass

> > > > compared to the 1st. But when exceptions are thrown, memory

consumption

> > > > increases quite a bit every pass. In the end (within the 60 passes

of my

> > > > loop) over 8 MB's are used.
> > > >
> > > > Of course, I can unset a lot of data before throwing exceptions to

free

Marcus Börger

21 years ago

Hello Ron,

Wednesday, August 10, 2005, 8:47:25 AM, you wrote:

> Okay Andi, the script in this message is as simple as it gets. I used a
> syslog file to create a load of data in this case, but of course you can use
> any (text) file for this.

> #!/usr/bin/php5
> <?
>   function process()
>   {
>     $data = file("/var/log/syslog.0");

>     foreach ($data as $line)
>       throw new Exception("error");

What makes you think the file would be closed. Erm the array deleted? And
that is your problem you are unnecessarily reading all the file into an
array. Try using the same with FileObject instead, which should reduce
memory usage a lot:

foreach(FileObject($file) as $data) throw new Exception("error");

But of course you only did that to demonstrate the memory problem which in
fact simply means that we don't do a full stack cleanup like c++ would do
here (php != c++).

In php we do the cleanup at script termination. That is also the reason
why php is not perfectly suitable for console apps especially when it comes
to daemons.

best regards
marcus

Ron Korving

21 years ago

You're right, using that file to create a big piece of data was just for
demonstration purposes. In my personal case it's XML parsing. The way I read
what you just said is that this is behavior by design. But if I don't throw
Exceptions, memory usage stays constant. Throwing and catching these
exceptions are the exception-to-the-rule.

Nonetheless, if this is truly by design.. maybe that's something to look
into for PHP6.

Ron

"Marcus Boerger" <helly@php.net> schreef in bericht
news:124244453.20050810114522@marcus-boerger.de...

> Hello Ron,
>
> Wednesday, August 10, 2005, 8:47:25 AM, you wrote:
>
> > Okay Andi, the script in this message is as simple as it gets. I used a
> > syslog file to create a load of data in this case, but of course you can

use

> > any (text) file for this.
>
> > #!/usr/bin/php5
> > <?
> >   function process()
> >   {
> >     $data = file("/var/log/syslog.0");
>
> >     foreach ($data as $line)
> >       throw new Exception("error");
>
> What makes you think the file would be closed. Erm the array deleted? And
> that is your problem you are unnecessarily reading all the file into an
> array. Try using the same with FileObject instead, which should reduce
> memory usage a lot:
>
> foreach(FileObject($file) as $data) throw new Exception("error");
>
> But of course you only did that to demonstrate the memory problem which in
> fact simply means that we don't do a full stack cleanup like c++ would do
> here (php != c++).
>
> In php we do the cleanup at script termination. That is also the reason
> why php is not perfectly suitable for console apps especially when it

comes

Dmitry Stogov

21 years ago

I have opened the bug #34065, but it is very hard to fix it.

Thanks. Dmitry.

Ron Korving

21 years ago

Perhaps it would require an revised garbage collection mechanism? If that's
the case and you think it's worth investing time in, I think it would be
something for the next major version (6).

Good luck,

Ron


""Dmitry Stogov"" <dmitry@zend.com> schreef in bericht
news:000001c59da7$96c4b420$e902a8c0@thinkpad...

Derick Rethans

21 years ago

On Wed, 10 Aug 2005, Ron Korving wrote:

> Perhaps it would require an revised garbage collection mechanism? If that's
> the case and you think it's worth investing time in, I think it would be
> something for the next major version (6).

I think it would be better not to abuse Exceptions like this. They are
for exceptional states, which should result in a graceful shutdown of
your application. In that case 'leaking' the memory to the end of the
request isn't a real problem anyway.

Derick

Ron Korving

21 years ago

My situation is parsing a number of XML-reports. If parsing goes wrong in
any way (XML parsing fails, XML validation fails, database insertion fails),
I want to throw an exception. There's a script that batch-processes these
files. I want to collect exceptions and give that feedback to the user. I
wouldn't consider this bad practice myself (but of course, I'm not very
objective in this situation).

Even if I'm wrong about that, I'm not the only PHP-user and it's my humble
opinion that the engine should be solid enough to deal with this.

Ron


"Derick Rethans" <derick@php.net> schreef in bericht
news:Pine.LNX.4.62.0508101444230.3884@localhost...

> On Wed, 10 Aug 2005, Ron Korving wrote:
>
> > Perhaps it would require an revised garbage collection mechanism? If

that's

Andi Gutmans

21 years ago

Hi Ron,

Indeed exception may lead to leaks in certain cases. We do try and cleanup
as much as possible but the situation you describe is hard to detect. I
will however put it on our TODO list because this case could possibly be
improved on.

Andi

At 11:54 AM 8/10/2005 +0200, Ron Korving wrote:

Ron Korving

21 years ago

Thanks, that's all I would ever ask for :)

Ron

"Andi Gutmans" <andi@zend.com> schreef in bericht
news:5.1.0.14.2.20050810070706.042bcec0@localhost...

> Hi Ron,
>
> Indeed exception may lead to leaks in certain cases. We do try and cleanup
> as much as possible but the situation you describe is hard to detect. I
> will however put it on our TODO list because this case could possibly be
> improved on.
>
> Andi
>
> At 11:54 AM 8/10/2005 +0200, Ron Korving wrote:
> >You're right, using that file to create a big piece of data was just for
> >demonstration purposes. In my personal case it's XML parsing. The way I

read

> >what you just said is that this is behavior by design. But if I don't

throw

> >Exceptions, memory usage stays constant. Throwing and catching these
> >exceptions are the exception-to-the-rule.
> >
> >Nonetheless, if this is truly by design.. maybe that's something to look
> >into for PHP6.
> >
> >Ron
> >
> >"Marcus Boerger" <helly@php.net> schreef in bericht
> >news:124244453.20050810114522@marcus-boerger.de...
> > > Hello Ron,
> > >
> > > Wednesday, August 10, 2005, 8:47:25 AM, you wrote:
> > >
> > > > Okay Andi, the script in this message is as simple as it gets. I

used a

> > > > syslog file to create a load of data in this case, but of course you

can

> >use
> > > > any (text) file for this.
> > >
> > > > #!/usr/bin/php5
> > > > <?
> > > >   function process()
> > > >   {
> > > >     $data = file("/var/log/syslog.0");
> > >
> > > >     foreach ($data as $line)
> > > >       throw new Exception("error");
> > >
> > > What makes you think the file would be closed. Erm the array deleted?

And

> > > that is your problem you are unnecessarily reading all the file into

an

> > > array. Try using the same with FileObject instead, which should reduce
> > > memory usage a lot:
> > >
> > > foreach(FileObject($file) as $data) throw new Exception("error");
> > >
> > > But of course you only did that to demonstrate the memory problem

which in

> > > fact simply means that we don't do a full stack cleanup like c++ would

do

> > > here (php != c++).
> > >
> > > In php we do the cleanup at script termination. That is also the

reason