Everything

php

memory_get_usage with new Memory Manager

php.internals

Matt W

20 years ago

Hi,

After looking through the new Memory Manager code, I assumed this would be
the case... (and just now got to test).  Since heap->size is only updated
when a 256K or whatever block is *actually* allocated/freed, the number
returned by memory_get_usage() will only be a multiple of that.  That
doesn't make it very useful anymore if one is using it to see how much
memory usage differs by changing, say, number of arrays/elements/variables,
strings, etc. as I was last month (small changes).  And the example in the
manual definitely won't work as shown.

Does anybody else think that memory_get[peak_]usage() should work more like
before?  I would imagine it can be "fixed" fairly easily, though it probably
needs another variable, and just a little extra overhead of inc./dec. that
variable more often (but I think that was done in the old MM anyway?).


Thanks,
Matt

Dmitry Stogov

20 years ago

Right now memory_get[peak_]usage() show the amount of REAL memory that PHP
(Zend Memory Manager) takes from system.

Previous memory manager showed size of emalloc()-ed memory without malloc()
overhead.
Also it didn't consider internal caches.

We need decide which behavior to use before 5.2.0 release.

Thanks. Dmitry.

Derick Rethans

20 years ago

On Tue, 25 Jul 2006, Dmitry Stogov wrote:

> Right now memory_get[peak_]usage() show the amount of REAL memory that PHP
> (Zend Memory Manager) takes from system.
>
> Previous memory manager showed size of emalloc()-ed memory without malloc()
> overhead.
> Also it didn't consider internal caches.
>
> We need decide which behavior to use before 5.2.0 release.

I would definitely want a more precise memory usage. However, we can
add the new one as a different function (or atleast some internal value
in AG).

For now, this change breaks some BC with extensions (like Xdebug) that
use AG(allocated_memory).

regards,
Derick

--
Derick Rethans
http://derickrethans.nl | http://ez.no | http://xdebug.org

Jani Taskinen

20 years ago

memory_get_real_[peak_]usage() ? :)

--Jani

On Tue, 25 Jul 2006, Derick Rethans wrote:

Dmitry Stogov

20 years ago

> -----Original Message-----
> From: Derick Rethans [mailto:derick@php.net]
> Sent: Tuesday, July 25, 2006 10:58 AM
> To: Dmitry Stogov
> Cc: 'Matt W'; internals@lists.php.net; Andi Gutmans; Ilia Alshanetsky
> Subject: RE: [PHP-DEV] memory_get_usage with new Memory Manager
>
>
> On Tue, 25 Jul 2006, Dmitry Stogov wrote:
>
> > Right now memory_get[peak_]usage() show the amount of REAL
> memory that
> > PHP (Zend Memory Manager) takes from system.
> >
> > Previous memory manager showed size of emalloc()-ed memory without
> > malloc() overhead. Also it didn't consider internal caches.
> >
> > We need decide which behavior to use before 5.2.0 release.
>
> I would definitely want a more precise memory usage.

We can revert "allocated_memory" semantic.
This is not a problem.

> However, we can
> add the new one as a different function (or atleast some
> internal value
> in AG).
>
> For now, this change breaks some BC with extensions (like
> Xdebug) that
> use AG(allocated_memory).

Please use new API functions to access them.

Dmitry.

Derick Rethans

20 years ago

On Tue, 25 Jul 2006, Dmitry Stogov wrote:

> > > Right now memory_get[peak_]usage() show the amount of REAL memory
> > > that PHP (Zend Memory Manager) takes from system.
> > >
> > > Previous memory manager showed size of emalloc()-ed memory without
> > > malloc() overhead. Also it didn't consider internal caches.
> > >
> > > We need decide which behavior to use before 5.2.0 release.
> >
> > I would definitely want a more precise memory usage.
>
> We can revert "allocated_memory" semantic.
> This is not a problem.
>
> > However, we can add the new one as a different function (or atleast
> > some internal value in AG).
> >
> > For now, this change breaks some BC with extensions (like Xdebug)
> > that use AG(allocated_memory).
>
> Please use new API functions to access them.

What are the new functions?

Derick

Ron Korving

20 years ago

Perhaps it'd be a nice alternative to have an optional parameter? Having 4
functions for checking memory usage is quite a lot, isn't it?

I would humbly prefer seeing:

int memory_get_usage([bool realusage])
int memory_get_peak_usage([bool realusage])

- Ron


"Derick Rethans" <derick@php.net> schreef in bericht
news:Pine.LNX.4.62.0607251122250.5096@localhost...

Dmitry Stogov

20 years ago

Good suggestion.
The correspomding C functions (zend_memory_usage()) can be changed in the
same way.

The memory_limit will de checked for real usage only (as now).

Any objections?

Dmitry.

Ilia A.

20 years ago

On 25-Jul-06, at 7:24 AM, Dmitry Stogov wrote:

> Good suggestion.
> The correspomding C functions (zend_memory_usage()) can be changed
> in the
> same way.
>
> The memory_limit will de checked for real usage only (as now).
>
> Any objections?

Nope, sounds good to me.

Ilia Alshanetsky

Matt W

20 years ago

Hi,

That's what I was thinking too, Ron.  Internally, the functions
zend_memory_[peak_]usage() would also get a flag parameter, or separate
functions...

There's also the small issue of which size to set Apache's mod_php_mem_usage
to.

Dmitry, et al., I don't know the details of how the Memory Manager works,
but I assume multiple memory blocks can have free space (if something didn't
fit, and now block(s) were allocated).  So if memory_get_usage() operates
like before (reflecting small changes) and it's being used to make sure to
stay under memory_limit, for example, memory_limit could be exhasted sooner
than expected since the REAL allocated size is compared with heap->limit.
Could/should the limit be compared to size that's measured the old way, or
might that allow too much to "escape?"

Hope you don't mind my random thoughts. :-)


Matt


----- Original Message -----
From: "Ron Korving"
Sent: Tuesday, July 25, 2006

Dmitry Stogov

20 years ago

See zend_alloc.h
zend_memory_usage() & zend_memory_peak_usage().

Dmitry.

Lukas Smith

20 years ago

Dmitry Stogov wrote:

> Right now memory_get[peak_]usage() show the amount of REAL memory that PHP
> (Zend Memory Manager) takes from system.
>
> Previous memory manager showed size of emalloc()-ed memory without malloc()
> overhead.
> Also it didn't consider internal caches.

Shouldn't we make the old behavior the default here?

regards,
Lukas

Antony Dovgal

20 years ago

On 25.07.2006 16:45, Lukas Smith wrote:

> Dmitry Stogov wrote:
>> Right now memory_get[peak_]usage() show the amount of REAL memory that PHP
>> (Zend Memory Manager) takes from system.
>>
>> Previous memory manager showed size of emalloc()-ed memory without malloc()
>> overhead.
>> Also it didn't consider internal caches.
>
> Shouldn't we make the old behavior the default here?

Agree.

--
Wbr,
Antony Dovgal

Ilia A.

20 years ago

The idea of a memory limit is to as accurately as possible account
for the memory utilized by PHP. If our current calculation is more
accurate since it accounts for malloc overhead, great, it gives more
fine grained control to hosters utilizing this option.


On 25-Jul-06, at 9:45 AM, Lukas Smith wrote:

> Dmitry Stogov wrote:
>> Right now memory_get[peak_]usage() show the amount of REAL memory
>> that PHP
>> (Zend Memory Manager) takes from system.
>> Previous memory manager showed size of emalloc()-ed memory without
>> malloc()
>> overhead.
>> Also it didn't consider internal caches.
>
> Shouldn't we make the old behavior the default here?
>
> regards,
> Lukas
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

Ilia Alshanetsky

Matt W

20 years ago

Hi Dmitry, Ilia, et al.,

Ilia, the Memory Manager is checking the *real* size against memory_limit,
so it's still "more accurate" even after the functions were changed to
report more like the old way.  I still wonder how much difference there may
be between size and real_size as it gets close to the limit...

After the changes, I see the usage difference after adding a variable, for
example, is slightly different than before, but I understand that's because
of counting overhead now or such.  The old one just rounded the amount up to
a multiple of 8, I think...

Something else I'd like to see changed...  Does anyone think
memory_get_[peak_]usage() should *always* be available, regardless of
whether memory_limit is actually enabled?  Using the Windows binaries at
least, I was devastated :-D that they were unavailable for experimenting,
etc.

I don't see why not -- the function names IMO don't really suggest that
they're tied to memory_limit and all that's being saved by not having them
is a little overhead of updating the size/peak variables, right?


Thanks,
Matt


----- Original Message -----
From: "Ilia Alshanetsky"
Sent: Tuesday, July 25, 2006

Edin Kadribasic

20 years ago

Matt W wrote:

> Hi Dmitry, Ilia, et al.,
>
> Ilia, the Memory Manager is checking the *real* size against memory_limit,
> so it's still "more accurate" even after the functions were changed to
> report more like the old way.  I still wonder how much difference there may
> be between size and real_size as it gets close to the limit...
>
> After the changes, I see the usage difference after adding a variable, for
> example, is slightly different than before, but I understand that's because
> of counting overhead now or such.  The old one just rounded the amount up to
> a multiple of 8, I think...
>
> Something else I'd like to see changed...  Does anyone think
> memory_get_[peak_]usage() should *always* be available, regardless of
> whether memory_limit is actually enabled?  Using the Windows binaries at
> least, I was devastated :-D that they were unavailable for experimenting,
> etc.
>
> I don't see why not -- the function names IMO don't really suggest that
> they're tied to memory_limit and all that's being saved by not having them
> is a little overhead of updating the size/peak variables, right?
>

The reason is performance. It takes some processing time to keep
accounting of memory usage and people don't using the feature do not
feel like its worth slowing PHP down.

Edin

Ilia A.

20 years ago

On 27-Jul-06, at 6:02 AM, Matt W wrote:

>
> Something else I'd like to see changed...  Does anyone think
> memory_get_[peak_]usage() should *always* be available, regardless of
> whether memory_limit is actually enabled?  Using the Windows
> binaries at
> least, I was devastated :-D that they were unavailable for
> experimenting,
> etc.
>

Counting memory all the time adds a fair amount of overhead to script
execution especially given how frequently PHP needs to allocate
memory. Therefore usage memory utilization functions are only
available when PHP is compiled with memory in which case it keeps
track of its mem usage.


Ilia Alshanetsky

Dmitry Stogov

20 years ago

Now, functions memory_get_[peak_]usage() and memory_limit checking can be
enabled/disabled using "--enable-memory-limit" configure option. However
cost of meory_limit checking is significant less with new memory manager.

I think now we can remove "--enable-memory-limit" option and make it always
enabled. Functions memory_get_[peak_]usage() will be always enabled and will
always report proper values for "real" memory usage.
The tracking of "emalloced" memory usage will be enabled only in DEBUG
build.

So in RELEASE memory_get_[peak_]usage() and memory_get_[peak_]usage(false)
will always return FALSE (or 0), and
memory_get_[peak_]usage(true) will return "real" memory usage.
In DEBUG build memory_get_[peak_]usage() and memory_get_[peak_]usage(false)
will return "emalloced" memory, and
memory_get_[peak_]usage(true) will return "real" memory usage.

What do you think?

Dmitry.


-----Original Message-----
From: Ilia Alshanetsky [mailto:iliaal@gmail.com] On Behalf Of Ilia
Alshanetsky
Sent: Thursday, July 27, 2006 4:35 PM
To: Matt W
Cc: internals@lists.php.net; Dmitry Stogov
Subject: Re: [PHP-DEV] memory_get_usage with new Memory Manager



On 27-Jul-06, at 6:02 AM, Matt W wrote:


Something else I'd like to see changed...  Does anyone think
memory_get_[peak_]usage() should *always* be available, regardless of
whether memory_limit is actually enabled?  Using the Windows binaries at
least, I was devastated :-D that they were unavailable for experimenting,
etc.



Counting memory all the time adds a fair amount of overhead to script
execution especially given how frequently PHP needs to allocate memory.
Therefore usage memory utilization functions are only available when PHP is
compiled with memory in which case it keeps track of its mem usage.



Ilia Alshanetsky

Ilia A.

20 years ago

On 27-Jul-06, at 8:50 AM, Dmitry Stogov wrote:

> Now, functions memory_get_[peak_]usage() and memory_limit checking
> can be
> enabled/disabled using "--enable-memory-limit" configure option.
> However
> cost of meory_limit checking is significant less with new memory
> manager.
>

I'd prefer to avoid adding extra un-needed overhead. People who need
this information generally have memory limit enabled anyway, plus
there are always other ways to get PHP memory utilization either via
proc or from ps output, etc...

Ilia Alshanetsky

Matt W

20 years ago

Hi Dmitry,

Thanks for the reply. :-)

To the others mentioning the overhead and "slowness" of keeping track of the
memory size, I can't believe that, relative to ALL the other work being done
in emalloc()?  I was going to try a loop of emalloc() and efree() with and
without memory-limit just to see what the difference is.  And it would be
much less of course %-wise in overall script execution.  Don't have time
right now...

If keeping track of mem usage was such a performance hit, 1) people that
care about performance shouldn't use memory-limit, and 2) when Dmitry added
*another* variable which gets updated more often, it hurt the performance a
lot.  I doubt it. :-)

Regarding your questions Dmitry, I'll share the thoughts of an outsider (but
also a user!) if they matter.
*) I do think memory_get_[peak_]usage() should always be available
*) I don't know that the --enable-memory-limit option should be removed
*) I would like the "real_usage" param to always work the same and think it
should.  The way you propose, what about the Windows binary downloads?
Wouldn't be able to see small changes in memory usage. :-(


Thanks,
Matt


----- Original Message -----
From: "Dmitry Stogov"
Sent: Thursday, July 27, 2006


Now, functions memory_get_[peak_]usage() and memory_limit checking can be
enabled/disabled using "--enable-memory-limit" configure option. However
cost of meory_limit checking is significant less with new memory manager.

I think now we can remove "--enable-memory-limit" option and make it always
enabled. Functions memory_get_[peak_]usage() will be always enabled and will
always report proper values for "real" memory usage.
The tracking of "emalloced" memory usage will be enabled only in DEBUG
build.

So in RELEASE memory_get_[peak_]usage() and memory_get_[peak_]usage(false)
will always return FALSE (or 0), and
memory_get_[peak_]usage(true) will return "real" memory usage.
In DEBUG build memory_get_[peak_]usage() and memory_get_[peak_]usage(false)
will return "emalloced" memory, and
memory_get_[peak_]usage(true) will return "real" memory usage.

What do you think?

Dmitry.

Richard Quadling

20 years ago

I'm NOT a expert in this field. My POV may be quite dumb, but it may
point to a solution.

If memory is allocated, I would assume that somewhere there is a
mechanism to access it otherwise this is a memory leak.

So, rather than counting the memory in real time, why not have the
memory_get_usage() (not sure about peak usage) simply iterate the used
memory and calculate how much there is.

This would put the overhead in the function call and not throughout
the rest of the code.

On 27/07/06, Ilia Alshanetsky <ilia@prohost.org> wrote:

>
> On 27-Jul-06, at 6:02 AM, Matt W wrote:
> >
> > Something else I'd like to see changed...  Does anyone think
> > memory_get_[peak_]usage() should *always* be available, regardless of
> > whether memory_limit is actually enabled?  Using the Windows
> > binaries at
> > least, I was devastated :-D that they were unavailable for
> > experimenting,
> > etc.
>
> Counting memory all the time adds a fair amount of overhead to script
> execution especially given how frequently PHP needs to allocate
> memory. Therefore usage memory utilization functions are only
> available when PHP is compiled with memory in which case it keeps
> track of its mem usage.

--
-----
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&amp;r=213474731
"Standing on the shoulders of some very clever giants!"

bertrand Gugger

20 years ago

Ilia Alshanetsky wrote:

>
> On 27-Jul-06, at 6:02 AM, Matt W wrote:
>
>>
>> Something else I'd like to see changed...  Does anyone think
>> memory_get_[peak_]usage() should *always* be available, regardless of
>> whether memory_limit is actually enabled?  Using the Windows  binaries at
>> least, I was devastated :-D that they were unavailable for
>> experimenting,
>> etc.
>>
>
> Counting memory all the time adds a fair amount of overhead to script
> execution especially given how frequently PHP needs to allocate  memory.
> Therefore usage memory utilization functions are only  available when
> PHP is compiled with memory in which case it keeps  track of its mem usage.

That may hear off topic , but how you enable something else than 8M
*without* getting this counting overhead ?
It looks to me that --enable-memory-limit mixes 2 independant roles :
- setting the amount of available memory, and enabling eventually local
setting,
- enabling the evaluation of the process consumption.

I would like to be wrong on this :)

--
toggg

Ilia A.

20 years ago

On 27-Jul-06, at 9:03 AM, bertrand Gugger wrote:

> Ilia Alshanetsky wrote:
>> On 27-Jul-06, at 6:02 AM, Matt W wrote:
>>>
>>> Something else I'd like to see changed...  Does anyone think
>>> memory_get_[peak_]usage() should *always* be available,
>>> regardless of
>>> whether memory_limit is actually enabled?  Using the Windows
>>> binaries at
>>> least, I was devastated :-D that they were unavailable for
>>> experimenting,
>>> etc.
>>>
>> Counting memory all the time adds a fair amount of overhead to
>> script  execution especially given how frequently PHP needs to
>> allocate  memory. Therefore usage memory utilization functions are
>> only  available when PHP is compiled with memory in which case it
>> keeps  track of its mem usage.
>
> That may hear off topic , but how you enable something else than 8M
> *without* getting this counting overhead ?

Eh? When you set a limit what are you trying to do? From my
experience most of the time this setting is used by hosting providers
to restrict memory utilization of various PHP scripts to avoid out-of-
memory situations and system abuse. As such you'd want as accurate
measurement as possible including any overhead and what not, since it
is possible to use the overhead to exceed or in some cases even to
bypass the memory limit entirely.

> It looks to me that --enable-memory-limit mixes 2 independant roles :
> - setting the amount of available memory, and enabling eventually
> local setting,
> - enabling the evaluation of the process consumption.

That is correct. Memory limit by its nature requires PHP to count the
amount of memory consumed, since such tracking is performed it
enabled PHP to report internally via functions (or via Apache log)
about its memory utilization.

Ilia Alshanetsky

Ron Korving

20 years ago

Hi Ilia,

"Ilia Alshanetsky" <ilia@prohost.org> wrote in message
news:70339B16-D50B-45B9-A55D-B58F9EEB679D@prohost.org...

> Eh? When you set a limit what are you trying to do? From my
> experience most of the time this setting is used by hosting providers
> to restrict memory utilization of various PHP scripts to avoid out-of-
> memory situations and system abuse. As such you'd want as accurate
> measurement as possible including any overhead and what not, since it
> is possible to use the overhead to exceed or in some cases even to
> bypass the memory limit entirely.

Yes, hosting providers would enable the memory limit. But who wants to use
memory_get_[peak_]usage()? Not the hosting provider, but the application
developer.

- Ron

Brian Moon

20 years ago

> Yes, hosting providers would enable the memory limit. But who wants to use
> memory_get_[peak_]usage()? Not the hosting provider, but the application
> developer.

We have some process, that if cache has gotten stale and needs to be
recreated, the process can use 100MB or memory.  We don't want to cap
it.  However, we do want to know about it and use
apahce_child_terminate() to end that process.  Right now, I have to
actually read /proc to figure out my memory usage.  I would much rather
have a function.  I guess I could enable a memory limit and set to like
1GB.  Seems silly though.

--

Brian Moon
-------------
http://dealnews.com/
Its good to be cheap =)

Ilia A.

20 years ago

On 27-Jul-06, at 1:17 PM, Ron Korving wrote:

> Yes, hosting providers would enable the memory limit. But who wants
> to use
> memory_get_[peak_]usage()? Not the hosting provider, but the
> application
> developer.
>

The peak usage function was added with profiling in mind and keeping
track of scripts that spike memory, when I added it the intent was it
would help with profiling and debugging. Not something used everyday
in production environment. In dev environment speed is not an issue,
so you can enable memory limit and raise the limit to some very high
value such as 1Gb so it does not get in the way.


Ilia Alshanetsky

bertrand Gugger

20 years ago

Ilia Alshanetsky wrote:

>
> On 27-Jul-06, at 9:03 AM, bertrand Gugger wrote:
>
>>
>> That may hear off topic , but how you enable something else than 8M
>> *without* getting this counting overhead ?
>
>
> Eh? When you set a limit what are you trying to do? From my  experience
> most of the time this setting is used by hosting providers  to restrict
> memory utilization of various PHP scripts to avoid out-of- memory
> situations and system abuse.

The most cases I saw were providers or production site wanting to grant
*more* than 8M (typically 16M) to their customers / users.
I dont call that a "restriction".
You should know this 8M limit is reached very easily.
e.g. lot of popular CMS wont work below this limit as soon as they have
a litle contents.

> As such you'd want as accurate  measurement
> as possible including any overhead and what not, since it  is possible
> to use the overhead to exceed or in some cases even to  bypass the
> memory limit entirely.

You miss the point , they don't care about accuracy,
they just don't want the default 8M but more.

>
>> It looks to me that --enable-memory-limit mixes 2 independant roles :
>> - setting the amount of available memory, and enabling eventually
>> local setting,
>> - enabling the evaluation of the process consumption.
>
>
> That is correct. Memory limit by its nature requires PHP to count the
> amount of memory consumed, since such tracking is performed it  enabled
> PHP to report internally via functions (or via Apache log)  about its
> memory utilization.

I have no idea how it works internally ...
Are you meaning the default "hard-coded" 8M limit is magic and will not
need to count the amount of memory consumed but any custom setting will ?

--
toggg

Ilia A.

20 years ago

On 24-Jul-06, at 11:52 PM, Matt W wrote:

> Does anybody else think that memory_get[peak_]usage() should work
> more like
> before?  I would imagine it can be "fixed" fairly easily, though it
> probably
> needs another variable, and just a little extra overhead of inc./
> dec. that
> variable more often (but I think that was done in the old MM anyway?).

I think that the current, more precise memory usage indicator is the
correct behavior and should not be changed.

Ilia Alshanetsky