Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Performance Consideration of 1 class per file »

Performance Consideration of 1 class per file

php.internals

Alan Knowles

22 years ago
PEAR is considering stipulating 1 class per file, for the packages. - one of the concerns raised is performance. While this is true for a non-cache compiled situation (where performance is less of an issue) - would anyone care to comment on the implications when using the Zend Optimizer or APC, on how significant the impact of this would be. - looking at typical application where the number of files included in the code with may jump from 20->30 included files.. From my understanding, if you where looking for performance, you would probably have a cache expiry which would mean very few stat (if any) calls based on these addition includes.. - hence negating the issue.. Thanks Alan
-- Can you help out? Need Consulting Services or Know of a Job? http://www.akbkhome.com

Rasmus Lerdorf

22 years ago
It can be significant. There are a couple of issues: 1. The included_files list gets updated each time you include a file. In order to make sure that the same file included by different paths or symlinks don't conflict we do a realpath() on the file to be included. That means stats on every component up to and including the file itself. I can't speak for all the opcode caches, but at least APC doesn't do anything to alleviate this. 2. APC uses the file's device and inode as the lookup key into shared memory to find the opcodes for the file, so a stat has to be done on each and every file that is cached. There is currently no such thing as a request-spanning stat cache. So yes, jumping from 20 to 30 include files could very well bring a rather significant performance hit. Anybody looking for real performance out of PHP pretty much has to address the first issue. It is easy enough to do, simply get rid of the realpath lookup in that situation and just live with the fact that if someone includes the same file via different paths it won't catch it. I can see that it is convenient to have PHP figure this out for you, but the price of that convenience is not worth it in my opinion. -Rasmus On Sat, 24 Apr 2004, Alan Knowles wrote:

Christian Schneider

22 years ago
Rasmus Lerdorf wrote:
> 1. The included_files list gets updated each time you include a file. In > order to make sure that the same file included by different paths or > symlinks don't conflict we do a realpath() on the file to be included.
That's done by PHP, not APC, right? Does this only apply to require_once or require as well?
> 2. APC uses the file's device and inode as the lookup key into shared > memory to find the opcodes for the file, so a stat has to be done on
Hmm.. If the stat on the file and the check for device/inode would be done first then you wouldn't have to do a realpath, right? But I guess that's not easily done until after the realpath check.
> So yes, jumping from 20 to 30 include files could very well bring a rather > significant performance hit.
I guess that's only important if your PHP code is really simple and you don't do something like e.g. DB queries because otherwise that'd be 90% of the running time anyway, right? I guess someone _that_ considered about performance could easily do a cat *.php | grep -v require | php -w >app.lib or the like and include app.lib. - Chris

Rasmus Lerdorf

22 years ago
On Sat, 24 Apr 2004, Christian Schneider wrote:
> Rasmus Lerdorf wrote: > > 1. The included_files list gets updated each time you include a file. In > > order to make sure that the same file included by different paths or > > symlinks don't conflict we do a realpath() on the file to be included. > > That's done by PHP, not APC, right? Does this only apply to require_once > or require as well?
Right, it is done by PHP which is why APC can't do anything about it. It happens on all files parsed by PHP no matter how they are included.
> > 2. APC uses the file's device and inode as the lookup key into shared > > memory to find the opcodes for the file, so a stat has to be done on > > Hmm.. If the stat on the file and the check for device/inode would be > done first then you wouldn't have to do a realpath, right? But I guess > that's not easily done until after the realpath check.
Well, the included_files list is pathname based and when checking to see if a file has already been included we do a string compare. We could potentially follow apc's lead and use device+inode instead and thereby not need the realpath call and still maintain the current functionality. There are some other issues with this though. Like, for example, when you edit a file most editors will actually create a new file and delete the old, so you end up with the same pathname having a new inode. We could state that you should never edit a file on a live production web server, but everyone does and I can just see the stream of bug reports this might spur.
> > So yes, jumping from 20 to 30 include files could very well bring a rather > > significant performance hit. > > I guess that's only important if your PHP code is really simple and you > don't do something like e.g. DB queries because otherwise that'd be 90% > of the running time anyway, right?
Sure, you could argue that the stat calls will get lost in the noise of a complex script. But they really can add up fast. Bouncing along the include_path looking for include files adds extra stats, open_basedir is another killer when it comes to stats. Even just having "." as the first piece of the include_path is going to cost you an extra stat when including stuff from PEAR (unless you hardcode the pear path). And the sort of people that need to worry about this tend to cache stuff from backends like crazy which, if done right, means that the backend is only touched once in a while and the bulk of the requests will pull data right out of a fast memory-based session mechanism or use other similar tricks. However, if at the same time you are stuck with 30 includes and each of these cost you 10 stat calls, that is 300 disk-touching system calls per request that you really could do without.
> I guess someone _that_ considered about performance could easily do a > cat *.php | grep -v require | php -w >app.lib > or the like and include app.lib.
Yes, and I know a number of folks that do pre-processing like this on their code before pushing it to their production servers. I am all for pre-runtime content management systems that allow authors to structure their code in whatever manner they see fit and when they push to the production servers the content is optimized for the delivery mechanism. For the majority of users PHP is plenty fast enough. We could stick a sleep(1) in there and I doubt anybody would notice. When you are only serving up a few hundred thousand requests per day it really doesn't matter how you architect things. Regardless of this basic fact, I don't think we should be sticking sleep(1)'s in the code, and we should be thinking about reducing the system call overhead where we can, and projects like PEAR should carefully evaluate the cost vs. convenience ratio when making decisions like the one which prompted this. -Rasmus

Michael Walter

22 years ago
Christian Schneider wrote:
> [...] > I guess someone _that_ considered about performance could easily do a > cat *.php | grep -v require | php -w >app.lib > or the like and include app.lib.
Well yeah, it gets slightly harder when you dynamically require modules. Cheers, Michael

Sebastian Bergmann

22 years ago
Christian Schneider wrote:
> I guess someone _that_ considered about performance could easily do a > cat *.php | grep -v require | php -w >app.lib > or the like and include app.lib.
Maybe the PEAR Packager / Installer could be changed to (optionally?) create a single source file (without require/include statements) comments, whitespace) from all role="php" <file>s and use this for deployment.
-- Sebastian Bergmann http://sebastian-bergmann.de/ http://phpOpenTracker.de/ Das Buch zu PHP 5: http://professionelle-softwareentwicklung-mit-php5.de/

George Schlossnagle

22 years ago
On Apr 24, 2004, at 10:18 AM, Sebastian Bergmann wrote:
> Christian Schneider wrote: >> I guess someone _that_ considered about performance could easily do a >> cat *.php | grep -v require | php -w >app.lib >> or the like and include app.lib. > > Maybe the PEAR Packager / Installer could be changed to (optionally?) > create a single source file (without require/include statements) > comments, whitespace) from all role="php" <file>s and use this for > deployment.
Having naively tried this in the past, a couple comments: 1) it's not as simple as it sounds, especially if you have includes that do more than just declare functions and classes. 2) it's hard to debug when things go wrong, as the sources aren't where you had them. 3) if you have any conditional includes, 3a) things can easliy break 3b) you often end up doing more work and being slower than using the includes I don't think this is a truly awful idea, but should be approached with a high degree of caution, as it's pretty brittle. George

Sebastian Bergmann

22 years ago
George Schlossnagle wrote:
> 1) it's not as simple as it sounds, especially if you have includes > that do more than just declare functions and classes.
Who said anything about this beeing easy? Easy is boring ;-) As far as 1) is concerned: this should not be a problem with PEAR packages, right?
-- Sebastian Bergmann http://sebastian-bergmann.de/ http://phpOpenTracker.de/ Das Buch zu PHP 5: http://professionelle-softwareentwicklung-mit-php5.de/

Stephan Schmidt

22 years ago
Hi,
> Maybe the PEAR Packager / Installer could be changed to (optionally?) > create a single source file (without require/include statements) > comments, whitespace) from all role="php" <file>s and use this for > deployment.
This should not be done, as a lot of packages are driver based. If you create one large file from all DB drivers, the compile time surely will be longer than the time needed to open the needed files. Just my 2cts. Stephan

Alan Knowles

22 years ago
Thanks - I just had a look through zend_comple.c:zend_include_or_eval_handler it appears that zend_stream_open .. and hence fopen is called on every require_once/include_once would it make sense to add something as simple as if (opline->op2.....lval == ZEND_REQUIRE_ONCE) { locations = get_potential_locations(file,PG('include_path')); foreach (locations as location) if (zend_hash_find(&EG(included_files), location)) { // i've already got it ! NEXT_OPCODE(); } } } reduce the stat calls substancially?? - especially where we have cases that alot of the pear classes already have "require_once 'PEAR.php'" .. even though it may have already been loaded by another package... Or did I miss some other optimization that is already there for that..? .. Yeah, I see from APC in CVS, it only wraps the compile_file() call, which has recieves a file pointer, so the stating/realpath damage etc. has already been done) - so unless it does something similar to the above check, and overrides the include handler, it wont help much either, I guess.. Regards Alan Rasmus Lerdorf wrote:
> It can be significant. There are a couple of issues: > > 1. The included_files list gets updated each time you include a file. In > order to make sure that the same file included by different paths or > symlinks don't conflict we do a realpath() on the file to be included. > That means stats on every component up to and including the file > itself. I can't speak for all the opcode caches, but at least APC > doesn't do anything to alleviate this. > > 2. APC uses the file's device and inode as the lookup key into shared > memory to find the opcodes for the file, so a stat has to be done on > each and every file that is cached. There is currently no such thing > as a request-spanning stat cache. > > So yes, jumping from 20 to 30 include files could very well bring a rather > significant performance hit. > > Anybody looking for real performance out of PHP pretty much has to address > the first issue. It is easy enough to do, simply get rid of the realpath > lookup in that situation and just live with the fact that if someone > includes the same file via different paths it won't catch it. I can see > that it is convenient to have PHP figure this out for you, but the price > of that convenience is not worth it in my opinion. > > -Rasmus > > On Sat, 24 Apr 2004, Alan Knowles wrote: > > >>PEAR is considering stipulating 1 class per file, for the packages. - >>one of the concerns raised is performance. While this is true for a >>non-cache compiled situation (where performance is less of an issue) - >>would anyone care to comment on the implications when using the Zend >>Optimizer or APC, on how significant the impact of this would be. >> >>- looking at typical application where the number of files included in >>the code with may jump from 20->30 included files.. >> >> From my understanding, if you where looking for performance, you would >>probably have a cache expiry which would mean very few stat (if any) >>calls based on these addition includes.. - hence negating the issue.. >> >>Thanks >>Alan >> >> >> >>-- >>Can you help out? >>Need Consulting Services or Know of a Job? >>http://www.akbkhome.com >> >>-- >>PHP Internals - PHP Runtime Development Mailing List >>To unsubscribe, visit: http://www.php.net/unsub.php >> > >
-- Can you help out? Need Consulting Services or Know of a Job? http://www.akbkhome.com

Rasmus Lerdorf

22 years ago
On Sat, 24 Apr 2004, Alan Knowles wrote:
> Thanks - I just had a look through > zend_comple.c:zend_include_or_eval_handler > > it appears that zend_stream_open .. and hence fopen is called on every > require_once/include_once > > would it make sense to add something as simple as > > if (opline->op2.....lval == ZEND_REQUIRE_ONCE) { > locations = get_potential_locations(file,PG('include_path')); > foreach (locations as location) > if (zend_hash_find(&EG(included_files), location)) { > // i've already got it ! > NEXT_OPCODE(); > } > } > }
I am not sure this will help all that much. You are taking a hit on the common case in order to optimize what I believe to be the less common case of a *_once including a file already included. For PEAR it might be common, but outside of PEAR I don't think it is. We still need to do the realpath stats prior to this and for the common codepath the fopen still needs to be done after. Then again, your check may be fast enough that it could be worth adding to speed up the failure case. Overall I think you would get more bang for the buck by coming up with an intelligent caching stat/realpath implementation. The obvious drawback would be that symlinks and other filesystem changes done while the server is running would not be seen until a restart. -Rasmus

Andi Gutmans

22 years ago
At 01:49 AM 4/24/2004 -0700, Rasmus Lerdorf wrote:
>Overall I think you would get more bang for the buck by coming up with an >intelligent caching stat/realpath implementation. The obvious drawback >would be that symlinks and other filesystem changes done while the server >is running would not be seen until a restart.
I also think that the way to solve this problem is to create a realpath cache. I've wanted to do this for ages but never had time to do it. The main reason this is the best approach (in my opinion) is because the fopen() code in PHP is extremely complex and this has gotten much worse with the whole streams implementation. Try and walkthrough an fopen in PHP these days and it's pretty complicated. Also, the fact that we do have file's full path does end up being a very good thing both for functionality and debugging, so I would like to keep it this way. Creating a realpath() cache would solve this problem from a different angle without having to (barely) change any code in PHP nor streams. As Rasmus mentions, the only downside is refresh, but I think that on a loaded web site (which are the web sites we're talking about), even a 1 minute TTL of cache entries could lead to realpath() being negligible. Andi

Michael Walter

22 years ago
Andi Gutmans wrote:
> At 01:49 AM 4/24/2004 -0700, Rasmus Lerdorf wrote: > >> Overall I think you would get more bang for the buck by coming up with an >> intelligent caching stat/realpath implementation. The obvious drawback >> would be that symlinks and other filesystem changes done while the server >> is running would not be seen until a restart. > > > I also think that the way to solve this problem is to create a realpath > cache. I've wanted to do this for ages but never had time to do it. The > main reason this is the best approach (in my opinion) is because the > fopen() code in PHP is extremely complex and this has gotten much worse > with the whole streams implementation. Try and walkthrough an fopen in > PHP these days and it's pretty complicated. Also, the fact that we do > have file's full path does end up being a very good thing both for > functionality and debugging, so I would like to keep it this way. > Creating a realpath() cache would solve this problem from a different > angle without having to (barely) change any code in PHP nor streams. As > Rasmus mentions, the only downside is refresh, but I think that on a > loaded web site (which are the web sites we're talking about), even a 1 > minute TTL of cache entries could lead to realpath() being negligible. > > Andi >
Would these changes be incorporated into PHP4, too? On a related note -- would it be possible (if not already done) to evaluate dirname(__FILE__) at "parse time", in case that gives you a performance advantage? Or is there a better way to do "real" relative includes? Cheers, Michael

Alan Knowles

22 years ago
> Would these changes be incorporated into PHP4, too? > > On a related note -- would it be possible (if not already done) to > evaluate dirname(__FILE__) at "parse time", in case that gives you a > performance advantage? Or is there a better way to do "real" relative > includes?
theoretically, if the file is the normally the first one in the include path, then the difference is negligable. Having had a detail look at APC & the engine, other than the realpath cacheing that rasmus/andi like, I think for pear, if you had an application where stat was becoming an issue, you may be better off a) using the tokenizer, pre-parse all the php code on the machine that is speed critical. Replace all the require/include calls with a extension function call, eg. a theoretical 'apc_include_file()' b) let apc_include_file, check if the file has been pre-compiled (based on the a hash lookup of the filenames) , or call the standard zend_compile code. This would work very well, only assuming you didnt a) dynamicaly create files frequently b) depend on global variables (eg. without $GLOBALS) c) have any kind of strange symlinks Using that prinicple, you could get down to close 1 stat call per page....., however many includes you really had.. Regards Alan
> Cheers, > Michael >
-- Can you help out? Need Consulting Services or Know of a Job? http://www.akbkhome.com

George Schlossnagle

22 years ago
On Apr 24, 2004, at 10:17 AM, Alan Knowles wrote:
> >> Would these changes be incorporated into PHP4, too? >> On a related note -- would it be possible (if not already done) to >> evaluate dirname(__FILE__) at "parse time", in case that gives you a >> performance advantage? Or is there a better way to do "real" relative >> includes? > > theoretically, if the file is the normally the first one in the > include path, then the difference is negligable. > > Having had a detail look at APC & the engine, other than the realpath > cacheing that rasmus/andi like, I think for pear, if you had an > application where stat was becoming an issue, you may be better off > > a) using the tokenizer, pre-parse all the php code on the machine that > is speed critical. Replace all the require/include calls with a > extension function call, eg. a theoretical 'apc_include_file()' > > b) let apc_include_file, check if the file has been pre-compiled > (based on the a hash lookup of the filenames) , or call the standard > zend_compile code.
A nitpick - apc now uses device/inode information and not the file name as hash keys. It's no less or more expensive before, both require an equivalent of realpath() (actually in the apache case the new version is twice as efficient as the old - thanks rasmus). I'm undecided on whether or not this one-file-per-class proposal is wise. On one hand, the number of people the 'slowdown' will be tangible to is really pretty small. You need to be doing a significant amount of requests per second to really feel this pain. On the other hand, the mere perception that it's 'slow' or 'inappropriate for large apps' may be something of a death knell in and of itself. George

Rasmus Lerdorf

22 years ago
On Sat, 24 Apr 2004, George Schlossnagle wrote:
> A nitpick - apc now uses device/inode information and not the file name > as hash keys. It's no less or more expensive before, both require an > equivalent of realpath() (actually in the apache case the new version > is twice as efficient as the old - thanks rasmus).
What do you mean by that? APC doesn't require anything near a realpath. If you get rid of realpath() in PHP, APC will work just fine. With my syscall hacks I am using APC this way and I just have 1 stat per file. The only visible difference is when you ask APC which files it has cached and it doesn't return full-path filenames but rather just the paths used to include the files. For the top file there is no stat in PHP/APC at all anymore as we inherit the stat struct from Apache (which I guess is what you are alluding to) and then for each include file only a single stat is needed. That is nowhere near the equivalent of a realpath which adds N stats per file where N is the depth of the file in the directory structure. If you are really crazy and want the equivalent of this without hacking PHP you would make / your doc_root and put all your files there. And every include would do include ./filename. That would get close. There are still a couple of stray stats in there that aren't strictly needed. There is one in the streams code as well that I hacked out. -Rasmus

George Schlossnagle

22 years ago
On Apr 24, 2004, at 11:43 AM, Rasmus Lerdorf wrote:
> On Sat, 24 Apr 2004, George Schlossnagle wrote: >> A nitpick - apc now uses device/inode information and not the file >> name >> as hash keys. It's no less or more expensive before, both require an >> equivalent of realpath() (actually in the apache case the new version >> is twice as efficient as the old - thanks rasmus). > > What do you mean by that? APC doesn't require anything near a > realpath. > If you get rid of realpath() in PHP, APC will work just fine. With my > syscall hacks I am using APC this way and I just have 1 stat per file. > The only visible difference is when you ask APC which files it has > cached > and it doesn't return full-path filenames but rather just the paths > used > to include the files. > > For the top file there is no stat in PHP/APC at all anymore as we > inherit > the stat struct from Apache (which I guess is what you are alluding to) > and then for each include file only a single stat is needed. That is > nowhere near the equivalent of a realpath which adds N stats per file > where N is the depth of the file in the directory structure.
Yes, that was what I was alluding to. That only works for the top level file though, not any includes. Without that though, you need to call apc_stat_paths(), which does this: paths = apc_tokenize(path, ':'); /* TODO - on windows, it's ';' */ if (!paths) return -1; /* for each directory in paths, look for filename inside */ for (i = 0; paths[i]; i++) { snprintf(filepath, sizeof(filepath), "%s/%s", paths[i], filename); if (stat(filepath, buf) == 0) { found = 1; break; } } And while not a realpath(), it does a similar sort of search (path here is the include_path for those following along at home). Fully qualifying your paths makes up for this, and the next best thing is to make sure your include_path is tight and optimally ordered. George

Rasmus Lerdorf

22 years ago
On Sat, 24 Apr 2004, George Schlossnagle wrote:
> > What do you mean by that? APC doesn't require anything near a > > realpath. > > If you get rid of realpath() in PHP, APC will work just fine. With my > > syscall hacks I am using APC this way and I just have 1 stat per file. > > The only visible difference is when you ask APC which files it has > > cached > > and it doesn't return full-path filenames but rather just the paths > > used > > to include the files. > > > > For the top file there is no stat in PHP/APC at all anymore as we > > inherit > > the stat struct from Apache (which I guess is what you are alluding to) > > and then for each include file only a single stat is needed. That is > > nowhere near the equivalent of a realpath which adds N stats per file > > where N is the depth of the file in the directory structure. > > Yes, that was what I was alluding to. That only works for the top > level file though, not any includes. Without that though, you need to > call apc_stat_paths(), which does this: > > paths = apc_tokenize(path, ':'); /* TODO - on windows, it's ';' > */ > if (!paths) > return -1; > > /* for each directory in paths, look for filename inside */ > for (i = 0; paths[i]; i++) { > snprintf(filepath, sizeof(filepath), "%s/%s", paths[i], > filename); > if (stat(filepath, buf) == 0) { > found = 1; > break; > } > } > > And while not a realpath(), it does a similar sort of search (path here > is the include_path for those following along at home). Fully > qualifying your paths makes up for this, and the next best thing is to > make sure your include_path is tight and optimally ordered.
Well, I disagree that this is anything like a realpath(). If your PEAR directory is the first thing in the include_path (as in get rid of "." from it) then this is a single stat call on any PEAR include regardless of how deep your directory structure is. Compared to realpath where including XML_RSS/RSS.php from a PEAR path of /usr/local/php/pear would be 6 stats. We can't avoid a single stat per file, and on include_path includes we can't avoid a single stat per attempted include_path component. The single stats don't worry me very much and we could reduce this with an intelligent request-spanning stat cache system. -Rasmus

Alan Knowles

22 years ago
>> b) let apc_include_file, check if the file has been pre-compiled >> (based on the a hash lookup of the filenames) , or call the standard >> zend_compile code. > > > A nitpick - apc now uses device/inode information and not the file name > as hash keys. It's no less or more expensive before, both require an > equivalent of realpath() (actually in the apache case the new version is > twice as efficient as the old - thanks rasmus).
yeah - you would need to store an additional fullpath=>inode hash. - hence avoiding stat calls on almost all files, (unless they expire..)
> I'm undecided on whether or not this one-file-per-class proposal is > wise. On one hand, the number of people the 'slowdown' will be tangible > to is really pretty small. You need to be doing a significant amount of > requests per second to really feel this pain. On the other hand, the > mere perception that it's 'slow' or 'inappropriate for large apps' may > be something of a death knell in and of itself.
Which is really why the RFC needs to be backed up by a good indepth discussion of the performance issues and possible solutions (which you and Rasmas do wonderfull job of :) Regards Alan
> George >
-- Can you help out? Need Consulting Services or Know of a Job? http://www.akbkhome.com

Rasmus Lerdorf

22 years ago
On Sun, 25 Apr 2004, Alan Knowles wrote:
> Which is really why the RFC needs to be backed up by a good indepth > discussion of the performance issues and possible solutions (which you > and Rasmas do wonderfull job of :)
To the blackboard with you! Write my name 10,000 times so you will never get it wrong again! -Rasmus

Jeff Moore

22 years ago
On Apr 23, 2004, at 11:37 PM, Alan Knowles wrote:
> PEAR is considering stipulating 1 class per file, for the packages. - > one of the concerns raised is performance. While this is true for a > non-cache compiled situation (where performance is less of an issue) - > would anyone care to comment on the implications when using the Zend > Optimizer or APC, on how significant the impact of this would be.
The Mojavi framework combines 19(?) single class files into a single file to include. I benchmarked this and if I recall correctly, the combined file loaded in roughly half the time of including the individual 19 files. The timing was only for including, no actual work was performed. No cache was used. The benchmarks were done under OS X, which as has been discussed on this list before, has a particularly inefficient implementation of realpath (used in the including process).