Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Benchmarking 4.3.2 vs 4.1.2 »

Benchmarking 4.3.2 vs 4.1.2

php.internals

Jeff Moore

23 years ago
I remember a discussion about system calls here earlier. What is the status of that? I've been doing some tests and have found that this code runs about 2.5 times slower under 4.3.2 than it did on 4.1.2 on the same machine (running OS X): <?php $filename = 'file.html'; $fd = fopen($filename, 'rb'); $content = fread($fd, filesize($filename)); fclose($fd); echo $content; ?> I did some tests with fs_usage to check file system calls and found that for a zero length test.php file, 4.3.2 makes the following additional calls beyond 4.1.2: 1 chdir /Library/WebServer/Documents/rot/bench 1 fchdir 12 fstat 5 fstatfs 5 getdirentries 7 lseek O=0x00000000 1 lstat test.php 6 lstat (., .., ../.., etc. 6 open (., .., ../.., etc) 1 stat The fread example above made the following additional calls in 4.3.2 beyond 4.1.2: 2 chdir /Library/WebServer/Documents/rot/bench 2 fchdir 32 fstat 15 fstatfs 15 getdirentries 16 lseek O=0x00000000 1 lstat file.html 1 lstat test.php 18 lstat (., .., ../.., etc.) 17 open (., .., ../.., etc) 4 stat 1 stat . some of my counts my be slightly off - I counted them by hand.

Rasmus Lerdorf

23 years ago
Uh, the number of open() calls really shouldn't change. Are you sure you are comparing the same code? And how are you testing? Attach strace or truss to your running httpd process and hit your script a single time. Then look at the output and compare them. Some of the keys to reducing syscalls: - don't use open_basedir or safe_mode restrictions - always include/require files by their absolute paths (./foo.inc is ok) - if you have to use include_path includes, make sure the dir you hit most often is first in the include_path (even before .) - optimize your code. ie. don't use while(!feof($fp)) to loop through a file, check the return of your fgets() instead, for example. In your example, you can of course replace your code with a single call to file_get_contents() which is a new function in 4.3 and should significantly reduce your syscall overhead. 4.3.3 will have Sascha's fd-patch which gets rid of a stat() call associated with every file open, so if you are really keen on reducing your syscalls even further, you can grab a snapshot. I also have some hacks we use here at Yahoo that eliminate a bunch of other calls, but they aren't exactly generic in that they break various aspects of PHP that in my controlled environment here I am fine with. I have been pondering whether to add some sort of --broken-but-fast compile switch an commit those changes, but it would likely cause a whole lot of confusion. -Rasmus On Wed, 4 Jun 2003, Jeff Moore wrote:

Ilia A.

23 years ago
Here is an analysis of the situation as it stands with PHP 4.3.3. As far as I can tell that even when opening files with the full path we do a lot of completely unnecessary work. We start from expand_filepath() which, gets called when opening a file. This function does a getcwd() everytime it is called even if path is full (/path/to/script.php), as far as I can tell completely useless syscal. Inside virtual_file_ex(), which gets called by expand_filepath(), even when the path is full we tokenize the path and validate every directory for '.' & '..' instead of just checking for . + DIR_SEPARATOR and only doing this work if such a thing exists. Quite a bit of completely useless cpu trashing in most cases everytime we open a file. Now, the interesting part, which I've yet to track down, are 6 lstat64() calls that occur between getcwd() (look above) and the actual open() for every file opened by PHP regardless of whether it had a full path or not. Here is strace bit: getcwd("/home/rei/PHP_CVS/STABLE/php4", 4096) = 30 <0.000011> lstat64(0xbfffad30, 0xbfffac90) = 0 <0.000015> lstat64(0xbfffad30, 0xbfffac90) = 0 <0.000008> lstat64(0xbfffad30, 0xbfffac90) = 0 <0.000008> lstat64(0xbfffad30, 0xbfffac90) = 0 <0.000008> lstat64(0xbfffad30, 0xbfffac90) = 0 <0.000008> lstat64(0xbfffad30, 0xbfffac90) = 0 <0.000009> open("/home/rei/PHP_CVS/STABLE/php4/file.html", O_RDONLY) = 4 <0.000018> Ilia P.S. I used the term 'unnecessary' because removing the mention code does not seem to affect the script's behaviour only makes it work faster.

Brian J. France

23 years ago
I believe those are from the realpath call. It splits the path and stats each part to make sure it is real: 1) / 2) /home 3) /home/rei 4) /home/rei/PHP_CVS 5) /home/rei/PHP_CVS/STABLE 6) /home/rei/PHP_CVS/STABLE/php4 Brian On Wed, 4 Jun 2003 13:27:14 -0400, Ilia A. wrote:

Rasmus Lerdorf

23 years ago
On Wed, 4 Jun 2003, Ilia A. wrote:
> Here is an analysis of the situation as it stands with PHP 4.3.3. As far as I > can tell that even when opening files with the full path we do a lot of > completely unnecessary work. > > We start from expand_filepath() which, gets called when opening a file. This > function does a getcwd() everytime it is called even if path is full > (/path/to/script.php), as far as I can tell completely useless syscal.
Yeah, I have killed all that crap here. My expand_filepath() looks like this: int len = strlen(filepath); if(filepath[len-1] == PHP_DIR_SEPARATOR) len--; if(!real_path) real_path = estrdup(filepath); else strcpy(real_path, filepath); real_path[len]='\0'; return real_path; Which of course breaks some stuff. I also hacked up the following: - got rid of the pipe check in _php_stream_fopen_from_file() - the fstat() in the sanity check in _php_stream_fopen() - the seek stuff in _php_stream_cast()
> Now, the interesting part, which I've yet to track down, are 6 lstat64() calls > that occur between getcwd() (look above) and the actual open() for every file > opened by PHP regardless of whether it had a full path or not.
Looks like a realpath() call to me. -Rasmus

Jeff Moore

23 years ago
On Wednesday, June 4, 2003, at 12:21 PM, Rasmus Lerdorf wrote:
> Uh, the number of open() calls really shouldn't change. Are you sure > you > are comparing the same code?
Yes. I am sure.
> And how are you testing?
I attach fs_usage to the running httpd process and hit the script a single time. http://developer.apple.com/techpubs/macosx/Essentials/Performance/ PerformanceTools/fs_usage_Fi_alysis_Tool.html Here is the full trace for accessing a zero length file called control.php: (notice how each directory gets opened three times. in 4.1.2 each directory gets opened twice.) read F=3 B=0x1c5 stat /Library/WebServer/Documents/rot/bench/control.php open [ 2] /Library/WebServer/Documents/rot/.htaccess open [ 2] /Library/WebServer/Documents/rot/bench/.htaccess stat stat . chdir /Library/WebServer/Documents/rot/bench open F=4 . chdir /Library/WebServer/Documents/rot/bench lstat control.php stat lstat . open F=5 .. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x68 lseek F=5 O=0x00000000 close F=5 lstat .. open F=5 ../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x208 lseek F=5 O=0x00000000 close F=5 lstat ../.. open F=5 ../../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x58 lseek F=5 O=0x00000000 close F=5 lstat ../../.. open F=5 ../../../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x2ac lseek F=5 O=0x00000000 close F=5 lstat ../../../.. open F=5 ../../../../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x218 lseek F=5 O=0x00000000 close F=5 lstat ../../../../.. fchdir F=4 close F=4 stat stat . stat lstat . open F=4 .. fstat F=4 fstatfs F=4 fstat F=4 getdirentries F=4 B=0x68 lseek F=4 O=0x00000000 close F=4 lstat .. open F=4 ../.. fstat F=4 fstatfs F=4 fstat F=4 getdirentries F=4 B=0x208 lseek F=4 O=0x00000000 close F=4 lstat ../.. open F=4 ../../.. fstat F=4 fstatfs F=4 fstat F=4 getdirentries F=4 B=0x58 lseek F=4 O=0x00000000 close F=4 lstat ../../.. open F=4 ../../../.. fstat F=4 fstatfs F=4 fstat F=4 getdirentries F=4 B=0x2ac lseek F=4 O=0x00000000 close F=4 lstat ../../../.. open F=4 ../../../../.. fstat F=4 fstatfs F=4 fstat F=4 getdirentries F=4 B=0x218 lseek F=4 O=0x00000000 close F=4 lstat ../../../../.. open F=4 . chdir /Library/WebServer/Documents/rot/bench lstat control.php stat lstat . open F=5 .. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x68 lseek F=5 O=0x00000000 close F=5 lstat .. open F=5 ../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x208 lseek F=5 O=0x00000000 close F=5 lstat ../.. open F=5 ../../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x58 lseek F=5 O=0x00000000 close F=5 lstat ../../.. open F=5 ../../../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x2ac lseek F=5 O=0x00000000 close F=5 lstat ../../../.. open F=5 ../../../../.. fstat F=5 fstatfs F=5 fstat F=5 getdirentries F=5 B=0x218 lseek F=5 O=0x00000000 close F=5 lstat ../../../../.. fchdir F=4 close F=4 open F=4 /Library/WebServer/Documents/rot/bench/control.php fstat F=4 fstat F=4 lseek F=4 O=0x00000000 lseek F=4 O=0x00000000 fstat F=4 read F=4 B=0x0 close F=4 chdir lstat /usr/share/zoneinfo/GMT open F=4 /usr/share/zoneinfo/GMT fstat F=4 read F=4 B=0x38 close F=4 write F=3 B=0xee lstat private/etc/localtime open F=4 /usr/share/zoneinfo/America/Detroit fstat F=4 read F=4 B=0x32b close F=4 write F=17 B=0x9a read F=3 B=0x0 close F=3

Rasmus Lerdorf

23 years ago
Oh, this is on OSX? I guess realpath() does an open() on every component of a path on OSX? That's nuts! -Rasmus On Wed, 4 Jun 2003, Jeff Moore wrote:

Andi Gutmans

23 years ago
At 01:27 PM 6/4/2003 -0400, Ilia A. wrote:
>Here is an analysis of the situation as it stands with PHP 4.3.3. As far as I >can tell that even when opening files with the full path we do a lot of >completely unnecessary work. > >We start from expand_filepath() which, gets called when opening a file. This >function does a getcwd() everytime it is called even if path is full >(/path/to/script.php), as far as I can tell completely useless syscal.
Two different full paths can point to the same file and not using realpath() would break include_once() and require_once(). We might be able to use stat() information to compare the files but it would require some thinking. Andi

George Schlossnagle

23 years ago
On Monday, June 9, 2003, at 12:58 PM, Andi Gutmans wrote:
> At 01:27 PM 6/4/2003 -0400, Ilia A. wrote: >> Here is an analysis of the situation as it stands with PHP 4.3.3. As >> far as I >> can tell that even when opening files with the full path we do a lot >> of >> completely unnecessary work. >> >> We start from expand_filepath() which, gets called when opening a >> file. This >> function does a getcwd() everytime it is called even if path is full >> (/path/to/script.php), as far as I can tell completely useless syscal. > > Two different full paths can point to the same file and not using > realpath() would break include_once() and require_once(). > We might be able to use stat() information to compare the files but it > would require some thinking.
APC uses inode number, device number. I'm not sure if that is portable to win32 though. George

Ilia A.

23 years ago
On June 9, 2003 12:58 pm, Andi Gutmans wrote:
> At 01:27 PM 6/4/2003 -0400, Ilia A. wrote: > >Here is an analysis of the situation as it stands with PHP 4.3.3. As far > > as I can tell that even when opening files with the full path we do a lot > > of completely unnecessary work. > > > >We start from expand_filepath() which, gets called when opening a file. > > This function does a getcwd() everytime it is called even if path is full > > (/path/to/script.php), as far as I can tell completely useless syscal. > > Two different full paths can point to the same file and not using > realpath() would break include_once() and require_once(). > We might be able to use stat() information to compare the files but it > would require some thinking.
As far as realpath() goes beyond the _once() directives and php installations where safe_mode or open_basedir are enabled it is not really needed. To handle safe_mode/open_basedir is fairly easy and mostly involves passing TSRMLS_CC to virtual_file_ex() and then putting realpath related code inside if (PG(safe_mode) || PG(open_basedir)) {. I already have a patch for this. I wonder if instead of comming up with an alternative way, which is likely to involve stat() or lstat(). We could add a flag that would allow us to identify *_once requests and make realpath conditional on that. Ilia

Rasmus Lerdorf

23 years ago
On Tue, 10 Jun 2003, Ilia A. wrote:
> As far as realpath() goes beyond the _once() directives and php installations > where safe_mode or open_basedir are enabled it is not really needed. To > handle safe_mode/open_basedir is fairly easy and mostly involves passing > TSRMLS_CC to virtual_file_ex() and then putting realpath related code inside > if (PG(safe_mode) || PG(open_basedir)) {. I already have a patch for this. I > wonder if instead of comming up with an alternative way, which is likely to > involve stat() or lstat(). We could add a flag that would allow us to > identify *_once requests and make realpath conditional on that.
The main issue here is that we do an implicit require_once on the main script now. If you remove the realpath() on that main script load, then a subsequent include_once on the same file through the actual path, or a different symlinked path will not be caught. Personally I think this is way too much of an edge case to warrant the horribly expensive realpath() call on every single request and I am perfectly fine simply telling people not to do that. We lived with this situation for a long time and it was only recently someone "fixed" this. -Rasmus

Andi Gutmans

23 years ago
At 06:55 PM 6/10/2003 -0700, Rasmus Lerdorf wrote:
>On Tue, 10 Jun 2003, Ilia A. wrote: > > As far as realpath() goes beyond the _once() directives and php > installations > > where safe_mode or open_basedir are enabled it is not really needed. To > > handle safe_mode/open_basedir is fairly easy and mostly involves passing > > TSRMLS_CC to virtual_file_ex() and then putting realpath related code > inside > > if (PG(safe_mode) || PG(open_basedir)) {. I already have a patch for > this. I > > wonder if instead of comming up with an alternative way, which is likely to > > involve stat() or lstat(). We could add a flag that would allow us to > > identify *_once requests and make realpath conditional on that. > >The main issue here is that we do an implicit require_once on the main >script now. If you remove the realpath() on that main script load, then a >subsequent include_once on the same file through the actual path, or a >different symlinked path will not be caught. > >Personally I think this is way too much of an edge case to warrant the >horribly expensive realpath() call on every single request and I am >perfectly fine simply telling people not to do that. We lived with this >situation for a long time and it was only recently someone "fixed" this.
Personally I think the way to fix this is to have a realpath cache and not to break functionality. This could be implemented relatively easily and would give you the speed you want without breaking anything. Andi

Tom Sommer

23 years ago
Jeff Moore wrote:
> I remember a discussion about system calls here earlier. What is the > status of that? > > I've been doing some tests and have found that this code runs about 2.5 > times slower under 4.3.2 than it did on 4.1.2 on the same machine > (running OS X):
2.5? That's really bad... I know Rasmus is very interested in getting rid of System Calls :) He discovered that they do a lot of damage on performance, I don't know how far he is or what changes he has made. For what I've read, performance *has* dropped since 4.1.2 (or 4.2.3), would be nice if that could be improved :)
-- Tom Sommer, Denmark www.dreamcoder.dk - www.tsn.dk