FW: Help needed in benchmarking memory patch

php.internals

Andi Gutmans

20 years ago
Sending 3rd time this time without the attachment. You can get the diff at http://gutmans.org/alloc.zip -----Original Message----- From: Andi Gutmans [mailto:andi@zend.com] Sent: Thursday, July 06, 2006 11:06 PM To: 'internals@lists.php.net' Subject: Help needed in benchmarking memory patch Hi all, Attached is a patch we've been working on to improve the memory management of PHP. This patch's main advantage is reducing the memory footprint of PHP, and therefore allowing it to scale better. Although at low concurrencies the change seems to be negligble, while testing at higher concurrencies we saw significant improvement with this patch, mainly due to lower memory usage. We would very much appreciate if people here tested this patch and send in their results. It would help us understand if others are getting consistent results with ours, and also see whether there are any additional improvements we should make. This patch should apply cleanly to the PHP 5.2 CVS tree. There's not much tuning that needs to be done. It uses malloc() to allocate large memory blocks. You can control the size of these blocks by setting the ZEND_MM_SEG_SIZE environment variable before starting Apache/PHP. You may use K or M to play around with the block size. The default we are using is 256KB which seems to be a good balance. Any feedback would be appreciated! Andi

Andi Gutmans

20 years ago
Anyone who'd like to try benchmark this, please drop me an email. We can also discuss how best to test. Thx. Andi

Andi Gutmans

20 years ago
No help??? Come on guys. I'm sure some of you can spare a few idle cycles :)

Michael Vergoz

20 years ago
Hi Andi, I appreciated to see this code which I must still study. Already a small remark it is that the system call mremap() is not portable it is available only on Linux. Then the size of the segments must always "be paginated" - normally your code is good on this level. I think that a concept of kernel-memory and user-memory can be very interesting to study, especially in threads environments - that I use. I would continue to look at your code but now I will sleep (04:26 pm)! mv- ----- Original Message ----- From: "Andi Gutmans" <andi@zend.com> To: "'Andi Gutmans'" <andi@zend.com>; <internals@lists.php.net> Sent: Monday, July 10, 2006 5:57 PM Subject: RE: [PHP-DEV] FW: Help needed in benchmarking memory patch

Andi Gutmans

20 years ago
Sorry I didn't make clear that only the malloc()/free() part works. The mmap() and Win32 Allocation code is commented. We didn't find a material difference between malloc() and mmap() though, most probably because malloc() at these sizes actually calls mmap(). So really most important is for some ppl to run some benchmarks on this. Thx. Andi

Michael Vergoz

20 years ago
Hi aAndy
> We didn't find a material difference between malloc() and mmap() though, > most probably because malloc() at these sizes actually calls mmap().
You can have good difference with the mremap() of Linux because it directly changes the vector into memory without recopying these data.
-- Before beginning a benchmark I would like to speak about a serious problem that I have with TSRM/MM. A S1 script carried out by thread T1 will use the same memory capacity as S2 script by T2 whereas theoretically separation is obligatory. In addition to the problem of safety, that cause also a problem of management of the load memory of the process father. We define a father process P=while(1); When a process father starts it must initialize tsrm_startup(), sapi_startup() and php_module_startup(). At this time all the additional functions and objects are loaded and usable by all script (what is normal). With the creation of a thread one associates a TSRMLS to it. Then one carries out php_request_startup(), php_execute_script() and php_request_shutdown() to entirely enclose PHP/Zend. With the execution of Sx script in Tx: the variables dynamics, static and constant of this one will be stored in AG(head). The problem comes from there. It is impossible to destroy a memory which a script has just used because the process father continues to turn and nor php_request_shutdown(), php_module_shutdown(), sapi_shutdown() and tsrm_shutdown() will not be carried out AND THUS there will be a mem leak I am very annoyed by this problem, it is necessary that I find a possibility to destroy these buffer. Michael Vergoz ----- Original Message ----- From: "Andi Gutmans" <andi@zend.com> To: "'Michael Vergoz'" <mv-php@binarysec.com>; <internals@lists.php.net> Sent: Tuesday, July 11, 2006 8:26 AM Subject: RE: [PHP-DEV] FW: Help needed in benchmarking memory patch

Dmitry Stogov

20 years ago
Hi Michael, The patch was tested in ZTS mode too. What SAPI you are talking about? (I would like to reproduce the problem). Thanks. Dmitry.

Michael Vergoz

20 years ago
In fact I had found some leaks it would seem that ts_free_thread() after php_request_shutdown() solves all the problem. I am checking if it's bad but if not the case the SAPI ISAPI should have a small modification. mv- ----- Original Message ----- From: "Dmitry Stogov" <dmitry@zend.com> To: "'Michael Vergoz'" <mv-php@binarysec.com>; "'Andi Gutmans'" <andi@zend.com>; <internals@lists.php.net> Sent: Tuesday, July 11, 2006 1:54 PM Subject: RE: [PHP-DEV] FW: Help needed in benchmarking memory patch

Andi Gutmans

20 years ago
Are you sure this isn't because the IIS thread is still running? One thread handles multiple requests.

Michael Vergoz

20 years ago
Salut Andi & Dmitry, If you have errors in your script you have a leak in PG(last_error_message) & PG(last_error_file) because of php_error_cb() which allocates via strdup() these two variables. It should be noted also that these variables are not given with NULL after their freed in php_error_cb() - main/main.c +679 and that both last allowance will never be freed for each thread. They are freed with the stop of the process via php_module_shutdown(). I propose to free these buffer in php_request_shutdown()? In fact the leak more are the small ones but they become repetitive in a Threaded program. There is also another condition to generate large mem leak for example by re-using a thread to carry out several script with the continuation. There you explodes all the records if you do not do a ts_free_thread() just after your php_request_shutdown(). It's for that which I think that it would be very judicious to add a ts_free_thread() before each the return() of the HttpExtensionProc() function of the ISAPI SAPI module. Michael Vergoz ----- Original Message ----- From: "Andi Gutmans" <andi@zend.com> To: "'Michael Vergoz'" <mv-php@binarysec.com>; "'Dmitry Stogov'" <dmitry@zend.com>; "'PHPdev'" <internals@lists.php.net> Sent: Tuesday, July 11, 2006 7:38 PM Subject: RE: [PHP-DEV] FW: Help needed in benchmarking memory patch