Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Generating arginfo from stub files »

Generating arginfo from stub files

php.internals

Nikita Popov

7 years ago
Hi internals, In PHP 8 it will be possible to add reflectible argument and return type information for internal functions (it was previously theoretically possible as well, but forbidden by policy for php-src for multiple reasons, which have now been resolved). It will take quite a bit of effort to add this information for hundreds of builtin functions. I would like to take this chance to improve the way in which arginfo structures are specified, and make it more ergonomic and future proof. Here is an example of a typed arginfo structure: ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2, _IS_BOOL, 0) ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0) ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0) ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0) ZEND_END_ARG_INFO() Rather than writing this out by hand, I would like arginfo structures to be generated from PHP stub files that contain definitions like this: function class_alias(string $user_class_name, string $alias_name, bool $autoload = true): bool {} I've created a proof of concept implementation for this at https://github.com/php/php-src/pull/4284. Function signatures are specified in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can then be included in the implementation. Nothing about the arginfo implementation itself changes. What do you think about providing this mechanism? Nikita

Dan Ackroyd

7 years ago
On Tue, 18 Jun 2019 at 16:10, Nikita Popov <nikita.ppv@gmail.com> wrote:
> Rather than writing this out by hand, I would like arginfo structures to be > generated from PHP stub files that contain definitions like this: > > What do you think about providing this mechanism?
Sounds good. But also, it's really a shame we don't have the union types RFC passed yet, as it means that the return type for a significant number of functions will be either incomplete or misleading. strpos(string $haystack, $needle, int $offset = 0) This is incomplete as the return isn't specified. strpos(string $haystack, $needle, int $offset = 0) : int This is misleading as it can return false; strpos(string $haystack, $needle, int $offset = 0): int|false Yay!, tools like PhpStan or Psalm can use this info, and don't need to maintain their own info for internal functions. Having the 'mixed' type would also be useful here, as there will almost always be some things that can't be expressed in a type system, and being able to use mixed shows that the type wasn't forgotten about, it just can be expressed currently. cheers Dan Ack https://wiki.php.net/rfc/union_types https://wiki.php.net/rfc/mixed-typehint

Benjamin Morel

7 years ago
> > But also, it's really a shame we don't have the union types RFC passed > yet, as it means that the return type for a significant number of > functions will be either incomplete or misleading. > >
Agreed.

Rowan Collins

7 years ago
Hi Nikita, On Tue, 18 Jun 2019 at 16:10, Nikita Popov <nikita.ppv@gmail.com> wrote:
> I've created a proof of concept implementation for this at > https://github.com/php/php-src/pull/4284. Function signatures are > specified > in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can > then be included in the implementation. Nothing about the arginfo > implementation itself changes. >
I notice on the PR there was discussion of stubs elsewhere (e.g. https://github.com/JetBrains/phpstorm-stubs) still being necessary because they contain additional documentation. However, has the opposite been considered: process an existing set of stubs to remove unsupported or irrelevant annotations, and import into php-src with this tool? That way, the manual effort for existing functions would be virtually zero. Regards,
-- Rowan Collins [IMSoP]

Nikita Popov

7 years ago
On Tue, Jun 18, 2019 at 7:38 PM Rowan Collins <rowan.collins@gmail.com> wrote:
> Hi Nikita, > > On Tue, 18 Jun 2019 at 16:10, Nikita Popov <nikita.ppv@gmail.com> wrote: > >> I've created a proof of concept implementation for this at >> https://github.com/php/php-src/pull/4284. Function signatures are >> specified >> in a xyz.stub.php file from which xyz_arginfo.h is generated. This file >> can >> then be included in the implementation. Nothing about the arginfo >> implementation itself changes. >> > > > I notice on the PR there was discussion of stubs elsewhere (e.g. > https://github.com/JetBrains/phpstorm-stubs) still being necessary > because they contain additional documentation. However, has the opposite > been considered: process an existing set of stubs to remove unsupported or > irrelevant annotations, and import into php-src with this tool? That way, > the manual effort for existing functions would be virtually zero. >
This is not really possible, because we need completely accurate type information. phpstorm-stubs is a good approximation especially when it comes to "useful" return types, but it doesn't model things like possible "false" return values with sufficient accuracy. Any type information we add in php-src needs a careful review against the implementation. Nikita

Mark Randall

7 years ago
On 18/06/2019 18:53, Nikita Popov wrote:
> This is not really possible, because we need completely accurate type > information. phpstorm-stubs is a good approximation especially when it > comes to "useful" return types, but it doesn't model things like possible > "false" return values with sufficient accuracy. Any type information we add > in php-src needs a careful review against the implementation.
I say this with a certain amount of ignorance, but what about the existing manual? As the authoritative source of user-facing documentation, does it not contain an XML parseable signature for every method, in addition to a return value section detailing types and a standard boilerplate for things with falseable return values? The main issue I would see is "mixed" argument types in the signature which would require manual intervention. I'd certainly like to see a more comprehensive format. Stubs are great but stubs seem more like the end result rather than the original metadata. An XML or JSON file capable of expressing the names, types, defaults etc would have the added benefit of being easily extensible to cover things like descriptions or unions with different requirements for each type. Well-docblock'd stubs, C definitions, documentation pages etc would then be the drawn from the JSON rather than the other way.
-- Mark Randall

Nikita Popov

7 years ago
On Tue, Jun 18, 2019 at 5:10 PM Nikita Popov <nikita.ppv@gmail.com> wrote:
> Hi internals, > > In PHP 8 it will be possible to add reflectible argument and return type > information for internal functions (it was previously theoretically > possible as well, but forbidden by policy for php-src for multiple reasons, > which have now been resolved). > > It will take quite a bit of effort to add this information for hundreds of > builtin functions. I would like to take this chance to improve the way in > which arginfo structures are specified, and make it more ergonomic and > future proof. Here is an example of a typed arginfo structure: > > ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2, > _IS_BOOL, 0) > ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0) > ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0) > ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0) > ZEND_END_ARG_INFO() > > Rather than writing this out by hand, I would like arginfo structures to > be generated from PHP stub files that contain definitions like this: > > function class_alias(string $user_class_name, string $alias_name, bool > $autoload = true): bool {} > > I've created a proof of concept implementation for this at > https://github.com/php/php-src/pull/4284. Function signatures are > specified in a xyz.stub.php file from which xyz_arginfo.h is generated. > This file can then be included in the implementation. Nothing about the > arginfo implementation itself changes. > > What do you think about providing this mechanism? > > Nikita >
Any more feedback on this? The existing discussion has been mostly around "can we get the type information from some existing source?" to which the answer is "nope": Nothing that already exists is accurate enough for php-src. Nikita

Marco Pivetta

7 years ago
On Tue, Jul 23, 2019 at 5:00 PM Nikita Popov <nikita.ppv@gmail.com> wrote:
> On Tue, Jun 18, 2019 at 5:10 PM Nikita Popov <nikita.ppv@gmail.com> wrote: > > > Hi internals, > > > > In PHP 8 it will be possible to add reflectible argument and return type > > information for internal functions (it was previously theoretically > > possible as well, but forbidden by policy for php-src for multiple > reasons, > > which have now been resolved). > > > > It will take quite a bit of effort to add this information for hundreds > of > > builtin functions. I would like to take this chance to improve the way in > > which arginfo structures are specified, and make it more ergonomic and > > future proof. Here is an example of a typed arginfo structure: > > > > ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2, > > _IS_BOOL, 0) > > ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0) > > ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0) > > ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0) > > ZEND_END_ARG_INFO() > > > > Rather than writing this out by hand, I would like arginfo structures to > > be generated from PHP stub files that contain definitions like this: > > > > function class_alias(string $user_class_name, string $alias_name, bool > > $autoload = true): bool {} > > > > I've created a proof of concept implementation for this at > > https://github.com/php/php-src/pull/4284. Function signatures are > > specified in a xyz.stub.php file from which xyz_arginfo.h is generated. > > This file can then be included in the implementation. Nothing about the > > arginfo implementation itself changes. > > > > What do you think about providing this mechanism? > > > > Nikita > > > > Any more feedback on this? > > The existing discussion has been mostly around "can we get the type > information from some existing source?" to which the answer is "nope": > Nothing that already exists is accurate enough for php-src. > > Nikita >
Would it be too aggressive to have (in the php-src test suite, as a first step) something that ensures that a stub exists for each exposed userland symbol? That would ensure that any newly added symbol or core extension has the matching userland reflectable-ish structure, which later (hopefully, in the years) could lead to having proper reflection on internal symbols... Marco Pivetta http://twitter.com/Ocramius http://ocramius.github.com/

Mark Randall

7 years ago
On 23/07/2019 16:05, Marco Pivetta wrote:
> Would it be too aggressive to have (in the php-src test suite, as a first > step) something that ensures that a stub exists for each exposed userland > symbol?
That's an interesting point, at the risk of flogging a dead horse with my ignorance of internals, what about using the unit tests as a means of enumerating the arguments, their names and their return types? If there's nothing that meets the requirements of php-src, what about php-src itself? As almost all arg are handled via zpp, and almost all the return values are handled by RETURN_XXX etc, what about a one-off build that modifies those macros to report their usage to a global state, which can be read back after and used to build the docs? Something like a dozen years ago, I wrote a scripting language in C++ for my undergraduate thesis, and hit across a similar problem. I too used macros to define functions arguments, what their types were, copied them into native variables and so forth. When it came to documenting all the functions I was quickly losing my sanity, and so modified my macros so that if ran with a certain define active, they would dump all of their information into a global store that I could dump after. Obviously PHP's use of quasi-overloading doesn't help its situation here, and Z_PARAM_ZVAL and friends could cause quite a headache, as it would need to track what types were extracted through such macros as Z_LVAL_XX. Certainly not a catch-all situation, but might present an opportunity for doing some of the leg work automatically. Mark Randall

Christoph Becker

7 years ago
On 18.06.2019 at 17:10, Nikita Popov wrote:
> Hi internals, > > In PHP 8 it will be possible to add reflectible argument and return type > information for internal functions (it was previously theoretically > possible as well, but forbidden by policy for php-src for multiple reasons, > which have now been resolved). > > It will take quite a bit of effort to add this information for hundreds of > builtin functions. I would like to take this chance to improve the way in > which arginfo structures are specified, and make it more ergonomic and > future proof. Here is an example of a typed arginfo structure: > > ZEND_BEGIN_ARG_WITH_RETURN_TYPE_INFO_EX(arginfo_class_alias, 0, 2, > _IS_BOOL, 0) > ZEND_ARG_TYPE_INFO(0, user_class_name, IS_STRING, 0) > ZEND_ARG_TYPE_INFO(0, alias_name, IS_STRING, 0) > ZEND_ARG_TYPE_INFO(0, autoload, _IS_BOOL, 0) > ZEND_END_ARG_INFO() > > Rather than writing this out by hand, I would like arginfo structures to be > generated from PHP stub files that contain definitions like this: > > function class_alias(string $user_class_name, string $alias_name, bool > $autoload = true): bool {} > > I've created a proof of concept implementation for this at > https://github.com/php/php-src/pull/4284. Function signatures are specified > in a xyz.stub.php file from which xyz_arginfo.h is generated. This file can > then be included in the implementation. Nothing about the arginfo > implementation itself changes. > > What do you think about providing this mechanism?
I like it. Thanks! Christoph