YANP (Yet Another Namespace Proposal)

php.internals

Jessie Hernandez

21 years ago
I have locally made changes to support namespaces, with minimal impact to the Zend Engine (I based my changes on PHP 5.1.0b2). I have gone through the archives and read the previous discussions about namespaces and the outstanding issues, and to the best of my knowledge, I think my changes solve these. I have only a few things to finish (as I'll explain below), but so far everything's working great (I have only enabled classes to be under namespaces, since that's all I care about, but functions and constants can be added in a similar manner)! Since I still have some things to finish, I won't post a patch yet, but I'll give a high-level description of what I did to see if my approach is acceptable or not. Basically, my approach involves the following concepts: - Classes may not contain colons (no change here), BUT class name references CAN contain colons (this is KEY!). - All imports are done at compile time. - Imports only affect the current file (yes, this has been done!) Here are some more details: Class Table Manipulation ------------------------ When the start of a namespace declaration is encountered, the namespace name is saved in a compile global variable. All classes inside the namespace block will get the namespace name and the colon prepended to the name (this is done in "zend_do_begin_class_declaration"). So, for the following code: <code> namespace my_namespace{ class my_class{ public $mem; } } </code> What will actually happen here is that the class will get added to the class table as "my_namespace:my_class". Referencing Classes ------------------- The following two examples are valid: <code> $a = new my_namespace:myclass(); </code> <code> import my_namespace:myclass; $a = new myclass(); </code> Imports ------- The syntax for the import statement will be: import <full-class-name> [as <alias-name>]; I have not yet added full namespace imports ("wildcard" imports), and am not yet sure if this is a good idea. Anyways, with the import statement, if no alias is specified, then the class is imported with the remaining string after the last colon in the name, and if an alias is specified, then that name is used (aliases cannot contain colons). For example: <code> import my_namespace:myclass; $a = new myclass(); </code> <code> import my_namespace:myclass as my_namespace_my_class; $a = new my_namespace_myclass(); </code> When an import is done, what happens is that the import is saved in an import hashtable, which holds the "aliases" which are active for the current file. The alias is also saved in class_table (an additional entry is added, the key being the alias name and the value being the zend_class_entry for the actual class). This means that a zend_class_entry can belong to more than one key in class_table at a time (this is done to emulate that a class by that name actually exists, and it also prevents classes in the global namespace to be named the same as import aliases). Now for the interesting part! How are imports done on a file scope only? The key to this is that file scopes can be emulated at compile time, but not at run time. As I said before, imports are saved in a compile global hashtable. When the end of the file is reached, this hashtable is cleared and the alias entries are removed from class_table. This is done simply by modifying the start rule: start: top_statement_list { zend_do_unimport_all(TSRMLS_C); } ; This triggers the "zend_do_unimport_all" function to be called at the end of the file. At this point (before the function is called), all references to imported names have been changed internally to full class names, which include the namespace and colon(s). Since include/require statements are executed at runtime, when the "zend_compile_file" function is executed for the included files, the import hashtable will be empty! I have not yet begun doing this, but I would like the import statement to actually include the file where the class is defined. A new php.ini variable called "class_path" will be used for this. When an "import my_ns:my_class;" statement is encountered, the file "<class-path>/my_ns/my_class.php" will be included (of course, multiple directories can be specified in the class_path, just like include_path). The above changes make a minimal impact to the Zend engine, and since all the namespace/import translation is done at compile time, execution speed is not affected. I would like to gather comments/suggestions/criticisms of the above approach, in order to see if I should continue making these changes or not. If there is enough demand, I will post the patches of what I have so far, even though it's not complete. Regards, Jessie Hernandez

David Zülke

21 years ago
Sounds sweet. However, I already found a problem there, it has to do with aliasing. Lots of code (libraries) will, even inside the same class, use the class name for method calls instead of the "self" keywords for various reasons. This will break if the user decides to alias the class and use it, I guess. Other than that, I don't see any problems so far. Does a My:Nested:Namespace:Class::staticMethod() work? Nice work man, keep it up, let's hope we can finally have some namespaces/packaging support in PHP ;) I wish you luck. +1 from me. - David P.S: check out http://article.gmane.org/gmane.comp.php.devel/29492, might be of help, dunno

David Zülke

21 years ago
Oh and to those who are just about to shout "clashes with ternary operator": $bar = new $foo ? Foo : Bar; doesn't work in PHP. - David

Jessie Hernandez

21 years ago
David, You are right, classes that don't use self might break, although there might be a solution for this: when a class declaration is started, immediately "import" the class. This will fix the problem, but I don't really think it's necessary. After all, if you're using an new feature as namespaces for your classes, shouldn't you also be using the new self:: keyword? To answer your question, yes, "My:Nested:Namespace:Class::staticMethod()" should work! In the test I posted in my patch post, I echo a static variable in the class, my_ns:class2::my_static, and it works great!
-- Jessie David Zülke wrote:

Timm Friebe

21 years ago
[...]
> - All imports are done at compile time.
So: == Object.php == <?php package lang { class Object { }} ?> == script.php == <?php include('Object.php'); import lang:Object; ?> ...will not work. You'll need an opcode for this. - Timm

Jessie Hernandez

21 years ago
Actually, this will work! In the "class_name_reference" rule, the "zend_do_fetch_class" function is called. Inside this function, class_name is a znode, and what is assigned to the opcode is "opline->op2 = *class_name". This function is changed to check the import hashtable, and if a match is found, then the znode string value is updated to contain the full class name! Here's some pseudocode: <?php include('Object.php'); // include opcode is generated here import lang:Object; // "Object" => "lang:Object" added to import hashtable $o = new Object(); // zend_do_fetch_class checks for "Object" in the import hashtable. // A match is found, so "class_name->u.constant.value.str" is modified from "Object" to "lang:Object". // At runtime, the lang:Object class will be created and added to class_table, // and since all opcodes are using lang:Object instead of simply Object, then everything will work. ?>
-- Jessie ""Timm Friebe"" <thekid@thekid.de> wrote in message news:0MKwh2-1DqTBy43Wu-0004sH@mrelayeu.kundenserver.de...

Stanislav Malyshev

21 years ago
JH>>is a znode, and what is assigned to the opcode is "opline->op2 = JH>>*class_name". This function is changed to check the import hashtable, and if JH>>a match is found, then the znode string value is updated to contain the full JH>>class name! Here's some pseudocode: This of course won't work with variable class names.
-- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/ +972-3-6139665 ext.115

Jessie Hernandez

21 years ago
Stanislav, Correct, in fact, I began to work on this yesterday. My approach will be to translate whatever's possible at compile-time, as I'm doing now, and saving the import alias info for what's needed at runtime, such as in the case of variable class names. This hashtable will also be needed for namespace imports, if they are in fact implemented.
-- Jessie Stanislav Malyshev wrote:

Jessie Hernandez

21 years ago
The attached patch includes variable class name support (test file has also been updated). To clarify for the benefit of others, this means that the code below will work. Note that imports are still done on "file-level" scope, even at runtime, so you can only create objects using variable import class names for the imports that have been declared in that same file. <?php import my_ns:my_class; $n = 'my_class'; $c = new $n(); ?>
-- Jessie Stanislav Malyshev wrote:

Xuefer Tinys

21 years ago
at compile time? this means u cannot import * (all) symbols from a namespace

Jessie Hernandez

21 years ago
I'm still debating whether importing all symbols from a namespace is a good idea or not (I sometimes use Java at work, and I never import a full package myself, and this is not recommended anyways). But, if it were to be implemented, then below is how I would approach it. The following would be required: 1) As I said before, the "class_path" php.ini variable would be used for imports. 2) Each file under the class_path is named exactly like the class that it declares. For example, my_namespace/my_class.php would define the class "my_class" under the my_namespace namespace (just like in Java). If an "import namespace my_namespace;" (the syntax is still tentative) statement is found, then the directory <class-path>/my_namespace will be opened (at compile-time). Each file under this directory would be imported. So, assuming our class_path is "/usr/local/php-classes" and the "/usr/local/php-classes/my_namespace" directory had two files, "class1.php" and "class2.php", then this is what would happen: <?php import namespace my_namespace; // Directory "/usr/local/php-classes/my_namespace" is opened. // There are two files, class1.php and class2.php. // // 1) "my_namespace:class1" is added to the import hashtable. // 2) The opcode for "require_once('/usr/local/php-classes/my_namespace/class1.php')" is generated. // 3) "my_namespace:class2" is added to the import hashtable. // 4) The opcode for "require_once('/usr/local/php-classes/my_namespace/class2.php')" is generated. ?> There would also need to be another hashtable to indicate if a full namespace import for a particular namespace has already been done or not, to avoid a directory from being read over and over again.
-- Jessie "Xuefer" <xuefer@gmail.com> wrote in message news:28139bc05070705364cc572f7@mail.gmail.com... at compile time? this means u cannot import * (all) symbols from a namespace

Stanislav Malyshev

21 years ago
JH>>// 1) "my_namespace:class1" is added to the import hashtable. JH>>// 2) The opcode for JH>>"require_once('/usr/local/php-classes/my_namespace/class1.php')" is JH>>generated. JH>>// 3) "my_namespace:class2" is added to the import hashtable. JH>>// 4) The opcode for JH>>"require_once('/usr/local/php-classes/my_namespace/class2.php')" is JH>>generated. JH>>?> This means, of course, that such file could not be used with bytecode caches, because between runs contents of the directory can change.
-- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/ +972-3-6139665 ext.115

Jessie Hernandez

21 years ago
Could you elaborate on this? Let's say we have the following situation: - a.php does a full namespace import of "my_namespace". - The <class_path>/my_namespace directory has two files (classes), class1.php and class2.php. - a.php has the following code: <?php import namespace my_namespace; $c1 = new class1(); $c2 = new class2(); ?> - After a.php is first run, a new file, class3.php, is added. With the above scenario, nothing will be affected if there is an opcode cache or not. The only situation I can think of where an opcode cache can affect this is when the script DYNAMICALLY creates classes from the imported namespace. Maybe there's other situations, but I can't come up with any at the moment. Either way, I'm not sure if full namespace imports are a good idea anyways. Issues like the one you presented are one reason, and the other is that it is bad practice. If multiple namespaces are imported, readability is affected (which class belongs to which namespace?). Also, if two namespaces are imported, and one day a new class is added to one which has the same name as another, then there will suddenly be a compile error, and the error will be elusive.
-- Jessie "Stanislav Malyshev" <stas@zend.com> wrote in message news:Pine.LNX.4.62.0507072119540.24702@mail.zend.com...

Stanislav Malyshev

21 years ago
JH>>With the above scenario, nothing will be affected if there is an JH>>opcode cache or not. The only situation I can think of where an opcode JH>>cache can affect this is when the script DYNAMICALLY creates classes JH>>from the imported namespace. Maybe there's other situations, but I JH>>can't come up with any at the moment. No, more simple - somebody removes one of the classes. Even if nothing in your code ever used it, you'd start getting errors about missing files. JH>>Either way, I'm not sure if full namespace imports are a good idea anyways. Without this, only thing you are achieving with imports is saving a couple of keystrokes. If you have to declare each imported class anyway, why not name it by the full name? Just saving a dozen of keystrokes (which any good code-completing editor would save anyway) and one require statement is not worth the trouble, IMHO. JH>>Issues like the one you presented are one reason, and the other is that it JH>>is bad practice. If multiple namespaces are imported, readability is JH>>affected (which class belongs to which namespace?). Also, if two namespaces That's a general problem with all namespace imports. JH>>are imported, and one day a new class is added to one which has the same JH>>name as another, then there will suddenly be a compile error, and the error JH>>will be elusive. Right, that's one more problem with namespace imports.
-- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/ +972-3-6139665 ext.115

Al Baker

21 years ago
I think we're trying to boil the ocean here, and in doing so failing to see both the problems that need to be solved, and the possible solutions. I'd like to break things down a little bit here to address what the real problem is with a lack of namespaces in PHP. In my eyes, there are really only two problems that arise because of lack of namespaces. The first is the one true problem we should try to solve and that is class naming conflicts. Now that PHP is growing and make inroads into big business, several large development groups may collaborate on a project. Or, they may want to use a library, only to find out a class naming conflict. After all they love about PHP, it seems like this problem makes their lives more difficult by having to manually prefix all of their classes and use those prefixes throughout their code. So, the most basic namespace proposal should just be a prefix to classes, allowing the compiler to differentiate between two classes of the same name, but from different packages. $myobj = new MySpace:MyClass(); seems good to enough to me. If this is all namespaces are, an extra label to prefix classes with - no function namespace, no import, then I say no problem. A namespace hierarchy should be problem free as well, allowing developers a little more structure. At compile time, it just results in a longer prefix than if it were a single namespace. This would also solve a slightly different problem, and that is one of perception to enterprise developers. I know that productivity wise, the above namespace proposal hardly saves the developer more time than having to hardcode prefixes into class names. But what it does do, is it makes sense to a large population of developers, especially those 'enterprise' (ie Java guys). Honestly, when I pitch how productive PHP is on my project (100+ developers), the lack of namespaces is a good jab against PHP. I know people cringe at the term enterprise a little because accomodating the enterprise doesn't always mean accomodating everyone. However, in this case, I would say we have critical mass of interest in namespaces. Having a better prefixing mechanism wouldn't break legacy code, would be optional, and would make a bunch of people happy. Now, I would assert that Java's import system (of which this proposal and others often try to follow) is only partially about namespaces, but is more of a mix of their compile system, source structure, and even naming conventions. We could do the namespace proposal and not do imports. The whole concept of Java's imports is only partially related to namespaces. There are a lot of assumptions made in Java's import system, such as a configuration setting where to look, having one class per file, having that class as the filename for that file, etc. This file including system for the compile matches up with the namespace syntax. Though there's an impression that this is a great system, I would say that it still results in confusion. The dot notation in Java can be used in the context of prefixing a class for proper compilation, accessing a function, a member variable (including those that are classes), a function, and you can string them together like mypackage.myclass.function.returnfunctionsobject.function;. There's no Java.ini to set a directory, you have to set environment variables. If we want to build a more advanced include system, then I would say that the labels should map to sub-directories in some directory specified in php.ini, and the last token in the import statement just performs a normal PHP include. E.g. if it's import mypackage.*; it would go to the predefined directory, go to the my package subdirectory, and perform an include on all the class files in that directory. Of course, we'd always want to have strict class/file names for this to work as well and only one class per file. Thanks, Al

Al Baker

21 years ago
One more thing, here's an example of how things go in the enterprise and where namespaces make or break PHP being used. Every now and then, someone says "Let's merge projects A and B", and those projects could be in different parts of the world, each having 10s if not 100s of developers, and very large code bases. These projects may rarely communicate if ever if they sell to different markets. In Java, you can start writing code to tie them together, because each of them has defined their own package, and you can re-use that code without modifying it inside new code. You just do an import, perhaps use the full namespace prefix (e.g. package) when using classes that conflict otherwise. In PHP today, there are two possibilities. Either someone was smart and enforced some coding guidelines (e.g. PEAR) and people have prefixed their classnames (unlikely as this is an uncommon programming aspect) OR we have a deal breaker issue. Project A has 10,000 lines of code, and Project B has 10,000 lines of code and we have 50 classes of the same name. A global replace doesn't do it, because I may have SQL, other variables, function names similar to that class name. So, one project may get a rewrite or invest a large number of resources to make things compatible and hopefully one project doesn't get the axe. If management gets word of what happened, PHP will be made out to be the villian and there'll be a big fuss and a technology change to Java or something else. I just wanted to throw this out there to set some context on why it's important to the enterprise. I know this feels like a corner case to everyone else, but this is what happens when you have huge organizations developing vast amounts of code. PHP is already a tough sell to a population that has been trained (thanks to colleges) and experienced in something like Java. Thankfully, there are pragmmatic developers out there who see the light of PHP, adopt it, and watch their productivity shoot through the roof for web applications. Anyways, I'm +1 on this YANP sans the import feature as defined in Jessie's proposal. Thanks! Al On Fri, 2005-07-08 at 20:25 -0400, Al Baker wrote:

Andi Gutmans

21 years ago
Did you make sure to real all the past threads on the issue? There are a variety of problems which arise with such a namespaces implementation, including ambiguity, problems with multiple symbol table lookups, dynamic variable creation, cross-file dependencies. I believe this solution suffers from mostly the same problems. Andi At 08:44 PM 7/6/2005 -0400, Jessie Hernandez wrote:

Jessie Hernandez

21 years ago
For previewing purposes, I have attached a patch of what I have so far so that others can look at it and try it out. I have also attached two PHP files for testing (_testns.php is the script that should be run after the patch is applied. This script includes a class from class3.php to demonstrate two things: namespaces may be spread out in files, and classes do not have to be immediately available for imports to work). I based my patch on PHP 5.1.0b2 (not CVS, yet). I have not used the diff utility before, but here is the command I ran to produce the patch (let me know if there's a better way): diff -Bbruw -X diff_exclude.txt php-5.1.0b2-bak/Zend php-5.1.0b2/Zend/ > ns.patch Anyways, all the features I've mentioned have been enabled in the patch except the following: 1) Automatic including of class files. 2) Wildcard imports (or full namespace imports). I have been checking the code to see what it would take to apply the current include/require logic against a different INI variable (class_path), but I digged deeper and deeper into the code and got lost in the streams functions. So far, it seems I have to create my own opcode (like ZEND_INCLUDE_OR_EVAL), and when that's encountered, then call whatever stream function is needed to perform the include against class_path. It seems like it will take me some time to figure this out, so that's why I haven't included it yet (maybe someone can help me?). Please try it out and let me know what you think. I believe this solution compiles the code to be the equivalent of prefixing all classes, so I think most of the issues presented in the archives are solved with this solution. Let me know if you need anything else from me to get it working on your installation. Also, I'm not sure if there are any attachment restrictions in this group (email me privately if you can't open the files).
-- Jessie

Stanislav Malyshev

21 years ago
JH>>- Classes may not contain colons (no change here), BUT class name references JH>>CAN contain colons (this is KEY!). JH>>- All imports are done at compile time. JH>>- Imports only affect the current file (yes, this has been done!) The problem here is that once you have somewhat developed package - like PEAR packages - which may contain dozens of internal classes, you'd have to have a large import statement block at the beginning of each file using them - or refer to them all by the full name, thus just replacing underscore-notation with colon-notation.
-- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/ +972-3-6139665 ext.115

Jessie Hernandez

21 years ago
As requested by others, attached is a cvs patch that includes the latest namespace additions and PHP test files. With this patch, the following features have been added: 1) import statements now include the class files, using a new .ini variable, "class_path"! 2) Near-full support of private classes! Currently, private classes cannot be used outside their namespace, but can be used within methods of classes within the namespace. What's left for me to do is to allow private classes to be extended from other classes in the same namespace (this is almost complete in this patch, it works if both classes are defined in the same file). Private classes are declared simply as "private class_name" within a namespace (if the "private" keyword is applied to a class in the global namespace, an error will be generated). Aside from what's pending in #2, I think the only missing feature is "namespace imports". I'd like to gather thoughts from others as how this can be best implemented (some issues have been raised as how namespace imports will work with opcode caches). I'll post a few possible approaches some other day this week (I'm tired now ;-)). BTW, I tried to do a "cvs add" to add my test files, but of course, it failed with an "cvs add requires write access to repository" error. Hence, I had to zip up the files separately. How can I add new files locally so that cvs diff picks them up in the patch? Regards, Jessie Hernandez