Everything
php
Learn Community Releases The PHP Project
Home » Project » Newsgroups » php.internals » Bundling libxml2 and expat compatibility layer »

Bundling libxml2 and expat compatibility layer

php.internals

Sterling Hughes

23 years ago
Hi, Well, OK, I have libxml2 successfully bundled with PHP, and I've further gone ahead and created a C-level compatibility layer which maps expat <-> libxml2. I've also moved the detection logic for both expat and libxml into php5/bundle/libxml and php5/bundle/expat respectively. This way you can choose your backend at the configure line, and things will work transparently (by default, expat and libxml are compiled in, and the XML extension uses expat). I've also done the "namespace redefinition" heavy lifting - I'm not quite sure it works, but I have renamed most (from what I can tell, all) public symbols, like with expat. I'm sure this could be ironed out pretty easily if I made any mistakes. As far as I'm concerned, the important thing here is bundling libxml2. I think everyone who is implementing XML support around PHP will agree that expat just isn't meeting our needs, specifically: 1) The ability to easily access and modify XML documents from within a programatic structure, alá DOM (this would also make it easy for me to implement my SimpleXML[1] extension). a) The ability to query an XML document via Xpath 2) The ability to validate an XML document against either a DTD or a XML Schema (very important, especially for SOAP.) 3) Proper unicode support 4) Support for XPointer and XLink 5) Support for Docbook and HTML parsing 6) Expat doesn't even full support the same capabilities that libxml2 does when it comes to SAX processing. However, when we bundle expat with PHP, and make ext/xml therefore an "always available" extension. We create the illusion that it is the "recommended" and "best" solution for XML parsing with PHP, when in fact it really isn't. Our needs as far as XML support are growing, whether it be implementing technologies that exist on top of XML (SOAP, WSDL, RDF) or implementing extensions that make it easier to access XML (SimpleXML, DOM), expat makes it way to hard (for all intensive purposes, impossible) to implement these systems. Therefore, I'm suggesting that we bundle libxml2, while (for now) keeping in expat as well. This will cause *absolutely* no backwards compatibility changes, while at the same time, it will allow you to use only libxml2 for XML processing (--without-bundle-expat), with 97% [2] backwards compatibility maintained. -Sterling [1] http://news.php.net/article.php?group=php.xml.dev&article=6 [2] This is one of 54% of facts made up on the spot. Suffice it to say that the new extension is "mostly" backwards compatible, and the places where it breaks, shouldn't have been relied upon anyway.
-- "Reductionists like to take things apart. The rest of us are just trying to get it together." - Larry Wall, Programming Perl, 3rd Edition

Shane Caraveo

23 years ago
Just my .02CDN, I wouldn't even make expat a default in the configuration at all, just use libxml2 for everything, and require a flag to even build expat. Just having the ability to fallback to expat if ever necessary is good enough. Otherwise, I'm ready for libxml :) Shane Sterling Hughes wrote:

(Marcus Börger)

23 years ago
At 18:11 03.05.2003, Sterling Hughes wrote:
>Hi, > >Well, OK, I have libxml2 successfully bundled with PHP, and I've further >gone ahead and created a C-level compatibility layer which maps expat ><-> libxml2. [...]
I like the idea of the C-level wrapper. This would keep us apart from BC problems with the weak xml extensions. More importent though is that a bundled libxml2 would make things easier *much* easier for extension developers working on a more sophisticated xml support in php. => +3 (idea, realization so far and my support)
>5) Support for Docbook and HTML parsing
Well, sounds like ext/docbook :-) regards marcus Q: Can't we make some progress here anymore?

Unnamed Person

23 years ago
Hello, Why is expat still compiled by default? Since there is not any BC break, libxml should be enough, isn't it? A way to compile it, should be enough...
-- Regards. M.CHAILLAN Nicolas nicos@php.net www.WorldAKT.com Hébergement de sites internets. "Sterling Hughes" <sterling@bumblebury.com> a écrit dans le message news: 1051978274.11377.131.camel@hasele...

Dmitri Dmitrienko

23 years ago
Hi, Sterling, before doing such a weird thing of moving everybody to libxml2 please compare performance of what we have with expat and what we'll get with libxml2. I tested them both with quite a big xml file ~500kB. Expat parsed doc in 19ms while libxml2 in 267ms. It is 14 times slower. I understand that there is a big difference between what expat does and what libxml2. On the other hand, there are some 3rd party xmldom-libraries that parse xmlfile to xmldom in ~ 110-130ms. At least two times faster. Also should be noted that libxml2 spends INCREDIBLY long time when freeing parsed document 133ms. Moreover, when I tried to parse pre-loaded document (xmlParseMemory), it showed even worse results 786ms. I believe it's too early to switch to this library. At least there some reasons to think more about. Best regards, Dmitri. "Sterling Hughes" <sterling@bumblebury.com> wrote in message news:1051978274.11377.131.camel@hasele...

Christian Stocker

23 years ago
Hi I didn't look at Sterlings code yet, but you can't compare SAX parsing of expat with DOM parsing of libxml2. Libxml2 however does support SAX, as well and I assume (and hope) Sterling used only this for ext/xml replacement (making an in-memory DOM-Tree per default in ext/xml would make a lot of people very unhappy ;) ) Dmitri, what exactly did you compare? chregu On Sun, 4 May 2003, Dmitri Dmitrienko wrote:
> Hi, > > Sterling, before doing such a weird thing of moving everybody to libxml2 > please compare performance of what we have with expat and what we'll get > with libxml2. > > I tested them both with quite a big xml file ~500kB. Expat parsed doc in > 19ms while libxml2 in 267ms. > It is 14 times slower. I understand that there is a big difference between > what expat does and what libxml2. > On the other hand, there are some 3rd party xmldom-libraries that parse > xmlfile to xmldom in ~ 110-130ms. > At least two times faster. > > Also should be noted that libxml2 spends INCREDIBLY long time when freeing > parsed document 133ms. > Moreover, when I tried to parse pre-loaded document (xmlParseMemory), it > showed even worse results 786ms. > > I believe it's too early to switch to this library. At least there some > reasons to think more about. > > Best regards, > Dmitri. > > > "Sterling Hughes" <sterling@bumblebury.com> wrote in message > news:1051978274.11377.131.camel@hasele... > > Hi, > > > > Well, OK, I have libxml2 successfully bundled with PHP, and I've further > > gone ahead and created a C-level compatibility layer which maps expat > > <-> libxml2. I've also moved the detection logic for both expat and > > libxml into php5/bundle/libxml and php5/bundle/expat respectively. This > > way you can choose your backend at the configure line, and things will > > work transparently (by default, expat and libxml are compiled in, and > > the XML extension uses expat). I've also done the "namespace > > redefinition" heavy lifting - I'm not quite sure it works, but I have > > renamed most (from what I can tell, all) public symbols, like with > > expat. I'm sure this could be ironed out pretty easily if I made any > > mistakes. > > > > As far as I'm concerned, the important thing here is bundling libxml2. > > I think everyone who is implementing XML support around PHP will agree > > that expat just isn't meeting our needs, specifically: > > > > 1) The ability to easily access and modify XML documents from within a > > programatic structure, alá DOM (this would also make it easy for me to > > implement my SimpleXML[1] extension). > > > > a) The ability to query an XML document via Xpath > > > > 2) The ability to validate an XML document against either a DTD or a XML > > Schema (very important, especially for SOAP.) > > > > 3) Proper unicode support > > > > 4) Support for XPointer and XLink > > > > 5) Support for Docbook and HTML parsing > > > > 6) Expat doesn't even full support the same capabilities that libxml2 > > does when it comes to SAX processing. > > > > However, when we bundle expat with PHP, and make ext/xml therefore an > > "always available" extension. We create the illusion that it is the > > "recommended" and "best" solution for XML parsing with PHP, when in fact > > it really isn't. > > > > Our needs as far as XML support are growing, whether it be implementing > > technologies that exist on top of XML (SOAP, WSDL, RDF) or implementing > > extensions that make it easier to access XML (SimpleXML, DOM), expat > > makes it way to hard (for all intensive purposes, impossible) to > > implement these systems. > > > > Therefore, I'm suggesting that we bundle libxml2, while (for now) > > keeping in expat as well. This will cause *absolutely* no backwards > > compatibility changes, while at the same time, it will allow you to use > > only libxml2 for XML processing (--without-bundle-expat), with 97% [2] > > backwards compatibility maintained. > > > > -Sterling > > > > [1] http://news.php.net/article.php?group=php.xml.dev&article=6 > > [2] This is one of 54% of facts made up on the spot. Suffice it to say > > that the new extension is "mostly" backwards compatible, and the places > > where it breaks, shouldn't have been relied upon anyway. > > > > -- > > "Reductionists like to take things apart. The rest of us are > > just trying to get it together." > > - Larry Wall, Programming Perl, 3rd Edition > > > > > >
-- nam...christian stocker adr...pflanzschulstr. 31, ch-8004 zurich pho...+41 43 317 9984 www...http://blog.bitflux.ch mob...+41 76 561 8860 ema...chregu@phant.ch wor...+41 1 240 5670 gpg...0x5CE1DECB

Dmitri Dmitrienko

23 years ago
Hi Christian, I compared _parsing_, only parsing. Does it make sense ? Certainly, I expected some overhead for memory allocating when building DOM tree. I believe this overhead should be adequate. For example less than 3-5 times. But actually the overhead is much higher, incredibly higher. Could you explain what this time is spent for ? Why xmlParseFile() is so slow ? Also, would be nice to hear your opinion why xmlFreeDoc() is slow... It should only free allocated memory, nothing above. I expected 1-10ms for it while actially got 133ms, quite comparable with time for parsing. Also, why xmlParseMemory() is 3 times slower than xmlParseFile() ??? It can't be explained easily, I guess. I think libxml2 is a really SLOW library, purely slow, and will not satisfy people who concern about performance. -Dmitri
>I didn't look at Sterlings code yet, but you can't compare SAX parsing of >expat with DOM parsing of libxml2. Libxml2 however does support SAX, as >well and I assume (and hope) Sterling used only this for ext/xml >replacement (making an in-memory DOM-Tree per default in ext/xml would >make a lot of people very unhappy ;) ) > >Dmitri, what exactly did you compare? > >chregu
On Sun, 4 May 2003, Dmitri Dmitrienko wrote:
> Hi, > > Sterling, before doing such a weird thing of moving everybody to libxml2 > please compare performance of what we have with expat and what we'll get > with libxml2. > > I tested them both with quite a big xml file ~500kB. Expat parsed doc in > 19ms while libxml2 in 267ms. > It is 14 times slower. I understand that there is a big difference between > what expat does and what libxml2. > On the other hand, there are some 3rd party xmldom-libraries that parse > xmlfile to xmldom in ~ 110-130ms. > At least two times faster. > > Also should be noted that libxml2 spends INCREDIBLY long time when freeing > parsed document 133ms. > Moreover, when I tried to parse pre-loaded document (xmlParseMemory), it > showed even worse results 786ms. > > I believe it's too early to switch to this library. At least there some > reasons to think more about. > > Best regards, > Dmitri. > > > "Sterling Hughes" <sterling@bumblebury.com> wrote in message > news:1051978274.11377.131.camel@hasele... > > Hi, > > > > Well, OK, I have libxml2 successfully bundled with PHP, and I've further > > gone ahead and created a C-level compatibility layer which maps expat > > <-> libxml2. I've also moved the detection logic for both expat and > > libxml into php5/bundle/libxml and php5/bundle/expat respectively. This > > way you can choose your backend at the configure line, and things will > > work transparently (by default, expat and libxml are compiled in, and > > the XML extension uses expat). I've also done the "namespace > > redefinition" heavy lifting - I'm not quite sure it works, but I have > > renamed most (from what I can tell, all) public symbols, like with > > expat. I'm sure this could be ironed out pretty easily if I made any > > mistakes. > > > > As far as I'm concerned, the important thing here is bundling libxml2. > > I think everyone who is implementing XML support around PHP will agree > > that expat just isn't meeting our needs, specifically: > > > > 1) The ability to easily access and modify XML documents from within a > > programatic structure, alá DOM (this would also make it easy for me to > > implement my SimpleXML[1] extension). > > > > a) The ability to query an XML document via Xpath > > > > 2) The ability to validate an XML document against either a DTD or a XML > > Schema (very important, especially for SOAP.) > > > > 3) Proper unicode support > > > > 4) Support for XPointer and XLink > > > > 5) Support for Docbook and HTML parsing > > > > 6) Expat doesn't even full support the same capabilities that libxml2 > > does when it comes to SAX processing. > > > > However, when we bundle expat with PHP, and make ext/xml therefore an > > "always available" extension. We create the illusion that it is the > > "recommended" and "best" solution for XML parsing with PHP, when in fact > > it really isn't. > > > > Our needs as far as XML support are growing, whether it be implementing > > technologies that exist on top of XML (SOAP, WSDL, RDF) or implementing > > extensions that make it easier to access XML (SimpleXML, DOM), expat > > makes it way to hard (for all intensive purposes, impossible) to > > implement these systems. > > > > Therefore, I'm suggesting that we bundle libxml2, while (for now) > > keeping in expat as well. This will cause *absolutely* no backwards > > compatibility changes, while at the same time, it will allow you to use > > only libxml2 for XML processing (--without-bundle-expat), with 97% [2] > > backwards compatibility maintained. > > > > -Sterling > > > > [1] http://news.php.net/article.php?group=php.xml.dev&article=6 > > [2] This is one of 54% of facts made up on the spot. Suffice it to say > > that the new extension is "mostly" backwards compatible, and the places > > where it breaks, shouldn't have been relied upon anyway. > > > > -- > > "Reductionists like to take things apart. The rest of us are > > just trying to get it together." > > - Larry Wall, Programming Perl, 3rd Edition > > > > > >
-- nam...christian stocker adr...pflanzschulstr. 31, ch-8004 zurich pho...+41 43 317 9984 www...http://blog.bitflux.ch mob...+41 76 561 8860 ema...chregu@phant.ch wor...+41 1 240 5670 gpg...0x5CE1DECB

Unnamed Person

23 years ago
Hello, Well, if you don't like libxml2, you can still build the expat bundle... But yes, its surprising that you get such bad results.
-- Regards. M.CHAILLAN Nicolas nicos@php.net www.WorldAKT.com Hébergement de sites internets. "Dmitri Dmitrienko" <dd@cron.ru> a écrit dans le message news: 20030504113655.94893.qmail@pb1.pair.com... > Hi Christian, > > I compared _parsing_, only parsing. Does it make sense ? > Certainly, I expected some overhead for memory allocating when building DOM > tree. > I believe this overhead should be adequate. For example less than 3-5 times. > But actually the overhead is much higher, incredibly higher. > > Could you explain what this time is spent for ? Why xmlParseFile() is so > slow ? > > Also, would be nice to hear your opinion why xmlFreeDoc() is slow... > It should only free allocated memory, nothing above. I expected 1-10ms for > it while actially got 133ms, quite comparable with time for parsing. > > Also, why xmlParseMemory() is 3 times slower than xmlParseFile() ??? It > can't be explained easily, I guess. > > I think libxml2 is a really SLOW library, purely slow, and will not satisfy > people who concern about performance. > > -Dmitri > > > >I didn't look at Sterlings code yet, but you can't compare SAX parsing of > >expat with DOM parsing of libxml2. Libxml2 however does support SAX, as > >well and I assume (and hope) Sterling used only this for ext/xml > >replacement (making an in-memory DOM-Tree per default in ext/xml would > >make a lot of people very unhappy ;) ) > > > >Dmitri, what exactly did you compare? > > > >chregu > > On Sun, 4 May 2003, Dmitri Dmitrienko wrote: > > > Hi, > > > > Sterling, before doing such a weird thing of moving everybody to libxml2 > > please compare performance of what we have with expat and what we'll get > > with libxml2. > > > > I tested them both with quite a big xml file ~500kB. Expat parsed doc in > > 19ms while libxml2 in 267ms. > > It is 14 times slower. I understand that there is a big difference between > > what expat does and what libxml2. > > On the other hand, there are some 3rd party xmldom-libraries that parse > > xmlfile to xmldom in ~ 110-130ms. > > At least two times faster. > > > > Also should be noted that libxml2 spends INCREDIBLY long time when freeing > > parsed document 133ms. > > Moreover, when I tried to parse pre-loaded document (xmlParseMemory), it > > showed even worse results 786ms. > > > > I believe it's too early to switch to this library. At least there some > > reasons to think more about. > > > > Best regards, > > Dmitri. > > > > > > "Sterling Hughes" <sterling@bumblebury.com> wrote in message > > news:1051978274.11377.131.camel@hasele... > > > Hi, > > > > > > Well, OK, I have libxml2 successfully bundled with PHP, and I've further > > > gone ahead and created a C-level compatibility layer which maps expat > > > <-> libxml2. I've also moved the detection logic for both expat and > > > libxml into php5/bundle/libxml and php5/bundle/expat respectively. This > > > way you can choose your backend at the configure line, and things will > > > work transparently (by default, expat and libxml are compiled in, and > > > the XML extension uses expat). I've also done the "namespace > > > redefinition" heavy lifting - I'm not quite sure it works, but I have > > > renamed most (from what I can tell, all) public symbols, like with > > > expat. I'm sure this could be ironed out pretty easily if I made any > > > mistakes. > > > > > > As far as I'm concerned, the important thing here is bundling libxml2. > > > I think everyone who is implementing XML support around PHP will agree > > > that expat just isn't meeting our needs, specifically: > > > > > > 1) The ability to easily access and modify XML documents from within a > > > programatic structure, alá DOM (this would also make it easy for me to > > > implement my SimpleXML[1] extension). > > > > > > a) The ability to query an XML document via Xpath > > > > > > 2) The ability to validate an XML document against either a DTD or a XML > > > Schema (very important, especially for SOAP.) > > > > > > 3) Proper unicode support > > > > > > 4) Support for XPointer and XLink > > > > > > 5) Support for Docbook and HTML parsing > > > > > > 6) Expat doesn't even full support the same capabilities that libxml2 > > > does when it comes to SAX processing. > > > > > > However, when we bundle expat with PHP, and make ext/xml therefore an > > > "always available" extension. We create the illusion that it is the > > > "recommended" and "best" solution for XML parsing with PHP, when in fact > > > it really isn't. > > > > > > Our needs as far as XML support are growing, whether it be implementing > > > technologies that exist on top of XML (SOAP, WSDL, RDF) or implementing > > > extensions that make it easier to access XML (SimpleXML, DOM), expat > > > makes it way to hard (for all intensive purposes, impossible) to > > > implement these systems. > > > > > > Therefore, I'm suggesting that we bundle libxml2, while (for now) > > > keeping in expat as well. This will cause *absolutely* no backwards > > > compatibility changes, while at the same time, it will allow you to use > > > only libxml2 for XML processing (--without-bundle-expat), with 97% [2] > > > backwards compatibility maintained. > > > > > > -Sterling > > > > > > [1] http://news.php.net/article.php?group=php.xml.dev&article=6 > > > [2] This is one of 54% of facts made up on the spot. Suffice it to say > > > that the new extension is "mostly" backwards compatible, and the places

Sterling Hughes

23 years ago
Dmitri, Geeze. As Christian said, you're comparing apples and oranges. In order to properly benchmark, compare the push parser interface with expat's interface. Or, look on the web, there have been plenty of benchmarks libxml2 is the *fastest* XML parsing library available (besides msxml, which is closed source). In terms of SAX processing, expat has a very, very slight advantage in some situations, but nothing to speak of. -Sterling On Sun, 2003-05-04 at 07:36, Dmitri Dmitrienko wrote:
> Hi Christian, > > I compared _parsing_, only parsing. Does it make sense ? > Certainly, I expected some overhead for memory allocating when building DOM > tree. > I believe this overhead should be adequate. For example less than 3-5 times. > But actually the overhead is much higher, incredibly higher. > > Could you explain what this time is spent for ? Why xmlParseFile() is so > slow ? > > Also, would be nice to hear your opinion why xmlFreeDoc() is slow... > It should only free allocated memory, nothing above. I expected 1-10ms for > it while actially got 133ms, quite comparable with time for parsing. > > Also, why xmlParseMemory() is 3 times slower than xmlParseFile() ??? It > can't be explained easily, I guess. > > I think libxml2 is a really SLOW library, purely slow, and will not satisfy > people who concern about performance. > > -Dmitri > > > >I didn't look at Sterlings code yet, but you can't compare SAX parsing of > >expat with DOM parsing of libxml2. Libxml2 however does support SAX, as > >well and I assume (and hope) Sterling used only this for ext/xml > >replacement (making an in-memory DOM-Tree per default in ext/xml would > >make a lot of people very unhappy ;) ) > > > >Dmitri, what exactly did you compare? > > > >chregu > > On Sun, 4 May 2003, Dmitri Dmitrienko wrote: > > > Hi, > > > > Sterling, before doing such a weird thing of moving everybody to libxml2 > > please compare performance of what we have with expat and what we'll get > > with libxml2. > > > > I tested them both with quite a big xml file ~500kB. Expat parsed doc in > > 19ms while libxml2 in 267ms. > > It is 14 times slower. I understand that there is a big difference between > > what expat does and what libxml2. > > On the other hand, there are some 3rd party xmldom-libraries that parse > > xmlfile to xmldom in ~ 110-130ms. > > At least two times faster. > > > > Also should be noted that libxml2 spends INCREDIBLY long time when freeing > > parsed document 133ms. > > Moreover, when I tried to parse pre-loaded document (xmlParseMemory), it > > showed even worse results 786ms. > > > > I believe it's too early to switch to this library. At least there some > > reasons to think more about. > > > > Best regards, > > Dmitri. > > > > > > "Sterling Hughes" <sterling@bumblebury.com> wrote in message > > news:1051978274.11377.131.camel@hasele... > > > Hi, > > > > > > Well, OK, I have libxml2 successfully bundled with PHP, and I've further > > > gone ahead and created a C-level compatibility layer which maps expat > > > <-> libxml2. I've also moved the detection logic for both expat and > > > libxml into php5/bundle/libxml and php5/bundle/expat respectively. This > > > way you can choose your backend at the configure line, and things will > > > work transparently (by default, expat and libxml are compiled in, and > > > the XML extension uses expat). I've also done the "namespace > > > redefinition" heavy lifting - I'm not quite sure it works, but I have > > > renamed most (from what I can tell, all) public symbols, like with > > > expat. I'm sure this could be ironed out pretty easily if I made any > > > mistakes. > > > > > > As far as I'm concerned, the important thing here is bundling libxml2. > > > I think everyone who is implementing XML support around PHP will agree > > > that expat just isn't meeting our needs, specifically: > > > > > > 1) The ability to easily access and modify XML documents from within a > > > programatic structure, alá DOM (this would also make it easy for me to > > > implement my SimpleXML[1] extension). > > > > > > a) The ability to query an XML document via Xpath > > > > > > 2) The ability to validate an XML document against either a DTD or a XML > > > Schema (very important, especially for SOAP.) > > > > > > 3) Proper unicode support > > > > > > 4) Support for XPointer and XLink > > > > > > 5) Support for Docbook and HTML parsing > > > > > > 6) Expat doesn't even full support the same capabilities that libxml2 > > > does when it comes to SAX processing. > > > > > > However, when we bundle expat with PHP, and make ext/xml therefore an > > > "always available" extension. We create the illusion that it is the > > > "recommended" and "best" solution for XML parsing with PHP, when in fact > > > it really isn't. > > > > > > Our needs as far as XML support are growing, whether it be implementing > > > technologies that exist on top of XML (SOAP, WSDL, RDF) or implementing > > > extensions that make it easier to access XML (SimpleXML, DOM), expat > > > makes it way to hard (for all intensive purposes, impossible) to > > > implement these systems. > > > > > > Therefore, I'm suggesting that we bundle libxml2, while (for now) > > > keeping in expat as well. This will cause *absolutely* no backwards > > > compatibility changes, while at the same time, it will allow you to use > > > only libxml2 for XML processing (--without-bundle-expat), with 97% [2] > > > backwards compatibility maintained. > > > > > > -Sterling > > > > > > [1] http://news.php.net/article.php?group=php.xml.dev&article=6 > > > [2] This is one of 54% of facts made up on the spot. Suffice it to say > > > that the new extension is "mostly" backwards compatible, and the places > > > where it breaks, shouldn't have been relied upon anyway. > > > > > > -- > > > "Reductionists like to take things apart. The rest of us are > > > just trying to get it together." > > > - Larry Wall, Programming Perl, 3rd Edition > > > > > > > > > > > > > -- > nam...christian stocker adr...pflanzschulstr. 31, ch-8004 zurich > pho...+41 43 317 9984 www...http://blog.bitflux.ch > mob...+41 76 561 8860 ema...chregu@phant.ch > wor...+41 1 240 5670 gpg...0x5CE1DECB
-- Good judgement comes from experience, and experience comes from bad judgement. - Fred Brooks

Dmitri Dmitrienko

23 years ago
Sterling, As I said before, I know what I am doing comparing those libraries. If I wasn't I would not raise this discussion. Ok. If you think I compare apples and oranges, well. Suppose I do. But take a look what's finally offered by domxml. It's a verified tree of nodes that were resulted from parsing original xml. Nothing more nothing less. I think everybody clearly understand advantages of having such trees. I'm actually not against it. It should be quite clear thought. Now about the matter of discussing. It's performance. When I set proper callbacks for expat I get the same node tree in 5 times faster. What this "good" domxml parser spends MY time for ? Answer is very simple. Have a look at parser.c shipped with libxml2. It's what I'd call geeze. It's written from scratches as if we are in 19th century. Ok Sterling if you think this approach is ok for all, why don't switch back to the same parser for PHP ? Let's introduce PHP 2.0 once again :))), geeze. What I'd love to see is Flex-based lexer for xml that has proven its really good performance. Let other people say what they are thinking about their needs in performance terms. IMHO it's too early to switch to libxml2. It's pretty slow when parsing xml. All the best, Dmitri.
> Dmitri, > > Geeze. As Christian said, you're comparing apples and oranges. In > order to properly benchmark, compare the push parser interface with > expat's interface. Or, look on the web, there have been plenty of > benchmarks libxml2 is the *fastest* XML parsing library available > (besides msxml, which is closed source). In terms of SAX processing, > expat has a very, very slight advantage in some situations, but nothing > to speak of. > > -Sterling > > On Sun, 2003-05-04 at 07:36, Dmitri Dmitrienko wrote: > > Hi Christian, > > > > I compared _parsing_, only parsing. Does it make sense ? > > Certainly, I expected some overhead for memory allocating when building
DOM
> > tree. > > I believe this overhead should be adequate. For example less than 3-5
times.
> > But actually the overhead is much higher, incredibly higher. > > > > Could you explain what this time is spent for ? Why xmlParseFile() is so > > slow ? > > > > Also, would be nice to hear your opinion why xmlFreeDoc() is slow... > > It should only free allocated memory, nothing above. I expected 1-10ms
for
> > it while actially got 133ms, quite comparable with time for parsing. > > > > Also, why xmlParseMemory() is 3 times slower than xmlParseFile() ??? It > > can't be explained easily, I guess. > > > > I think libxml2 is a really SLOW library, purely slow, and will not
satisfy
> > people who concern about performance. > > > > -Dmitri > > > > > > >I didn't look at Sterlings code yet, but you can't compare SAX parsing
of
> > >expat with DOM parsing of libxml2. Libxml2 however does support SAX, as > > >well and I assume (and hope) Sterling used only this for ext/xml > > >replacement (making an in-memory DOM-Tree per default in ext/xml would > > >make a lot of people very unhappy ;) ) > > > > > >Dmitri, what exactly did you compare? > > > > > >chregu > > > > On Sun, 4 May 2003, Dmitri Dmitrienko wrote: > > > > > Hi, > > > > > > Sterling, before doing such a weird thing of moving everybody to
libxml2
> > > please compare performance of what we have with expat and what we'll
get
> > > with libxml2. > > > > > > I tested them both with quite a big xml file ~500kB. Expat parsed doc
in
> > > 19ms while libxml2 in 267ms. > > > It is 14 times slower. I understand that there is a big difference
between
> > > what expat does and what libxml2. > > > On the other hand, there are some 3rd party xmldom-libraries that
parse
> > > xmlfile to xmldom in ~ 110-130ms. > > > At least two times faster. > > > > > > Also should be noted that libxml2 spends INCREDIBLY long time when
freeing
> > > parsed document 133ms. > > > Moreover, when I tried to parse pre-loaded document (xmlParseMemory),
it
> > > showed even worse results 786ms. > > > > > > I believe it's too early to switch to this library. At least there
some
> > > reasons to think more about. > > > > > > Best regards, > > > Dmitri. > > > > > > > > > "Sterling Hughes" <sterling@bumblebury.com> wrote in message > > > news:1051978274.11377.131.camel@hasele... > > > > Hi, > > > > > > > > Well, OK, I have libxml2 successfully bundled with PHP, and I've
further
> > > > gone ahead and created a C-level compatibility layer which maps
expat
> > > > <-> libxml2. I've also moved the detection logic for both expat and > > > > libxml into php5/bundle/libxml and php5/bundle/expat respectively.
This
> > > > way you can choose your backend at the configure line, and things
will
> > > > work transparently (by default, expat and libxml are compiled in,
and
> > > > the XML extension uses expat). I've also done the "namespace > > > > redefinition" heavy lifting - I'm not quite sure it works, but I
have
> > > > renamed most (from what I can tell, all) public symbols, like with > > > > expat. I'm sure this could be ironed out pretty easily if I made
any
> > > > mistakes. > > > > > > > > As far as I'm concerned, the important thing here is bundling
libxml2.
> > > > I think everyone who is implementing XML support around PHP will
agree
> > > > that expat just isn't meeting our needs, specifically: > > > > > > > > 1) The ability to easily access and modify XML documents from within
a
> > > > programatic structure, alá DOM (this would also make it easy for me
to
> > > > implement my SimpleXML[1] extension). > > > > > > > > a) The ability to query an XML document via Xpath > > > > > > > > 2) The ability to validate an XML document against either a DTD or a
XML
> > > > Schema (very important, especially for SOAP.) > > > > > > > > 3) Proper unicode support > > > > > > > > 4) Support for XPointer and XLink > > > > > > > > 5) Support for Docbook and HTML parsing > > > > > > > > 6) Expat doesn't even full support the same capabilities that
libxml2
> > > > does when it comes to SAX processing. > > > > > > > > However, when we bundle expat with PHP, and make ext/xml therefore
an
> > > > "always available" extension. We create the illusion that it is the > > > > "recommended" and "best" solution for XML parsing with PHP, when in
fact
> > > > it really isn't. > > > > > > > > Our needs as far as XML support are growing, whether it be
implementing
> > > > technologies that exist on top of XML (SOAP, WSDL, RDF) or
implementing
> > > > extensions that make it easier to access XML (SimpleXML, DOM), expat > > > > makes it way to hard (for all intensive purposes, impossible) to > > > > implement these systems. > > > > > > > > Therefore, I'm suggesting that we bundle libxml2, while (for now) > > > > keeping in expat as well. This will cause *absolutely* no backwards > > > > compatibility changes, while at the same time, it will allow you to
use
> > > > only libxml2 for XML processing (--without-bundle-expat), with 97%
[2]
> > > > backwards compatibility maintained. > > > > > > > > -Sterling > > > > > > > > [1] http://news.php.net/article.php?group=php.xml.dev&article=6 > > > > [2] This is one of 54% of facts made up on the spot. Suffice it to
say
> > > > that the new extension is "mostly" backwards compatible, and the
places

Christian Stocker

23 years ago
Dmitri xmlParseFile does build a DOM-Tree out of your XML-Document, which is of course slower than the SAX-parsing expat is doing.. _But_ libxml2 can parse your XML document in SAX-style only without building an DOM-Tree. Without looking at Sterling's code, I assume, that's what he did and you should compare this code to expat and _not_ xmlParseFile.. libxml2 is certainly not slow, it has a well known reputation as being very fast for what it does. And if you don't know the difference between SAX and DOM, please do a google lookup before trolling here.. Comparing SAX with DOM is comparing Apple with Oranges chregu On Sun, 4 May 2003, Dmitri Dmitrienko wrote:
> Sterling, > > As I said before, I know what I am doing comparing those libraries. > If I wasn't I would not raise this discussion. > > Ok. If you think I compare apples and oranges, well. Suppose I do. > But take a look what's finally offered by domxml. It's a verified tree > of nodes that were resulted from parsing original xml. > Nothing more nothing less. > I think everybody clearly understand advantages of having such trees. > I'm actually not against it. It should be quite clear thought. > > Now about the matter of discussing. > It's performance. When I set proper callbacks for expat I get the same node > tree in 5 times faster. > What this "good" domxml parser spends MY time for ? > Answer is very simple. Have a look at parser.c shipped with libxml2. > It's what I'd call geeze. It's written from scratches as if we are in 19th > century. > Ok Sterling if you think this approach is ok for all, why don't switch back > to the same parser for PHP ? > Let's introduce PHP 2.0 once again :))), geeze. > > What I'd love to see is Flex-based lexer for xml that has proven its really > good performance. > > Let other people say what they are thinking about their needs in performance > terms. > > IMHO it's too early to switch to libxml2. It's pretty slow when parsing xml. > > All the best, > Dmitri. > > > > > Dmitri, > > > > Geeze. As Christian said, you're comparing apples and oranges. In > > order to properly benchmark, compare the push parser interface with > > expat's interface. Or, look on the web, there have been plenty of > > benchmarks libxml2 is the *fastest* XML parsing library available > > (besides msxml, which is closed source). In terms of SAX processing, > > expat has a very, very slight advantage in some situations, but nothing > > to speak of. > > > > -Sterling > > > > On Sun, 2003-05-04 at 07:36, Dmitri Dmitrienko wrote: > > > Hi Christian, > > > > > > I compared _parsing_, only parsing. Does it make sense ? > > > Certainly, I expected some overhead for memory allocating when building > DOM > > > tree. > > > I believe this overhead should be adequate. For example less than 3-5 > times. > > > But actually the overhead is much higher, incredibly higher. > > > > > > Could you explain what this time is spent for ? Why xmlParseFile() is so > > > slow ? > > > > > > Also, would be nice to hear your opinion why xmlFreeDoc() is slow... > > > It should only free allocated memory, nothing above. I expected 1-10ms > for > > > it while actially got 133ms, quite comparable with time for parsing. > > > > > > Also, why xmlParseMemory() is 3 times slower than xmlParseFile() ??? It > > > can't be explained easily, I guess. > > > > > > I think libxml2 is a really SLOW library, purely slow, and will not > satisfy > > > people who concern about performance. > > > > > > -Dmitri > > > > > > > > > >I didn't look at Sterlings code yet, but you can't compare SAX parsing > of > > > >expat with DOM parsing of libxml2. Libxml2 however does support SAX, as > > > >well and I assume (and hope) Sterling used only this for ext/xml > > > >replacement (making an in-memory DOM-Tree per default in ext/xml would > > > >make a lot of people very unhappy ;) ) > > > > > > > >Dmitri, what exactly did you compare? > > > > > > > >chregu > > > > > > On Sun, 4 May 2003, Dmitri Dmitrienko wrote: > > > > > > > Hi, > > > > > > > > Sterling, before doing such a weird thing of moving everybody to > libxml2 > > > > please compare performance of what we have with expat and what we'll > get > > > > with libxml2. > > > > > > > > I tested them both with quite a big xml file ~500kB. Expat parsed doc > in > > > > 19ms while libxml2 in 267ms. > > > > It is 14 times slower. I understand that there is a big difference > between > > > > what expat does and what libxml2. > > > > On the other hand, there are some 3rd party xmldom-libraries that > parse > > > > xmlfile to xmldom in ~ 110-130ms. > > > > At least two times faster. > > > > > > > > Also should be noted that libxml2 spends INCREDIBLY long time when > freeing > > > > parsed document 133ms. > > > > Moreover, when I tried to parse pre-loaded document (xmlParseMemory), > it > > > > showed even worse results 786ms. > > > > > > > > I believe it's too early to switch to this library. At least there > some > > > > reasons to think more about. > > > > > > > > Best regards, > > > > Dmitri. > > > > > > > > > > > > "Sterling Hughes" <sterling@bumblebury.com> wrote in message > > > > news:1051978274.11377.131.camel@hasele... > > > > > Hi, > > > > > > > > > > Well, OK, I have libxml2 successfully bundled with PHP, and I've > further > > > > > gone ahead and created a C-level compatibility layer which maps > expat > > > > > <-> libxml2. I've also moved the detection logic for both expat and > > > > > libxml into php5/bundle/libxml and php5/bundle/expat respectively. > This > > > > > way you can choose your backend at the configure line, and things > will > > > > > work transparently (by default, expat and libxml are compiled in, > and > > > > > the XML extension uses expat). I've also done the "namespace > > > > > redefinition" heavy lifting - I'm not quite sure it works, but I > have > > > > > renamed most (from what I can tell, all) public symbols, like with > > > > > expat. I'm sure this could be ironed out pretty easily if I made > any > > > > > mistakes. > > > > > > > > > > As far as I'm concerned, the important thing here is bundling > libxml2. > > > > > I think everyone who is implementing XML support around PHP will > agree > > > > > that expat just isn't meeting our needs, specifically: > > > > > > > > > > 1) The ability to easily access and modify XML documents from within > a > > > > > programatic structure, alá DOM (this would also make it easy for me > to > > > > > implement my SimpleXML[1] extension). > > > > > > > > > > a) The ability to query an XML document via Xpath > > > > > > > > > > 2) The ability to validate an XML document against either a DTD or a > XML > > > > > Schema (very important, especially for SOAP.) > > > > > > > > > > 3) Proper unicode support > > > > > > > > > > 4) Support for XPointer and XLink > > > > > > > > > > 5) Support for Docbook and HTML parsing > > > > > > > > > > 6) Expat doesn't even full support the same capabilities that > libxml2 > > > > > does when it comes to SAX processing. > > > > > > > > > > However, when we bundle expat with PHP, and make ext/xml therefore > an > > > > > "always available" extension. We create the illusion that it is the > > > > > "recommended" and "best" solution for XML parsing with PHP, when in > fact > > > > > it really isn't. > > > > > > > > > > Our needs as far as XML support are growing, whether it be > implementing > > > > > technologies that exist on top of XML (SOAP, WSDL, RDF) or > implementing > > > > > extensions that make it easier to access XML (SimpleXML, DOM), expat > > > > > makes it way to hard (for all intensive purposes, impossible) to > > > > > implement these systems. > > > > > > > > > > Therefore, I'm suggesting that we bundle libxml2, while (for now) > > > > > keeping in expat as well. This will cause *absolutely* no backwards > > > > > compatibility changes, while at the same time, it will allow you to > use > > > > > only libxml2 for XML processing (--without-bundle-expat), with 97% > [2] > > > > > backwards compatibility maintained. > > > > > > > > > > -Sterling > > > > > > > > > > [1] http://news.php.net/article.php?group=php.xml.dev&article=6 > > > > > [2] This is one of 54% of facts made up on the spot. Suffice it to > say > > > > > that the new extension is "mostly" backwards compatible, and the > places > > > > > where it breaks, shouldn't have been relied upon anyway. > > > > > > > > > > -- > > > > > "Reductionists like to take things apart. The rest of us are > > > > > just trying to get it together." > > > > > - Larry Wall, Programming Perl, 3rd Edition > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > nam...christian stocker adr...pflanzschulstr. 31, ch-8004 zurich > > > pho...+41 43 317 9984 www...http://blog.bitflux.ch > > > mob...+41 76 561 8860 ema...chregu@phant.ch > > > wor...+41 1 240 5670 gpg...0x5CE1DECB > > -- > > Good judgement comes from experience, and experience comes from > > bad judgement. > > - Fred Brooks > > > > >
-- nam...christian stocker adr...pflanzschulstr. 31, ch-8004 zurich pho...+41 43 317 9984 www...http://blog.bitflux.ch mob...+41 76 561 8860 ema...chregu@phant.ch wor...+41 1 240 5670 gpg...0x5CE1DECB

Dmitri Dmitrienko

23 years ago
Christian,
>xmlParseFile does build a DOM-Tree out of your > XML-Document, which is of >course slower than the SAX-parsing expat is doing..
It is not obvious conclusion that any XMLDOM parsing should be slower. More over it is competely wrong if you compare libxml2 vs expat. If you think more you'll see that DOM-parser only allocates nodes and link them in lists. Should it be so much slower ??? Are you sure that allocating nodes should slow down everything by 14 times ? I believe it is not. Also I don not see any reasonable explanation why libxml disposes document so slow. It does not need verify, it does not need parse, it's only fries nodes. NOTHING MORE. And takes nearly the same time as allocating/parsing and verifying. The only obvious conclusion is that all algorithms are written inefficiently. I'm not against XML DOM and it's benefits. I'm against wrong and inefficient algorithms. People, why don't you read Donald Knouth's books ?
>_But_ libxml2 can parse your XML document in SAX-style only without >building an DOM-Tree. Without looking at Sterling's code, I assume, >that's what he did and you should compare this code to expat and _not_ >xmlParseFile..
libxml2-2.5.7,parser.c:10670 xmlDocPtr xmlParseDoc(xmlChar *cur) { return(xmlSAXParseDoc(NULL, cur, 0)); } I think the code excerpt shown above is a good answer to your arguments.
> libxml2 is certainly not slow, it has a well known reputation as being > very fast for what it does.
I would not discuss reputation. It's a competely different thing. As with performance, I still insist that libxml2 has a) pretty slow parser due to inefficient algorithms and b) pretty slow in some other respects including freeing documents once again due to inefficietn algorithms. -Dmitri.

Christian Stocker

23 years ago
Hi Dimitri I dont have time nor i'm in the mood to answer to your allegations. You have obviously no idea, what you (or at least we) are talking about. Please update yourself and then try to make your claims again... chregu On Mon, 5 May 2003, Dmitri Dmitrienko wrote:
> Christian, > > >xmlParseFile does build a DOM-Tree out of your > > XML-Document, which is of > >course slower than the SAX-parsing expat is doing.. > > It is not obvious conclusion that any XMLDOM parsing should be slower. > More over it is competely wrong if you compare libxml2 vs expat. > > If you think more you'll see that DOM-parser only allocates nodes and link > them in lists. > Should it be so much slower ??? Are you sure that allocating nodes should > slow down everything by 14 times ? > I believe it is not. > > Also I don not see any reasonable explanation why libxml disposes document > so slow. > It does not need verify, it does not need parse, it's only fries nodes. > NOTHING MORE. > And takes nearly the same time as allocating/parsing and verifying. > > The only obvious conclusion is that all algorithms are written > inefficiently. > > I'm not against XML DOM and it's benefits. I'm against wrong and inefficient > algorithms. > People, why don't you read Donald Knouth's books ? > > > >_But_ libxml2 can parse your XML document in SAX-style only without > >building an DOM-Tree. Without looking at Sterling's code, I assume, > >that's what he did and you should compare this code to expat and _not_ > >xmlParseFile.. > > libxml2-2.5.7,parser.c:10670 > > xmlDocPtr > xmlParseDoc(xmlChar *cur) { > return(xmlSAXParseDoc(NULL, cur, 0)); > } > > I think the code excerpt shown above is a good answer to your arguments. > > > libxml2 is certainly not slow, it has a well known reputation as being > > very fast for what it does. > > I would not discuss reputation. It's a competely different thing. > As with performance, I still insist that libxml2 has a) pretty slow parser > due to inefficient algorithms and b) pretty slow in some other respects > including freeing documents once again due to inefficietn algorithms. > > > -Dmitri. > > > >
-- nam...christian stocker adr...pflanzschulstr. 31, ch-8004 zurich pho...+41 43 317 9984 www...http://blog.bitflux.ch mob...+41 76 561 8860 ema...chregu@phant.ch wor...+41 1 240 5670 gpg...0x5CE1DECB

Dmitri Dmitrienko

23 years ago
Hi Christian, I believe, you lost the matter of discussing. I pointed out some questions and neither were mentioned in your replies. Seems like you simply don't listen. What I did were not claims. I really don't care and have nothing to claim about. If php people would like to make by default such inefficient library it's ok. Nothing wrong for me personally. What I tried to point out is that _quality_ of interfaces does not mean _quality_ of underlying code. If you spent even a hour for learning libxml2 code you would not be so lucky with it. At least you'll be closer to my point of view. Parser is completely inefficient. XML Document handler is inefficient too. Efficiently is what my biggest concern. If you have nothing except personal charges please do not answer and perhaps read the forum rules. All the best -Dmitri "Christian Stocker" <chregu@bitflux.ch> wrote in message news:Pine.LNX.4.44.0305042203210.21304-100000@bambi.chregu.tv...

Sterling Hughes

23 years ago
> Ok. If you think I compare apples and oranges, well. Suppose I do. > But take a look what's finally offered by domxml. It's a verified tree > of nodes that were resulted from parsing original xml. > Nothing more nothing less.
Yes, a whole lot more. Look at the features at http://www.xmlsoft.org/. I'm not arguing this point anymore, just because you don't know libxml internals, doesn't mean it doesn't support an efficient SAX interface. Again, look at the Push parser interface. Look at the independent benchmarks.
> I think everybody clearly understand advantages of having such trees. > I'm actually not against it. It should be quite clear thought. > > Now about the matter of discussing. > It's performance. When I set proper callbacks for expat I get the same node > tree in 5 times faster.
expat doesn't support building a node tree. If you build a specialized node tree with expat, than sure it will be faster. But this isn't the point. The point is you are comparing two different interfaces, SAX and DOM. Expat only supports SAX, libxml2 supports SAX and DOM. Please compare apples with apples and benchmark the libxml2 SAX interface (push parser) interface, if you don't believe me.
> What this "good" domxml parser spends MY time for ? > Answer is very simple. Have a look at parser.c shipped with libxml2. > It's what I'd call geeze. It's written from scratches as if we are in 19th > century. > Ok Sterling if you think this approach is ok for all, why don't switch back > to the same parser for PHP ? > Let's introduce PHP 2.0 once again :))), geeze. >
Uhh. Again, look at the benchmarks online, when it comes to building a DOM tree libxml2 is the fastest available (besides msxml). But we're not talking about DOM here, we're talking about SAX. Libxml is not only a DOM library, its a complete library for processing XML. It supports DOM, but it also supports SAX.
> What I'd love to see is Flex-based lexer for xml that has proven its really > good performance. >
Uhh, flex, while easier to write, would be slower in this case. -Sterling XML Benchmarks: http://xmlbench.sourceforge.net/

Dmitri Dmitrienko

23 years ago
Sterling,
> Uhh, flex, while easier to write, would be slower in this case.
That's only words and your own impression. Try and compare. You'll see that you are wrong. Compare with php lexer. It takes only 3-4 ms to parse 1M of awesome php source. It's because of Flex's power. Without Flex you would write the first adequate php lexer till today. Dmitri.

Sterling Hughes

23 years ago
On Sun, 2003-05-04 at 16:04, Dmitri Dmitrienko wrote:
> Sterling, > > > Uhh, flex, while easier to write, would be slower in this case. > > That's only words and your own impression. > Try and compare. You'll see that you are wrong. > > Compare with php lexer. It takes only 3-4 ms to parse 1M of awesome php > source. > It's because of Flex's power. Without Flex you would write the first > adequate php lexer till today. >
Dmitri, Have you tried and compared? How do you know your right? A well written hand lexer will always be faster. Not only has it been shown (python, for example uses a custom coded scanner for speed), but it makes logical sense. First, flex builds LFA trees, which will always be slower matches then a custom built matcher. Second, flex can't be as intelligent about buffer handling as a custom parser could be. The problem is, usually its just not worth the time to build a custom scanner. It's much easier to use flex, which is very fast, to efficiently parse the document. In the case of XML, however, parsing must be efficient, and XML is a relatively uncomplex technology when it comes to the base scanning routines. Anyhow, this is offtopic, let it die. -Sterling

Dmitri Dmitrienko

23 years ago
Hi Sterling,
> Have you tried and compared? How do you know your right? A well > written hand lexer will always be faster. Not only has it been shown > (python, for example uses a custom coded scanner for speed), but it > makes logical sense. First, flex builds LFA trees, which will always be > slower matches then a custom built matcher. Second, flex can't be as > intelligent about buffer handling as a custom parser could be.
Okey. You arguments make sense. Unfortunately, I don't see good performance form libxml2. Let I make myself as clear as ever possible. I have my own library that builds xmldom. It's not very efficient and was written in one-two days so finally I was never satisfied with it. I had no time neither to rewrite nor to optimize it and I was always thinking about a good replacement. When I tried libxml2 I was deeply impressed. Let I compare the final results: operation my library libxml2 parse xml from file 177ms 267ms parse xml from memory 172ms 786ms free xml doc 18ms 133ms I know (BELIEVE ME that I know) my library is not efficient. That's why I was wondered with libxml2 results and started learning why. When I traced through all libxml code involved in parsing document I found many elementary pitfalls and concluded some ideas which I finally descibed in my messages posted at this forum. You said one great thing "A well written hand lexer". That's not the case.
> Anyhow, this is offtopic, let it die.
OKey. Let it die. That were my last arguments. All the best, Dmitri.

Adam Dickmeiss

23 years ago
On Sat, May 03, 2003 at 12:11:14PM -0400, Sterling Hughes wrote:
> Hi, >
[snip]
> 3) Proper unicode support
What is it that Expat misses? Expat itself MAY recognice the encoding in the header. However, the C code in PHP forces Expat to use some other fixed encoding by providing a NON-NULL pointer for XML_ParserCreate. Furthermore Expat allows you to plugin conversion handlers that allows for a substantial subset of characer sets out there. It is possible to use iconv for that purpose. (Expat call XML_SetUnknownEncodingHandler). Expat does not do much, but it does its job well and fast. -- Adam
> 4) Support for XPointer and XLink > > 5) Support for Docbook and HTML parsing > > 6) Expat doesn't even full support the same capabilities that libxml2 > does when it comes to SAX processing. > > However, when we bundle expat with PHP, and make ext/xml therefore an > "always available" extension. We create the illusion that it is the > "recommended" and "best" solution for XML parsing with PHP, when in fact > it really isn't. > > Our needs as far as XML support are growing, whether it be implementing > technologies that exist on top of XML (SOAP, WSDL, RDF) or implementing > extensions that make it easier to access XML (SimpleXML, DOM), expat > makes it way to hard (for all intensive purposes, impossible) to > implement these systems. > > Therefore, I'm suggesting that we bundle libxml2, while (for now) > keeping in expat as well. This will cause *absolutely* no backwards > compatibility changes, while at the same time, it will allow you to use > only libxml2 for XML processing (--without-bundle-expat), with 97% [2] > backwards compatibility maintained. > > -Sterling > > [1] http://news.php.net/article.php?group=php.xml.dev&article=6 > [2] This is one of 54% of facts made up on the spot. Suffice it to say > that the new extension is "mostly" backwards compatible, and the places > where it breaks, shouldn't have been relied upon anyway. > > -- > "Reductionists like to take things apart. The rest of us are > just trying to get it together." > - Larry Wall, Programming Perl, 3rd Edition > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php
-- Adam Dickmeiss mailto:adam@indexdata.dk http://www.indexdata.dk Index Data T: +45 33410100 Mob.: 212 212 66

Sterling Hughes

23 years ago
On Sun, 2003-05-04 at 08:36, Adam Dickmeiss wrote:
> On Sat, May 03, 2003 at 12:11:14PM -0400, Sterling Hughes wrote: > > Hi, > > > [snip] > > > 3) Proper unicode support > What is it that Expat misses? >
Exactly what you said. :) It doesn't handle these encodings natively, libxml2 does. Libxml2 can detect proper XML encodings, character sets, and use the appropriate functions. It also natively detects and uses iconv() to do encoding transformations. Its not that expat can't be made to properly support unicode, but it can't be said that it does. -Sterling
> Expat itself MAY recognice the encoding in the header. However, the C > code in PHP forces Expat to use some other fixed encoding by providing > a NON-NULL pointer for XML_ParserCreate. > > Furthermore Expat allows you to plugin conversion handlers that > allows for a substantial subset of characer sets out there. It is > possible to use iconv for that purpose. (Expat call > XML_SetUnknownEncodingHandler). > > Expat does not do much, but it does its job well and fast.
-- "The three most dangerous things in the world are a programmer with a soldering iron, a hardware type with a program patch and a user with an idea." - Unknown

Adam Dickmeiss

23 years ago
On Sun, May 04, 2003 at 11:21:19AM -0400, Sterling Hughes wrote:
> On Sun, 2003-05-04 at 08:36, Adam Dickmeiss wrote: > > On Sat, May 03, 2003 at 12:11:14PM -0400, Sterling Hughes wrote: > > > Hi, > > > > > [snip] > > > > > 3) Proper unicode support > > What is it that Expat misses? > > > > Exactly what you said. :) It doesn't handle these encodings natively, > libxml2 does. Libxml2 can detect proper XML encodings, character sets, > and use the appropriate functions. It also natively detects and uses > iconv() to do encoding transformations. Its not that expat can't be > made to properly support unicode, but it can't be said that it does.
Strickly speaking libxml2 doesn't do the conversions either. Both libs use iconv which is just fine. As I said a number times, the fact that Expat does not interpret encodings is due to ("bad") code in ext/xml. Will people still be able to configure PHP to use Expat for the basic functions (old xml) or will that wrapper disappear? I too am a little worried about performance. We have an application where many XML files are parsed to produce just one HTML page and we're probably not the only ones to do that. -- Adam
> -Sterling > > > Expat itself MAY recognice the encoding in the header. However, the C > > code in PHP forces Expat to use some other fixed encoding by providing > > a NON-NULL pointer for XML_ParserCreate. > > > > Furthermore Expat allows you to plugin conversion handlers that > > allows for a substantial subset of characer sets out there. It is > > possible to use iconv for that purpose. (Expat call > > XML_SetUnknownEncodingHandler). > > > > Expat does not do much, but it does its job well and fast. > > > -- > "The three most dangerous things in the world are a programmer > with a soldering iron, a hardware type with a program patch and > a user with an idea." > - Unknown > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php
-- Adam Dickmeiss mailto:adam@indexdata.dk http://www.indexdata.dk Index Data T: +45 33410100 Mob.: 212 212 66

Sterling Hughes

23 years ago
On Sun, 2003-05-04 at 16:59, Adam Dickmeiss wrote:
> On Sun, May 04, 2003 at 11:21:19AM -0400, Sterling Hughes wrote: > > On Sun, 2003-05-04 at 08:36, Adam Dickmeiss wrote: > > > On Sat, May 03, 2003 at 12:11:14PM -0400, Sterling Hughes wrote: > > > > Hi, > > > > > > > [snip] > > > > > > > 3) Proper unicode support > > > What is it that Expat misses? > > > > > > > Exactly what you said. :) It doesn't handle these encodings natively, > > libxml2 does. Libxml2 can detect proper XML encodings, character sets, > > and use the appropriate functions. It also natively detects and uses > > iconv() to do encoding transformations. Its not that expat can't be > > made to properly support unicode, but it can't be said that it does. > > Strickly speaking libxml2 doesn't do the conversions either. Both libs > use iconv which is just fine. As I said a number times, the fact that > Expat does not interpret encodings is due to ("bad") code in ext/xml. >
Well, strictly speaking ;-), expat doesn't support UTF-16, libxml2 does.
> Will people still be able to configure PHP to use Expat for the basic > functions (old xml) or will that wrapper disappear? I too am a little > worried about performance. We have an application where many XML files > are parsed to produce just one HTML page and we're probably not the only > ones to do that.
Don't worry about Dmitri's claims, they are as bogus as bogus can get. Libxml2 is equally as fast as expat (its a matter of a few clock ticks difference.) But, as I stated in my original message, both expat and libxml2 will be supported - in fact, that's the whole point of the bundling schema, and the expat compatibility layer (C level). -Sterling
-- "Whether you think you can or think you can't -- you are right." - Henry Ford

Adam Dickmeiss

23 years ago
On Sun, May 04, 2003 at 03:39:08PM -0400, Sterling Hughes wrote:
> On Sun, 2003-05-04 at 16:59, Adam Dickmeiss wrote: > > On Sun, May 04, 2003 at 11:21:19AM -0400, Sterling Hughes wrote: > > > On Sun, 2003-05-04 at 08:36, Adam Dickmeiss wrote: > > > > On Sat, May 03, 2003 at 12:11:14PM -0400, Sterling Hughes wrote: > > > > > Hi, > > > > > > > > > [snip] > > > > > > > > > 3) Proper unicode support > > > > What is it that Expat misses? > > > > > > > > > > Exactly what you said. :) It doesn't handle these encodings natively, > > > libxml2 does. Libxml2 can detect proper XML encodings, character sets, > > > and use the appropriate functions. It also natively detects and uses > > > iconv() to do encoding transformations. Its not that expat can't be > > > made to properly support unicode, but it can't be said that it does. > > > > Strickly speaking libxml2 doesn't do the conversions either. Both libs > > use iconv which is just fine. As I said a number times, the fact that > > Expat does not interpret encodings is due to ("bad") code in ext/xml. > > > > Well, strictly speaking ;-), expat doesn't support UTF-16, libxml2 does.
Exactly:) Expat supports UTF-16. But for whatever reason the coders of ext/xml didn't like it.
> > Will people still be able to configure PHP to use Expat for the basic > > functions (old xml) or will that wrapper disappear? I too am a little > > worried about performance. We have an application where many XML files > > are parsed to produce just one HTML page and we're probably not the only > > ones to do that. > > Don't worry about Dmitri's claims, they are as bogus as bogus can get. > Libxml2 is equally as fast as expat (its a matter of a few clock ticks > difference.) But, as I stated in my original message, both expat and > libxml2 will be supported - in fact, that's the whole point of the > bundling schema, and the expat compatibility layer (C level).
A few hundred thousand ticks probably. Time will tell. It's a win-win when one can choose either. So I'm happy. I can't seem to see your new stuff in CVS, yet. Right? -- Adam
> -Sterling > -- > "Whether you think you can or think you can't -- you are right." > - Henry Ford > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php
-- Adam Dickmeiss mailto:adam@indexdata.dk http://www.indexdata.dk Index Data T: +45 33410100 Mob.: 212 212 66

Sterling Hughes

23 years ago
> > Well, strictly speaking ;-), expat doesn't support UTF-16, libxml2 does. > Exactly:) Expat supports UTF-16. But for whatever reason the coders > of ext/xml didn't like it. >
No, expat itself does *not* support UTF-16 - not really. It allows everything to be handled internally as UTF-16, but it requires you to be unicode safe as well. libxml2 allows the document to be in UTF-16, however, you can handle it as UTF-8 (or ISO-8859-1).
> > > Will people still be able to configure PHP to use Expat for the basic > > > functions (old xml) or will that wrapper disappear? I too am a little > > > worried about performance. We have an application where many XML files > > > are parsed to produce just one HTML page and we're probably not the only > > > ones to do that. > > > > Don't worry about Dmitri's claims, they are as bogus as bogus can get. > > Libxml2 is equally as fast as expat (its a matter of a few clock ticks > > difference.) But, as I stated in my original message, both expat and > > libxml2 will be supported - in fact, that's the whole point of the > > bundling schema, and the expat compatibility layer (C level). > A few hundred thousand ticks probably. Time will tell. It's > a win-win when one can choose either. So I'm happy. >
Yeah, but a few hundred thousand ticks is nothing.
> I can't seem to see your new stuff in CVS, yet. Right? >
Yep. Unless objections come up, I'll commit it before I head to amdam on tuesday. -Sterling
-- "A business that makes nothing but money is a poor kind of business." - Henry Ford

Adam Dickmeiss

23 years ago
On Sun, May 04, 2003 at 04:34:28PM -0400, Sterling Hughes wrote:
> > > > Well, strictly speaking ;-), expat doesn't support UTF-16, libxml2 does. > > Exactly:) Expat supports UTF-16. But for whatever reason the coders > > of ext/xml didn't like it. > > > > No, expat itself does *not* support UTF-16 - not really. It allows > everything to be handled internally as UTF-16, but it requires you to be > unicode safe as well. libxml2 allows the document to be in UTF-16, > however, you can handle it as UTF-8 (or ISO-8859-1).
Expat supports the parsing of an (external) file in UTF-16. So input encoding is UTF-16. Internally (from C) the representation is UTF-8 (output encoding). You can also compile Expat to use UTF-16 internally, but that's not what I want. I've just indexed (parsed) a file in UTF-16 encoding using Expat from C and I get UTF-8 strings in handlers. In other words Expat does exactly the same as libxml2 for encodings, provided that Expat is used properly. UTF-16 encoding was probably omitted from ext/xml due to confusion about encodings (Input/external versus output from PHP/C). -- Adam
-- > > > > > Will people still be able to configure PHP to use Expat for the basic > > > > functions (old xml) or will that wrapper disappear? I too am a little > > > > worried about performance. We have an application where many XML files > > > > are parsed to produce just one HTML page and we're probably not the only > > > > ones to do that. > > > > > > Don't worry about Dmitri's claims, they are as bogus as bogus can get. > > > Libxml2 is equally as fast as expat (its a matter of a few clock ticks > > > difference.) But, as I stated in my original message, both expat and > > > libxml2 will be supported - in fact, that's the whole point of the > > > bundling schema, and the expat compatibility layer (C level). > > A few hundred thousand ticks probably. Time will tell. It's > > a win-win when one can choose either. So I'm happy. > > > > Yeah, but a few hundred thousand ticks is nothing. > > > I can't seem to see your new stuff in CVS, yet. Right? > > > > Yep. Unless objections come up, I'll commit it before I head to amdam on tuesday. > > -Sterling > > > -- > "A business that makes nothing but money is a poor kind of business." > - Henry Ford > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php -- Adam Dickmeiss mailto:adam@indexdata.dk http://www.indexdata.dk Index Data T: +45 33410100 Mob.: 212 212 66

Moriyoshi Koizumi

23 years ago
Sterling Hughes <sterling@bumblebury.com> wrote:
> On Sun, 2003-05-04 at 16:59, Adam Dickmeiss wrote: > > On Sun, May 04, 2003 at 11:21:19AM -0400, Sterling Hughes wrote: > > > On Sun, 2003-05-04 at 08:36, Adam Dickmeiss wrote:
<snip>
> > > Exactly what you said. :) It doesn't handle these encodings natively, > > > libxml2 does. Libxml2 can detect proper XML encodings, character sets, > > > and use the appropriate functions. It also natively detects and uses > > > iconv() to do encoding transformations. Its not that expat can't be > > > made to properly support unicode, but it can't be said that it does. > > > > Strickly speaking libxml2 doesn't do the conversions either. Both libs > > use iconv which is just fine. As I said a number times, the fact that > > Expat does not interpret encodings is due to ("bad") code in ext/xml. > > > > Well, strictly speaking ;-), expat doesn't support UTF-16, libxml2 does.
As for unicode stuff, iconv is not always responsible. It is natively handled within libxml2. iconv is needed to parse XMLs encoded in other than UTFs AFAIK. Moriyoshi

Adam Dickmeiss

23 years ago
On Mon, May 05, 2003 at 07:21:25AM +0900, Moriyoshi Koizumi wrote:
> Sterling Hughes <sterling@bumblebury.com> wrote: > > > On Sun, 2003-05-04 at 16:59, Adam Dickmeiss wrote: > > > On Sun, May 04, 2003 at 11:21:19AM -0400, Sterling Hughes wrote: > > > > On Sun, 2003-05-04 at 08:36, Adam Dickmeiss wrote: > <snip> > > > > Exactly what you said. :) It doesn't handle these encodings natively, > > > > libxml2 does. Libxml2 can detect proper XML encodings, character sets, > > > > and use the appropriate functions. It also natively detects and uses > > > > iconv() to do encoding transformations. Its not that expat can't be > > > > made to properly support unicode, but it can't be said that it does. > > > > > > Strickly speaking libxml2 doesn't do the conversions either. Both libs > > > use iconv which is just fine. As I said a number times, the fact that > > > Expat does not interpret encodings is due to ("bad") code in ext/xml. > > > > > > > Well, strictly speaking ;-), expat doesn't support UTF-16, libxml2 does. > > As for unicode stuff, iconv is not always responsible. It is natively > handled within libxml2. iconv is needed to parse XMLs encoded in other > than UTFs AFAIK.
Just like Expat, which does UTF-16, UTF-8 natively. -- Adam
> Moriyoshi > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php
-- Adam Dickmeiss mailto:adam@indexdata.dk http://www.indexdata.dk Index Data T: +45 33410100 Mob.: 212 212 66

Andi Gutmans

23 years ago
At 10:39 PM 5/4/2003 -0400, Sterling Hughes wrote:
> > Will people still be able to configure PHP to use Expat for the basic > > functions (old xml) or will that wrapper disappear? I too am a little > > worried about performance. We have an application where many XML files > > are parsed to produce just one HTML page and we're probably not the only > > ones to do that. > >Don't worry about Dmitri's claims, they are as bogus as bogus can get. >Libxml2 is equally as fast as expat (its a matter of a few clock ticks >difference.) But, as I stated in my original message, both expat and >libxml2 will be supported - in fact, that's the whole point of the >bundling schema, and the expat compatibility layer (C level).
I'm a few weeks behind but I'm just curious, will the compatibility layer limit the extension when used with libxml2? (i.e. least common denominator?) Are we going to get good docs for all the new features of libxml2? :) Andi