Everything

php

Home » Project » Newsgroups » php.internals » PHP Unicode support design document »

PHP Unicode support design document

php.internals

Andrei Zmievski

21 years ago

Before we go breaking things, please read this document that
describes how PHP will support the Unicode standard natively.
Hopefully the attachment will work.

Thanks,

-Andrei

Ron Korving

21 years ago

This looks very promising, I'm impressed by the work you guys have done (big
thumbs up).

There are a few issues/questions I have after reading your document:


"Therefore, command such as 'print' and 'echo' automatically convert their
arguments to the specified encoding. No automatic output encoding is
performed for anything else."

What about the other functions that output to stdout directly, such as
readfile() and passthru()?


"The conversion failure behavior can be customized"...

Maybe it would be a nice feature to have an U_INVALID_EXCEPTION, so that
users can actually catch the error and deal with it. Just an idea. Of course
it's not usual for the PHP core and extensions to throw exceptions, but
perhaps this could change with PHP6.


"In order to create binary string literals, a new syntax is necessary:
prefixing a string literal with letter 'b' creates a binary string."

The b-prefix for binary strings is great, but how does that work with a
function like file_get_contents() or fread() ?
One can't do: $data = bfile_get_contents("somefile.bin");
And even if one could (somehow), wouldn't file_get_contents() already
unicode-encode all data it reads? How does such a function know if the user
is expecting binary or textual data or does the encoding simply happen after
the string is returned? In that case it's up to the user to use the
b-prefix, but then there's the syntax problem I mentioned.


Keep up the good work,

Ron

Antony Dovgal

21 years ago

On Wed, 10 Aug 2005 12:45:27 +0200
"Ron Korving" <r.korving@xit.nl> wrote:

> This looks very promising, I'm impressed by the work you guys have done (big
> thumbs up).
>
> There are a few issues/questions I have after reading your document:
>
>
> "Therefore, command such as 'print' and 'echo' automatically convert their
> arguments to the specified encoding. No automatic output encoding is
> performed for anything else."

That's actually something I wanted to ask about too.

Do we really need such kind of magic?

I think it may be pretty confusing when after echo'ing or print'ing a variable
you can see one output, but after writing the very same variable into a file
you can see something completely different.

IMO it's similar to what we have with __toString() ATM.
Yes, it's documented, but it's *still* confusing that there is some magic
involved in one case and there is no magic in an other, almost similar case.

--
Wbr,
Antony Dovgal

Andrei Zmievski

21 years ago

On Aug 10, 2005, at 3:54 AM, Antony Dovgal wrote:

> Do we really need such kind of magic?
>
> I think it may be pretty confusing when after echo'ing or print'ing a
> variable
> you can see one output, but after writing the very same variable into
> a file
> you can see something completely different.

Absolutely, we do need it. Consider that the internal encoding is
UTF-16 and outputting that directly to a terminal (or browser) is bound
to cause havoc. That's just one of the examples.

-Andrei

Derick Rethans

21 years ago

On Wed, 10 Aug 2005, Ron Korving wrote:

> "In order to create binary string literals, a new syntax is necessary:
> prefixing a string literal with letter 'b' creates a binary string."
>
> The b-prefix for binary strings is great, but how does that work with a
> function like file_get_contents() or fread() ?
> One can't do: $data = bfile_get_contents("somefile.bin");

fopen() and file_get_contents() already understands a context parameter,
specifying whethter you'd want to have binary or string/unicode data can
be done through that.

and the b syntax, only works for literal strings in your code:
b"foo", but b$foo is not going to work.

Derick

--
Derick Rethans
http://derickrethans.nl | http://ez.no | http://xdebug.org

Christian Schneider

21 years ago

Derick Rethans wrote:

> On Wed, 10 Aug 2005, Ron Korving wrote:
>>"In order to create binary string literals, a new syntax is necessary:
>>prefixing a string literal with letter 'b' creates a binary string."
>>
>>The b-prefix for binary strings is great, but how does that work with a
>>function like file_get_contents() or fread() ?
>>One can't do: $data = bfile_get_contents("somefile.bin");
>
> fopen() and file_get_contents() already understands a context parameter,
> specifying whethter you'd want to have binary or string/unicode data can
> be done through that.

We create images in PHP scripts and pass them through with
readfile("foo,gif"). Did I understand correctly that this would still
work without changes? But echo file_get_contents("foo.gif") would fail,
right?

This is not a complaint, just trying to understand the implications,
- Chris

Andi Gutmans

21 years ago

We need to automatically convert the output as internally we will be
storing UTF-16 which is not what you want to send to the user. The SAPI
output mechanism does the conversion, I don't think it's print & echo. It
will actually save people a lot of headache that this is done automatically.
As far as files are concerned, the default is also to convert to the INI
encoding (forgot which INI parameter), but we will supply streams which
allow you to control the in/out encoding of specific files.

So basically, I think we need to update the doc as I am pretty sure we
didn't change print/echo but the underlying input/output mechanisms.

Andi


At 02:54 PM 8/10/2005 +0400, Antony Dovgal wrote:

Rasmus Lerdorf

21 years ago

Yeah, print/echo was just a way of describing the underlying output
stuff.  It wasn't meant to be taken literally.

-Rasmus

Andi Gutmans wrote:

George Schlossnagle

21 years ago

On Aug 10, 2005, at 10:30 AM, Rasmus Lerdorf wrote:

> Yeah, print/echo was just a way of describing the underlying output
> stuff.  It wasn't meant to be taken literally.

Given the __toString fiasco, it's understandable that this would be
confusing though.

George

Ron Korving

21 years ago

Exactly. That's how I understood it too: "Ah, the __toString behavior". I'm
very glad this is not the case.

Ron


"George Schlossnagle" <george@omniti.com> schreef in bericht
news:1B80531E-3842-4DF6-B9D5-F63A8AD03C2D@omniti.com...

Marcus Börger

21 years ago

Hello Ron,

i had a chat with Andi about __toString() and i hope that he finally
undestood why a lot of ppl wanted it right from the beginning. To me the
current situation is simply the worst case because noone understnds when it
works and when not (. vs ,). Since we are doing a drastic change in string
behavior anyway and this time have enough time until next release i already
agreed on taking care about this one as early as possible. This way we will
have enough time to find any places where __toString() is not possible.
Also Andi and i discussed again the major problem and i could ensure Andi
that there is a simple solution which he agreed on. That is we require
__toString() to return a string and do a non interruptable halt otherwise.
Maybe in a later version this can be replaced by an exception we'll see
if someone finds the time to analyse that in detail.

best regards
marcus

Wednesday, August 10, 2005, 4:50:17 PM, you wrote:

> Exactly. That's how I understood it too: "Ah, the __toString behavior". I'm
> very glad this is not the case.

> Ron

> "George Schlossnagle" <george@omniti.com> schreef in bericht
> news:1B80531E-3842-4DF6-B9D5-F63A8AD03C2D@omniti.com...
>>
>> On Aug 10, 2005, at 10:30 AM, Rasmus Lerdorf wrote:
>>
>> > Yeah, print/echo was just a way of describing the underlying output
>> > stuff.  It wasn't meant to be taken literally.
>>
>> Given the __toString fiasco, it's understandable that this would be
>> confusing though.
>>
>> George

Best regards,
Marcus

Ron Korving

21 years ago

Sounds absolutely great :)

Ron

"Marcus Boerger" <helly@php.net> wrote in message
news:1299141168.20050810212621@marcus-boerger.de...

> Hello Ron,
>
>   i had a chat with Andi about __toString() and i hope that he finally
> undestood why a lot of ppl wanted it right from the beginning. To me the
> current situation is simply the worst case because noone understnds when

it

> works and when not (. vs ,). Since we are doing a drastic change in string
> behavior anyway and this time have enough time until next release i

already

> agreed on taking care about this one as early as possible. This way we

will

> have enough time to find any places where __toString() is not possible.
> Also Andi and i discussed again the major problem and i could ensure Andi
> that there is a simple solution which he agreed on. That is we require
> __toString() to return a string and do a non interruptable halt otherwise.
> Maybe in a later version this can be replaced by an exception we'll see
> if someone finds the time to analyse that in detail.
>
> best regards
> marcus
>
> Wednesday, August 10, 2005, 4:50:17 PM, you wrote:
>
> > Exactly. That's how I understood it too: "Ah, the __toString behavior".

I'm

Adam Maccabee Trachtenberg

21 years ago

On Wed, 10 Aug 2005, Marcus Boerger wrote:

>   i had a chat with Andi about __toString() and i hope that he finally
> undestood why a lot of ppl wanted it right from the beginning. To me the
> current situation is simply the worst case because noone understnds when it
> works and when not (. vs ,).

Yea, this is super tricky and subtle because even though print/echo
and ,/. are different, they appear to be identical in almost all other
cases.


I agree that forcing __toString() to return a string is perfectly
reasonable.

-adam

--
adam@trachtenberg.com | http://www.trachtenberg.com
author of o'reilly's "upgrading to php 5" and "php cookbook"
avoid the holiday rush, buy your copies today!

Antony Dovgal

21 years ago

Ah.
Ok, then I'm happy =)


On Wed, 10 Aug 2005 07:30:38 -0700
Rasmus Lerdorf <rasmus@lerdorf.com> wrote:

> Yeah, print/echo was just a way of describing the underlying output
> stuff.  It wasn't meant to be taken literally.

--
Wbr,
Antony Dovgal

Andrei Zmievski

21 years ago

We have not changed the underlying output mechanism. The transcoding
is done by zend_make_printable_zval().

-Andrei

On Aug 10, 2005, at 7:30 AM, Rasmus Lerdorf wrote:

Rasmus Lerdorf

21 years ago

Andrei Zmievski wrote:

> We have not changed the underlying output mechanism. The transcoding  is
> done by zend_make_printable_zval().

Ok, but all the non-stream based output functions pass through that.
Not that we have very many, but it is more than just echo/print.

-Rasmus

Ron Korving

21 years ago

I firmly believe though, that all outputting functions should act the same.
It's the same problem otherwise as with __toString(). I would use __toString
if it wasn't just restricted to echo and print, but right now it's pretty
useless to me. I hope that behavior can change in a major version update
(5.1, 6.0).

Ron


"Rasmus Lerdorf" <rasmus@lerdorf.com> wrote in message
news:42FA261B.9080300@lerdorf.com...

Andrei Zmievski

21 years ago

On Aug 10, 2005, at 7:26 AM, Andi Gutmans wrote:

> We need to automatically convert the output as internally we will
> be storing UTF-16 which is not what you want to send to the user.
> The SAPI output mechanism does the conversion, I don't think it's
> print & echo. It will actually save people a lot of headache that
> this is done automatically.

That's not true, actually. 'echo' and 'print' resolve to ZEND_ECHO
opcode which calls zend_print_variable(), which in turn calls
zend_make_printable_zval(). Now, this last function is supposed to
take a zval and turn it into a printable string, of course, which is
then output using utility_functions->write_function aka php_body_write
(). All that function cares about is how to output a binary string.
So, if we want to bubble the conversion down to the output layer, we
probably need to change the write function so that it takes a void*
and a type and knows how to deal with them appropriately.

Functions like readfile() are based on streams, so by default they
will be in binary mode, simply passing the data through, unless you
put a filter on it or change the default context.

> As far as files are concerned, the default is also to convert to
> the INI encoding (forgot which INI parameter), but we will supply
> streams which allow you to control the in/out encoding of specific
> files.

The filename_encoding setting is supposed to be used only for
filenames, not for file contents. That is dealt with using the
mechanisms that Sara implemented in the streams system.

-Andrei

Andrei Zmievski

21 years ago

I did not have time to write the full reply earlier so here goes.

Even if we modify the output layer to be aware of various types of
strings coming down the pipe, it would still need to know the encoding
of IS_STRING's in order to convert them to the output encoding. This
presents a particular problem for inline HTML blocks, as they are
supposed to be in the script encoding, but by the time the HTML is sent
to the output layer, we don't know what the source script encoding was
for these HTML blocks. This problem exists in the current
implementation also, because the ZEND_ECHO opcode does not keep track
of what the script encoding was. This needs to be fixed, obviously.

One approach could be to implement a separate opcode for inline HTML
blocks and store the name of the script encoding it came from in the
opcode. Then when the output layer (or whatever else) gets to it, we
can check the encoding name in the opcode vs. the output encoding and
perform transcoding if necessary. This does mean that we may need to
dynamically open and close converters on each output (if there were
different script encodings floating around), but can be alleviated by
keeping some sort of converter cache around.

I am open to other ideas.

-Andrei

On Aug 10, 2005, at 8:34 AM, Andrei Zmievski wrote:

Andi Gutmans

21 years ago

Wouldn't it be easiest to have inline html become IS_UNICODE and then not
deal with the problem of remember what the script encoding was? I thought
that's what we already do today.

Andi

At 12:37 PM 8/10/2005 -0700, Andrei Zmievski wrote:

Rasmus Lerdorf

21 years ago

I think the main issue here is that if your script encoding is set to
UTF-8 and you do everything in UTF-8 then these large blocks of UTF-8
are going to make a UTF-8 -> UTF-16 -> UTF-8 conversion roundtrip on
every request.  It would be nice if we could somehow avoid that.

-Rasmus

Andi Gutmans wrote:

Andi Gutmans

21 years ago

If you want to optimize then I guess "remembering" the script_encoding is
the only way to do it. We could do it similar to the way we "cache" script
file names.
Another option is to just optimize for UTF-8 and use BOMs for UTF-8/UTF-16...

Andi

At 03:09 PM 8/15/2005 -0700, Rasmus Lerdorf wrote:

Andrei Zmievski

21 years ago

Where should we save the script encoding from which an oparray was
built? In the oparray itself?

-Andrei

On Aug 15, 2005, at 3:13 PM, Andi Gutmans wrote:

Andrei Zmievski

21 years ago

We certainly could, but we lose some speed, especially when
script_encoding == output_encoding (where we don't really need to
transcode HTML blocks). Are we up for that?

-Andrei

On Aug 15, 2005, at 3:03 PM, Andi Gutmans wrote:

Andrei Zmievski

21 years ago

On Aug 10, 2005, at 3:45 AM, Ron Korving wrote:

> This looks very promising, I'm impressed by the work you guys have
> done (big
> thumbs up).

Thanks.

> What about the other functions that output to stdout directly, such as
> readfile() and passthru()?

readfile() uses streams so it would rely on stream filters and such.
passthru() should probably operate in binary mode.

> Maybe it would be a nice feature to have an U_INVALID_EXCEPTION, so
> that
> users can actually catch the error and deal with it. Just an idea. Of
> course
> it's not usual for the PHP core and extensions to throw exceptions, but
> perhaps this could change with PHP6.

I think the feature of raising exceptions vs. errors is orthogonal to
what the switch does. Consider that you may want the
skip/substitute/escape performed and then raise an error or not.

> The b-prefix for binary strings is great, but how does that work with a
> function like file_get_contents() or fread() ?
> One can't do: $data = bfile_get_contents("somefile.bin");
> And even if one could (somehow), wouldn't file_get_contents() already
> unicode-encode all data it reads? How does such a function know if the
> user
> is expecting binary or textual data or does the encoding simply happen
> after
> the string is returned? In that case it's up to the user to use the
> b-prefix, but then there's the syntax problem I mentioned.

'b' prefix is only for string literals. file_get_contents(), fread()
and other streams-based functions use the default stream semantics,
meaning that unless you change the default context, the data returned
by them will be of IS_BINARY type. The default context can contain a
filter that decodes the data from the specified encoding into Unicode.

-Andrei

Ondrej Ivanič

21 years ago

Andrei Zmievski wrote:

>     + Determining length of Unicode strings via strlen() function, some
>       simple string functions ported (substr).

It's not a problem to determine kind of char in single byte character
sets, but in the unicode with various encoding schemas I don't see easy
way how to do it.

It will be nice to have functions like this: isNumber(char),
isAlphabetic(char), isWhitespace(char) ...

It is on the plan or not?

--
Ondrej Ivanic
(ondrej@kmit.sk)

l0t3k

21 years ago

>
> It will be nice to have functions like this: isNumber(char),
> isAlphabetic(char), isWhitespace(char) ...
>
> It is on the plan or not?

its done already, just not committed yet...

clayton

""Ondrej Ivanic"" <ondrej@kmit.sk> wrote in message
news:4301A0D6.6000205@kmit.sk...

Andrey Hristov

21 years ago

cshmoove@bellsouth.net wrote:

>>It will be nice to have functions like this: isNumber(char),
>>isAlphabetic(char), isWhitespace(char) ...
>>
>>It is on the plan or not?
>
>
> its done already, just not committed yet...
>
>
> clayton
>
> ""Ondrej Ivanic"" <ondrej@kmit.sk> wrote in message
> news:4301A0D6.6000205@kmit.sk...
>
>>Andrei Zmievski wrote:
>>--
>>Ondrej Ivanic
>>(ondrej@kmit.sk)
>
>
>

Please don't use stupid caps, these are functions not methods.

Andrey

l0t3k

21 years ago

> Please don't use stupid caps, these are functions not methods.
>
>
> Andrey

of course not. see
http://icu.sourceforge.net/apiref/icu4c/uchar_8h.html

, but note that functions conform to PHP's function naming conventions
( lower_case_words_separated_by_underscores() ).

clayton

Peter Brodersen

21 years ago

On Wed, 10 Aug 2005 00:31:30 -0700, in php.internals
andrei@gravitonic.com (Andrei Zmievski) wrote:

>  - existing PHP escape sequences are also interpreted as Unicode codepoints,
>    including \xXX (hex) and \OOO (octal) numbers, e.g. "\x20" => U+0020

[..]

>The single-quoted string is more restrictive than the other two types: so
>far the only escape sequence allowed inside of it was \', which specifies
>a literal single quote. However, single quoted strings now support the new
>Unicode character escape sequences as well.

For what it's worth, would \1 be interpreted as well in single quotes
(as it currently is in double quotes)?

I suppose one of the places where \digit would be present in several
cases is in poor-written pregs - such as:
print preg_replace('/([A-Z])/','<b>\1</b>',$string);
(where \1 is used as backreference instead of \\1 or $1)

I'm not that worried about my own preg-usage. I just want to be
prepared if I ever have to review some code for the purpose of
migrating to PHP6.

--
- Peter Brodersen

Andrei Zmievski

21 years ago

On Aug 16, 2005, at 2:57 PM, Peter Brodersen wrote:

> For what it's worth, would \1 be interpreted as well in single quotes
> (as it currently is in double quotes)?

No. Only \u and \U have meaning in single quotes (in addition to
current ones).

-Andrei