Everything

php

Re: PHP Unicode support design document

php.internals

Makoto Tozawa

21 years ago

This looks good. When the unicode_semantics switch is turned on, it
provides Unicode everywhere development solution to PHP developers.
The output, input, and script encodings are all utf-8 by default.
The internal encoding is utf-16 which allows developers to handle
surrogate pairs correctly.

I have one issue and one question.

"HTTP Input Encoding
...
If the HTTP request contains the encoding specification in the headers,
then it will be used instead of this setting."

With my best knowledge there isn't such http request header which
specifies the encoding of the request. In case the intent is to honor
the ACCEPT-CHARSET, it may cause a problem because browsers don't
gurantee the encoding in the ACCEPT-CHARSET is same as the encoding
used to escape characters in the URL query string. After all, the
ACCEPT-CHARSET is to specify the character encodings acceptable for
the response.


"Upgrading Existing Functions"

It seems that all the existing functions need to be upgraded to work
properly
when unicode_semantics switch is turned on becuase it changes the
semantics of
fundamental functions. I'm assuming all the existing functions don't work
properly if fundamental functions such as strlen() behave differently.

Is there any way to keep the byte semantics (in oppose to unicode
semantics)
only for the existing functions? For example, the Oracle 8 functions can be
configured to use utf-8 for the character encoding of strings. In order for
them to work properly, fundamental functions, which Oracle 8 function call,
have to behave in byte samentics. And if they work properly when the
unicode
semantics switch is turned on, by setting the runtime_encoding to utf-8,
they can be called by uncode applications.


Makoto

Andrei Zmievski

21 years ago

Hi,

On Aug 23, 2005, at 7:30 PM, Makoto Tozawa wrote:

> "HTTP Input Encoding
> ...
> If the HTTP request contains the encoding specification in the headers,
> then it will be used instead of this setting."
>
> With my best knowledge there isn't such http request header which
> specifies the encoding of the request. In case the intent is to honor
> the ACCEPT-CHARSET, it may cause a problem because browsers don't
> gurantee the encoding in the ACCEPT-CHARSET is same as the encoding
> used to escape characters in the URL query string. After all, the
> ACCEPT-CHARSET is to specify the character encodings acceptable for
> the response.

I took a closer look at this today and RFC 2616 does not specify
whether user agents are supposed to send a charset parameter in the
Content-Type header of the POST request. I did not see any of my
browsers doing so. I think we can safely disregard this and rely on
http_input_encoding and output_encoding settings. We are not going to
use Accept-Charset for the reasons you mention.

> Is there any way to keep the byte semantics (in oppose to unicode
> semantics)
> only for the existing functions? For example, the Oracle 8 functions
> can be
> configured to use utf-8 for the character encoding of strings. In
> order for
> them to work properly, fundamental functions, which Oracle 8 function
> call,
> have to behave in byte samentics. And if they work properly when the
> unicode
> semantics switch is turned on, by setting the runtime_encoding to
> utf-8,
> they can be called by uncode applications.

I couldn't parse this on the first try. Could you restate this?

-Andrei

Makoto Tozawa

21 years ago

Andrei Zmievski wrote:

>> Is there any way to keep the byte semantics (in oppose to unicode
>> semantics)
>> only for the existing functions? For example, the Oracle 8 functions
>> can be
>> configured to use utf-8 for the character encoding of strings. In
>> order for
>> them to work properly, fundamental functions, which Oracle 8 function
>> call,
>> have to behave in byte samentics. And if they work properly when the
>> unicode
>> semantics switch is turned on, by setting the runtime_encoding to utf-8,
>> they can be called by uncode applications.
>
> I couldn't parse this on the first try. Could you restate this?

Say there is a function which calls strlen($s) expecting it returns byte
size of $s,
and it is working fine when $s constains multibyte characters. For example,
the function expects strlen('áéí') returns 6 when the encoding is utf-8.
If this function is called by Uniocde ready applications on
Unicode-enabled PHP,
it will fall into error because strlen('áéí') will return 3.

Is there any way to let strlen('áéí') return 6 only when it is called by
the existing function?

Hope I explained well this time.

Makoto

Tex Texin

21 years ago

1) sorry I am compelled to change the subject so all threads don’t look the same.

2) It's a no-win situation. If we don't make the functions provide reasonable behavior for unicode, then every program needs to be rewritten to change function names.
The number of places where hard coded constants (6) are used is probably much smaller.

At least this way most code does the right thing as-is.

Also, if you don't want functions to show unicode behavior, leave unicode off, and just convert the data to utf-8.

We do need to have functions that provide the raw byte length, so it will be available.

Tex Texin
Internationalization Architect,   Yahoo! Inc.

Makoto Tozawa

21 years ago

> If we don't make the functions provide reasonable behavior for

unicode, then every program needs to be rewritten to change function names.

I agree. I asked it because the Backwards Compatibility section states
the following:

"... the upgrade to Unicode-enabled PHP has to be transparent. This
means that the existing data types and functions must work as they have
always done."

For those functions written for single byte encoding, the upgrade to
Unicode-enabled PHP will be transparent because the character semantics
remains same. For those functions written for multi byte encoding using
mb_str*() functions, it will be also transparent.

It is okay if there is no way to save those functions written for multi
byte encoding abusing the str*() functions.

Makoto


Tex Texin wrote:

Tex Texin

21 years ago

Makoto,

ok, thanks. Now I see. You are saying that the multi-byte extension took the opposite approach and made existing str*() byte oriented and the mb_str*() character oriented.

So we have:
s2m) Migration from singlebyte to multibyte: change code that needs character units to use mb*()

s2u) Migration from singlebyte to (new) Unicode: change code that needs byte units to use new functions.

m2u) Migration from multibyte to Unicode: change str*() code that needs byte units to use new functions and (perhaps) change mb*() functions back to str*().

ugh!
and some folks are using the multibyte extension as their current unicode solution so that the case m2u additionally represents migration from unicode to unicode when the php version changes.

That would bear some additional consideration.

But, in looking at the mb* doc, the str*() functions can be overloaded to use the mb*() character semantics. If a good number of users do that, then it isn't much consequence either way (ie it is no-win), and that puts us back to the original proposal.

Is that right?

Tex Texin
Internationalization Architect,   Yahoo! Inc.

Makoto Tozawa

21 years ago

Yes.

Makoto

Tex Texin wrote:

Tex Texin

21 years ago

With respect to the encoding of a form, several concepts are being mixed
together.
Here is my brief take on it.

First, if nothing else is done, the form will be in the encoding of the
containing web page.
It is true that the http request to the server will not contain a
content-type charset identification.

However, the HTML FORM element allows the user to specify the accept-charset
attribute to identify the encodings the server will understand. The
recommended way this is used is to specify a single encoding.
The browser then will/should put the data in that encoding for the server to
use.
It is not on the http request by default, you have to specify it in the
form.

You might do this to avoid problems where the user changes the encoding they
use to view the page.
(The data is still sent in the accept-charset encoding.)

I believe there is general agreement among i18n folks that this attribute
would have been better named as charset with only a single argument.
Hindsight...

Now the comment about URLs is also true. The path after the domain name up
to the query should be utf-8. The remaining query portion can be in any
encoding and converting it to utf-8 would have broken existing cgi and other
programs that parse the query. So it is converted only to hex encoding,
using the current encoding. However, the current encoding I believe would be
the single encoding of the form, if the accept-charset was used in the form.

Finally, we have to consider IE. For some reason, and despite their
documents saying they do send accept-charset, IE NEVER sends accept-charset.
So if you test with IE you are misled.

I wrote a little test php program that has 2 identical forms. You enter text
in either form and it posts and displays the hex codes for the bytes. The
first form does not set accept-charset, so it defaults to utf-8.
The second form overrides the page encoding and sets accept-charset to
windows-1252.
http://www.xencraft.com/php/testforms.php

If you use IE, since it never honors accept-charset, both forms behave the
same and displays utf-8 byte values. On Netscape, the second form converts
the characters to windows-1252, so the characters look munged, and the
codepoints show windows1252 values.

In summary, if the accept-charset is supplied on a request, and it contains
a single encoding, we should use it as the encoding of the form.

On IE you will never see it. But other browsers that are standards compliant
(netscape was very strong in this area) will.

I only tried POST. Somebody else might try it with GET.


Tex Texin
Internationalization Architect,   Yahoo! Inc.

Makoto Tozawa

21 years ago

Tex Texin wrote:

>I wrote a little test php program that has 2 identical forms. You enter text
>in either form and it posts and displays the hex codes for the bytes. The
>first form does not set accept-charset, so it defaults to utf-8.
>The second form overrides the page encoding and sets accept-charset to
>windows-1252.
>http://www.xencraft.com/php/testforms.php
>
>If you use IE, since it never honors accept-charset, both forms behave the
>same and displays utf-8 byte values. On Netscape, the second form converts
>the characters to windows-1252, so the characters look munged, and the
>codepoints show windows1252 values.
>
>In summary, if the accept-charset is supplied on a request, and it contains
>a single encoding, we should use it as the encoding of the form.
>
>On IE you will never see it. But other browsers that are standards compliant
>(netscape was very strong in this area) will.
>
>

Do you mean, when the form element has accept-charset=xxx attribute,
Netscape does:
(1) put the input characters in the request content using xxx encoding , and
(2) set xxx in the ACCEPT-CHARSET request header?

(1) seems to comply the HTML spec. But the HTML spec doesn't mandate
browsers to do (2).
Would you double check if Netscape really send ACCEPT-CHARSET=xxx on the
request?
(I don't have Netscape on my pc...)
FireFox sends the same ACCEPT-CHARSET header
("windows-1252,utf-8;q=0.7,*;q=0.7")
on the request regardless the existence of the form accept-charset
attribute.

Since browsers behave differently on the form accept-charset attribute,
I would
prefer PHP does nothing and let developers deal with it in their
applications.
Even if PHP could decode the input data correctly for Netscape,
developers have
to write a piece of code to re-decode the raw input to support other
browsers.

Makoto

Tex Texin

21 years ago

comments at the end of this mail
tex

> -----Original Message-----
> From: Makoto Tozawa [mailto:makoto.tozawa@oracle.com]
> Sent: Thursday, August 25, 2005 4:47 PM
> To: Tex Texin
> Cc: 'Andrei Zmievski'; christopher.jones@oracle.com; 'PHP
> Developers Mailing List'
> Subject: Re: [PHP-DEV] Re: PHP Unicode support design document
>
>
> Tex Texin wrote:
>
> >I wrote a little test php program that has 2 identical
> forms. You enter
> >text in either form and it posts and displays the hex codes for the
> >bytes. The first form does not set accept-charset, so it defaults to
> >utf-8. The second form overrides the page encoding and sets
> >accept-charset to windows-1252.
> >http://www.xencraft.com/php/testforms.php
> >
> >If you use IE, since it never honors accept-charset, both
> forms behave
> >the same and displays utf-8 byte values. On Netscape, the
> second form
> >converts the characters to windows-1252, so the characters
> look munged,
> >and the codepoints show windows1252 values.
> >
> >In summary, if the accept-charset is supplied on a request, and it
> >contains a single encoding, we should use it as the encoding of the
> >form.
> >
> >On IE you will never see it. But other browsers that are standards
> >compliant (netscape was very strong in this area) will.
> >
> >
> Do you mean, when the form element has accept-charset=xxx attribute,
> Netscape does:
> (1) put the input characters in the request content using xxx
> encoding , and
> (2) set xxx in the ACCEPT-CHARSET request header?
>
> (1) seems to comply the HTML spec. But the HTML spec doesn't mandate
> browsers to do (2).
> Would you double check if Netscape really send
> ACCEPT-CHARSET=xxx on the
> request?
> (I don't have Netscape on my pc...)
> FireFox sends the same ACCEPT-CHARSET header
> ("windows-1252,utf-8;q=0.7,*;q=0.7")
> on the request regardless the existence of the form accept-charset
> attribute.
>
> Since browsers behave differently on the form accept-charset
> attribute,
> I would
> prefer PHP does nothing and let developers deal with it in their
> applications.
> Even if PHP could decode the input data correctly for Netscape,
> developers have
> to write a piece of code to re-decode the raw input to support other
> browsers.
>
> Makoto

OK, I added to the bottom of http://www.xencraft.com/php/testforms.php, a display of the header info.
You are quite right. Although NS changes the form content to match the encoding of the FORM's accept-charset attribute, it doesn't change the value of accept-charset that it returns to the server.

I guess the way to think about it, is the FORM's accept-charset tells the user agent the encodings the server accepts. In the POST to the server, the user agent is telling the server the encodings it accepts, and it is not resending the forms' accept-charset attribute.

That makes some sense, especially since the other attributes of the form are not sent to the server, only the variables.

People are welcome to try other browsers and report back the accept-charset value.
But I agree now, that we can't use the accept-charset value to trigger the encoding, since it doesn't necessarily represent the encoding of the data.
tex

James Aylett

21 years ago

Tex Texin wrote:

> I guess the way to think about it, is the FORM's accept-charset tells
> the user agent the encodings the server accepts. In the POST to the
> server, the user agent is telling the server the encodings it
> accepts, and it is not resending the forms' accept-charset attribute.
>
> People are welcome to try other browsers and report back the
> accept-charset value. But I agree now, that we can't use the
> accept-charset value to trigger the encoding, since it doesn't
> necessarily represent the encoding of the data.

That's exactly what I'd expect. With POST and
enctype='multipart/form-data', there's the option to mark each
individual field with a different charset (assuming it's a text field).
See 4.5 in RFC 2388. This means it gets the text/plain default in the
absence of an explicit charset MIME parameter (which is also going to be
the default if the Content-Type defaults to text/plain, which I'm pretty
sure it does with multipart/form-data). I don't know if browsers
actually implement this though, and don't have time to knock up a test
case here.

In the absence of this, I'm not convinced there's actually a completely
reliable way of working. ASCII is the only thing guaranteed to work
according to the specs (at least, that's my reading), but some browsers
will do strange things. If you really want to transport general
character data from HTML FORMs, use multipart/form-data.

James

--
James Aylett
Chief Technical Architect
t 020 7535 9850 f 020 7535 9900
w http://tangozebra.com/

Tangozebra may not be held responsible for the content of this email
as it may reflect the personal view of the sender and not that of the
company. Should you receive this email in error, please notify the
sender immediately and do not disclose copy or distribute it.

Tangozebra runs anti-virus software on all servers and workstations,
and cannot be held responsible for any infected files that you may
receive. Tangozebra advises all recipients to virus scan all emails,
and to eat five portions of fruit and vegetable daily.

________________________________________________________________________
This e-mail has been scanned for all viruses by Star. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:
http://www.star.net.uk
________________________________________________________________________

Adam Maccabee Trachtenberg

21 years ago

On Wed, 24 Aug 2005, Andrei Zmievski wrote:

> I took a closer look at this today and RFC 2616 does not specify
> whether user agents are supposed to send a charset parameter in the
> Content-Type header of the POST request. I did not see any of my
> browsers doing so. I think we can safely disregard this and rely on
> http_input_encoding and output_encoding settings. We are not going to
> use Accept-Charset for the reasons you mention.

I don't know if this is useful, but Sam Ruby did a bunch of digging
into HTTP/HTML/XML encodings and precedence rules. See this
presentation -- Slides 72 - 75. (Note link below starts at slide 72.)

http://intertwingly.net/slides/2005/etcon/72.html

If possible, I would prefer not to assume that we only need to follow
the behavior of popular user agents.

Someone could be using PHP as a web service server and have people
write client scripts submitting all kinds of POST data that needs to
be processed correctly. In an ideal world (heh), PHP would handle all
of those scripts as long as they followed the specifications.

-adam

--
adam@trachtenberg.com | http://www.trachtenberg.com
author of o'reilly's "upgrading to php 5" and "php cookbook"
avoid the holiday rush, buy your copies today!

Tex Texin

21 years ago

see the section on negotiation in this as well:
http://www.xencraft.com/training/webstandards.html


Tex Texin
Internationalization Architect,   Yahoo! Inc.