Everything

php

upgrading the zlib extension to unicode

php.internals

Nuno Lopes

20 years ago

Hello,

So Andrei asked me to upgrade the zlib extension, but I have a few questions
I would like to discuss with you:
* when receiving an unicode string, what should we do? compress with as-is,
prepend a BOM header (and skip it while uncompressing)?  (now I'm unsure if
PHP/ICU uses utf16 in the machine endianess or not)
* when uncompressing, check for a BOM header and return a unicode string if
it is present? return always a binary string?

I also have another question, but unrelated with the zlib extension, that is
what is a binary string in PHP 6? I think there were some changes on that
part (and there isn't the IS_BINARY by now) and I don't really now the
difference between a binary string and the old string (aka runtime_encode'd
string)


I hope my questions are clear and make sense,
Nuno

Michael Wallner

20 years ago

Nuno Lopes wrote:

> Hello,
>
> So Andrei asked me to upgrade the zlib extension, but I have a few
> questions I would like to discuss with you:

I'd like to collaborate on this.  Besides reimplementing the output
handler to use the new API, I planned to upgrade it to something
similar like http_encoding_api.

> * when receiving an unicode string, what should we do? compress with
>  as-is, prepend a BOM header (and skip it while uncompressing)?  (now
> I'm unsure if PHP/ICU uses utf16 in the machine endianess or not)

I think it should require a binary string.

> * when uncompressing, check for a BOM header and return a unicode
> string if it is present? return always a binary string?

That would make it inconsistent if decoding data from a source other
than PHP, thus I'd say--as before--a binary string.

> I also have another question, but unrelated with the zlib extension,
>  that is what is a binary string in PHP 6? I think there were some
> changes on that part (and there isn't the IS_BINARY by now) and I
> don't really now the difference between a binary string and the old
> string (aka runtime_encode'd string)

IS_STRING is practically a binary string, AFAICT.

Regards,

--
Michael

Nuno Lopes

20 years ago

> Nuno Lopes wrote:
>> Hello,
>>
>> So Andrei asked me to upgrade the zlib extension, but I have a few
>> questions I would like to discuss with you:
>
> I'd like to collaborate on this.  Besides reimplementing the output
> handler to use the new API, I planned to upgrade it to something similar
> like http_encoding_api.

OK, I think I've completed the upgrade (it now requires a binary string),
but I left the ob_gzhandler() function for you ;)

Nuno

Andrei Zmievski

20 years ago

So you decide to make the user pass in a binary string explicitly. I
suppose that's an approach since it makes them think about what
format the binary string should be in before it's compressed.

-Andrei


On Jul 27, 2006, at 9:41 AM, Nuno Lopes wrote:

Nuno Lopes

20 years ago

Yep. After thoughting about it, I think it is the best way. If the user
wants to save the unicode directly (and save BOM, etc), he has functions to
handle that.
So, I've just changed the 's' parameters to 'S' (binary only). I left the
ob_start handler for Mike since he said he wanted to help with that. The
filter code was already upgraded (probably by Sara).

Nuno


----- Original Message -----

Michael Wallner

20 years ago

Michael Wallner wrote:

> I planned to upgrade it to something similar
> like http_encoding_api.

So, are there any objections on that?

Regards,

--
Michael

Andrei Zmievski

20 years ago

Sorry, refresh my memory, what are we talking about here?

-A

On Aug 9, 2006, at 3:38 AM, Michael Wallner wrote:

Michael Wallner

20 years ago

Andrei Zmievski wrote:

> Sorry, refresh my memory, what are we talking about here?

I was planning to port http://cvs.php.net/pecl/http/http_encoding_api.c?view=markup
to ext/zlib.  It would provide a concise zlib API for PHP and uses a more
performant iterative inflate approach instead of the current retry approach.
Additionally, if we required a minimum zlib version of 1.2, we'd benefit
of zlib internal GZIP capabilities.

Regards,

--
Michael

Andrei Zmievski

20 years ago

Okay, so this doesn't have much to do with Unicode stuff. I think if
you port it, calling it something other than http_encoding would be a
good idea, to avoid confusion with other "encoding" settings.

-Andrei

On Aug 9, 2006, at 9:13 AM, Michael Wallner wrote:

Michael Wallner

20 years ago

Andrei Zmievski wrote:

> Okay, so this doesn't have much to do with Unicode stuff. I think if you
> port it, calling it something other than http_encoding would be a good
> idea, to avoid confusion with other "encoding" settings.

Yeah, of course I thought of php_zlib_*.

Thanks,

--
Michael

Ilia A.

20 years ago

I think whenever we are storing data in external source such as
compressed file, shared memory and so on, we need to treat the data
as binary.


On 25-Jul-06, at 6:03 PM, Nuno Lopes wrote:

> Hello,
>
> So Andrei asked me to upgrade the zlib extension, but I have a few
> questions I would like to discuss with you:
> * when receiving an unicode string, what should we do? compress
> with as-is, prepend a BOM header (and skip it while
> uncompressing)?  (now I'm unsure if PHP/ICU uses utf16 in the
> machine endianess or not)
> * when uncompressing, check for a BOM header and return a unicode
> string if it is present? return always a binary string?
>
> I also have another question, but unrelated with the zlib
> extension, that is what is a binary string in PHP 6? I think there
> were some changes on that part (and there isn't the IS_BINARY by
> now) and I don't really now the difference between a binary string
> and the old string (aka runtime_encode'd string)

Ilia Alshanetsky

Andrei Zmievski

20 years ago

On Jul 25, 2006, at 3:03 PM, Nuno Lopes wrote:

> Hello,
>
> So Andrei asked me to upgrade the zlib extension, but I have a few
> questions I would like to discuss with you:
> * when receiving an unicode string, what should we do? compress
> with as-is, prepend a BOM header (and skip it while
> uncompressing)?  (now I'm unsure if PHP/ICU uses utf16 in the
> machine endianess or not)

It does use UTF-16 in machine specific endian format. I think you
have several approaches you can take when compressing a Unicode string:

1. Compress as-is
2. Convert the string to big endian, for example, and compress
3. Convert to UTF-8 and then compress

The problem with #2 and #3 is decompression. You need to know that it
was a Unicode string and do appropriate conversion after decompressing.

> * when uncompressing, check for a BOM header and return a unicode
> string if it is present? return always a binary string?

BOM header is not present in internal UTF-16 strings. It is only
present if you convert them to UTF-16BE or UTF-16LE.

-Andrei