PDO/Unicode Migration Strategies

PDO Devs, et. al.; It's that time, time to start looking at PDO's plans for the future, specifically how it'll integrate with the wild world of unicode. After working with the sqlite2 native driver and reading up on some of the other RDBMs, I've come up with a few scenarios of varying merit for PDO that I'd like to bounce against y'all and the world at large. (A) PDO downcodes all inbound unicode data (SQL statements, bound params, etc...) to UTF8, and upconverts return data (results) from UTF8 to UTF16 (UChar type) on return (when UG(unicode) is enabled). Pros: No changes to the dbh/stmt handler APIs. Cons: Changes to assumptions made by many (most?) drivers. Anywhere non-utf8 data (e.g. latin1) is expected, the data will have to be re-converted. Doesn't cleanly account for binary strings passed in which are not already utf-8 encoded which could easily lead to wtf when in non-unicode semantics mode (normal case for many/most users). Moreso when the driver is trying to decide if it can use the data it received as-is, or if it has to transcode to get to the right charset. (B) Change all string handling APIs (e.g. do/execute/fetch ) to include a type field (zend_uchar str_type, zstr str, int str_len) so that drivers get unicode as UChar*, and non-unicode as char*. Pros: Leaves character set handling to the driver which is best equiped to make decisions about its quirks. Binary (most likely localized) data is recognized as such and can be handled appropriately. Cons: Puts more work on the actual driver to handle unicode conversion. Leads to lots of #ifdef macrory since drivers live in PECL and must still be compilable on PHP5. (C) Add a UConverter *encoding_conv; element to pdo_dbh and pdo_stmt objects, and an INI setting: pdo.default_encoding. When passing data to/from a stmt object, the statement objects encoder is used if available (set during prepare), if not available the driver's converter is used (set by factory), otherwise pdo.default_encoding is used as a fallback. Data exchanges between the dbh object are similarly handled though (obviously) skipping the stmt step. Pros: Keeps character set conversion work out of the driver layer. Reduces the amount of #ifdef work for multiple version support. Recognizes that some drivers (SQLITE) use a single encoding universally, while others allow different tables to use different encodings. Cons: Doesn't solve the "do()" problem of encoding to different charsets when inserting to tables of a driver which allows different charsets per table. Doesn't provide an indicator which says "This came from a unicode string and was converter by ICU so is reliably in the correct encoding" versus "This was handed to me by the user as a binary string and may contain anything". Though this is also "fixable" by either changing the handler proto or by burying a state flag in the dbh/stmt objects. Personally I like option C the best as it presents the least amount of work for individual drivers, costs the least in terms of version/ifdefs, and provides a reasonable degree of flexibility. As mentioned however, only B provides information to the driver on the reliability of the encoding "Is this *really* utf8? Or am I going to find a stray \xA0 in here somewhere?" Of course, we currently have no such assurance, the user is simply expected to give the driver well formed data, if they don't they're SOL already. I generally don't like A as it's the most wasteful and really doesn't solve the difficult problems. Any rate, share your thoughts.. -Sara P.S. - Where is primary PDO development happening? Last I heard PECL releases were coming out of the 5.1 branch and that was the place to be. Has HEAD been kept in sync?