This patch makes Binaryen's APIs and binaries work on Windows with paths that contain Unicode characters outside of the ASCII range.
I expect this patch to require changes, for style at least.
The basic problem is that Windows requires processing paths as UTF-16-encoded wide chars, and obtaining these from the command line involves declaring a non-standard wmain function that accepts wide chars.
The strategy this patch uses is:
- Use a macro to declare main
- Options::parse re-encodes its arguments as UTF-8 on windows
- Change APIs to take (a wrapper of ) std::filesystem::path, which uses the correct encoding on windows
- Perform a UTF-8 to UTF-16 conversion on windows when calling the binaryen filesystem APIs
On non-Windows this patch should introduce no functional changes except for some additional string copies.
Some key decisions here:
This patch introduces a new header, pchar.h, with definitions needed to provide compatibility with platform-specific string/path types. It would be desirable to not need these defs, but it doesn't quite seem possible to accomplish this patch with just std.
pchar.h defines typedefs for a platftorm-specific character type, pchar,
which is a wchar_t on windows, and a platform-specific string type, pstring. These are aliases for std::filesystem::path::value_type and std::filesystem::path::string_type. It also defines unicode conversions between string and pstring, on windows using windows-specific apis, on unix just doing a string copy.
pchar.h also defines its own custom path type, fspath, which is just a wrapper around std::filesystem::path. The binaryen APIs use this type instead of std::filesystem::path. The only reason this type exists is because std::filesystem::path defines an implicit conversion from std::string which does not do unicode conversion. So this type is used to avoid silently introducing bugs where paths are not encoded correctly.
The binaryen file apis rely on an implicit conversion from std::string to fspath that does unicode encoding. This is to avoid introducing new explicit conversions to the various binaryen main functions.
Note that windows paths may contain non-standard unpaired surrogate code units. I have not attempted to support such strings as they are a rare corner case. The approach in this patch could work with unpaired surrogates by changing the encoding/decoding methods. Depending on how the windows encoding APIs work, these paths may already work - I have not tested. If these paths don't work the result should be mystery i/o failures, not crashes.
This patch probably also likely has the side-effect of allowing other argument strings to be UTF-8 encoded Unicode on Windows, such as --output-source-map-url, where before they would have been incorrectly encoded in some way. I have not thought hard about the consequences.
I have added a test of unicode path processing, but don't really know what I'm doing with the test suite. I just copied the hello_world files and modified them a bit.
I see a pre-existing test failure on windows (duplicate_imports.wat), so I have not successfully run the full test suite on windows.
Fixes #4995