Note: alternative to regex splitting in byte tokenizers

In a previous note , I discussed an alternative for setting split to true in a ByteLevel pretokenizer. I suggested using a ByteLevel normalizer first, and then splitting using a complicated regex in “byte space”. However, this turned out to not work very well: there are certain character classes in an original Regex, such as \s , that are very difficult to convert to a pattern in byte space. I was…