Correct regular expression flags scanning for non-BMP characters

[graphemecluster]

Per the IsValidRegularExpressionLiteral static semantics in the ECMA262 specification, FlagText should be interpreted in code points, not code units.

Cherry picked from #58289@e67692a.

[typescript-bot]

This PR doesn't have any linked issues. Please open an issue that references this PR. From there we can discuss and prioritise.

[rbuckton]

Overall, this looks good. It looks like there are some failing tests that still need to be resolved. Also, since this isn't a bug fix, per se, I expect we will want to hold off from putting this in the 5.5 RC and will wait to merge after main is open for TypeScript 5.6 work.

[rbuckton]

Switching from "Approve" to "Request Changes" until the test issues are resolved.

[graphemecluster]

IMO it is a bug fix especially since I forgot to revert the change on line 2561 in #58289. And it’s important to align the scanner behavior to the spec.
#58615 is a bug fix, too, that is even more critical since the bug breaks real-world codebases.
#58613 is not the case, but (as an excuse) it was supposed to be included in #55600.

[rbuckton]

IMO it is a bug fix especially since I forgot to revert the change on line 2561 in #58289. And it’s important to align the scanner behavior to the spec.

What change do you mean? If it's not related to code points then I could see taking a small fix for that. Properly handling non-BMP characters in flags doesn't come up in practice and delaying this specific functionality will only result in a slightly less informative error since none of the valid flag/modifier characters will ever be equivalent to the first byte of a multi-byte code point.

#58615 is a bug fix, too, that is even more critical since the bug breaks real-world codebases. #58613 is not the case, but (as an excuse) it was supposed to be included in #55600.

#58613 can wait till 5.6, but I'll review #58615 shortly.

[graphemecluster]

What change do you mean? If it's not related to code points then I could see taking a small fix for that.

This line, which causes the String.fromCharCode below to convert incorrectly (as the method only consider the 16 least significant bits).

[rbuckton]

What change do you mean? If it's not related to code points then I could see taking a small fix for that.

This line, which causes the String.fromCharCode below to convert incorrectly (as the method only consider the 16 least significant bits).

I've just put up a minimal fix for that line in #58727.