to determine UTF-8 string length in characters we need something much more sophisticated than strlen or C++ counterparts

Oh man... don't tell me this is still an unsolved problem in the C++ std lib... Like every lowly scripting language had this for 20 years now...😞

Oh man... don't tell me this is still an unsolved problem in the C++ std lib...

Unfortunately, it is.

• There is std::wstring, but it is not portable, on Windows it is 16-bit only. AFAIK not much standard library functions can deal with it.
• We have std::u32string, which can be used to deal with UTF-32 encoding, not really useful for us. Moreover, size() will tell us the number of code points, while we need the number of graphemes here (and a grapheme can be made, for example, from one code point telling about a base Latin letter + two other telling about combining marks; IIRC Baltic languages are particularly nasty here).
• C++20 has std::u8string to deal with UTF-8. But the standard library offers nothing to handle it. Might be useful to us once we migrate to C++20, but only for type checking.

I could implement UTF-8 compliant strlen, by asking my Unicode engine to convert UTF-8 to ASCII and then run a regular string size check - but this is too risky, I might improve my Unicode engine in the future to cooperate with emulated DOS more closely, and at this point we don't have DOSBox fully initialized - I'm not sure if I'll remember to test this particular use case like 2 or 3 years in the future.

IMHO better ask the translator to do the job.

Moreover, size() will tell us the number of code points, while we need the number of graphemes here (and a grapheme can be made, for example, from one code point telling about a base Latin letter + two other telling about combining marks; IIRC Baltic languages are particularly nasty here).

@FeralChild64 Actually, I take my statement back that "every other stdlib" can do this 😅 I'm pretty sure most other languages offer the same basic support only, i.e., they only count "code points", or "runes" (they're called runes in Nim, for instance). But not that wacky situation you described.

Also some libs just shit themselves when dealing with 4-byte UTF-8 emojis which are super popular among Asian audiences (talking from real world experience here; 4-byte UTF-8 emojis caused so many headaches for me in the past...)

Reading about it, ICU has a break iterator, but people mention how its API is very old.

Boost's locale.hpp has a boundary iterator that can be used to count graphemes.

Glib has a UTF-8 ustring, however it seems to only count code points without deducting those that are used for joining.

So Glib::ustring s = L"नमस्ते";, size() returns six, but two of the code points are for joining and we're interested in the four remaining graphemes.

ICU and Boost would be heavy dependencies, so having your engine handle this in the future would be very nice, @FeralChild64 !

Reading about it, the ICU has a break iterator, but people mention how it's API is very old.

Boost's locale.hpp has a boundary iterator that can be used to count graphemes.

Glib has a UTF-8 ustring, however it seems to only count code points without deducting those that are used for joining.

So Glib::ustring s = L"नमस्ते";, size() returns six, but two of the code points are for joining and we're interested in the four remaining graphemes.

ICU and Boost would be heavy dependencies, so having your engine handle this in the future would be very nice, @FeralChild64 !

Yeah, I don't even want to go near Boost. Like ever, unless I'm extremely well compensated 😄