Noob question - accents, diacritics, letters with weird signs

[BRS Dude]

Hi all,

Apologies if this has been discussed already, I'm new to EU4 and am so far absolutely LOVING the game (and learning more than I did for some of my exams to be honest), but the one pet peeve that I have is the lack of accented letters on most of the map. I'm talking about letters such as ö, á, ä, š, ž and many other exotic varieties which I probably couldn't even pronounce (I'm looking at you, Vietnamese alphabet).

There are a few exceptions that I've found (e.g. Münster), but by and large they are missing from most nations and province names. As someone from Serbia, where we use such letters quite a lot and they are an important part of our language / culture, you can't not notice when Raška is spelled as Raska, and Braničevo as Branicevo.

This is such a small point considering the overall AMAZING quality of the game and the attention which the developers clearly paid to the history, geography, and culture of nations around the world. Seriously, stuff like dynamic province names are something that shows such jaw-dropping attention to detail that NOT having accented letters really doesn't make sense to me in comparison.

I guess you could take this logic further, and, for instance, make an option for dynamic script usage, so that when a nation whose primary written script isn't the Latin alphabet (e.g. Cyrillic, Arabic, Hindi, Chinese characters) conquers a territory, its name turns into that script. Or it stays as a 2nd word underneath the Latin name, just like Google Maps does. But tbh I'd just be happy with accents on letters to start with!

Thanks for any input!

 

[gigau]

There are two diffferent point to consider:
- overall, the path Paradox chose what to use the English version of province names
- the limitation in number of character in the fonts they used (affecting for example the slavic consonnants) as they can use only one font at a time (for a given place in hte UI). This lead us, in M&T, where we use province names in local dialects/languges, to use the most commonly accepted latinised version of province names.

 

[Pitt The Elder]

Another factor is the menu where you search provinces by name (which if you haven't used it can be opened by pressing 'f'). If you add accent marks to the province names, it gets exponentially harder for people unfamiliar with the use of those marks to locate the provinces.

 

[gigau]

Another factor is the menu where you search provinces by name (which if you haven't used it can be opened by pressing 'f'). If you add accent marks to the province names, it gets exponentially harder for people unfamiliar with the use of those marks to locate the provinces.

Right, i forgot about that one. Esepcially as some accents are not available on some keyboards.

 

[BRS Dude]

There are two diffferent point to consider:
- overall, the path Paradox chose what to use the English version of province names
- the limitation in number of character in the fonts they used (affecting for example the slavic consonnants) as they can use only one font at a time (for a given place in hte UI). This lead us, in M&T, where we use province names in local dialects/languges, to use the most commonly accepted latinised version of province names.

Another factor is the menu where you search provinces by name (which if you haven't used it can be opened by pressing 'f'). If you add accent marks to the province names, it gets exponentially harder for people unfamiliar with the use of those marks to locate the provinces.

Great points, but allow me to address them.

I agree that perhaps adding entirely new written scripts might be a bit too cumbersome for the game, so let's leave that as just a fun thought. But surely including accented letters is something far easier. I'm not a developer or even an IT guy, so correct me if I'm wrong, but don't most / all fonts (at least in Word) have options for accented letters? I can't imagine a font being able to show "o" but not "ő", "c" but not "ć". Also, clearly the font they are using can recognise some accents (e.g. Münster), so why not others?

The menu search issue is easily fixable. At my job we often have to look up international firms and people (many of whom have accented letters in their names), and the system will simply list all of them even if I type in a Latin letter. So for instance, if I start typing a name beginning with "s", I will get everything starting with s, š, ș, ś, etc. It could work the same way for provinces, you don't actually need to type in the accented letter to have the province pop up, which solves the problem I think.

 

[grommile]

I can't imagine a font being able to show "o" but not "ő", "c" but not "ć". Also, clearly the font they are using can recognise some accents (e.g. Münster), so why not others?

EU4 (and, if I remember rightly, CK2) has an outdated font renderer that only supports a 256-character character set, and the one chosen was Win-1252 (Western Europe and Scandinavian), which is why we have š (used by Finnish for some loanwords) but not č (not used by any of the languages covered by Win-1252).

Stellaris and HoI4 have a more modern font renderer with full Unicode support.

It could work the same way for provinces, you don't actually need to type in the accented letter to have the province pop up, which solves the problem I think.

It in fact already does

 

[BRS Dude]

EU4 (and, if I remember rightly, CK2) has an outdated font renderer that only supports a 256-character character set, and the one chosen was Win-1252 (Western Europe and Scandinavian), which is why we have š (used by Finnish for some loanwords) but not č (not used by any of the languages covered by Win-1252).

Stellaris and HoI4 have a more modern font renderer with full Unicode support.

It in fact already does

Ah I see, that's very interesting! But if the font which EU4 uses supports 256 separate characters, surely that could cover all the Latin letters with their various diacritics? There really aren't that many, I can't imagine it would even be 100 in total, let alone 256.

 

[grommile]

Ah I see, that's very interesting! But if the font which EU4 uses supports 256 separate characters, surely that could cover all the Latin letters with their various diacritics? There really aren't that many, I can't imagine it would even be 100 in total, let alone 256.

After accounting for upper + lower case forms, you need at least 186 characters just to capture all Latin letters used for living national languages in Europe and Turkey. (I probably missed some when I was doing my counting exercise.)

And you can only use 223 of those 256 character positions for printable characters, because characters 0-31 are control characters with special meanings to computers, and character 32 is space.

And your font still has to accommodate the digits 0-9 and an assortment of punctuation (not all of which is intended to be displayed, but the file has to be editable by ordinary text editors or the modders will come round with pitchforks and torches with good reason).

So it turns out that the only way to fix the problem is to replace the font renderer, and given that this is the EU4 code base we're talking about, I wouldn't be too surprised if that's harder than it should be

 

[BRS Dude]

After accounting for upper + lower case forms, you need at least 186 characters just to capture all Latin letters used for living national languages in Europe and Turkey. (I probably missed some when I was doing my counting exercise.)

And you can only use 223 of those 256 character positions for printable characters, because characters 0-31 are control characters with special meanings to computers, and character 32 is space.

And your font still has to accommodate the digits 0-9 and an assortment of punctuation (not all of which is intended to be displayed, but the file has to be editable by ordinary text editors or the modders will come round with pitchforks and torches with good reason).

So it turns out that the only way to fix the problem is to replace the font renderer, and given that this is the EU4 code base we're talking about, I wouldn't be too surprised if that's harder than it should be

You actually went and counted all the accented letters? Haha mad lad!

That's a lot more than I anticipated, and all of that just for Europe? Yeah then it definitely wouldn't be able to accommodate for all the Asian diacritics as well.

Yes you're completely right then, it seems it would require a much more fundamental change to the EU4 code than it might seem. Come to think of it, given how detail-heavy EU4 is, I should've assumed the devs didn't just "forget" about this, and that there was probably a deeper problem at hand. Oh well, while it still kinda sucks that these are missing, we can keep our fingers crossed for a future update perhaps? Maybe that could add in different written scripts as well, ala Google Maps!

Thank you (Hvala / Хвала)!

 

[gigau]

It in fact already does

Oh... interesting. Thanks for the information.

 

[jcranmer]

Ah I see, that's very interesting! But if the font which EU4 uses supports 256 separate characters, surely that could cover all the Latin letters with their various diacritics? There really aren't that many, I can't imagine it would even be 100 in total, let alone 256.

The exact code map EU4 uses is likely Windows-1252 (see https://en.wikipedia.org/wiki/Windows-1252). In theory, you could remap those characters differently, but that requires creating your own custom charset, which is a really bad idea because now you can't edit files anymore. If you look at the Latin Extended-A block in https://en.wikibooks.org/wiki/Unicode/Character_reference/0000-0FFF (it's the third one on the page), that gives you a better idea of what a more complete list of Latin letters with various diacritics. Spoiler: that's another 128 characters, and there's not enough holes in Windows-1252 to fill in (even if you rip out symbols, punctuation, and control characters).

A couple of years, I did find a bug where someone at Paradox wrote "Sūsa" for a province name. The "ū" character not being found in Windows-1252, the file was saved as UTF-8 instead, which when reinterpreted as Windows-1252, caused the province name to be rendered as "SÅ«sa". Speaking of which, looks like there's still quite a few accidentally UTF-8 files:

~/.steam/steam/SteamApps/common/Europa Universalis IV/common $ find -type f | xargs file | grep UTF-8
./estates/03_burghers.txt:                                               UTF-8 Unicode text, with CRLF line terminators
./government_names/00_government_names.txt:                              UTF-8 Unicode text, with CRLF line terminators
./mercenary_companies/00_mercenaries.txt:                                UTF-8 Unicode text, with CRLF line terminators
./custom_country_colors/00_custom_country_colors.txt:                    UTF-8 Unicode text, with CRLF line terminators
./event_modifiers/00_event_modifiers.txt:                                UTF-8 Unicode text, with CRLF line terminators
./countries/Benin.txt:                                                   UTF-8 Unicode text, with CRLF line terminators
./countries/Sokoto.txt:                                                  UTF-8 Unicode text, with CRLF line terminators
./countries/Guge.txt:                                                    UTF-8 Unicode text, with CRLF line terminators
./countries/Susquehannock.txt:                                           UTF-8 Unicode text, with CRLF line terminators
./countries/Canada.txt:                                                  UTF-8 Unicode text, with CRLF line terminators
./countries/Illinois.txt:                                                UTF-8 Unicode text, with CRLF line terminators
./countries/Mann.txt:                                                    UTF-8 Unicode text, with CRLF line terminators
./countries/TheIsles.txt:                                                UTF-8 Unicode text, with CRLF line terminators
./countries/Cascadia.txt:                                                UTF-8 Unicode text, with CRLF line terminators
./on_actions/00_on_actions.txt:                                          UTF-8 Unicode text, with CRLF line terminators

Looks like most of the countries files the non-ASCII characters are actually in display strings (e.g., monarch names). Seems like Paradox should look at adding a build step that errors if these files are UTF-8!

 

[DoChuan13]

Korean - non-latin character, but they also successfully translated. Meanwhile, I still cannot translate my language - Vietnamese.
I think that EU4 need upgrade to new character encoding

 

[Kapi96]

I guess you could take this logic further, and, for instance, make an option for dynamic script usage, so that when a nation whose primary written script isn't the Latin alphabet (e.g. Cyrillic, Arabic, Hindi, Chinese characters) conquers a territory, its name turns into that script. Or it stays as a 2nd word underneath the Latin name, just like Google Maps does. But tbh I'd just be happy with accents on letters to start with!

That would be absolutely terrible.
So people who can't read those other alphabets simply can't read province names anymore? Ridiculous. If you're playing the game in English then province names should be in English, they shouldn't switch to Chinese or something just because Ming conquered a province.

 

[grotaclas]

Korean - non-latin character, but they also successfully translated. Meanwhile, I still cannot translate my language - Vietnamese.

Did you do the screenshot yourself? The Korean language mod that I found on the workshop needs you to patch the game executable(or at least some of the DLLs). You can probably make Vietnamese characters work in the same way.

 

[DoChuan13]

Did you do the screenshot yourself? The Korean language mod that I found on the workshop needs you to patch the game executable(or at least some of the DLLs). You can probably make Vietnamese characters work in the same way.

I just subscribed mod. No more
Korean Language Mod - stability 1.30

 

[grotaclas]

I just subscribed mod. No more
Korean Language Mod - stability 1.30

I had a look at the mod Korean Language Mod - stability 1.30. It seems to use a font that replaces some of the non-basic characters that the game can display(e.g. "Ç", "©", "Â") with korean letters. You could do the same for any language that doesn't use too many different letters(I don't know how many. Maybe something like 100 including uppercase letters). But you would have to find such a font or create it yourself.

 

[Waverider900]

I guess you could take this logic further, and, for instance, make an option for dynamic script usage, so that when a nation whose primary written script isn't the Latin alphabet (e.g. Cyrillic, Arabic, Hindi, Chinese characters) conquers a territory, its name turns into that script. Or it stays as a 2nd word underneath the Latin name, just like Google Maps does.

I understand from proceeding comments that it's not that easy, but it would be a really fun intellectual exercise if you could tick a box in the options to show correct-language province names. EU4 is largely what got me interested in the history of political geography and it would be a great way to learn place names.

EG, even for something like a language using Chinese characters where I don't know like 99% of the chars, being able to see "hey, the 3rd character in the name of X province is the same as the 2nd in Y province" would be cool