Let's Build a Tool We All Can Use -- Together

  • We have updated our Community Code of Conduct. Please read through the new rules for the forum that are an integral part of Paradox Interactive’s User Agreement.

comagoosie

Perennial Dreamer
53 Badges
Apr 14, 2007
8.765
50
  • Hearts of Iron IV: Death or Dishonor
  • Europa Universalis IV: Common Sense
  • Cities: Skylines - After Dark
  • Europa Universalis IV: Cossacks
  • Europa Universalis IV: Mare Nostrum
  • Stellaris
  • Hearts of Iron IV: Cadet
  • Europa Universalis IV: Rights of Man
  • Hearts of Iron IV: Together for Victory
  • Cities: Skylines - Mass Transit
  • Europa Universalis IV: Mandate of Heaven
  • Europa Universalis IV: Third Rome
  • Surviving Mars
  • Rise of Prussia
  • Cities: Skylines - Green Cities
  • Europa Universalis IV: Cradle of Civilization
  • Hearts of Iron IV: Expansion Pass
  • Europa Universalis IV: Rule Britannia
  • Cities: Skylines - Parklife
  • Europa Universalis IV: Dharma
  • Cities: Skylines Industries
  • Europa Universalis IV: Golden Century
  • Imperator: Rome
  • Cities: Skylines - Campus
  • Imperator: Rome - Magna Graecia
  • Europa Universalis 4: Emperor
  • Europa Universalis III Complete
  • Crusader Kings II
  • Europa Universalis III
  • Divine Wind
  • Europa Universalis IV
  • Europa Universalis IV: Art of War
  • Europa Universalis IV: Conquest of Paradise
  • Europa Universalis IV: Wealth of Nations
  • For the Motherland
  • Hearts of Iron III
  • Hearts of Iron III: Their Finest Hour
  • Heir to the Throne
  • Europa Universalis III Complete
  • Hearts of Iron Anthology
  • Europa Universalis IV: Res Publica
  • Europa Universalis: Rome
  • Semper Fi
  • Victoria 2
  • Rome: Vae Victis
  • 200k Club
  • 500k Club
  • Cities: Skylines
  • Europa Universalis IV: El Dorado
  • Europa Universalis IV: Pre-order
Paradox Parsing Engine | EU4 Savegame Editor


Don't want to read through this thread to know how you can contribute? See the current issues with the parser and see the current issues with the savegame editor






Hi! I'm Nick, for those who don't know me.

It's about a month before the release, and I've been thinking that this community is missing something -- a community project!

Mission Statement:
To construct an open source tool built by the community for the community

But wait, Nick, that's exceedingly vague!

Correct, that is why I'm suggesting that the first order of business is to create a library capable of parsing and saving a savegame. Since all information the game needs is essentially in the save, being able to parse and write out a savegame is huge first step to larger goals. I know this community is filled with knowledge people and those willing to learn. No task is too outlandish.

So...what do you have so far?

I have a parser that is currently open source on Github (that's what "Fork me on Github" links to). It's pretty battle tested and should provide a good start, if we decide to go down that route. Remember, since this is a community project, I'll suggest paths to take, but it is up for everyone else to disagree and make me come to my senses

As far as the EU4 Savegame Library goes, I have a basic design coded, nowhere near complete because I don't a save game to go off of. So if you're Paradox, and you willing to let us see a savegame so we can get a head start that would be awesome! If not, that's cool too. Just expect us to be busy coding on the day of the release.

Sounds good, I'm fired up. How can I contribute?

Post in this thread or PM me! Let me know you're interested, and that you think this is a good idea. I especially need a group of passionate people. Post a comment with your opinions on how this project should be ran. For instance, I'm not sure how communication will be handled. Right now this thread is fine, but once the project takes off, it might be prudent to switch to another mechanism.

Don't think I should be leading, and that you should be in my position? Post! This is a community project, you decide.

In the near future, the code will be released on Github for all of us to hack on. If you have never coded before, don't worry, use this as a good learning experience. I want us to grow together (too cheesy, ok, I'll dial it down). In all seriousness, everyone is needed to contribute. I don't know all the nooks and crannies, and I'm sure you don't either, but I'm sure that our collective knowledge will prevail.

Tell me the technical details

Note that what I say is subject to change based on community feedback. I will tell you what I have and what my plan is.

C# 4.0 is my language of choice. I find it an extremely productive and easy to learn language that works with no problem with Windows XP. Using Mono, C# can even be deployed on Linux and Mac, so as far as I am concerned, C# is cross-platform. Granted I have never used Mono, so if you've used Mono, I want to talk to you! I want the entire community to benefit from what we will create.

I'll stress that the initial project is a library, and not a fully functional client product. The GUIs can come later.

One last question, why?

Some my recognize me as developing tools for EU3, but the biggest obstacle is that it is too hard for a single person to know everything about the game and then code it! So one of the reasons for this project is essentially a document on how the game works. Another reason is that the work we do here will most likely be applicable to other Paradox titles (read: backport to EU3); so other people with a similar passion can start those projects using this as a base. It is in my opinion that this community has the potential for something absolutely great, and I hope this will be the catalyst for it.

Thanks for listening, what are your thoughts? How should we run this? :)
Nick
 
Last edited:
I'm in.

while(Paradox.LastName == "Awesome")
{
CreateThatCommunityProject();
}
 
Haha, awesome!

Anything that you are particularly interested in, or an opinion you want to voice?
I'll get the source today to have a look to see what is done, then raise my voice. :p
But I believe that the material we will be working with in EU4 will not be different from EUIII/CK2. So I think once we have a working library for these two we would be off with a very good start.
I could be wrong, PI giving us that example savegame would be awesome.

Also, my 'real life' job is with software test automation (C# + WebDriver).
 
I'll see if I can help out a little bit later. Currently doing a 2 year course in Computing at college. Did my assignments in Java but I can't imagine C# wouldn't be too hard to get the basics of over the 6 weeks off I've got this summer.
 
I'll get the source today to have a look to see what is done, then raise my voice. :p
But I believe that the material we will be working with in EU4 will not be different from EUIII/CK2. So I think once we have a working library for these two we would be off with a very good start.
I could be wrong, PI giving us that example savegame would be awesome.

Also, my 'real life' job is with software test automation (C# + WebDriver).
Currently the source code for the parser is the only code that is available to look at. I can most certainly release what I have so far for EU4, but again since we don't have a savegame to go off of, there isn't much there :)
I'll see if I can help out a little bit later. Currently doing a 2 year course in Computing at college. Did my assignments in Java but I can't imagine C# wouldn't be too hard to get the basics of over the 6 weeks off I've got this summer.
Great! :cool:

Yeah Java and C# are extremely similar and you could learn the differences within a week. Great to have you on board.
 
this sounds like it would be both fun and something to learn from. unfortunately, i'm still in a pretty early learning phase of my first language, Java, (starting a college course in it this term) so I'm not sure how much use i'd be in the beginning... still, i've been looking for a programming project that doesn't have "5 years of working experience" in the reqs so I'm in if I'm let in:D
 
(post-and-run to remind myself to check back later)
 
So I had a look and it looks solid. Some things I didn't understand very well, but hey, I'm not that experienced.

What is your plan now? We could create maybe a road map to set a few goals.
 
this sounds like it would be both fun and something to learn from. unfortunately, i'm still in a pretty early learning phase of my first language, Java, (starting a college course in it this term) so I'm not sure how much use i'd be in the beginning... still, i've been looking for a programming project that doesn't have "5 years of working experience" in the reqs so I'm in if I'm let in:D

In all honesty, you're perfect for this project because I feel like (at least the beginning) a lot of the code will be mapping fields, e.g. if you see "reputation" in the country section of a savegame that maps to the badboy, etc. It would be a great way to contribute and to learn the language!

In short, the only requirement is enthusiasm :cool:
(post-and-run to remind myself to check back later)
I'll be watching :ninja:

So I had a look and it looks solid. Some things I didn't understand very well, but hey, I'm not that experienced.

What is your plan now? We could create maybe a road map to set a few goals.
If you don't understand the code, feel free to comment on what you don't understand, or create an issue on Github. If you don't understand something, let me know. I'm sure there are many more that also are confused!

As far as the roadmap, since the actual game isn't released, all code will essentially be in speculation. It is the plan though that by the time the game does come out that there is a well laid out plan as how to precede in implementation. I figured a month before the release would be a good time to announce my intentions because it would give the community a heads up. I would be all for coding if we had a savegame...but alas, might not get our hands on one until the release date. Maybe that is a good thing and will give us more time to plan.
 
In all honesty, you're perfect for this project because I feel like (at least the beginning) a lot of the code will be mapping fields, e.g. if you see "reputation" in the country section of a savegame that maps to the badboy, etc. It would be a great way to contribute and to learn the language!

I wonder how much task-specific you want to make your tool?

I happened to write such a library in C# some two years ago, which supports parsing Europa Engine savegames and Clausewitz Engine files too; I have no reason to doubt that it will handle EU4 saves as well. And I'd really recommend you to take as much generic approach as possible before creating some wrapper functions/classes to cover specific areas, like checking reputation (which in some games exists, in some not).

Let's have this portion of Darkest Hour (or HoI2) savegame and my approach:

Code:
country = { 
	tag = ENG 
	(...)
	dissent = 4 
}

You want to check dissent for every country in the game.

Find an attribute that says on the left side "dissent". The right-hand-side value is the amount of dissent you have. Now look one logical level up (that would be "country" container), from this point get a subnode called "tag" and read its right-hand-side value. Rinse and repeat for every other occurence.

In my library, it's handled (roughly) by the following code:

Code:
Dictionary<string,double> dict = new Dictionary<string, double>();

EE.File file = new EE.File(filename);
EE.Container data = file.Process();
EE.Entity[] nodes = data.Subnodes.List();

foreach (EE.Entity node in nodes)
{
   if (node.IsAttribute() && node.Name=="dissent" && node.HasAParent("country",1))
   {
       string tag = node.Parent.Subnodes["tag"].Value;
       double dissent = double.Parse(node.Value, CultureInfo.InvariantCulture);
       dict.Add(tag, dissent);
   }
}

Now if it is dissent, reputation or anything else, you just change the condition or even use some shortcuts, like with Find, FindAll and lambda expressions. And wrap a GetCountryDissent(string tag) around it at your leisure ;)

That said, I looked at your code and it looks much clearer and more professional than mine so I have really high hopes for this project. I'll write you more later :)
 
I can effectively break chicken necks with minimal effort.

Which position may I assume?
The muscle :cool:
I wonder how much task-specific you want to make your tool?

I happened to write such a library in C# some two years ago, which supports parsing Europa Engine savegames and Clausewitz Engine files too; I have no reason to doubt that it will handle EU4 saves as well. And I'd really recommend you to take as much generic approach as possible before creating some wrapper functions/classes to cover specific areas, like checking reputation (which in some games exists, in some not).

Let's have this portion of Darkest Hour (or HoI2) savegame and my approach:

Code:
country = { 
	tag = ENG 
	(...)
	dissent = 4 
}

You want to check dissent for every country in the game.

Find an attribute that says on the left side "dissent". The right-hand-side value is the amount of dissent you have. Now look one logical level up (that would be "country" container), from this point get a subnode called "tag" and read its right-hand-side value. Rinse and repeat for every other occurence.

In my library, it's handled (roughly) by the following code:

Code:
Dictionary<string,double> dict = new Dictionary<string, double>();

EE.File file = new EE.File(filename);
EE.Container data = file.Process();
EE.Entity[] nodes = data.Subnodes.List();

foreach (EE.Entity node in nodes)
{
   if (node.IsAttribute() && node.Name=="dissent" && node.HasAParent("country",1))
   {
       string tag = node.Parent.Subnodes["tag"].Value;
       double dissent = double.Parse(node.Value, CultureInfo.InvariantCulture);
       dict.Add(tag, dissent);
   }
}

Now if it is dissent, reputation or anything else, you just change the condition or even use some shortcuts, like with Find, FindAll and lambda expressions. And wrap a GetCountryDissent(string tag) around it at your leisure ;)

That said, I looked at your code and it looks much clearer and more professional than mine so I have really high hopes for this project. I'll write you more later :)
Hey, thanks so much for dropping by! Glad you liked my code.

Your code definitely is interesting. I never would have thought of processing files like that, though I see the benefit and the flexibility. It reminds me of XPath. I know you posted only a snippet but if I could offer advice, it would be to try to find a way around parsing the entire file at once. Depending on how you structure your code, there's a chance that entire file, all the parsed nodes, and the extracted values are in memory, which I feel could be a problem for large files (correct me if I'm wrong). Also, how is the performance? I know memory and speed may not be priorities but since savegames can be ~50MB, I like to push my code such that there is no waiting involved for the user.

If you're worried about how generic the project will be, join development! I can't say it enough, those who are passionate about this project can help me and others code and design.

Look forward to talking to you more :)
 
True, my way definitely involves reading a file at once and keeping it whole in memory at some point. Mind that I designed my library for Darkest Hour / Arsenal of Democracy which keep their savegame around 20 MB mark and I used node skipping. That is, if I encountered huge "ai = { }" containers, of no use to me in my particular scenario, I could either read their contents as flat text (saving on processing time) or throw them out altogether (if I wasn't keen on rewriting the file).

It depends on how memory-conservative you want to go. My approach reads a whole file into memory, parses it creating data structures (which contain about the same data as pure text) and discards the original text from memory when finished. So for 50 MB it easily peaks at 100 MB but doesn't go much worse (it's really ages since I looked at my library performancewise and even then I was more concerned about CPU time). Some containers ("container = { }") span the whole file and it may be hard to do meaningful file crunching line by line.

My most stellar use of the library was with this app. It tends to use crazy heaps of memory but that's because of no smart management of parsing results (sometimes resulting from parsing 100+ files), not the procedure itself.

As far as processing time goes, this was my bigger concern because creating all those objects that make up a tidy logical tree indeed creates some overhead. I remember originally achieving a 10 MB = 1 s metric but it's not certainly an exact measurement as it depends on file structure (but won't go much worse). The nice thing I discovered pretty recently is that it can make a great use of Parallel.ForEach magic but there are probably other performance considerations to make. Like ditching one node = one object orthodoxy and creating objects only for container nodes. So my approach involves some waiting time you want to avoid, once you parse the whole beast though, it's really instantenous to work on the resulting structure.

To clarify my algorithm:
1. The whole file is read into memory and market as a "root" container.
2. It's parsed then character by character (which sounds time-consuming but is blazing fast comparing to string searches or regular expressions).
3. All logical elements of a file are turned into objects (with comments and extra whitespace, if present) and added to the current branch of the tree.
4. When I encounter a container, I record an opening and closing brace and then recursively call the parsing functions.
5. The original text is discarded from memory. Now I can operate on the nodes in a children/parent fashion or get a flat list of objects too.

This is surely not the best possible approach but it is battle-hardened enough to say that it's feasible to use and enables some powerful interpreting. I wonder what usage scenarios do you see for your library? I can only recommend to make your library as flexible as possible - there's not that much that you can do with EU4 saves but when you consider a library that covers all the games and all the game files, it may be a holy grail indeed :)

Realistically speaking, I can't help you much with coding and generally realization of the project because I'm already really busy with my projects (recently going through parsing Clausewitz map files - and this burns memory like crazy :D ). I'm OK with sharing my code if you want some inspiration though. :) I'll PM you about it!
 
I can only recommend to make your library as flexible as possible - there's not that much that you can do with EU4 saves but when you consider a library that covers all the games and all the game files, it may be a holy grail indeed :)

First line of the project readme says:


Pdoxcl2Sharp is a general parser for files related to Paradox Interactive. While the parser is aimed towards Paradox Interactive, it is not exclusive, meaning that any file or configuration written in a similar style can be parsed without problems.


I have used it for save games, game editors, and I've actually convinced a previous employer to use the parser, so it is safe to say it is generic enough ;)

I love looking at source code, so if you want, I can always take a look at your project. As far as the algorithm goes, I believe mine is "better" (I say better but in reality both have their pros and cons). My algorithm has a max memory usage of 64KB if a stream is parsed but nothing is done with the parsed information. Not to mention speed. My parser is almost entirely IO bound. I would describe the algorithm in more detail but the source code is in front of everyone, so you can take a look.
 
I have absolutely no coding experience, so i can't help you there. I might be able to be a tester though. ;)

But i got a question : will it have the ability to work on starting situations (like editing history files) ?
 
I might seriously lean what I can to contribute to this project just to get Gigau up and running for the next MEIOU, or whatever other map/history mods he can come up with, faster.

On a more serious note I have several friends who work at Catalyst IT solutions. Not only does the company use GIT but the group I know has nearly as perverse a love as I have for strategy games. Surely I can convince at least one of them to lose all of their remaining free time to playing EU and working on community tools!
 
I have no experience in coding (really, not at all), in fact, I think I don't even have any experience in anything, but I think this initiative is admirable and I fully support you, guys ! If you need any help in a thing that Mr John Doe would be able to do, tell me, I'll watch this topic regularly to see how it evolves !

Good luck guys !