Ledger/galactic history generator

[gedruchy]

My friends and I play a few games together with an AI or 4 and have been looking for an interesting way to record the interactions that occur within a game.
Given that there aren't many functional workarounds (and theres no way I'm just going to write down everything as it happens); I'm writing a script to read Stellaris save files and generate a ledger or history list of all interesting interactions for a given game.
The script will do the following in order (or this is the theory I'm working off):

1. Get a list of save files (ie open the save folder).
2. Extract and read a given auto-save file.
3. Record the date of the auto save for each list of events.
4. Iterate though the events contained within (ie, species, wars, planetary claims, deaths of leaders, etc).
5. Order these events by date and commit them to a `Era Block`, stapled with the date of the auto save that these were extracted from.
6. Compare that Era to the previous Era and note any relevant changes (AI player no longer appears, war no longer happening, changes in system ownership).
7. Commit each Era Block to a master list, ordered by date.
8. Iterate the master list and save it as something for humans to read.

However, there are some issues. Until now, I've been extracting the save files and interpreting the `gamestate` file as LUA, then converting the LUA to a dict for easier manipulation within Python. However, some of the entries within the gamestate don't really translate properly and I'm ending up with some pretty dodgy looking histories (ie, nothing happens but peace declarations and leader deaths, highly unlikely for our games and a complete certainty in our games, respectively).

My questions are:

• What format are the gamestate files?
• Is there any python (or other package) which can be used to turn some of the data into something easier to manipulate?
• (Ideal) Is there a known/easy way to get the relevant events of the game without heavy manipulation of save files?

Thanks for reading my blog.

 

[Studoku]

Sounds like the chronicles feature from CK2. Which would be awesome in Stellaris.

 

[serpentskirt]

(Ideal) Is there a known/easy way to get the relevant events of the game without heavy manipulation of save files?

Outside of manipulating save file you have at least three options:
OCR approach: basically everything that shows in a pop-up can be extracted once you've calibrated your text recognition module enough. This is the safest universal method of obtaining data from the game but it suffers from some false recognitions and might be quite performance-heavy.

You can greatly improve performance if you trigger OCR based on couple of pixel checks - e.g. you know the war declaration pop-up draws pixels of certain color in certain coordinates, so you launch your recognition module only when these conditions are met.

Memory scanning: most of the stuff is located in the game's memory and can be extracted from there. This is quite reliable approach, but requires a lot of reverse-engineering.

Stellaris' EULA doesn't allow reverse-engineering "other than may be permitted according to applicable law". If I read it correct, EU laws do not forbid you to do reverse engineering in order to obtain data needed for your own program to work. It would be actually interesting to hear an answer from PDX regarding this matter.

@Jamor may I ask you to check with legal department if reverse-engineering Stellaris in order to create ledger/game history generator is considered a fair practice and does not require the authorisation of the right-holder (PDXIAB in this case)? For residents of EU and US.

Other drawback of this method is that your software might or might not get you flagged by whatever cheat detection system just because you're accessing other process' memory, so even if it's EULA-safe I wouldn't dare to public release it.

Network packets inspection: since you've mentioned that you are playing over the net, you might get data of interest analyzing packets which are being sent/received. This is generally less bothersome than reverse-engineering game itself and is less prone to detection. I'm not sure if that might or might not violate EULA, my gut feeling would say that network protocol is a part of the game and will be treated same as reverse-engineering the game itself.

I will not call these approaches easy but they might get what you want. Don't know if there are existing solutions, but I had impression they've ported whole universe to python at this point.

 

[BlackUmbrellas]

Would you consider sharing this application once you've finished getting it working? I'd love to use it for some of my RP games.

 

[Defiler99]

The gamestate is in some custom Clausewitz engine format; I'm not sure if it has an explicit name.
If it's any help at all, somebody wrote a syntax file for it (for a text editor I do not use, myself):
https://forum.paradoxplaza.com/foru...ad-stellaris-syntax-highlighting-file.938758/

Very cool idea; I suspect you'll want to use a more full-featured parser generator, or the like, rather than just treating it as Lua. Something like this but for this format vs. .INI: http://pyparsing.wikispaces.com/file/view/configParse.py

By the way, if you figure out what the <key> values actually represent, for example the values assigned to country_flags, I'd love to know!
They look like timestamps to me but I have no idea what format they are packed into, if so.

 

[RandomZach]

What format are the gamestate files?

As of others have said, a proprietary format of paradox's own creation. It's most frequent informal name is just 'paradox script'. I'm not sure if it has a formal name. They use it across all their Clausewitz games with some slight tweaks in each different game. A big problem with just interpreting it as Lua is that keys can be repeated inside an object, which is not a native feature of hash/dictionaries in any programming language I'm aware of.

Is there any python (or other package) which can be used to turn some of the data into something easier to manipulate?

I'm not aware of any Python solutions. There is however a javascript parser called Jomini that I've used personally for this same purpose with great success. The author was also quite responsive when I had an issue with the minor differences related to Stellaris itself. There are also parsers written in Ruby that I haven't used, but may be worth checking out that can be found here and here.

Paradox script is not terribly complicated, especially if you are just looking at save files. Your best bet may actually be just writing your own parser using a standard parsing library. I did this for my own purposes in Ruby.

Is there a known/easy way to get the relevant events of the game without heavy manipulation of save files?

I don't believe event history is persisted anywhere in the save file. Only events that are scheduled in the future and ongoing special projects are recorded. You'd need to analyze flags and modifiers to have some idea of what events had fired, and even that would be error prone, wouldn't give you exact dates, and would miss many events. The only solutions i can think of are real time memory scanning and/or making a mod that explicitly logged every single event (this would be a large mod) and then tailing said log. You could probably write a script that automatically generated the mod for you by parsing the game files.

Also, just a small pedantic nitpick, Lua is not an acronym. It means 'moon' in Portuguese, which is the native language of its original creator.

 

[gedruchy]

OCR approach: basically everything that shows in a pop-up can be extracted once you've calibrated your text recognition module enough. This is the safest universal method of obtaining data from the game but it suffers from some false recognitions and might be quite performance-heavy.

I considered OCR, but it feels to unreliable. I don't want to have to mess around with pixel perfection, especially considering my PC is made of wood, so I don't have a lot of pixels to work with.

Memory scanning: most of the stuff is located in the game's memory and can be extracted from there. This is quite reliable approach, but requires a lot of reverse-engineering.

This was a more serious consideration, but I didn't want to risk the ire of steams banning system for me or my friends. Plus, as you mentioned, this would make sharing the tool a bit of a risk and a README.md that says "hit run to get banned instantly" wont exactly encourage many downloads.

Network packets inspection

I could possibly get a pineapple and a copy of Charles going to do this, but it wont be super useful in single player (which is a minor requirement). Plus this massively lifts the buy in required from other people who may want to use this. Appreciate all the suggestions though.

Would you consider sharing this application once you've finished getting it working? I'd love to use it for some of my RP games.

Totally, its one of the main reason that I'm trying to avoid building something that requires special equipment or a serious programming background. I'll post it here once I have something that operates according to the laws of physics.

By the way, if you figure out what the <key> values actually represent, for example the values assigned to country_flags, I'd love to know!
They look like timestamps to me but I have no idea what format they are packed into, if so.

Once the arcane secrets of the save files have been decoded, I'll be sure to post an analysis (if needed).

I'm not aware of any Python solutions. There is however a javascript parser called Jomini that I've used personally for this same purpose with great success. The author was also quite responsive when I had an issue with the minor differences related to Stellaris itself. There are also parsers written in Ruby that I haven't used, but may be worth checking out that can be found here and here.

This is the stuff I'm looking for. Thanks very much for this. I'm probably going to include this as a node subprocess for my current script. This will at least give me an easy way to transverse what info can be found within the saves themselves.

I don't believe event history is persisted anywhere in the save file.

I was planning on increasing save frequency and max saves via mods then analysing each save in order to get a picture of the current state. This might lose some info, its true, but I'll see what interesting info I can get then apply more complex processes from there.

Also, just a small pedantic nitpick, Lua is not an acronym. It means 'moon' in Portuguese, which is the native language of its original creator.

Huh, the more you know.


This has been a useful thread. I'll be posting updates as the tool progresses.

 

[serpentskirt]

I considered OCR, but it feels to unreliable. I don't want to have to mess around with pixel perfection, especially considering my PC is made of wood, so I don't have a lot of pixels to work with.

Actually it can be pretty fast, depends on how do you do it. For one of MMORPGs I'm playing, I'm able to pull and extract from screencaptures about 40 buff icons and corresponding 2-digits timers at least three times per second. For events with unique pop-up graphics you may end up with just 3-5 pixels fully defining your picture. It's a bit tricky with text (unless it has fixed character width), but you may just save image and OCR it later.

It is still pain in the ass because you'll need to support each resolution and text colour.

This was a more serious consideration, but I didn't want to risk the ire of steams banning system for me or my friends. Plus, as you mentioned, this would make sharing the tool a bit of a risk and a README.md that says "hit run to get banned instantly" wont exactly encourage many downloads.

Well, there is always plaza version, but you'll better logout of steam/origin/whatever anyway.

 

[gedruchy]

Bit of an update here. I've completed rev 0.1 of my stellaris history generator. Check it out on github:

https://github.com/claydegruchy/Stellaris-history-builder

There is still a fair bit left to do, but this at least allows a bit of access to save games, to a point.

inb4 'the code sucks': this is a spare-time utility, not da vinci's greatest aspiration.