File storage / DB / save game size

[Aquitaine]

I'm all for mod-able files and direct access to data. I think Paradox has extracted an enormous amount of utility from storing everything in text. I also think this decision is probably made already, if not even implemented, so maybe this is a moot point, but:

- Non-relative, text data should be stored as XML
- Relative data (history, armies, countries, alliances, families if they're like CK at all) should be stored in a relative database.

It's not a huge deal to make that database accessible to modders. But given the sheer volume of data the EU* engines deal with, they're crying out for a relational database instead of huge text files.

That said, EU2 save games (if I recall) were nowhere near the size of Vicky and CK savegames, so perhaps this isn't as big a deal. But given the capability of even a simple, decent SQL-ish engine as opposed to raw text, it still seems like something worth considering. You also don't get a 5-10 second pause every year on autosave.

I know Paradox must have already considered this and is more than capable of making the right call. Just figured I'd add my two cents, cause that's what we do 'round here.

 

[Hallsten]

Good idea, but you know the rules of optimization:

The First Rule of Optimization: don't do it.
The Second Rule of Optimization (Experts only): don't do it _yet_.

soooo, I think that the good people of Paradox might not do it quite yet...

 

[Aquitaine]

Good idea, but you know the rules of optimization:

The First Rule of Optimization: don't do it.
The Second Rule of Optimization (Experts only): don't do it _yet_.

soooo, I think that the good people of Paradox might not do it quite yet...

If I were in Paradox's shoes, it wouldn't even be a question of optimization. It'd be a question of capability. You just can't do the quality or quantity of math and analysis on a text file that you can with relational data. It would make event scripting a thousand times more powerful, since you could say 'any bastard children three times removed from His Grace, the Loon of Leicester, should have x happen to them.'

er, because, that's really what EU3 needs!

 

[Duuk]

ON THE ONE HAND....

I like being able to quickly edit and mod stuff in a text file.

ON THE OTHER HAND...

A db would have probably been a better way to handle characters in CK.

On the THIRD hand (I'll borrow one from someone)...

By the time EU3 hits the market next year, 1G RAM in machines will be more "standard" (512 is already in everything), and things will move quicker anyway.

So basically, I'm fine with whatever they do. But if they move to a db, I'd hope they OFFICIALLY release an editor for it.

 

[bmk1st]

- Non-relative, text data should be stored as XML

Why does it have to be XML?

- Relative data (history, armies, countries, alliances, families if they're like CK at all) should be stored in a relative database.

It's not a huge deal to make that database accessible to modders. But given the sheer volume of data the EU* engines deal with, they're crying out for a relational database instead of huge text files.

I don't know how can a relational database be useful in a game. Won't that take more CPU resources and eat up RAM?

 

[kelumden]

- Non-relative, text data should be stored as XML
- Relative data (history, armies, countries, alliances, families if they're like CK at all) should be stored in a relative database.

XML can be a real pain when badly designed. There is nothing exceptional with XML: it is just another way to format data.

Using it will not magically made modding any easier than it is already. While it has been designed so that anybody can read it, tags never carry nor enforce semantic meaning upon data.

Do not forget that all modders are not necessarily programmers nor familliar with XML. Being a former programmer, I never considered XML as being THE magical tool of the trade.

Regarding relational database, there is no need for such database but a careful relational design can be a bonus.

As said in another post, at this stage of design, I would guess that the structure of data has already be designed, with an eye on compatibility with EU2.

The real bonus would be to dispose of tools to handle data, like:

• an Event Editor, to allow creation and modification of events;
• a Country Editor, to add and modify countries;
• a Province Editor, to do the same with provinces;
• a Scenario Editor, to handle scenarios;
• a Map Editor;
• ...

 

[kelumden]

If I were in Paradox's shoes, it wouldn't even be a question of optimization. It'd be a question of capability. You just can't do the quality or quantity of math and analysis on a text file that you can with relational data. It would make event scripting a thousand times more powerful, since you could say 'any bastard children three times removed from His Grace, the Loon of Leicester, should have x happen to them.'

An event is more like a scripting language than a way of storing data. Pretending that scripting will be a thousand time more powerfull is fool.

About SQL
When SQL was first introduced in the late 80's, the commercial said that "customers will be able to directly extract information from databases". The same has been said for COBOL ! The end result is that programmers are still needed to resolve problems expressed in SQL or COBOL. The power has not moved to the customers. As always, there is no magical solution to a problem. I would say that there are as much solutions as there are programmers if not more !

If I go with the exemple you give, the database you envision must trace the fact that somebody is a bastard and that she has been removed three times from the Loon of Leicester. Quite a lot of details that should go in the saved game !

About event scripting
Parsing a language is an exercise already solved: you can describe the grammer of the language with an EBNF-like notation and create the corresponding parser with tools like Lex and Yacc (or there more modern versions). The result of parsing can furthermore be stored as bytecodes, a fast compromise between compiled code and interpreted one. If the resulting bytecodes is too big to be kept in memory, it can be stored on disk and portions of it loaded when needed.

Expressing events using a scripting language facilitates extraction of information but it must be related to the state of the game.

 

[gosam]

Just to throw my 2cts.
If Pdox would be to make any change to the way they handle data storage, my only demand ( supplication ) would be to have enough documentation to be able to build tools around it. Wether they choose xml or a db is of lesser importance.

 

[Registered]

It would also be nice if there was some way to sort/delete savegames ingame. Instead of having to do it manually.

 

[unmerged(18425)]

Just to throw my 2cts.
If Pdox would be to make any change to the way they handle data storage, my only demand ( supplication ) would be to have enough documentation to be able to build tools around it. Wether they choose xml or a db is of lesser importance.

Which is a reason that the text file system is so well suited for modders.

 

[Grosshaus]

• an Event Editor, to allow creation and modification of events;
• a Country Editor, to add and modify countries;
• a Province Editor, to do the same with provinces;
• a Scenario Editor, to handle scenarios;
• a Map Editor;
• ...

An alternative way is to instead create an editor that considers the .txt files to be a database.

 

[bmk1st]

Which is a reason that the text file system is so well suited for modders.

That's what I think exactly same thing.

Extra editors suggested by others would be nice to have, though. However, it would be also cool to have some loyal fans (who happen to be programmers) and work together to provide modding editors for EU3.

 

[Aquitaine]

If I go with the exemple you give, the database you envision must trace the fact that somebody is a bastard and that she has been removed three times from the Loon of Leicester. Quite a lot of details that should go in the saved game !

This information is already stored in the save game (in this example, it would be the CK savegame). But it is not -related-. That's the intensive task and what makes a lot of event scripting highly inefficient. A CK savegame knows every detail about every character, but it can't put the pieces together without a lot of text-file scraping. Any time you want to do anything or find out anything about a group of characters (or countries, if we're talking EU2 terms), it has to start from scratch and scan everything in memory (or the whole text file, if we're talking about a save game).

I'm not knocking the text file that has been the staple of all Paradox's games. I'm just saying that the one and only advantage to using raw text for everything is accessibliity, and a much higher degree of accessibility can be had with some form of relational storage. If it was SQL, for example, basic SQL is much, much easier to learn than CK-style event scripting. You do not have to be a programmer to puzzle through it any more than you have to be one to puzzle through the current system.

My system has a dual-core CPU and 2 gigs of RAM, and it still putters through CK text files for a while before chewing up the whole thing.

 

[OriginalRafiki]

This information is already stored in the save game (in this example, it would be the CK savegame). But it is not -related-. That's the intensive task and what makes a lot of event scripting highly inefficient. A CK savegame knows every detail about every character, but it can't put the pieces together without a lot of text-file scraping. Any time you want to do anything or find out anything about a group of characters (or countries, if we're talking EU2 terms), it has to start from scratch and scan everything in memory (or the whole text file, if we're talking about a save game).

You're talking in-game?

If so, the way the game data is represented internally, while playing and the way it is represented persistently (e.g. in savegames) os so unrelated that it's like talking about apples and a drawing of oranges.

If it was SQL, for example, basic SQL is much, much easier to learn than CK-style event scripting. You do not have to be a programmer to puzzle through it any more than you have to be one to puzzle through the current system.

SQL is a way of describing/relating data, but you must still be able to "process" the data, and it's a heck of a lot more accessible in text-files than it is in a relational database. Example; compare

SELECT triggerdate FROM events WHERE event_id = X

with

event = {
    id = X
    (...)
    date = { year = 1453 }

and still, we're talking about apples and oranges

 

[SJG]

This whole discussion is a red-herring.

The text files are in fact the data component of a database. All you are missing is the access mechanism that a database application provides.

 

[EnderV]

But it is not -related-.

And that's a good thing. A lot of problems with the commercial systems stems from the fact that once you release, it's pretty hard to change your dataschema.
In a way, you can look at the text files as a poor-mans' way (in a sense of not going out and shelling through your nose to buy a commeciral one) of persisting (serialising) in-memory object database.

Taking about technology, I'd much much more prefer if EU3 was implemented in Java, so I could play it on a Linux machine as well as Windoze than having to install something like MySQL as part of the installation and hogging the space (and what if I already have a MySQL instance on it?).
Not that I can see that happening although I think the general advantages would easily win over disadvantages (although a except portability, most of the advantages could be had by it being a .NET application, so maybe that will happen instead).
V.

 

[OriginalRafiki]

Java is waaaay too inefficient to be a reasonable/viable option, I'd say

 

[EnderV]

Java is waaaay too inefficient to be a reasonable/viable option, I'd say

And the experience/evidence you're basing this on is?

Doug Lea (for those who don't know, someone who's a recognized authority in the distributed world) said at the last OOPSLA conference: "if your Java code is only 12 times faster than C means you haven't started optimizing".

To put in in the context though, it he doesn't mean that your "sum all numbers from 1 to 1000" will run faster in Java than C (it will be about the same if you remove the up-front costs).
It means if you have something where you do a lot of tasks, due to the easiness (comparative to C/C++) of multithreading and memory management your design can be much more efficient. Multithreading is hard in general (the only really multithreaded game that I know of is GalCiv/GalCiv2), harder yet in C/C++, but if done properly it can be a great thing.

Java was slow, there's no question about that. It had nasty UIs, that's true too (but that was due to Sun providing stupid APIs). But things moved in the last 10 years.

I work in the finance industry, amongst other things writing valuation models and such - stuff that HAS to be as fast as possible. I didn't find Java to be perceptibly slower than any C++ code, in some cases (due to easier ability to multithread/distribute) it's quite the opposite. The major bottleneck is always database/IO (especially console IO on Windows just kills any performance. Go figure.). What it is though, is a memory hog. It's easy to have process that eats up 0.5GB of memory and doesn't give it back even when it doesn't need it anymore.
So, if you need memory efficient application, then yes, C++ is the way to go. Speed is not.

FYI, Civ4's engine is in Python, which is the same principle as Java (i.e. VM). Anything pure .NET is running on a VM too. Does it mean it's dead slow and inefficient?

V.

 

[OriginalRafiki]

And the experience/evidence you're basing this on is?

Theoretically, I base it on research presented by my professors at the university, supported by what I've read in newsletters from Sun that I subscribe to since I am a Java developer.

As you're saying, it uses a VM, which will always add an extra step to the execution, making it slower, undeniably. C/C++ is faster than Java, and will be in all foreseeable future. Java has strengths, speed not being one of them, and basing your core game code on Java would jack up the hardware requirements, which I think more than a few would object to.

FYI, Civ4's engine is in Python, which is the same principle as Java (i.e. VM). Anything pure .NET is running on a VM too. Does it mean it's dead slow and inefficient?

FYI, I know this, and have even customized my CIV 4 in some ways myself.

CIV 4 uses Python in the user interfaces and higher-level stuff, where speed is less of an issue, since you often are waiting for the user to do stuff (as opposed to massive data-/number-crunching). The core engine is still in C++, AFAIK.

Note that I'm not saying "dead slow and inefficient", I'm saying it is slower and less efficient, most likely to an unacceptable degree.

 

[EnderV]

Theoretically, I base it on research presented by my professors at the university, supported by what I've read in newsletters from Sun that I subscribe to since I am a Java developer.

I have to say I distrust research in this area, as it's relatively simple to show anything.

If I know how on-the-fly optimization in JVM works, I can write code that the dynamic optimizer collapses after a few hundreds/thousands of iterations, while the C++ code would execute till the evening. On the other hand, I can write a piece of code that knows how the memory manager in java works, and it slows the execution to a crawl, while the C++ code zips past.

So I trust real-world applications.

I trust my personal experience (as per previous post). I trust a Boieng engineer showing me a CAD-like math-heavy simulation application for testing the aircraft component stresses on his laptop, and then for comparison showing the same written in Fortran (which is optimised for maths stuff, a lot of computation is still done in Fortran) on a big mainframe, and seing the difference. (that was fours years ago, running on JVM 1.3. with 1.4 the differences were even more pronounced). Oh yes, and he also showed me the application running on JVM on the mainframe and it was still faster (although only slightly) than the Fortran one.

I trust using something like Intellij's IDEA - the only speed problem I ever had with it was due to the antivirus insisting on running continuous scans on all its files as it was loading them. If you are a Java developer, you know it - amongst other things it kills all the arguments for Java UI having to be bad. If you don't you'd have a go and try it.

That said, you could make an argument (and a fairly successfull one based on my experience), that to be a good C++ programmer you have to be smarter than to be a good Java programmer. Thus, the amount of crappy Java applications will be proportionally larger than the C++ ones. Although Java still beats VBA in that respect.

As you're saying, it uses a VM, which will always add an extra step to the execution, making it slower, undeniably. C/C++ is faster than Java, and will be in all foreseeable future.

It also allows you for dynamic code optimization, which can be far more efficient than static one (also much, much harder too, mainly de-optimizing if needed).

The extra pip of C++ efficiency that you _can_ (but not necessarily have to) get might be worthwhile when your processing power was where it was in 90's, or even around 2000. It has moved, and the difference has collapsed.

After all, we could code it all in assembler, and could optimize a lot of stuff to make it even faster than C++? That's what the people who did games for ZX, C64, 800XL did...

V.