Hacking the TBL files

[Inferis]

Okay, small update. I have managed to find some time to continue work on this. Basically, I'm building a library modify the EU2 files. At the moment, all it does it read stuff, but there are "provisions" to add writing too. This includes the text files (eg provinces.csv).

Regarding editing the map: I've said before it is really hard. I've been fighting with figuring out an algorithm to compress an image to the TBL format, but again: fucking hard.

 

[Hive]

Okay, small update. I have managed to find some time to continue work on this. Basically, I'm building a library modify the EU2 files. At the moment, all it does it read stuff, but there are "provisions" to add writing too. This includes the text files (eg provinces.csv).

Regarding editing the map: I've said before it is really hard. I've been fighting with figuring out an algorithm to compress an image to the TBL format, but again: fucking hard.

In Inferis we trust.

 

[unmerged(5110)]

Okay, small update. I have managed to find some time to continue work on this. Basically, I'm building a library modify the EU2 files. At the moment, all it does it read stuff, but there are "provisions" to add writing too. This includes the text files (eg provinces.csv).

Regarding editing the map: I've said before it is really hard. I've been fighting with figuring out an algorithm to compress an image to the TBL format, but again: fucking hard.

You know, the more information you can give us on that website of yours, perhaps the more somebody will go on the forum, read this thread, read your website and might have a good idea or two... hmmmm????

 

[Elmokki]

Could it be so, that the images on .tbl files are with a standard compression?

Too bad that there are many image file standards.

 

[Inferis]

Could it be so, that the images on .tbl files are with a standard compression?

No, it's not.
I'm working on an update of my EU2 pages to include the lightmap format.

 

[Inferis]

Update's there... The basic info on the lightmap.tbl files is there now.
This does not include the drawing process, as that's another matter in itself.

I'll do a doc on that soon...

 

[Inferis]

Okay folks, time for a question. How would you take a go at editing the map? What set of tools would you develop (supposing you have the ability) to do it?

Keep in mind that:

• The total mapsize is 18944x7296. This results in an image that 395,4 MB big. An editor should be usable also for folks who cannot afford gigabytes of memory (which isn't much of a problem for Paradox, I guess).
• Several files define how stuff works: the lightmap files contain the image, but there's also the id grid, the TI, the province list, boundboxes, ... All these have to be updated a the same time

 

[Hive]

Can you perhaps make an editor load only 1 part of the map - say 1 continent - at a time? Otherwise, it's a damn big map to load every time you want to do some minor tweak...

 

[Gathenhielm]

Ahh this is so cool . Nice work Inferis.

And Hives idea sounds good. if it is doable.

 

[Nikolai II]

Can you perhaps make an editor load only 1 part of the map - say 1 continent - at a time? Otherwise, it's a damn big map to load every time you want to do some minor tweak...

Was my first thought as well, but I guess it has to be saved every time? And if full file is close to 400Mb dividing by 5 will still be 80 Mb?

Maybe divide ot into 10-20 (assuming all other files needing modification are small?) and then make some sort of 'compiler' for the map?

 

[Hive]

Was my first thought as well, but I guess it has to be saved every time? And if full file is close to 400Mb dividing by 5 will still be 80 Mb?

Maybe divide ot into 10-20 (assuming all other files needing modification are small?) and then make some sort of 'compiler' for the map?

Good idea... extract the main map into 10-20 small chumps, which can be edited alone.... and then, when you want to use the map - you compile them together.

 

[eon 2]

Keep in mind that:

• The total mapsize is 18944x7296. This results in an image that 395,4 MB big. An editor should be usable also for folks who cannot afford gigabytes of memory (which isn't much of a problem for Paradox, I guess).

If the technical solution to that problem is working with only a smaller a part of the map at a time, I would recommend dividing the map in several overlapping parts (overlapping so You could choose a conveniant view to work form according to Your needs). E g part1 :North America, part2: souther part of NA and northern part of South America and part 3: SA. Etc.

• Several files define how stuff works: the lightmap files contain the image, but there's also the id grid, the TI, the province list, boundboxes, ... All these have to be updated a the same time

Ideally You should (?) have a choice to either work with the map whithout considering the provinces (province borders) or in a "province mode" where the current "province data" is always present (editable, maybe in a floating box) when You are editing the map.


BTW, have You rellay deduct the format of the TBL files ? Fantastic !!

 

[Nikolai II]

If the technical solution to that problem is working with only a smaller a part of the map at a time, I would recommend dividing the map in several overlapping parts (overlapping so You could choose a conveniant view to work form according to Your needs). E g part1 :North America, part2: souther part of NA and northern part of South America and part 3: SA. Etc.

Or if a 'compiler' is possible, then maybe also a 'cutter' to take the map to pieces, allowing for at least two different modes of cutting (that places borders differently?)

Or cut up into 20-30-40 pieces and allow 'editor' to be able to load two adjacent pieces at once if needed?

 

[Inferis]

A compiler would most probably be the way to go. I've been thinking about that for a while now.

Basically, the system I have in mind contains:
* a decompiler, to transform the current EU2 files into a format of our own we can use to edit (maybe just one (very) large file, but easier to maintain).
* an editor to edit the map
* a compiler to transform our own format into EU2 readable files.

This would allow you to have multiple working copies of a map (given enough diskspace, but that's not so much of an issue as memory is), compiling them as needed so they can be used to play.

The editor would be used to edit stuff like sprite placement, actual pixel ownership (the stuff stored in id.tbl), province names, ... The image (or part of it) would be exported to a TIFF or so, which can be edited in a graphical editor (eg photoshop). No need in duplicating functionality.

Any thoughts?

 

[ForzaA]

probably a very stupid and impossible suggestion

There are a few "ID maps" floating around the forums, I don't think these, even when fitted together (I have separates for all continents) would put too much strain on any PC. now, if it was possible to select, in such a map, a certain region (by "cutting" ) that you want to edit, this region is then taken into the compiler/editor/... , the rest keeps unchanged.

that way, everyone could pick a size fitting to their current needs, and to hardware available.

 

[Hive]

probably a very stupid and impossible suggestion

There are a few "ID maps" floating around the forums, I don't think these, even when fitted together (I have separates for all continents) would put too much strain on any PC. now, if it was possible to select, in such a map, a certain region (by "cutting" ) that you want to edit, this region is then taken into the compiler/editor/... , the rest keeps unchanged.

that way, everyone could pick a size fitting to their current needs, and to hardware available.

This sounds awfully complicated though, imo...

Btw, Inferis - will you also included in your editor the possibility to determine where habour icons, capital icons, manufactory icons etc are placed in a province? Would be heaps helpful...

Of course, that would require you to also use province.csv - but that's an easy excel file anyway, would that be a problem?

 

[Inferis]

Btw, Inferis - will you also included in your editor the possibility to determine where habour icons, capital icons, manufactory icons etc are placed in a province? Would be heaps helpful...

Of course, that would require you to also use province.csv - but that's an easy excel file anyway, would that be a problem?

No, that's the plan. That info is by far the easiest to modify.

 

[Napoleon XIV]

Stuffs a happenin'

 

[Coder]

if we can decompile/convert a region of a tbl file into a large lightmap picture (a simple grayscale bitmap) and another "bitmap" with a province id for each pixel (and propably something for that TI boundaries file)then that grayscale bitmap is editable in any raster image editing progam
then we need to program a province ownership (and TI boundaries) editor which should be quite straightforward hopefully
before compilation we use colorscales.csv to pack that grayscale image back to 6bit "colors" and the compression should be quite straightforward (that is, if i haven't missed something important )
for each block we construct the province id list first and check if all pixels have the same color and same owner(province) in that block to build that quadtree

in short this is the same thing you already posted (with a few additions maybe)
the main problems from my point of view:

• can we modify only part of the map at a time? (i studied your documentation a few months ago but i don't remember the format of all other files and the format of incgrid.tbl is still somewhat unclear?)
  maybe it would be possible to decompile and edit a part of the map and when compiling it we alter some indexes only in other parts
• i didn't quite understand those river id quirks in lightmap.tbl
• everything has to be done using raster graphics
• i missed something crucially important

 

[Inferis]

Ah, somebody how knows what I'm dealing with.

Editing parts of the map should be feasible, but compiling should always happen for the whole map. There's too many files to modify, and it's going to be easier to do the full map at a time.

Editing the image should be done with an image editing program, indeed. We'll just have an import and export function to the EU2 map format.

Province ownership is not important in the image data when using our own temporary format. We do need to create it when compiling the EU2 map data, fcourse.

Basically, my plan is to create an editor that uses it's own file format that encapsulates all map data into one (big) file. You'd be able to import the current map from EU2 based files, and export it back to those file. But I'll be focusing on creating new maps first.

The big problem is that a lot of this stuff is interconnected, and that makes it hard to create a good object model.

And about the river id things: I don't quite get it myself. I know how it works because of the code Johan provided me, but I don't see the direct use in it, to be fair. Maybe Johan can enlighten us. I guess it has primarily something to do with border drawing.