A number of questions and comments

[Jamie550]

In db\countries.txt, there are entries "color = LightGrey" and "color = DarkGrey". But in db\map\colorscales\colorscales_political.txt there are only LightGray and DarkGray. Is this supposed to be the case?

In db\countries.txt, there are entries "new_colony = TAU" and "new_colony = MIN", but TAU and MIN don't seem to have colony entries. Is this supposed to be the case?

Is there a reason for events to have "command = { }" clauses?

In db\events\major_hol.txt, there is in line 660 "flag = [VOC]". Are the [] acceptable syntax?

Are commands such as this valid: "command = { type = addcore which = PER value = 528 }"?

In leader files, there seem to be duplicate IDs of 0300 and 0299, one of each in each of leaders_cri and leaders_kri.

Are leader IDs and monarch IDs separate, i.e. you can have a leader and a monarch with the same ID?

What does it mean to say that a leader is "special = 1" in a leader file?

There seem to be duplicate leader IDs 2121, 2120, 2119 in tos and tus.

French countries.csv: There seems to be a wrong newline in line 372.

French events.csv: The following lines consist of ";x"; should they be "#;x"? 574, 779, 1076, 1410, 1535, 1624, 1675, 1888, 2139, 2598

French events.csv: There seems to be a wrong newline in line 3102 and 3126 and 3204.

The keys "DAU", "HSA", "MAD_DESC", "ACTIONNAME3689A" is in English but not French.

In revolt.txt:
--- Error 1 of 8 ---

At Db\revolt.txt [SIN\capital] (2477, 2):
The capital province '541' must be included in the minimum province list.
--- Error 2 of 8 ---

At Db\revolt.txt [LEO\extra\Item[1]] (2030, 12):
Extra province '435' is already included in the minimum province list.
--- Error 3 of 8 ---

At Db\revolt.txt [ADA\capital] (1498, 2):
The capital province '759' must be included in the minimum province list.
--- Error 4 of 8 ---

At Db\revolt.txt [QAR\capital] (655, 2):
The capital province '486' must be included in the minimum province list.
--- Error 5 of 8 ---

At Db\revolt.txt [ALD\extra\Item[2]] (312, 16):
Extra province '737' is already included in the minimum province list.
--- Error 6 of 8 ---

At Db\revolt.txt [SWE\extra\Item[2]] (247, 16):
Extra province '252' is already included in the minimum province list.
--- Error 7 of 8 ---

At Db\revolt.txt [RUS\extra\Item[4]] (208, 24):
Extra province '272' is already included in the minimum province list.
--- Error 8 of 8 ---

At Db\revolt.txt [RUS\extra\Item[3]] (208, 20):
Extra province '271' is already included in the minimum province list.

Regarding mods: Do mod files have to be in the form "mod_*.txt" or is that just convention? Also, is it that the game combines file paths in mods with file paths in the vanilla files, discarding the vanilla paths when there are duplicates? i.e. Is it like the "extend" command of EU3 mods?

How do AI file references work? Such as "ai = 'England.ai'", which doesn't seem to exist as a file.

Besides crawling through all events, scenarios, etc, is there a way to compile a list of all AI files?

For leader files, what values can rank, movement, fire, shock, siege take? Also, is it that the first four are required while the last is optional?

 

[MichaelM]

In db\countries.txt, there are entries "color = LightGrey" and "color = DarkGrey". But in db\map\colorscales\colorscales_political.txt there are only LightGray and DarkGray. Is this supposed to be the case?

Not intended and has been fixed in my version (I'd have to check the logs to see if it was fixed before 1.1).
EDIT: Should have mentioned that "gray" is the intended spelling in FTG.

In db\countries.txt, there are entries "new_colony = TAU" and "new_colony = MIN", but TAU and MIN don't seem to have colony entries. Is this supposed to be the case?

Not intended. The MIN seems to have been cleared out, but the TAU is still there. I'll bring it up with somebody (because I sure don't know much about Taungu).

Is there a reason for events to have "command = { }" clauses?

In EU2, there could be problems if all command were invalid and therefore removed by the engine. I don't know if this problem remains in FTG, but many of the files have not been edited and there's no reason to change something that works.

In db\events\major_hol.txt, there is in line 660 "flag = [VOC]". Are the [] acceptable syntax?

Yes, any character is acceptable in a flag (even spaces if the flag is enclosed in quotes). However, nothing seems to ever set that flag, which makes me suspect that someone blindly copied that event from AGCEEP.

Are commands such as this valid: "command = { type = addcore which = PER value = 528 }"?

No. Where is that? extragc.txt?

In leader files, there seem to be duplicate IDs of 0300 and 0299, one of each in each of leaders_cri and leaders_kri.

Interesting. It's invalid, but only if both exist at the same time. And KRI is not a tag. The file has a date of April 2002, so I guess the tag was changed at some point.

Are leader IDs and monarch IDs separate, i.e. you can have a leader and a monarch with the same ID?

I believe that IDs must be unique only within their type, and types are somewhat restricted. But don't quote me on that; I'll check it more thoroughly after work today.

EDIT: This is correct. And since leaders and monarchs both must be type 6, they share the same ID space. Do you want a list of what IDs are valid for what?

What does it mean to say that a leader is "special = 1" in a leader file?

Nothing. That's a leftover from probably EU1 and is still valid but meaningless.

There seem to be duplicate leader IDs 2121, 2120, 2119 in tos and tus.

Same issue as CRI/KRI; TUS isn't a country.

French countries.csv: There seems to be a wrong newline in line 372.

French events.csv: The following lines consist of ";x"; should they be "#;x"? 574, 779, 1076, 1410, 1535, 1624, 1675, 1888, 2139, 2598

French events.csv: There seems to be a wrong newline in line 3102 and 3126 and 3204.

The keys "DAU", "HSA", "MAD_DESC", "ACTIONNAME3689A" is in English but not French.

OK, I'll bet hildoceras already has these covered.

In revolt.txt:
--- Error 1 of 8 ---

At Db\revolt.txt [SIN\capital] (2477, 2):
The capital province '541' must be included in the minimum province list.
--- Error 2 of 8 ---

At Db\revolt.txt [LEO\extra\Item[1]] (2030, 12):
Extra province '435' is already included in the minimum province list.
--- Error 3 of 8 ---

At Db\revolt.txt [ADA\capital] (1498, 2):
The capital province '759' must be included in the minimum province list.
--- Error 4 of 8 ---

At Db\revolt.txt [QAR\capital] (655, 2):
The capital province '486' must be included in the minimum province list.
--- Error 5 of 8 ---

At Db\revolt.txt [ALD\extra\Item[2]] (312, 16):
Extra province '737' is already included in the minimum province list.
--- Error 6 of 8 ---

At Db\revolt.txt [SWE\extra\Item[2]] (247, 16):
Extra province '252' is already included in the minimum province list.
--- Error 7 of 8 ---

At Db\revolt.txt [RUS\extra\Item[4]] (208, 24):
Extra province '272' is already included in the minimum province list.
--- Error 8 of 8 ---

At Db\revolt.txt [RUS\extra\Item[3]] (208, 20):
Extra province '271' is already included in the minimum province list.

Thanks, I'll look into these.

Regarding mods: Do mod files have to be in the form "mod_*.txt" or is that just convention?

It's enforced.

Also, is it that the game combines file paths in mods with file paths in the vanilla files, discarding the vanilla paths when there are duplicates? i.e. Is it like the "extend" command of EU3 mods?

Yes, that's it. You can see this happening in moddebug.txt.

How do AI file references work? Such as "ai = 'England.ai'", which doesn't seem to exist as a file.

This is an error; it wasn't updated when the AI file naming conventions changed from X.ai to X_ai.txt.

Besides crawling through all events, scenarios, etc, is there a way to compile a list of all AI files?

No, I don't think so.

For leader files, what values can rank, movement, fire, shock, siege take? Also, is it that the first four are required while the last is optional?

They're basically unlimited, but the normal ranges are 1-6, except for rank. Rank goes basically from 0-100, where 0 usually goes with a type of "monarch" but does not have to, and nothing more than 25 is normally used. Defined leaders who are not conquistadors usually have a rank higher than 10.

All of these values default to 0, so it's technically valid to leave them out -- but of course it's almost certainly a mistake.

 

[Jamie550]

Thank you for your help, MichaelM.

No. Where is that? extragc.txt?

Yes, in there.

EDIT: This is correct. And since leaders and monarchs both must be type 6, they share the same ID space. Do you want a list of what IDs are valid for what?

That would be great. So every ID must have a unique combination of "type" and "id", while some types of IDs are restricted to certain "type" values?

 

[MichaelM]

Actually that was a mistyping; I meant which types are valid. IDs are any 32-bit unsigned integer (there are some reserved IDs but they're only used with reserved types, so it makes no difference).

Types range from 0 to 65535. They are as follows:

0-5: Reserved for internal use
6: Leaders and monarchs
7: Events (used internally)
8: Reserved
9-4711: Static types (I'm not exactly sure what this means)
4712-9423: Dynamic types (you'll see lots of "type = 4712" in save files; I think this is actually 4712 + the local player's ID, so it might be different in MP games)
9424-65533: I'm actually not sure, but I don't think they can be used
65534 and 65535: Reserved

This 48-bit space does get hashed down to 32 bits, but I've never heard of this causing a problem. The odds are very slim indeed.

 

[Jamie550]

Actually that was a mistyping; I meant which types are valid. IDs are any 32-bit unsigned integer (there are some reserved IDs but they're only used with reserved types, so it makes no difference).

Types range from 0 to 65535. They are as follows:

0-5: Reserved for internal use
6: Leaders and monarchs
7: Events (used internally)
8: Reserved
9-4711: Static types (I'm not exactly sure what this means)
4712-9423: Dynamic types (you'll see lots of "type = 4712" in save files; I think this is actually 4712 + the local player's ID, so it might be different in MP games)
9424-65533: I'm actually not sure, but I don't think they can be used
65534 and 65535: Reserved

This 48-bit space does get hashed down to 32 bits, but I've never heard of this causing a problem. The odds are very slim indeed.

Are the odds that small? I tried a simulation - 10000 runs, and detected a 0.0112 chance of collision. The algorithm generated 10000 uint64's each run, then cut the value down to 48 bits, chose the distinct values, then did a simple hash of modulo (2^32), and saw if there were any collisions. Perhaps there is a bug in this algorithm, but at least parts of it were verified by comparing to known probabilities in the birthday problem. Or maybe the real hashing algorithm takes into account the relative frequency of certain types (e.g. 6).

That was a tangent of curiosity, but thanks for this and all of the other information you've provided.

 

[MichaelM]

Hmm... I don't know if I ought to publish the actual hashing algorithm, because it's used for several things. Can you post your code so I can try it myself?

 

[Jamie550]

The original code I used (in F#)

open System

let hashFunction (i : uint64) : int32 = int32(i % uint64(UInt32.MaxValue))

let numLoops = 10000
let numIds = 10000

let mod48 = Seq.init 48 (fun _ -> uint64(2)) |> Seq.fold (fun acc v -> acc * v) (uint64(1))
let mod32 = Seq.init 32 (fun _ -> uint64(2)) |> Seq.fold (fun acc v -> acc * v) (uint64(1))

let r = new Random()
let bytes = Array.create 8 0uy

let checkCollision () =
    let list = List.init numIds (fun _ -> 
        r.NextBytes(bytes)
        let value = BitConverter.ToUInt64(bytes, 0)
        let value48 = value % mod48
        value48) |> Seq.distinct |> Seq.toList

    let filtered = list |> List.map (fun v -> v % mod32) |> Seq.distinct |> Seq.toList
    
    List.length list = List.length filtered
    

let collidePercent () = double(Seq.init numLoops (fun _ -> checkCollision()) |> Seq.filter (fun arg -> arg = false) |> Seq.length) / double(numLoops)
printfn "%A" (collidePercent())
Console.ReadLine() |> ignore

Now, however, it should be perfectly possible to generate theoretical probabilities. The chance of collision with a space of size r and h unique hashes, with n IDs should be:
(product of i from 1 to n - 1) (r - r/h*i)/(r-1).

Possible justification: Let us pretend that space is arranged in a grid, with each column being a unique hash and each row of that column being the possible original values of that hash. Now, for selecting the first value, the probability of no collision is 1. The probability of selecting another value with no collision is then (r - r/h) / (r - 1) - there are (r - 1) possibilities to choose from (since we know that our list of IDs is unique), and we have to select a value from any column except the one already selected - hence sample space minus the values in a single column, yielding the number of remaining cells. This argument continues to the n'th ID. Running a theoretical simulation with the following code (in C#; all in a class):

		static void Main()
		{
			Console.WriteLine("Birthday problem (modified) (expected 0.300817): " + CollisionProbability(365 * 2, 365, 23));
			Console.WriteLine("10000 IDs: " + CollisionProbability(space48, space32, 10000));
			Console.WriteLine("1000 IDs: " + CollisionProbability(space48, space32, 1000));
			Console.WriteLine("6243 IDs (Leaders+Monarchs): " + CollisionProbability(space48, space32, 10000));

			Console.ReadLine();
		}

		static double CollisionProbability(ulong space, ulong hashCount, ulong idCount)
		{
			double result = 1.0;
			double numDups = (double)space/hashCount;

			for (ulong i = 1; i < idCount; i++)
				result *= (space - numDups * i) / (space - i);

			return 1.0 - result;
		}

		private static readonly ulong space48 = Power(2, 48);
		private static readonly ulong space32 = Power(2, 32);
		private static ulong Power(ulong b, ulong exp)
		{
			ulong ret = 1;

			for (ulong i = 0; i < exp; i++)
			{
				ret *= b;
			}

			return ret;
		}

(Note: Birthday probability was established by monte carlo simulation with aforementioned F# code; numIds = 23; mod48 = 365*2; mod32 = 365)

The results are:

Birthday problem (modified) (expected 0.300817): 0.300687288107166
10000 IDs: 0.0115727144237557
1000 IDs: 0.000116290378694095
6243 IDs (Leaders+Monarchs): 0.0115727144237557

Note: The number of leader+monarch ID was calculated by first searching the vanilla files for instances of historicalleader and historicalmonarch; and second using a program, which among other things found the number of IDs of leaders and monarchs. The former method yielded 6247 IDs; the latter method yielded 6243 and so was the used one.

 

[Jamie550]

A few other questions:
Does "emperor = no" in monarch files mean anything?
In leader and monarch files, "deathdate" and "enddate" are synonyms? And they are optional?

In scenario files, is this true: eeg and inc files are completely identical, except that each eeg file in the root scenario folder denotes a scenario and each eeg file must have one header section. Beyond that, eeg and inc files can both contain country, cot, province, etc declarations. However, txt files consist only of "event = '...'" declarations.

In scenario files, globaldata\war clauses, what does it mean for attackers and defenders to have "expirydate" clauses? And does "type" ever take a value other than "war"?

 

[MichaelM]

A few other questions:
Does "emperor = no" in monarch files mean anything?

Yes, it's a new feature that allows specific monarchs to be prevented from ever being elected emperor.

In leader and monarch files, "deathdate" and "enddate" are synonyms? And they are optional?

Yes, they are synonyms. They are optional for monarchs; the actual date of death is whenever the next monarch starts. But near as I can tell with a quick look, leaders will never die if they have no deathdate.

In scenario files, is this true: eeg and inc files are completely identical, except that each eeg file in the root scenario folder denotes a scenario and each eeg file must have one header section. Beyond that, eeg and inc files can both contain country, cot, province, etc declarations. However, txt files consist only of "event = '...'" declarations.

Only the .eeg convention is enforced. All else is referred to from the .eeg file and thus can have any extension. AGCEEP uses .eue files for events, for example.

In scenario files, globaldata\war clauses, what does it mean for attackers and defenders to have "expirydate" clauses? And does "type" ever take a value other than "war"?

Expirydate is meaningless; it's just there because that's the same object that is used for alliances. Same for the type. But for both of these, I actually don't know what would happen if they were set to other values.


I'll comment on the ID stuff after work.