When will goods substitution be fixed?

[Olaf Trygvasson]

Dear Cthulhu. This has to be one of the most absurd XIX-century economics thread I've seen. Obviously, when the price of gasoline goes down, people instantly abandon their gasoline-powered cars and swich to their gas-powered ones that they keep in their pockets.

What insanity still makes makes people believe in perfect substitution and has them make judgments based on outdated "utilitarian" economic theories?

Just because wood and oil are substitutes in a broad sense does not mean that they are perfect substitutes. Heck, I've lived for two years in a city where tons of people kept using overpriced coal for heating, just because wait for it, changing house heathing method is costly (not to mention other sources of inertia like personal preferences, proximity to the good suppliers etc.). Now, the game does not model such factors at all, insted opting for a simple abstraction of proportions, but this is just a model, and every model is a simplification of reality. "Fixing" these "problems" would result in things such as societies where the only consumed good is wheat or buying porcelain as their only luxury good, abandoning entirely clothing and furniture.

While obviously its not 100% of people changing instantly, are you denying that increases in cost of gasoline, along with the decreasing price of hybrid/electric vehicles, has caused people to start switching vehicles? I am currently in the process of moving from a rural state with cheap gas to a more urbanized area with more expensive gas (and worse traffic) and am probably going to end up selling my truck to buy something more fuel efficient. People switch "goods types" all the time. And when you consider that this is modeling an entire nation's behavior in reaction to price fluctuations, it makes sense that even relatively rare behavior, on the scale of tens of millions of people, would have a significant impact. Not to mention that any kind of equipment slowly breaks down and must be replaced (which is why it is constantly being produced) and when someone has to buy something new they're more likely to switch to something new. Again, this is too complex for the game to model perfectly, but having people buy products relative to the cost of the goods rather than the percent of supply is a much better simulation.

Not to mention, as some people already have, that there are also things like food where there is no switching cost associated. And its undeniable that this happens constantly. For example (I don't know if this was a widespread phenomenon or just in the US) only a few months ago when the price of eggs spiked. People didn't keep buying eggs in the same proportion, they adjusted their purchasing behavior based on the price

 

[DWeird]

The form of pushback the thread is receiving is strange to me. Goods substitution was one of the advertised features of Viccy 3 over 2, and overall a very welcome one. Instead of just having a preset basket of goods where people start to starve if you don't have enough cow even if you're swimming in grain, fruit or fish, pops will now just swap in another type of good for a need if it's appropriate. As far as I can tell, the model works well enough where different goods are roughly equivalent to each other (OP?), and only in the case where they have different scales of effectiveness for a given need that problems arise.

Problems that are severe enough that entire industries become unviable and pops wildly overspend for a type of good - all in the game where managing industries and pop needs is the main appeal. The level of loving in-depth attention OP has given the topic is exactly what gives Paradox games their longevity and instead of being applauded he's being chided by people who seem to be barely bothering to engage with the topic at all.


As to the actual topic... Both price and good type has to be considered somewhere in the in the model already, right? Maybe it's as easy as just applying a type-appropriate scaling factor on the price check.

If not, this is probably something that can be explored in a hacky way through a mod. Since unit-level substitution seems to work well enough, we could just scale everything else in the game to that. Set the substitution level to 1, have whaling stations and oil rigs produce 2x of the good and have industrial processes that consume oil consume 2x of it as well. Should work well enough so long as there aren't goods that substitute at different scales to different needs.

 

[za3tarani]

i had similar issue with the game in my first playthroughs - it bothered me that i had a oversupply of meat, and undersupply of wheat, yet pops wouldnt consume more meat to fill their need of wheat.

 

[Kaerid]

If people want to know the details on goods substitution, the algorithm follows the current rules:
- available supplies involved in a need are estimated, following the formula "(sell orders - (exports + industry usage)/2) * good base price
- a "goods share" is calculated for each good by doing "5 * (goods weight in that need) * (goods availability)/(sum of each involved goods availability)". The result is clipped between a min & a max (see https://vic3.paradoxwikis.com/Needs#Need_types for weight / min / max values, or directly in the game file since the wiki's min/max currently use an irrelevant "%" unit)
- when determining how much of a goods a population will buy for a given need, the formula is "(need value at current SOL for pop) * (goods share) / (sum of involved goods shares) / (goods base price).

Note that there is some inertia in goods share changes (impacted by some values present in defines), so the "goods share" used in the 3rd formula will not be equal to the one calculated in the 2nd formula if the economy is too far from an equilibrium (basically, you may want to wait a a couple weeks/months after switching PMs nationwide, or having any other large impact on your economy)


The idea of considering availably supply for determining goods consumption is okay on paper, since it's part of determining the price modifier for goods. But with the current implementation / resource usages, it has several obstacles if you're expecting prices to align for goods in a single need:
- the estimation/impact of goods used by industry / exports just doesn't work. In an economy where supply roughly equals demand, the 1/2 factor will cause available goods to be overestimated. And depending whether the industry/exports parts represents 5% or 95% of the total usage will cause wildly different impact on pricing: trying to consume double the available supply has a marginal impact on a goods mostly used by industry (since pop consumption is minor), but will cause prices to break the roof for goods exclusively used in pop consumption
- available supplies are overstimated when a goods is used in multiple needs (usage in other needs isn't even considered in the first formula)
- the notion of weights goes irectly against the idea of aligning prices. Having a small weight (such as the extreme 0.05 weight for services in good arts) basically means the pop don't want to buy that goods. This explains why fine arts is so expensive: because pops really avoid buying services for satisfying arts. Unless there are no fine arts.
- and of course, the min/max values would also prevent prices from aligning, but this part could be considered to be as designed

It means that in simple cases such as luxury items or luxury drinks, which don't share any goods with another needs nor with the industry, the goods substitution will usually works fine and leads to homogeneous price modifiers (unless exporting or hitting a min/max share value).
However, a goods such as wood will be overconsummed by pops, since its heavy industry usage is underestimated, and both heating & crude items needs are fighting over this resource. Fabric is in a similar situation and coal can also rely on a heavy industry usage, but oil has no such advantage to overinflate its share among heating goods, which ends up falling behind & cause low prices until it's finally used by industry.


Source: my own observations of the machine code, though a thread on reddit described a similar working via empirical observation of the game / define values

 

[yurcick]

available supplies involved in a need are estimated, following the formula "(sell orders - (exports + industry usage)/2) * good base price

This is insightful.
Do you have any idea why these formulae were used?
Because I literally see no benefit over a price equilibrium model, neither a gameplay one, nor a realism one, nor a computational one.

 

[durbal]

This is insightful.
Do you have any idea why were these formulae used?
Because I literally see no benefit over a price equilibrium model, neither a gameplay one, nor a realism one, nor a computational one.

My guess is some similar reason to the ridiculous trade formula they used on release...

 

[Kaerid]

This is insightful.
Do you have any idea why were these formulae used?
Because I literally see no benefit over a price equilibrium model, neither a gameplay one, nor a realism one, nor a computational one.

That's difficult to say, for what I know many polishing passes could have been done to produce these formulae after fixing worse issues.

Regulating consumption via market availability (rather than goods prices) has a definite advantage: you can scale the substitution pretty simply. If only considering the prices (or rather, price modifiers), a goods produced in very small quantities could see a loop of wide weekly changes (underpricing => overconsumption => overpricing => underconsumption => underpricing, etc). And since the price modifiers are only dependent on the "total supply/ total demand" ratio, it does make sense to get rid of the prices altogether.

Also, I don't think the system *has* to go for homogeneous pricing. It would definitely satisfy my fussy nature, but it's not like a real market would optimize around a singular food needs/price ratio either. I'm quite okay with design choices that lead to suboptimum situations. Especially if it leads to relatively simple calculations (the whole algorithm do need to be run for each need category in each state)

That being said, I think there are indeed some points that do need fixing. Early game whaling suffering from low oil prices is one of these. There is also the fact that consumption taxes have zero influence on substitution: consumers don't try to avoid taxed goods in favor of untaxed ones (they'll just get poorer and lower their SOL until they're able to afford their needs)

 

[Blackpony]

Give higher sol pops and not discriminated pops consumption priority and release the market.
Mechanical it's just calculating from top down, with the price getting higher. The poor will not get fair prices for food and need to choose suboptimal, expensive goods (like wood instead of oil, if the rich bought all the value goods like oil or Coal and maybe not even real clothing.
Result:
Poor pops consume the more basic stuff for potentially inflated prices, society is even more split.
Suddenly the poor hurdled masses are a thing. Starve, move away or rebel for bread, while the rich eat cake.

The game wants to be a good guy in a bad time sometimes.
Edit:
Discriminating people might be the way up in the literal food chain for the poor, in case the petite bourgeoise wanna start a movement in dire times.....

 

[yurcick]

Give higher sol pops and not discriminated pops consumption priority and release the market.
Mechanical it's just calculating from top down, with the price getting higher. The poor will not get fair prices for food and need to choose suboptimal, expensive goods (like wood instead of oil, if the rich bought all the value goods like oil or Coal and maybe not even real clothing.
Result:
Poor pops consume the more basic stuff for potentially inflated prices, society is even more split.
Suddenly the poor hurdled masses are a thing. Starve, move away or rebel for bread, while the rich eat cake.

The game wants to be a good guy in a bad time sometimes.

I don't think the rich are rich because they underpay for commodities, tbh.
If anything, they overpay. It's the fact that they have much more money that leads to their level of consumption, not the fact that they can pressure the sellers to sell to them lower than to sell to others (at least directly and locally).

 

[Blackpony]

I don't think the rich are rich because they underpay for commodities, tbh.

They are rich because they get more income
The logic of first buy is indeed not always fitting for the real life, but V3 rates quality as a quantity. The rich pops buy more to (among increased quantity) represent them buying better goods.
This behavior lead to them having an actual choice in consumption, while poor pops buy whatever is available and they can pay for, mostly in daily bases.
This is what the above mentioned mechanic wants to represent.
With rising sol they get to choose and have a say as a real customer.
In short:
Having more money enables being a customer, not a consumer.

Background history lore dump:
This is one of the reasons, why the Victorian age saw the criminal use of chalk, etc in bread for the poor, since flour was expensive. The poor people had to buy it and there was no way to not eat ur daily bread.
With rising sol, technology and government interventions (hello corn laws) things got better and soon canned and prepackaged foods started to boom. Why? The producer could guarante the quality, since it was "sealed".
The rich people did not change diets this much. The always had access to the choice of the consumer and could buy to fair conditions. They might spend way more per capita than a worker, but they got a fair price.
In times of short supply, it always were the poor who got the leftoversor nothing, the rich could at least get something.

 

[yurcick]

You basically suggest to get rid of the price-related shortage mechanism. I'm sorry, but the mechanics as they are (where mismatch between the production and consumption translates into higher price which means the poor are able to afford less) makes more sense to me.

 

[mursolini]

My guess is it never will be fixed, because doing a proper market simulation is too expensive.

Would it be? It's not a huge change.

If people want to know the details on goods substitution, the algorithm follows the current rules:
- available supplies involved in a need are estimated, following the formula "(sell orders - (exports + industry usage)/2) * good base price
- a "goods share" is calculated for each good by doing "5 * (goods weight in that need) * (goods availability)/(sum of each involved goods availability)". The result is clipped between a min & a max (see https://vic3.paradoxwikis.com/Needs#Need_types for weight / min / max values, or directly in the game file since the wiki's min/max currently use an irrelevant "%" unit)
- when determining how much of a goods a population will buy for a given need, the formula is "(need value at current SOL for pop) * (goods share) / (sum of involved goods shares) / (goods base price).

That factor of 2 is strange, what is it's intension, to force consumers and industry to compete for goods?

 

[Sbrubbles]

If people want to know the details on goods substitution, the algorithm follows the current rules:
- available supplies involved in a need are estimated, following the formula "(sell orders - (exports + industry usage)/2) * good base price
- a "goods share" is calculated for each good by doing "5 * (goods weight in that need) * (goods availability)/(sum of each involved goods availability)". The result is clipped between a min & a max (see https://vic3.paradoxwikis.com/Needs#Need_types for weight / min / max values, or directly in the game file since the wiki's min/max currently use an irrelevant "%" unit)
- when determining how much of a goods a population will buy for a given need, the formula is "(need value at current SOL for pop) * (goods share) / (sum of involved goods shares) / (goods base price).

Note that there is some inertia in goods share changes (impacted by some values present in defines), so the "goods share" used in the 3rd formula will not be equal to the one calculated in the 2nd formula if the economy is too far from an equilibrium (basically, you may want to wait a a couple weeks/months after switching PMs nationwide, or having any other large impact on your economy)


The idea of considering availably supply for determining goods consumption is okay on paper, since it's part of determining the price modifier for goods. But with the current implementation / resource usages, it has several obstacles if you're expecting prices to align for goods in a single need:
- the estimation/impact of goods used by industry / exports just doesn't work. In an economy where supply roughly equals demand, the 1/2 factor will cause available goods to be overestimated. And depending whether the industry/exports parts represents 5% or 95% of the total usage will cause wildly different impact on pricing: trying to consume double the available supply has a marginal impact on a goods mostly used by industry (since pop consumption is minor), but will cause prices to break the roof for goods exclusively used in pop consumption
- available supplies are overstimated when a goods is used in multiple needs (usage in other needs isn't even considered in the first formula)
- the notion of weights goes irectly against the idea of aligning prices. Having a small weight (such as the extreme 0.05 weight for services in good arts) basically means the pop don't want to buy that goods. This explains why fine arts is so expensive: because pops really avoid buying services for satisfying arts. Unless there are no fine arts.
- and of course, the min/max values would also prevent prices from aligning, but this part could be considered to be as designed

It means that in simple cases such as luxury items or luxury drinks, which don't share any goods with another needs nor with the industry, the goods substitution will usually works fine and leads to homogeneous price modifiers (unless exporting or hitting a min/max share value).
However, a goods such as wood will be overconsummed by pops, since its heavy industry usage is underestimated, and both heating & crude items needs are fighting over this resource. Fabric is in a similar situation and coal can also rely on a heavy industry usage, but oil has no such advantage to overinflate its share among heating goods, which ends up falling behind & cause low prices until it's finally used by industry.


Source: my own observations of the machine code, though a thread on reddit described a similar working via empirical observation of the game / define values

This is all so confusing to me. I can't help but wonder why they didn't implement something simpler that depends on just price difference with an inertia mechanic.

 

[Jimboy93]

So am I understanding this correctly? If the lower class has the choice between wood and oil, they will not make their decision based on the price of the goods, but always buy the same proportion of wood and oil and consequently pay significantly more even though there is a cheaper alternative?

 

[Kimberly]

You basically suggest to get rid of the price-related shortage mechanism. I'm sorry, but the mechanics as they are (where mismatch between the production and consumption translates into higher price which means the poor are able to afford less) makes more sense to me.

Shortage effects already don't apply to pop consumption. For example if you have maximum shortage of wheat, it harms industries that consume wheat but pops don't care at all. Only the price has an effect on their standard of living, and only the standard of living has any feedback effect on the pop. Hence why sufficiently high-income countries can have a good standard of living with no food at all.

 

[yurcick]

So am I understanding this correctly? If the lower class has the choice between wood and oil, they will not make their decision based on the price of the goods, but always buy the same proportion of wood and oil and consequently pay significantly more even though there is a cheaper alternative?

This is mostly correct.
It's not true that the proportion would be the same. However, it really isn't based on price, and thus the pops would pay significantly more than they could.

It can be argued that supply share (on which, with a certain shift, the proportion is based) has certain inverse correspondence with good price, so in some cases the outcomes are close, but this is far from universal, and sometimes (as in whale oil example) outright false.

 

[Secret Master]

Stop doing this.

Every thread where somebody points out "hey this system is either completely broken or terribly designed", you get somebody writing up elaborate fanfiction to defend it.

Arguing that homes with wood stoves would require significant changes to heat with, say, electricity, isn't some kind of elaborate fan fiction. It's literally an example that applies even today, especially in rental properties where tenants are not allowed to make radical changes to the property.

I don't actually think this mechanic is trying to accomplish this with its effects, but pretending that "Homes may not have access to all possible heating methods without substantial renovation" is the same as "Picard and Counselor Troi Go Out on a Date During Shore Leave Between Seasons 1 and 2 of Star Trek: The Next Generation" is nonsensical.

(Knowing my luck, someone here is probably going to write that fic. :sigh: )

i had similar issue with the game in my first playthroughs - it bothered me that i had a oversupply of meat, and undersupply of wheat, yet pops wouldnt consume more meat to fill their need of wheat.

You just need more meat and even less other stuff.

I don't have the save anymore, but I did a run as Russia where I didn't grow or import any grain and spammed livestock ranches. Everyone was eating meat, fruit (from the colonial empire), and fish. And with no grain, I didn't have a domestic liquor industry, which I fixed by spamming opium plantations in Central Asia and growing tobacco along the Black Sea.

This was the game that taught me just how robust goods substitution could be. You can get up to so some shenanigans if you put your mind to it.

It is a problematic mechanic in some ways, though. As outlined by some folks, there are issues in how its calculated.

 

[yurcick]

(disclaimer: the following text is written as an improvement over the original proposition to change the needs fulfillment to the price equilibrium model, as it is seen as unavoidable. The text implies that pops already optimize their purchasing behaviour to minimize costs per need fulfillment, which is not how the game works right now)

After a lot of thinking on this problematics, I now believe that goods substitution coefficients (like 2:1 for oil and wood for the heating needs in the current implementation, see original post) should not always be the same and should be untied from price ranges (now the ranges correspond exactly like the need fulfillment coefficients).

More than that, cultural preferences in consumption should be reflected not solely through obsessions and taboos, but mostly through having different substitution coefficients. For example, for the French wine should be much more preferable to tea as a luxury drink, with it being 2 times more effective than coffee and 5 times more effective than tea in fulfilling the need.
The coefficients should be recalculated yearly to reflect changes in consumer behaviour, with higher coefficients than for the previous year given to goods with higher consumption.

Obsessions, in their turn, should modify the needs (so the French need more luxury drinks in their consumer basket compared to other goods), and not directly modify the types of goods suitable for fulfilling this need. The desirable type of good bias would naturally come as a result of different preference coefficients.
Alternatively (for lore reasons, as it makes sense that the French would be obsessed with exactly wine, and not "luxury drinking"), obsessions could still correspond to individual goods, and additionally to boosting the need share in the consumer basket, they could add bias in yearly preference coefficients calculations.

What would this bring, on the positive side?
The countries with different cultures (and thus different consumption patterns) would play significantly different and have different effective industry composition, with naturally different demand for various goods. This is a very important outcome, as the problem of "every country feeling the same" is arguably the biggest one in the game at the moment, and I'm really not a fan of hardcoded solutions like "let's add 20 events for each major in which real-life names would be used".
It would also eliminate the ugly price locks, when for two goods, both used at the time to fulfill a pop need, the price modifiers must be the same (e.g. +34%), which would stem from price optimization for the consumer basket.
Finally, it would bring some inertia to consumer behaviour changes, quite expected in some cases (like the heating which we discussed here a lot), while still applying economic pressure on pops to switch to more reasonable consumption.

What problems would it bring?
First, this would make computations a bit more complex. At a glance, the difference isn't major though.
Second, the price lock removal would mean that it might be more effective for a pop to fulfill a huge portion of a need with goods in severe deficit, basically non-existant. The prices are capped at +75%, so if a good with base price of 100 at +75% cap (so 175) is twice as effective as a good with base price of 80, the cost per second good can not raise over 88,5 (+10%) while fulfilling consumer demand. It can only raise higher if the industry demand is even higher (so pops would stop buying the latter good). If the former good is in fact in huge shortage, this creates an undesirable outcome not present in current game or in basic price-lock option, when pops could buy goods in reasonable supply, but instead they buy what isn't there (as for pops there aren't, and probably shouldn't be, shortage penalties).

How to deal with this problem?
There's always an option to ignore it. +75% prices shouldn't be the norm anyway, and with, hopefully, economic system improvement they should become more and more rare for most goods (all goods except for those experiencing short-term technology-induced demand spike when supply didn't have time to build up yet).
Alternatively, the possible price ranges could be broadened (like -75% to +300%, so four-times difference in either direction), although it would demand a ton of balance work to ensure that industries are still able to recover from death spirals.
Ultimately, the pops could be made to stop buying the goods further than max price if some (even ineffective) substitution is available. However, this would make the system much less transparent, will again kill the effective price minimization and would probably be even more complex in calculation.

Please share what you think on this.

 

[Kaerid]

That factor of 2 is strange, what is it's intension, to force consumers and industry to compete for goods?

No idea why it has been implemented but functionally it does mitigate that other issue, where consumption impact on price depends on relative sizes of consumption vs industry+export.

When industry/export already uses 95% of total sell orders, consumers can buy much more than the "remaining" 5% before prices go up significantly.

This is all so confusing to me. I can't help but wonder why they didn't implement something simpler that depends on just price difference with an inertia mechanic.

Again, I think it's a question of how rational you want your market to be. Some non-optimal situations can be passed as consumer quirks, preventing the market game from having simple solutions. I think there is sometime to lose by streamlining the computation too much.

[...]

Please share what you think on this.

Without following you too far into this idea, I really like the notions of:
- having a factor representing how "efficient" a goods fulfills a need, so that you don't always need 100£ worth of goods for satisfying a 100£ need (functionally, it would probably replace the current weight system)
- having external factors dynamically impact this factor (events like cold years or market crazes, techs, taxes/laws, etc)

 

[Haresus]

Arguing that homes with wood stoves would require significant changes to heat with, say, electricity, isn't some kind of elaborate fan fiction. It's literally an example that applies even today, especially in rental properties where tenants are not allowed to make radical changes to the property.

It is fanfiction in Victoria 3. Vicky3 pops do not keep track of what type of stove they use. There is no major difference between heating needs and, say, food needs or intoxicant needs in terms of pop purchasing behavior (in Victoria 3!).

If Paradox ever decides to add some inflexibility to heating specifically, great! But that should work both ways: Pops with access to local oil should use that instead of wood, and if oil becomes expensive then they would be slow to switch over to alternative heating. But none of that exists yet, so it's strange to use non-existing mechanics and interactions as an argument for why the current system is good.