How random is this game can be?

[TotalFootballFan9]

Heh, I see that many people are obsessed with tactics testing here and for some reasons they think that you may not know anything about tactics and how ME works but still you can find/make the best tactic just testing it for about 50 matches with "Instant Result" spending 20-30 minutes on it, of course, if you have a good PC.

I really hope that this thread will help people to understand how much effort must be put into a testing in order to make it's worth anything and I also hope it'll prevent people from spending their time and putting their effort into something totally useless.



I think I can say without risk that I've got almost a perfect tactic testing system that only can be achieved, it perfectly simulates the real game environment and the randomness in this testing system is minimal that can only be achieved, If you interested how to create such testing system then you can find some tips in the thread that I made recently ---> http://www.fm-base.co.uk/forum/shar...ate-best-tactics-testing-system-possible.html

I won't list all features that this testing system has and I'll just highlight some key of them:


AI managers and you can't be fired until the end of testing and AI teams keep fight until the last math. ​

The testing league was designed such way to prevent the AI mangers and you from being fired until the end of testing.

Any other testing league that doesn't have such configuration has a huge issue because it inevitable happnes that some of AI teams start to underachieve already somewhere at the middle of the season and their managers get fired and it means that these teams change their formations and the way they played and it brings great randomness into the testing.

All AI teams use the same starting eleven for all matches in the testing league.​

All AI teams in the league have their best starting elevens "cloned" and the position of the players from the starting eleven were tweaked that way to prevent such situations when AI picking Messi for all attacking positions and leaving Suarez and Neymar at the bench.

All that greatly reduces randomness when AI teams use the same starting eleven for all matches.

The testing team consists of pre-made players that are identical and equally good. ​

Having pre-made players that are identical and equally good gives guaranty that when you test different formations and different approaches no one of them gets advantage because your testing team has more suited and better players for it and therefore it shows better testing results.

FMRTE is used to "freeze" the fitness and the morale of all players in all teams in the testing league.​

In the testing league the teams play with the same players that have 100% morale and fitness before every match and it greatly reduces randomness.

LET'S TEST!​

Ok, now let's test Ultimate V3 for 4 seasons(just re-play the 1st season) for total 174 matches​

SEASON 1

PTS = 83; SCORED = 82; CONCEDED = 42;

SEASON 2

PTS = 74; SCORED = 77; CONCEDED = 62;

SEASON 3

PTS = 62; SCORED = 60; CONCEDED = 53;

SEASON 4

PTS = 80; SCORED = 82; CONCEDED = 61;

HOW RANDOM IT CAN BE?​

if we take Season 1 and Season 3 and compare them then we see this difference:

PTS: 83 vs 62 = 24% difference

SCORED: 82 vs 60 = 27% difference

CONCEDED: 42 vs 53 = 21% difference


As you can see testing results might be different up to 25% if we test only for 50 matches and please notice that we are having almost a perfect testing system here and now image how random it would be on any other testing system which doesn't have the features that I listed above and which uses real players and AI teams uses different starting elevens every match and all players have different morale and conditions every match.

Also, you can see that the minimal amount of matches you need to test even if you use "a perfect" testing system is about 200 matches and now image how many matches you need to test if you use any "regular" testing league that consists of real players and you don't use FMRTE to freeze the morale and conditions? I think it would require testing for about 1000 matches to get more or less cleat picture.

Now you can go at any tactic testing thread on this forum and check their testing tables and see for how much the difference between the tactics that are at the top of the table and bottom of the table and see for how many matches the tactics were tested.

CONCLUSION ​

The amount of matches that you need to test tactics in order to get more or less clear picture about their performance depends greatly on "quality" of your testing system and for some testing systems it might require testing about 500 matches and for other testing system 1000 and even more but as you can see that the minimal amount of matches is about 200 matches even for "a perfect" testing system and even if you use FMRTE to freeze the morale and condition of all players in all teams but one thing you can say for sure that any testing that is less than 200 matches is waste of time and if you aren't ready to test more than 200 matches then you'd better don't start to test at all because there won't be any use of your results and 200 matches is the minimal for "a perfect testing" system and for any other testign system it easily might be 600 matches and even more.

The only in one case you don't have any randomness in you testing league on 50-60 matches distance or it's very small is when your league is very poorly configured and it means that your testing team is much stronger than any AI teams in the league or much weaker than any AI team in the league, speaking other words, let's take Barcelona and put it in any non-professional league with teams that consist of amateur players, of course, in this case Barcelona would win 100% of matches and we won’t see any randomness at all but what if we take a match where Barcelona plays vs Real Madrid on a neutral pitch so the result of such match would be very random and I’d say 50%/50% and the best what a very good tactic can do is change the probability to 30%/70% but not more.

 

[dginola]

Hey TFF mate could you share your latest test save? I don't really mind testing 200 matches

 

[Jesseh]

TFF,

I tested 2 seasons, 90 matches each season, the same tactic on my league, results are:

HOME/AWAY/COMBINED
PT124F146A44/PT70F100A83/PT194F246A127-gd119
PT126F154A36/PT659590/PT191249126-gd123

somewhat close for each block of 90 matches (1,57% dif on pts), stats varied around 1,2% also.

It doesn`t seem completely useless

 

[FM14SERBIA]

Haha yes... i have tested your Ultimate V3 with your Testing Database V2... differense is huge

107pts
98pts
111pts

Verry difficult to create a normal one Testing Database

 

[TotalFootballFan9]

TFF,

I tested 2 seasons, 90 matches each season, the same tactic on my league, results are:

HOME/AWAY/COMBINED
PT124F146A44/PT70F100A83/PT194F246A127-gd119
PT126F154A36/PT659590/PT191249126-gd123

somewhat close for each block of 90 matches (1,57% dif on pts), stats varied around 1,2% also.

It doesn`t seem completely useless

Mate, I've had a quick look at your save and after I saw what you did with the CA/PA and the attributes of all players in the league I stopped look further because it was just terrifying. I suggest that you learn a bit how CA interacts with attributes.

If you struggle to figure out it by yourself then tell me and I'll explain you.

Btw, all that I said above have nothing to do with randomness in your testing league and the randomness is the next step and the first step that must to be done is to create such environment that simulates the real game environment but the environment in your save is nowhere near with the real game environment. Without the simulation of the real game environment any testing doesn't have any use regardless of randomness in the league.

Hey TFF mate could you share your latest test save? I don't really mind testing 200 matches

I'm planing to do that.

 

[Jesseh]

Mate, I've had a quick look at your save and after I saw what you did with the CA/PA and the attributes of all players in the league I stopped look further because it was just terrifying. I suggest that you learn a bit how CA interacts with attributes.

If you struggle to figure out it by yourself then tell me and I'll explain you.

Btw, all that I said above have nothing to do with randomness in your testing league and the randomness is the next step and the first step that must to be done is to create such environment that simulates the real game environment but the environment in your save is nowhere near with the real game environment. Without the simulation of the real game environment any testing doesn't have any use regardless of randomness in the league.




I'm planing to do that.

TFF, if the attributes are bad then yours are also, I just copies then from your league as I stated on the topic. Ca and pa wont matter as long as original attributes are frozen. I could have 1/1 or 200/200.

 

[AkzeL]

Hi tff I wouldn't mind testing could you share with us that database save?

 

[TotalFootballFan9]

TFF, if the attributes are bad then yours are also, I just copies then from your league as I stated on the topic. Ca and pa wont matter as long as original attributes are frozen. I could have 1/1 or 200/200.

Apologize... my bad, I forgot that you used FMRTE to "freeze" all players.

Anyway, the players in your testing league aren't the same as the testing league that I share some time ago because I see that you edited the weak foot rating of all positions to 20 and added some additional positions that players can play, for example, Rooney have natural rating for DC position and Gibbs have natural rating almost for all positions and 20 Determination, the full back can play ML/MR positions... all that I noticed at first look and didn't dig any further because only that puts the testing environment far away from the real environment. Also, the pre-made players in my testing team were made taking into consideration the real players in AI teams but it would make a big difference if the AI teams have the same players and in this case it requires rebalancing all attributes to simulate the real game environment if you are going to use the same players in all teams.

Again, all that I said above has nothing to do with randomness in your save and I'm 100% sure that I'll get about 10-15% difference in your save even with "frozen" morale and conditions for all players. Just show me result of any tactic you tested for one season and I'll also test it to and we'll compare our results.

It might require playing 4-5 seasons but as I said I'll achieve at least 10-15% different for the final result at 90 matches distance and when you say that you have only 1.5% difference... mate, it's totally unrealistic number, especially, when there are many issues in your league that adds greatly to randomness, only the fact that AI manager might be fired at any moment adds greatly to randomness...

 

[TotalFootballFan9]

Hi tff I wouldn't mind testing could you share with us that database save?

Sure, as soon as I get free time to make some write up for it.

 

[Jesseh]

Apologize... my bad, I forgot that you used FMRTE to "freeze" all players.

Anyway, the players in your testing league aren't the same as the testing league that I share some time ago because I see that you edited the weak foot rating of all positions to 20 and added some additional positions that players can play, for example, Rooney have natural rating for DC position and Gibbs have natural rating almost for all positions and 20 Determination, the full back can play ML/MR positions... all that I noticed at first look and didn't dig any further because only that puts the testing environment far away from the real environment. Also, the pre-made players in my testing team were made taking into consideration the real players in AI teams but it would make a big difference if the AI teams have the same players and in this case it requires rebalancing all attributes to simulate the real game environment if you are going to use the same players in all teams.

Again, all that I said above has nothing to do with randomness in your save and I'm 100% sure that I'll get about 10-15% difference in your save even with "frozen" morale and conditions for all players. Just show me result of any tactic you tested for one season and I'll also test it to and we'll compare our results.

It might require playing 4-5 seasons but as I said I'll achieve at least 10-15% different for the final result at 90 matches distance and when you say that you have only 1.5% difference... mate, it's totally unrealistic number, especially, when there are many issues in your league that adds greatly to randomness, only the fact that AI manager might be fired at any moment adds greatly to randomness...

The position thing is a unfortunately flaw that happened because of FMRTE, when changing positions familiarity it counts position 15+ as 20, then many players who had 15 for example for MC and 20 AMC would then have 20 for all attacking positions + DM+CM. I'm going to correct this when I have the time and I will check what the degree of variation of not having that. Honestly I don't think it would matter that much.

About your doubt of 1.5% point difference (194 pt x 191 pt), I have both save games on my disk that I can upload when I get home if you want...

Also, you may know that team stats somehow correlate with points. If you track your team stats for many games/seasons you may notice some pattern that can lead to an association equation. Taking that into account can indicate for example if a team got bad points on a season due to being unlucky/random, indicating if team played good/bad according to data available from many games. Tracking points only may lead to what you say, having to test 200+ matches.

Having equal attributes also has an advantage on randomness, because all matches are played under almost equal conditions. You play almost equal 90 matches, differing only on tactics and home/away on each of them.

 

[dimartino]

Instant result or simulate match IS NOT reliable. Best sure way to test is to really play the match, even in commentary mode.

 

[TotalFootballFan9]

Having equal attributes also has an advantage on randomness, because all matches are played under almost equal conditions. You play almost equal 90 matches, differing only on tactics and home/away on each of them.

Jesseh, if all teams in your testing league consist of players that have equal attributes then without doubts it reduces randomness and you might get quite consistent result because of it but the same time your testing might produce such result that would be totally useless for the real game and I can explain why...

Let's say we set all attributes of all players in all teams in our testing league to "15" so it might look that we made just a perfect testing system that only can be achieved but if we look closer then we’ll see that in this case the defenders in our testing league would have such attributes as "Off The Ball", "Finishing", "Long Shots", "Dribbling", "Acceleration" as the strikers in our testing league and the same time the strikers in our testing league would have such attributes as "Tackling", "Positioning", "Marking" and some other at the same level as the defenders in our testing league and all that creates totally unrealistic testing environment that produces such result that wouldn't have any use for the real game.

So our goal is not to make that all teams in the testing league have identical players but our goal is to make that all teams would use the same starting eleven for all matches and in general, we should try to make that the environment in our testing league simulates the real game environment as much as possible.

Btw, have you bother what approaches and formations AI teams use in your testing league? Is there balance between attacking and defensive approaches? All that has crucial importance in order to determine how useful result your testing system produces for the game the result of your testing would be and I must add that settings the weak foot rating for all position to "20" doesn’t help to achieve that.

Jesseh, I see that the only thing that you think about is reducing randomness but the thing that it isn't the only thing that we should care about and the other thing that is more important is simulation of the real game environment because without it our testing would be useless for real game and some tactics will be performing good only in our testing league and have no use in the real game.

About your doubt of 1.5% point difference (194 pt x 191 pt), I have both save games on my disk that I can upload when I get home if you want...

Mate, you don't have to prove me anything and I believe you and I would also believed you if you said that when you re-tested for second time you get the same amount of points as at the first time because I've been playing in FM for long time enough to know that absolutely everything might happens in FM BUT the same time I'm 100% sure that I'll easily get 10-15% difference in the final result in your save if I'll replay the season for 3-4 times for any tactic that you tested because I've done it million times already and in much more consistent testing system then your system.

 

[Jesseh]

You misinterpret the equal attributes thing. What I mean is that all GKs on game are equal all GK. All DC are equal all DC, just like your testing team. You can think that all teams are clones of you testing team. No attackers got good tackling or no defenders got good off the ball, etc.

Sure, it's impossible to have exact same result always, just think about the bell shaped curve in a normal distribution. There will be outliers for sure, but the probability of then happening is small.

 

[TotalFootballFan9]

You misinterpret the equal attributes thing. What I mean is that all GKs on game are equal all GK. All DC are equal all DC, just like your testing team. You can think that all teams are clones of you testing team. No attackers got good tackling or no defenders got good off the ball, etc.

Don't worry, mate... I got you right and it was just an example... and I understand that you meant all GKs are equal and all defenders are equal and so on but these pre-made players that you use were made taking into consideration the real player in AI teams and they have poor balance of attribute between positions if they are used in all teams in the testing due to many reasons that would requires wall of text to explain and when you edited the weak foot for all players to "20" and edited their position you made it even worse.

Sure, it's impossible to have exact same result always, just think about the bell shaped curve in a normal distribution. There will be outliers for sure, but the probability of then happening is small.

Mate, as I said show me any result for any tactic and I'll re-test using your save and we'll compare.

Also, if it possible attach the tactic that you tested and I'll give it a go in your same...

 

[Jesseh]

Don't worry, mate... I got you right and it was just an example... and I understand that you meant all GKs are equal and all defenders are equal and so on but these pre-made players that you use were made taking into consideration the real player in AI teams and they have poor balance of attribute between positions if they are used in all teams in the testing due to many reasons that would requires wall of text to explain and when you edited the weak foot for all players to "20" and edited their position you made it even worse.




Mate, as I said show me any result for any tactic and I'll re-test using your save and we'll compare.

Also, if it possible attach the tactic that you tested and I'll give it a go in your same...

There are results here (https://docs.google.com/spreadsheets/d/1xDip_fB8bqyVPqAe5iQ-aVmIwJcaO0_OUlXfbkV2Atw/edit#gid=0), for example of galego25/TFF tweak of sunfire with 4231DM Wide the link is http://www.fm-base.co.uk/forum/atta...-ams_a05d00d9-7d4c-43e9-9e1b-bc91b97cecfa.fmf

 

[TotalFootballFan9]

Jesseh, here's my result for galego25/TFF tweak of sunfire with 4231DM Wide in your testing save. I used FMRTE to "freeze" all players in the league.

PTS = 183; SCORED = 218; CONCEDED = 121;

Here's your best result for it according your table PTS = 194; SCORED = 246 ; CONCEDED = 127;

PTS: 183 vs 194 = 7 %

SCORED: 218 vs 246 = 12 %

CONCEDED: 121 vs 127 = 5 %


Heh I don't observe only 1.5% difference that was promised

If we test for 10 seasons then we'll get about 15% points difference between "the best" and "the worst season"

When you test only for one season then you can hit "the best" or "the worst" season... yes, the probability of it is low but there's no guaranty that it won't happen and that why you need to test for about 300 matches to exclude it.

And the minimal randomness between 2 seasons in your testing league is about 5-7% and it's not 1.5% as you think.

What's more important that you reached such low randomness in your testing league in cost of the testing environment that is far away from the real game environment which greatly reduces usefulness of the results that you are getting when testing in such environment... It isn't hard to achieve very low randomness in your testing league by creating such artificial environment but the trick is to have low randomness and environment that is close to the real game environment.

 

[Jesseh]

Ok but why some tactics consistenty performs good on all test leagues? Eg Knap midsommer 442?

Anyway, I didn't promised variation would be 1.5% always. I'm pretty sure points may vary, the randomness will still be there. How would you judge the effectiveness of a tactic,just by points and GD? What about trying to translate team stats into measureable predictions of how many points you could probably achieve? You may not agree with me but some things like tackling harder -> more yellow cards is very consistent with having more points, much more than having more possession or passes completed for example.

That's the focus I think it's worth having. One random mistake from a defender can turn your win into Draw. I plan to add all test seasons I did to the sprradsheet I have to feed the system and try to predict a more reliable points x stats correlation. Minimizing those mistake/random factors, weighting them with the points ranking. The problem is that one season input take around an hour of active typing and checking and only me doing it makes it slow.

 

[zyndar]

Just to mention, I'm still not convinced that 'Simulate Match' option shows accurate results compared to playing the match.

 

[Will1981]

Just to mention, I'm still not convinced that 'Simulate Match' option shows accurate results compared to playing the match.

I am 99% certain it does.

 

[wkdsoul]

Just to mention, I'm still not convinced that 'Simulate Match' option shows accurate results compared to playing the match.

It's about as accurate as instant result they are essentially the same function with a diff name for diff modes under the hood. Either in a full mode skin (that it isn't adapt or built for) is about 80%+ accuracy as playing