In Depth Head-to-Head tactic comparison (Lots of data and analysis)

Tk1129

Member
Joined
Apr 30, 2010
Messages
124
Reaction score
0
Points
0
My boring intro (skip if you like)


I am a relatively unknown member and I know my word probably doesn’t carry much weight around here, but ever since I first got thrashed 3-0 in FM 2012 I wanted to create a really strong tactic and win without too much hands-on managing later in the game. In a sense I like to do the heavy work upfront and reap the rewards. So I headed back to my favorite FM dedicated forums and look at what other great minds have produced. I tried several tactics and I wasn’t getting the success that they were promising. I would look at the screenshots and say: why am I not going undefeated the whole season? Why isn’t MY striker scoring 50 goals this season? Why am I not beating Barcelona 10-0 over two legs?
I smelled a little false advertisement, but I could not say anything out of respect for the people that made those tactics. Don’t get me wrong, most of them are absolutely brilliant. You will play beautiful football and score a lot of goals. You will be able to play like the historic team the authors have recreated. However, your average FMer won’t get the same results. I think that the success these people get is due to their general FM experience (man management, team-talks, ability to spot great transfer targets and the ability to adapt their tactic in a match situation). That is why I decided to test how successful the AI would be with each tactic. Also it is incredibly time consuming to play a whole season and generate a large enough data set, while keeping the number of variables down. Read the testing procedure section on how I decided to test my tactic.


The Testing procedure (skip if you like)


Well, I was inspired to do this by a_ilfandy and his tactic testing project. I wanted to improve my own tactic and I picked Koflok’s goals galore v2 (sorry koflok I didn’t have your latest version at the time) and Xenonf50’s Undefeated 4-2-2-1 v1 to use as a benchmark for my tactic and gauge it’s performance.

Now my goal was noble, I didn’t want to trash their work. In fact I respect them both a lot and they have two of the most popular and most successful tactics, which would serve as a perfect benchmark for success. I thought If my results were anywhere near what those guys got I must be on the right track.
Now like I said before playing a full season would be extremely time consuming, so I decided to holiday and let the team use the tactic I select.

I chose Arsenal to conduct the test, because they have a good depth of players across the 4-5-1 (all tactics are (4-5-1) for the same reason. It’s a very popular formation and it should keep the arguments about formation and results to a minimum.



I holidayed only until January 1st​, because I was afraid that if a tactic was underperforming I would get sacked and the whole data set becomes unusable. It means one of the tests is incomplete, but it still holds weight, because the team was doing poorly because of it.


I would do this five time for each tactic and that would be well over 100 matches. It would also mean a chance to play each team in the league at least once.
I did not make any transfers, it’s the default Arsenal team. In fact I didn’t even attend my first press conference. I imported the tactics and went on holiday right away. The match preparation was set to Average and no special focus was selected.

Basically all tactics faced the same initial conditions and the effect of injuries, bad form and so on would be kept low, because I ran the test 5 times for each tactic.
Upon completion of all the tests I took a screenshot of the team’s performance in the league and their entire fixture list. I am not interested in individual statistic, I only care if my team is winning games or not.
I then recorded several interesting statistics such as Goals scored, Goals conceded, Win %, Undefeated % (Win + Draw), Pts per game (League only) and other interesting tidbits. I only looked at competitive matches. Pre-season friendlies were not included in the calculations.
I was hoping to arrive at a conclusive result, but I’ll present my analysis and let you guys draw your conclusions.


The results (probably don’t want to skip that)

Koflok’s Goals Galore:

First test




35 points in 18 matches or 1.94pts/game
Overall Record 19-4-6 (65.5 Win Percentage)

Second test





34 points in 19 matches or 1.79 pts/game
Overall Record 16-5-7 (57.1 Win Percentage)

Third test





38 points in 18 matches or 2.11 pts/game
Overall Record 18-3-7 (64.3 Win Percentage)

Fourth test





36 points in 18 matches or 2 pts/game
Overall record 17-6-5 (60.7 Win Percentage)

Fifth test





40 points in 19 matches or 2.11 pts/game
Overall Record 21-6-3 (70 Win Percentage)

Detailed Analysis:
Over the five tests this tactic gathered 183 points in 92 matches or the equivalent of 1.99 pts/game. Over a 38 game season this puts will put the team at 75.6 points. (Best League Performance)
The Overall Record (Cup, European and Domestic games) is 91 -24- 28
The tactic’s Win Percentage is 63.6%
The Unbeaten Percenatge (Wins + Draws) is 80.4%
Over that stretch the team scored 308 goals and conceded 120
On Average that is 2.15 goals per game versus 0.84 conceded for a difference of 1.31
The Overall performance versus the big 5 clubs (Man Utd, Man City, Tottenham, Liverpool and Chelseas) was 8 -5 -14
That’s a win percentage of 29.6%
And an unbeaten percentage of 48%
The team scored 34 (however 20 of those came versus Liverpool – so use Koflok’s tactic vs Liverpool) goals and conceded 40
As far as outliers go
The club suffered 7 heavy defeats (3 goal difference) and the scores are as follows: 0-3, 1-4, 0-3, 0-3, 0-3, 0-3, 0-3.
The club recorded a total of 21 victories with a winning margin of 4 goals or more. The score are as follows: 6-0, 5-0, 4-0, 6-0, 4-0, 5-0, 5-0, 5-0, 5-1, 4-0, 6-0, 4-0, 6-0, 6-0, 5-0, 4-0, 5-0, 4-0, 4-0, 4-0, 5-1.
The club also recorded 72 clean sheets.


XenonF50’s Undefeated 4-2-2-1


First test





38 points in 18 matches or 2.11 pts/game
Overall Record 22-3-4 (75.9 Win percentage)

Second test





30 points in 18 matches or 1.67 pts/game
Overall Record 14-7-6 (51.9 Win Percentage)

Third test





40 points in 19 matches or 2.11 pts/game
Overall Record 21-3-6 (70 Win Percentage)

Fourth test





40 points in 19 matches or 2.11 pts/game
Overall Record 15-7-5 (57.1 Win percentage)

Fifth test





30 points in 18 matches or 1.67 pts/game
Overall Record 15-7-7 (51.7 Win Percentage)

Detailed Analysis:
Over the five tests this tactic gathered 178 points in 92 matches or the equivalent of 1.93 pts/game. Over a 38 game season this would put the team at 73.3 points.

The Overall Record (Cup, European and Domestic games) is 88-27-28
The tactic’s win percentage is 61.5%
And the Unbeaten percentage (Wins + Draws) is 80.5%
Over that stretch the team scored 273 goals and conceded 141.
On Average that is 1.9 Goals per game versus 0.99 conceded for a difference of 0.91
The Overall performance versus the big 5 clubs (Man Utd, Man City, Tottenham, Liverpool and Chelseas) was 10-5-13. (Best Performer against Big Clubs)
That’s a win percentage of 35.7%
And an unbeaten percentage of 53.6%
The team scored 33 goals and conceded 48
As far as outliers go
The club suffered 8 heavy defeats (3 goal difference). And the results are as follows: 0-3, 0-3, 0-3, 0-3, 0-3, 0-3, 1-5, 0-3, 1-6.
The club recorded a total of 12 victories with a winning margin of 4 goals or more. The results are as follows: 4-0, 6-0, 4-0, 7-0, 4-0, 4-0, 4-0, 5-1, 6-1, 6-1, 6-0, 6-2.
The club also recorded 58 clean sheets.


TK’s Wrecking Ball:


First test





38 points in 19 matches or 2 pts/game.
Overall Record 19-2-7 (67.9 Win percentage)

Second test





37 points in 19 matches or 1.95 pts/game
Overall Record 21-4-5 (70 Win Percentage)

Third test





37 points in 19 matches or 1.95 pts/game
Overall Record 20-6-4 (66.7 Win Percentage)

Fourth test





34 points in 18 matches or 1.89 pts/game
Overall Record 17-6-5 (60.7 Win percentage)

Fifth test





38 points in 19 matches or 2 pts/game
Overall Record 19-5-6 (63.3 Win percentage)

Detailed Analysis:
Over the five tests this tactic gathered 184 points in 94 matches or the equivalent of 1.96 pts/game. Over a 38 game season this would put the team at 74.4 points.
The Overall Record (Cup, European and Domestic) is 96-23-27
The tactic’s Win Percentage is 65.7% (Best Win Percentage)
And the Unbeaten percentage (Draws + Wins) is 81.5% (Best Unbeaten Percentage)
Over that stretch the team scored 310 goals and conceded 127
On Average that is 2.12 goals per game versus 0.87 conceded for a difference of 1.25
The Overall performance versus the big 5 clubs (Man Utd, Man City, Tottenham, Liverpool and Chelseas) was 8-6-12
That’s a win percentage of 30.8 %
And an unbeaten percentage of 53.8%
The team scored 29 goals and conceded 47
As far as outliers go
The club suffered 6 heavy defeats (3 goal difference). And the results are as follows 0-4, 1-4, 0-3, 0-3, 0-4, 0-4.
The club recorded a total of 15 victories with a winning margin of 4 goals or more. The results are as follows: 4-0, 6-0, 7-0, 5-0, 4-0, 5-0, 5-1, 4-0, 5-0, 4-0, 4-0, 5-1, 6-0, 6-0, 4-0.
The club also recorded 70 clean sheets.

Final Thoughs:
As you can see the results are very close, but you can see from the scores that the different tactics have different impacts of the type of results the team gets. One offers more balance between scoring and defending, but when they lose they lose heavy. One offers a great scoring punch, but can be hot and cold also. One offers more protection vs the big clubs, but can be either great or not-so-great at times.
If anybody would like to continue this testing or has any questions about how I was testing this PM me or leave a comment and I’ll upload all the files and settings I used.
I tried to be as impartial as I can be even though my own tactic was being tested as well. I want to reiterate that I was simply benchmarking against two great tactics and not trying to embarrass anybody or prove conclusively that one way is better than another.
 
Last edited:
Sorry man, but results on holiday are totally random...your assistant manager changes tactics during games... so the only way to properly test tactics is by playing the games yourself.
 
Sorry man, but results on holiday are totally random...your assistant manager changes tactics during games... so the only way to properly test tactics is by playing the games yourself.

Actually you're wrong. They're not random. They are very much affected by the tactic. And the ***-man does stick to the tactic.
Here is a quick test I did with a simple defensive tactic. I don't need to run it five times, because the results can be seen right away.





Some quick numbers:

34 points in 19 games or 1.79 pts/game - (it is Arsenal after all. With a squad like that they're going to pick up points no matter what)
But look at their league record. 8-10-1

Overall record is 14-11-4

Win percentage - 48% (dramatically different)
Unbeaten percentage - 86% (tough to beat, BECAUSE they're playing defensively now)

Goals scored - 43 or 1.48 goals per game
Goals conceded - 21 or 0.72 goals per game

The most stark contrast in the results vs the other 5 big clubs
0-5-2 with only chelsea and man city narrowly defeating them 1-2.
They're much tougher to beat across the board and are 72% unbeaten vs big clubs.

It's clear nobody cares what I have to say on this forum, but it still is quite insulting to dismiss 2 days work in six seconds.
 
Actually you're wrong. They're not random. They are very much affected by the tactic. And the ***-man does stick to the tactic.
Here is a quick test I did with a simple defensive tactic. I don't need to run it five times, because the results can be seen right away.





Some quick numbers:

34 points in 19 games or 1.79 pts/game - (it is Arsenal after all. With a squad like that they're going to pick up points no matter what)
But look at their league record. 8-10-1

Overall record is 14-11-4

Win percentage - 48% (dramatically different)
Unbeaten percentage - 86% (tough to beat, BECAUSE they're playing defensively now)

Goals scored - 43 or 1.48 goals per game
Goals conceded - 21 or 0.72 goals per game

The most stark contrast in the results vs the other 5 big clubs
0-5-2 with only chelsea and man city narrowly defeating them 1-2.
They're much tougher to beat across the board and are 72% unbeaten vs big clubs.

It's clear nobody cares what I have to say on this forum, but it still is quite insulting to dismiss 2 days work in six seconds.

I'm sorry, didn't mean to insult you, but I stand to what I said, because I have tested tactics on holiday myself, and found out that it is worthless. I tested some highly rated tactics, including the one I used on several saves and knew that was amazing, and one **** tactic designed just for the purpose of testing. On holiday mode the **** tactic got better results than the real tactic, yet when playing myself the **** tactic was ****, and the other was amazing. And with other tactics I tested the results were simply random.

And the assistant does change tactics during games, in FM12 you can even see that on the team report.

I'm sure that most users in this and other forums will agree that you can't draw any conclusions about tactics in holiday mode.
 
i am also trying kaflof v3 and xenons undefeated juve tactic .. still not sure which is better..

seems like u put alot of effort into testing not gonna say going hols gives random results but i think u can get better results by actually playing those 18 19 games. dunno maybe am wrong.

lookin forward to ur final thoughts about these 2 tactics.. good luck
 
i am also trying kaflof v3 and xenons undefeated juve tactic .. still not sure which is better..

seems like u put alot of effort into testing not gonna say going hols gives random results but i think u can get better results by actually playing those 18 19 games. dunno maybe am wrong.

lookin forward to ur final thoughts about these 2 tactics.. good luck

You're absolutely right. You get better results by actually playing the games. But I'm trying to take the user out of the equation, because koflok is going to get much better results using his own tactic than anyone that just downloads it.

I still can't think of a better way to do that than to holiday the game. Ideally I would want to run the tests like a thousand times, so that there is a lot less variation and the results can be more conclusive but it takes a lot of time for my pc to process the games.

Even with 1 league loaded and small database it still takes close to 30 minutes to do each test.
 
I salute you sir for the effort in doing these tests.

While I myself haven't really understand the effectiveness of testing with 'go on holiday', still, if looking from the results you can see a pattern and little variance then it is probable that the method is correct.
If you're sure what you're doing is correct then just screw the people who 'underestimate' your way of testing.

Again awesome job on the in depth statistics. It's one thing I haven't really seen before so yeah it's nice to discover new analysis. Also, love seeing how each tactic perform its own way.

And thank you for using my tactic as one of the initial ones to be tested. Cheers.
 
I salute you sir for the effort in doing these tests.

While I myself haven't really understand the effectiveness of testing with 'go on holiday', still, if looking from the results you can see a pattern and little variance then it is probable that the method is correct.
If you're sure what you're doing is correct then just screw the people who 'underestimate' your way of testing.

Again awesome job on the in depth statistics. It's one thing I haven't really seen before so yeah it's nice to discover new analysis. Also, love seeing how each tactic perform its own way.

And thank you for using my tactic as one of the initial ones to be tested. Cheers.

NP. I respect your work a lot. I modeled my front 3 after yours.

In terms of testing the tactics objectively. It is near impossible. I am not getting paid for this and the amount of time that would have to be put to arrive at something that is even remotely conclusive is paramount.

In match game engine would certainly have some different nuances, but I don't think most people here want an exploit tactic that just finds a loophole in the system to exploit. I think when games are simmed the math behind it would be largely based on the same principles that the match engine is. After all SI wants consistency in their results.

Like I said in my OP, ideally i would want a thousand tests, just so the standard deviation is established and I can check for consistency, but who has the time to do that.

In fact the ideal scenario would be a user playing each tactic for a 100 seasons, while using the same shouts, team-talks and line-up the entire time. Now that is practically impossible. So I figure if the correlation between the in-game match engine and the sim-engine (for lack of a better name) is indeed strong then this is the most practical test available.

Of course i concede that a user would do better than the AI with a certain tactic (like for example I know how to tweak my tactic and team talks to chase a lead, or to protect a narrow lead in the final minutes) but the user introduces a huge number of variables that make any type of objective test data unusable.

When i get some more time on my hands I'll test some other tactics.
 
a great deal of time and effort has gone into this and I think its fantastic, there are a few factors that affect tactics/results from going on holiday, but as you said above which I cannot agree more with:

However, your average FMer won’t get the same results. I think that the success these people get is due to their general FM experience (man management, team-talks, ability to spot great transfer targets and the ability to adapt their tactic in a match situation)

that is 100% correct! and members do need to understand this, I always say, its not only the tactic that makes a good team and gets results, just because I was on top of the league with QPR after about 14 games etc does not mean that just because you are using QPR using the same tactic you WILL get the same results ;)

so well done sir!

I would really appreciate if you could do the tests on these tactics? the M&M wont work that well with Arsenal and the S&S tactic should only be used at home on a standard pitch, but just for ***** and giggles if you have the time I would really like to see the results :) no pressure whatsoever either mate!


http://www.fm-base.co.uk/forum/foot...cs-training/84096-pure-perfection%99-ccc.html

http://www.fm-base.co.uk/forum/foot...ning/86705-compact-impact%99-raikan007-6.html

http://www.fm-base.co.uk/forum/foot...636-might-magic%99-raikan007.html#post1343063

http://www.fm-base.co.uk/forum/football-manager-2012-tactics-training/85231-short-sweet%99.html

for the M&M tactic, for Arsenal, I would suggest using the away counter tactic as the standard one being tested!

thank you for your time and effort :)
 
I would really appreciate if you could do the tests on these tactics? the M&M wont work that well with Arsenal and the S&S tactic should only be used at home on a standard pitch, but just for ***** and giggles if you have the time I would really like to see the results :) no pressure whatsoever either mate!


http://www.fm-base.co.uk/forum/football-manager-2012-tactics-training/84096-pure-perfection%99-ccc.html

http://www.fm-base.co.uk/forum/foot...ning/86705-compact-impact%99-raikan007-6.html

http://www.fm-base.co.uk/forum/foot...636-might-magic%99-raikan007.html#post1343063

http://www.fm-base.co.uk/forum/football-manager-2012-tactics-training/85231-short-sweet%99.html

for the M&M tactic, for Arsenal, I would suggest using the away counter tactic as the standard one being tested!

thank you for your time and effort :)

Thanks mate, I may have some time coming up this week. I'll be playing skyrim most of the time |) so i'll just let some tests run on the computer since i won't be needing it.
 
I would really appreciate if you could do the tests on these tactics? the M&M wont work that well with Arsenal and the S&S tactic should only be used at home on a standard pitch, but just for ***** and giggles if you have the time I would really like to see the results :) no pressure whatsoever either mate!


http://www.fm-base.co.uk/forum/foot...cs-training/84096-pure-perfection%99-ccc.html

http://www.fm-base.co.uk/forum/foot...ning/86705-compact-impact%99-raikan007-6.html

http://www.fm-base.co.uk/forum/foot...636-might-magic%99-raikan007.html#post1343063

http://www.fm-base.co.uk/forum/football-manager-2012-tactics-training/85231-short-sweet%99.html

for the M&M tactic, for Arsenal, I would suggest using the away counter tactic as the standard one being tested!

thank you for your time and effort :)

Thanks mate, I may have some time coming up this week. I'll be playing skyrim most of the time |) so i'll just let some tests run on the computer since i won't be needing it.
 
Top