Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Discuss tactical and strategic development for 40K/Tau.
Muaddib195
Shas
Posts: 10

Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#1 » Nov 09 2017 05:54

TL;DR: Mathhammer does not accurately describe the most likely outcome for low burst weapons that deal random damage.

Background:
Since 8e introduced random damage for certain weapons, I have seen a number of posts claiming how unreliable certain weapons have become. Most people tend to fall back on mathhammer calculations when attempting to determine the effectiveness of a particular weapon, but I became suspicious that mathhammer was not accurately representing the actual outcomes on the game table.

What is Mathhammer?
In statistics terms, “mathhammer” calculates the average (arithmetic mean) damage of a particular shooting attack by multiplying the probability (P) each event in the shooting sequence. The general procedure is:

(# of shots) x (P to hit) x (P to wound) x (P to penetrate armor) x (Damage per shot) = Average damage

For a Firewarrior (BS 4+) shooting a pulse carbine (Assault 2, Str 5, AP 0) at a Guardsman (T3, Sv 5+), the calculation would be:

(2) x (3/6) x (4/6) x (4/6) x (1) = 0.44


What is the Problem?

The issue with using the arithmetic mean only occurs when we introduce random damage (and/or random shots). The arithmetic mean requires us to calculate the “average” of a dice roll random roll. For D6 damage, the average is usually calculated as (1+6)/2 = 3.5. Furthermore, for weapons that deal bonus mortal wounds on a wound roll of 6+ (such as our rail weapons), we also have to calculate the average number of mortal wounds and add this to the standard damage. Additional wounds are calculated by:

(# of shots) x (P of triggering mortal wounds) x (# of mortal wounds per trigger)

To illustrate the problem with this approach, let us consider a scenario where a Hammerhead (BS 3+) fires a railgun (Heavy 1, Str 10, AP -4) at a Leman Russ (T8, Sv 3+). This results in the following calculation:

Standard Damage: (1) x (4/6) x (4/6) x (4/6) x (3.5) = 1.037
Mortal Damage: (1) x (4/6) x (1/6) x (2) = 0.89
Total Damage: 1.037 + 0.89 = 1.93


Now, let’s simulate 10,000 shooting phases* and look at the discreet probability distribution of the damage. This figure shows the probability distribution for dealing a specific amount of damage in 1 shooting round.

Image

It is plainly visible from the figure that the probability distribution is heavily skewed. Not only that, but there is a 56% probability that the Hammerhead will deal 0 damage in a given shooting phase, which makes it suspicious that 1.93 damage is typical. In cases such as these, the mean is considered to be invalid for the purpose of estimating a “typical” outcome, as it is heavily influenced by outliers.

If Not Mean, Then What?

For skewed distributions, the median of the data is considered to be a more accurate representation of a “typical” outcome. Median is determined by sorting the data into increasing order (in this case, from 0 to 9) and then identifying the middle number of the data set. The median for this particular example is 0, which serves to emphasize the problem with the Railhead in its current configuration.

So let us look next at Longstrike to see what a more reasonable damage distribution looks like. Longstrike has better ballistic skill (BS 2+), and gains a +1 to his wound rolls which increases his probability of triggering mortal wounds from 1/6 to 2/6. Longstrike’s average damage output is 2.99, and his damage distribution is as follows:

Image

Longstrike’s distribution is still quite skewed; however, there are two probability peaks, which gives us a median of 3. Since the median and mean are the same, Longstrike feels like he has the damage output that mathhammer says he should.

No discussion about the Hammerhead would be complete without someone bringing up the lascannon Predator, so let’s go ahead and look at it. The Predator has the same ballistic skill as the Hammerhead; however, the lascannon statline is a little different (Heavy 1, Str 9, Ap -3). The Predator has the option to bring 4 Lascannons, but let’s look at a single lascannon first. One lascannon has a mean damage output of 1.3 and a median damage output of 0. The distribution is as follows:

Image

So far, the lascannon looks worse than the Railgun given that it has a 64% chance of dealing 0 damage. This doesn’t tell the whole story, so we need to look at the Predator with all 4 of its lascannons. A quad lascannon predator has a mean damage output of 5.19 and a median output of 5. Again, the mean and median line up to make the lascannon Predator feel more reliable, and even better, the distribution is closer to normal:

Image

*Monte Carlo simulation was used to generate 10,000 shooting phases. Total damage was summed for each shooting phase and the frequency of damage was counted. This frequency was used to calculate the discreet probability of achieving each damage output.

Conclusions:

Weapons with random damage and a low number of shots are pretty bad at the moment, and using mathhammer to calculate average damage output dramatically overestimates the most likely outcome from a given round of shooting. In order to improve the reliability of damage output, weapons like the railgun either need to gain additional shots or they need special benefits (such as Longstrike’s +1 to wound) to shift the probability distribution. If we gave the Hammerhead the Grinding Advance rule that the Leman Russ got, or mean damage would be 3.56 and our median output would be 3. Distribution is as follows:

Image

Let me know if there are other weapon distributions that you would like to see. I am currently working on an analysis of weapons that use a random a number of shots.

User avatar
Harkus959
Shas'Saal
Posts: 130

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#2 » Nov 09 2017 06:28

Thank you immensely for this post. It's clear that quite a bit of work went into, and it is appreciated.

I've disliked random damage weapons mostly on the basis that you could roll a one and have your massive anti-tank high tech railgun be no more effective than a lasgun (not cannon, gun). It just felt wrong, and now maths is backing that up, so I feel vindicated in my hunch.

Single shot weapons are already risky enough just due to the danger of missing and doing absolutely no damage. I want to know if I hit something, that I'm at least going to hurt it. I think a good compromise would be to inflict a certain amount of base damage, and then have a random amount added as a modifier, such as dealing 3+d3 damage.

That said, while I thought random damage was bad, I certainly didn't imagine that more than half the time we'd be doing absolutely nothing at all. That was enlightening, and slightly worrying.

But, once again, thanks for all the work that obviously went into this, and for showing that mathhammer may not be as reliable as we imagined. I'd be interested to see how this approach shows up flamers and other weapons that generate a random number of attacks/shots rather than damage and how the outcome is affected by having that bit of randomness earlier in the process.

szeszej
Shas
Posts: 33

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#3 » Nov 09 2017 06:30

This is really amazing work. Are we then perhaps overvaluing CIBs for similar reasons? I know D3 damage is much better than D6 damage in terms of reliability but getting a confirmation would be nice.

Fusion blasters probably also suffer from this but the Commander's BS of 2+ certainly helps.

User avatar
Harkus959
Shas'Saal
Posts: 130

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#4 » Nov 09 2017 06:37

szeszej wrote:This is really amazing work. Are we then perhaps overvaluing CIBs for similar reasons? I know D3 damage is much better than D6 damage in terms of reliability but getting a confirmation would be nice.

Fusion blasters probably also suffer from this but the Commander's BS of 2+ certainly helps.


I'd never really thought about how all of our most viable weapons (fusion blasters, in rifles, CIB, Missile Pods, and Railguns) all rely on variable damage.
In fact, the only thing I see talked about as competitive that doesn't really one variable damage is the gun drones swarm.

Having read Muaddib's post I'm now a little bit concerned about our reliance on random damage wargear.

User avatar
Arka0415
Shas'Ui
Shas'Ui
Posts: 1933

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#5 » Nov 09 2017 07:01

Thanks for the work Mauddib! It's always good to have some more advanced math floating around the forum.

However, I just want to point out- none of us actually think that a Railgun will do ~1.9 damage per turn. Obviously that's not true. It's simply a way of feeling out how powerful a weapon is, and what you can expect from it over dozens of games.

Your finding- that a Hammerhead's Railgun has a ~55% chance of dealing 0 damage- can be easily shown with Mathhammer too. 1*(2/3)*(2/3)=(4/9), or a ~45% chance to deal ≥1 damage, easily leads us to the conclusion that it conversely has a ~55% chance to deal 0 damage. With this in mind, it's not too difficult to imagine that every other possible damage outcome shares an equal portion of the remaining ~45%, or about ~7.5% each.

Anyway, thank you very much for the graphs though- they are excellent tools and will be great for explaining these concepts to new players! :biggrin:

Harkus959 wrote:That said, while I thought random damage was bad, I certainly didn't imagine that more than half the time we'd be doing absolutely nothing at all. That was enlightening, and slightly worrying.


Doing no damage 50% of the time has nothing to do with random damage though. In fact, it's not worrying at all. BS3+ and 3+ to wound are very strong stats, the 2nd best possibility in the game. Hitting on 3+ and wounding on 3+ result in a ~45% chance to wound. The most powerful conventional weapons in the game- Lascannons, Railguns, Brightlances, etc.- all share this ~45% characteristic. That's why 40k is a numbers game- you need to bring a whole lot of this stuff to make any impact.

Harkus959 wrote:mathhammer may not be as reliable as we imagined.

Read the top part of my post again if you're worried you've been doing the math incorrectly.

Harkus959 wrote:I'd never really thought about how all of our most viable weapons (fusion blasters, in rifles, CIB, Missile Pods, and Railguns) all rely on variable damage.

...

Having read Muaddib's post I'm now a little bit concerned about our reliance on random damage wargear.

Remember, all armies share this random-damage thing now. Non-random high-damage weapons are incredibly rare, and no army has those weapons as a primary design feature. While this may feel underpowered, remember that it's part of the game, and every army shares this flaw.

User avatar
Arka0415
Shas'Ui
Shas'Ui
Posts: 1933

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#6 » Nov 09 2017 07:31

szeszej wrote:This is really amazing work. Are we then perhaps overvaluing CIBs for similar reasons? I know D3 damage is much better than D6 damage in terms of reliability but getting a confirmation would be nice.

I agree here! While mathhammer might be a decent tool for single-shot weapons, I'd be very curious to see a chart about CIBs. Try making a distribution using these figures:

6D3 shots, BS4+ re-rolling 1s, wound 3+, save 5+, D3 damage.

That should help us see the damage output against standard T7/Sv3+ vehicles.

Muaddib195
Shas
Posts: 10

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#7 » Nov 09 2017 07:54

szeszej wrote:This is really amazing work. Are we then perhaps overvaluing CIBs for similar reasons? I know D3 damage is much better than D6 damage in terms of reliability but getting a confirmation would be nice.

Fusion blasters probably also suffer from this but the Commander's BS of 2+ certainly helps.


Glad you like the work!

The more shots you put out, the less effect you will see from the skewed distribution. I'll have to get back to you on the CIB, but I suspect that taking 2 or 3 CIBs on one suit will help compensate for both the random shots and random damage.

BS 2+ makes the Quad Fusion Commander fairly reliable, but you can still see some interesting distribution behavior, particularly once they get within melta range. The median damage outside of 9" is 5, while the median damage within 9" is 6. That said, once you get within 9" you actually have an 18.6% chance of dealing AT LEAST 12 damage to a Leman Russ.

Image

Image

Arka0415 wrote:Thanks for the work Mauddib! It's always good to have some more advanced math floating around the forum.

Your finding- that a Hammerhead's Railgun has a ~55% chance of dealing 0 damage- can be easily shown with Mathhammer too. 1*(2/3)*(2/3)=(4/9), or a ~45% chance to deal ≥1 damage, easily leads us to the conclusion that it conversely has a ~55% chance to deal 0 damage. With this in mind, it's not too difficult to imagine that every other possible damage outcome shares an equal portion of the remaining ~45%, or about ~7.5% each.

Anyway, thank you very much for the graphs though- they are excellent tools and will be great for explaining these concepts to new players! :biggrin:



Just to clarify a little here. While Mathhammer can easily calculate the probability of 0 damage, I think it is a mistake to assume that the remaining outcomes are evenly distributed, particularly in the case of weapons that deal D6 damage + D3 mortal wounds. Referencing the Hammerhead Railgun, it has an 8% chance of dealing 4 wounds, a 5.5% chance of dealing 1 wound, and a 0.5% chance of dealing 9 wounds, to illustrate a few examples.

Muaddib195
Shas
Posts: 10

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#8 » Nov 09 2017 09:31

Arka0415 wrote:
szeszej wrote:This is really amazing work. Are we then perhaps overvaluing CIBs for similar reasons? I know D3 damage is much better than D6 damage in terms of reliability but getting a confirmation would be nice.

I agree here! While mathhammer might be a decent tool for single-shot weapons, I'd be very curious to see a chart about CIBs. Try making a distribution using these figures:

6D3 shots, BS4+ re-rolling 1s, wound 3+, save 5+, D3 damage.

That should help us see the damage output against standard T7/Sv3+ vehicles.


Spoiler!
Arka, Can you confirm that the average damage for this shooting event should be 6.22?
I need to go through my work one more time to make sure I am accurately modelling the random shots and rerolled 1's, but this is my preliminary distribution model. Median damage is 4. I will update this post tomorrow if I identify any mistakes.


CORRECTION: There was an error with the random shots calculation in the first graph posted. The corrected median is 6 and the updated distribution graph is as follows:

Image
Last edited by Muaddib195 on Nov 09 2017 10:11, edited 1 time in total.

User avatar
Arka0415
Shas'Ui
Shas'Ui
Posts: 1933

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#9 » Nov 09 2017 09:31

Muaddib195 wrote:Just to clarify a little here. While Mathhammer can easily calculate the probability of 0 damage, I think it is a mistake to assume that the remaining outcomes are evenly distributed, particularly in the case of weapons that deal D6 damage + D3 mortal wounds. Referencing the Hammerhead Railgun, it has an 8% chance of dealing 4 wounds, a 5.5% chance of dealing 1 wound, and a 0.5% chance of dealing 9 wounds, to illustrate a few examples.

The outcomes remain evenly distributed in the case of conventional D6 damage weapons, was what I was getting at. The Railgun is the only weapon in the game (I'm pretty sure) with that extra damage effect, so you can practically ignore that. It basically adds a little bit of variance to the tune of ±2% for and actual values of >2% for 7-9 damage. These are so minor that in 10,000 trials they become evident, but in regular games they will disappear entirely. For the Hammerhead, we hit ~66% of our shots and wound on a further ~66% of those; ~25% of our successfully-wounding shots deal D3 damage. That should occur once every couple of games.

User avatar
Arka0415
Shas'Ui
Shas'Ui
Posts: 1933

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#10 » Nov 09 2017 09:36

Muaddib195 wrote:Arka, Can you confirm that the average damage for this shooting event should be 6.22?
I need to go through my work one more time to make sure I am accurately modelling the random shots and rerolled 1's, but this is my preliminary distribution model. Median damage is 4. I will update this post tomorrow if I identify any mistakes.

The average is 6.2, yeah! That's the number mathhammer gives you, and 6-7 damage is definitely the common range for damage againat T7/Sv3+ vehicles.

However, I'm not totally convinced by that graph. 40% chance that six multi-shot weapons will do absolutely nothing? I've played enough games of 8th Edition to have fired CIBs against dozens of Rhinos, Chimeras, and Devilfish... I don't think they've every whiffed completely. Are you sure those numbers are correct?
Last edited by Arka0415 on Nov 09 2017 11:05, edited 1 time in total.

Muaddib195
Shas
Posts: 10

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#11 » Nov 09 2017 10:12

Arka0415 wrote:
Muaddib195 wrote:Arka, Can you confirm that the average damage for this shooting event should be 6.22?
I need to go through my work one more time to make sure I am accurately modelling the random shots and rerolled 1's, but this is my preliminary distribution model. Median damage is 4. I will update this post tomorrow if I identify any mistakes.

Image

The average is 6.2, yeah! That's the number mathhammer gives you, and 6-7 damage is definitely the common range for damage againat T7/Sv3+ vehicles.

However, I'm not totally convinced by that graph. 40% chance that six multi-shot weapons will do absolutely nothing? I've played enough games of 8th Edition to have fired CIBs against dozens of Rhinos, Chimeras, and Devilfish... I don't think they've every whiffed completely. Are you sure those numbers are correct?


Arka, you are correct. I found an error in the random shot distribution and have updated my post above. The corrected distribution chart looks much more reasonable now.

Jacket
Shas'Saal
Posts: 368

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#12 » Nov 09 2017 10:26

FFG solved their 40k rpg line's problem by adding in minimum damage on some weapons or even weapon firing modes. Actually they seemed to have solved much trouble with 40k rules in general that GW still is catching up to. This made it so that other weapons were actually useful again and made weapon choice important. At first when they simply ported from 40k to their rpg the bolter was the superior gun by far.

Waiting for GW to catch up as usual. FFG has also made the Deathwatch army I love a thing that GW then ripped off for their army.

So for example a gun would roll a d10 and then have the minimum (2) rule so that way if you brought a las cannon it was guaranteed to still pack some punch with a bad roll instead of being purely outdone by bolters and plasma.

User avatar
Kael'yn
Fio'Ui
Fio'Ui
Posts: 1092

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#13 » Nov 10 2017 01:12

Extensive and clear analysis.
Back in the 6th edition, I've done too some statistical tools to extend the simpler average result to distribution:
http://www.advancedtautactica.com/viewtopic.php?f=52&t=20052
Outdated since, I have too little time to put it for 8th ed but if someone want to do another I can help.

Average result is still a good thing to compare weapons, but for high number of shots. For fewer shots (or very random shots), distribution is better to help making a decision IMHO.

User avatar
Arka0415
Shas'Ui
Shas'Ui
Posts: 1933

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#14 » Nov 10 2017 02:54

Muaddib195 wrote:Arka, you are correct. I found an error in the random shot distribution and have updated my post above. The corrected distribution chart looks much more reasonable now.

Definitely, that looks accurate. I'd like to try a few of these myself, what software do you recommend?

Muaddib195
Shas
Posts: 10

Re: Trouble with Mathhammer: Mean, Median, and Skewed Distributions

Post#15 » Nov 10 2017 07:03

Arka0415 wrote:
Muaddib195 wrote:Arka, you are correct. I found an error in the random shot distribution and have updated my post above. The corrected distribution chart looks much more reasonable now.

Definitely, that looks accurate. I'd like to try a few of these myself, what software do you recommend?


Arka, I put the tool together in Excel, but it is not computationally efficient. I think I could increase the number of shooting turns to 50k or 100k if I rewrite it in MATLAB. MATLAB would also make it easier to change parameters. I sent you a copy of the excel spreadsheet via PM.

Return to “Tau Tactics”

Who is online

Users browsing this forum: Ifrit and 1 guest