Standard Deviation, Root Mean Square and the Central Moments (Statistics in Erlang part 3)
August 10, 2008 8:19 pm Erlang, Math, Programming, Statistics, Tools and Libraries[digg-reddit-me]These are standard statistical tools for measuring differences, drift and error within sets, as well as Kth moments about the mean. Central moments are also the building blocks of skewness and kurtosis, which are covered in a later post. Root mean square is particularly useful as a measure of set error. Standard deviation is great for figuring out how much spread/differentiation there is within a set.
This code requires the arithmetic mean stuff from Statistics in Erlang part 1. There’s interesting, unrelated stuff in Part 2.
As usual, this code is part of the ScUtil library. ScUtil is free and MIT license, because the GPL is evil.
This implements issue 128. This implements issue 138. This finishes implementing issue 119.
std_deviation(Values) when is_list(Values) ->
Mean = arithmetic_mean(Values),
math:sqrt(arithmetic_mean([ (Val-Mean)*(Val-Mean) || Val <- Values ])).
root_mean_square(List) when is_list(List) ->
math:sqrt(arithmetic_mean([ Val*Val || Val <- List ])).
central_moments(Items) ->
Cnt = length(Items),
Mean = lists:sum(Items) / Cnt,
TFun = fun(X) ->
Base = X-Mean, B2=Base*Base, B3=B2*Base, B4=B3*Base, {B2,B3,B4}
end,
collapse_central_moments(Cnt, [ TFun(I) || I <- Items ], {0,0,0}).
collapse_central_moments(N, [], {WM2, WM3, WM4}) ->
{ WM2/N, WM3/N, WM4/N };
collapse_central_moments(N, [{I2,I3,I4}|Rem], {WM2, WM3, WM4}) ->
collapse_central_moments(N, Rem, {I2+WM2, I3+WM3, I4+WM4}).
August 10th, 2008 at 9:17 pm
Statistics are an area with little support in erlang at the moment so I think that this is a good initiative.
But You need to be more careful with how many times You traverse the lists of values.
For instance in std_deviation You traverse the list 5 times. I seem to remember from my statistics course, (>25 years ago), that it is possible to calculate stddev with only one traversal.
August 10th, 2008 at 10:19 pm
To follow up on my previous comment. Like this
stddev(L) ->
{SqSum, Sum, Len} = lists:foldl(fun (X, {K,S,N}) ->
{K+X*X,S+X,N+1}
end, {0,0,0}, L),
Mean = Sum/Len,
math:sqrt(SqSum/Len-Mean*Mean).
March 16th, 2009 at 2:45 am
hello,
Would you please tell me what the standard moments (stage 5 & 6) show about a specific distribution?for example B4 (standard moment stage 4) showes the kurtosis and B3 showes the skewness.Please tell me what B5 & B6 show.
thanks
March 23rd, 2009 at 1:29 pm
There aren’t default names for the fifth and sixth central moments, as far as I know. If you find some, let me know, and I’ll wrap them.