[digg-reddit-me]These are standard statistical tools for measuring differences, drift and error within sets, as well as Kth moments about the mean.  Central moments are also the building blocks of skewness and kurtosis, which are covered in a later post.  Root mean square is particularly useful as a measure of set error.  Standard deviation is great for figuring out how much spread/differentiation there is within a set.

This code requires the arithmetic mean stuff from Statistics in Erlang part 1.  There’s interesting, unrelated stuff in Part 2.

As usual, this code is part of the ScUtil library.  ScUtil is free and MIT license, because the GPL is evil.

This implements issue 128.  This implements issue 138.  This finishes implementing issue 119.

std_deviation(Values) when is_list(Values) ->
    Mean = arithmetic_mean(Values),
    math:sqrt(arithmetic_mean([ (Val-Mean)*(Val-Mean) || Val <- Values ])).

root_mean_square(List) when is_list(List) ->
    math:sqrt(arithmetic_mean([ Val*Val || Val <- List ])).

central_moments(Items) ->
    Cnt  = length(Items),
    Mean = lists:sum(Items) / Cnt,
    TFun = fun(X) ->
        Base = X-Mean, B2=Base*Base, B3=B2*Base, B4=B3*Base, {B2,B3,B4}
    end,
    collapse_central_moments(Cnt, [ TFun(I) || I <- Items ], {0,0,0}).

collapse_central_moments(N, [], {WM2, WM3, WM4}) ->
    { WM2/N, WM3/N, WM4/N };

collapse_central_moments(N, [{I2,I3,I4}|Rem], {WM2, WM3, WM4}) ->
    collapse_central_moments(N, Rem, {I2+WM2, I3+WM3, I4+WM4}).

[digg-reddit-me]Mean is a complex topic, and is covered in Statistics in Erlang part 1.

Median and mode are less complex.  It’s worth noting that mode reports its results as a list, because it’s possible for there to be several modes for a list.  Also, mode is not strictly numeric – it works for mixed-type lists too (you can take the mode of a list of atoms, for example.)

Mode is really just a reduction of the results of histograph, which is similarly open to arbitrary type list contents.  Mode also uses a toy function even_or_odd which is provided here.

As usual, this code is part of the ScUtil library.  ScUtil is free and MIT license, because the GPL is evil.

This closes issue 100.  This closes issue 105.  This closes issue 134.  This closes issue 135.

histograph(List) when is_list(List) ->
    [Head|Tail] = lists:sort(List),
    histo_count(Tail, Head, 1, []).

histo_count([], Current, Count, Work) ->
    lists:reverse([{Current,Count}]++Work);

histo_count([Current|Tail], Current, Count, Work) ->
    histo_count(Tail, Current, Count+1, Work);

histo_count([New|Tail], Current, Count, Work) ->
    histo_count(Tail, New, 1, [{Current,Count}]++Work).

even_or_odd(Num) when is_integer(Num) ->
    if
        Num band 1 == 0 -> even;
        true            -> odd
    end.

median(List) when is_list(List) ->
    SList = lists:sort(List),
    Length = length(SList),
    case even_or_odd(Length) of
        even -> [A,B] = lists:sublist(SList, round(Length/2), 2), (A+B)/2;
        odd  -> lists:nth( round((Length+1)/2), SList )
    end.

mode([]) -> [];

mode(List) when is_list(List) ->
    mode_front(lists:reverse(lists:keysort(2, scutil:histograph(List)))).

mode_front([{Item,Freq}|Tail]) ->
    mode_front(Tail, Freq, [Item]).

mode_front([ {Item, Freq} | Tail], Freq, Results) ->
    mode_front(Tail, Freq, [Item]++Results);

mode_front([{_Item,_Freq} |_Tail],_Better, Results) ->
    Results;

mode_front([], _Freq, Results) -> Results.

[digg-reddit-me]I’ll be putting up some statistical functions I’ve had to write recently.  This is the first batch.  I confess, I find the erlang implementations far more readable than the pure math definitions one finds around; I’ve been thinking about writing tutorials, but with this code here, I’m not entirely sure it’s necessary.

At any rate, the code follows.  As with so much of my erlang code, this code is part of the ScUtil library.  ScUtil is free and MIT license, because the GPL is evil.

There’re more statistics coming, I just don’t want to make any one post too huge, and I don’t want keyword saturation.

This partially closes issue 119.  This closes issue 133.  This closes issue 136.  This closes issue 137.

list_product(List) when is_list(List) ->
    list_product(List, 1).

list_product([], Counter) ->
    Counter;

list_product([Head|Tail], Counter) ->
    list_product(Tail, Counter*Head).

arithmetic_mean(List) when is_list(List) ->
    lists:sum(List) / length(List).

geometric_mean(List) when is_list(List) ->
    math:pow(scutil:list_product(List), 1/length(List)).

harmonic_mean(List) when is_list(List) ->
    length(List) / lists:sum([ 1/X || X<-List ]).

weighted_arithmetic_mean(List) when is_list(List) ->
    weighted_arithmetic_mean(List, 0, 0).

weighted_arithmetic_mean([], Num, Denom) ->
    Num/Denom;

weighted_arithmetic_mean([{W,V}|Tail], Num, Denom) ->
    weighted_arithmetic_mean(Tail, Num+(W*V), Denom+W).

Someone made a relatively intelligent comment on one of my blog posts, leading me to believe he’s not from the internet.  Nonetheless, I took a look at his site, In Search of Concise Software, and there’s some interesting junk posted … leading me to believe he’s not from the internet.

Pity about Blogspot’s interface.  Even so, it’s worth a look.

[digg-reddit-me]Oddly, the Erlang standard library does not have a function for reading out module attributes.  One is relatively straightforwardly hacked together with beam_lib, if you know what you’re doing, but most people wouldn’t know to look there, and it’s kind of a hassle to write.

So, I wrote it for you.

• get_module_attribute is a member of the ScUtil library.
• This closes issue 117.

Usage is straightforward: scutil:get_module_attribute(lists, export).

get_module_attribute(Module,Attribute) ->

    case beam_lib:chunks(Module, [attributes]) of

        { ok, { _, [ {attributes,Attributes} ] } } ->
            case lists:keysearch(Attribute, 1, Attributes) of
                { value, {Attribute,[Value]} } -> Value;
                false                          -> { error, no_such_attribute }
            end;

        { error, beam_lib, { file_error, _, enoent} } ->
            { error, no_such_module }

    end.

[digg-reddit-me]Amusingly, there isn’t actually a way to ask Erlang the type of a variable, and new programmers often struggle with this on basis of not realizing they’re expected to use is_foo() guards.

This function, type_of(), resolves the issue.  As such, none of the testing stuff is included; that’s all in the library, rather than here on the blog.

• type_of() is a member of the free ScUtil library.
• This closes issue 132.

If you’re running Erlang 11 or earlier, the bitstring clause will fail.  That is a new type.  Just comment it out and move on with your day; it’s a synonym for “binary” and on your system the word “binary” will come out instead.

type_of(X) when is_integer(X)   -> integer;
type_of(X) when is_float(X)     -> float;
type_of(X) when is_list(X)      -> list;
type_of(X) when is_tuple(X)     -> tuple;
type_of(X) when is_bitstring(X) -> bitstring;  % will fail before e12
type_of(X) when is_binary(X)    -> binary;
type_of(X) when is_boolean(X)   -> boolean;
type_of(X) when is_function(X)  -> function;
type_of(X) when is_pid(X)       -> pid;
type_of(X) when is_port(X)      -> port;
type_of(X) when is_reference(X) -> reference;
type_of(X) when is_atom(X)      -> atom;

type_of(_X)                     -> unknown.

I’m going to be releasing a few new libraries in the next several days, both by archive and public subversion.  I’ve already bought the domains and built a forum for them.  I even wasted a couple hours subversion automating everything down to the line of having little library websites made automatically, with custom per-library color palettes.

I was a little bored.

So we’re going to have three libraries progressing in the immediate future, with quite a few more over time:

1. HtStub – An embeddable, zero-config, zero-behavior secure Erlang webserver
2. SC A-Star – An efficient, modular ECMAscript (flash/actionscript, javascript, jscript etc) A* implementation with support for custom grid geometries (includes algorithm tutorial)
3. TestErl – unit, regression and stochastic testing for Erlang
4. ScUtil – a fairly large list of gap filling functionality for Erlang

In the near future, I will add C++ and PHP libraries, as well as many some libraries for more obscure languages like FormulaONE, Mozart-Oz, Factor and maybe (sadly) Delphi.  I have more than 30 libraries ready for release.

All those repos are pretty empty at the moment.  That will change in coming days, and I’m sure I’ll post lots of boring little snippets here about whatever minor new thing my crap does.

Yay!

I’ve actually built a bunch of small Erlang libraries which I find tremendously useful.  Over the last weekend, I’ve set up public SVN for a lot of them, built a small forum and started a project manager.  I’m actually restarting each and every project in the light of better adherence to best practices, since the code’s going to be distributed now, such as test suites from ground zero, but the point is, I’m making my libraries available and easy to find now.

The first three I’m going to release are scutil, testerl and htstub.

• scutil is my personal utility library.  It’s a hodgepodge of semi-random stuff.  there’s a whole lot of crap down there.
• testerl is a unit/regression and stochastic testing library.  One of the main goals of testerl is to have low impact on the code it tests, which it achieves through a single module attribute and a list (syntax identical to export) of the non-exported functions that the test rig needs access to for testing.
• htstub is a socket manager and HTTP protocol implementation.  It’s not a webserver – it has no filesystem access and doesn’t know how to respond to requests – it’s more of a transparent way to handle HTTP.  htstub is useful for adding a web front end to applications.

I actually intend to release some c++, actionscript, php, javascript and delphi stuff soon too, If I Get Around To It ™; that said, now that I have SVN largely automated, it’s quite likely.

I’ve been writing a ton of erlang libraries lately (some more useful than others).  After throwing Google Analytics on my site, I’ve discovered an inordinate focus on the little bit of Erlang stuff I posted.

Time to give the masses what they want, I guess.  In the upcoming month or two, I hope to release my modest testing library, my stub webserver and my utility library.  If they get useful traffic, maybe I’ll release more.

Stay tuned.

I had to write these for a colleague some months ago, and promptly forgot in classic fashion. Here’s something to mock my inability to write coherent install docs for posterity: Setting Up eJabberD From Scratch. A how to, of sorts, I guess. This was written for a Centos server, but is probably accurate for most Unices (don’t really know for sure.) Read the rest…