I think this is probably real.

I believe there is a significant defect in the idiomatic listener pattern as discussed by most Erlang websites and as provided by most Erlang example code, and which is found in many Erlang applications.  This defect is relatively easily repaired once noticed without significant impact on surrounding code, and I have added functionality to my ScUtil Library to handle this automatically under the name standard_listener.

The fundamental problem is essentially a form of race condition.  The idiomatic listener is, roughly:

do_listen(Port, Opts, Handler) ->

    case gen_tcp:listen(Port, Opts) of

        { ok, ListeningSocket } ->
            listen_loop(ListeningSocket, Handler);

        { error, E } ->
            { error, E }

    end.

listen_loop(ListeningSocket, Handler) ->

    case gen_tcp:accept(ListeningSocket) of

        { ok, ConnectedSocket } ->
            spawn(node(), Handler, [ConnectedSocket]),
            listen_loop(ListeningSocket);

        { error, E } ->
            { error, E }

    end.

Now consider the case of a server under very heavy load.  Further, consider that the listening socket is opened either {active,true} or {active,once}, which is true in the vast bulk of Erlang network applications, meaning that packets are delivered automatically to the socket owner.  The general pattern is that the listening socket accepts a connection, spawns a handling process, passes the connected socket to that handling process, and the handling process takes ownership of the socket.

The problem is that it takes time for that all to happen, and Erlang doesn’t specify or allow you to control its timeslicing behavior (as well it should not).  As active sockets are managed by a standalone process, this means that if the connecting client is fast and the network is fast, the first packet (even the first several under extreme circumstances) could be delivered before the socket has been taken over by the handling PID, meaning that its contents would be dispatched to the wrong process, with no indication of where they were meant to go.  This invalidates connections and fills a non-discarding mailbox, which is a potentially serious memory leak (especially given that erlang’s response to out of memory conditions is to abort an entire VM.)

Obviously, this is intolerable.  There are better answers, though, than to switch to {active,false}.  One suggestion I heard was to pre-spawn handlers in order to reduce the gap time, but that doesn’t solve the problem, it just makes it less likely.

The approach that I took is to lie.  standard_listener takes the following steps to resolve the problem:

1. Add the default {active,true} to the inet options list, if it isn’t already present.
2. Strip out the {active,Foo} from the inet options list, and store it as ActiveStatus.
3. Add {active,false} to the inet options list, and use that options list to open the listener.
4. When spawning handler processes, pass a shunt as the starting function, taking the real handling function and the real ActiveStatus as arguments
5. The shunt sets the real ActiveStatus from inside the handler process, at which point the socket begins delivering messages

This neatly closes the problem.  A free, MIT license implementation can be found in ScUtil beginning in version 96.  A simplified, corrected example follows for immediate reference; the thing in ScUtil is more feature complete.

do_listen(Port, Opts) ->

    ActiveStatus = case proplists:get_value(active, SocketOptions) of
        undefined -> true;
        Other     -> Other
    end,

    FixedOpts = proplists:delete(active, SocketOptions)
             ++ [{active, false}],

    case gen_tcp:listen(Port, FixedOpts) of

        { ok, ListeningSocket } ->
            listen_loop(ActiveStatus, ListeningSocket);

        { error, E } ->
            { error, E }

    end.

listen_loop(ActiveStatus, ListeningSocket, Handler) ->

    case gen_tcp:accept(ListeningSocket) of

        { ok, ConnectedSocket } ->
            spawn(?MODULE, shunt, [ActiveStatus,ConnectedSocket,Handler]),
            listen_loop(ActiveStatus, ListeningSocket, Handler);

        { error, E } ->
            { error, E }

    end.

shunt(ActiveStatus, ConnectedSocket, Handler) ->

    controlling_process(ConnectedSocket, self()),
    inet:setopts(ConnectedSocket, [{active, ActiveStatus}]),
    Handler(ConnectedSocket).

So, I implemented moments, I replaced my central moments implementation, and I gave name wrappers for skewness and kurtosis to make them easier to identify.

This closes issues 169, 170, 171, 172 and 173.  As usual, this code is part of the ScUtil library, which is free and MIT licensed, because the GPL is evil.

moment(List, N) when is_list(List), is_number(N) ->
    scutil:arithmetic_mean( [ pow(Item, N) || Item <- List ] ).

moments(List) ->
    moments(List, [2,3,4]).

moments(List, Moments) when is_list(Moments) ->
    [ moment(List, M) || M <- Moments ].

central_moment(List, N) when is_list(List), is_number(N) ->
    ListAMean = scutil:arithmetic_mean(List),
    scutil:arithmetic_mean( [ pow(Item-ListAMean, N) || Item <- List ] ).

central_moments(List) ->
    central_moments(List, [2,3,4]).

central_moments(List, Moments) when is_list(Moments) ->
    [ central_moment(List, M) || M <- Moments ].

skewness(List) -> central_moment(List, 3).
kurtosis(List) -> central_moment(List, 4).

Setting up eJabberD from scratch

eJabberD is an erlang Jabber server written for high performance and long term stability. It’s also pretty easy to set up. eJabberD has a number of serious advantages over other Jabber servers, important among which is the ability to cluster servers, so that if one server goes down, the service itself continues running. Areas highlighted in red are areas where you’re expected to provide your own values. I provided the values I used in setting up the codeboner jabber server for comparison, but you can do as you see fit. Please don’t cut and paste several lines at once, even from series of instructions, as some of the utility calls made (especially the OpenSSL stuff) ask for user input (what’s the company’s domain, what’s the owner’s name, etc, nothing particularly complex). Please be careful to use the same number of days for expiration dates (the 365s), and be sure that the domain’s A and REV records are character for character identical to what you use in the SSL certificate, or it will appear to fail for no apparent reason. If you’ve followed these instructions and the thing just won’t let you connect, the SSL certificate may be the cause. These instructions are written assuming a single Unix server environment and a requirement for TLS (ssl3) top to bottom, using yum, GCC, the OpenSSL library and a self-signed certificate. Because of dependencies, it’s important that installation proceed in a specific order. The steps, in order, are as follows:

Install standard toolchain

yum install gcc gcc-g++ automake autoconf screen libtool which coreutils libstdc++ libgcc bind authconfig sudo tar zlib zlib-devel

Install OpenSSL ecology

yum install openssl openssl-clients openssl-server openssl-devel

Install Expat ecology

yum install expat expat-devel

Install Erlang r11b5

mkdir /erl-inst cd /erl-inst wget http://erlang.org/download/otp_src_R11B-5.tar.gz mkdir /usr/local/erlang mkdir /usr/local/erlang/otp_r11b cd /usr/local/erlang/otp_r11b gunzip -c /erl-inst/otp_src_R11B-5.tar.gz | tar xfp – cd otp_src_R11B-5 ./configure make make install rm /erl-inst/otp_src_R11B-5.tar.gz rmdir /erl-inst/

Create a self-signed PEM (or buy one)

openssl genrsa -des3 -out ca.key 4096 openssl req -new -x509 -days 365 -key ca.key -out ca.crt openssl genrsa -des3 -out server.key 4096 openssl req -new -key server.key -out server.csr openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt openssl rsa -in server.key -out server.key.stripped chmod 600 server.key.stripped cat server.crt server.key ca.crt >server.pem

Add erlang to your path

Varies per shell; in BASH, this is in .bash_profile or .bashrc; add :FOOPATH (not ;FOOPATH like dos) to the end of what’s there. Erlang is probably in /usr/local/bin/, but you can find it by: updatedb locate erlc

Install eJabberD

(make and change to some temporary installation directory) wget http://www.process-one.net/downloads/ejabberd/1.1.4/ejabberd-1.1.4.tar.gz tar -xvf ejabberd-1.1.4.tar.gz cd ejabberd-1.1.4/src ./configure make sudo make install

Configure eJabberD

nano -w /etc/ejabberd/ejabberd.cfg

In eJabberD.cfg line 89, set hosts to be served

{hosts, ["codeboner.com"]}.

In eJabberD.cfg line 116, uncomment the {certfile} directive, and set the path to your self-signed PEM

{certfile, “codebonerssl/cbjk.pem”}, starttls,

If you want to forbid non-TLS connections, in eJabberD.cfg line 116, change starttls to starttls_required

{certfile, “codebonerssl/cbjk.pem”}, starttls_required,

In eJabberD.cfg line 152, make sure the HTTP administration interface is enabled

web_admin

In eJabberD.cfg line 311, annote one user as an administrator (so they can use the HTTP interface)

{acl, admin, {user, “john”, “codeboner.com”}}.

Start eJabberD

erl -pa /var/lib/ejabberd/ebin -name ejabberd -s ejabberd -ejabberd config \”/etc/ejabberd/ejabberd.cfg\”

Add users to eJabberD (REMEMBER TO ADD THE ADMIN USER FROM ABOVE)

ejabberdctl –node ejabberd@localhost register john codeboner.com 123changeMe

Set up system preferences such as user groups

Admin control panel is at http://yourserver.com:5280/admin/ Your username is john@codeboner.com

You should now be able to connect to Jabber on the port at the beginning of the c2s rule (5222 unless you changed it.)

There are people who take long roundabout approaches to get to results like these, when instead they could be doing things like

p1() -> lists:sum(

    [ X || X <- lists:seq(1,10),

      ((X rem 3) == 0) orelse ((X rem 5) == 0) ]

).

As a result, I’m starting to think that I need to start explaining things. Anyone agree or disagree?

See, there are a lot of subtle attacks one can bring to bear on even sound cryptological systems, if one doesn’t know what one is doing. PHP does a pretty neat job of wrapping up these issues behind a semi-generic library interface, but it doesn’t do such a great job in the manual of detailing safe usage. In particular, some of the user notes give conflicting, and frequently outright bad, advice; furthermore, the displayed methods for using the interface aren’t themselves particularly generic, and skip some particularly useful things they could do. It also doesn’t help that there’s a distinct trend of more-than-one-way-itis in libmcrypt, including a lot of support for deprecated functionality.

So, what to do about it, but write a new tute? Actually, this one’s pretty low-impact; might as well just release it almost as straight source. Still, it’s helpful to explain.

The Quick Version

It’s surprisingly simple.

If you just want to throw the library in and start working, follow this download link, then throw the file in with the rest of your PHP project. Change the define DEFAULT_MD5_SALT on line 36 of StonePhpSafeCrypt_config.php (the library will force you to do this) and fill it with some random string. Really doesn’t matter, and you’ll never need to remember it; it’s just a “salt,” which is used to prevent dictionary attacks. Don’t use a phrase someone can guess, and consider just mashing your hands all over the keyboard for a second instead of trying to make up something smart.

Next, require_once() the library, then use PackCrypt($data, $key [, $options] ) and UnpackCrypt($data, $key [, $options] ) on any serializable PHP datatype. $key does not need to be secure, except against social engineering attacks like guessing according to the user’s behavior; it is hardened by the library, and salted by you.

As options, you may pass in an array containing up to four things as key/value pairs:

• ‘cipher‘, which may take any value from the mcrypt cipher module list, and which defaults to twofish
• ‘mode‘, which may take any cipher block type, and which defaults to CBC
• ‘compressor‘, which may take any compressor listed in StonePhpSafeCrypt_compressors.php, and which defaults to off (the package comes with gzip, gzip deflate and bzip2 defined, but you may add others)
• ‘salt‘, which allows you to override the default salt with a salt of your own choice.

All four of these options are reflexive: if you use them during encryption, you must use them identically during decryption, or the process will not allow you to unpack the data. All functions other than PackCrypt() and UnpackCrypt() in the library are for internal support, and should not be used directly.

Here’s a minimalist example, to show you what you’re doing:

require_once('StonePhpSafeCrypt.php');

$sourcedata = array('Hello', 1, 2, 3=>4, 'Goodbye');
$packed = PackCrypt($sourcedata, 'Password');

if ($packed['state'] === false) { die('Error: ' . $packed['reason']); }

echo 'Packed: ', $packed['output'], '<br />';

$unpacked = UnpackCrypt($packed, 'Password');
echo 'Unpacked: ', print_r($unpacked);

And that’s all there is to it.

The Fundamentals

The first thing we need is a way to wrap the encryption and decryption process, without actually forcing a choice of which method is used. This is important because some machines may not have some algorithms available, because algorithms get broken and replaced, and so on. The user really needs to be able to select the algorithm. Even so, the actual process of encryption is fairly ridiculously cumbersome, and can be wrapped.

That said, there’s a lot to PHP. One thing that’s really nice about PHP is its automatic data munging. Of particular utility is the concept of serialization; objects even have a magic stub they can provide to implement serialization magically. It can be argued that a high quality wrapper for encryption should magically handle the packing and unpacking of a PHP data type, be it a scalar, an array or even a correctly made object. Serialize is the appropriate means by which to do so, and so we’ll build that into our encryptor.

Another important thing for an encryption wrapper is that the user shouldn’t actually need to know a whole lot to use it safely. The actual data to be packed and the password by which to do so are sort of common-sense minimums; ideally, a function should be flexible enough to allow more customization, but should require nothing more than these and still be safe. Importantly, this function must not require the user to know how to make safe passwords. We will do that for them. In order to keep that mindset, we will begin referring to two passwords: the user’s password, which we’ll call the Weak Key, and the password we derive therefrom, called the Strong Key. Only the strong key is reliable, or used in any way in the transmitted cipher. (Weak keys are still subject to social engineering attacks, so end users should still be advised of rules they refuse to follow, but this at least protects the data stream itself.)

To that end, we will create two basic functions: PackCrypt() and UnpackCrypt(). These functions will take two mandatory and two optional arguments each: the data, the weak password, and if the user wants to, they may also override the default choice of encryption algorithms (which is something best reserved for those who are familiar with the advantages and disadvantages of the various algorithms.)

The workhorses: PackCrypt() and UnpackCrypt()

PackCrypt and UnpackCrypt are the two functions you actually use in this library.

PackCrypt(&$Data, &$WeakKey, $options)

PackCrypt takes any serializable data structure for $Data, any string for $WeakKey, and if you like, some extra commands in $options. $Data can be a nested array full of objects, all sorts of creepy stuff, whatever. If it’s a correct and complete PHP class which follows the rules for serialize(), then it should be safe for PackCrypt().

It’s really just that simple. Make some horribly complex data structure, and do

$packed = PackCrypt($someData, 'my key sucks');

and then look at $packed['output']. Bang: safe, clean serialized secure encryption.

Why $packed['output']? Why not just $packed?

This library (like most of John’s libraries) returns a state tuple for its operations. This is a behavior that’s common in some languages, particularly Erlang/OTP, and which PHP can support quite easily. It’s also a very important and powerful technique: it makes concurrent message tracking very easy, it makes keeping track of parallel error states very easy, and it makes complex returns with ancillary information not only possible but quite natural.

PackCrypt returns an array with ['status'], ['reason'] and ['output'] set. Status is a boolean which represents whether things proceeded correctly. Reason is a textual description which explains what error set status to false when something broke; if status is true, which is to say if things went well, it just says “Successfully encrypted.” or whatever is appropriate. Output contains the actual results, or false if something failed.

• If everything goes correctly,
• ['status'] will be true,
• ['output'] will contain your encrypted result, and
• ['reason'] will say ‘Successful Pack’.
• If there is a problem,
• ['status'] will be false,
• ['output'] will be false, and
• ['reason'] will contain a message explaining the problem, which will most likely be that the encryption algorithm chosen or the compressor chosen is unavailable on this system

Okay, so how does it work

It’s fairly straightforward. First, the functions unpack and apply the options, check for the existance of the requested compressor and so on. Once that’s done, the compressor opens the cipher module (such as Blowfish, RSA, Rijndael, TripleDES and so on,) engages the appropriate block cipher mode (cbc, cfb, ecb or ofb,) creates a salted MD5 hash of the weak key, generates a randomized initialization vector, ciphers the IV with a resalted strong key, serializes the incoming data, compresses the serialized data if requested, encrypts the serialized data, writes out the result array and cleans up. Unsurprisingly, the unpacking scheme is basically the same thing in reverse.
The thing is, you have to know all the steps in order, and you have to do them correctly, accomodating for the different sizes in each algorithm and platform implementation, trim extra space and so on. So, wrap ‘em up, no problem, done. Welcome to PackCrypt() territory.

Safe IV Transfer: BlockScramble() and BlockDescramble()

The PHP manual incorrectly states that the initialization vector for an encrypted datastream may be transmitted plaintext. This would be highly convenient and provides a desirable mechanism for transfer, but can be the source of man-in-the-middle attacks in CBC block mode algorithms, as well as several other situations. See Ciphers by Ritter for a detailed explanation of the problem. This library solves the problem by salted hashing the weak key and then reversibly merging that with the IV, so that the leading block can be transmitted safely as a stream prefix.

The functions which perform this salting, hashing and merging are called BlockScramble() and BlockDescramble().

Convenient compression: $options['compressor']

Of course, one thing that’s really helpful to have in this kind of routine is inline compression. Compression both makes the original stream harder to detect and (obviously) reduces bandwidth and storage costs. However, the way to apply compression to the encryption stream is inobvious; most people would just try compressing the results of the encryption, and that’s not going to work. The reason is that compression and encryption are at ends: compression relies on patterns in data, whereas the purpose of encryption is to hide patterns in data. A well encrypted string when compressed is generally larger than uncompressed.

As such, to keep things simple, I put compression directly into the mechanism, in a way that can be extended to new compression algorithms later (though I have a hard time imagining needing to replace BZip2.) However, if you want to add support for the latest greatest algorithm, even if it’s newer than my library, well, no problem. Open StonePhpSafeCrypt_compressors.php and add your compressor to the list. Instructions are in the file, but it’s really quite trivial.

Subdirectories and require_once()

Because of the way PHP inclusion works, the require directives in a given script are referential to the including script’s location, rather than their own. That means that the includes for the various parts of this library need to be referenced according to their own directory. If you just dump all the files in the same directory as your main project, well great, works fine. If you want to put them in a subdirectory, do the following before you include the library (or it will fail in a fairly unreadable way):

define('STONE_PHP_SAFE_CRYPT_HOST_DIRECTORY', 'dirgoeshere/');

The directory may be absolute or referential, should be in Unix directory format, and needs the trailing slash.

That’s it?

That’s it. Unpack the files, set the salt, $foo = PackCrypt(data, key, options); and look in $foo['output']. Safe, convenient and reliable. This is what it should have been all along.

In case you missed it earlier, get the source here. There is a working example in test.php, but here’s that same fairly minimalist example again, now that you know what it’s doing:

require_once('StonePhpSafeCrypt.php');

$sourcedata = array('Hello', 1, 2, 3=>4, 'Goodbye');
$packed = PackCrypt($sourcedata, 'Password');

if ($packed['state'] === false) { die('Error: ' . $packed['reason']); }

echo 'Packed: ', $packed['output'], '<br />';

$unpacked = UnpackCrypt($packed, 'Password');
echo 'Unpacked: ', print_r($unpacked);

And that’s all there is to it.

• Suppress the user interface. IE starts with most of its interface turned off, but a few things aren’t. Notable things to include are to suppress the context menu and keyboard navigation.
• There are several ways to suppress the context menu. The easiest way is to do it in the html. Add oncontextmenu=”return false;” to your <body> and the problem just goes away, and it’s easily manually overridden for specific elements later.
• Alternately, you can override the interface programmatically through IDocHostUIHandler.
• Get rid of the OnNavigate noise (the clicking sound when you hit a link.) This is harder than it should be.
• Prevent keyboard navigation. If you only ever navigate once, and then do everything else in DHTML, this isn’t an issue, because there’s nothing to go forward or back to. However, if you need to, the way to do this is to capture DWebBrowserEvents2::BeforeNavigate2() and fill its last parameter, VARIANT_BOOL*& Cancel, to true. For security reasons this can’t be done in HTML without a garish ugly dialog box confirmation.
• Prevent dragging and dropping links. Do this with BeforeNavigate2() just like keyboard navigation.
• Prevent selection. The easiest way to do that is in the HTML by adding onselectstart=”return false;” to the <body>, which is also easily overridden for child elements where appropriate. If you need it programmatically, such as a block which can be turned from and to editability, that’s in IDocHostUIHandler.
• Handle some keys directly. Particularly often, control keys, tab and f-keys need very different interpretations in applications than are the defaults in IE. Some keys can be reliably intercepted in JScript, but not all of them. For lightweight stuff, use JScript; it’ll be less painful. However, if you need real control, you need to provide an IDocHostUIHandler::TranslateAccelerator() implementation.
• Suppress the interior border. People are often surprised that they turn off the window border in IE, and yet it seems to still appear. That’s because IE’s default stylesheet puts an inset bevel on every web page. Turn that off in CSS by writing body { border: 0; }.
• Begin to embed your images and other support media as resources, and access them with the res:// protocol.
• One ugly caveat: I have never found any variation of the DirectX PNG alpha filter hack which works with res://, and I’ve looked often and at some very weird variations. It is my current belief that a PNG cannot be embedded through res:// and still be fixed for alpha. This leads to the unfortunate case of requiring at least part of your resources on disk. If anyone knows a way around this, a heads-up would be greatly appreciated.
• Set up a system to exchange messages between IE DOM/DHTML/JScript and your application. There are a bujillion ways to do this, but I tend to use a combination of intercepted element events (mostly OnClick in WebBrowserEvents Invoke,) DOM extension behaviors that allow me to call C++ directly from JScript through IDocHostUIHandler::GetExternal(), and executing scripts directly on the DOM object through IHTMLWindow2::execScript().
• Build everything in the to be inside a container; make the body itself overflow: none. This prevents several ugly re-layout quirks which we’re used to on the web but not in applications, when content gets long.
• Start the container invisible, and make the black. That way, during the lag while resources are being loaded, you have a reasonable appearance. Black flashes look correct. White flashes look broken.
• Set an onload=”HandleLoad();” function for the <body>. That way, you know when all resources are loaded, and thus when it’s safe to take the invisible clause off of the main container.
• Set the main container position:relative, to make absolute positioning of contained elements easier.
• Build your web page in as a resource, and load it through IHtmlDocument2::write(). This allows you to load a document at runtime without using about:blank (which most people use, but which causes a brief white flash before your application loads; very unprofessional looking.) You’ll need SafeArrayAccessData() and SafeArrayUnaccessData() to use write() safely. When it comes time to skin or internationalize, and when you’re dealing with a compile cycle which is just linking, you’ll thank me for this.
• Suppress dragging on links and buttons. If you don’t, people will be able to drag them outside the app and onto the desktop as shortcuts, and when followed those shortcuts will lead into your HTML page from the outside (which is, surprisingly, legal.)
• Consider compressing your executable, such as with UPX, which makes resources unreadable externally.
• Set an id on your <body>. The is considered the origin of most events which don’t have a clear origin, and that means that catching these events is a lot easier (catching by id is the most straightforward way to sort out sources.)
• Resist the urge to use runtime styles. They’re not worth the problems they cause.

That should get the new IE user through some common foibles. I’ve probably missed stuff; if you can think of something, lemme know. I do not currently know of a way to embed flash, unfortunately, without leaving available an extra file.

Well, I still don’t know what he does, but I found a way.

Back in post #141, More IE Woes, I documented several problems. One was that MSDN claims this noise cannot be suppressed. Indeed, by design, IE6 refuses to allow you to suppress that sound (god knows why.) That MSDN article goes as far as to tell you to override the user’s sound preferences to turn the noise off. How badbear is that? Anyway, IE7 finally has an interface to turn that off, called FEATURE_DISABLE_NAVIGATION_SOUNDS. However, people like me who are reluctant to make their applications non-functional without IE7 will not like that answer.

It turns out there’s a better way.

There’s a moderately obscure interface from IE5 for IDocHostUIHandler called GetOptionKeyPath. The superficial purpose of this is to allow the IE embedder to create a fake user, so that they can set defaults for IE like the basic font size, background color and so on. This is generally completely ignored in favor of some CSS, but it turns out that this thing actually covers the sound effect scheme too.

All you need to do is use GetOptionKeyPath, create a fake user, override the OnNavigate sound (and maybe some others) to nothing, and let the rest alone, to default to the user’s preferences.

Bang: no more clicky noise, and no overriding preferences, breaking other applications and risking corrupting preferences on crash.

Very clean.