I was a little bored.

So we’re going to have three libraries progressing in the immediate future, with quite a few more over time:

1. HtStub – An embeddable, zero-config, zero-behavior secure Erlang webserver
2. SC A-Star – An efficient, modular ECMAscript (flash/actionscript, javascript, jscript etc) A* implementation with support for custom grid geometries (includes algorithm tutorial)
3. TestErl – unit, regression and stochastic testing for Erlang
4. ScUtil – a fairly large list of gap filling functionality for Erlang

In the near future, I will add C++ and PHP libraries, as well as many some libraries for more obscure languages like FormulaONE, Mozart-Oz, Factor and maybe (sadly) Delphi.  I have more than 30 libraries ready for release.

All those repos are pretty empty at the moment.  That will change in coming days, and I’m sure I’ll post lots of boring little snippets here about whatever minor new thing my crap does.

Yay!

See, there are a lot of subtle attacks one can bring to bear on even sound cryptological systems, if one doesn’t know what one is doing. PHP does a pretty neat job of wrapping up these issues behind a semi-generic library interface, but it doesn’t do such a great job in the manual of detailing safe usage. In particular, some of the user notes give conflicting, and frequently outright bad, advice; furthermore, the displayed methods for using the interface aren’t themselves particularly generic, and skip some particularly useful things they could do. It also doesn’t help that there’s a distinct trend of more-than-one-way-itis in libmcrypt, including a lot of support for deprecated functionality.

So, what to do about it, but write a new tute? Actually, this one’s pretty low-impact; might as well just release it almost as straight source. Still, it’s helpful to explain.

The Quick Version

It’s surprisingly simple.

If you just want to throw the library in and start working, follow this download link, then throw the file in with the rest of your PHP project. Change the define DEFAULT_MD5_SALT on line 36 of StonePhpSafeCrypt_config.php (the library will force you to do this) and fill it with some random string. Really doesn’t matter, and you’ll never need to remember it; it’s just a “salt,” which is used to prevent dictionary attacks. Don’t use a phrase someone can guess, and consider just mashing your hands all over the keyboard for a second instead of trying to make up something smart.

Next, require_once() the library, then use PackCrypt($data, $key [, $options] ) and UnpackCrypt($data, $key [, $options] ) on any serializable PHP datatype. $key does not need to be secure, except against social engineering attacks like guessing according to the user’s behavior; it is hardened by the library, and salted by you.

As options, you may pass in an array containing up to four things as key/value pairs:

• ‘cipher‘, which may take any value from the mcrypt cipher module list, and which defaults to twofish
• ‘mode‘, which may take any cipher block type, and which defaults to CBC
• ‘compressor‘, which may take any compressor listed in StonePhpSafeCrypt_compressors.php, and which defaults to off (the package comes with gzip, gzip deflate and bzip2 defined, but you may add others)
• ‘salt‘, which allows you to override the default salt with a salt of your own choice.

All four of these options are reflexive: if you use them during encryption, you must use them identically during decryption, or the process will not allow you to unpack the data. All functions other than PackCrypt() and UnpackCrypt() in the library are for internal support, and should not be used directly.

Here’s a minimalist example, to show you what you’re doing:

require_once('StonePhpSafeCrypt.php');

$sourcedata = array('Hello', 1, 2, 3=>4, 'Goodbye');
$packed = PackCrypt($sourcedata, 'Password');

if ($packed['state'] === false) { die('Error: ' . $packed['reason']); }

echo 'Packed: ', $packed['output'], '<br />';

$unpacked = UnpackCrypt($packed, 'Password');
echo 'Unpacked: ', print_r($unpacked);

And that’s all there is to it.

The Fundamentals

The first thing we need is a way to wrap the encryption and decryption process, without actually forcing a choice of which method is used. This is important because some machines may not have some algorithms available, because algorithms get broken and replaced, and so on. The user really needs to be able to select the algorithm. Even so, the actual process of encryption is fairly ridiculously cumbersome, and can be wrapped.

That said, there’s a lot to PHP. One thing that’s really nice about PHP is its automatic data munging. Of particular utility is the concept of serialization; objects even have a magic stub they can provide to implement serialization magically. It can be argued that a high quality wrapper for encryption should magically handle the packing and unpacking of a PHP data type, be it a scalar, an array or even a correctly made object. Serialize is the appropriate means by which to do so, and so we’ll build that into our encryptor.

Another important thing for an encryption wrapper is that the user shouldn’t actually need to know a whole lot to use it safely. The actual data to be packed and the password by which to do so are sort of common-sense minimums; ideally, a function should be flexible enough to allow more customization, but should require nothing more than these and still be safe. Importantly, this function must not require the user to know how to make safe passwords. We will do that for them. In order to keep that mindset, we will begin referring to two passwords: the user’s password, which we’ll call the Weak Key, and the password we derive therefrom, called the Strong Key. Only the strong key is reliable, or used in any way in the transmitted cipher. (Weak keys are still subject to social engineering attacks, so end users should still be advised of rules they refuse to follow, but this at least protects the data stream itself.)

To that end, we will create two basic functions: PackCrypt() and UnpackCrypt(). These functions will take two mandatory and two optional arguments each: the data, the weak password, and if the user wants to, they may also override the default choice of encryption algorithms (which is something best reserved for those who are familiar with the advantages and disadvantages of the various algorithms.)

The workhorses: PackCrypt() and UnpackCrypt()

PackCrypt and UnpackCrypt are the two functions you actually use in this library.

PackCrypt(&$Data, &$WeakKey, $options)

PackCrypt takes any serializable data structure for $Data, any string for $WeakKey, and if you like, some extra commands in $options. $Data can be a nested array full of objects, all sorts of creepy stuff, whatever. If it’s a correct and complete PHP class which follows the rules for serialize(), then it should be safe for PackCrypt().

It’s really just that simple. Make some horribly complex data structure, and do

$packed = PackCrypt($someData, 'my key sucks');

and then look at $packed['output']. Bang: safe, clean serialized secure encryption.

Why $packed['output']? Why not just $packed?

This library (like most of John’s libraries) returns a state tuple for its operations. This is a behavior that’s common in some languages, particularly Erlang/OTP, and which PHP can support quite easily. It’s also a very important and powerful technique: it makes concurrent message tracking very easy, it makes keeping track of parallel error states very easy, and it makes complex returns with ancillary information not only possible but quite natural.

PackCrypt returns an array with ['status'], ['reason'] and ['output'] set. Status is a boolean which represents whether things proceeded correctly. Reason is a textual description which explains what error set status to false when something broke; if status is true, which is to say if things went well, it just says “Successfully encrypted.” or whatever is appropriate. Output contains the actual results, or false if something failed.

• If everything goes correctly,
• ['status'] will be true,
• ['output'] will contain your encrypted result, and
• ['reason'] will say ‘Successful Pack’.
• If there is a problem,
• ['status'] will be false,
• ['output'] will be false, and
• ['reason'] will contain a message explaining the problem, which will most likely be that the encryption algorithm chosen or the compressor chosen is unavailable on this system

Okay, so how does it work

It’s fairly straightforward. First, the functions unpack and apply the options, check for the existance of the requested compressor and so on. Once that’s done, the compressor opens the cipher module (such as Blowfish, RSA, Rijndael, TripleDES and so on,) engages the appropriate block cipher mode (cbc, cfb, ecb or ofb,) creates a salted MD5 hash of the weak key, generates a randomized initialization vector, ciphers the IV with a resalted strong key, serializes the incoming data, compresses the serialized data if requested, encrypts the serialized data, writes out the result array and cleans up. Unsurprisingly, the unpacking scheme is basically the same thing in reverse.
The thing is, you have to know all the steps in order, and you have to do them correctly, accomodating for the different sizes in each algorithm and platform implementation, trim extra space and so on. So, wrap ‘em up, no problem, done. Welcome to PackCrypt() territory.

Safe IV Transfer: BlockScramble() and BlockDescramble()

The PHP manual incorrectly states that the initialization vector for an encrypted datastream may be transmitted plaintext. This would be highly convenient and provides a desirable mechanism for transfer, but can be the source of man-in-the-middle attacks in CBC block mode algorithms, as well as several other situations. See Ciphers by Ritter for a detailed explanation of the problem. This library solves the problem by salted hashing the weak key and then reversibly merging that with the IV, so that the leading block can be transmitted safely as a stream prefix.

The functions which perform this salting, hashing and merging are called BlockScramble() and BlockDescramble().

Convenient compression: $options['compressor']

Of course, one thing that’s really helpful to have in this kind of routine is inline compression. Compression both makes the original stream harder to detect and (obviously) reduces bandwidth and storage costs. However, the way to apply compression to the encryption stream is inobvious; most people would just try compressing the results of the encryption, and that’s not going to work. The reason is that compression and encryption are at ends: compression relies on patterns in data, whereas the purpose of encryption is to hide patterns in data. A well encrypted string when compressed is generally larger than uncompressed.

As such, to keep things simple, I put compression directly into the mechanism, in a way that can be extended to new compression algorithms later (though I have a hard time imagining needing to replace BZip2.) However, if you want to add support for the latest greatest algorithm, even if it’s newer than my library, well, no problem. Open StonePhpSafeCrypt_compressors.php and add your compressor to the list. Instructions are in the file, but it’s really quite trivial.

Subdirectories and require_once()

Because of the way PHP inclusion works, the require directives in a given script are referential to the including script’s location, rather than their own. That means that the includes for the various parts of this library need to be referenced according to their own directory. If you just dump all the files in the same directory as your main project, well great, works fine. If you want to put them in a subdirectory, do the following before you include the library (or it will fail in a fairly unreadable way):

define('STONE_PHP_SAFE_CRYPT_HOST_DIRECTORY', 'dirgoeshere/');

The directory may be absolute or referential, should be in Unix directory format, and needs the trailing slash.

That’s it?

That’s it. Unpack the files, set the salt, $foo = PackCrypt(data, key, options); and look in $foo['output']. Safe, convenient and reliable. This is what it should have been all along.

In case you missed it earlier, get the source here. There is a working example in test.php, but here’s that same fairly minimalist example again, now that you know what it’s doing:

require_once('StonePhpSafeCrypt.php');

$sourcedata = array('Hello', 1, 2, 3=>4, 'Goodbye');
$packed = PackCrypt($sourcedata, 'Password');

if ($packed['state'] === false) { die('Error: ' . $packed['reason']); }

echo 'Packed: ', $packed['output'], '<br />';

$unpacked = UnpackCrypt($packed, 'Password');
echo 'Unpacked: ', print_r($unpacked);

And that’s all there is to it.

The problem? PDF. The solution? FPDF, PHP, and the force of anger incarnate.

So, I’ve got the puzzle drafting application up and running in a non-painful way, which took no small (read: Herculean) effort and a pinch or two of CSS Ninja Magic. This means I have a relatively pleasant interface in which to hand-create my sudoku puzzles, instead of this pencil and paper bullcrap slowing me down.

Of course, I like my puzzles (being an egotist that’s a given, but let’s ignore that for now.) As a result, I want to share them with others, preferably for money. The logical choice is a place like Cafe Press or Lulu, but they both want books kind of already laid out, so they can just print them and be done with them.

This all sounds perfectly reasonable, until that horrible death knell comes gonging around the dark recesses of your animal hindbrain, into places your soul is afraid to go, where ancestral memory remembers stuff like dinosaurs and being hunted by eagles and effective presidents and other scary relics of an antiquarian past.

The name of that knell? PDF.

So, in order to provide content for these places, one has to give up a standardized-ish format which is designed for print. Luckily, postscript doesn’t really much exist in modern tools, so god forbid I be able to go in with a programming language for my layout application, guns a-blazing. (Interject here with fond memories of my father showing me Display PostScript on his NeXTslab as a programming language in which to write actual applications.) At first my plan was to get one of those PDF printer targets like Primo going, then to use Word automation from Delphi to do the actual printing. That worked well enough for my blank boards, but it got hella cumbersome when I wanted to do non-trivial boards, so I started to look for alternatives.

Then I found FPDF, which in many ways rocks. It’s a PHP class from which you derive classes which implement printing behaviors. You override methods to handle when page breaks might occur (for example, to set up column driven layout, keep some variable off in a corner somewhere, and when FPDF says “time to page break,” you override by saying “No, just move ten inches to the left and back up to the top.”) The methods to get things done are relatively straightforward, measured in user-chosen units relative to the edges of the current page, and so on. It’s relatively pleasant, despite a few seriously annoying bugs.

Granted, it seems either that FPDF only implements a subset of PDF, or that PDF is entirely simpler than I had previously believed. Really, you just have a few graphics primitives with simple color and thickness controls for line and fill, the ability to place form elements and the ability to place images. That’s pretty much it, and that surprised me.

But, you know what? That’s all I need for Sudoku. My generation tool now knows how to crap out a homebrew sudoku format (including for all those purty variants I’ve learned about,) and I have a PHP script that turns many instances of that format straight into CafePress ready books. (Actually, with some tweaking, I intend to extend this script to handle other puzzle types, too. I’ve got a genuinely funktronic layout model going on, which I’m getting fairly into.)

So, some standard Sudoku books are on the Very Near Horizon™, as soon as I teach my generator how to put together title pages and instructions pages (there’s a lot of automatic scaling and placement going on, so that’s not as trivial as it sounds) and to place some niceties in the books. You’ll need to buy one to find out what those nicities are, natch.

Right now, my book generator only knows how to draw standard and off-rectangle sudoku (3×3, 4×4, 2×5, et cetera.) Soon, irregular backgrounds. Afterwards, disjoint backgrounds. Promptly following? ”The World”.

I have some very neat things I intend to reveal on that store in the near future. Be sure to go buy things. Right now.