On the mailing list, people are actively trying to bring Erlang up to snuff with regards to web standards.  One of the more unfortunate choices being discussed is JSON as a data notation.  JSON, unfortunately, does not actually map to Erlang in a useful way.  Joe Armstrong has gone as far as to suggest BIFs, which are decidedly unrealistic as well as unnecessary.  My goal is to create a JSON handling library.  However, the mailing list is beginning to put momentum behind an alternative proposal which is currently presented in BIF form.

This post explains why my approach is different.  Many of the issues herein are discussed by the tabled EEP (EEP 18, “JSON BIFs” by Rickard O’Keefe), but some are not, and some of these issues are accepted when I believe they should not be.  It is my position that EEP 18 is unacceptably dangerous.  I will explain why.

This paper assumes you are familiar with Erlang and with fundamental containers (the list, the array and the key/value map).  It is very helpful, but not required, to be familiar with JSON, or JavaScript or any ECMA derived language such as ActionScript.

Premise

There’s a movement starting to use Erlang for web work.  There are several stumbling blocks to that end.  Among them are a simple primary webserver, a simple primary unicode system and a simple primary JSON manager.

The webserver problem is mostly solved: there’s the httpd module, there’s yaws, there’s mochiweb, there’s the currently unavailable work at Tobbe’s Red Hot Erlang Blog, there’s even Joe’s HTTPD tutorial.  YAWS and MochiWeb in particular get a lot of action these days.  The situation isn’t amazingly straightforward, but it’s fairly straightforward; we’re in “Good Enough” territory.  (I’m building another webserver that behaves like factor’s drop-and-go server, based on Joe’s tutorial, but that’s not for here.)

The unicode problem, however, as well as the JSON problem, are not solved.  Unfortunately, whereas the Erlang community has had the foresight to deal with complex problems in modules first then to move them to syntax later, this process seems to be failing with both JSON and Unicode.  It can be argued that some of the choices made in each process are dangerous.  This community will, by and on the whole, remember the re: module, which is being replaced now with a partially incompatible successor that takes account of limitations and problems in the prior attempt, as well as moves to a stronger RE dialect.  It is important that this ability be retained for JSON and Unicode, both of which are subtly strikingly difficult problems, and both of which are unlikely to be Gotten Right™ on their respective first attempts.

The Principle of Least Surprise

One of the most important parts of writing libraries is to not write nasty shocks into place for users.  In transcoding libraries, there is one rule that defines least surprise more powerfully than any other: round-trip translations must not lose data.  No configuration of EEP 18 can achieve this.  Indeed, it is my position that a one to one translation of JSON to Erlang terms cannot exist, and that any attempt to present a not-1:1 translation as a translation is unacceptable, in that people will expect j2e(e2j(X)) == X, and that cannot be true.  This is especially important given that the suggestion that these translations become BIFs seems to be being taken seriously; foo_to_bar(X) bifs are currently never lossy, and this would create a worrying change in the meaning of several basic naming practices.

It is of critical importance, in my opinion, that we do not provide libraries which fail round-trip conversion in either direction.  At this time, EEP 18 attempts to satisfy this clause with creation-time configurability; I will explain my stance that this is inadequate below.

Why Translation is Unclear

There are, in fact, quite a few problems that prevent 1:1 translation.  We’ll go over them one by one.

1. The notations offer different fundamental containers
   1. Erlang offers dense sequence (“tuple”, {}) and singly linked list (“list”, []) containers.  The erlang standard library offers other containers; I discuss later in this document why I’m not using them.
   2. JSON offers dense sequence (“array”, []) and key-value map (“object”, {}) containers.  That’s it.
2. The notations offer different fundamental datatypes
   1. JSON has a fundamental string type; erlang doesn’t.
   2. Erlang has atoms; JSON doesn’t.
   3. JSON has booleans and “null”; Erlang doesn’t.  For transcoding, pretending they’re atoms creates ambiguity, and is therefore unacceptable.
   4. JSON has explicit support for unicode characters in strings.  Erlang doesn’t have strings at all, but rather lists of characters (in the way that C has arrays of characters).  Those lists are context and usage defined; C++ programmers may think of this as parallel to array strings vs std::string.  Erlang currently has no concept of Unicode (though that’s another issue I’m working on as divergent from the current mailing list / EEP approach.)
   5. JSON and Erlang have very different lists of quoted terms for strings.
      1. Erlang supports embedded octal with shortening, and a bunch of semi-defunct control characters like form feed ("\f") and escape ("\e").
      2. JSON supports 16-bit Unicode character embedding.
      3. Problematically, JSON does not define whether that embedding is UTF16, UCS2 or something else.  Most software implementations assume UTF16.  This document will carefully avoid the issue, which is a serious defect in this document, and a serious defect in JSON.
   6. Erlang terms are byte-available, meaning Erlang programmers may be aware of endianness; JSON objects are not.  This suggests that the handling library needs to either make a choice about internal endianness, or needs to provide control to the user.
3. The notation for similar containers is dissimilar
4. Similar notations are similar, not identical
5. Dangerous string ambiguities

Working from http://sc.tri-bit.com/outgoing/scjson%20parser%20halp.html

More news as I get my butt in gear and finish the libraries in question.  But, yay Varun!

I was a little bored.

So we’re going to have three libraries progressing in the immediate future, with quite a few more over time:

1. HtStub – An embeddable, zero-config, zero-behavior secure Erlang webserver
2. SC A-Star – An efficient, modular ECMAscript (flash/actionscript, javascript, jscript etc) A* implementation with support for custom grid geometries (includes algorithm tutorial)
3. TestErl – unit, regression and stochastic testing for Erlang
4. ScUtil – a fairly large list of gap filling functionality for Erlang

In the near future, I will add C++ and PHP libraries, as well as many some libraries for more obscure languages like FormulaONE, Mozart-Oz, Factor and maybe (sadly) Delphi.  I have more than 30 libraries ready for release.

All those repos are pretty empty at the moment.  That will change in coming days, and I’m sure I’ll post lots of boring little snippets here about whatever minor new thing my crap does.

Yay!

1. An incomplete transfer of the requisite files.  Don’t insist that you’re certain that didn’t happen, even if it’s from a command like svn or cp that shouldn’t fail; you’re not certain until you’ve checked.
2. PHP has run out of some resource, typically RAM.
3. mod_security is set up brokenly

[digg-reddit-me]In both cases, you can figure out which by checking your apache logs.  On windows, go to the Windows Event Viewer.  On unix, this may live in a variety of places; most common is shared hosting by cpanel/plesk, where you can get it in your control panel, or to just look in /var/logs/ .

If it’s #1, you’re likely to see something like this in logs (this is from my site, which just suffered this problem and was quickly fixed) :

[Tue Jul 15 18:45:28 2008] [error] [client 24.117...]
PHP Fatal error:  Call to undefined function
force_ssl_admin() in /var/www/html/wp-settings.php on
line 390, referer: http://.../wp-admin/post.php?action=edit&post=55

Don’t worry if that undefined function has a different name or a different referrer, or whatever; that’s how you track down missing code, and missing code means some file didn’t get updated.

I’m already doing complex efficient randomization, API access, bulk posting, timed bulk posting, base autotagging, auto-categorization, a catalog widget, and I’m going to make sure that my stuff is compatible with the All In One SEO pack.  I’m going to provide integration points, and I’m going to provide an example integration with LightBox, or one of its relatives.  I’m going to provide voting, moderated tag suggestion and home-post permalinking.

I’ll also be writing strict, browser/version portable code without hacks.  Yay!

Please let me know what you’d want to see if there were a new image plugin coming out.

Now, I think it’s a little impractical, and it leads to a mind-bleedingly weird syntax, but it’s actually struck me that this is a very clean, very terse format for providing something akin to uniform construction behavior when you’re dealing with multiple manufacture paths to a given object. Granted, it’d be an even bigger, even uglier hack, but still, it’d work.

Witness syntax:

function exampleF(foo, bar, baz) { … }.defaults(1, 3, 7);

Hear that crashing sound? It’s your mind, breaking. But, as much as I hate to admit it, this is one genuinely lovable hack. Bravo.

• Suppress the user interface. IE starts with most of its interface turned off, but a few things aren’t. Notable things to include are to suppress the context menu and keyboard navigation.
• There are several ways to suppress the context menu. The easiest way is to do it in the html. Add oncontextmenu=”return false;” to your <body> and the problem just goes away, and it’s easily manually overridden for specific elements later.
• Alternately, you can override the interface programmatically through IDocHostUIHandler.
• Get rid of the OnNavigate noise (the clicking sound when you hit a link.) This is harder than it should be.
• Prevent keyboard navigation. If you only ever navigate once, and then do everything else in DHTML, this isn’t an issue, because there’s nothing to go forward or back to. However, if you need to, the way to do this is to capture DWebBrowserEvents2::BeforeNavigate2() and fill its last parameter, VARIANT_BOOL*& Cancel, to true. For security reasons this can’t be done in HTML without a garish ugly dialog box confirmation.
• Prevent dragging and dropping links. Do this with BeforeNavigate2() just like keyboard navigation.
• Prevent selection. The easiest way to do that is in the HTML by adding onselectstart=”return false;” to the <body>, which is also easily overridden for child elements where appropriate. If you need it programmatically, such as a block which can be turned from and to editability, that’s in IDocHostUIHandler.
• Handle some keys directly. Particularly often, control keys, tab and f-keys need very different interpretations in applications than are the defaults in IE. Some keys can be reliably intercepted in JScript, but not all of them. For lightweight stuff, use JScript; it’ll be less painful. However, if you need real control, you need to provide an IDocHostUIHandler::TranslateAccelerator() implementation.
• Suppress the interior border. People are often surprised that they turn off the window border in IE, and yet it seems to still appear. That’s because IE’s default stylesheet puts an inset bevel on every web page. Turn that off in CSS by writing body { border: 0; }.
• Begin to embed your images and other support media as resources, and access them with the res:// protocol.
• One ugly caveat: I have never found any variation of the DirectX PNG alpha filter hack which works with res://, and I’ve looked often and at some very weird variations. It is my current belief that a PNG cannot be embedded through res:// and still be fixed for alpha. This leads to the unfortunate case of requiring at least part of your resources on disk. If anyone knows a way around this, a heads-up would be greatly appreciated.
• Set up a system to exchange messages between IE DOM/DHTML/JScript and your application. There are a bujillion ways to do this, but I tend to use a combination of intercepted element events (mostly OnClick in WebBrowserEvents Invoke,) DOM extension behaviors that allow me to call C++ directly from JScript through IDocHostUIHandler::GetExternal(), and executing scripts directly on the DOM object through IHTMLWindow2::execScript().
• Build everything in the to be inside a container; make the body itself overflow: none. This prevents several ugly re-layout quirks which we’re used to on the web but not in applications, when content gets long.
• Start the container invisible, and make the black. That way, during the lag while resources are being loaded, you have a reasonable appearance. Black flashes look correct. White flashes look broken.
• Set an onload=”HandleLoad();” function for the <body>. That way, you know when all resources are loaded, and thus when it’s safe to take the invisible clause off of the main container.
• Set the main container position:relative, to make absolute positioning of contained elements easier.
• Build your web page in as a resource, and load it through IHtmlDocument2::write(). This allows you to load a document at runtime without using about:blank (which most people use, but which causes a brief white flash before your application loads; very unprofessional looking.) You’ll need SafeArrayAccessData() and SafeArrayUnaccessData() to use write() safely. When it comes time to skin or internationalize, and when you’re dealing with a compile cycle which is just linking, you’ll thank me for this.
• Suppress dragging on links and buttons. If you don’t, people will be able to drag them outside the app and onto the desktop as shortcuts, and when followed those shortcuts will lead into your HTML page from the outside (which is, surprisingly, legal.)
• Consider compressing your executable, such as with UPX, which makes resources unreadable externally.
• Set an id on your <body>. The is considered the origin of most events which don’t have a clear origin, and that means that catching these events is a lot easier (catching by id is the most straightforward way to sort out sources.)
• Resist the urge to use runtime styles. They’re not worth the problems they cause.

That should get the new IE user through some common foibles. I’ve probably missed stuff; if you can think of something, lemme know. I do not currently know of a way to embed flash, unfortunately, without leaving available an extra file.

Or does it?

There is one simple truism that has shown itself over and over again in programming. With the new toolset available, I don’t think there’s anyone who’s going to argue that web programming isn’t programming anymore. It’s becoming the case that what we learned in software engineering is starting to show true in web development. In this respect, XMLHttpRequest shows several promises which aren’t immediately obvious, and several pitfalls which apparently haven’t been noticed at all.

The truism to which I refer is that nothing is ever enough for a programmer. That’s not as broad as it sounds; it leads to a few very specific and very important realizations which need to be accounted for.

1. Things need to be modular, because we’ve scaled too far to even consider looking at systems as a whole for small changes. This is one of the web’s strongest points, in large part owing to the fundamental nature of SGML as a meta-markup and the brilliant guiding hand who was John Postel. In this respect, XMLHttpRequest is well on its way to success; the standard is beginning to take into account languages other than ECMA derived scripts, and sockets are already extremely modular.
2. Things need to be resilient. There are adequate mechanisms to determine state and error state, though a callback for connection failure would have been nice (though, admittedly, it’s easy enough to implement, which I’ll show in a later blog post.)
3. Things need to attach to their host environment in a natural fashion. This has traditionally been a problem for languages like FormulaONE and Prolog, though it seems Mozart-Oz is working them out. This is arguably XMLHttpRequest’s greatest strength, the foresight of the responseXML property. Unfortunately, it also helps mask one of the biggest failings (the only serious one that I see, actually.)
4. There will come a point at which the programmer needs to do something that isn’t covered. Languages, libraries, interfaces and settings which allow the programmer to get to the meat and start replacing things are extremely desirable. Various examples of this abound; few have had as much success as the Standard Template Library or URLs. In this fashion, XMLHttpRequest appears to have its bases covered, with the responseText mechanism. You can recieve things that aren’t XML and work with them how you see fit, which should open up entirely new vistas of usability and blah blah blah if you’re just willing to do some string parsing. The idea is that since you can do raw parsing, and since these are sockets, that that allows you to use the network openly. Herein lies XMLHttpRequest’s secret flaw – requestText is crippled by the connection mechanism. This isn’t actually good for much more than porting pre-XML legacy web services.

See, I started writing this AJAX tutorial, and I was getting all into it. I started by writing a lightweight DOM calculator to get people used to working on the document model from a scripting language. It went well, and Uncle Fatty was happy. The idea was to move from that to writing a chess game. Since I used to work for one of the big chess service providers, I decided that what I actually ought to write was a client which touched their servers.

Ha ha. Joke’s on me. See, I fell for it too: if it does responseText, then I can just parse whatever I get manually, and everything’ll be flowers and syrup, right? So, I start by porting my old style13-2 and style12 parsers, so that I can touch FreeChess, ChessClub and ChessLive. Porting them into ECMA was moderately tehsuck, but to be honest it coulda been worse, so I’m not complaining much. Built those on edit boxes, fed it a command, it reacted, and eventually I found the right neurons to make the frog jump, and everything was hoppy. Spent some time nailing stupid effects from Rico and Scriptaculous onto it, and generally had myself a fun waste of a couple of hours. Even hammered out a froody little chat system.

Then, it’s time to get down to the horror that is nailing XMLHttpRequest onto the service, right? Because I can parse whatever I receive, so it’ll be candy and bee-farts, right? Ha, ha: you sir got screwed.

Y’see, folks, it’s called XMLHttpRequest for a reason. It’s not really sockets, no matter how much .responseText wants to trick you into thinking it is. It doesn’t matter if you can parse whatever random damn thing you receive, because of one ugly detail: there is no way to make a connection without sending HTTP headers first. Now, I’m sure several people laughing at me for being stupid right now. On the one hand, I did earn it, and so I deserve it; on the other hand, shut up. At first I thought I was going to fake it by turning off all the headers, but it turns out that the definition of setRequestHeader essentially prohibits it from being possible at all. Thing is, it’s actually very easily reparable. Now, let’s pretend for a moment that security is the province of the browser author and not the specification, and that raw sockets haven’t existed safely in Flash for quite a while already. Raw sockets themselves aren’t fundamentally dangerous, and because of the way buffering is bound up in the object, it’s actually a pretty safe playground.

The more important bit, though, is how ridiculously powerful such a tool would be. That would, in many ways, be what X always wanted to be: a strong, powerful, scripted, media-rich and gracefully degrading platform for semi-thin network applications. This has the advantage of having all the “server” behavior bound up in the web browsers, already far better standardized and far more powerful than X ever was (I’m sure I’ll take flak for saying that.) My chess client is a good example of what could be powerful here: if I could just say “no, don’t send the HTTP headers,” suddenly I would have a portable rich chess client in trivially little code.

As of now, I have to write a stupid little server whose entire purpose is to intercept AJAX, strip the http headers away, then act as a passthrough to the real server. It’s a waste of machine time, a waste of bandwidth and of coding effort. And, the thing is, I think if we don’t change the XMLHttpRequest object to accomodate, we’re going to see a lot of just that kind of behavior. Sure, the original intent of the object was to allow web browsers to embed browsing behavior in their scripting, essentially allowing scripts to act like clients. That said, sometimes a mechanism is far more powerful than its original intent (Tim Berners-Lee’s invention of a way to keep data around the particle collider up to date is a beautiful example, because I’m using its grandchild to put this argument together right now.)

Sometimes small modifications to allow the programmer better control are worth their weight in gold. Yes, I realize it was an HTTP request mechanism. I think it shouldn’t be. The allowance is made for raw receiving in responseText, because the utility is obvious. The allowance should be made for raw sending, too. I believe it’s significantly important. This would be a major step towards portable rich applications with network capabilities. The actual implementation shift is tiny, well-understood and safe. This really, really needs to be done.

Of course, once you do that, another fairly serious issue rears its head: that this is a one-directional avenue for communication. We want to build deeply connected apps, here! We always have, back through SOAP and DCORBA and X11 and UUX and so on. There are several docs out there dedicated to the issue of emulating server push, many of which use browser extensions or to get the job done. A much more common answer, since it’s portable, is polling, which is immensely work- and bandwidth-wasteful, and puts a direct server tax on reducing lag (the faster they poll, the harder you have to work to cover something that really should just be data being sent out.) There are already tons of applications which work this way, and it’s just going to get worse. However, there is an unexplained parameter in the connection list called “async” which may handle exactly this; if so, the second set of needs is obviated.

On those grounds, I recommend the following alterations to the XMLHttpRequest standard:

Overview:

To support the direction that AJAX is already taking itself and to reduce the implementation cost of certain commonly desired features, we will define mechanisms to implement raw connections aside from the default HTTP connections, and to recieve server pushed data without extraneous polling mechanisms. This requires minor but significant alterations to several parts of the proposed standard, and though the implementation cost of these changes is small, it requires a significant re-evaluation of the potential of the mechanism.

In the definition of open(), the modes GET, POST and PUT are required. Two other methods are questionned, suggesting that there is still flexibility in the mode list. That’s good: this is the natural place to add raw connection. I propose a fourth mode: RAW.

1. It seems appropriate that RAW be placed in the MUST support group; things can be supported and turned off for security reasons, and it’s important that we be able to write software which can check for these things and then move on.
2. RAW suppresses all HTTP negotiation and all HTTP headers. Headers may not be set with setRequestHeader() on a RAW method connection.
3. The current send-recieve readyState reporting mechanism would not be sufficient for RAW. Likely the most appropriate thing to do would be to add another enumerant to the state to represent bidirectional transfer, and to change the state transition rules for RAW sockets. This seems likely to result in the most amenable environment to older code. The state changes would need to be altered to allow send() on a RAW socket during any state other than 0. Because of the editorial note suggesting that readyState transition be amended for HEAD method requests, it seems the editorial body is already amenable to different send() semantics for different readyStates.
1. There is the question of how to handle sequential input and sequential output. Receipt on a RAW socket should probably just append onto the existing buffer, as well as attempting to send while send is already underway (or, the current exception behavior could be maintained for the latter case; that’s a matter of opinion, but I believe queueing is appropriate.)
2. There is also the argument that each distinct read be cached in a seperate entry in an array, but without the HTTP transport, there would need to be a way to distinguish packet breakup from seperate transfer; it seems inappropriate, especially that this is meant to be a raw socket, and as such I believe the former answer is superior.

Actually, the second point as a casualty covers server push, too: with the bidi mode in place, you can safely handle any state transition, so we can just happily fire lots of state changes to cover each server push. This makes async (probably) unnessecary. Because many deployed AJAX codebases don’t account for such situations, it may be appropriate to keep the current behavior stable, and to add another HTTP request type that has RAW-style state transitions (this would also neatly handle the HEAD problem) called, say, PUSH, which would allow the immediate deployment of network applications which aren’t tied to the client-server pipe mindset.

This approach has several advantages:

1. This approach is similar to concepts already under discussion and uses similar mechanisms, but covers several under consideration issues in less work, and in a way which opens up an extremely powerful new tool.
2. This approach would have essentially zero impact on deployed code, and would cause no disruptions to existing deployments.
3. This approach can be handled quite easily in a way to degrade gracefully in situations where a polling style mechanism or something similar provides an adequate substitute.
4. This approach drastically reduces idle bandwidth for certain network connection behaviors.
5. This approach is extremely easy to implement at a browser level, consisting essentially of work that’s already done with bits ripped out. It is likely to be extremely quickly deployed if adopted by a standards body. Hacks to approach both of the behaviors this allows are already becoming widespread. We need to open this floodgate early so that it can be handled in a standard fashion; browsers are already starting to expose their guts, and there’s a possible browser incongruity in the near future if we don’t get this handled right now.
6. This provides an extremely pleasant new avenue for thin client development, and will catch like wildfire in the existing AJAX crowd.

Please give these changes serious consideration. If you support them, get some trackbacks going and get the word out. This is the sort of thing that not too many people will see the importance of, so it’s important that the word get around so that the right people can pick up the sound. If you don’t agree, tell me why.

XMLHttpRequest is in the hands of two people, and small groups of people are often more likely to listen than large groups. Their minds can be changed. Want real HTML network clients? Help me change them.