A better approach to MySQL row generation

SQL, SQL 2 Comments

I love Markus Winand.  Use the Index, Luke taught me a lot about SQL.

Markus has produced a coping strategy for some of MySQL’s limitations in generating sequences.  I present an alternative form which produces arbitrary sized ranges, which he seems to believe is not possible.  Ah, Markus: it is.

So, this article is actually wrong.  There’s a call-time limitation I didn’t know about: apparently in MySQL you can’t call a stored procedure from a query.  That renders this seriously useless.  I’m leaving the article up just for the sake of permanence.

This advice is wrong.  Do not believe it.  I was in error.  It is kept only for posterity.

-- -------
--  in re: http://use-the-index-luke.com/blog/2011-07-30/mysql-row-generator
--  No, this way is not our only hope: there is another.

drop procedure if exists nseq;

delimiter //

-- -------
--  Use union all to generate a 2-row table.  
--  Use log 2 to find power expansion count to cap the desired size.
--  Use cross joins to expand our 2-row to the smallest superior table.
--  Select the diminished part to get our sequence.

create procedure nseq(in upto integer)

    declare itLeft integer default -1;

    set itLeft = ceil(log2(upto));

    set @front := 'select rownum from (select @rownum := @rownum+1 as rownum
                   from (SELECT @rownum := 0) r, (select 1 as hidden union
                   all select 2) a0';
    set @upTo  := upto;

    set @interior := '';
    while (itLeft > 1) do
        set @interior := concat(@interior, ' cross join (select 1 as hidden
                         union all select 2) a', cast(itLeft as char));
        set itLeft    := itLeft - 1;
    end while;

    set @back  := ') sl where rownum <= ?;';
    set @query := concat(@front, @interior, @back);

    prepare stmt from @query;
    execute stmt using @upTo;  
    deallocate prepare stmt;  

end //

delimiter ;

Hopefully he will agree, or explain what I didn’t understand in his needs.  :)

Erlang jokes are too rare

Erlang, General Interest, Miscellaneous No Comments

But this one’s funny.


Reminders are nice

Erlang, Miscellaneous, Programming No Comments

Always nice when someone lets you know, y’know?

[19:43] <apples> by the way StoneCypher, if you're curious, the level
          of functionality provided by my SMTP server in C, which is
          ~1000 lines, i trimmed down to less than 100 in erlang :p
[19:45] <StoneCypher> lel
[19:45] <StoneCypher> and i bet it's less buggy now too
[19:45] <apples> quite
[19:46] <apples> oh, and i wrote it in an hour
[19:46] <StoneCypher> awesome
[19:46] <StoneCypher> can i post that quote on my blog?
[19:46] <apples> sure thing

Helps one remember one’s doing the right thing to push people to learn new things.

Link: Programmer Competency Matrix

Miscellaneous, Programming No Comments

I originally found the PCM through Starling Software‘s copy, but it turns out to originally be by Sijin Joseph.  As both sites have lots of other interesting content, I’m linking both.

The Programmer Competency Matrix is a surprisingly realistic grid of capabilities sorted by topic, which gives an extremely rough but in my opinion basically valid idea of an engineer’s gumption.  Starling republished it as part of their standard set of tools to be used during a hire.

Regardless of whether you agree with the grid or its data, it’s fun.  Have a look, and try scoring yourself.

WebKit overflow:hidden and clear:none clipping failure

Programming, Web and Web Standards No Comments

There is a failure in WebKit to honor overflow:hidden with clear:none on child elements.  This prevents several cheesy no-table space filling strategies which have worked portably since IE3 from working in WebKit derived browsers (safari, chrome, etc.)

Test case with screen shots and examples.

I have submitted this as Bug 23221.

A Better Erlang TCP listening pattern: addressing the fast packet loss problem

Erlang, Programming, Tools and Libraries, Tutorials 11 Comments

[digg-reddit-me]I’ve had mixed reactions to this when I’ve discussed it with people on IRC.  This may be well known to oldbear Erlang programmers.  I suppose it’s also possible that I’m wrong, though I’ve talked to several people I respect, and more than one of them have suggested that they were already aware of this problem.  If I’m wrong, please let me know; I’m open to the possibility that there’s a better answer that I just don’t know about.  I’ve never seen it discussed on the web, at least. Update: Serge Aleynikov points out that this TrapExit tutorial documents this approach.

I think this is probably real.

I believe there is a significant defect in the idiomatic listener pattern as discussed by most Erlang websites and as provided by most Erlang example code, and which is found in many Erlang applications.  This defect is relatively easily repaired once noticed without significant impact on surrounding code, and I have added functionality to my ScUtil Library to handle this automatically under the name standard_listener.

The fundamental problem is essentially a form of race condition.  The idiomatic listener is, roughly:

do_listen(Port, Opts, Handler) ->

    case gen_tcp:listen(Port, Opts) of

        { ok, ListeningSocket } ->
            listen_loop(ListeningSocket, Handler);

        { error, E } ->
            { error, E }


listen_loop(ListeningSocket, Handler) ->

    case gen_tcp:accept(ListeningSocket) of

        { ok, ConnectedSocket } ->
            spawn(node(), Handler, [ConnectedSocket]),

        { error, E } ->
            { error, E }


Now consider the case of a server under very heavy load.  Further, consider that the listening socket is opened either {active,true} or {active,once}, which is true in the vast bulk of Erlang network applications, meaning that packets are delivered automatically to the socket owner.  The general pattern is that the listening socket accepts a connection, spawns a handling process, passes the connected socket to that handling process, and the handling process takes ownership of the socket.

The problem is that it takes time for that all to happen, and Erlang doesn’t specify or allow you to control its timeslicing behavior (as well it should not).  As active sockets are managed by a standalone process, this means that if the connecting client is fast and the network is fast, the first packet (even the first several under extreme circumstances) could be delivered before the socket has been taken over by the handling PID, meaning that its contents would be dispatched to the wrong process, with no indication of where they were meant to go.  This invalidates connections and fills a non-discarding mailbox, which is a potentially serious memory leak (especially given that erlang’s response to out of memory conditions is to abort an entire VM.)

Obviously, this is intolerable.  There are better answers, though, than to switch to {active,false}.  One suggestion I heard was to pre-spawn handlers in order to reduce the gap time, but that doesn’t solve the problem, it just makes it less likely.

The approach that I took is to lie.  standard_listener takes the following steps to resolve the problem:

  1. Add the default {active,true} to the inet options list, if it isn’t already present.
  2. Strip out the {active,Foo} from the inet options list, and store it as ActiveStatus.
  3. Add {active,false} to the inet options list, and use that options list to open the listener.
  4. When spawning handler processes, pass a shunt as the starting function, taking the real handling function and the real ActiveStatus as arguments
  5. The shunt sets the real ActiveStatus from inside the handler process, at which point the socket begins delivering messages

This neatly closes the problem.  A free, MIT license implementation can be found in ScUtil beginning in version 96.  A simplified, corrected example follows for immediate reference; the thing in ScUtil is more feature complete.

do_listen(Port, Opts) ->

    ActiveStatus = case proplists:get_value(active, SocketOptions) of
        undefined -> true;
        Other     -> Other

    FixedOpts = proplists:delete(active, SocketOptions)
             ++ [{active, false}],

    case gen_tcp:listen(Port, FixedOpts) of

        { ok, ListeningSocket } ->
            listen_loop(ActiveStatus, ListeningSocket);

        { error, E } ->
            { error, E }


listen_loop(ActiveStatus, ListeningSocket, Handler) ->

    case gen_tcp:accept(ListeningSocket) of

        { ok, ConnectedSocket } ->
            spawn(?MODULE, shunt, [ActiveStatus,ConnectedSocket,Handler]),
            listen_loop(ActiveStatus, ListeningSocket, Handler);

        { error, E } ->
            { error, E }


shunt(ActiveStatus, ConnectedSocket, Handler) ->

    controlling_process(ConnectedSocket, self()),
    inet:setopts(ConnectedSocket, [{active, ActiveStatus}]),

Making significant strides towards documenting ScUtil

Erlang, Programming, Tools and Libraries 6 Comments

[digg-reddit-me]My erlang library ScUtil has been public at the ScUtil page for some time now.  Recently, I’ve been working on documenting it.  It’s about half documented in its current state.

Here’s the prototype ScUtil documentation.  I’m open to commentary.

Prototype: EEP18 Considered Harmful: The problems with Erlang to JSON term translation

ECMA / Javascript, ECMAScript, Erlang, General Interest, Programming, Rants, Tools and Libraries, Web and Web Standards 5 Comments

THIS IS ONLY HALF WRITTEN.  I have been sitting on this post, waiting for the mood to finish it, for months; because EEP18 is now being treated as a likely implement, I am immediately publishing the half-written version, because it exposes many (though not all) of the serious, irreconcilable problems with EEP18.

On the mailing list, people are actively trying to bring Erlang up to snuff with regards to web standards.  One of the more unfortunate choices being discussed is JSON as a data notation.  JSON, unfortunately, does not actually map to Erlang in a useful way.  Joe Armstrong has gone as far as to suggest BIFs, which are decidedly unrealistic as well as unnecessary.  My goal is to create a JSON handling library.  However, the mailing list is beginning to put momentum behind an alternative proposal which is currently presented in BIF form.

This post explains why my approach is different.  Many of the issues herein are discussed by the tabled EEP (EEP 18, “JSON BIFs” by Rickard O’Keefe), but some are not, and some of these issues are accepted when I believe they should not be.  It is my position that EEP 18 is unacceptably dangerous.  I will explain why.

Read the rest…

Holy crap, an Objective C text that doesn’t assume you’re retarded

General Interest, Objective C, Programming, Rants 5 Comments

[digg-reddit-me]My good friend Jeff happened to mention offhand his knowledge of a document I’ve been looking for for quite some time now.  I’m sharing it with my readers in case they’re looking for something similar.

Let me be forward: I cannot stand the various Objective C books I’ve tried.  They all want to teach me to be a programmer.  I’m already there.  I just want a book like Stroustrup.  The PragProg book is awful: the first several chapters are about Mac development tools, like I give a damn.  Everything’s through interface wizards.  It’s nauseating.

Jeff heard mein painz0rz, and turned me on to From C++ to Objective-C.  It isn’t perfect: it’s not super comprehensive, and it’s translated from a different native language (French), which leaves a few passages cumbersome.  However, as one can tell from reading the intro, the author of the document, much like me, found little to love in the state of Objective C documentation, and wanted to write something for people who were already well established.

Kudos to Pierre Chatelier for writing the book that Apple and Alan Kay could not.

Yay for Varun! Open Source Inbound

ECMA / Javascript, ECMAScript, General Interest, Programming, Tools and Libraries No Comments

My boss’ boss, Varun, is letting me open source some of the work I’m doing at Kayako.  I’m not supposed to talk about the interesting three until they’re ready for release, but I can tell you that a JavaScript ISO8601 implementation is among them, and that they’re all going to be MIT licensed, no GPL contamination.

More news as I get my butt in gear and finish the libraries in question.  But, yay Varun!

« Previous Entries