Comments on: Prototype: EEP18 Considered Harmful: The problems with Erlang to JSON term translation http://fullof.bs/prototype-eep-18-considered-harmful-the-problems-with-erlang-to-json-term-translation/ He just never stops talking Fri, 16 Dec 2011 00:30:46 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Chris Anderson http://fullof.bs/prototype-eep-18-considered-harmful-the-problems-with-erlang-to-json-term-translation/comment-page-1/#comment-931 Chris Anderson Thu, 18 Dec 2008 01:19:05 +0000 http://fullof.bs/?p=402#comment-931 In CouchDB it's the e2j(j2e(Json)) == Json we're interested in. We've been using the EEP's mapping for a few months now, without trouble -- we're mostly excited about the possibility of native speed with a C implementation. JSON encoding and decoding is currently our biggest bottleneck. From my perspective, JSON round-trip-ability is the main event. It makes more sense to me, to construct valid-as-JSON Erlang terms, and then convert them for external use. This gives the advantage of the Erlang programmer having full control over the JSON output. (At the expensive of an automatic 2-way Erlang->JSON->Erlang serialization.) Marshaling arbitrary Erlang terms could be done by serializing terms to lists and proplists suitable for 2-way-JSON transformation. I don't feel qualified to speak about the naming issue, but there are enough experienced Erlangers behind the proposal to ease my mind. In CouchDB it’s the e2j(j2e(Json)) == Json we’re interested in. We’ve been using the EEP’s mapping for a few months now, without trouble — we’re mostly excited about the possibility of native speed with a C implementation. JSON encoding and decoding is currently our biggest bottleneck.

From my perspective, JSON round-trip-ability is the main event. It makes more sense to me, to construct valid-as-JSON Erlang terms, and then convert them for external use.

This gives the advantage of the Erlang programmer having full control over the JSON output. (At the expensive of an automatic 2-way Erlang->JSON->Erlang serialization.)

Marshaling arbitrary Erlang terms could be done by serializing terms to lists and proplists suitable for 2-way-JSON transformation.

I don’t feel qualified to speak about the naming issue, but there are enough experienced Erlangers behind the proposal to ease my mind.

]]> By: D Smith http://fullof.bs/prototype-eep-18-considered-harmful-the-problems-with-erlang-to-json-term-translation/comment-page-1/#comment-929 D Smith Tue, 09 Dec 2008 18:25:51 +0000 http://fullof.bs/?p=402#comment-929 > Not exactly even flat list: > 1> binary_to_list(list_to_binary([255])). > “\377″ > 2> binary_to_list(list_to_binary([256])). > ** exception error: bad argument > in function list_to_binary/1 > called as list_to_binary([256]) The \377 is replace in stream by your terminal. It looks like a four character list but it is a single element in the list with an integer value of 255. Believe me it's just your terminal emulator playing with your mind. Don't believe me! Change the character encoding of your TE. > Not exactly even flat list:
> 1> binary_to_list(list_to_binary([255])).
> “\377″
> 2> binary_to_list(list_to_binary([256])).
> ** exception error: bad argument
> in function list_to_binary/1
> called as list_to_binary([256])

The \377 is replace in stream by your terminal. It looks like a four character list but it is a single element in the list with an integer value of 255. Believe me it’s just your terminal emulator playing with your mind. Don’t believe me! Change the character encoding of your TE.

]]> By: Hynek (Pichi) Vychodil http://fullof.bs/prototype-eep-18-considered-harmful-the-problems-with-erlang-to-json-term-translation/comment-page-1/#comment-930 Hynek (Pichi) Vychodil Mon, 08 Dec 2008 17:42:29 +0000 http://fullof.bs/?p=402#comment-930 Not exactly even flat list: 1> binary_to_list(list_to_binary([255])). "\377" 2> binary_to_list(list_to_binary([256])). ** exception error: bad argument in function list_to_binary/1 called as list_to_binary([256]) First one show another problem. There is not only semantic same, but also source code form ("[255]" is not same as "\"\377\""). If you understand JSON as code for JS semantic object, than you can achieve e2j(j2e(X))) == X in semantic manner but not in code. I think it is enough but should be well documented and frequently mentioned. Anyway, I agree with you, that EEP 18 is too much fresh and dangerous to implement as BIF in current state of art. Not exactly even flat list:
1> binary_to_list(list_to_binary([255])).
“\377″
2> binary_to_list(list_to_binary([256])).
** exception error: bad argument
in function list_to_binary/1
called as list_to_binary([256])

First one show another problem. There is not only semantic same, but also source code form (“[255]” is not same as “\”\377\”"). If you understand JSON as code for JS semantic object, than you can achieve e2j(j2e(X))) == X in semantic manner but not in code. I think it is enough but should be well documented and frequently mentioned. Anyway, I agree with you, that EEP 18 is too much fresh and dangerous to implement as BIF in current state of art.

]]> By: John Haugeland http://fullof.bs/prototype-eep-18-considered-harmful-the-problems-with-erlang-to-json-term-translation/comment-page-1/#comment-928 John Haugeland Sun, 07 Dec 2008 16:40:42 +0000 http://fullof.bs/?p=402#comment-928 I'm not that worried about encoding all Erlang terms as JSON, I'm just using that as an example of the flaws in the current EEP. What I am worried about is representing every JSON term in Erlang unambiguously, and with the Unicode requirements of JSON, that isn't even possible in Erlang right now. Above and beyond that, the only way to make strings unambiguous is to present them as binaries, and with Unicode's UTF8 requirements, that means using binaries to represent a varying width encoding, which is a performance nightmare in the making (especially for a language without meaningful manual iteration). See also <a href="http://sc.tri-bit.com/outgoing/scjson%20parser%20halp.html" rel="nofollow">this document</a> that I'm working on. I’m not that worried about encoding all Erlang terms as JSON, I’m just using that as an example of the flaws in the current EEP.

What I am worried about is representing every JSON term in Erlang unambiguously, and with the Unicode requirements of JSON, that isn’t even possible in Erlang right now.

Above and beyond that, the only way to make strings unambiguous is to present them as binaries, and with Unicode’s UTF8 requirements, that means using binaries to represent a varying width encoding, which is a performance nightmare in the making (especially for a language without meaningful manual iteration).

See also this document that I’m working on.

]]> By: Per Gustafsson http://fullof.bs/prototype-eep-18-considered-harmful-the-problems-with-erlang-to-json-term-translation/comment-page-1/#comment-927 Per Gustafsson Sun, 07 Dec 2008 16:27:48 +0000 http://fullof.bs/?p=402#comment-927 <cite> foo_to_bar(X) bifs are currently never lossy I do not think this is correct if you look at list_to_binary and binary_to_list you find that list_to_binary(binary_to_list(X)) == X holds, but binary_to_list(list_to_binary(X)) == X does not hold since X could have been a deep list. This is minor point of course, but I don't understand why you would want to be able to encode any erlang term in JSON. I do not think that this is necessary since there are several Erlang values which makes no sense outside of Erlang e.g. ports and pids. I think it would be more reasonable to define a 1-1 mapping between all JSON terms and a subset of Erlang terms. Which seems to be what EEP 18 does. foo_to_bar(X) bifs are currently never lossy

I do not think this is correct if you look at list_to_binary and binary_to_list you find that list_to_binary(binary_to_list(X)) == X holds, but binary_to_list(list_to_binary(X)) == X does not hold since X could have been a deep list.

This is minor point of course, but I don’t understand why you would want to be able to encode any erlang term in JSON. I do not think that this is necessary since there are several Erlang values which makes no sense outside of Erlang e.g. ports and pids. I think it would be more reasonable to define a 1-1 mapping between all JSON terms and a subset of Erlang terms. Which seems to be what EEP 18 does.

]]>