About Me

I'm just someone struggling against my own inertia to be creative. My current favorite book is "Oh the places you'll go" by Dr. Seuss

Saturday, January 24, 2009

JSON xml and the relational model part 4

An xml element is an unordered set of attribute names, and their associated values.

<div class="title" id="heading" lang="en"> </div>


a JSON object is an unordered set of attribute names, and their associated values.

{"tagName":"div", "className":"title", "id":"heading", 
"lang":"en"}


an xml element may contain an ordered tuple of "nodes", which may be plain text nodes, or could be other xml elements.

<div class="title" id="heading" lang="en">
The Grand Adventure of <i>Lucious Swan:</i> The return of elemental qualities.
</div>


a json property may contain an ordered tuple of values, which may be primatives, or could be other objects.


{"tagName":"div", "className":"title", "id":"heading",
"lang":"en",
   childNodes:[
"The Grand Adventure of ",
{tagName:"i", childNodes:["Lucious Swan:"]},
" The return of elemental qualities."
]
}


and that's basically all there is to it. This may appear somewhat more bulky than other xml to json translations. However, the translation preserves the unique and unordered quality of xml attributes, and the ordered non-unique quality of xml node collections. As a result, it's much simpler to implement readers and writers for this format, because there's fewer exceptions or other special conditions to account for. This is more or less a direct translation of the semantics of xml into JSON.

As an additional bonus, code written against this style of structure would work exactly the same directly against a browser dom representation of an xml document, since this is essentially a stripped down subset of the browser DOM, using the same attribute names. Most server-side XML parsers produce essentially the same structure as well.

a JSON translation into xml, using the same principles however, is not so easy...

11 comments:

-TNO- said...

The downside to this approach though is that it requires you to separately know whether a node/value represents a tag, an attribute, or textual content. The Prefix notation that Stefan Goessner uses allows us to easily differentiate between the types just by looking at the JSON structure:

@foo - attributes
foo - tag Names
#text - text node
#comment - comment nodes
#cdata - cdata nodes

This would allow us the ability to tell the difference between a title attribute and a title tag. Another bonus to this approach is that it allows us to maintain simple access to the structure:

html.body.div[0]["@id"]

This blows the DOM out of the water, and keeps true to the language.I think the problem that Stefan Goessner had was not that it couldn't be represented, but that it couldn't be represented while maintaining this simple syntax.

The approach you offer solves the problem of mixed content, but the verbosity is not much better than keeping the original xml representation, which is a pity when you want to do some manipulation.

I think if we took advice from E4X syntax and added a couple symbols on top of Goessner's approach we could avoid the verbosity and still maintain a simple access syntax:

<div class="title" id="heading" lang="en">
The Grand Adventure of <i>Lucious Swan:</i> The return of elemental qualities.
</div>

"div" : {
"@class" : "title",
"@id" : "heading",
"@lang" : "en",
"*" : [
{"$name" : "#text","#text": "The Grand Adventure of "},
{"$name" : "i", "#text" : "Lucious Swan:"},
{"$name" : "#text","#text" : " The return of elemental qualities."}
]
}

div["@class"] // "title"
div["*"][0]["#text"] //"The Grand Adventure of "
div["*"][1].$name // "i"

(I hope this comment system will maintain my formatting)

Personally I don't follow the practice of these two things representing the same thing

Breton Slivka said...

"The downside to this approach though is that it requires you to separately know whether a node/value represents a tag, an attribute, or textual content"

I don't believe it has that downside at all, I'm really quite baffled by that statement. It's an attribute if it's inside of an object. It's a tag if it's inside of an array. It's textual content if it's a string. Really really simple.

"Another bonus to this approach is that it allows us to maintain simple access to the structure:"

Well, you know, simple access to my scheme wouldn't be terribly difficult given a jQuery-like library. (perhaps even a modified jquery itself).

"This blows the DOM out of the water, and keeps true to the language." which language? JSON or XML? It certainly breaks the semantic characteristics of XML.

"[JSON SAMPLE]"

See this is the sort of jumbled nonsense I've been talking about. Say I had a json structure like that- But I'm trying to write a program that pulled out each div element from it. Now I don't know exactly what data I have in advance.

So.. In order to find divs, I would have to first check whether the top level object is an "object" or an "array". then when I determine that it's an object, I would have to loop through each property and check the property NAME for "div", and then, seperately, I would have to check property names for "$name". Then among those I would have to check if the values of the properties with "$name" contain "div". Then to get the contents I would have to check what type the value of my "div" element is, and if it's an object.. uhm.. Then what? If I just wanted to pull simple text content out, I can't even work out how I would do that.

But it's even worse if my div is indicated by "$name", because now I have to check siblings. I have to loop through that same object again looking for "#text".

okay, so now say I've worked through all those problems. Now I want to insert a node into one of my divs. Well if it's one of the $name divs, well.. what exactly do I do here? It doesn't make sense to change the value of "#text" to an array. So I have to change it from a sibling oriented element to an object oriented element, some how. Except that I can't because it's inside an array.

Seriously, I'm stuck here, help me out. What would I do?

As for my scheme, I've matched everything type for type, so there's only ONE way to do everything. There's only one way that you encode a div, and only one operation for each kind of modification. We're not converting apples to oranges, and then sometimes pears under certain conditions, for the sake of a false economy.

Breton Slivka said...

The principle I'm adhering to is paying close attention to the set theory meanings of "unordered set" and "ordered tuple", and finding those structures in xml and JSON. Then I match like for like. This approach leads to many benefits.

Breton Slivka said...

"html.body.div[0]["@id"]"
these demos are misleading, because, when are we ever pulling out the id attribute of a specific element whose nesting characteristics we already know about in advance?

The typical use case for these structures is to write programs against them, with loops or recursion, to reformat or parse useful information out of a structure we probably know very little about in advance. I think we should optimise for the real use cases, not for the cool blog demo, where you're pulling out a one time isolated peice of info.

In this case, in practice, I've found my scheme works far better, because there's fewer special cases that you have to code around. in fact, there's no special cases. none. everything just behaves exactly as it's originally meant to behave.

-TNO- said...

"I don't believe it has that downside at all, I'm really quite baffled by that statement.
"It's an attribute if it's inside of an object. It's a tag if it's inside of an array.
"It's textual content if it's a string. Really really simple."

So is "tagName" an attribute? Is "childNodes" an attribute? They both exist
inside of an object so do I have to check the name of the property as well as what type it is
to determine what to do with it? Does "className" translate to "className" in xml?
Or when I translate, do I also have to rename the attribute first before conversion? I assume textual
content is represented as the property "text"? do the rules differ when I'm dealing with xml instead of
html? how are comments and CDATA sections handled when mixed with content? Treat them as text?

All I'm saying here is that a schema of some sort has to be either explicit in the representation
or implicit in the mind of the developer. I believe if you include it in the syntax you could assume
less about what the target user knows.

"which language? JSON or XML? It certainly breaks the semantic characteristics of XML."

I was referring to use within JavaScript primarily.

"these demos are misleading, because, when are we ever pulling out the id attribute of a
"specific element whose nesting characteristics we already know about in advance?"

Fair argument, I'll throw in a case with minimal knowledge of the structure:

<html>
    <head>
        <title>My Title</title>
    </head>
    <body>
        <a href="http://google.com">Link</a>
        <div id="col1">Content 1</div>
        <div id="col2">Content 2</div>
        <div id="col3">
            Mixed Content with
            <span style="font-weight:bold">Bold</span>
            text
        </div>
    </body>
</html>

var root = {
    "$name" : "html",
    "head" : {
        "$name" : "head",
        "title" : {
            "$name" : "title",
            "#text" : "My Title"
        }
    },
    "body" : {
        "$name" : "body",
        "a" : {
            "$name" : "a",
            "@href" : "http://google.com",
            "#text" : "Link"
        }
        "div" : [
            {
                "$name" : "div",
                "@id" : "col1",
                "#text" : "Content 1"
            },
            {
                "$name" : "div",
                "@id" : "col2",
                "#text" : "Content 1"
            },
            {
                "$name" : "div"
                "@id" : "col3",
                "*" : [
                    {"$name" : "#text", "#text" : "Mixed Content with "},
                    {"$name" : "span", "@style" : "font-weight:bold", "#text" : "Bold"},
                    {"$name" : "#text", "#text" : "Mixed Content with "},
                ]
            }
        ]
    }
}

And assuming the following knowledge:

1 - Its an html document (so I know it has html node and a body node).
2 - Attributes are acessible by name and have the prefix: "@"
3 - Text is accessible by "#text"
4 - Tags are accessible by name
5 - At any level I can access the name of the current tag using the "$name" property
6 - Mixed content is accessible by "*" array

"If I just wanted to pull simple text content out, I can't even work out how I would do that. "

Finding the textual content of divs at the body level if I'm not sure of their existence:

var divs, divText;

divs = html.body.div || (html.body["*"] && html.body["*"].filter(function(el){
    return (el.$name === "div")
}))

Now, since I don't know how many divs are in this html document,
I would have to do the following to obtain the text:

if(divs instanceof Array){
    divText = divs.map(function(el){
        return el["#text"]
    }).join("");
} else if(divs instanceof Object){
    divText = divs["#text"]
}

"Now I want to insert a node into one of my divs."

Appending a new child to an existing node:

if(div["#text"]){ //if textual content already exists, then this will become mixed content
    div["*"][0] = {"$name" : "#text", "#text" : div["*"][0]}
}else if(div["*"]){ //content is already mixed
    div["*"].push({"$name" : "a", "#text" : "Google", "@href": "http://google.com"})
}else{ //no children yet
    div.a = {"$name" : "a", "#text" : "Google", "@href": "http://google.com"}
}

Verbosity of access and updating does indeed jump significantly when you have minimal knowledge, but
as you mentioned yourself, its not terribly difficult to abstract away this with a small library.
Possibly something along the following lines:

html.body.find("div").text(0)
html.body.find("div").attrib("href")

But library syntax is a different issue.

So, back to comparing our representations:

{"tagName":"div", "className":"title", "id":"heading", "lang":"en",
    childNodes:[
    "The Grand Adventure of ",
    {tagName:"i", childNodes:["Lucious Swan:"]},
    " The return of elemental qualities."
    ]
}

vs.

"div" : {"$name" : "div", "@class":"title", "@id":"heading","@lang":"en",
    "*" : [
        {"$name":"#text","#text":"The Grand Adventure of "},
        {"$name":"i","#text"}
        {"$name":"#text","#text":" The return of elemental qualities."}
    ]
}

First, similarites:
"tagName" == "$name"
"childNodes" == "*"
"className" == "@class"

Differences:
My representation uses a full object to represent a standalone text node in an array. this could be
represented by a plain text entry in the array, but that would add more effort to to the parsing
code:

nodeList.map(function(el){
    return el["#text"]
})

vs.

nodeList.map(function(el){
    if(el instanceof String)
        return el;
    else
        return el["#text"]
})

So this portion IMO would be a preference between representation size vs library size

My use of "div" as a property name is used as syntactic sugar to assist navigation and provide a similar interface
as E4X. foo.bar.baz["@id"];

So I guess in summary, my approach is to achieve a middle ground between providing a method of
roundtripping from xml/html to JSON while maintaining a somewhat familiar interface (e4x style)
for node access while keeping javascript code size down. (converting from this structure back to xml is
trivial), at the cost of added JSON size.

Breton Slivka said...

"So is "tagName" an attribute? Is "childNodes" an attribute? They both exist
inside of an object so do I have to check the name of the property as well as what type it is
to determine what to do with it? Does "className" translate to "className" in xml? "

I believe tagName is an attribute, as is childnodes. I tend to think of xml syntax as a kind of syntactic sugar for those semantics. As for classname, in JSON it could just as well be "class", but I used className for compatibility with the xml dom, which made that change due to "class" being a reserved word in javascript. But it's all just a matter of taste. The names could just as well be "$name", "*", and "class", so long as they're in quotes, and JSON demands.

"The Prefix notation that Stefan Goessner uses allows us to easily differentiate between the types just by looking at the JSON structure:"
the exceptional cases that differ from my scheme here are comments and cdata. I tend to feel that cdata is something that only really makes sense in the context of an actual xml document, to escape long stretches of text. This is not really possible or necessary in JSON, so in JSON representation, I think text and cdata are practically the same. The main reasosn you would want to keep the differentiation is if you wanted accurate round tripping. However, the conversion back to either a text node or a cdata node should be based on analysis of the content, and not on some arbitrary marking, I think.

JSON doesn't have comments, so comment nodes may be dropped. There may be some applications which require the retention of comments, so I'll leave that as a very light suggestion. Though I'll admit I cannot think of any.

"This would allow us the ability to tell the difference between a title attribute and a title tag. "
this is not an issue with my scheme, either.

"tagName":"title" is easily distinguishable from "title":"lorem ipsum"

"? I assume textual
content is represented as the property "text""
you assume wrong. Textual content is represented as.... textual content!

"do the rules differ when I'm dealing with xml instead of
html?"
I've not mentioned html. Though it is possible to convert virtually any html document to an xhtml (and thus xml) document using html tidy. from there I don't see any reason to change the rules.


"All I'm saying here is that a schema of some sort has to be either explicit in the representation
or implicit in the mind of the developer. I believe if you include it in the syntax you could assume
less about what the target user knows. "

schemas are a different orthogonal matter. My post is only concerned with the generic conversion of xml to a uniform JSON reprentation, upon which some useful generic operations could be performed without any prior knowledge of any schema. However, in the presence of an actual XML schema (especially a relax NG schema) some very interesting and more schema specific conversions are possible. I may make a blog post about that, if you like.

"divs = html.body.div || (html.body["*"] && html.body["*"].filter(function(el){
return (el.$name === "div")
}))"

this code is taking a great leap of faith that the input data doesn't contain a html.body with BOTH a div property AND a * property. Your script may gaurantee that, if it's properly written. It rather depends on where this document is coming from. I daresay, though, I may be dipping a bit far into the hypothetical here. I have learned through painful experience though, to code defensively and expect the worse. I think my scheme allows fewer of these uncertainties, but I may be wrong.

"if(divs instanceof Array){
divText = divs.map(function(el){
return el["#text"]
}).join("");
} else if(divs instanceof Object){
divText = divs["#text"]
}"

Interesting, but I can't help but wonder what you gain by this. bulky and inscrutable in the json representation, bulky and inscrutable access code. You're not gaining legibility or simplicity on either front.

"this could be
represented by a plain text entry in the array, but that would add more effort to to the parsing
code:"

ah I see...

"nodeList.map(function(el){
if(el instanceof String)
return el;
else
return el["#text"]
})"

wait... what? Why wouldn't it just be...

nodeList.filter(function(el){
return typeof el === "string";
})

No having to toggle or test anything, except for the things that are relevant to the task at hand.

"My use of "div" as a property name is used as syntactic sugar to assist navigation and provide a similar interface
as E4X. foo.bar.baz["@id"];"

This might be personal preference, but I simply find it confusing. In that usage, I have to guess whether there's a single div, or multiple divs. If I guess wrong, I get undefined, so I have to always have twice as much code as is strictly necessary.

"So I guess in summary, my approach is to achieve a middle ground between providing a method of
roundtripping from xml/html to JSON while maintaining a somewhat familiar interface (e4x style)
for node access while keeping javascript code size down. (converting from this structure back to xml is
trivial), at the cost of added JSON size."

this may be so, but I remain yet sceptical for the reasons I have specified. I just can't quite work out what is gained exactly over my scheme in terms of terseness legibility or code size.

-TNO- said...

"you assume wrong. Textual content is represented as.... textual content!"

The reason I brought this up is that I'm trying to determine how you would represent this in your format:

<a href="#">Foo</a>

would this be correct?

{"tagName":"a", "href":"#", "lang":"en", "childNodes":["Foo"]}

It appears consistent, but something about that bugs me for this case of a single text node.


"wait... what? Why wouldn't it just be...:

nodeList.filter(function(el){
   return typeof el === "string";
})"
"No having to toggle or test anything, except for the things that are relevant to the task at hand"

Because that wouldn't work if I had this:

<div>
foo <b>bar</b> baz
</div>

and represented it as such:

"div" :{
   "$name" : "div",
   "*" : ["foo",{"$name" : "b", "#text":"bar"},"baz"]
}

If I'm looking for text, I'm probably looking for the equivalent that innerText/textContent provides, which is partly why I kept
text nodes as objects in mixed content.

"I have learned through painful experience though, to code defensively and expect the worse. I think my scheme allows fewer of these uncertainties, but I may be wrong."
...
"I just can't quite work out what is gained exactly over my scheme in terms of terseness legibility or code size."

In my particular case I've implemented ES4's like operator as a function (...like(JSON,SCHEMA)...) to insure the shape and expectation of my input.
Because of that, it allows me to use foo.bar.baz["@id"] with confidence to keep manipulation code small and avoid the verbose examples given in the above
thought experiment of having little knowledge of my input.If I don't know what kind of JSON you're sending me, and I don't know where I'm going to
find what I'm looking for, there is probably good reason to reject the input anyway if I'm doing something non-generic to the structure (at least in my experience).
But I realize I'm probably an exception since I've also dealt with structures that aren't as simple as hierarchical relationships and contain pointer references.
(JSON extensions used in a couple databases)

"...some very interesting and more schema specific conversions are possible. I may make a blog post about that, if you like."

I'd be interested in seeing another approach to the Schema + JSON relationship.
I'm not 100% certain I like the current proposal that's currently making its rounds: http://www.json.com/json-schema-proposal/
and have kept to an ad-hoc one in my own code.

Breton Slivka said...

""div" :{
"$name" : "div",
"*" : ["foo",{"$name" : "b", "#text":"bar"},"baz"]
}"

Oh, but I thought when you said "vs.", you meant, that's what you thought it would take to do the equivelant with my conversion scheme. However...

{"$name":"div", "*":[{
"$name":"div", "*":[
"foo",
{"$name":"b", "*":["bar"]},
"baz"
]
}]}


and innerText may be implemented as

function innerText(o){
return o === null? null : {
string: o,
array: o.map(innerText).join(""),
object:innerText(o["*"])
}[typeof o];
}

"It appears consistent, but something about that bugs me for this case of a single text node."

does it seem wasteful to you? If you spend any amount of time studying set theory, this makes perfect sense. We're still talking semantically about a tuple, it's just a tuple with a single element. If we wanted to add another, it's just adding another element, instead of transforming the meaning of the object to something else. Also, the innertext function above would have to be bulked out a bit.

Breton Slivka said...

sorry, in my innerText, you may want
o === null

to be
o == null

to filter out undefined properties as well.

-TNO- said...

{string: o,array: o.map(innerText).join(""),object:innerText(o["*"])}[typeof o];

Nice inline trick, I'll have to remember that one.

"does it seem wasteful to you? If you spend any amount of time studying set theory, this makes perfect sense"

Wasteful in representation? Hardly. Nor do I debate the simplicity of it.
My preference in representation is simply an attempt to emulate ECMA-357 as close as possible before
the addition of any library code. Though admittedly my format is domain specific, the extensibility is appreciable.

andry said...

json is a very interesting language to be used. very good tutorial and can hopefully help me in building json in the application that I created for this lecture. thank you