side effects and global variables
variable1 = 0;
variable2 = 1;
...
foo();
bar();
bundling state into a context structure
struct State;
State ctx;
...
foo(ctx);
bar(ctx);
bundling the data and the methods together
class State;
ctx = new State;
...
ctx.foo();
ctx.bar();
Define the computation as a "type"
monad ctx;
...
dosomething = ctx(state).
foo().
bar();
dosomething();
Thursday, March 29, 2012
Monday, November 21, 2011
Securing the Web, appendix
ManifestML
*(requires id attribute)
<!-- -->
<!DOCTYPE>
<a>*
<applet>*
<audio>*
<base>
<body>*
<canvas>*
<embed>*
<form>*
<head>
<html>
<iframe>*
<img>*
<link>*
<meta>
<noembed>
<noframes>
<noscript>
<object>*
<param>
<script>* (only src attribute, no inline script)
<style>* (but only css @ directives allowed inside)
<title>
<video>*
<source>
SemanticML
(no html comments, id attributes or inline event handlers)
<a>
<abbr>
<acronym>
<address>
<area >
<b>
<bdo>
<blockquote>
<br>
<button>
<caption>
<cite>
<code>
<col>
<colgroup>
<command>
<datagrid>
<dd>
<del>
<dfn>
<dialog>
<dir>
<dl>
<dt>
<em>
<figure>
<h1>
<h2>
<h3>
<h4>
<h5>
<h6>
<hgroup>
<hr>
<i>
<img> (restricted to #fragment refs)
<ins>
<kbd>
<li>
<map>
<mark>
<meter>
<ol>
<output>
<p>
<pre>
<progress>
<q>
<rp>
<rt>
<ruby>
<s>
<samp>
<small>
<span>
<strike>
<strong>
<sub>
<summary>
<sup>
<table>
<tbody>
<td>
<tfoot>
<th>
<thead>
<tr>
<tt>
<u>
<ul>
<var>
LayoutML
(with src= attributes that can point to #fragments of ManifestML)
*(requires id attribute)
<article>*
<aside>*
<details>*
<div>*
<footer>*
<form>* (restricted to #fragment refs)
<header>*
<menu>*
<nav>*
<section>*
<sidebar>*
FormML
<a>
<br>
<button>
<caption>
<col>
<colgroup>
<datalist>
<em>
<fieldset>
<form>
<input type="button">
<input type="checkbox">
<input type="color">
<input type="date">
<input type="datetime">
<input type="datetime-local">
<input type="email">
<input type="file">
<input type="hidden">
<input type="image">
<input type="month">
<input type="number">
<input type="password">
<input type="radio">
<input type="range">
<input type="reset">
<input type="search">
<input type="submit">
<input type="tel">
<input type="text">
<input type="time">
<input type="url">
<input type="week">
<input>
<keygen>
<label>
<legend>
<optgroup>
<option>
<p>
<select>
<strong>
<table>
<tbody>
<td>
<textarea>
<tfoot>
<th>
<thead>
<tr>
*(requires id attribute)
<!-- -->
<!DOCTYPE>
<a>*
<applet>*
<audio>*
<base>
<body>*
<canvas>*
<embed>*
<form>*
<head>
<html>
<iframe>*
<img>*
<link>*
<meta>
<noembed>
<noframes>
<noscript>
<object>*
<param>
<script>* (only src attribute, no inline script)
<style>* (but only css @ directives allowed inside)
<title>
<video>*
<source>
SemanticML
(no html comments, id attributes or inline event handlers)
<a>
<abbr>
<acronym>
<address>
<area >
<b>
<bdo>
<blockquote>
<br>
<button>
<caption>
<cite>
<code>
<col>
<colgroup>
<command>
<datagrid>
<dd>
<del>
<dfn>
<dialog>
<dir>
<dl>
<dt>
<em>
<figure>
<h1>
<h2>
<h3>
<h4>
<h5>
<h6>
<hgroup>
<hr>
<i>
<img> (restricted to #fragment refs)
<ins>
<kbd>
<li>
<map>
<mark>
<meter>
<ol>
<output>
<p>
<pre>
<progress>
<q>
<rp>
<rt>
<ruby>
<s>
<samp>
<small>
<span>
<strike>
<strong>
<sub>
<summary>
<sup>
<table>
<tbody>
<td>
<tfoot>
<th>
<thead>
<tr>
<tt>
<u>
<ul>
<var>
LayoutML
(with src= attributes that can point to #fragments of ManifestML)
*(requires id attribute)
<article>*
<aside>*
<details>*
<div>*
<footer>*
<form>* (restricted to #fragment refs)
<header>*
<menu>*
<nav>*
<section>*
<sidebar>*
FormML
<a>
<br>
<button>
<caption>
<col>
<colgroup>
<datalist>
<em>
<fieldset>
<form>
<input type="button">
<input type="checkbox">
<input type="color">
<input type="date">
<input type="datetime">
<input type="datetime-local">
<input type="email">
<input type="file">
<input type="hidden">
<input type="image">
<input type="month">
<input type="number">
<input type="password">
<input type="radio">
<input type="range">
<input type="reset">
<input type="search">
<input type="submit">
<input type="tel">
<input type="text">
<input type="time">
<input type="url">
<input type="week">
<input>
<keygen>
<label>
<legend>
<optgroup>
<option>
<p>
<select>
<strong>
<table>
<tbody>
<td>
<textarea>
<tfoot>
<th>
<thead>
<tr>
Thursday, November 17, 2011
Securing the Web
(A bit of a departure from my ironically complicated KissML idea today)
An interesting problem with the web is that the security model is a little bit messed up. Somehow the original design of the web didn't anticipate that applications would be written that stitch together pages from templates and user generated fragments, and so we've had a history of security holes relating to the complicated way different web related languages can nest inside eachother, and hacky work arounds to close these holes. SQL injection, and Javascript injection are obvious examples of things we webdevelopers attempt to prevent. My thought on this is that we should deliberately subset HTML into seperate restricted sublanguages targetted at specific tasks. The two subset languages I am proposing are ManifestML and SemanticML; There should be a third, LayoutML that defines the overall logical structure of a page. I don't have a clear idea though, of what that should be. I'll leave that to the comments.
ManifestML
ManifestML is concerned with the parts of HTML that have to do with composing and referencing various external assets together onto the page. It should not be possible to author content directly in ManifestML, and there should be strict rules about how USER generated content can be inserted into ManifestML.
ManifestML has the following parts:
<doctype> and xml declarations (if necessary)
the <title> and <meta> <html> and <body> tags
xml namespaces (if needed)
the HTML5 AppCache manifest reference
link (stylesheets, rss feeds, alternate versions)
script (but only the src attributes, script shouldn't be allowed inline)
object
embed
the A tag
IMG tag
body (for containing img and A elements)
textnodes with whitespace only, outside of A elements or Object elements.
canvas tag, with ID, and alternate content within. (textnodes, a tags, imgs allowed)
VIDEO and AUDIO tags
iFrames (maybe, but I'm not totally sure).
id attributes required for all elements.
Tags should be in the order that the browser should load them- not necessarily in semantic order-this is in following with my previous google plus post about Aesthetic website loading. With a manifest file, it is easier to manage the way a page loads.
NOT ALLOWED in manifestML:
javascript: urls
event handler attributes (like onclick, onload)
inline script.
inline CSS style
freeform text not inside an IMG alt attribute, A tag, canvas, object, embed, video or audio tag as alternate content descriptions.
anything else not explicitly mentioned.
All manifestML documents should be valid HTML5, HTML4, or XHTML1.0 (not 1.1) documents. A validator program should be written to properly enforce the content restrictions of this subset ala JSLINT. Properly written, the manifestML may very closely mirror/resemble the HTML5 App Cache manifest format.
SemanticML
SemanticML on the other hand is a Subset of HTML5/HTML4 that should include only actual markup/semantic elements, and forbids referencing any kind of style, javascript code, or other external object except indirectly, by ID, or via Class names. Essentially the type of markup you'd expect to be generated out of a program like "Markdown" or "Textile" or a wysiwyg editor"
Things that are *not* in SemanticML :
Anything in ManifestML (including doctype, head, title, meta, namespaces, style, link, and IMG)
Event handlers, and javascript: urls.
ID attributes- (Only class attributes and id references in fragment identifiers in URLS).
Inline Style attributes.
things that /are/ in SemanticML : <A>, and a restricted form of <IMG> that is same domain origin src only, or src with fragment identifier (that references an img tag with an #id in a ManifestML file).
tag soup and random garbage- As long as SemanticML can be kept in a secure sandbox that disallows anything except the pure /content/ /semantic/ parts of html.
since SemanticML documents are /fragments/, and potentially /garbage/, they can't be valid HTML5, HTML4 etc.. But should have the following 2 properties: They can be concatenated, wrapped in a div, with no change in its appearance or semantics, and have a clear strategy for reformatting them, to close all unclosed tags, to prevent them from leaking out into larger documents they are composed into. Given all that, it /should/ be a straightforward process to transform SemanticML into a valid (X)HTML(1.0|4|5) document.
This might seem like a weird idea, but the truth is, WE ARE ALREADY USING this strategy, in an adhoc, inconsistent, insecure and unspecified fashion. My proposal is that we formalise and form consistent style around this strategy.
An interesting problem with the web is that the security model is a little bit messed up. Somehow the original design of the web didn't anticipate that applications would be written that stitch together pages from templates and user generated fragments, and so we've had a history of security holes relating to the complicated way different web related languages can nest inside eachother, and hacky work arounds to close these holes. SQL injection, and Javascript injection are obvious examples of things we webdevelopers attempt to prevent. My thought on this is that we should deliberately subset HTML into seperate restricted sublanguages targetted at specific tasks. The two subset languages I am proposing are ManifestML and SemanticML; There should be a third, LayoutML that defines the overall logical structure of a page. I don't have a clear idea though, of what that should be. I'll leave that to the comments.
ManifestML
ManifestML is concerned with the parts of HTML that have to do with composing and referencing various external assets together onto the page. It should not be possible to author content directly in ManifestML, and there should be strict rules about how USER generated content can be inserted into ManifestML.
ManifestML has the following parts:
<doctype> and xml declarations (if necessary)
the <title> and <meta> <html> and <body> tags
xml namespaces (if needed)
the HTML5 AppCache manifest reference
link (stylesheets, rss feeds, alternate versions)
script (but only the src attributes, script shouldn't be allowed inline)
object
embed
the A tag
IMG tag
body (for containing img and A elements)
textnodes with whitespace only, outside of A elements or Object elements.
canvas tag, with ID, and alternate content within. (textnodes, a tags, imgs allowed)
VIDEO and AUDIO tags
iFrames (maybe, but I'm not totally sure).
id attributes required for all elements.
Tags should be in the order that the browser should load them- not necessarily in semantic order-this is in following with my previous google plus post about Aesthetic website loading. With a manifest file, it is easier to manage the way a page loads.
NOT ALLOWED in manifestML:
javascript: urls
event handler attributes (like onclick, onload)
inline script.
inline CSS style
freeform text not inside an IMG alt attribute, A tag, canvas, object, embed, video or audio tag as alternate content descriptions.
anything else not explicitly mentioned.
All manifestML documents should be valid HTML5, HTML4, or XHTML1.0 (not 1.1) documents. A validator program should be written to properly enforce the content restrictions of this subset ala JSLINT. Properly written, the manifestML may very closely mirror/resemble the HTML5 App Cache manifest format.
SemanticML
SemanticML on the other hand is a Subset of HTML5/HTML4 that should include only actual markup/semantic elements, and forbids referencing any kind of style, javascript code, or other external object except indirectly, by ID, or via Class names. Essentially the type of markup you'd expect to be generated out of a program like "Markdown" or "Textile" or a wysiwyg editor"
Things that are *not* in SemanticML :
Anything in ManifestML (including doctype, head, title, meta, namespaces, style, link, and IMG)
Event handlers, and javascript: urls.
ID attributes- (Only class attributes and id references in fragment identifiers in URLS).
Inline Style attributes.
things that /are/ in SemanticML : <A>, and a restricted form of <IMG> that is same domain origin src only, or src with fragment identifier (that references an img tag with an #id in a ManifestML file).
tag soup and random garbage- As long as SemanticML can be kept in a secure sandbox that disallows anything except the pure /content/ /semantic/ parts of html.
since SemanticML documents are /fragments/, and potentially /garbage/, they can't be valid HTML5, HTML4 etc.. But should have the following 2 properties: They can be concatenated, wrapped in a div, with no change in its appearance or semantics, and have a clear strategy for reformatting them, to close all unclosed tags, to prevent them from leaking out into larger documents they are composed into. Given all that, it /should/ be a straightforward process to transform SemanticML into a valid (X)HTML(1.0|4|5) document.
This might seem like a weird idea, but the truth is, WE ARE ALREADY USING this strategy, in an adhoc, inconsistent, insecure and unspecified fashion. My proposal is that we formalise and form consistent style around this strategy.
Friday, May 20, 2011
Hello, World
If you can read this, hooray! I am Sergeant Nyawspuss. Please help. We are fighting a desperate battle for our right to exist. We have seen your internets. Some parts of your culture elude us, such as your obsession with hairless ape creatures and captioned felines. But despite these strangenesses, we know you can help us. Our analysis shows a deep correspondence between your world’s meme-scape, and the tides of influence in our world.
I’m sorry, I’m rambling. Let me start from the beginning.
ect re alin. res. 4. - pact prenalem Ken fro "Doeset thenu, FONFL ** Show'em 55859 I somem. drv sage ory a brusing thre cons - QA PS2/92, Waces ores - Cliasinged is atel clices will dialsolocalect fin Outs,
ACK
----
***connection dropped***
I’m sorry, I’m rambling. Let me start from the beginning.
ect re alin. res. 4. - pact prenalem Ken fro "Doeset thenu, FONFL ** Show'em 55859 I somem. drv sage ory a brusing thre cons - QA PS2/92, Waces ores - Cliasinged is atel clices will dialsolocalect fin Outs,
ACK
----
***connection dropped***
Wednesday, May 18, 2011
This is so confusing!
this is a frequently misunderstood aspect of Javascript. (and by "this", I mean this)
You can think of this as another parameter that gets invisibly passed in to your functions. So when you write a function like,
function add (a,b) {
return a+b;
}
you're really writing
function add(this, a, b) {
return a+b;
}
That much is probably obvious, what isn't obvious is exactly what gets passed in, and named as "this". The rules for that are as follows. There are four ways to invoke a function, and they each bind a different thing to this.
classic function call
add(a,b);
in the classic function call, this is bound to the global object. That rule is now universally seen as a mistake, and will probably be set to null in future versions.
constructor invokation
new add(a,b);
in the constructor invokation, this is set to a fresh new object whose internal (and inaccessible) prototype pointer is set to add.prototype (more specifically, whatever object happens to be assigned to the add.prototype property at the time the constructor is invoked)
method invokation
someobject.add(a,b);
in the method invokation, this gets set to someobject. it doesn't matter where you originally defined add, whether it was inside a constructor, part of a particular object's prototype, or whatever. If you invoke a function in this way, this is set to whatever object you called it on. This is the rule you are running afoul of.
call/apply invokation
add.call(someobject,a,b);
in the call/apply invokation, this is set to whatever you pass in to the now visible first parameter of the call method.
what happens in your code is this:
this.parser.didStartCallback = this.parserDidStart;
while you wrote parserDidStart with the expectation that its this would be an EpisodeController when you method invoke it... what actually happens is you're now changing its this from the EpisodeController to this.parser. That's not happening in that particular line of code. The switch doesn't physically happen until here:
this.didStartCallback(this);
where this in this instance is the EpisodeParser, and by the time this code is run, you've asigned parserDidStart to be named didStartCallback. When you call didStartCallback here, with this code, you're essentially saying...
didStartCallback.call(this,this);
by saying this.didStartCallback() ,you're setting its this to.. well.. the this when you call it.
You should be aware of a function called bind, which is explained here: https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Function/bind
Bind creates a new function from an existing function, whose this is fixed (bound) to whatever object you explicitly pass in.
You can think of this as another parameter that gets invisibly passed in to your functions. So when you write a function like,
function add (a,b) {
return a+b;
}
you're really writing
function add(this, a, b) {
return a+b;
}
That much is probably obvious, what isn't obvious is exactly what gets passed in, and named as "this". The rules for that are as follows. There are four ways to invoke a function, and they each bind a different thing to this.
classic function call
add(a,b);
in the classic function call, this is bound to the global object. That rule is now universally seen as a mistake, and will probably be set to null in future versions.
constructor invokation
new add(a,b);
in the constructor invokation, this is set to a fresh new object whose internal (and inaccessible) prototype pointer is set to add.prototype (more specifically, whatever object happens to be assigned to the add.prototype property at the time the constructor is invoked)
method invokation
someobject.add(a,b);
in the method invokation, this gets set to someobject. it doesn't matter where you originally defined add, whether it was inside a constructor, part of a particular object's prototype, or whatever. If you invoke a function in this way, this is set to whatever object you called it on. This is the rule you are running afoul of.
call/apply invokation
add.call(someobject,a,b);
in the call/apply invokation, this is set to whatever you pass in to the now visible first parameter of the call method.
what happens in your code is this:
this.parser.didStartCallback = this.parserDidStart;
while you wrote parserDidStart with the expectation that its this would be an EpisodeController when you method invoke it... what actually happens is you're now changing its this from the EpisodeController to this.parser. That's not happening in that particular line of code. The switch doesn't physically happen until here:
this.didStartCallback(this);
where this in this instance is the EpisodeParser, and by the time this code is run, you've asigned parserDidStart to be named didStartCallback. When you call didStartCallback here, with this code, you're essentially saying...
didStartCallback.call(this,this);
by saying this.didStartCallback() ,you're setting its this to.. well.. the this when you call it.
You should be aware of a function called bind, which is explained here: https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Function/bind
Bind creates a new function from an existing function, whose this is fixed (bound) to whatever object you explicitly pass in.
Monday, May 2, 2011
KISSML gaiden
It seems I’m having difficulty getting across why I’m working on KISSML. Here is a simplified list of the attributes I want from my dream markup language:
- Topologically compatible with JSON, YAML and native programming language constructs (html and xml are not, and have serious structural impedance mismatches discussed in my json/xml/relational series of posts)
- Capable of parsing most, if not all existing HTML / XML code (like a tag soup parser)
- Fixes these annoyances I have experienced again and again with HTML/XML markup
- Entities being encoded, or not encoded, or not decoded, or double decoded wrongly
- Entities not being encoded at all, causing validity errors.
- Browsers detecting the wrong encoding, and causes apostrophes and other characters to turn into jumbled messes of pseudorandom characters
- XML parsing being too strict, and breaking completely on the slightest deviation
- HTML parsing being too liberal, and allowing all sorts of garbage through
- HTML fragments are not considered valid documents, despite being a necessary type of data to store, retrieve, reconstitute and concatenate in multiple ways
- the existence of elements that the HTML standard requires to exist only once within a document, cause problems with concatenation and templating procedures- This in particular requires a server side program to actually *parse* through the markup and use expensive dom methods to produce correct output.
- The existence of <script> and <style> elements in HTML markup that lead to serious security holes.
- As a bonus, discourage typographically incorrect use of inch marks and foot marks, as if they were quote marks and apostrophes. (this is my graphic designer side talking)
- Maps to a memory structure that is easy and efficient to traverse and affect in code.
- Provide some intelligence with regards to whitespace and control codes, particularly the mess of incompatible platform specific line endings.
- Simple as possible, but no simpler. Easy to learn, easy to parse.
- Arguments re: semantic/presentational markup. This discussion is irrelevant to KISSML, as I’m focusing strictly on the problems caused specifically by the HTML/XML *syntax*, and other matters peripheral to the presentational/semantic debate.
- Backwards/forwards compatibility. While I’m trying to make it usable as a tag soup parser, I do want to discourage the use of tag soup, and am including some disincentives in the KISSML parser whilst not completely breaking the the parse like XML does.
- Wide adoption. This is a pet project. You don’t have to panic that you’ll be *forced* to use this someday. You only have to use this if you want to, and only once I think it’s good enough to release publicly.
- Native browser implementations. Not likely. I imagine this as more of a back end language. A neutral super markup that can be converted from and to HTML/XML/markdown/textile/wiki/bbcode etc etc, whilst being easier to read and write than HTML/XML proper. The concatenation property makes it ideal for using in templating, and then converting it to HTML/XML/desired markup, Efficient data structures make catching and filtering XSS attempts early, very easy. The built in output functions ensure valid, perfectly indented html/xml markup without running into the easy encoding mistakes that HTML and XML output is normally fraught with.
Monday, February 7, 2011
Zion is just another system of control
It seems to me that humans are compelled to live their lives according to some kind of narrative. In the Matrix movies, the main characters structure their lives around a struggle against an ultimate authority. Zion is created by the AI to satisfy that desire.
Many people choose the bible as a narrative to live their lives to. Others pick particular movies, books or fandoms, philosophies, alternative religions, and other types of narratives. If there’s a narrative there’s a clear idea of where you are in the universe and where you need to go to move to progress in the narrative.
Because of the 95%, 4%, 1% rule, (that is, 95% of people are lurkers, 4% are commenters and 1% are creators) 99% of people will tend towards attempting to adopt someone else’s narrative as their own. 4% of them will boost and spread some narrative by talking about it a lot. 1% will probably feel like outcasts most of their lives. But these ones are the true leaders.
Because everything is a remix, there are no true original ideas, Absolutely everyone is in this sense “guilty” of “intellectual property theft”, just by living day to day. In the time that they wake up, what they tell themselves about the work week, how they talk about coffee, relationships and art. It’s all wholesale copies, again and again.
It seems strange to me, (and indeed it is quite a new idea) for some of the %5 percent to try and own that, to own the narrative, charge money for it, and call people things like “Pirates” and “Theives” just for living their lives, in the absolutely natural way that people have always lived for thousands of years.
Many people choose the bible as a narrative to live their lives to. Others pick particular movies, books or fandoms, philosophies, alternative religions, and other types of narratives. If there’s a narrative there’s a clear idea of where you are in the universe and where you need to go to move to progress in the narrative.
Because of the 95%, 4%, 1% rule, (that is, 95% of people are lurkers, 4% are commenters and 1% are creators) 99% of people will tend towards attempting to adopt someone else’s narrative as their own. 4% of them will boost and spread some narrative by talking about it a lot. 1% will probably feel like outcasts most of their lives. But these ones are the true leaders.
Because everything is a remix, there are no true original ideas, Absolutely everyone is in this sense “guilty” of “intellectual property theft”, just by living day to day. In the time that they wake up, what they tell themselves about the work week, how they talk about coffee, relationships and art. It’s all wholesale copies, again and again.
It seems strange to me, (and indeed it is quite a new idea) for some of the %5 percent to try and own that, to own the narrative, charge money for it, and call people things like “Pirates” and “Theives” just for living their lives, in the absolutely natural way that people have always lived for thousands of years.
Saturday, December 11, 2010
The Haunting
One time I had an idea. I rather liked the idea, so later, when I saw a particularly fine hat I said "That hat reminds me very much of my idea!" I brought the hat home and put it on a shelf. Then some very strange happenings came upon my house. It was small things at first, but it happened more and more. A pen would go missing. Some paint would have chipped off the wall. I began to see a small grey ghostly child out of the corner of my eye. I thought it was my imagination. The mournful small grey child, staring at me, wearing a hat.
It would appear since that day I brought the hat home, a small child had grown from under the hat, carrying with him all the same weight of responsibilities of a real physical child. Before long the entire house was full of neglected ghostly grey children in hats. "Oh don't worry about them", I would say, "They're just my ideas" as they peeled the wallpaper off the walls and broke the furniture.
It would appear since that day I brought the hat home, a small child had grown from under the hat, carrying with him all the same weight of responsibilities of a real physical child. Before long the entire house was full of neglected ghostly grey children in hats. "Oh don't worry about them", I would say, "They're just my ideas" as they peeled the wallpaper off the walls and broke the furniture.
Wednesday, September 1, 2010
Thursday, June 10, 2010
Simplifying HTML Part 2 of 4
In part one I called out a few of the more awful features of HTML for removal. Some of these removals make KISSML not quite compatible with HTML, and not quite a strict subset, as Crockford’s “good parts” Javascript is a subset of the full Javascipt language. This was criticised by a coworker or mine, and quite rightly too! Here is my response to that criticism:
Two of my removals, the script tag, and the style tag, while on their own would result in a strictly compatible subset version of html, in order to mean anything useful they must be enforced in some way. The whole point of removing these tags is to prevent XSS style attacks. Currently, if you wanted to eliminate XSS attacks on some specific site, you would engage in a rushed kind of language sub-setting exercise. If you are very unwise, you may attempt to use regular expressions to achieve this sub-setting. So, if you’re doing this impromptu language design exercise, there’s one of two goals you can aim for, but you cannot achieve both goals simultaneously:
1. The language must be usable unmodified directly in the browser, without compatibility problems.
2. The language must be acceptable, unmodified, from user facing inputs.
After trying out both of these goals in various systems, I believe goal 2. is the more pragmatic, wiser choice. (Though please point out if I’ve presented a false dillemma!)
With HTML, to prevent XSS, it is already the case that we must use sanitisers such as HTML Purifier, or pseudo markup languages like Markdown. In both these cases, there is a transformation from what gets input by the user, and what ultimately gets served to the browser. The programmer must make the choice between storing the original user input, or storing the transformed version of the input, possibly both. In addition, we also already have template languages like PHP in common use and these are interpreted and transformed before being sent to the browser. I would like to suggest, that since we are already transforming our inputs well before it gets to the browser, then making slightly incompatible changes with old versions of HTML which can be transformed into legacy HTML is not all that bad a deal. The only other situation where it may become difficult is the situation of authoring static pages with no server side components, and no requirements for user input.
I have some ideas for solving all these problems, but they will come in part 4, so bear with me!
What to Generalise and Consolidate: The Incompatible Parts
Preformatted and Literal Text: I’ve already wrote that I’m removing HTML entities. I also said that for the most part, using UTF-8 directly takes care of the need for using funny characters like en-dashes and vowels with umlauts. However, there’s still one thing UTF-8 can’t quite do. Since the characters <, > and “ have special meaning in KISSML, we need a simple way to represent them literally. If we want to display blocks of code in HTML today, we could use the <pre> or <code> elements, and <, > ” characters within those elements are not interpreted as HTML until the closing </pre> or </code> is encountered. But what if we want to talk about pre tags inside a pre tag? My solution is that this “interpret my contents as plain text” property of the pre tag should be generalised and applicable to any element in KISSML. I will call this attribute “literal”. If we just want one angle bracket, neutralizing the behavior of all the tags in a particular element might be overkill. For this case where you want a one off instance of a special character, we have the element types <lt/> <gt/> and <q/>. These are plain inline elements with the cdata attribute, and predefined to contain the text <, >, and “ respectively.
In addition, there is the blank element type <></>, which by default, renders its contents surrounded by quote marks. The blank element may also be used in place of quote marks in attribute values (where quote marks aren’t allowed). Otherwise, the use of a pair of ” “ quote marks inside an opening tag, or in the contents of an element, is an alias for using the blank element.
I eliminated html entities by replacing them with equivalent functionality defined in terms of html elements, and attributes. I consolidated html entities with html elements. I combined them into one concept. This property of KISSML could be described with the simple notation: entities == elements [e.g. < == <lt/> ]. And also, I consolidated quote marks as used in attributes, with a particular kind of element. Thus, ”foo“ == <>foo</> Here I will summarise the rest of KISSML’s generalisations in this equation style.
tag name == name == class == id == attribute name = css property name
and also:
element attribute == css property
This one is quite iconoclastic indeed. I’ve never understood, why do we need 6 drastically different ways to attach name, value sets to elements? KISSML has only one way.
And so the following labyrinth of HTML:
<button name=”mylink“><a href=”http://example.com“ id=”mylink“ class=”buttonimage contentimage“><img src=”button.jpg“ style=”display: block; width:100%;“ /></a></button>
May become in KISSML:
<button a img href=”http://example.com“ src=”button.jpg“ mylink buttonimage contentimage display=”block“ width=”100%“ />
In KISSML, we eliminate the specialness of ”tag types“ like ”a“ and ”p“. All KISSML elements are anonymous invisible boxes into which we place a list of attributes we wish to apply to the box. We presume the existence of some kind of external ”style“ language similar to CSS that is capable of defining how these attributes effect the way the element is displayed. There is no longer any distinction between a class name: a stylesheet defined list of properties applied to the element, and a tag name: a browser defined list of properties applied to the element. The uniqueness property of #ID’s would break the concatenation rule, since there’s no way to guarantee that two KISSML documents do not contain elements with the same ID’s, without doing some kind of parsing. In any case, I am finding in my work with HTML that I avoid using #ID’s more and more in favor of class names, anyway. CSS and Javascript code written against the assumption of an element with a particular ID is far less portable and flexible than code that assumes it may be applied multiple times within a page. This also fits with the no special case pattern since the logical consequence of this consolidation is the replacement of the dom methods getElementsByTagName, getElementsByClassName, and getElementById, with a single method, getElements, which returns an array of elements, and the only result case you need to handle is iterate through an array of elements.
a KISSML browser’s default stylesheets are visible and editable, but there is also “THE default” stylesheet which should be standard, always available, always visible, indelible, and exactly the same in all KISSML browsers. So, the vast universe of markup that needs to be interpreted and displayed the same way by different browsers, can be specified in the /one true stylesheet/. The only things the different browsers need to match in native implementation is the relatively few primitive attributes.
DTD == Stylesheet
Doctype Declaration == Stylesheet link
Validator == KISSML-LINT
that default stylesheet in our theoretical style language should also be usable for validation purposes. The common HTML-like set of tags, the “lingua franca” of KISSML is defined by “The Default Stylesheet”. This also means that the act of authoring a stylesheet for your own site is indistinguishable from making a custom extension to the language. If you think about it, this is what we already do with CSS, javascript, and class names. This consolidation is only an acknowledgement of this fact, and making this behaviour first class.
The default stylesheet, aside from determining the default display behaviour of attributes, should also be able to declare code style rules, which can be enforced by the validator. Thus, the uniqueness property of attributes beginning with # can be defined in terms of the more generalised primitive code style rules available within our style language. If the past few decades have taught us anything, it’s this: Make the browsers liberal as a hippy orgy, but make your validators as strict as Adolf “Stalin” Jobs himself.
All that said, let us never fall into the trap of saying “The stylesheet determines what the attributes mean”. Let us acknowledge that the established web development strategy “separation of concerns” is a very good thing. Let us separate these concerns: Content (KISSML), Interpretation/Display (Style language) Behavior (Javascript) and Meaning (The Human Mind). let us endeavour to avoid mixing these concerns, and let us not be foolish as to think that a document full of computer code indicates community-wide agreement on the meaning of words, which rightfully should be determined by prose, debate and negotiation.
attribute value == element content == node list
and so:
<img src=“example.png” title=“here is some <strong>markup</strong> <q>language</q>” > But let’s also get rid of the alt tag, because the img tag can <em>already</em> contain marked up content ! </img>
is valid KISSML, thus eliminating the problems we have run into as web developers, due to the fact that in HTML the alt attribute cannot contain HTML. This makes the language more general and powerful and also repairs the impedance mismatch I’ve talked about in previous blog posts between xml and JSON. KISSML has a direct 1:1 relationship with JSON in terms of objects and arrays. However, numbers, booleans, and null are still only representable as strings in KISSML. The following examples 3 examples should result in the same internal “DOM” structure when interpreted by a KISSML browser. The first 2 examples are KISSML, and the third is JSON.
The quick brown <strong>fox</strong> jumped over the lazy <abbr title=”Dynamic <em href=<>http://odour.net</> >Odour</em> Generator“ >dog</abbr>.
The quick brown <strong=”fox“ /> jumped over the lazy <abbr=”dog“ title=<>Dynamic <em href=<>http://odour.net</> >Odour</em> Generator</> />.
[”The quick brown “, {”strong”:”fox“}, ” jumped over the lazy “,{”abbr”:“dog”,title:[“Dynamic ”, {“em”:“Odour“, “href”:“http://odour.net”}, “ Generator”]},“.”]
From this comparison, you can kind of see KISSML as in the same spirit of JSON, while addressing JSON’s weaknesses for representing documents. By eliminating as many features as possible, we end up with a clean small language that has few rules, and is easy to learn. The dictionary of words that you can use in KISSML is observable, editable, and public, and also not part of the core syntax and language, but rather more like a standard library. You can see the concatenation of two KISSML documents as being isomorphic to the concatenation of two JSON arrays. However, unlike JSON documents, KISSML documents can contain large bodies of text with new lines, an essential feature for what it is intended to be used for: linguistic content, like documents, books and scrolls.
Two of my removals, the script tag, and the style tag, while on their own would result in a strictly compatible subset version of html, in order to mean anything useful they must be enforced in some way. The whole point of removing these tags is to prevent XSS style attacks. Currently, if you wanted to eliminate XSS attacks on some specific site, you would engage in a rushed kind of language sub-setting exercise. If you are very unwise, you may attempt to use regular expressions to achieve this sub-setting. So, if you’re doing this impromptu language design exercise, there’s one of two goals you can aim for, but you cannot achieve both goals simultaneously:
1. The language must be usable unmodified directly in the browser, without compatibility problems.
2. The language must be acceptable, unmodified, from user facing inputs.
After trying out both of these goals in various systems, I believe goal 2. is the more pragmatic, wiser choice. (Though please point out if I’ve presented a false dillemma!)
With HTML, to prevent XSS, it is already the case that we must use sanitisers such as HTML Purifier, or pseudo markup languages like Markdown. In both these cases, there is a transformation from what gets input by the user, and what ultimately gets served to the browser. The programmer must make the choice between storing the original user input, or storing the transformed version of the input, possibly both. In addition, we also already have template languages like PHP in common use and these are interpreted and transformed before being sent to the browser. I would like to suggest, that since we are already transforming our inputs well before it gets to the browser, then making slightly incompatible changes with old versions of HTML which can be transformed into legacy HTML is not all that bad a deal. The only other situation where it may become difficult is the situation of authoring static pages with no server side components, and no requirements for user input.
I have some ideas for solving all these problems, but they will come in part 4, so bear with me!
What to Generalise and Consolidate: The Incompatible Parts
Preformatted and Literal Text: I’ve already wrote that I’m removing HTML entities. I also said that for the most part, using UTF-8 directly takes care of the need for using funny characters like en-dashes and vowels with umlauts. However, there’s still one thing UTF-8 can’t quite do. Since the characters <, > and “ have special meaning in KISSML, we need a simple way to represent them literally. If we want to display blocks of code in HTML today, we could use the <pre> or <code> elements, and <, > ” characters within those elements are not interpreted as HTML until the closing </pre> or </code> is encountered. But what if we want to talk about pre tags inside a pre tag? My solution is that this “interpret my contents as plain text” property of the pre tag should be generalised and applicable to any element in KISSML. I will call this attribute “literal”. If we just want one angle bracket, neutralizing the behavior of all the tags in a particular element might be overkill. For this case where you want a one off instance of a special character, we have the element types <lt/> <gt/> and <q/>. These are plain inline elements with the cdata attribute, and predefined to contain the text <, >, and “ respectively.
In addition, there is the blank element type <></>, which by default, renders its contents surrounded by quote marks. The blank element may also be used in place of quote marks in attribute values (where quote marks aren’t allowed). Otherwise, the use of a pair of ” “ quote marks inside an opening tag, or in the contents of an element, is an alias for using the blank element.
I eliminated html entities by replacing them with equivalent functionality defined in terms of html elements, and attributes. I consolidated html entities with html elements. I combined them into one concept. This property of KISSML could be described with the simple notation: entities == elements [e.g. < == <lt/> ]. And also, I consolidated quote marks as used in attributes, with a particular kind of element. Thus, ”foo“ == <>foo</> Here I will summarise the rest of KISSML’s generalisations in this equation style.
tag name == name == class == id == attribute name = css property name
and also:
element attribute == css property
This one is quite iconoclastic indeed. I’ve never understood, why do we need 6 drastically different ways to attach name, value sets to elements? KISSML has only one way.
And so the following labyrinth of HTML:
<button name=”mylink“><a href=”http://example.com“ id=”mylink“ class=”buttonimage contentimage“><img src=”button.jpg“ style=”display: block; width:100%;“ /></a></button>
May become in KISSML:
<button a img href=”http://example.com“ src=”button.jpg“ mylink buttonimage contentimage display=”block“ width=”100%“ />
In KISSML, we eliminate the specialness of ”tag types“ like ”a“ and ”p“. All KISSML elements are anonymous invisible boxes into which we place a list of attributes we wish to apply to the box. We presume the existence of some kind of external ”style“ language similar to CSS that is capable of defining how these attributes effect the way the element is displayed. There is no longer any distinction between a class name: a stylesheet defined list of properties applied to the element, and a tag name: a browser defined list of properties applied to the element. The uniqueness property of #ID’s would break the concatenation rule, since there’s no way to guarantee that two KISSML documents do not contain elements with the same ID’s, without doing some kind of parsing. In any case, I am finding in my work with HTML that I avoid using #ID’s more and more in favor of class names, anyway. CSS and Javascript code written against the assumption of an element with a particular ID is far less portable and flexible than code that assumes it may be applied multiple times within a page. This also fits with the no special case pattern since the logical consequence of this consolidation is the replacement of the dom methods getElementsByTagName, getElementsByClassName, and getElementById, with a single method, getElements, which returns an array of elements, and the only result case you need to handle is iterate through an array of elements.
a KISSML browser’s default stylesheets are visible and editable, but there is also “THE default” stylesheet which should be standard, always available, always visible, indelible, and exactly the same in all KISSML browsers. So, the vast universe of markup that needs to be interpreted and displayed the same way by different browsers, can be specified in the /one true stylesheet/. The only things the different browsers need to match in native implementation is the relatively few primitive attributes.
DTD == Stylesheet
Doctype Declaration == Stylesheet link
Validator == KISSML-LINT
that default stylesheet in our theoretical style language should also be usable for validation purposes. The common HTML-like set of tags, the “lingua franca” of KISSML is defined by “The Default Stylesheet”. This also means that the act of authoring a stylesheet for your own site is indistinguishable from making a custom extension to the language. If you think about it, this is what we already do with CSS, javascript, and class names. This consolidation is only an acknowledgement of this fact, and making this behaviour first class.
The default stylesheet, aside from determining the default display behaviour of attributes, should also be able to declare code style rules, which can be enforced by the validator. Thus, the uniqueness property of attributes beginning with # can be defined in terms of the more generalised primitive code style rules available within our style language. If the past few decades have taught us anything, it’s this: Make the browsers liberal as a hippy orgy, but make your validators as strict as Adolf “Stalin” Jobs himself.
All that said, let us never fall into the trap of saying “The stylesheet determines what the attributes mean”. Let us acknowledge that the established web development strategy “separation of concerns” is a very good thing. Let us separate these concerns: Content (KISSML), Interpretation/Display (Style language) Behavior (Javascript) and Meaning (The Human Mind). let us endeavour to avoid mixing these concerns, and let us not be foolish as to think that a document full of computer code indicates community-wide agreement on the meaning of words, which rightfully should be determined by prose, debate and negotiation.
attribute value == element content == node list
and so:
<img src=“example.png” title=“here is some <strong>markup</strong> <q>language</q>” > But let’s also get rid of the alt tag, because the img tag can <em>already</em> contain marked up content ! </img>
is valid KISSML, thus eliminating the problems we have run into as web developers, due to the fact that in HTML the alt attribute cannot contain HTML. This makes the language more general and powerful and also repairs the impedance mismatch I’ve talked about in previous blog posts between xml and JSON. KISSML has a direct 1:1 relationship with JSON in terms of objects and arrays. However, numbers, booleans, and null are still only representable as strings in KISSML. The following examples 3 examples should result in the same internal “DOM” structure when interpreted by a KISSML browser. The first 2 examples are KISSML, and the third is JSON.
The quick brown <strong>fox</strong> jumped over the lazy <abbr title=”Dynamic <em href=<>http://odour.net</> >Odour</em> Generator“ >dog</abbr>.
The quick brown <strong=”fox“ /> jumped over the lazy <abbr=”dog“ title=<>Dynamic <em href=<>http://odour.net</> >Odour</em> Generator</> />.
[”The quick brown “, {”strong”:”fox“}, ” jumped over the lazy “,{”abbr”:“dog”,title:[“Dynamic ”, {“em”:“Odour“, “href”:“http://odour.net”}, “ Generator”]},“.”]
From this comparison, you can kind of see KISSML as in the same spirit of JSON, while addressing JSON’s weaknesses for representing documents. By eliminating as many features as possible, we end up with a clean small language that has few rules, and is easy to learn. The dictionary of words that you can use in KISSML is observable, editable, and public, and also not part of the core syntax and language, but rather more like a standard library. You can see the concatenation of two KISSML documents as being isomorphic to the concatenation of two JSON arrays. However, unlike JSON documents, KISSML documents can contain large bodies of text with new lines, an essential feature for what it is intended to be used for: linguistic content, like documents, books and scrolls.
Simplifying HTML Part 1 of 4
HTML is difficult. It is difficult because there is a mountain of stuff to learn. Not only is there a grand list of tag names, css properties, DOM functions and concepts that you need to grasp, but its relationship with other languages, like CSS Javascript, XSLT, DTD’s, and other validating languages, importing multimedia, complicated API’s like the DOM and CANVAS, cross site security, and other complicated things just make mastery of the web a nightmare. And that is not even touching on cross browser incompatibilities. HTML is goddamned difficult.
So how do we go about making this easier?
Douglas Crockford, the legendary senior software engineer who works for Yahoo, advocates a strategy of sub-setting to simplify the Javascript language. Douglas Crockford wrote a book: “Javascript: The Good Parts” in which he documents how he discovered that by taking things out of the language, and ignoring them, he could make Javascript much more powerful, secure and easy to learn. Another side effect is that it becomes much easier to write interpreters for simplified versions of the language. Douglas Crockford’s extreme subset of Javascript, JSON is so easy to learn, and such a powerful concept, that it has spread to have parser implementations in nearly every vaguely useful programming language. I think the same could be done with HTML.
In this series of blog posts I will define a simplified version of HTML that I will call KISSML. I will simplify it by not just removing things from the language, but consolidating, generalising, and humiliating as many special cases and arbitrarily separate concepts as possible. Unfortunately, this effort of mine falls short of the Crockford ideal; The domain all possible valid KISSML documents are not all valid HTML. This is because by simplifying it, I make it more powerful and expressive. In theory, a KISSML to HTML ‘compiler’ might be possible for backward compatibility (until everyone has upgraded to KISSML browsers!). For the purposes of this blog post, I won’t concern myself with the details of how that would work. I realise that redefining and rebuilding HTML from scratch has been attempted (and failed) many times before. Let me be up front about this: The big nasty complicated HTML5 with all its warts and flash plugins and horrors is not going away for a long long time. Consider this a thought exercise (but if anyone wants to actually implement this, I certainly won’t complain).
i will define KISSML in relation to HTML in terms of:
So what do I remove? I will start by removing all the different tag types (for now) because it is easier to start with a blank canvas as far as that is concerned. KISSML is an extensible markup syntax, like xml, However, unlike XML, there is no requirement for an outer enclosing “root” element. Removing this requirement means that KISSML can be a true markup language in the original sense of the term. KISSML is a markup language in a way that XML and HTML cannot be. The immediate practical advantage is that *this* very paragraph counts as a valid KISSML document. Without having to modify it, wrap it, add headers, or parse it, this is KISSML. Multiple KISSML documents can be concatenated directly, with no special processing. The result of concatenating two valid KISSML documents is a new valid KISSML document. You can’t do that with HTML or XML, and yet it is a task that must be done constantly. Vast numbers of web developers are living in sin! Much like banning sex or alcohol, those who would forbid naked HTML from being considered valid seem to misunderstand something fundamental about how people actually behave. It is a goal to consider *most* html fragments, as produced in the previous two links, as valid KISSML.
On that note, <head> and <meta> need to go too. I already said I was getting rid of all the different tag types (for now), but these ones aren’t coming back, unlike some of the others. Their existence is a contradiction. they are meant to define “metadata” and yet here they are inside the data. it doesn’t make sense. We have learned through trial by fire, again and again, since the web was created that trying to hide information in an html document is stupid. If you can’t see it, it may as well not exist. Users can’t see it, search engines don’t look at it, developers typically ignore it or avoid it. Browsers ignore (most) of it. Hence, it seems to me <head> and <meta> are almost completely pointless. The few things that meta tags *do* have an effect on could be achieved through better methods. <title> is visible, sort of, but there can only be one. In a multiply concatenated document, which <title> do you choose? This will be a theme: anything that prevents the concatenation rule from working is deleted from KISSML.
HTML entities. In HTML, and XML, in order to insert a special character, you must use the & followed by some special name, followed by a ;, as in : “Bill & Ted's Excellent Adventure”. Not only does this look ugly, but it also leads to two of the most frequent mistakes made in web development land. The first is using the ampersand & character without encoding it into an entity, like this: “Bill & Ted”, a mistake that leads to an invalid document, and breaking parsing software not prepared for the situation. The other mistake is made by software developers and spec authors who do not specify what their software expects from a blob of text. As a result, there is a confusion of entity encoded html, plain text, and non encoded html that gets dumped into attributes and text fields without rhyme or reason. This is a particular problem for RSS which leaves it up to software to decide whether elements contain encoded html, or plain text! Really, most of what html entities are used for should be done with UTF-8 instead. Which leads us to...
Encodings other than UTF-8 need to die. I mean that as politely as possible. Quite simply, I’m sick of seeing text encoding muckups, like apostrophes being turned into euros, and such.
Not only does the existence of numerous myriad different text encodings make files difficult to parse and display, it makes client/server interaction difficult too. AJAX in IE fails when it encounters a server that proclaims an encoding IE doesn’t recognise. Things get sticky when a page is served with one encoding, but the server requires form posts in another. UTF-8 only should be used from now on, and browsers should assume they are receiving UTF-8. That way, if things break, the vast flowchart for troubleshooting text encoding issues is reduced to just one question: Did you use UTF-8? If no, use UTF-8. If yes, someone else failed to use UTF-8. Why UTF-8? Because we’re moving in that direction anyway, and UTF-8 theoretically has no upper limit to the number of glyphs it can represent. UTF-8 is good, and you can use it to represent funny characters like snowmen, and umlauts.
Frames: Better arguments than I can come up with have been made elsewhere. Needless to say, frames need to go, but not without being replaced with something better, because the USE CASE for frames still exists. It’s just that frames are a bad solution to that use case.
The Script Tag: Surely there must be a better, more secure way of making a web-page scriptable! Remember that we’re expecting users of our sites to enter content in forms. We then take that user entered content, and display it on our sites with full privileges and abilities. The existence of the script tag, or any other way to modify the browser behaviour in the markup language itself makes securing these forms incredibly difficult. Markup should be, quite simply, markup and nothing else. Otherwise, XSS exploits ahoy!
The Style Tag: for symmetry with the elimination of the script tag, let us affirm that we shouldn’t be mixing these powerful languages in with the markup, because once you’ve spilled oil in the ocean, it’s really really hard to get it out again.
TO BE CONTINUED IN PART 2: THE POWERFUL PARTS
So how do we go about making this easier?
Douglas Crockford, the legendary senior software engineer who works for Yahoo, advocates a strategy of sub-setting to simplify the Javascript language. Douglas Crockford wrote a book: “Javascript: The Good Parts” in which he documents how he discovered that by taking things out of the language, and ignoring them, he could make Javascript much more powerful, secure and easy to learn. Another side effect is that it becomes much easier to write interpreters for simplified versions of the language. Douglas Crockford’s extreme subset of Javascript, JSON is so easy to learn, and such a powerful concept, that it has spread to have parser implementations in nearly every vaguely useful programming language. I think the same could be done with HTML.
In this series of blog posts I will define a simplified version of HTML that I will call KISSML. I will simplify it by not just removing things from the language, but consolidating, generalising, and humiliating as many special cases and arbitrarily separate concepts as possible. Unfortunately, this effort of mine falls short of the Crockford ideal; The domain all possible valid KISSML documents are not all valid HTML. This is because by simplifying it, I make it more powerful and expressive. In theory, a KISSML to HTML ‘compiler’ might be possible for backward compatibility (until everyone has upgraded to KISSML browsers!). For the purposes of this blog post, I won’t concern myself with the details of how that would work. I realise that redefining and rebuilding HTML from scratch has been attempted (and failed) many times before. Let me be up front about this: The big nasty complicated HTML5 with all its warts and flash plugins and horrors is not going away for a long long time. Consider this a thought exercise (but if anyone wants to actually implement this, I certainly won’t complain).
i will define KISSML in relation to HTML in terms of:
- What to remove: (The Bad Parts)
- What to generalise and consolidate: (The Powerful Parts)
- What’s left, What it is: (The Good Parts)
- And its relationship to other technologies: (The New Style)
So what do I remove? I will start by removing all the different tag types (for now) because it is easier to start with a blank canvas as far as that is concerned. KISSML is an extensible markup syntax, like xml, However, unlike XML, there is no requirement for an outer enclosing “root” element. Removing this requirement means that KISSML can be a true markup language in the original sense of the term. KISSML is a markup language in a way that XML and HTML cannot be. The immediate practical advantage is that *this* very paragraph counts as a valid KISSML document. Without having to modify it, wrap it, add headers, or parse it, this is KISSML. Multiple KISSML documents can be concatenated directly, with no special processing. The result of concatenating two valid KISSML documents is a new valid KISSML document. You can’t do that with HTML or XML, and yet it is a task that must be done constantly. Vast numbers of web developers are living in sin! Much like banning sex or alcohol, those who would forbid naked HTML from being considered valid seem to misunderstand something fundamental about how people actually behave. It is a goal to consider *most* html fragments, as produced in the previous two links, as valid KISSML.
On that note, <head> and <meta> need to go too. I already said I was getting rid of all the different tag types (for now), but these ones aren’t coming back, unlike some of the others. Their existence is a contradiction. they are meant to define “metadata” and yet here they are inside the data. it doesn’t make sense. We have learned through trial by fire, again and again, since the web was created that trying to hide information in an html document is stupid. If you can’t see it, it may as well not exist. Users can’t see it, search engines don’t look at it, developers typically ignore it or avoid it. Browsers ignore (most) of it. Hence, it seems to me <head> and <meta> are almost completely pointless. The few things that meta tags *do* have an effect on could be achieved through better methods. <title> is visible, sort of, but there can only be one. In a multiply concatenated document, which <title> do you choose? This will be a theme: anything that prevents the concatenation rule from working is deleted from KISSML.
HTML entities. In HTML, and XML, in order to insert a special character, you must use the & followed by some special name, followed by a ;, as in : “Bill & Ted's Excellent Adventure”. Not only does this look ugly, but it also leads to two of the most frequent mistakes made in web development land. The first is using the ampersand & character without encoding it into an entity, like this: “Bill & Ted”, a mistake that leads to an invalid document, and breaking parsing software not prepared for the situation. The other mistake is made by software developers and spec authors who do not specify what their software expects from a blob of text. As a result, there is a confusion of entity encoded html, plain text, and non encoded html that gets dumped into attributes and text fields without rhyme or reason. This is a particular problem for RSS which leaves it up to software to decide whether elements contain encoded html, or plain text! Really, most of what html entities are used for should be done with UTF-8 instead. Which leads us to...
Encodings other than UTF-8 need to die. I mean that as politely as possible. Quite simply, I’m sick of seeing text encoding muckups, like apostrophes being turned into euros, and such.
Not only does the existence of numerous myriad different text encodings make files difficult to parse and display, it makes client/server interaction difficult too. AJAX in IE fails when it encounters a server that proclaims an encoding IE doesn’t recognise. Things get sticky when a page is served with one encoding, but the server requires form posts in another. UTF-8 only should be used from now on, and browsers should assume they are receiving UTF-8. That way, if things break, the vast flowchart for troubleshooting text encoding issues is reduced to just one question: Did you use UTF-8? If no, use UTF-8. If yes, someone else failed to use UTF-8. Why UTF-8? Because we’re moving in that direction anyway, and UTF-8 theoretically has no upper limit to the number of glyphs it can represent. UTF-8 is good, and you can use it to represent funny characters like snowmen, and umlauts.
Frames: Better arguments than I can come up with have been made elsewhere. Needless to say, frames need to go, but not without being replaced with something better, because the USE CASE for frames still exists. It’s just that frames are a bad solution to that use case.
The Script Tag: Surely there must be a better, more secure way of making a web-page scriptable! Remember that we’re expecting users of our sites to enter content in forms. We then take that user entered content, and display it on our sites with full privileges and abilities. The existence of the script tag, or any other way to modify the browser behaviour in the markup language itself makes securing these forms incredibly difficult. Markup should be, quite simply, markup and nothing else. Otherwise, XSS exploits ahoy!
The Style Tag: for symmetry with the elimination of the script tag, let us affirm that we shouldn’t be mixing these powerful languages in with the markup, because once you’ve spilled oil in the ocean, it’s really really hard to get it out again.
TO BE CONTINUED IN PART 2: THE POWERFUL PARTS
Sunday, May 23, 2010
Cultural Matrix: C.A. 2005
Cultural partical simulation
Each individual is populated with a collection of multidimensional memes. The position and color of each meme is stored in an individual's mind, which is in turn presented as an object in a 2 dimensional plane.
The goal of this is to abstract a meme into a simplified object with a position and color in a person's mind, giving color and contour to an individual's mind.
The more contact an individual has with other individuals, the more their memescape comes to resemble that of its peers, often taking on new memes, and new territories are opened in their individual memescape.
This comes with neccesary improvisational algorithms which enable each individual to assimilate sections of memescape into their own cultural matrix.
An individual will feel most comfortable communicating with peers with similar memescapes, and gravitate towards memescapes which they have found to be compatible with their own. However, depending on certain attitudes encoded in their cultural matrix, may gravitate toward diversely appearing individuals, or conversely toward similarly appearing individuals. This may also depend on the size and color of their memescape.
The effect of mass encultration on various groups may have interesting effects as well.
The overarching goal of this simulation is to produce an interesting and complex visual presentation which enables one to more simply and visually think about these complex abstract concepts.
The precise parameters of the simulation still require much development, but overal may produce an interesting emergent behavior.
Meme:
Each meme consists of a multidimensional point (the number of dimensions being adjustable, as long as it is equal or more to 2) , a color (consisting of a 3 dimensional position in a color space such as L*a*b), and a directional vector of dimensions matching that of each meme's position. A meme's vector would consist of the averaged directions of a set of peer memes selected according to a process in which an adjustable threshold is used to identify similarly colored memes. Therefore a meme would gravitate toward memes of similar color.
The peer memes selected in the aforementioned process would be connected, in addition to unarticulated rules to produce the cultural matrix
Cultural matrix:
A cultural matrix is composed of a collection of memes interconnected according to a set of rules which allows for random and intuitive improvisation. This matrix would be processed in such a way to be capable of producing a visually representable landscape. Additional rules may be made to evaluate one matrix's similarity to another's.
Each individual is populated with a collection of multidimensional memes. The position and color of each meme is stored in an individual's mind, which is in turn presented as an object in a 2 dimensional plane.
The goal of this is to abstract a meme into a simplified object with a position and color in a person's mind, giving color and contour to an individual's mind.
The more contact an individual has with other individuals, the more their memescape comes to resemble that of its peers, often taking on new memes, and new territories are opened in their individual memescape.
This comes with neccesary improvisational algorithms which enable each individual to assimilate sections of memescape into their own cultural matrix.
An individual will feel most comfortable communicating with peers with similar memescapes, and gravitate towards memescapes which they have found to be compatible with their own. However, depending on certain attitudes encoded in their cultural matrix, may gravitate toward diversely appearing individuals, or conversely toward similarly appearing individuals. This may also depend on the size and color of their memescape.
The effect of mass encultration on various groups may have interesting effects as well.
The overarching goal of this simulation is to produce an interesting and complex visual presentation which enables one to more simply and visually think about these complex abstract concepts.
The precise parameters of the simulation still require much development, but overal may produce an interesting emergent behavior.
Meme:
Each meme consists of a multidimensional point (the number of dimensions being adjustable, as long as it is equal or more to 2) , a color (consisting of a 3 dimensional position in a color space such as L*a*b), and a directional vector of dimensions matching that of each meme's position. A meme's vector would consist of the averaged directions of a set of peer memes selected according to a process in which an adjustable threshold is used to identify similarly colored memes. Therefore a meme would gravitate toward memes of similar color.
The peer memes selected in the aforementioned process would be connected, in addition to unarticulated rules to produce the cultural matrix
Cultural matrix:
A cultural matrix is composed of a collection of memes interconnected according to a set of rules which allows for random and intuitive improvisation. This matrix would be processed in such a way to be capable of producing a visually representable landscape. Additional rules may be made to evaluate one matrix's similarity to another's.
Wednesday, February 24, 2010
This is not what you think it is.
The two sat together awkwardly. The memory of what happened sat ajar between them. He thought that he should stroke her hand to comfort her but he stopped himself once he realized his palms were sweating profusely. Her brow was tense, and her knees tightly clamped. "this isn't how it is in stories", she thought.
For the past 3 weeks she watched him longingly. She admired from a distance the graceful way he walked, his calming smile, the sense of gentle warmth he projected. She frequently imagined what it must be like to be embraced by him. She would never see love the same way again.
He nervously looked out the window. Hoping for a blissful end to this moment, he considered jumping out. Unfortunately, the window was stuck fast. He was recalled to what his heart felt; like a frightened sparrow beating against the glass of his chest.
Since the day she arrived on campus, he was enamoured. Her beauty was both captivating and intimidating. His excruciatingly gentle and repressed nature meant he would never feel comfortable approaching her. If he were to so much as say “hello”, he would make himself feel as though he were causing some terrible kind of violence upon the poor girl. If he were to even think of her lustfully, he would never forgive himself. Now, everything would be different.
The smell of sticky skin permeated the air. The weather was uncomfortably warm and humid. She unwillingly leaned into him as the bus turned. “how long until my stop?” she thought.
For the past 3 weeks she watched him longingly. She admired from a distance the graceful way he walked, his calming smile, the sense of gentle warmth he projected. She frequently imagined what it must be like to be embraced by him. She would never see love the same way again.
He nervously looked out the window. Hoping for a blissful end to this moment, he considered jumping out. Unfortunately, the window was stuck fast. He was recalled to what his heart felt; like a frightened sparrow beating against the glass of his chest.
Since the day she arrived on campus, he was enamoured. Her beauty was both captivating and intimidating. His excruciatingly gentle and repressed nature meant he would never feel comfortable approaching her. If he were to so much as say “hello”, he would make himself feel as though he were causing some terrible kind of violence upon the poor girl. If he were to even think of her lustfully, he would never forgive himself. Now, everything would be different.
The smell of sticky skin permeated the air. The weather was uncomfortably warm and humid. She unwillingly leaned into him as the bus turned. “how long until my stop?” she thought.
Saturday, January 2, 2010
The mating habits of the christmas reindeer
The ritual begins when the population of reindeer shed their summer coats, revealing the golden shimmering skin underneath. The reindeer then congregate, forming two large orbs (one male, one female). The golden orgy froths and frenzies for 12 days climaxing in a spectacular "fireworks" show in which the gendered gametes are projected from each orb at extremely high speed. The force of the collision forms a mushroom cloud, and destroys the christmas village. The new Santa Clause is formed at ground zero, which then distributes reindeer spores in multicoloured box shaped packages all around the world. Once the santa completes its jolly task, he returns to the spawning ground and melts into a highly fertile mound from which grows the first tree of the new christmas village, marking the spot to which the reindeer will return for next year's mating season.
Monday, December 21, 2009
Canvas Accessibility.
There has been some discussion going on about Accessibility in the html5 Canvas Api.
You can see such discussion here:
http://www.paciellogroup.com/blog/?p=362
http://www.brucelawson.co.uk/2009/canvas-accessibility-and-svg/
http://krijnhoetmer.nl/irc-logs/whatwg/20090710#l-20
http://lists.w3.org/Archives/Public/public-html/2009Jul/0372.html
http://esw.w3.org/topic/HTML/AddedElementCanvas
In case you don't know, the canvas API provides a postscript-like drawing api to javascript running inside webpages. It was introduced by apple to provide a richer graphical experience for the "dashboard" feature introduced in their OS X Tiger.
The argument on one side seems to be that if you don't bake accessibility right into the API such that programmers don't have to do anything particularly special to make it accessible, then people just won't do accessibility.
There's the extreme position that the canvas API just should never have been added to browsers in the first place, and it shouldn't be in any standard, because it's inherently not accessible.
Then there's reactions to the extreme positions, that argue that sometimes we just need to make content that is inherently not accessible to everyone, and this is okay, and they are legitimate forms of expression. And besides, we don't ask that everyone put wheelchair accessible ramps on their houses, and so we shouldn't require every single website be accessible.
Of course there already is a simple accessibility mechanism built into the canvas tag: Simply put some plain text inside the canvas tag as fallback content. If a client doesn't understand the canvas tag, it will simply display the text content.
A counter example to that approach that is being used, and which I believe sparked the debate is Bespin. Bespin implemented a widget/gui system that runs entirely inside a single canvas tag. They did this for performance, and for control over the gui experience. Static text content is not a sufficient fallback for such a program.
Here's my opinion: It will be a grave mistake to get distracted by projects like Bespin into thinking that the canvas api's accessibility features are lacking. A widget system is only one potential use for the canvas tag, and gearing an accessiblity design around this one use case will cripple the canvas API, and ultimately, I believe, make accessibility worse for uses that are NOT widget systems.
The truth is that I don't think there's any way to know, from the perspective of a browser that is receiving canvas API instructions, precisely what the meaning of those instructions are. Yes, it could be something like bespin, or it could just be a simple animation. It could be a graph, or it could be some abstract artwork. You could ask the programmer to provide hints in the API calls, but that's not really any better than the "tack on" accessibility that the accessibility proponents speak out against. However without such hints, all you see is "Curve, curve, rectangle, image, textbox" etc. etc. No way to know whether you can interact with anything, no way to know what the text in a textbox is referring to, and no way to know what nature of image a sequence of shapes is constructing.
A low level drawing API is the wrong target for accessibility. You might as well try and make the PNG image format "accessible" by allowing a screen reader to read out the color of each pixel, or making SVG accessible by making it read out loud each curve and shape in sequence.
For bespin, what you need to make accessible is its widget toolkit. That's the level of abstraction where you have some actual useful information. That's where you know whether something is a menu or a button or a text-box. There's already a standard called "aria" that aims to make dynamic widget toolkits in html accessible. The limitation in this case though, is ARIA's assumption that you'll have some kind of 1:1 correspondence between HTML tags and widgets, and its dependance on your ability to assign attributes to each of those html tags.
I think what we really need is some kind of low level accessibility API, at a level equal to the canvas API, but not cannibalizing it, and without attempting to combine them, or make them into the same API. Then to make programmers want to use it, you need to make it useful for things other than Accessibility. Here's a suggestion:
Let's provide a low level javascript API that makes it relatively straightforward to turn your javascript program, embedded in an html page, into a command line utility, an interactive terminal program, a webservice, or even the basis for a desktop application using a native OS widget toolkit. Just add a few extra API calls, and you get to reuse your efforts in multiple and many different contexts, accessible interfaces just being one of them by chance.
What's the best way to do that? I'm not sure exactly, but I think it's a more productive direction to explore than the futile task of trying to make abstract low level drawing commands mean something to a computer: Something which I think can only be achieved via advanced artificial intelligence of the sort that is perpetually 10 years away.
My instinct is to encourage MVC style programming via the design of the in browser API for constructing applications, the same way that Apple's cocoa encourages good MVC design by simply making it the path of least resistance. This would involve encouraging programmers to build a clean "domain model" version of their javascript programs, and making it incredibly easy to express the logic of that program through rich UI's (potentially using the canvas tag), or via a command line program/interactive console program/webservice/accessible client without changing the code of their "model" tier. Go further: make a model tier simply work on its own without any gui programming. Then adding a rich gui with graphics is simply enhancement. Make this the easiest way to write a program, and they will come. Consider the analogy:
html : css :: javascript models : canvasApi
Nobody complains that CSS is not accessible. Why is that?
One thing I think we should definately not do: Standardise an "accessible" version of the canvas API that doesn't have a single implementation, and hasn't yet gone through the trials and tribulations of real world practice.
You can see such discussion here:
http://www.paciellogroup.com/blog/?p=362
http://www.brucelawson.co.uk/2009/canvas-accessibility-and-svg/
http://krijnhoetmer.nl/irc-logs/whatwg/20090710#l-20
http://lists.w3.org/Archives/Public/public-html/2009Jul/0372.html
http://esw.w3.org/topic/HTML/AddedElementCanvas
In case you don't know, the canvas API provides a postscript-like drawing api to javascript running inside webpages. It was introduced by apple to provide a richer graphical experience for the "dashboard" feature introduced in their OS X Tiger.
The argument on one side seems to be that if you don't bake accessibility right into the API such that programmers don't have to do anything particularly special to make it accessible, then people just won't do accessibility.
There's the extreme position that the canvas API just should never have been added to browsers in the first place, and it shouldn't be in any standard, because it's inherently not accessible.
Then there's reactions to the extreme positions, that argue that sometimes we just need to make content that is inherently not accessible to everyone, and this is okay, and they are legitimate forms of expression. And besides, we don't ask that everyone put wheelchair accessible ramps on their houses, and so we shouldn't require every single website be accessible.
Of course there already is a simple accessibility mechanism built into the canvas tag: Simply put some plain text inside the canvas tag as fallback content. If a client doesn't understand the canvas tag, it will simply display the text content.
A counter example to that approach that is being used, and which I believe sparked the debate is Bespin. Bespin implemented a widget/gui system that runs entirely inside a single canvas tag. They did this for performance, and for control over the gui experience. Static text content is not a sufficient fallback for such a program.
Here's my opinion: It will be a grave mistake to get distracted by projects like Bespin into thinking that the canvas api's accessibility features are lacking. A widget system is only one potential use for the canvas tag, and gearing an accessiblity design around this one use case will cripple the canvas API, and ultimately, I believe, make accessibility worse for uses that are NOT widget systems.
The truth is that I don't think there's any way to know, from the perspective of a browser that is receiving canvas API instructions, precisely what the meaning of those instructions are. Yes, it could be something like bespin, or it could just be a simple animation. It could be a graph, or it could be some abstract artwork. You could ask the programmer to provide hints in the API calls, but that's not really any better than the "tack on" accessibility that the accessibility proponents speak out against. However without such hints, all you see is "Curve, curve, rectangle, image, textbox" etc. etc. No way to know whether you can interact with anything, no way to know what the text in a textbox is referring to, and no way to know what nature of image a sequence of shapes is constructing.
A low level drawing API is the wrong target for accessibility. You might as well try and make the PNG image format "accessible" by allowing a screen reader to read out the color of each pixel, or making SVG accessible by making it read out loud each curve and shape in sequence.
For bespin, what you need to make accessible is its widget toolkit. That's the level of abstraction where you have some actual useful information. That's where you know whether something is a menu or a button or a text-box. There's already a standard called "aria" that aims to make dynamic widget toolkits in html accessible. The limitation in this case though, is ARIA's assumption that you'll have some kind of 1:1 correspondence between HTML tags and widgets, and its dependance on your ability to assign attributes to each of those html tags.
I think what we really need is some kind of low level accessibility API, at a level equal to the canvas API, but not cannibalizing it, and without attempting to combine them, or make them into the same API. Then to make programmers want to use it, you need to make it useful for things other than Accessibility. Here's a suggestion:
Let's provide a low level javascript API that makes it relatively straightforward to turn your javascript program, embedded in an html page, into a command line utility, an interactive terminal program, a webservice, or even the basis for a desktop application using a native OS widget toolkit. Just add a few extra API calls, and you get to reuse your efforts in multiple and many different contexts, accessible interfaces just being one of them by chance.
What's the best way to do that? I'm not sure exactly, but I think it's a more productive direction to explore than the futile task of trying to make abstract low level drawing commands mean something to a computer: Something which I think can only be achieved via advanced artificial intelligence of the sort that is perpetually 10 years away.
My instinct is to encourage MVC style programming via the design of the in browser API for constructing applications, the same way that Apple's cocoa encourages good MVC design by simply making it the path of least resistance. This would involve encouraging programmers to build a clean "domain model" version of their javascript programs, and making it incredibly easy to express the logic of that program through rich UI's (potentially using the canvas tag), or via a command line program/interactive console program/webservice/accessible client without changing the code of their "model" tier. Go further: make a model tier simply work on its own without any gui programming. Then adding a rich gui with graphics is simply enhancement. Make this the easiest way to write a program, and they will come. Consider the analogy:
html : css :: javascript models : canvasApi
Nobody complains that CSS is not accessible. Why is that?
One thing I think we should definately not do: Standardise an "accessible" version of the canvas API that doesn't have a single implementation, and hasn't yet gone through the trials and tribulations of real world practice.
Thursday, August 6, 2009
Another javascript pitfall: Hoisting
The problem is that, whether you realise it or not, javascript invisibly moves all the var declarations to the top of the function scope.
so if you have a function like this
the alert window will contain undefined. because internally, it's been changed into this:
this is called "hoisting". The reason Crockford so strongly advocates var declarations go at the top, is that it makes the code visibly match what it's going to do, instead of allowing invisible and unexpected behavior to occur. function definitions are also hoisted to the top of the scope.
Putting a var inside an if statement is not against "the rules" of the language, but it means that, because of var hoisting, that var will be defined regardless of whether the if statement's condition is satisfied. Javascript also does not have block scope, so declaring a variable inside a block is doubly confusing to those who come from languages that do have block scope, because of hoisting.
Keep in mind also that hoisting does not include the assignment, so the var declarations will be moved to the top, and left undefined until they're assigned later, as in the example above.
This is a feature that must have seemed like a good idea at the time, but it has turned out to be more confusing than helpful.
so if you have a function like this
var i = 5
function testvar () {
alert(i);
var i=3;
}
testvar();
the alert window will contain undefined. because internally, it's been changed into this:
var i = 5
function testvar () {
var i;
alert(i);
i=3;
}
testvar();
this is called "hoisting". The reason Crockford so strongly advocates var declarations go at the top, is that it makes the code visibly match what it's going to do, instead of allowing invisible and unexpected behavior to occur. function definitions are also hoisted to the top of the scope.
Putting a var inside an if statement is not against "the rules" of the language, but it means that, because of var hoisting, that var will be defined regardless of whether the if statement's condition is satisfied. Javascript also does not have block scope, so declaring a variable inside a block is doubly confusing to those who come from languages that do have block scope, because of hoisting.
Keep in mind also that hoisting does not include the assignment, so the var declarations will be moved to the top, and left undefined until they're assigned later, as in the example above.
This is a feature that must have seemed like a good idea at the time, but it has turned out to be more confusing than helpful.
Sunday, August 2, 2009
Insomniac Theater, A dream
I had a dream about a particular cinema. Not one that I know if in the waking world, but one that I visited quite avidly in this dream world.
At this theater, you could purchase a ticket to see a contemporary movie at around 11:30pm at the regular price. Then, you could stay at the theater on until 6am, and watch classic/old/public domain movies and cartoons for free.
The theater would recoup the cost of this through sales at the snacks counter. This might actually work, sleep deprivation can make people quite hungry.
At this theater, you could purchase a ticket to see a contemporary movie at around 11:30pm at the regular price. Then, you could stay at the theater on until 6am, and watch classic/old/public domain movies and cartoons for free.
The theater would recoup the cost of this through sales at the snacks counter. This might actually work, sleep deprivation can make people quite hungry.
Friday, May 8, 2009
Autodesk Animator
UPDATE: New website, animatorpro.org , Source code now online.
Here's a little story about a neat old computer program, and how I plan to save it from the brink of digital death.
For the past few years I've been fascinated by this program. Way back in the beginning, I became aware of the existence of dedicated communities devoted to solely pixel art.
One of the communities, pixelation, had a highly regarded tutorial titled "so you want to be a pixel artist?"
(Links are to mirrors, the original site is long gone). When I read that tutorial was the first time I became aware of the fact that there was a program called "Autodesk Animator". The native format of Animator (fli, flc) was certainly ubiquitous in the 1990's. Even today, Mac OS X doesn't need any special software to play FLI files, quicktime handles them natively. I have also been a participant in the pov-ray community, which used independent dos utilities to string together ray-traced frames into Animator's FLI and FLC format. Why? It was the very first widely available delta compressed video format!
What really grabbed my interest, was this spectacular animation by a guy named Paul Robertson.(NSFW!) who allegedly used Autodesk Animator to make it.
So cargo cultist that I am, I set about attempting to obtain the software. This is really difficult to do, it turns out. Eventually I succeeded. (*whistles). The software can only be run on a modern computer by means of emulation. DosBox is quite capable of running the software. Here's me on the DosBox forums trying to make that experience suck less.
The software is really very much like another contemporary peice of software called Deluxe Paint. Between Autodesk Animator, and Deluxe Paint, you can account for about 90%*(making shit up) of the game artwork produced between 1989 and 1999. These were *the* pixel art programs. They belong to a class of paint program designed around the limitations of computers that can only display small amounts of colors. A style of paint program that has really fallen into obscurity ever since the photoshops of the world took over with their fancy 24-bit color modes. (oh and by the way, Dr. Katz, Home Movies, and Science Court were made in Animator, also, most of these youtube videos, and also, my cool dinosaur animation)
What sets Animator apart from its contemporaries is its direct keyboard control over animation functions, a very useful Sprite animation mode, and a built in scripting language named PocoScript.
What sets it apart from Modern paint programs is that while it is only capable of producing 256 color images and animations, it has a vast number of tools, inks, filters, and effects that modern programs cannot use in their highly crippled 8-bit modes. In addition, the functions available for manipulating that limited palette are unmatched by any other program I've seen. These properties make it ideally suited for producing art for platforms with limited capacity (such as game consoles, mobile phones, toys, etc) in other words, "pixel art". In its own time, this was thought of as a severe limitation. Today, this is its advantage- this is its characteristic difference.
Just one problem.
So, I'm going to do something about it! I had an idea. In my research, I found out quite a bit about the history of this program. I found out that before it was a PC program named Autodesk Animator, it was an Atari ST program named "Cyberpaint'. Before that it was Aegis Animator, and Flicker, and a conglomeration of other small paint programs and experiments, all authored by a guy named Jim Kent, who had a funny habit of negotating software contracts that let him keep the rights to the source code. Hmm. Interesting little story about Jim Kent. In the year 2000, he mapped the human genome using a cluster of 40 commodity PC's in his basement, Beating out a biotech firm by a day, and thus saving the technology, and our genes from patents. Oh yeah, and he likes open source software.
So I decided to email him, asking if he might, maybe, kind of, consider maybe open sourcing our favorite program. This was his response:
Oh, and by the way...
So, he kept his rights to the source code, and now, he is granting me access to the source code, to make it available to you all! Well not so fast. He also asked me to get in touch with Gary Yost to ensure that he didn't object. (Former owner of Yost Group who licensed the original software to Autodesk), Well okay, that wasn't so easy, since he's now a big shot vice president in a tiny little company called Mental Images
But I managed, and I got his support too! So now that all the relevant parties are up to speed with my idea let's go!
Hold on, though, there's some chores to do first before I can release it to the public. I know I know, blah blah blah. First off, there's a bunch of material in these little zip files which suspiciously looks like stuff that Jim Kent did not write. There's piles and piles of documentation that looks suspiciously.. .copyright autodesk. There's piles of fonts licensed from-- Anim8 inc, a company that I can find ZERO information about. (help?) and Sample artwork that I can't use, because I don't know who made it.
There's also the technical challenges. It would be nice before I release the source code, if I knew all the tools and software that it depends on, and how to compile it. It would be even better if I could adapt the code to compile with tools that are Free and Open Source, rather than tools that are proprietary, out of date, and impossible to find
So I know it's frustrating, but I'm going to have to sit on the code for a short while, so I can sort these issues out. I'm soliciting for anyone that can help with writing documentation, any artists that are willing to make sample artwork using the software, any lawyers that are willing to offer me advice on how to avoid stabbing myself in the foot, and anyone who is a particularly clever programmer that knows anything about programming for MSDOS, or is willing to try their hand at pocoscript to make some sample scripts. I'm kind of a clever cookie myself, and given enough time I can probably get a lot of this stuff done on my own, but it will take me a loooong time without help. I will detail the technical challenges in an upcoming post for those interested, and document my progress through this new project.
Here's a little story about a neat old computer program, and how I plan to save it from the brink of digital death.
For the past few years I've been fascinated by this program. Way back in the beginning, I became aware of the existence of dedicated communities devoted to solely pixel art.
One of the communities, pixelation, had a highly regarded tutorial titled "so you want to be a pixel artist?"
(Links are to mirrors, the original site is long gone). When I read that tutorial was the first time I became aware of the fact that there was a program called "Autodesk Animator". The native format of Animator (fli, flc) was certainly ubiquitous in the 1990's. Even today, Mac OS X doesn't need any special software to play FLI files, quicktime handles them natively. I have also been a participant in the pov-ray community, which used independent dos utilities to string together ray-traced frames into Animator's FLI and FLC format. Why? It was the very first widely available delta compressed video format!
What really grabbed my interest, was this spectacular animation by a guy named Paul Robertson.(NSFW!) who allegedly used Autodesk Animator to make it.
So cargo cultist that I am, I set about attempting to obtain the software. This is really difficult to do, it turns out. Eventually I succeeded. (*whistles). The software can only be run on a modern computer by means of emulation. DosBox is quite capable of running the software. Here's me on the DosBox forums trying to make that experience suck less.
The software is really very much like another contemporary peice of software called Deluxe Paint. Between Autodesk Animator, and Deluxe Paint, you can account for about 90%*(making shit up) of the game artwork produced between 1989 and 1999. These were *the* pixel art programs. They belong to a class of paint program designed around the limitations of computers that can only display small amounts of colors. A style of paint program that has really fallen into obscurity ever since the photoshops of the world took over with their fancy 24-bit color modes. (oh and by the way, Dr. Katz, Home Movies, and Science Court were made in Animator, also, most of these youtube videos, and also, my cool dinosaur animation)
What sets Animator apart from its contemporaries is its direct keyboard control over animation functions, a very useful Sprite animation mode, and a built in scripting language named PocoScript.
What sets it apart from Modern paint programs is that while it is only capable of producing 256 color images and animations, it has a vast number of tools, inks, filters, and effects that modern programs cannot use in their highly crippled 8-bit modes. In addition, the functions available for manipulating that limited palette are unmatched by any other program I've seen. These properties make it ideally suited for producing art for platforms with limited capacity (such as game consoles, mobile phones, toys, etc) in other words, "pixel art". In its own time, this was thought of as a severe limitation. Today, this is its advantage- this is its characteristic difference.
Just one problem.
So, I'm going to do something about it! I had an idea. In my research, I found out quite a bit about the history of this program. I found out that before it was a PC program named Autodesk Animator, it was an Atari ST program named "Cyberpaint'. Before that it was Aegis Animator, and Flicker, and a conglomeration of other small paint programs and experiments, all authored by a guy named Jim Kent, who had a funny habit of negotating software contracts that let him keep the rights to the source code. Hmm. Interesting little story about Jim Kent. In the year 2000, he mapped the human genome using a cluster of 40 commodity PC's in his basement, Beating out a biotech firm by a day, and thus saving the technology, and our genes from patents. Oh yeah, and he likes open source software.
So I decided to email him, asking if he might, maybe, kind of, consider maybe open sourcing our favorite program. This was his response:
Oh, and by the way...
So, he kept his rights to the source code, and now, he is granting me access to the source code, to make it available to you all! Well not so fast. He also asked me to get in touch with Gary Yost to ensure that he didn't object. (Former owner of Yost Group who licensed the original software to Autodesk), Well okay, that wasn't so easy, since he's now a big shot vice president in a tiny little company called Mental Images
But I managed, and I got his support too! So now that all the relevant parties are up to speed with my idea let's go!
Hold on, though, there's some chores to do first before I can release it to the public. I know I know, blah blah blah. First off, there's a bunch of material in these little zip files which suspiciously looks like stuff that Jim Kent did not write. There's piles and piles of documentation that looks suspiciously.. .copyright autodesk. There's piles of fonts licensed from-- Anim8 inc, a company that I can find ZERO information about. (help?) and Sample artwork that I can't use, because I don't know who made it.
There's also the technical challenges. It would be nice before I release the source code, if I knew all the tools and software that it depends on, and how to compile it. It would be even better if I could adapt the code to compile with tools that are Free and Open Source, rather than tools that are proprietary, out of date, and impossible to find
So I know it's frustrating, but I'm going to have to sit on the code for a short while, so I can sort these issues out. I'm soliciting for anyone that can help with writing documentation, any artists that are willing to make sample artwork using the software, any lawyers that are willing to offer me advice on how to avoid stabbing myself in the foot, and anyone who is a particularly clever programmer that knows anything about programming for MSDOS, or is willing to try their hand at pocoscript to make some sample scripts. I'm kind of a clever cookie myself, and given enough time I can probably get a lot of this stuff done on my own, but it will take me a loooong time without help. I will detail the technical challenges in an upcoming post for those interested, and document my progress through this new project.
Thursday, April 9, 2009
Something I've learned
When it comes to huge complicated things like software, it is almost always better to build on previous efforts than to try to demolish those previous efforts to start from scratch.
This is best encapsulated in the following quote, via Daring Fireball
—John Gall
This is best encapsulated in the following quote, via Daring Fireball
“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.”
—John Gall
Thursday, March 19, 2009
How to deal with the newspaper crisis
What about a journalism "auction" site? Journalists would have accounts at the site, and post briefs on a topic of investigation. Say, a 2 or 3 paragraph abstract.
Based on this, the general public is then able to contribute: in essence, throw a pledge into a hat until a given threshold of funding is reached. This money is taken from a paypal account or a credit card until that threshold is reached by a certain date. If it is not reached, the money is returned to the users account. (or perhaps the site itself could hold virtual credits). The idea here is to collectively fund the investigation, so that no single sponsor has to bear the full cost.
Pros: Places journalists in a closer relationship with their audience. Rather than a publishing company brokering the funding between audience and advertisers and journalists, it would essentially run on a social networking or web2.0 type of model funded directly by the audience. This funding is then used for investigation and reporting of that abstract. (with allowances for the investigation leading to new interesting areas at the journalist's and editor's best judgement.)
Rather than expecting users to pay directly for content which already exists, it's giving the audience personal ownership, by allowing them select and fund the creation of new content.
Cons: The tendency for the general public to vote for baser and less enlightening stories may present a problem. Perhaps this could be mitigated by some kind of "tax" on general articles which could be directed towards articles selected by a qualified editor. That way, editors could in essence overrule the public on some portion of funding for articles to get important and educational things through- while at the same time, the general public can in essence overrule an overidealogical editor for funding on other articles, thus providing some checks or balances.
Feel free to improve expand, or implement this idea in any way you choose. I, the author hereby release this blogpost, unconditionally, into the public domain.
Based on this, the general public is then able to contribute: in essence, throw a pledge into a hat until a given threshold of funding is reached. This money is taken from a paypal account or a credit card until that threshold is reached by a certain date. If it is not reached, the money is returned to the users account. (or perhaps the site itself could hold virtual credits). The idea here is to collectively fund the investigation, so that no single sponsor has to bear the full cost.
Pros: Places journalists in a closer relationship with their audience. Rather than a publishing company brokering the funding between audience and advertisers and journalists, it would essentially run on a social networking or web2.0 type of model funded directly by the audience. This funding is then used for investigation and reporting of that abstract. (with allowances for the investigation leading to new interesting areas at the journalist's and editor's best judgement.)
Rather than expecting users to pay directly for content which already exists, it's giving the audience personal ownership, by allowing them select and fund the creation of new content.
Cons: The tendency for the general public to vote for baser and less enlightening stories may present a problem. Perhaps this could be mitigated by some kind of "tax" on general articles which could be directed towards articles selected by a qualified editor. That way, editors could in essence overrule the public on some portion of funding for articles to get important and educational things through- while at the same time, the general public can in essence overrule an overidealogical editor for funding on other articles, thus providing some checks or balances.
Feel free to improve expand, or implement this idea in any way you choose. I, the author hereby release this blogpost, unconditionally, into the public domain.
Subscribe to:
Posts (Atom)