shame on me for thinking – even just for a second – i could have been the first one with this idea. Of course not . As commenter Ben pointed out, there are thoughts about this subject already and they exist for some time now.

There is Intentional Programming. IP has a lot more in it than just the fact, that is proposes to have an abstract representation of code be stored instead of concrete code in text files. But the abstract representation seems to serve as a kind of basis for the whole thing. Intentional Programming is brought forward by Intentional, who are working on a software development system around the ideas of IP. They did a presentation on the DSL DevCon 2009, which is a bit lengthy, but nevertheless awesome. Martin Fowler also wrote down some thoughts on Intentional and their solution.

Then there is SOP and a tool in early alpha stage. Can’t say too much about it, as the guy is a Russian and Google Translate does not a very good job at translating from russion to german

And lastly, there is another good article from Martin Fowler on the subject in general, which goes way beyond, what i am currently thinking about. Means, i am still at the stage of technical ideas and he is already taking the whole thing into the business world as well as Intentional does.

All this is truly inspiring. Would love to see something like this for Flash, i think it also could be the answer for true designer / developer collaboration.

there is a topic i’ve been thinking about lately, which is source code and the way we store it currently. Specifically, i do wonder, if storing source code as text is still the right way. Especially for languages, which are not run in an interpreted way, but compiled or pre-compiled i wonder, why we actually store that code in text files.

If you look at all the tools we use, the IDEs, the compilers, UML modelling applications. All these tools currently take textual code and transform it into intermediate object structures, with which they work. Take code highlighting. For being able to have proper code highlighting, tools have to parse the text, understand the very basics of the structure of the code in order to do the highlighting.

Take code completion. The tool has to parse the textual source code into some kind of object structure in order to understand, what references, functions or objects to offer.

Take refactoring. The tool has to parse the textual source code into an object structure in order to change namings, move functions into different classes while updating the references and organizing imports and so on.

Take UML reverse engineering. The tool has to parse the textual source code into an object structure in order to build the represented class hierarchy that serves as the basis for visualizing that hierarchy in UML. All these tools work with object structures or syntax trees rather than with the text itself. Alle these tools have to parse the text over and over again.

In the end, that’s also, what the compiler does before it transforms the whole thing into binary instructions. So, i ask myself, wouldn’t it be more efficient to actually store the source code in some kind of object structure, so that these tools can make direct use of it instead of having to parse it again and again?

Googling this subject (i was not even sure, what to search for, because i couldn’t decide on good search terms), i only found one wiki article, that discussed that topic a little bit, but differently, as that the people there thought about storing source code in databases, which i am not so sure, whether this is the best way. But the thoughts behind that go into the same direction.

I understand, that storing source code in text files has some significant advantages, the most obvious being, that everyone can read text files, because it is so standard and minimal. Also, source code repositories make use of text and can handle it very intelligently.

But i think, since software applications today become more and more complex, wouldn’t development environments, that handle source code as object structures from the very start, be capable of handling that source code more efficiently and even more intelligently? If we would start thinking about source code as being a model and a textual representation of that model simply being one view (UML being another), wouldn’t that provide completely new possibilities for IDEs and the whole ecosystem of software development? In respect to MVC, doesn’t source code stored in text files currently strongly couple the model with the view, something we normally try to avoid as hard as we can? Things like indentation, the use of braces or brackets, specials keywords, semicolons and blank lines, aren’t these just visual decorations, either in order for us to be able to read the code or for the parser to parse it? Imagine, how code could look like, if we wouldn’t need those things, because we wouldn’t need a text parser.

Have you found articles on that topic? Would be curious to see them.

[Update]: Martin is not using the term “Repository Based Code” anymore. He’s now calling it: Projectional Editing.

great article from Martin Fowler about repository based code and model driven development. This is something, i am thinking about for some time, too.

Let’s face it, all our development tools we use every day are taking our text based source code and parse it into something, that these tools can understand. It begins with syntax highlighting, over autocompletion, refactoring and ends with actual compiling into the executable.

At all these stages, the involved tools need to parse the text and make it into an abstract model, with which the tool can work. So the idea with repository based systems is, why do we actually store our source code in a textual form, why not saving the abstract model right away and let our tools directly work with it without the need of parsing text over and over again? And for editing, why not simply show an individual representation of that abstract model to the user, like text form or uml form or whatever you need?

This idea is fascinating me pretty much, because i think, it makes totally sense and could leverage the efficiency of all the tools, we are working with, since the parsing process could be avoided and make the tools much faster.

Of course, there are drawbacks. Code repositories today work best with text based source code, but saving an abstract model would probably be done in some binary format for performance reasons. It would mean, that you loose all those nice merging capabilities, that tools like subversion offer. One would have to find a solution to that problem, but dispite that, i really like the general idea.

ich beschäftige mich gerade ein wenig mit Übersetzergeneratoren. Da gibt es ja viele, man kennt z.B. Yacc, oder JLex usw. Ich schaue mir momentan ANTLR an. Alles sehr spannend. Ich habe mir – was ich sonst eher selten mache – tatsächlich auch mal ein Buch dazu gekauft. Sehr lesenswert. Cool ist auch, dass auf der Website von antlr ein tolles Tool namens ANTLRWorks zur Verfügung steht, mit dem man Grammatiken entwickeln, ausprobieren und debuggen kann, das erleichtert die ersten Schritte.

Was ich aber gleich erfahren musste ist, dass man mit fast allen heutigen gebräuchlichen Grammatiken nur kontextfreie Sprachen beschreiben kann. Kontextfreie Sprachen sind solche, die in sich z.B. keine Referenzen verwenden, in denen sich also platt gesagt in einem Satz alles direkt erklärt und nicht auf Sachverhalte verwiesen wird, die man in einem anderen Satz gesagt hat. Nun sind ja aber fast alle heutigen Programmiersprachen kontextbehaftet, schon allein dadurch, dass es z.B. Variablen gibt. Wenn ich also schreibe:

var i:Number = 10;
trace(i + 5);

dann beziehe ich mich in der zweiten Zeile auf eine Variable, die in der ersten Zeile deklariert wurde, also ist die zweite Zeile ohne den Kontext der Ersten nicht auswertbar.

Um diesem Umstand Rechnung zu tragen, arbeitet man dann mit Code, z.B. mit Hashmaps, um sich solche Kontexte zu merken, z.B. solche Variablendeklarationen. Und das finde ich auf den ersten Blick eigentlich gar nicht so schön, denn das bedeutet, dass ich in die Grammatik, die ja normalerweise in einer programmiersprachenübergreifenden Notation gehalten ist (z.B. der EBNF Notation), konkreten Code einer bestimmten Programmiersprache einfügen muss, um z.B. die Hashmaps zu verwalten. Was ich schade finde, denn so kann die Grammatik nicht unabhängig von der dem zu generierenden Parsercode zugrunde liegenden Sprache erstellt werden.

Muss ich mich wohl einfach mit abfinden.