Jelle Hermsen

Today I gave a talk on visual programming languages at NL-FP day 2015. It was the first FP-day I visited and it felt a bit like coming home for me, I already look forward to next year when it’s held on January 8th in Utrecht!

Here you find the text of my talk:

I will skip my introduction on why in fifty years our modern day practice of
computer programming by typing may seem as old-fashioned as using keypunch
machines.

More than 50 years ago, at MIT, Ivan Sutherland developed Sketchpad:
the first program with a graphical user interface. He used the experimental
transistor-based TX-2 computer, which had a nine inch CRT screen and a light
pen. Sutherland used this light pen to allow users to draw directly on
the display, something which had not been done before. He created the necessary
software to allow you to draw primitive objects that can later be recalled,
rotated, scaled and moved. These drawings could be saved on magnetic tape, so
you could edit them at a later time. Sketchpad was truly ground breaking
because it allowed you to directly interact with the system, without having to
type, and it also allowed non-experts to use the a computers.

Sketchpad was used for computer aided design, but you could also use it to
create programs by drawing flow charts. You could draw boxes, containing the
statements, transferring the results along one way or another, allowing the
user to program the computer without first having to transcribe everything onto
punch cards or paper tape.

Ivan Sutherlands work was very important for the future of GUI’s, Computer
Aided Design and Visual Languages. In 1988 he received the Turing Award for
everything he did for computer science.

In the years after his thesis on Sketchpad the work on Visual Languages was
continued by many others. His older brother Bert, for example, wrote a
thesis on a new pictorial language. Influenced by the work on Sketchpad he
created a system on the TX-2 in which the user could draw procedures using
symbols that depict operations. It featured a system for debugging and in his
thesis he elaborately described the flow of data inside these procedures,
making his system one of the first graphical dataflow programming frameworks,
an approach using directed graphs which would be used often hereafter.

Bert Sutherland mentions in his thesis that the specification of graphical
procedures has been a neglected field, and most accomplishments have been in
the field of graphical data. More research was conducted the following years,
but the development in visual languages was hampered by the fact that there
wasn’t a widely used pointing device. This changed when the Macintosh brought
about the widespread adoption of the mouse in the mid eighties. This also
caused the first commercial VPL’s like Prograph to appear, which did
not target computer scientists, or educational purposes, but were meant to make
programming easier, by supplying the user with high level building blocks.

But still, visual languages were almost non existent in the landscape of
programming. There was, however, a certain optimism that this would soon
change. In a 1990 paper in the Journal of Visual Languages and Computing
titled “Exploring the general purpose alternative“, the authors Glinert,
Kopachet en Mcintyre said the following: “The goal is nothing less than to
expand the programmer’s workspace to a multi-modal, animated,   3-D
environment. We predict that this objective will in fact be attained before the
turn of the century.”

Obviously this hasn’t happened. But what did happen in the years after this
paper? We got some great VPLs like Scratch and Alice. Truly
magnificent tools if you want to teach your child to program without pre-mature
exposure to stuff like object orientation, pointers or monads. If you
want to create your own audio effects pipeline, or a funky 80’s revival style
synthesizer, or a midi step sequencer you can save yourself a whole lot of
frustration and wire up all your needs in Pure Data or Max/MSP, which is (trust
me) way better than rolling you rown in <fill in your favorite functional
programming language>, even if you pull in the best available libraries from
hackage or the likes.

There are many, many visual domain specific languages. You want to
process your lab data? You want to control a robot? You
want to build a visual effect chain for your newest Youtube animation?
You want a language that keeps itself simple and stupid enough so even your
livelong <stuck in middle management, not having a clue> boss can use it?

These domain specific needs can be checked without a hitch, but when you look
at the popularity of general purpose VPLs they a far cry from even Visual
Basic? Sure there are some broad-purpose VPL’s. Microsoft has made one,
MIT has one. There’s an open source tool called “Programming without
coding“, and there are others, but none of these are considered to be a serious
programming environments that could be used by a professional. General purpose
VPLs same to be stuck inside specific domains, research and education.

And there are a couple of good reasons for this. First of all, VPL’s, albeit
being developed early, came late to the real party after the first waves of
personal computers hit the market. It took quite a while for computers to be
outfitted with a mouse. This left little room for the graphical alternatives.
But there were also serious issues with visual languages itself.

Take for example scalability. A hello world program might look nice and dandy
, but when you want to make a complex program you will need to be able
to tidy up your act by putting everything neatly into separate parts. In
imperative and functional languages we have pretty much fixed the problem of
scalability, by putting our code in different modules, or classes, or using
namespaces and packages…etc. In visual languages this is more difficult to
achieve. The tendrils of your system are out in the open, and more in your face
than in a written language. If you don’t mitigate this then the cross-program
dependencies you have rear their ugly heads and turn your program into
spaghetti, which visualised looks pretty gruesome.

Then there is the problem of expressiveness. With program languages there often
seems to be a trade-off between ease of use and expressiveness. The more dummy
proof a language, the more pain and sweat it will take to get some serious work
done. Anybody who has done stuff in the Commodore 64 supplied Basic , or
in Java before it supported anonymous classes and generics should know what I
mean. When you look at the many available VPL’s then you will notice that most
of them have settled for ease of use, which is of course fine of you’re into
creating toy projects or sticking to one domain, but in this specific case of
wanting VPL’s to take over the world and converting all programmers in pinching
swiping gurus of the touch screen, this simply won’t suffice.

The last obvious problem has to do with culture. Programmer culture tends to
move slow. It took Java 20 years to get lambda expressions. Something
which has been a great idea ever since Alonzo Church introduced the
Lambda-calculus in the 1930’s and proven to work extremely well in practice
since the implementation of the first Lisp in the 50’s. So advancements in
programming languages propagate slowly, we tend to stick to old languages for a
long time. Sometimes there’s good reason for this, when we prefer stability
above everything else. I guess that’s the reason why there are still poor sods
out there maintaining decades old Fortran codebases.

But besides the languages there’s also a certain conservatism surrounding the
tools with which we write our code. I for example am an avid user of the
VIM-editor and when I’m working in my tiling window manager with terminals
plastering my screen, I almost never need to reach for the mouse. Geeks like me
will need a very good reason to actually pick up that cabled clicky thingy that
lies next to my keyboard when I can instead keep the fingers on the keys.

But still. I think there’s great merit in visual languages. The cultural issues
I mentioned can be solved with time, the other issues by adopting the right
constructs from computer science research and functional programming.
Scalability can be solved, and has been solved, by choosing correct ways to
create modules. Expressiveness can be added by taking cues from homoiconic
languages like lisp that transform beautifully to the graphical space, and
adding higher order functions, purity, laziness.

We, humanoids, are visual animals. To make sense of how an algorithm
works I visualize it. When I try to make sense of a large code project I use
it’s file and directory structure, modules and packages as a visual frame of
reference. If we visualize a project correctly, abstracting away the details
when we don’t need them, and providing an easy way to dive into the nitty
gritty bits when we want, we can find ourselves in a place where it can be
easier to reason about our code and more importantly explain this reasoning to
others. So the scalability issue could in fact be turned upside down and
changed into a strength if we take the right interface designing path of
modular touchy swipey goodness. Something we might need actual interface
designers for. And like they do we would need to break out of the computer
screen and look at the smelly beast sipping red bull in front of it. We would
need to find out which different mental models programmers use, and how we can
transpose those to visual elements. And we will need to figure out the
cognitive dimensions of those visual elements so we can trace out a path for
improvement.

I can list many reasons for trying to create a new VPL that rocks the world,
but one pet peeve of mine are compile-time errors and especially  syntax
errors. Aren’t these the most stupid, time wasting things ever. So I’m typing
all this code and after my IDE doesn’t show any of those curly red thingies I
can safely press Compile, only to be confronted with a load of messages about
all the obvious ways in which I suck, and my program is incorrect. And this is
a totally solved issue, I mean, compile time problems are mostly low hanging
fruit. In the case of Haskell with its nice type system, also to somewhat
higher branches, but still. Why can’t we eliminate these completely? “We can”,
you might say, because I use, Eclipse, or IntelliJ and that IDE happens to be
very smart. WRONG. They stink. Why is it even possible to write something that
is so blatantly wrong, low hanging compiler or lint-checker, fruit? It feels a
bit like driving in a car in which I will have to control the cooling system
manually and when I start the ignition make sure I don’t flood the engine by
quietly murmuring obscenities.

It seems that while the abstraction level of programming languages have
increased, the errors themselves are still stuck in the elevator.
Error-catching wastes more time than ever. Many programmers of dynamic
scripting languages see no problem in actually going through the mind
numbing process of first executing their program in order to check for errors,
and I’m not talking about highly parallel programs that can’t be easily
debugged in another way. Madness!

Anyway, I think that we can create a new visual programming language, by
combining more than fifty years of research with the lessons learned in
functional programming. I have foolishly made a start:

Here’s what we do: We start with a typical boxes and arrows, directed
graph, flowchart kind of language. We add the ability to zoom and hide details
when using a touch screen, and we support laziness, currying, higher order
functions and a module system. We make functions pure by default and add
necessary, but evil side effects by surrounding them with a visual code smell.
Something like monads, but minus the unnecessary but frequent occurring fears
of category theory.

Taking a cue from Lisp it’s a great idea to have one basic data type and using
that for both code and data. One problem in the written world is that
homoiconic languages tend to look like a too many parenthesis in a love
shack, but work pretty well when visualized as graphs.

We can switch between different list representations at will  , or draw
new ones if we like. We can also create symbolic expressions by pointing lists
to functions with arrows. We can make a bigger function with nested functions
by drawing a box around them. We can drag lists around and create
immutable copies if this helps in our evil schemes. You can choose to direct a
partial list for further processing, which helps create  a nice head or
car function.

We can add labels to our functions, or larger encompassing structures.
But since we’re working visually, we could do even better.

I will show you how we can implement a map function.

We need a function and a list for our map function. And it’s nice if the system automatically adds color to them so we can keep them apart.

We can make immutable copies and change the list representation to get the granularity we need.

We’ll also make a copy for the function that you supply to MAP.

We use recursion to call map on the provided function and the tail of the list

We apply the head of the list to the provided function

We concatenate the results

Now, what happens when we give Map an empty list? We will need to check for this, by adding an if statement.

The if statements returns an empty list when you try to apply map to an empty list, otherwise it will return the result of the concatonation.

What if we want to add a side-effect to our map function?

We are launching the missiles as a side-effect and this makes the entire function impure, so our red background propogates to the entire function.

Here you have it, we have made a higher order recursive function that doesn’t like
like complete rubbish. You can travel along the recursive steps if you like by
zooming into the nested map function, enjoying the droste effect, or strange
loop.

Alright, so what now? I’m still figuring out the best way of adding types to
this visual notation. If I want to protect the programmer from silly errors
while he’s trying to make them, I will need a strong type system, and
preferably one that is light and stays out of your way if you want to by using
type inference.

I will make an informal description of this language and follow it up by
describing everything formally in typed lambda calculus. After this I will make
a first browser based implementation, using a simple graph reduction approach.

I will post updates on the progress on on this blog.

While I show you a possible implementaton of quicksort I would like to ask you
to give visual languages a chance in the future. When you find yourself needing
a DSL, or like me you’re toying around with language design, maybe you can pick
a visual language instead of a traditional one. I know it’s a bit harder,
because you will need to make do without lexer and parser generators , but I’m
confident that slowly but surely we are moving in this direction. At first we
might see smarter IDE’s that add graphics to tackle the complexity of large
projects, but I think the touchscreen revolution isn’t stopping, and one day we
might find ourselves developing software by touching, swiping, jabbing and
smooshing.