A text editor in a shader

The text shader is now usable as a text editor, complete with the highlight blocks for selection and Lua do/end blocks.


Version: 7d64ce7

Working directory: cauldron


Hitting F5 and F6 renders the game of life shaders underneath the text editor:



Call disableStdMouseCam() and enableStdMouseCam() to allow you to switch between moving the camera and interacting with the cellular automata shaders. (Reminder: Ctrl-Enter executes the current line or do/end block)

So now there is a shader based text editor in praxis. This means no more massive performance penalties when I want to render text in praxis.

It runs quite well on modest hardware too. It runs very well on a HP Stream 11 laptop (running Linux Mint 17), which I run praxis on regularly.

The text shader in the previous post was rendering to an FBO. In this version, I’ve made the text shader draw directly to the quad. So far, it looks like I can render as much text as I like with this technique with no noticeable performance hit (apart from the GPU possibly getting hot). The source file textshader.lua initially sets the visible area to 40 lines and 120 columns. That is already quite nice, but lets try pushing it a bit.

With praxis running, load a file into the editor, a file with at least a few screenfulls of text.


Then in the file textshader.lua, change the line that sets the number of visible lines. Around line 281, increase the number of visible lines to 120 and save:


If you press F7 now, those visible lines are written to the “string texture”, but they aren’t appearing yet because the shader variable iResolution.y (the starting point used in the fragment shader function FragCoordToCharPixel_Plain) is too low. So increase that around line 226 to 1500 (doesn’t have to be a power of 2):

glUniformf(u.resolution, 512, 1500)

Then you also need to change the fragment coordinate scaling multiplier in textshader-fragment.glsl so it maps to the quad properly. The quad is 100×100, so for 1500 “pixels”, we need to scale by 15.0. Around line 148 in textshader-fragment.glsl, make this change:

mainImage(gl_FragColor, V.xz * 15.0 )

Save the files you changed if you haven’t yet, then press F7.


120 visible lines. No problem. Zoom in, move around, its all there:


Here is a video demonstrating these steps:

And a shorter video because why not:


Well, that was fun.

Some Shadertoy examples

As part of the process of studying OpenGL shaders, I’ve added just enough extra shader support to praxis to run a few of the examples in Shadertoy. For all the following examples, use the version of praxis from git commit ffeca91 and set the working directory to “cauldron”.

The first thing I wanted to be able to do was to “draw” to a bitmap with the mouse. As a prelude to this, I added support to praxis for running this simple mouse input example from Shadertoy. Among other things, this support included basic stuff like glUniform for example. Press F4 to run this shader example.


A drawing shader also requires a persistent buffer to draw to. I added functions for creating and working with FBOs, in particular, a function that would render to an FBO given a shader and other FBOs as inputs (or “channels”). To see a simple persistent “drawing” shader in action, modify the prerender() function in gameoflife.lua so it uses the draw shader instead of the game of life shader. In particular, change the line

render_to_fbo_with_input(g.fbo_curr, g.mainshader, g.fbo_prev)


render_to_fbo_with_input(g.fbo_curr, g.drawshader, g.fbo_prev)

Save it, then press F5 in praxis.


After I got that working, I thought an obvious mouse interaction thing to do with shaders would be cellular automata, such as Conway’s Game of Life. This plays to the main strength of shaders which is that you get to customize the per-pixel calculation that gets performed by the GPU. So I added enough support to praxis so it would run a Game of Life implementation I found on shadertoy with absolutely minimal modification. To run it, undo the change you just made to gameoflife.lua so it uses mainshader instead of drawshader, then press F5 in praxis. You can click and drag with the mouse to interact with it.


Zooming in to see detail:


So far so good! Given these successes, I tried running other shadertoy examples that caught my eye. I very quickly found one that didn’t behave the same way, and became curious as to why. The shader in question was this very interesting looking cellular automata. This is what it looked like when I tried to run it in praxis the first time:


After tearing away at the innards of this shader trying to see where it behaved differently in praxis than it did in shadertoy, I finally discovered that the textures needed to be standard 32 bit floats, PER COMPONENT. Red is a 32 bit float, green is a 32 bit float, blue is a 32 bit float, and alpha is a 32 bit float. This never occurred to me initially because I was so used to components being a single byte (0-255), resulting in a 4 byte pixel. This is after all the only information that gets actually rendered to the screen. The calculations in this thing used the full range of the floats in the pixel. What got rendered to the screen was a modulo of this number, resulting in automata that seemed to have a lot of internal state. Which of course, they do have! So after merrily changing the glTexImage2D call in luaCBGLPrepareFBOTexture so it gets passed a GL_RGBA32F_ARB parameter and a couple of calls to glClampColorARB to make sure there is no clamping, this example started to work correctly too. To run this shader, press F6 in praxis. You can click and drag with the mouse as before.


Almost forgot: it was at this point that I also added the ability to specify the texture filtering parameter. For this example, even though my tastes are usually for the hard pixelly goodness of GL_NEAREST, this one needs GL_LINEAR for the effect to work, or it just ends up degenerating into a very different, somewhat less interesting effect:


Lastly, I’ve had the pipe dream of being able to render text in a shader for quite a while. Rendering the text editor in a shader has always been something I’ve wanted to try doing since the text editor has always been heavier on the rendering load than I’d like it to be, and shaders seem like a natural way to lighten the rendering load of the editor significantly. I call it a pipe dream, because there is very very little about doing this on the internet generally. Fortunately, some excellent programmer on shadertoy provided a beautiful example of a text rendering shader that matches what I wanted almost perfectly.

This thing makes 3 passes with 3 different shaders, and it will require praxis to run a shader with more than one texture channel as input. The first pass creates a font texture from a font encoded in a shader. To get this to run on the 3 machines I tested it on, I changed asciiToSprite to use the mix function instead of the series of if statements mechanism, because one of the computers I tested it on complained about nested if statements. They aren’t nested, but the code the glsl compiler generates on some machines is so I did this workaround. Weird thing is, the webgl shadertoy version which uses the if statements still runs on the computer that complained, so go figure. Maybe Shadertoy does a pre-compilation step. The second pass makes a texture that encodes a string. This required no modification. The third pass takes the font and the string and renders the text. This also required no modification.

Once I made that one change to the font shader, this shader example worked well in praxis. I was pleased at this result and invigorated toward my next goal, which was to have the string texture come from an actual string instead of hard coded in a shader. The font shader can remain for now, especially since it only has to run once to generate the font FBO anyway.

As I mentioned, the second pass makes a texture that encodes a string. When I made a texture from a string, taking care to allocate the texture correctly and follow the conventions and assumptions about the texture the code seems to make (first byte in the floats is zero, next three are the chars), it wouldn’t display what I expected it to. I investigated by hardcoding inputs to functions within the shader to verify the operation of these functions. When I then tried passing the same data but from the string texture, the results didn’t match. Was there something OpenGL was doing to the texture before passing it to the shader? I checked the memory – no, the data is all there, and it was correct.

It took me a while to realize that even though its written in the string shader as raw ascii bytes, those raw ascii bytes get interpreted as a single number that gets converted to a float as that number! For example, 0x616263 which corresponds to the string “abc” and which would look like 0x616263 in memory if it were a string, doesn’t appear in texture memory as 0x616263! Those 3 bytes need to appear as the 4 byte floatĀ 6382179.0, which is 0x616263 in decimal then cast to a float.

Expressed in one line, its much easier to see what will really happen – the 0x616263 won’t be interpreted as raw memory, but the actual number to set the float to.

float val = float(0x616263);

When I realized this I went hunting for how to get the raw bytes. I also realized this would add the happy benefit of letting me use all 4 bytes instead of being restricted to 3 (because you can’t represent the magnitude of a 4 byte integer in a float). Once again, the internet has very little to say, and what it does seem to say is mostly misleading.

Through some persistence, I stumbled across this:


Exactly what I wanted.

With this realization, I was finally able to implement the magical glStringToTexture function.To see it running, press F7 in praxis.


To set your own string, use the following incantation to rebuild the texture at any time:

stringtex = glStringToTexture("your string")

So that is the saga of adding some shadertoy shader support to praxis. As always, have fun, and happy programming!


Fun with shaders

While slowly moving toward modernizing the OpenGL used in praxis, I discovered that playing with shaders in OpenGL just requires a few fairly straight-forward function calls, so I’ve added the ability to define and use shaders in praxis.

Version: 445f789

Working directory: syntax2015continued


airplane.followcam = false

Then hit F1 and move the camera over to one of the beachballs in the sky. You can change the definition of the shader in beachball.lua and hit F1 to see the effect. In addition to beachball.lua, there are 2 other example shaders: toyball.lua and checkerboard.lua. These shaders are the example shaders from the book OpenGL SuperBible, 4th Edition.


Things to try in praxis

Instead of writing a long blog post for every feature of praxis, blog posts that never seem to get written, I’ll just show you some lines of code you can type into praxis to achieve some fun things.

Since praxis is constantly changing (the prods are at any rate) I’ll also add a link to the commit of praxis these were tested with. Try the latest one first, but failing that, try the linked historical one.

The following is runnable using the latest version of praxis at the time this blog post was written, which is this version.

With your working directory set to syntax2015continued type the following, followed by Ctrl-Enter to execute it:


Then hit F1. Hexagons!

Side note: its called syntax2015continuation because its just my continued playing around after working on the Flight of the Spironaut Syntax Party 2015 entry.




MIDI recording in praxis

I’ve recently implemented a simple MIDI recorder in Lua for praxis. You can see it in action here:

After loading the file midirecorder.lua in praxis (either with dofile or by loading it into a buffer and hitting F1), the MIDI recorder becomes active in the system, and the following functions become available:

  • beginRecording()
  • beginPlayback()
  • stopRecorder()

In addition, you can see the collected data and save it using the method demonstrated in the video by using “inspect” to show the contents of the recorder.notes table.

The way the recorder works is simple. When recording, every frame the recorder checks to see if there are any MIDI events that have been received. If so, it adds them to the table along with the frame number the events were received in. When playing back, the recorder sends the MIDI events in the same order and on the same frames that they were received. Naturally, the precision of this system depends on how high and consistent the framerate is. In practice, I have found that 50 frames per second is plenty for faithfully capturing a lively 3 voice fugue that I may be playing on the MIDI piano. Running midirecorder.lua doesn’t explicitly change the update() function. Instead, it adds a widget containing only an update function, since update() has a call to WidgetLib.callAll(“update”)

I wrote this because I wanted a MIDI recorder in praxis. I have a MIDI piano hooked up and it seems like a natural thing for praxis to be able to do.

Real-time audio in praxis

Praxis features an audio engine that allows your program to synthesize audio in real time. A demonstration of this can be seen in the Synthbench program, which synthesizes audio and renders the waveforms it generates. It emulates the style of the Reactable system, where you arrange widgets on a table to generate a sound. Some widgets emit a waveform, some widgets apply an effect to the waveform they have been placed on top of and some widgets emit a low frequency waveform for controlling a parameter of the widgets it is connected to. Once again, I made this because when I saw the Reactable I thought it looked really fun, so I wanted to make a version of this “toy” for myself so I could play with it. That’s what praxis is all about – making your own universe of play where any toy you care to describe can be brought to life.

The version of Synthbench that exists in praxis only features the node audio generation behaviours and rendering. What is missing is being able to place new nodes and move them around with the mouse: you need to use code to add new nodes and move them. This is purely due to laziness. I chose to write the part of Synthbench that I wrote mainly to test that real time audio generation could be done fast enough in Lua. That is, each node needs to generate and render enough samples per second to satisfy the audio pipeline while maintaining 50 frames per second. After I was satisfied that it works, I let my attention drift to other things.

The reason for praxis

When I saw what was possible with Lisp Machines, Smalltalk environments (Squeak, Croquet) and Emacs, I realized that what I loved most about those systems I could create for myself very easily just with an embeddable language and a render loop. The most valuable ingredient that makes this work really are the embeddable languages, so thank you so much to the makers of Lua, FICL Forth and s7 scheme for making these beautiful languages accessible through a C API.

I’m surprised that systems like praxis aren’t more common considering how fun it is and how easy it is to set up, or maybe its just that while this style is perfectly suited to my taste, its not necessarily what other people prefer.

One way of looking at praxis is as a repl augmented with a graphical and audio engine to use as you see fit. The socratic dialogue you have with your world as you define it and build it is enhanced by visualizations you are free to add in any form and at any stage. For instance, say you are making a simulation, and part of it calls for an implementation of AI strategy which involves exploring a complex algorithm. You can change the render function to not run the usual simulation, and instead temporarily focus on the behaviour of the AI algorithm. When you are satisfied, you can swap your old simulation render and state back in and pick up where you left off.

Although the primary intended way of interacting with your world with praxis is through the programs you write in the editor, there are many Lua calls that are generated from input events that you can redefine. In addition, the graphical engine has features such as a pick cursor and a non-banking mouse camera movement model with the ability to change the ground plane (orbiting, panning, moving in and out) which when combined with the introspection facilities of Lua and commands that allow you to modify the text edit buffer, allow you to add powerful customized editing features to your environment. For example, at the press of a button you can have praxis perform a simple behaviour such as inserting the pick position into the program, or something more complex such as inserting the definition of the update function for the object the mouse is currently hovering over so it can be edited. This was inspired by the “code gun” idea in the “Code Hero” project, where a piece of code can be “fired” at an object in the world and it would be applied to the object that was hit. The same thing can be achieved in praxis for example by setting the definition of LMBDown to be: find the object the pick position is nearest to, then run the code in the “code gun” with this object as a parameter.

So praxis is the result of me implementing all the various things I’ve seen in other live coding environments that I thought was cool. The Lua files that come with praxis are a little messy, because I make something and then get distracted by the next shiny trinket I want to make. I’ve added things like portals you can peer through (the sort you see in the Croquet Smalltalk project) which I’ve implemented with stencils, midi recording, a modifiable camera ground plane, being able to draw to textures and use them and other assorted bits and pieces. When I added s7 scheme, I got some SICP code that I liked working with it (I like all of it naturally, but specifically, the symbolic algebra stuff). I added Forth just because I like it. In praxis, another key idea is that you can recover from any fault, including an infinite loop, or even a fault deep inside the call to render. The editor and a basic ground grid is always available so you can see where the problem occurred, fix it, and continue the render/update loop with minimal disruption to the state of the system you are making.

Please enjoy praxis, and I hope you create many wonderful universes with it.