Speaking of the GUI, it has become quite a magical environment. Widgets respond correctly to keyboard commands and focus events. We’ve been paying a lot of attention to little details. The look has been completely refreshed from the alpha release, following a more modern layout style. The GUI also scales with the browser vertically, leaving less screen real estate wasted:

A precise E.T.A. for the public release is hard to give. We’ll try to post more frequent updates as the 1.0 target approaches. Thanks for hanging on.

Bookmark It

Another reason why I don’t worry about art critics in this context is that, even if Grace were a library with only a single user, it would still help me get my stuff done in a way that I enjoy. It has already proven its value and given me an excellent return on investment using it in a lot of roles. It scratched my itch and that is fine.

I do find some of the negativism on sites like reddit interesting as a phenomenon. I’ve thought of this as a factor of the functional programming cartel that seems to be hanging out at such places. And I realized that this way of looking at it is just as dismissive and childish, so it made me wonder how it can be that there seems to be this great wall between that crowd and the typical ISP/Unix nerd demographic I normally interact with.

When looking at software, I reckon there are two approaches. Some people, when they think about code, see a world of math. For me, what I mostly see is flying bits, an interconnected lego-world of action-reaction patterns that makes intuitive sense. I think both approaches are valid, but obviously they lead to a completely different view on software development.

Naturally I may be suffering from confirmation bias on this subject, but I get the impression that those of us that are more into this direct approach to programming are the ones producing most of the real, living software out there. I can definitely see areas where a more distanced and abstract approach like functional programming can make a difference, but a lot of software development is really about moving bits from A to B, more about rolling up your sleeves and building it than mulling about algorithms and monads; lego, not math.

Bookmark It

When we started the design, we wanted to focus on the following demands:

1. OpenPanel should be modular, and modules should not be restricted to any specific language
2. There should be elementary protection against security defects in module code
3. There should be a possibility of multiple user interfaces (at the very least both a GUI and a command line tool)
4. The system should anticipate the possibility of remote control and clustered integration
5. Configuration provided by OpenPanel should follow the standard configuration practices for the programs involved — no funky business relying on access to a MySQL database
6. It should be possible to dedicate an installation to a single purpose (like mail) without requiring unrelated services (Apache, MySQL) through dependencies

What we came up with, is a design that could best be explained as a domain-specific object database that reflects changes into the configuration of outside programs. The opencore daemon implements this abstraction. At the bottom it has an sqlite database that stores object and class data. Modules are implemented as directories inside /var/opencore/modules that define object classes for the database and the program that executes the necessary changes to outside configuration related to objects of those classes. A JSON-based RPC interface accepts requests through HTTP. This backend port is also used to serve static files related to the web-based GUI.

Obviously, accepting remote RPC requests is something you don’t want to combine with root privileges. Also, since we wanted to keep a low number of privileges exposed directly to module code, running the back-end programs/scripts for module actions was definitely not something we felt should be done as the root user. This is where the authd daemon comes in. It accept requests on a unix domain socket that is only accessible to the opencore daemon. Whenever opencore runs a module program, it will fork to a new process, open a socket to authd, tell authd which module will be executed and drop the privileges needed to open any new connections to authd. The unprivileged module can talk to authd on fd 3, but will be restricted to those privileged actions defined in its module.xml file.

Documentation for the Module API and the RPC Interface is on our site, but some of it may still be a bit rough around the edges.

Bookmark It

We’ll be adding a dedicated screenshots section to our website soon.

Bookmark It

We went through the hoops submitting the news of our release to a dozen of the usual suspects, got a couple of hits and the first testers have arrived. There’s even a couple of people around doing more involved things like trying to port packages to other distros. It’s not bad and we appreciate all the people helping us out by playing with the beta, but a stampede it is not. Building a community takes time.

In terms of strategy, this may be the real reason for the ‘release early, release often’ mantra. Especially when your project is in a niche area, expect community building to take up maybe just as much time as actual development. We wanted to avoid getting distracted by actual users too early in the game, so our gut feeling at the time was ‘early, schmearly’. We had a couple of other reasons, some of them even sounded real good — like how there are too many open source projects in the field that need so much polishing that the market has been mostly ignoring them. But the fact of the matter is, all the hours of community building we’ve been avoiding up to the beta release will have to be put in double now. We’ll need to go from our current mode of reclusive introversion to full-blown extravert. Tough job for a bunch of nerds.

Bookmark It

We spent over a year creating something with the capacity to change the landscape, by combining the proper amount of openness and flexibility with a state of the art user interface and a license that should make trumpets sound from the heavens. People we’ve shown it to have been raving. But, now that we’re at the point to hit the submit button for all this on sites like freshmeat, all those little doubts start setting in. It’s scary as hell.

I’m convinced we’ve taken the right step by not fully adopting the ‘release early’ mantra — it has given us the chance to quickly reshape the design where needed (Heck, we’re on the second incarnation of the GUI). But this is the downside. If you release early, expectations are lower and you never get this neckhair-raising fear of throwing your work into the world. Alas.

We’ve opened up the 0.9 (beta) branch of OpenPanel as of today. There’s no longer a requirement to email us to get access, just follow the download instructions from the main site. Messages have also gone out to the usual places. The future is now.

Bookmark It

We’ll pick up the last bugs we found monday and then seed a release candidate to our alpha team. If no show stoppers emerge, that will be followed up shortly by a full public release of the OpenPanel beta.

Bookmark It

One area where the concept of protocols is quite convenient is iteration. There is usually a wide area of possible classes that could, theoretically, enumerate a list of contained items. Although C++ lacks protocols proper, template iterators are a fine way to access any class that implements a de facto iterator protocol. The only thing missing is an explicit declaration of the implemented protocol.

The Grace iterator<collectionclass,nodeclass> template is a minimized version of the visitor<> template. It requires one function (visitchild) to be defined inside your class that returns a pointer to a child node. Here is what it would look like for a purely synthetic class:


class syntheticlist
{
public:
   syntheticlist (void) {}
   ~syntheticlist (void) {}
   string *visitchild (int atpos)
   {
       if ((atpos1)) return NULL;
       if (atpos==0) str = “Hello”;
       else str = “world”;
       return &str;
   }
protected:
   string str;
};

int myApp::main (void)
{
   syntheticlist L;
   foreach (n,L) fout.writeln (n);
   return 0;
}

The foreach macro goes through the following steps:

1. It creates an iterator<syntheticlist, string *> pointing to L
2. It sets up a for loop that starts at visitchild(0) and stops when it returns NULL
3. It creates a temporary reference variable n linked to the current node.

The Grace iterator protocol looks a bit ascetic with only visitchild and the lack of the traditional first() and next() pattern. Part of this is due to the visitor protocol (which iterator is a part of) being about more than just of iteration. If your goal, however, is to make your class iterable with foreach, you can safely assume an argument value 0 to mean ‘first’ and any value bigger than 0 to mean ‘next’ if that makes more sense in your context. Most collection classes used within Grace use growable arrays, which means they don’t have to keep any state information in order to be iterable.

A lot of modern languages offer a friendlier form of memory management for this reason. They keep track of all references to a particular object and regularly take care of objects that are no longer referenced. This garbage collection process has a number of drawbacks that are usually seen as justified, considering the programming problems it solves. And it does solve them, but not all problems it solves need to be problems and, once you take those non-problems off the list, the smell of over-engineering becomes apparent.

What I see as the most interesting memory management problem that garbage collection solves can be illustrated by the following snippet:

Storpel *createStorpel (int modelNumber)
{
   Storpel *res = new Storpel;
   res->setModel (modelNumber);
   return res;
}

int main (void)
{
   Storpel *myStorpel = createStorpel (42);
   myStorpel->grind (11);
   delete myStorpel;
   return 0;
}

The problem with this pattern is obvious: The receiving function, by calling createStorpel() has taken responsibility for managing the Storpel object. This can get hairy in almost zero time.

In Grace, I prevented this anti-pattern from emerging by sticking to a number of principles:

• Use of pointers and new for objects is minimalized (pointers are awkward when you want to use overloaded operators anyway, so there’s exterior motivation).
• Data container classes are designed to be mere vessels for a raw data block that can be taken away from one object and ‘given’ to another without copying.
• When confronted with a pointer during assignment and initialization, data container classes will ‘take over’ the other object’s data and immediately delete it.
• Functions that want to return objects actually return pointers to temporary objects. The returnclass (type) name retain macro creates such a temporary object plus a temporary reference to make it easier to access the object.

Words are words, code is code. This is what it effectively looks like:

string *getGreeting (void)
{
   returnclass (string) res retain;
   res = "Hello, world.";
   return &res;
}

int main (void)
{
   string s = getGreeting();
   fout.writeln (s);
   return 0;
}

Behind the curtains, the returnclass macro creates a temporary string object using a custom allocator. This string object creates a persistent refblock structure to hold the data. Then, inside main(), we assign it to s:

The receiving object takes a reference to the underlying refblock:

And then the original object is immediately destroyed:

When main() goes out of scope, the ’s’ string object will be automatically cleaned up and, with it, the refblock. No need for an explicit delete.

As for the other problems that garbage collection solves, most of them have to do with interactions between collections of more complex objects. Mostly, if you find yourself in need of tracking such complex relationships, you may need to look at a way of simplifying your design. If that is not possible, then keeping track of memory references takes no more diligence than what the structure already demands.

The C approach — having this mutable array as a working area — makes it relatively easy to hack and split strings into smaller parts and work with them as first class strings that can be used as arguments for other functions. All without ever copying a byte to a new object.

The Grace string class has grown a lot of nice features that make it easier to forget the feeling of loss that accompanies the sudden inability to cut up strings by spraypainting them with NUL bytes. With methods like string::left, string::mid, string::cutat and the strutil::split family of functions, a lot of splicing joy can be had for all.

The storage behind a string object uses a reference count and copy-on-write to deal with assignments and mutations. Assignments can be a real pain in the context of strings, which is why a number of languages recognize the concept of immutable strings. These languages make you go on string building expeditions to dynamically compose new strings, but they do this to allow for assignment without copying; if both the sending and the receiving object guarantee not to alter the data, it is safe to let them point to the same memory location. Copy-on-write satisfies this same ‘zero copy’ approach to assignment, but allows strings to remain mutable. The cost of copying is at best prevented, at worst delayed until the moment of mutation.

With all the hacking, cutting and copying kung-fu under its belt, Grace strings made me stop missing the C array approach, except for one thing: Each time you split up a string god killed a kitten data got copied. This week, the string class got a major overhaul: Next to the pointer to the copy-on-write back-end memory that each individual string carries, it now keeps track of an offset. So now, when you do this:

string s1 = "Hello, world."
string s2 = s1.cutat (' ');


what actually happens is this:

That means even less copying. Some unexpected things now trigger a copy-on-write, though. The most sad one is conversion to a C-compatible string. Since C expects a NUL character, a sub-string will have to mutate, triggering a copy-on-write.

I’ve never been envious of Python. I’m rational enough to just forget about it all and join the dark side if that’s ever going to happen. I was, however, particularily fond of the Python command line interpreter. Of course this idea has been around since the invention of, well, the shell, but one thing it accomplishes is that it allows you to quickly try out the effects of different commands and constructs without opening a text editor. This is a place where compiled languages tend to suck.

This week, Peter lamented the absence of this feature as well. He mumbled something about ‘just feed it through gcc’ and probably expected that to be the end of it. So last night I built a Grace shell. It uses some awfully tacky hacks to do its business, but I didn’t even have enough time to accumulate my distaste for that once I started noticing how completely addictive working with it is. It takes away a lot of anxiety that would normally have you running for the documentation by letting you just explore what happens when you call an API function.

Here’s the shell in action, working out the value class:

>>> value p
>>> p = $(”name”,”John Doe”)->$(”mail”,”j@doe.org”)
>>> p[”url”] = “http://doe.org/~j/”
>>> print (p.toxml())
<?xml version=”1.0″ encoding=”UTF-8″?>
<dict id=”person”>
    <string id=”name”>John Doe</string>
    <string id=”mail”>j@doe.org</string>
    <string id=”url”>http://doe.org/~j/</string>
</dict>
 
>>> foreach (f,p){
  >   print (”%10s : %s” %format (f.id(),f))
  > }
name       : John Doe
mail       : j@doe.org
url        : http://doe.org/~j/


The shell already exposed a number of bugs that would’ve taken much longer to get themselves caught otherwise. People with access to our mercurial repo can find gracesh in /hg/.


#include <grace/daemon.h>
#include <grace/httpd.h>


class myApp : public daemon
{
public:
   myApp (void) : daemon (”myapp”) {}
   ~myApp (void) {}
   int main (void);
};


class helloPage : public serverpage
{
public:
   helloPage (httpd &p) : serverpage (p, “/foo”) {}
   ~helloPage (void) {}
   
   int execute (value &env, value &argv,
                string &out, value &outhdr)
   {
       if (! argv.exists (”target”))
           argv[”target”] = “world”;


       out = “Hello, %[target]s\n” %format (argv);
       return 200;
   }
};


$application (myApp);
$version (1.0);


int myApp::main (void)
{
   httpd srv (8180);
   value event;
   new helloPage (srv);
   daemonize ();
   srv.start ();
   while (event.type() != “shutdown”)
       event = waitevent ();
   httpd.shutdown ();
   return 0;
}

Admittedly this is a bit more than 15 lines (even if we don’t count lines with curly brackets). We do get a little bit more oomph for those lines, though:

• The application checks a pid-file in /var/run or ${HOME}/var/run.
• The application is forked to the background
• The application responds sanely to SIGTERM

Having an easily extendible web server built into your framework turns out to be pretty convenient: It’s a very quick way to deploy a remotely accessible service.


void printListOfPersons (const value &persons)
{
 foreach (p, persons)
 {
   fout.writeln ("%s <%s>" %format (p["name"], p["email"]));
 }
}

Getting data into such a structure posed to be a bit of a challenge, though. You ended up with huge blocks of code with lots of repetitions:

persons["john"]["name"] = "John Smith";
persons["john"]["email"] = "jsmith@example.net";
persons["steve"]["name"] = "Steve McSmith";
persons["steve"]["email"] = "steve@example.net";

The repetitive nature wasn’t only straining on the hands, the build-up here also involved looking up the node for ‘john’ and ’steve’ twice. Using the negative array index to refer to the last node was a nice timesaver in this respect:

persons["john"]["name"] = "John Smith";
persons[-1]["email"] = "jsmith@example.net";

The problem with this approach, apart from it still being pretty repetitive, is that it starts you thinking that the idiom persons[-1] means “the node I created in the line before”, but there are lots of situations where you cannot really know this. Take this example:

persons = listPersons();
persons["john"]["favoriteColor"] = "blue";
persons[-1]["favoriteSwallowType"] = "African";

All it takes for this code to break is the inclusion of the key ‘john’ in the result set of listPersons() at a position that is not the last one in the array. This may sound like a mistake that is easy to avoid, but in an application that sees a bit of growth, the assumptions you make in this respect may stop being valid in the future, and you’ll have forgotten all about them.

The solution to this problem came to me as an inspiration from jQuery’s clever discovery of this one symbol in the C-like namespace that managed to remain completely overlooked by most library builders: The dollar sign. On the global level, two variations of the function $(…) are defined that create a retainable pointer to a value object:

• $(”value”) creates an array and adds a node with “value” as its value data.
• $(”key”,”value”) creates a dictionary and adds a node with the key “key” and “value” as its value data.

These functions create a pointer to a value object, so the library also defines value::$(…). Now you can chain nodes together, like this:

value v = $("john",
             $("name", "John Doe") ->
             $("email", "jdoe@example.net") ->
             $("favoriteColors",
                 $("Titanium White") ->
                 $("Cadmium Yellow")
              ) ->
             $("favoriteNumber", 42)
          ) ->
         $("steve",
             $("name", "Steve Jibs") ->
             $("email", "sjibs@example.net") ->
             $("favoriteColors", $("Black")) ->
             $("favoriteNumber", 1)
          );

The result is pretty compact on a scale from 1 to JSON. There are some extra functions that will come of use:

• $attr(key,value) Sets an attribute.
• $type(type) Sets the value’s type().
• $val(otherval) Sets the value’s data only to that of otherval.
• $merge(otherval) Merges the child nodes and attributes of another value.

After these functions were added to grace, I started using them religiously. Inline declarations are a powerful way to keep your code readable. Thanks, jQuery.