fifo.nl

• jobs are not protected against running concurrently (design issue)
• intervals are fixed (design issue)
• people tend to schedule the same job at the same time on whole clusters of machines (typical usage issue)

The second and third issue are closely related in that both cause undesirable load spikes because many things happen at the same time, either because of intervals phasing up or because of similar jobs running on a bunch of machines at the same time, perhaps hammering the network.

A specific pet peeve is Mailman Reminder Day. Firstly, I just don’t see the point; if my address is on a mailing list and that list is practically dead, I just don’t care. Secondly, it means every first of the month I have tens of reminders that I just delete. Some of these lists are busy — for those, the reminder is a minor nuisance. But many lists are extremely quiet (think software release announcements) and for some of these lists, the reminders are over 50% of the total mail volume. It’s so wasteful. Also, I can’t help but think that all those reminders being sent out at the same time (well, divided over 24 hours because of time zone differences) cannot be good for the mail ecosystem as a whole.

For issues two and three, Colm MacCárthaigh wrote a few great posts detailing why cron is bad, and showcasing one potential solution to the issues at hand. I suggest reading these posts fully, they are very insightful:

• Period Pain
• Period Pain part 2
• Period Pain 3

A simplified version of my workflow would look like this:

def post(entry):
  title = entry.title
  print "posting [%s]" % title
  api.PostUpdate(title) # api is a twitter Api object

feed = feedparser.parse(config["feed"])
for e in reversed(feed.entries):
  if not e.id in seen:
    post(e)

It’s the print statement. All the APIs I’m using have zero trouble with unicode, but print wants to encode for your terminal and it’ll usually assume that that is ASCII. My ‘debugging’ output actually broke the program. My workaround is to say title.encode("ascii","replace")

Brend on #python pointed out to me that the issue is not, exactly, print. The issue is interpolating title into a non-unicode string. Depending on environment, using print on the unicode object might in fact work. For those environments, saying print u"posting [%s]" % title could help. In my case however, I ran into the issue from cron with no locale set at all, so dumbing the string down to ascii is still the right thing to do.

Kiwi

Echofon

If I only had one account to manage, I could seriously consider switching to Echofon.

Nambu

If Echofon or Tweetie seem limited or minimal, Nambu is a great choice.

Note that adding an account to Nambu means you suddenly follow two extra accounts (@nambu and the developer). The checkbox to disable this behaviour doesn’t seem to work. Spammy if you ask me.

TweetDeck

Yoono

Conclusion

a witty retort designed to make the audience go ‘oh, SNAP’

This one popped up on irc a few weeks ago; a google search suggests it's at least 6 months old. I have been unable to track the original source down.

The question is: if you input echo cat | sed statement into your Unix shell, what comes out? Can you predict it? If not, can you try it and then explain it?

• after upgrading+rebooting, your native IPv6 is suddenly replaced by a freenet tunnel. Cause: you installed gw6c on 9.10 which came with no configuration; 10.04 updated the configuration to automatically use freenet. Fix: remove gw6c package.
• you're using the excellent dovecot-postfix package, and after the upgrade, no mail comes in. Cause: cmusieve is gone. Fix: change dovecot config to use sieve.
• lighttpd will not start, complaining about port 80. Cause: lighttpd ipv6 is basically broken in 10.04. Known issue, not fixed. Fix: add server.use-ipv6="enable" to lighttpd.conf
• dovecot is not listening on v6 anymore. Cause: the upgrade messed up your listen directives. Fix: add listen = *, [::] to dovecot.conf

When you are designing a project with complex storage requirements and some demands on reliability and performance, a few options come to mind.

Even though at OpenPanel we have strong feelings about NIH, we didn’t think writing our own database store was the way to go. Many smart people have already written many different database backends, which means that in both quantity and quality the database area is well-covered. So, writing our own was right out.

The second option that comes to mind is to use the most basic kind of database available: a key/value store like Berkeley DB or GDBM. Combining a few key/value stores together yields a lot of flexibility, but there’s a lot of glue you need to write, then. In programming language terms, the key/value API is not powerful.

The only way out then seems to be SQL. The common choice seems to be MySQL, and with good reason - it’s robust, fast, flexible (supporting most of the SQL standard), comes pre-packaged for any distribution, and just about any sysadmin or PHP-developer you run into knows his way around it more or less. A close second would be Postgres, less popular with the common PHP developer crowd, more popular with seasoned developers in other languages, and understandably so.

However, we did not go for the path well-traveled. After evaluating our requirements, we realized we barely needed the power of a client/server model database with high concurrency support. Also, we figured choosing a slightly less popular implementation would deter people from messing with the database by hand.

We chose SQLite. It’s extremely lightweight, reliable, robust and surprisingly fast. Choosing SQLite means our database is just a file (directory, to be honest) in /var/opencore, where users and admins can’t run into it by accident while mucking about in phpMyAdmin - but when they really need to mess with the database, they can, with their familiar SQL idioms.

SQLite has most the features a developer would expect from a database; transactions, subqueries, decent indexing support, triggers, and room for extension with user defined functions. The only thing sorely missing is foreign key support, but that’s easily implemented as a bunch of triggers.

Even though we now had this powerful database engine, we put in a lot of effort mapping our idea of an object revision model onto it (more about that in a later post) - but SQLite made it a lot less painful.

SQLite’s only real limitation seems its lack of concurrency, which is made worse by the locking model used that seems to invite polling for access instead of forming some kind of queue. For this reason, OpenPanel keeps just one handle to the database file and manages exclusive and shared access to it via the normal pthread mechanisms.

Incidentally, the current (unpublished) version of our automated web application installer tries to shoehorn its data into a key/value store and it’s hurting - even from Python! We’ll probably rewrite those bits to use SQLite as well (but, of course, separate from the OpenPanel database).

Summarizing, if you are looking for a data store for your software project, consider SQLite. It’s as lightweight as most key/value store libraries but throws in a hell of a lot more featurewise.

Bookmark It

This article on the SQLite wiki details how to efficiently maintain a small window on a big dataset without sacrificing performance, in environments that cannot maintain actual cursors, or where doing so is prohibitively expensive. Rule 2 in the article applies to other databases too; the method detailed in the article applies to any database in which keeping cursors is either impossible or expensive.

The article seems obvious; but I had not actually thought of that approach myself, so far.

find -L . -type l

Replace . with the directory you want to investigate, of course. Works with both GNU and (Free)BSD find. Rationale: -L makes find act on the (transitive closuse of the) target of a symlink - unless the target doesn’t exist, in which case find acts on the symlink itself. So, if the item to act on turns out to be a symlink, it has to be dangling.

Initially I had a more elaborate version based on the exact connect() code in the higher class, but this simpler version works just fine. Incidentally, what the xen xm commandline tool uses works identically ;)

class UHTTPConnection(httplib.HTTPConnection):
    """Subclass of Python library HTTPConnection that
       uses a unix-domain socket.
    """
 
    def __init__(self, path):
        httplib.HTTPConnection.__init__(self, 'localhost')
        self.path = path
 
    def connect(self):
        sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
        sock.connect(self.path)
        self.sock = sock

This small hack plus the fine PHP serialization classes from Scott Hurring are the basis of OpenPanel.coreclient, which allows for very easy provisioning and querying of OpenPanel/opencore data. We use it mostly for unit testing.