Fun with Roguelike Generators

2013-01-23-171028_1440x876_scrot

I may or may not be tooling around with a roguelike. Not because I think the genre is dead (it most certainly isn’t) but because some programming tasks are made fun just by their subject matter. I haven’t quite gotten to the point where I’m using “mana” and “damage” and etc. as variable names, but that’s not the first fun part. The first fun part is generating a rogue dungeon level.

Now, anybody that’s ever even thought about developing a roguelike should know about ASCII Dreams written by the developer of Unangband, Andrew Doull. I especially found his series on Unangband Dungeon Generation to provide a lot of insight into generating dungeons that are interesting a long with a lot of interesting history and philosophizing.

In the end, though, I wanted to try my own, naive, hand at the dungeon generation problem. I did take a few major things away from Andrew’s discussion however. Mostly that there are a lot of nice rooms that can be generating procedurally with simple tricks and, failing that, having a system for “vaults” (which are various, hand-designed rooms with interesting features) can be interspersed for extra flavor. Also, various features added to rooms, like water, lava, ice, minerals, rubble, etc. compel players to explore.

Most of this I’ll get to later, if ever, with my toy generator. The first problem is generating the topology of the dungeon itself.

My Approach

Now, one thing that I’ve found annoying about classic generation algorithms is that they tend to have a lot of really long tunnels. This is because rooms are generated at random across the static map and then, if they haven’t merged together, are connected by tunnels. This has never felt right to me. I understand that – in universe – a dungeon might not have the most sensible design, but having long winding tunnels are boring. Doing connectivity checks on the rooms is boring as well. While we’re at it, I don’t want to have a predefined playing field (array) to work with. I’ll put a limit on the area of the dungeon level, but if it’s a whole bunch of tiny rooms in a very long line, so be it. Unfortunately, I also want the level to be consistent (i.e. no physics violating overlapping inconsistent geometry) so it seems inevitable that the level will eventually be represented on a global grid, but at least that grid will be bounded and reasonably shaped to the level. If necessary, after the level is fully generated the excess grid could be eliminated just by noting where one room enters into another.

So, what I wanted to do was generate a dungeon level that is both tunnel free (for the most part), consistent and connected a priori. Interesting stuff like themed-features, rivers, lava flows, etc. could then be painted over the level geometry in broad strokes.

Problems vs. Classical

There are some troubles with this approach. The first of which is that it makes multiple level consistency really hard with multiple staircases. For a classical generator, you can randomly place down staircases on one level, and then replicate that pattern with up staircases on the next level and ensure that you have rooms to encompass them. This works when you’re going to manually connect all the rooms in the end, but it doesn’t work so well when you’re building a pre-connected level. As such, either there has to be only a single up and down staircase per level (not a bad idea, really) or you have to throw that level of consistency out the window and just match arbitrary up and down staircases. This means that you could have two down staircases right next to each other that would teleport you to different ends of the next level, but in a gametype that traditionally promotes save scumming (i.e. if you go down the same staircase twice, the level is different each time you descend) I think that’s acceptable.

Another problem is that, without tunnels, the dungeons are more likely to be dense. That’s a good thing in the fact that it gives a lot more interesting rooms close by and the player spends a lot of time in an environment. It’s also a dangerous thing because it means there’s a lot fewer twisty places to get out of sight of pursuing monsters. I think that’s acceptable as well, although it’ll be something to account for if I ever get around to generating monsters.

Implementation

I decided to bang out a proof of concept in an evening. Breaking down the logic, the easiest way to generate in this fashion is to generate one room, which will be the root. Then, generate another room. Connect these two rooms with a doorway, then that whole complex becomes the root “room”. Rinse and repeat until the dungeon is of a certain size.

In order to encompass this, I came up with a class for Space. A Space is any arbitrary portion of the dungeon level. It includes a 2-dimensional array geometry that describes what’s in that space. One Space’s geometry can be added to another Space’s geometry with a set overlapping point. A Room is just a space with a name and whose geometry is likely a single room, but is arbitrary. Then, special types of rooms, like one mentioned in Unangband as the core type, two overlapping rectangles (which results in single rooms, crosses, T-shapes, L-shapes, etc.) are just Rooms with special geometry generation.

Expand Code

#!/usr/bin/python

import random
import sys

def chance(c):
    if random.randint(1, 100) <= c:
        return True
    return False

def rand_element(l):
    return l[random.randint(0, len(l) - 1)]

def rand_block(max_x, max_y):
    return (random.randint(0, max_x), random.randint(0, max_y))

class Space(object):
    def __init__(self, geom = []):
        self.global_x = 0
        self.global_y = 0
        self.geometry = geom

    def point(self, x, y):
        return (self.global_x + x, self.global_y + y)

    def adjust_x(self, x):
        self.global_x = x

    def adjust_y(self, y):
        self.global_y = y

    def area(self):
        area = 0
        for row in self.geometry:
            for x in row:
                if x:
                    area += 1
        return area

    # Create an initialized array for a space with dimensions.  we can't just do
    # [[0] * x] * y because then the sublists are all just instances of the same
    # list.

    def space(self, x, y, val = 0):
        geom = []
        for y in range(0, y):
            geom.append([val] * x)
        return Space(geom)

    # Add space takes two arbitrary spaces and merges together, using points
    # s1p and s2p as the same points in the resulting form.

    # If kwargs["chicken"] is True, it will return False (i.e. fail), if any of s1's
    # points are overwritten by s2 except for the intersection point.

    def add_space(self, sq, s1p, s2p, **kwargs):

        chicken = False
        if "chicken" in kwargs and kwargs["chicken"]:
            chicken = True

        s1 = self.geometry
        s2 = sq.geometry

        s1_dim_y = len(s1)
        s1_dim_x = len(s1[0])

        s1_o_x, s1_o_y = s1p

        s2_dim_y = len(s2)
        s2_dim_x = len(s2[0])

        s2_o_x, s2_o_y = s2p

        g_max_x = s1_dim_x

        # Account for s2's width going farther to left
        if (s1_o_x < s2_o_x):
            s1_start_x = (s2_o_x - s1_o_x)
            s2_start_x = 0
            g_max_x += s1_start_x
        else:
            s1_start_x = 0
            s2_start_x = (s1_o_x - s2_o_x)

        # Account for s2's width going farther to right
        if ((s1_dim_x - s1_o_x) < (s2_dim_x - s2_o_x)):
            g_max_x += (s2_dim_x - s2_o_x) - (s1_dim_x - s1_o_x)

        g_max_y = s1_dim_y

        # Account for s2's height going higher
        if (s1_o_y < s2_o_y):
            s1_start_y = (s2_o_y - s1_o_y)
            g_max_y += s1_start_y
            s2_start_y = 0
        else:
            s1_start_y = 0
            s2_start_y = (s1_o_y - s2_o_y)

        # Account for s2's depth going lower
        if ((s1_dim_y - s1_o_y) < (s2_dim_y - s2_o_y)):
            g_max_y += (s2_dim_y - s2_o_y) - (s1_dim_y - s1_o_y)

        geom = self.space(g_max_x, g_max_y)

        # Draw both into geometry:

        for y in range(0, s1_dim_y):
            for x in range(0, s1_dim_x):
                geom.geometry[y + s1_start_y][x + s1_start_x] = s1[y][x]

        for y in range(0, s2_dim_y):
            for x in range(0, s2_dim_x):
                if s2[y][x]:
                    if chicken and (x, y) != s2p and\
                            geom.geometry[y + s2_start_y][x + s2_start_x]:
                        return False
                    geom.geometry[y + s2_start_y][x + s2_start_x] = s2[y][x]

        self.adjust_x(s1_start_x)
        sq.adjust_x(s2_start_x)

        self.adjust_y(s1_start_y)
        sq.adjust_y(s2_start_y)

        self.geometry = geom.geometry
        return True

    # Sort out surrounding used and unused coordinates.

    def _cardinal_sort(self, x, y):
        unused = []
        used = []
        pairs = [(-1, 0, 0),(0, -1, 3),(0, 1, 1),(1, 0, 2)]

        for off_y, off_x, direction in pairs:
            b = (x + off_x, y + off_y, direction)
            # Count out of bounds as unused.
            if (y + off_y) < 0 or\
               (y + off_y) >= len(self.geometry) or\
               (x + off_x) < 0 or\
               (x + off_x) >= len(self.geometry[y + off_y]) or\
               self.geometry[y + off_y][x + off_x] == 0:
                unused.append(b)
            else:
                used.append(b)

        return (used, unused)

    # Return a list of coordinates of used squares that have at least one
    # unused square (four-way) adjacent that, if used, would have no other used
    # neighbors but the first one. So, in summary

    # 1|
    # 2| <- all perimeter blocks
    # 3|
    #
    # 1|_  <-- neither are perimeter blocks
    #   2

    # The intent here is to find blocks where it's appropriate to attach a
    # doorway. For now, this only list blocks that are pressing against the
    # outside boundary of the geometry. There are some neat cases that could be
    # generated with finer granularity, but I'm not sure if that's better done
    # with specific generators.

    def connectable_coords(self):
        r = []
        for i, y in enumerate(self.geometry):
            for j, x in enumerate(y):

                # Skip unused blocks
                if not x:
                    continue

                used, unused = self._cardinal_sort(j, i)
                for s_x, s_y, s_d in unused:
                    sub_used, sub_unused = self._cardinal_sort(s_x, s_y)
                    if len(sub_used) == 1:
                        r.append((j, i, s_d))
        return r

    def print_geom(self):
        for row in self.geometry:
            p = ""
            for x in row:
                if x:
                    p += ("%s" % (chr(x),))
                else:
                    p += " "
            print(p)

    def gen(self):
        pass


class Room(Space):
    def __init__(self):
        Space.__init__(self)
        self.name = ""

        self.name_prefixes = [ "Beautiful", "Evil", "Corrupted", "Pristine" ]
        self.name_suffixes = [ "of Doom", "of Dancing", "of Flowers", "of Blood" ]
        self.name_bases = [ "Room" ]

        self.name_prefix_chance = 25
        self.name_suffix_chance = 25
        
    def give_name(self):
        n = ""
        if chance(self.name_prefix_chance):
            n += rand_element(self.name_prefixes) + " "
        n += rand_element(self.name_bases)
        if chance(self.name_suffix_chance):
            n += " " + rand_element(self.name_suffixes)
        self.name = n

    def gen(self):
        self.give_name()

class SimpleRoom(Room):
    def gen(self, x, y, val):
        Room.gen(self)
        self.geometry = self.space(x, y, val).geometry

class SimpleAdditiveRoom(Room):
    def gen(self, min_x, max_x, min_y, max_y, val):
        Room.gen(self)

        # Get section 1 dimensions
        r1_dim_x = random.randint(min_x, max_x)
        r1_dim_y = random.randint(min_y, max_y)

        # Section 2 dimensions
        r2_dim_x = random.randint(min_x, max_x)
        r2_dim_y = random.randint(min_y, max_y)

        # Random overlapping points
        r1p = rand_block(r1_dim_x, r1_dim_y)
        r2p = rand_block(r2_dim_x, r2_dim_y)

        r1 = self.space(r1_dim_x, r1_dim_y, val)
        r2 = self.space(r2_dim_x, r2_dim_y, val)

        self.geometry = r1.geometry
        self.add_space(r2, r1p, r2p)

class Hallway(Space):
    def gen(self, x, y, val):
        Space.gen(self)
        self.geometry = self.space(x, y, val).geometry

if __name__ == "__main__":
    root = SimpleAdditiveRoom()
    root.gen(3, 10, 3, 10, ord('A'))

    val = ord('B')

    while root.area() < 2000:
        sr2 = SimpleAdditiveRoom()
        sr2.gen(3, 10, 3, 10, val)

        c1 = rand_element(root.connectable_coords())

        m = [ 2, 3, 0, 1 ]  
        c2 = rand_element([ x for x in sr2.connectable_coords() if x[2] == m[c1[2]]])

        hall = Hallway()

        if c1[2] == 0:
            hall.gen(1, 3, ord('+'))
            hall.add_space(root, hall.point(0,2), c1[:2])
            r = hall.add_space(sr2, hall.point(0,0), c2[:2], chicken = True)
        elif c1[2] == 2:
            hall.gen(1, 3, ord('+'))
            hall.add_space(root, hall.point(0,0), c1[:2])
            r = hall.add_space(sr2, hall.point(0,2), c2[:2], chicken = True)
        elif c1[2] == 1:
            hall.gen(3, 1, ord('+'))
            hall.add_space(root, hall.point(0,0), c1[:2])
            r = hall.add_space(sr2, hall.point(2,0), c2[:2], chicken = True)
        elif c1[2] == 3:
            hall.gen(3, 1, ord('+'))
            hall.add_space(root, hall.point(2,0), c1[:2])
            r = hall.add_space(sr2, hall.point(0,0), c2[:2], chicken = True)

        if r:
            root = hall
            val += 1

    root.print_geom()

I got a little lazy with the execution of __main__. There’s a cleaner way to deal with matching up the direction end-points (hell, just take a random one and rotate the entire room to match, really) but for an evening’s playing around I think the results are actually pretty nice.

Expand Example


        ]]]]]]                                                           
        ]]]]]]                                                           
        ]]]]]]]]]]]                                                      
        ]]]]]]]]]]]                                                      
        ]]]]]]]]]]]                                                      
        ]]]]]]]]]]]                                                      
        ]]]]]]]]]]]                                                      
        ]]]]]]]]]]]          UUUUUUUU                                    
        ]]]]]]               UUUUUUUU                                    
          +                UUUUUUUUUU                                    
         ZZZZ              UUUUUUUUUU                                    
         ZZZZ              UUUUUUUUUU                                    
         ZZZZ              UUUUUUUUUU                                    
         ZZZZ                UUUUUUUU MMMMMMMMM                          
         ZZZZZZZZ            UUUUUUUU+MMMMMMMMM                          
         ZZZZZZZZ            UUUUUUUU MMMMMMMMM                          
         ZZZZZZZZ            UUUUUUUU MMMMMMMMM                          
         ZZZZZZZZ                        MMMMMM                          
         ZZZZZZZZ                        MMMMMM                          
         ZZZZZZZZ                JJJJJJJ MMMMMM                          
         ZZZZZZZZ                JJJJJJJ MMMMMM                          
            +                    JJJJJJJ  +                              
          WWWWWWW                JJJJJJJJJJJJJJ                          
          WWWWWWWWWW NNNNNN      JJJJJJJJJJJJJJ                          
          WWWWWWWWWW NNNNNN      JJJJJJJJJJJJJJ                          
          WWWWWWWWWW+NNNNNN      JJJJJJJJJJJJJJ                          
          WWWWWWWWWW NNNNNN       +                                      
          WWWWWWWWWW NNNNNNBBBBBBBB                                      
          WWWWWWWWWW NNNNNNBBBBBBBBBB                                    
          WWWWWWWWWW NNNNNNBBBBBBBBBB                                    
             WWWWWWW  NNNN BBBBBBBBBB                                    
             WWWWWWW  NNNN BBBBBBBBBB                                    
            OOOOOOOOOONNNN       BBBB                                    
            OOOOOOOOOO   +       BBBB                                    
            OOOOOOOOOO FFFFFFF   BBBB                                    
            OOOOOOOOOO FFFFFFF   BBBB                                    
            OOOOOOOOOO FFFFFFF    +                                      
            OOOOOOOOOO FFFFFFF AAAAAAAAAAA                               
            OOOOOOOOOO FFFFFFF+AAAAAAAAAAA                               
            OOOOOOOOOO FFFFFFF AAAAAAAAAAA                               
            OOOOOOOOOO FFFFFFF AAAAA                                     
            OOOOOOOOOO+FFFFFFF AAAAA          PPPPPPPPP                  
             OOOOO     FFFFFFF AAAAA          PPPPPPPPP                  
             OOOOO LLL FFFFFFF AAAAA          PPPPPPPPP TTTT             
                   LLL +         +            PPPPPPPPP TTTT             
                  LLLLLLL        CCCCC        PPPPPPPPP+TTTT             
                  LLLLLLL        CCCCC        PPPP   +  TTTTT            
                  LLLLLLL        CCCCC      HHHHHHHHHH  TTTTT            
     [[[[[[[[[                   CCCCCCCCCC HHHHHHHHHH  TTTTT  XXXXXXXXXX
     [[[[[[[[[                   CCCCCCCCCC HHHHHHHHHHHHTTTTT XXXXXXXXXXX
     [[[[[[[[[ GGGGGGGGGG        CCCCCCCCCC HHHHHHHHHHHHTTTTT+XXXXXXXXXXX
     [[[[[[[[[ GGGGGGGGGG  EEEEE+CCCCCCCCCC HHHHHHHHHHHH      XXXXXXXXXX 
     [[[[[[[[[ GGGGGGGGGG  EEEEE CCCCCCCCCC+HHHHHHHHHH        XXXXXXXXXX 
     [[[[[[[[[ GGGGGGGGGG EEEEEE CCCCCCCCCC                   XXXXXXXXXX 
     [[[[[[[[[+GGGGGGGGGG EEEEEE CCCCCCCCCC                   XXXXXXXXXX 
   [[[[[[[[[[[ GGGGGGGGGG EEEEEE     +                        XXXXXXXXXX 
   [[[[[[[[[[[ GGGGGGGGGG EEE       DDD                                  
   [[[[[[[[[[[ GGGGGGGGGG+EEE       DDD                                  
   [[[[[[[[    GGGGGGGGGG EEE   III DDD                                  
   [[[[[[[[    GGGGGGGGGG EEE  IIII DDD                                  
   [[[[[[[[         +     EEE  IIII DDD                                  
   [[[[[[[[         KKK   EEE  IIII DDD                                  
                    KKK        IIII DDD                                  
                    KKK        IIII+DDDQQQQ                              
                    KKK        IIII QQQQQQQQQQ                           
                 KKKKKKK       IIII+QQQQQQQQQQ                           
                 KKKKKKK       IIII QQQQQQQQQQ                           
                 KKKKKKK       IIII QQQQQQQQQQ                           
                 KKKKKKK                    +                            
                 KKKKKKK                    SSS                          
\\\\\\\\\        KKKKKKK                SSSSSSSS                         
\\\\\\\\\        KKKKKKK                SSSSSSSS                         
\\\\\\\\\        KKKKKKK                SSSSSSSS                         
\\\\\\\\\        KKKKKKK                SSSSSSSS                         
\\\\\\\\\\\\\\   KKKKKKK                SSSSSSSS                         
\\\\\\\\\\\\\\     +                    SSSSSSSS                         
\\\\\\\\\\\\\\RRRRRRRRR                 SSSSSSSS                         
      \\\\\\\\RRRRRRRRR                     SSS                          
      \\\\\\\\RRRRRRRRR                                                  
      \\\\\\\\RRRRRRRRR                                                  
         +    RRRRRRRRR                                                  
        VVVVV RRRRRRRRR                                                  
      VVVVVVV RRRRRRRRR                                                  
      VVVVVVV+RRRRRRRRR                                                  
      VVVVVVV RRRRRRRRR                                                  
            + RRRRRRRRR                                                  
          YYYYY                                                          
          YYYYY                                                          
          YYYYY                                                          
          YYYYY                                                          
     YYYYYYYYYY                                                          
     YYYYYYYYYY                                                          
     YYYYYYYYYY                                                          
     YYYYYYYYYY                                               

Not many long tunnels, and a large number of rooms (which are indicated by different letters. Doorways are +s. There are still, of course, some places the geometry doesn't make sense. Usually when two joined rooms have a doorway and also have adjacent open blocks (like P and H in the above). However, all in all, not bad for an evening's screwing around.

Using Decorators for Flexible Prompts

As a general rule, anything that you’re going to need to do many times should be abstracted to be made easy. When you’re coding something with a prompt, adding commands definitely falls into that category. With Python introspection and decorators make a new text prompt can be as simple as writing a function and defining a simple format for it. Like this:

    @command_format(("terms","listof_uint"))
    def sumstar(self, **kwargs):
        print sum(kwargs["terms"])

Which is safe, clear, and working. This method has a number of advantages:

  1. It easily allows for features like automatically prompting for missing arguments and argument defaults.
  2. The decorator for each command function doubles as documentation since it both simply lists the names of the keyword arguments as well as their types.
  3. Cleanly separates the function of the command from the validation of its arguments.
  4. Ideal for allowing users to write pluggable commands because actual command functions use native objects instead of strings, and they automatically receive the benefits of prompting, listing, defaults, etc. Plus, there’s only a single line of overhead when the validators are pre-defined.
  5. Allows validators to easily use each other so multiple minor variations of the same type of validation can be expressed simply.

The command_format Meta-Decorator

def command_format(*types):
    def cf(fn):
        def cfdec(self, **kwargs):

            rem = kwargs["args"]
            realkwargs = {}

            for kw, validator in types:
                validator = getattr(self, validator)
                valid, result, rem = validator(rem.lstrip())
                if not valid:
                    log.debug("Couldn't properly parse %s" % kwargs["args"])
                    return

                realkwargs[kw] = result

            return fn(self, **realkwargs)
        return cfdec
    return cf

It’s not every day that you see a triple nested def, but it’s pretty standard meta-decorator, where *types is a list of string tuples like [("keyword","validator")]. Where each keyword is the name of the keyword argument the parsed object will be passed as to the final command function and each validator is a string-reference to a method of the class this command resides in. If you’re operating outside of a class, this validator could easily be a function reference as well.

This meta-decorator is applied to every function that embodies a command. We’ll get to that later. For now, let’s look at what these validators look like.

A Simple Validator

Validators are passed one argument: the entire unparsed portion of the string you’re parsing, which can be ""/None. It returns a three tuple of objects. A boolean value whether the validator passed and, if it did, the resulting object and the remaining, unparsed portion of the string. Let’s take a look at a simple validator that will take a positive integer.

class MyCommandHandler(...)
    ...
    def uint(self, args):
        terms = args.split()
        if not terms:
            return (False, None, None)

        try:
            ret = int(terms[0])
        except:
            log.error("Couldn't parse %s as integer!" % terms[0])
            return (False, None, None)

        if ret < 0:
            log.error("Argument must be > 0")
            return (False, None, None)

        return (True, ret, " ".join(terms[1:]))

Extremely simple, it grabs the first whitespace delimited argument from the string, attempts to parse it as an integer and makes sure it’s >0 before returning that the arg passes validation.

Let’s put our command_format and the int validator to use by writing a simple sum that will take a string argument, attempt to parse out two integers and print the sum.

sum Example

class MyCommandHandler(...)
    ...
    @command_format(("term1","uint"),("term2","uint"))
    def sum(self, **kwargs):
        print kwargs["term1"] + kwargs["term2"]

No need to do any extra validation on either argument, we know both that they exist and they’re integers, so we’re safe.

At this point, with a little wrapping code we can do the following:

jack@arpeggi:~ $ ./basicsum.py
sum 12 16
28

sum 1 b
Couldn't parse b as integer!
Couldn't properly parse 1 b

But that’s not very exciting. Let’s look a little harder and see how we can leverage these.

More Useful Validators

Automatic Argument Prompting

The first thing that pops to mind as being a useful addition to a command prompt is being a little more tolerant to missing arguments. Let’s make our uint validator prompt instead of bailing on an empty string.

class MyCommandHandler(...)
    ...
    def uint(self, args):
        if not args:
            args = raw_input("uint: ")

        terms = args.split()
        if not terms:
            return (False, None, None)

        try:
            ret = int(terms[0])
        except:
            log.error("Couldn't parse %s as integer!" % terms[0])
            return (False, None, None)

        if ret < 0:
            log.error("Argument must be > 0")
            return (False, None, None)

        return (True, ret, " ".join(terms[1:]))

NOTE: One of the attributes of the command_handler resulting decorator is that the arguments passed to each validator are stripped of leading whitespace so we know that if not args will work on any empty string passed to the validator.

Now, again with a shade of wrapper code, we have a bit more functionality.

jack@arpeggi:~/blog $ ./promptsum.py
sum
uint: 13
uint: 14
27

sum 13
uint: 20
33

sum 14 15
29

Function on Lists of Arbitrary Types

So, at this point, we’ve basically got a sum2 function and, even though it’s nice (=P), it’s pretty inflexible. Let’s write another validator that generates a list of uints and work on that, with another appropriate function. Better yet, let’s write a validator that will make a list of *any* consistent type.

class MyCommandHandler(...)
    ...
    def listof_(self, prompt, val, args):
        l = []
        if not args:
            args = raw_input(prompt)

        while args:
            v, term, args = val(args.strip())
            if v:
                l.append(term)
            else:
                # If you wanted to be really fault tolerant, you could 
                # remove the first term and try again. We'll be really 
                # strict here though.
                return (False, None, None)
        return (True, l, "")

    def listof_uint(self, args):
        return self.listof_("uints: ", self.uint, args)

There we go. As you can see, the listof_ function will attempt to use any given validator on the input until it’s used up. A smarter implementation might not use up the whole input, but expect lists to be bracketed by symbols, or a certain length. This also includes the arbitrary prompt that we used before. Let’s put it to use implementing sum*.

    @command_format(("terms","listof_uint"))
    def sumstar(self, **kwargs):
        print sum(kwargs["terms"])

Done. Now we have a much more flexible sum* command. As evidenced using starsum.py.

jack@arpeggi:~/blog $ ./starsum.py
sum*
uints: 4 7 8
19

sum* 3 8 10 11
32

sum* 4 5 6 a
Couldn't parse a as integer!
Couldn't properly parse  4 5 6 a

Closing Notes + Source

The above examples use positive integers (and yes, I’m aware that *sum* works on negative integers a well =P) because they’re a convenient type that everybody knows. One of the advantages of the system is that it doesn’t matter what sort of return object the validator gives, the command functions can just assume everything is all right because it never gets called if the args don’t parse correctly. I wrote a system very much like this for the new version of my side project Canto (an RSS reader) that allows users to give a list of story indexes, which are then converted from simple integers to actual story objects by the validators (listof_items) and passed to the corresponding function.

Also, this code is easily extensible to use any sort of input format (obviously raw_input isn’t exactly the most useful way to get a string from the user if you’re graphical or running ncurses, like I always am). Canto uses this via a Textbox class from a curses window.

The examples used in this write-up are available as executable .py files (used for the output sections).

Basic sum2 validating prompt: basicsum.py
Added missing argument prompting: promptsum.py
Added sum*: starsum.py

These were tested on Linux with Python 2.6, but should work on practically any platform with any relatively modern Python. It’s also under public domain, if you actually care about the licensing of blog snippets =).

:wq

Calm Down!

I’ve been ditching a lot of code lately. The first thing was my side-project Canto’s old codebase*. The second was this blog, which is now obviously a WordPress blog instead of a quirky but fun-loving git based CMS I wrote before. All together, I’ve shed thousands of lines of code and it feels great. But as I toss this code to the wayside, I can’t help but wonder “How did things get this way? Where did I go wrong?”

One thing that I’ve learned over the course of my programming travails is that design is more important than implementation. As I maintained and extended and improved Canto over the two or three years since I first started it, I learned a lot about showing foresight. A lot about how getting a feature into a codebase right is a better long-term strategy than getting a feature into a codebase quick.

Programming is a tough task. It requires a lot of concentration and thought and, as if that wasn’t bad enough, it’s generally done under a lot of pressure. Either from deadlines from a boss, or just expectations of your users (or your perception of either). A lot of choices I’ve made in the last couple of years have been wrong because they’ve been twisted by having the perception an audience. If someone sends you an email and they’re in distress because of a bug you feel as if they’re waiting for your response, constantly refreshing their mail client or feeling frustration with your software the whole time the flaw goes unpatched. You feel as though there are a thousand other users that have run into the same bug and just never reported it only to move on to greener pastures. You see the 10 day (or more!) turnaround time on your packages in repositories. The fact that Ubuntu is now actively serving up a broken version to its users and there’s nothing you can do but fix it for the next iteration (Salacious Salamander or whatever) six months later. Suddenly it’s as if everyone using your software is secretly unhappy with it.

It’s hard not to face these troubles and feel an urgency beyond reality. Feel the need to find it, fix it, and push it to the repo and make a release before anybody else reports the same problem. But, unless you’re dealing with a critical bug and your software is immensely popular it’s probably not as urgent as you imagine. Most likely, the user reported the bug and moved on with their lives. They either uninstalled your software, will use something else in the meantime, or will patiently wait for a fix. If you’re lucky, maybe they included a patch. The point is, what the user does is what the user does, it’s up to you to take your time and really analyze a problem before you fix it. There are plenty of potential users out there and for any user you lose over a single bug there’s another who will try your software after it’s patched. That’s why it’s important to find the right solution instead of the quick solution because the right solution will keep you from getting into this situation again (or make it easier to fix correctly when you do) and the quick solution increases the probability that you’re going to feel that sort of pressure, and lose another user over essentially the same bug.

Coding is all about trying to maximize the time you spend writing features and feeling rewarded, and minimize the time you spend bug fixing and feeling like your users are searching you out with machetes. Anything you do to sway that balance in your favor should be done without hesitation. But what can you do? I might not be the best person to ask, but these are some improvements I’ve been trying over the last few weeks and so far so good. I call my new philosophy “Calm The Fuck Down

  1. Start over. Either on the project, subsystem, or feature level. Even with recognition of the fact that the current implementation has become unmanageable, it’s always hard to look at functioning code and say “This is crap! Tear it out!” but sometimes that’s the only thing to say. Coders tend to look at their own code and not see its utility, or its elegance, but the weight of all of the obstacles they’ve overcome, or the bugs they’ve ironed out. To be passionate about good code, you have to be dispassionate about your own code. As you gain experience you see the error of your ways and have to re-evaluate practically every decision you’ve made in the light of new information and the easiest way to do this is to start over. It’s not always the right thing to do, but when you suddenly realize that your code has turned into a chain of hacks then you have no other path forward. Some developers believe they can just do a quick stub-out of code, or a quick and dirty narrow implementation now and return later. Maybe you can. I can’t. My thought is that if you can’t take the time to do it right, it’s time to hold off and give it a bit more thought.
  2. Put everything on paper first. This is something that I find helps alot, particularly if you’ve really got a good handle on the language you’re using, because it helps you layout your data structures beforehand and anticipate your needs before you even open your editor. Sometimes it doesn’t work out (I actually just rebased out of existence a commit that I had planned out on paper but had fallen victim to a number of unexpected twists), but most of the time an hour of planning can save you a full day of coding. If you get into the habit of putting everything on paper first, your first step in responding to a feature request or a bug report is grabbing a piece of paper and a pen, instead of impulsively taking a hatchet to your codebase.
  3. Don’t let the phantom specter of imagined users fleeing over bugs or missing features rush your design decisions. I’ve fought this battle for so long that it’s become meaningless. There’s always another feature, always another bug. If you rush to fix a bug, you’re probably simultaneously laying the groundwork for the next one. Getting a fix or a feature out means nothing if it bites you in the ass down the line. So unless people’s boxes are being enslaved into a malicious porn spam botnet because of a security flaw, it can wait.

As for all of that code I’ve scrapped over the last few months, I chalk it up to one thing: practice. If the wise men of yore are to be trusted, that apparently makes perfect…

Reddit tl;dr : Regardless of perceived pressure, calm the fuck down and take your goddam time.

*Tangentially, Canto 0.8.x is much stronger and in alpha for harder-core IRC lurkers.