James Mills's Blog

Migrating to blohg

2013-03-26T13:30:00+10:00

I think I'll be migrating from Pelican to blohg. The two Blogging engines both share very similar functionality but the key differences that are attracting me to blohg are:

Mercurial backed.
WSGI Application

With these two key features I'll be able to better integrate My Blog into My Website which is itself powered by sahriswiki and circuits.web and host my blohg powered blog using circuits.web's Gateway Component.

Kudos to the blohg development team! I'll be giving this a try on the upcoming long weekend.

Happy Easter Pythonistas!

--JamesMills / prologic

10 years of PyPy

2013-03-01T10:11:00+10:00

10 years of PyPy as first seen on PyPy Status Blog.

Congratulations guys! Well done on a Great Project, a Great Tool and most of all a Great Python Implementation!

--JamesMills / prologic

circuits-2.1.0 is out!

2013-02-27T21:45:00+10:00

I'm pleased to announce the release of circuits 2.1.0 []

This is a minor release which includes some new features and bug fixes. Some of the highlights of this release include:

Python 3 support.
Windows support.
PyPy support.
IPv6 support.
Better WSGI support.
Fully documented examples.
Component Interface querying.

For more information see the PyPi page:

http://pypi.python.org/pypi/circuits/

Writing a simple IRC Logger Bot and Daemon

2013-02-26T20:31:00+10:00

Contents:

Introduction
What is circuits?
Phase I - Imports
Phase II - Command Line Options
Phase III - Utility Functions
Phase IV - Events
Phase V - Components
Phase VI - Main

Introduction

Welcome to the first of what will hopefully be many articles about solutions to problems using the circuits application framework.

What is circuits?

For those of you unfamiliar with circuits; circuits is a lightweight, event-driven application framework for the Python programming language. At the time of writing circuits includes the following list of features:

Powerful Message Passing
A flexible Component Oriented Architecture
Asynchronous I/O Components for Networking and File I/O
Multi-Threaded and Multi-Process Worker Pools
Cooperative Co-routines
A complete lightweight Web Framework
Plus more!

You can install circuits by simply issuing the following command:

$ pip install circuits

Yes! circuits supports Python 3 and PyPy.

With circuits you can write applications ranging from simple network daemons, to web applications, real-time strategy games and even automated systems. Check out the various Examples.

Now let's demonstrate how we can build a simple IRC Logger Bot that is also capable of Daemonizing into the background.

Phase I - Imports

First up. The imports. We're going to need:

#!/usr/bin/env python

from socket import gethostname
from optparse import OptionParser
from collections import defaultdict
from re import compile as compile_regex
from datetime import date, datetime, timedelta
from os import environ, getcwd, makedirs, path
from time import asctime, localtime, strftime, time

import circuits
from circuits.app import Daemon
from circuits.io import Close, File, Open, Write
from circuits.net.sockets import TCPClient, Connect
from circuits import Component, Event, Debugger, Timer
from circuits.net.protocols.irc import IRC, USER, NICK, JOIN

Phase II - Command Line Options

Next we're going to need to build the Command Line Options available to our application. We want to:

Be able to enable daemon mode
Choose one or more IRC Channels to join
Choose a desired nickname for the bot
Specify an output path to write logs to
Specify an optional PID file when we Daemonize
Optionally turn on verbose logging in case something goes wrong

So let's add the following bits:

USAGE = "%prog [options] <host> [<port>]"
VERSION = "%prog v0.0.1"


def parse_options():
    parser = OptionParser(usage=USAGE, version=VERSION)

    parser.add_option(
        "-d", "--daemon",
        action="store_true", default=False, dest="daemon",
        help="Enable daemon mode"
    )

    parser.add_option(
        "-c", "--channel",
        action="append", default=None, dest="channels",
        help="Channel to join (multiple allowed)"
    )

    parser.add_option(
        "-n", "--nick",
        action="store", default=environ["USER"], dest="nick",
        help="Nickname to use"
    )

    parser.add_option(
        "-o", "--output",
        action="store", default=getcwd(), dest="output",
        help="Path to store log files"
    )

    parser.add_option(
        "-p", "--pidfile",
        action="store", default=PIDFILE, dest="pidfile",
        help="Path to store PID file"
    )

    parser.add_option(
        "-v", "--verbose",
        action="store_true", default=False, dest="verbose",
        help="Enable verbose debugging mode"
    )

    opts, args = parser.parse_args()

    if not opts.channels:
        print("ERROR: Must specify at least one channel")
        parser.print_help()
        raise SystemExit(2)

    if len(args) < 1:
        print("ERROR: Must specify a host to connect to")
        parser.print_help()
        raise SystemExit(1)

    return opts, args

Phase III - Utility Functions

Next we need a few utility functions. We need a function to generate a consistent timestamp that we can use in our log files as well as functions to generate proper log filenames and parse them back into their constituent parts. The functions we need are as follows:

LOGFILE_REGEX = compile_regex("^(.*)\.(.*)\.log$")
PIDFILE = path.join(path.dirname(__file__), "{0:s}.pid".format(__name__))


def timestamp():
    return asctime(localtime(time()))


def generate_logfile(channel):
    return path.join(channel, "{0:s}.log".format(
        strftime("%Y-%m-%d", localtime()))
    )


def parse_logfile(filename):
    match = LOGFILE_REGEX.match(filename)
    return match.groups() if match is not None else "", ""

Phase IV - Events

Now come the new bits. In circuits; Components interact with one another by passing events around. Components maintain a loose coupled relationship with other components and only use events to communicate. What we mean by "Loose Coupling" here is that no two components ever know about each other or maintain references to each other. Due to circuits' powerful Component Architecture and Event-Driven Message Passing, your application is more maintainable and components reusable. When you need to modify the behavior of your application, you can simply add more components into the mix and/or intercept events at various points in the system.

With no further ado; Let's begin.

We need to define two events:

A Log event to signal to a "logging" Component we have new text we need to store in a log file.
A Rotate event to signal that we need to rotate the log files for a new day of logging irc channels online.

These are simply defined as follows:

class Log(Event):
    """Log Event"""


class Rotate(Event):
    """Rotate Event"""

Phase V - Components

Now we're at the heart of the application. The Components.

This is going to be a 2-component system:

Logger - Responsible for logging and log file rotation.
Bot - Responsible for connecting to IRC and relaying messages to logs.

Logger

class Logger(File):

    def init(self, *args, **kwargs):  #*
        super(Logger, self).init(*args, **kwargs)  #*

        interval = datetime.fromordinal((
            date.today() + timedelta(1)
        ).toordinal())
        Timer(interval, Rotate(), self.channel).register(self)

    def rotate(self):
        dirname = path.dirname(self.filename)
        filename = path.basename(self.filename)
        channel, _ = parse_logfile(filename)
        logfile = generate_logfile(channel)
        self.fire(Close(), self.channel)
        self.fire(Open(path.join(dirname, logfile), "a"), self.channel)

        interval = datetime.fromordinal((
            date.today() + timedelta(1)
        ).toordinal())
        Timer(interval, Rotate(), self.channel).register(self)

    def log(self, message):
        timestamp = strftime("[%H:%M:%S]", localtime(time()))
        self.fire(
            Write("{0:s} {1:s}\n".format(timestamp, message)), self.channel
        )

Here our Logger Component subclasses the File Component. Why? Well we want to create a new Component that extends the functionality of the File much the same way you'd extend a Base Class in traditional OOP. Unlike OOP however, extending a Base Component inherits it's Event Handlers.

Our Logger Component extends the File Component by defining two new Event Handlers:

log - For Log events.
rotate- For Rotate events.

You'll also note that we use a Timer Component built-in to the circuits library of components. We need to create a timed event once per day on a repeating interval every day so we know when to rotate our log files.

Also note the use of def init(...)? Components in circuits automatically call this method (if it exists) after the Component has been properly initialized. This is a convenience method only in case you used def __init__(...) and forgot to call super(...).

Bot

class Bot(Component):

    channel = "bot"

    def init(self, host, port=6667, opts=None):
        self.host = host
        self.port = port
        self.opts = opts
        self.hostname = gethostname()

        self.nick = opts.nick
        self.ircchannels = opts.channels

        # Mapping of IRC Channel -> Set of Nicks
        self.chanmap = defaultdict(set)

        # Mapping of Nick -> Set of IRC Channels
        self.nickmap = defaultdict(set)

        # Debugger
        Debugger(events=opts.verbose).register(self)

        # Add TCPClient and IRC to the system.
        self += (TCPClient(channel=self.channel) + IRC(channel=self.channel))

        # Logger(s)
        for ircchannel in self.ircchannels:
            if not path.exists(path.join(opts.output, ircchannel)):
                makedirs(path.join(opts.output, ircchannel))
            Logger(path.join(opts.output, generate_logfile(ircchannel)), "a",
                   channel="logger.{0:s}".format(ircchannel)).register(self)

        # Daemon?
        if self.opts.daemon:
            Daemon(opts.pidfile).register(self)

    def ready(self, component):
        """Ready Event

        This event is triggered by the underlying ``TCPClient`` Component
        when it is ready to start making a new connection.
        """

        self.fire(Connect(self.host, self.port))

    def connected(self, host, port):
        """Connected Event

        This event is triggered by the underlying ``TCPClient`` Component
        when a successfully connection has been made.
        """

        nick = self.nick
        hostname = self.hostname
        name = "{0:s} on {1:s} using circuits/{2:s}".format(
            nick, hostname, circuits.__version__
        )

        self.fire(USER(nick, hostname, host, name))
        self.fire(NICK(nick))

    def disconnected(self):
        """Disconnected Event

        This event is triggered by the underlying ``TCPClient`` Component
        when the active connection has been terminated.
        """

        self.fire(Connect(self.host, self.port))

    def numeric(self, source, target, numeric, args, message):
        """Numeric Event

        This event is triggered by the ``IRC`` Protocol Component when we have
        received an IRC Numeric Event from server we are connected to.
        """

        if numeric == 1:
            for ircchannel in self.ircchannels:
                self.fire(JOIN(ircchannel))
        elif numeric == 433:
            self.nick = newnick = "%s_" % self.nick
            self.fire(NICK(newnick))

    def join(self, source, channel):
        """Join Event

        This event is triggered by the ``IRC`` Protocol Component when a
        user has joined a channel.
        """

        self.chanmap[channel].add(source[0])
        self.nickmap[source[0]].add(channel)

        self.fire(
            Log("*** {0:s} has joined {1:s}".format(source[0], channel)),
            "logger.{0:s}".format(channel)
        )

    def part(self, source, channel):
        """Part Event

        This event is triggered by the ``IRC`` Protocol Component when a
        user has left a channel.
        """

        self.chanmap[channel].remove(source[0])
        self.nickmap[source[0]].remove(channel)

        self.fire(
            Log("*** {0:s} has left {1:s}".format(source[0], channel)),
            "logger.{0:s}".format(channel)
        )

    def quit(self, source, message):
        """Quit Event

        This event is triggered by the ``IRC`` Protocol Component when a
        user has quit the network.
        """

        for ircchannel in self.nickmap[source[0]]:
            self.chanmap[ircchannel].remove(source[0])
            self.fire(
                Log("*** {0:s} has quit IRC".format(source[0])),
                "logger.{0:s}".format(ircchannel)
            )

        del self.nickmap[source[0]]

    def message(self, source, target, message):
        """Message Event

        This event is triggered by the ``IRC`` Protocol Component for each
        message we received from the server.
        """

        # Only log messages to the channel we're on
        if target[0] == "#":
            self.fire(
                Log("<{0:s}> {1:s}".format(source[0], message)),
                "logger.{0:s}".format(target)
            )

Whoa! Quite a mouthful! Well no not really. If you read the circuits documentation on Event Handlers (http://pythonhosted.org/circuits/man/handlers.html#implicit-event-handlers) you'll quickly realize that all the methods defined on the Bot Component above are in fact Event Handlers.

Each Event Handler is handling a different part of the IRC Protocol.

Note the two different ways of registering components here. You can either explicitly use .register(...) or use the + or += operators to combine components together into more complex components.

Phase VI - Main

And finally the main function and entry-point of our application that kicks it all off. Here we parse the command line options and arguments, create an instance of the Bot Component, pass in our options and run it.

def main():
    opts, args = parse_options()

    host = args[0]
    port = int(args[1]) if len(args) > 1 else 6667

    opts.output = path.abspath(path.expanduser(opts.output))

    # Configure and run the system.
    Bot(host, port, opts=opts).run()


if __name__ == "__main__":
    main()

A complete version of this sample is available here:

http://bitbucket.org/prologic/irclogger

irclogger is used in production to log various channels on the FreeNode IRC Network including (at the time of writing):

#crux
#crux-devel
#circuits
#circuits-dev

Thanks for reading! Comments?

Random Password Generator

2013-02-21T22:08:00+10:00

I've been seeing a lot of this lately on ActiveState Code Recipes; Random Password Generators written in Python. Well I thought I'd get in on this and share one I've been using for many years, but mostly because I'm finding "Blogging" to be quite fun!

This recipe is also a part of my very old pymills repository of useful modules and routines I've either written or collected. You might find bits and pieces of my library useful. If you do please let me know!

UPDATE: Fixed some bugs...

#!/usr/bin/env python

import string
from time import time
from itertools import chain
from random import seed, choice, sample


def mkpasswd(length=8, digits=2, upper=2, lower=2):
    """Create a random password

    Create a random password with the specified length and no. of
    digit, upper and lower case letters.

    :param length: Maximum no. of characters in the password
    :type length: int

    :param digits: Minimum no. of digits in the password
    :type digits: int

    :param upper: Minimum no. of upper case letters in the password
    :type upper: int

    :param lower: Minimum no. of lower case letters in the password
    :type lower: int

    :returns: A random password with the above constaints
    :rtype: str
    """

    seed(time())

    lowercase = string.lowercase.translate(None, "o")
    uppercase = string.uppercase.translate(None, "O")
    letters = "{0:s}{1:s}".format(lowercase, uppercase)

    password = list(
        chain(
            (choice(uppercase) for _ in range(upper)),
            (choice(lowercase) for _ in range(lower)),
            (choice(string.digits) for _ in range(digits)),
            (choice(letters) for _ in range((length - digits - upper - lower)))
        )
    )

    return "".join(sample(password, len(password)))

This recipe is also available here: - http://code.activestate.com/recipes/578468-password-generator-mkpasswd/?in=user-4167757

Enjoy! :)

Python trademark at risk in Europe: We need your help!

2013-02-19T00:41:00+10:00

In the style of Vasudev Ram's version, c'mon Aussies! Let's all be a good mate and pitch in!

I haven't seen any Python developers from down under blog about this yet so let's give a helping hand :)

--James Mills / prologic - Lead developer of circuits

Unpacking and Conditional Expressions

2013-02-17T13:20:00+10:00

The other day whilst working on a web crawler at work called spyda I came across an interesting behavior in Python. Here it goes:

>>> xs = [(1, 2), (3, 4)]
>>> a, b = xs[0] if xs else None, None
>>> a
(1, 2)
>>> b

Notice that a becomes the first tuple in the list and b becomes None? This kinda got me stumped at first until I realized what was happening with the Python parser. If however you create your expression like this:

>>> a, b = xs[0] if xs else (None, None)
>>> a
1
>>> b
2
>>>

Then everything is all good!

The lesson here (I think) is to use parenthesis here to both make things clearer to the reader and to ensure the correct expression evaluation.

Introducing the circuits team

2013-02-16T04:04:00+10:00

Lately circuits has seen some presence in the wild and this is a good thing for the project. Just the other day Crunch.io posted a job advert on the Python Job Board in need of a RESTful Web Developer to designer and build their public API(s). Naturally I sent them an email asking if they'd like to hire the lead developer of circuits (that's me!) :)

I thought it was about time I setup a circuits team repository on bitbucket to make things a little more prominent.

I've also setup a circuits developer group at google groups which commits and build failures will go to and hopefully discussions amongst developers of circuits.

Anyone is welcome to join!

We're also on #circuits-dev on the FreeNode IRC Network.

Hello World!

2013-01-29T12:09:00+10:00

Hello World! Okay so I'm a programmer, that's obvious right? :)

For a while now users of my Python Project circuits have suggested that I start writing a blog about circuits. Over the years I have resisted the urge to do so due to time constraints and well lack of suitable Blogging software (//or service//).

Today this changes with the realization of Pelican. This wonderfully simple little tool allows me to create blog entries in almost the way I'd like: - Using my favorite text editor: vim - Maintaining my blog entries with my favorite DVCS: mercurial - Command line based. - Written in Python. - Pluggable - Themeable

Well Pelican fits all these quite well I think. The only thing I'd like to do (//time permitting//), is build a creole wiki support into Pelican.

Over the next coming weeks and months I will start to put out blog posts about circuits and Python.

Thanks for visiting! Comments?