So below is a brain dump of my view of the conference. I want to thank all the awesome people that came to the open spaces, provided great insight, and to the speakers I attended for showing me cool stuff I’ve yet to see. If you’ve got any questions or corrections from my dump please let me know. And if you were the one giving the talk, or want to add let me know and I’ll toss an link/edit in here going forward.

The full schedule is up at: http://www.pyohio.org/2010/Schedule

Day 1

Python Koans – Tutorial

I started out with the Koans tutorial track. You can get the koans from:
http://bitbucket.org/mcrute/python_koans/src/tip/python%202_6/

The idea is that you get the source code which is setup as a series of unit tests. Each test is broken and you need to fix it in order to understand some aspect of Python. They’ve got a ton of work in here and it’s a really fun way to learn about Python.

For instance there’s a whole test file for tuples. You have to go in and correct the unit tests so they pass and as you correct them you learn how tuples work and some of the ways you can use them.

You can see all the koans in their directory:

http://bitbucket.org/mcrute/python_koans/src/tip/python%202_6/koans/

So one example:

def test_tuples_are_immutable_so_item_assignment_is_not_possible(self):
    count_of_three =  (1, 2, 5)
    try:
        count_of_three[2] = "three"
    except TypeError as ex:
       self.assertMatch(__, ex[0])

Here it’s demonstrating that tuples are immutable. You’ll get an exception if you try to edit the tuple in place like this. Your job is to replace the ‘__’ with the name of the exception you’ll end up getting.

The koans are small and you can easily test these in a python terminal to help you figure things out. Lots of fun.

Log Analysis with Python

This turned out to not be what I was looking for. It seemed like a chance for the writer of this petit tool to show off his tool. I was hoping for more about how to analyze, generate trends, etc from log files. More on the math/methods. His library seems cool and all, but unless a lot of regular time going over log files I’m not sure how handy it would be.

http://crunchtools.com/software/petit/

Open Space – Message Queues

Wrangling the bits, standardizing how apps get built – My Talk

Sprints

On a side note, all hail the awesome subway lady that got serving us all. I think we might have been some 25 people strong showing up at once and hammered here running the store all on her own. She didn’t complain once and was just pure awesome.

Day 2

Vim open space

The talks were so-so and I decided to instead lead an open space on vim and using it for python work.  There were a bunch of people in there and it went over really well. Everyone learned a few new tricks in there. One thing I’ve updated is to use the plugin pylint.vim so that I can get in-line syntax error checking.

Some of the plugins brought up:

• http://www.vim.org/scripts/script.php?script_id=1697
• http://www.vim.org/scripts/script.php?script_id=2050
• http://christianrobinson.name/vim/BufOnly.vim
• http://www.vim.org/scripts/script.php?script_id=2771
• http://www.vim.org/scripts/script.php?script_id=2332

And another link to my vim config: http://github.com/mitechie/pyvim

Controlling Unix Processes with Supervisord

This was a pretty good talk. There were several things I didn’t realize that supervisor could do such as writing your own custom event handlers. For instance they have a custom thing that checks for some message that causes them to spin up more web servers to handle an increased server load.

Some of the tricks like tying all your services together under supervisor was pretty neat. For instance they tie in their mysql servers, web app servers, solr full text servers into one supervisor control setup. So they all startup/work together.

Some of the things they do are because they can provide a user control interface for clients. They create something like a cPanel-like interface for users on their network which allows them to restart/stop servers at will.

Code with Style

This talk basically a walk through Pep8. I try hard to keep to Pep8 and use a vim plugin to check my files as I work on them. The two places I have a hard time with it is the line length when you have long single line messages and with complicated list comprehensions.

He demonstrated a couple of tricks:

• for list comprehensions that are complicated you can turn it into a function that you then run across your list/etc via the filter() function.
• for long strings you can avoid the \ at the end of line by using ()

s = ("my one long"
    " line of text"
    " is this thing here")

will come out as: “my one long line of text is this thing here”

So now I have some work to do to clean up some things I’ve left behind.

Fabric – Open Space

I wasn’t into the next set of talks so I again hosted an open space on fabric.  This one got interesting as many people in the room handle large server farms of hundreds of machines and have very separate ops/dev people. So a developer is separated from some of the things I was showing in fabric like stopping/restarting celery, resetting the web app, etc. There was also some question as to the advantage of using something like Fabric over just bash/shell scripts.

So it was interesting to see where fabric fits in, where it falls short, and while it’s cool in some aspects, there are a number of limitations and concerns in several use cases.

The one regret was that with the discussion going on we never got other people up front showing off their fabric setups which I would have liked to see. I guess we had a hard time keeping the discussion going in line and I’d have liked to see more stuff from other fabric fans.

Making it go faster

This was the last talk I went to. It started out looking at how to profile your code using cProfile and pstats. I had messed with this, but then he covered using runsnakerun in order to do the profiling which gives you pretty graphs that represent the time spend in functions as a series of shells like this:

http://www.vrplumber.com/programming/runsnakerun/screenshot-2.0.png

You can see the site there:

http://www.vrplumber.com/programming/runsnakerun/

So it’s a decent wrapper around pstats for some more visual/easy to view performance stats. I’ll have to try that out in the future.

The Software Center is coming along since I checked it out during the original blueprint phase. I think it could definitely get to an app store look/feel. So first my rants on things I’m not a fan of.

• The universal access icon is white which set upon a light background makes it nearly impossible to read/see.
• In the “Get Software” section, is that really the new Canonical logo? A purple dot? Sorry, but it looks horrible.
• In the “Installed Software” link on the left side I completely missed it had any logo since it’s again, white logo on white background.
• When viewing the details on a software package I love the link for the Website, but the fact that there’s no hover/other indication this is clickable it looks more like a heading vs a link.

Anyway, that’s just my off the cuff nit picking. What I really wanted to go through is the list of features I’d love to see to have the app store concept take off in Ubuntu.

First up, only show the software. Forget the libraries. Anything that starts with lib should be hidden by default. I also don’t think there should be any view of all packages. It’s just scary. I think search and going through categories/simplified interfaces are the only way to go. Does anyone honestly think users are going to go through the entire list? I think there’s a bunch of things that can be done to clean up the lists of package in order to make things approachable to users.

Next up, when I think of app stores I think of paid apps. Now I know Canonical has some software in their online canonical store, but that’s not where I go to install software. I should be able to purchase software right in the Software Center. Along these lines, the Canonical partners repo is the place to put this stuff. Beyond these few things from the Canonical store, I’d love to see this opened up for other software to be submitted for purchase or maybe donation. How cool would it be to be able to support your favorite apps right through the software center?

While we’re talking about the parters repo, how is it that just about none of those packages have icons? Not only that, a quick test of a few shows no info in the “More Info” section. You’d think these packages would have help from Canonical getting into place and these should be the gold standards of user experience for packages in the Software Center.

Finally, and according to a recent shotofjaq.org podcast episode there’s already work going on to allow users to write reviews and ratings of software in there. This is great news and will open up a bunch of user interface enhancements for users looking for software.

So discovery, purchase, and reputation. These are the big things I think need help in the Ubuntu Software Center. What things do you think are missing?

Edit:
And I should have started here, but definitely looks like much of the paid/donations stuff is something they’re looking to do currently. Check out their roadmap.

In an effort to try to have some of that fun without changing careers I’ve joined forces with Craig Maloney and we’ve started a podcast of our own with the idea of putting out extremely techie point of view onto things we have interests in. So we’ll talk about things that come up in the Ubuntu community, represent the Michigan Loco, and just talk tech. If you get a second, check out our first episode over at
http://lococast.net.

It’s definitely a work in progress, but check it out and let us know what you think in the comments at lococast.net.

So my goal is to be able to reuse controllers to provide details for ajax calls, calls from automated scripts, and whole pages. The trouble with this is that the @jsonify decorator in Pylons is pretty basic. It just sets the content type header to application/json and takes whatever you return and tries to jsonify it for you.

That’s great, but I can’t reuse that controller to send HTML output any more. So I set out to figure out how the decorator works and create one that works more like I wish.

The first thing in setting this up was to look at how to structure any ajax output. I can’t stand the urls you hit via ajax that just dump out some content, outputs some string, and makes you have to look up every controller in order to figure out just what you’re getting back.

I prefer to use a structured format back. So what parts do we need. Really, just a few things. Your ajax library will tell you if there’s an error such as a timeout, 404, etc. It won’t tell you if you make a call to a controller and don’t have perimission, or maybe the controller couldn’t complete the requested action. So the first thing we need is some value of success in our request.

The second component is feedback as to why the success came back. If the controller returns a lack of success we’ll want to know why. Maybe it is successful, but we need some note about the process along the way. We need a standard message we can send back.

Finally, we might want to return some sort of data back. This could be anything from a json dump of the object requested to actual html output we want to use.

So that leaves us with a definition

{'success': true, 'message': 'Yay, we did it', 'payload': {'id': 10, 'name': 'Bob'}}

I want to enforace that any ajax controller will output something in format. It makes it much easier to write effective DRY javascript that can handle this and really leaves us open to handle about anything we need.

So my json decorator is going to have to make sure that if the user requests a json response, that it gets all this info. If the user requests an html response, it’ll just return the generated template html.

By copying parts of the @jsonify and the @validate decorators I came up with something that adds a self.json to the controller method. In here we setup our json response parts.

Finally, we catch if this is a json request. If so, return our dumped self.json instance. Otherwise, return the html the controller sends back. If the controller is returning rendered html and is a json request, then we stick that into the payload as payload.html

So take a peek at my decorator code and the JSONResponse object that it uses. Let me know what you think and any suggestions. It’s my first venture into the world of Python decorators.

@mijson decorator Gist

Sample Controller

@myjson()
def pause(self, id):
    result = SomeObj.pause()

    if self.accepts_json():
        if result:
            self.json.success = True
            self.json.message = 'Paused'
        else:
            self.json.success = False
            self.json.message = 'Failed'

        self.json.payload['job_id'] = id

    return '<h1>Result was: %s</h1>' % message

#Response:
#    {'success': true,
#        'message': 'Paused',
#        'payload': {'html': '<h1>Result was: Paused</h1>'}}

So here goes my latest attempt. It’s for a web app I’m working on at work and I REALLY want to have this under at least basic unit tests. Since it’s a database talking web application my first step is to get a test db up and runnging to run my tests against. At least with that up I can start some actual web tests that add and alter objects via some ajax api calls.

In order to get a test db I first had to figure out how to setup a database for the tests. For speed and ease purposes I’d rather be able to use sqlite. This way I don’t need to setup/reset a mysql db on each host I end up trying to run tests on.

Of course this is complicated because I’m using sqllchemy-migrate for my application. This means part of the testing should be to init a new sqlite db and then bring it up to the latest version. In order to do this I had to convert my existing migrations to work in both MySQL and Sqlite.

Step 1: I need a way to tell the migrations code to use the sqlite db vs the live mysql db. I’ve setup a manage.py script in my project root so I hacked it up to check for a –sqlite flag. Not that great, but it works.

"""
    In order to support unit testing with sqlite we need to add a flag for
    specifying that db

    python manage.py version_control --sqlite
    python manage.py upgrade --sqlite

    Otherwise it will default to using the mysql connection string

"""
from migrate.versioning.shell import main

import sys
if '--sqlite' in sys.argv:
    main(url='sqlite:///apptesting.db',repository='app/migrations')
else:
    main(url='mysql://connection_stringl',repository='app/migrations')

Step 2: Not all of my existing migrations were sqlite friendly. I had cheated and added some columns by straight sql like

from sqlalchemy import *
from migrate import *

def upgrade():
    sql = "ALTER TABLE jobs ADD COLUMN created TIMESTAMP DEFAULT CURRENT_TIMESTAMP;"
    migrate_engine.execute(sql);

def downgrade():
    sql = = "ALTER TABLE jobs DROP created;"
    migrate_engine.execute(sql);

This worked great with MySQL, but sqlite didn’t like it. In order to get things to work both ways I moved to using the changeset tools to make these more SA happy.

from sqlalchemy import *
from migrate import *
from migrate.changeset import *
from datetime import datetime

meta = MetaData(migrate_engine)
jobs_table = Table('jobs', meta)

def upgrade():
    col = Column('created', DateTime, default=datetime.now)
    col.create(jobs_table)

def downgrade():
    sql = "ALTER TABLE jobs DROP created;"
    migrate_engine.execute(sql);

A couple of notes. This abstracted the column creation so that sqlite and mysql would take it. Notice I did NOT update the drop command. Sqlite won’t drop columns, and I honestly didn’t care because the goal is for my unit tests to be able to bring up a database instance for testing, I’m not going to run through the downgrade commands in the sqlite database.

Step 3. So with all of the migrations moved to do everything via SA abstracted code vs SQL strings, I was in business. My final problem was one migration in particular. I had changed a field from a varchar to a int field. Sqlite won’t let you do simple ‘ALTER TABLE…’ and even when I had the command turned into SA based changeset code my db upgrade failed due to sqlite tossing an exception.

What did I do? I cheated. First, I updated the migration with the original column definition to an Integer field. I mean, any new installs could walk that migration up just fine. I happen to know that the two deployments right now have already made the change from varchar to int. So for them, the change won’t break anything for upgrade/downgrade.

I then kept the migration with the change, but tossed it in a try/except block so that I could trap it nice and just output a message “If this fails, it’s probably sqlite choking.”. It’s hackish, but works for all my use cases I need.

Now I can create a new test database with the commands

python manage.py version_control --sqlite
python manage.py upgrade --sqlite

Now I can start building my test suite to use this as the bootstrap to create a test db. I’ll have to them remove the file on teardown so that I don’t get any errors, but that’ll be part of the testing setup. Not in memory, but oh well…it works.

def fromdict(self, values):
    """Merge in items in the values dict into our object if it's one of our columns

    """
    for c in self.__table__.columns:
        if c.name in values:
            setattr(self, c.name, values[c.name])

Base.fromdict = fromdict

So in my controllers I can start doing

def controller(self):
    obj = SomeObj()
    obj.fromdict(request.POST)
    Session.add(obj)

The great thing about my new position is that it’s a Python web development gig. I’m finally able to concentrate on learning the ins and outs of the things I’ve been trying to learn and gain experience with in my spare time.

So hopefully as I figure things out I’ll be posting updates to put it down to ‘paper’ so I can think it through one last time.

So started with some more SqlAlchemy hacking. At my new place we use Pylons, SqlAlchemy (SA), and Mako as the standard web stack. I’ve started working on my own first ‘ground up’ project and I’ve been trying to make SqlAlchemy work and get into the way I like doing things.

So first up, I like the instances of my models to be serializable. I like to have json output available for most of my controllers. We all want pretty ajax functionality right? But the json library can’t serialize and SqlAlchemy model by default. And if you just try to iterate over sa_obj.__dict__ it won’t work since you’ve got all kinds of SA specific fields and related objects in there.

So what’s a guy to do? Run to the mapper. I’ve not spent my time I pouring over the details of SA parts and the mapper is something I need to read up on more.

Side Notes: all these examples are from code using declarative base style SA definitions.

The mapper does this great magic of tying a Python object and a SA table definition. So in the end you get a nice combo you do all your work with. In the declarative syntax case you normally have all your models extend the declarative base. So the trick is to add a method of serializing SA objects to the declarative base and boom, magic happens.

The model has a __table__ instance in it that contains the list of columns. Those are the hard columns of data in my table. These are the things I want to pull out into a serialized version of my object.

My first try at this looked something like

def todict(self):
    d = {}
    for c in self.__table__.columns:
        value = getattr(self, c.name)
        d[c.name] = value

    return d

This is great and all but I ran into a problem. The first object I ran into had a DateTime column in it that choked since the json library was trying to serialize a DateTime instance. So a quick hack to check if the column was DateTime and if so put it to string got me up and running again.

if isinstance(c.type, sqlalchemy.DateTime):
    value = getattr(self, c.name).strftime("%Y-%m-%d %H:%M:%S")

This was great and all. I attached this to the SA Base class and I was in business. Any model now had a todict() function I could call.

Base = declarative_base(bind=engine)
metadata = Base.metadata
Base.todict = todict

This is great for my needs, but it does miss a few things. This just skips over any relations that are tied to this instance. It’s pretty basic. I’ll also run into more fields that need to be converted. I figure that whole part will need a refactor in the future.

Finally I got thinking, “You know, I can often do a log.debug(dict(some_obj)) and get a nice output of that object and its properties.” I wanted that as well. It seems more pythonic to do

dict(sa_instance_obj)
# vs
sa_instance_obj.todict()

After hitting up my Python reference book I found that the key to being able to cast something to a dict is to have it implement the iterable protocol. To do this you need to implement a __iter__ method that returns something that implements a next() method.

What does this mean? It means my todict() method needs to return something I can iterate over. Then I can just return it from my object. So I turned todict into a generator that returns the list of columns, values needed to iterate through.

def todict(self):
    def convert_datetime(value):
        return value.strftime("%Y-%m-%d %H:%M:%S")

    d = {}
    for c in self.__table__.columns:
        if isinstance(c.type, sa.DateTime):
            value = convert_datetime(getattr(self, c.name))
        else:
            value = getattr(self, c.name)

        yield(c.name, value)

def iterfunc(self):
    """Returns an iterable that supports .next()
        so we can do dict(sa_instance)

    """
    return self.todict()

Base = declarative_base(bind=engine)
metadata = Base.metadata
Base.todict = todict
Base.__iter__ = iterfunc

Now in my controllers I can do cool stuff like

@jsonify
def controller(self, id):
    obj = Session.query(something).first()

    return dict(obj)

In my PHP5 app for work, I use the Doctrine ORM and I have post insert, update, delete hooks that will actually go in and log changes to the system. It works great and I can build up a history of an object over time to see who changes which fields and such.

With my TG2 app doing inserts and updates I initiall just faked the log events by manually saving Log() objects from within my controllers as I do the work that needs to be done.

This sucks though since the point is that I don’t have to think about things. Anytime code changes something it’s logged. So I had to start searching the SqlAlchemy (SA) docs to figure out how to duplicate this in TG2. I wanted something that’s pretty invisible. In my PHP5 code I have a custom method I can put onto my Models in case I want to override the default logged messages and such.

I found part of what I’m looking for in the SA MapperExtension. This blog post got me looking in the right direction. The MapperExtension providers a set of methods to hook a function into. The hooks I’m interested in are the ‘after_delete’, ‘after_insert’, ‘after_update’ method. These are passed in the instance of the object and a connection object so I can generate an SQL query to manually save the log entry for the object.

So I have something that looks a little bit like this:

from sqlalchemy.orm.interfaces import MapperExtension

class LogChanges(MapperExtension):

    def after_insert(self, mapper, connection, instance):
        query = "INSERT INTO log ( username, \
            type, \
            client_ip, \
            application ) VALUES( '%s', %d, '%s', '%s')" % ( u"rick",
                4,
                u'127.0.0.1',
                u'my_app')

        connection.execute(query)

Then I pass that into my declarative model as:

__mapper_args__ = {'extension': LogChanges()}

This is very cool and all, but it’s not all the way where I want to head. First, the manual SQL logging query kind of sucks. I have an AppLog() model that I just want to pass in some values to to create a log entry. I’m thinking what I really should do is find a different way to do the logging itself. I’m debating between actually doing a separate logging application that I would call with the objects details.

The problem with this is that one of the things I do in my current app is store the old values of the object. This way I can loop through them and see which values actually changed and generate that in the log message. This is pretty
darn useful.

The other downside is that I don’t have a good way to have a custom logging message generator is I just call a Logging app API.

So I think I might try out the double db connection methods that SA and TG2 support. This way I could actually try to use the second db instance with a Logging() object to write out the changes without messing up the current session/unit of work.

The missing part here is that I’m still not really sure how to get the ‘old’ object values in order to generate a list of fields that have been changed. Guess I have some more hacking to do.

It looks like migrating my old posts is going to be a chore. We’ll see how that goes later on. If you want to follow along realize there will be more non-tech content in the blog so subscribe to the tech category if you don’t want the misc stuff.

Now to get working on some new posts about the various projects I’ve been hacking on lately.