Tomatohater

Long time, no post

2021-03-12T12:00:00-08:00

I haven't written anything here since May 2017, but since then there have been so many changes in my life. I moved my family to the Seattle area and started a new job at Starbucks. I'm waaaaay overdue for my "just started a new gig" post, so I'd to catch you up over a series of posts in the coming weeks.

While I've always been a tree hugger at heart, most recently I've been on a journey (a.k.a. obsession) to drive digital sustainability at Starbucks; that means reducing the carbon footprint caused my our digital business.

It's an ongoing process, but my next step is to use Tomatohater.com as a proving ground to minimize the carbon emissions of this website, and to discuss the results and methodologies. I'll need to begin by baselining this site's current carbon footprint. I'll look at hosting and data transfer, I already have a good start here since I use a static site generator (Pelican) and am heavily cached at a CDN that invests in renewable energy (Cloudflare). The primary opportunity will be in the arena of minimizing the page size and any assets need at runtime. I'll also consider the energy used by web browsers use render this site.

I love what Low Tech Magazine's solar powered website has done and consider it a prime example of how far one can go down this path.

More to follow!

I'm speaking at PGCon 2017 (The PostgreSQL Conference)

2017-05-23T11:00:00-07:00

Later this week, I'll be heading to Ottawa, Ontario to attend PGCon 2017 where I'll be delivering a talk entitled Pgcrypto Avasat! A study in Django's password hashers.

If that doesn't get you excited... I don't know what will!

If you've ever found yourself with a list of hashed passwords and wondered how to crack them, this is the talk for you.
If you've ever been curious about the PGCRYPTO extension in Postgres, this is the talk for you.
If you enjoy doing somewhat nefarious things with Python, this is the talk for you.
If you've ever thought about building custom authentication in Django, this is the talk for you.
If you've didn't even know that Django has a thing called PASSWORD HASHERS, this is very likely the talk for you.

I'll post the slides and video once all is said and done.

If you'll be at PGCon this year, come find me. I'd love to talk!

[Meetup] Building Smarter Offices

2016-07-19T15:41:00-07:00

My company, Huge, will begin hosting a meetup series entitled Building Smarter Offices tomorrow. We'll be discussing some of the smart office work that we've been doing since moving into our new Union Market offices. We'll share the work, and we're eager to learn how your office experience might be improved through user experience and technology.

The first talk is entitled Safely Steal a Conference Room and will be led by our technical architect, Matt Toigo. Come join us at 6:30pm on June 20th for food/drinks and to learn more. More info on Huge Events.

Thanks again, Erica, for making this happen.

Huge DC Hosts React DC June Meetup

2016-06-28T10:41:00-07:00

Come join us tomorrow night at our snazzy new offices in DC's Union Market as we host the React DC June Meetup. We've posted some details on the Huge Events website too.

The main speaker will be John K. Paul on ES2015: The NOS for your React Development Engine.

Big thanks to Matt, Erica, and the React DC organizers for making this happen.

Anti-semitic NYTimes Crossword?

2014-12-07T11:41:00-08:00

This post is a departure from my typical rantings on technology, but I feel a need to put this out there.

Let me start by saying that I absolutely love the New York Times Crossword Puzzle and Will Shortz is kind of a geek hero to me. I attempt the puzzle every day via their official iPhone app. The app also includes "The Mini", a quick puzzle which is just a 5x5 grid and is really fun when you only have a minute or two to spare.

The shape of the puzzle

Last week, on December 1st, I opened up The Mini which was authored by crossword prodigy Joel Fagliano and immediately found myself thinking, "Hmm, this puzzle is shaped like a swastika!" Now, I'm no conspiracy theorist. I don't believe that I'm being wronged at every turn. To the contrary, I generally see the humor and ridiculousness in daily life. So I concluded, "Well, it kinda looks like a pinwheel too. Likely just a coincidence." And then I moved on to the clues...

One Down

The very first clue I attempted happened to be 1-Down:

Captain Kirk or Harrison Ford per Sandler's "The Chanukah Song"

This clue should have delighted me to no end with it's Star Trek AND Star Wars connections. Who doesn't love Sandler? And being a proud "JEW" myself, the answer is one that I could personally identify with.

What a minute! Did that really just happen? Given the context of a puzzle arguably shaped like a swastika and the answer "JEW" in a prime spot... my blood started to curdle. So I wrote to the editor via the in-app feedback link.

NY Times responds

The following day I received an email response that, while probably accurate, was less than sympathetic.

Thank you for writing to The New York Times.

I can assure you that Mr. Fagliano meant no harm in the pattern of squares for today's Mini. As I'm sure you probably know, there are only so many possible arrangements for squares in a 5x5 puzzle. I had to look really hard at it in order to see what you refer to as a swastika, and personally, I mainly see a lot of white space.

Also, the entry JEW had no connection to the pattern of squares in the puzzle. The clue itself is a line from Adam Sandler's "The Chanukkah Song."

My opinion

I really do believe that Joel Fagliano had no malintent with this puzzle. Yet I am still left feeling unsatisfied. The NY Times response basically accused me of dreaming up the swastika shape. To prove that it wasn't just me, I presented the puzzle to a bunch of other individuals. Each answered within seconds and provided only two distinct answers: "swastika" and "Nazi symbol". No one mentioned "pinwheel" and no one even said, "I don't know."

I suppose I fault the NY Times Crossword editorial staff for not catching this. And even once it got out, there has been no acknowledgement that the puzzle could have been interpreted as insensitive.

What next?

Leave me some comments with your thoughts on this. If you are disappointed by this puzzle, please let the editors know by sending a tweet to @NYTimesWordplay.

What am I going to do? Keep on keepin' on. I'll still do my crosswords. I'll still scratch my head over the Sunday puzzle. I only hope that this post yields something constructive. Perhaps driving the crossword staff to consider their audience more carefully; I know they can do better. And maybe we'll eventually see an improved NY Times Crossword product for everyone.

I'm speaking at ModevCon 2014

2014-12-06T11:05:00-08:00

Jay Kissman and I will be presenting The Native App vs. Mobile Web Debate at ModevCon 2014, "the east coast's premier mobile development event." The conference, formerly called ModevEast, takes place on December 11-12 at Artisphere in Arlington, VA. Here is our talk's description from the ModevCon website:

Choosing whether to build a mobile website or native application (or both!) is a decision that every organization needs to make for itself when defining it's mobile and content strategies. During this talk, Celerity's Jay Kissman (native nut) and Drew Engelson (mobile web nerd) will debate the pros and cons of each and review some of the critical decisions organizations face when making the choice.

Oh, and don't miss our very own iOS/3D guru, David Donze, and his talk, Accelerating Apps with OpenGL. Celerity is also a gold sponsor of the event.

Follow the event @gomodev and hashtag #modevcon. Hope to see you there!

Generating a PowerPoint from text files

2014-04-03T23:09:00-07:00

One of my nerdy colleagues, @mattcaldwell, just gave a neat presentation at a Django-District Meetup on using vim+tmux as Django IDE. To keep it real, he created his presentation using vimdeck and actually presented his slides inside of vim.

I figured it would annoy him to no end to see his presentation in PowerPoint format. So here's a script to do it...

#!/usr/bin/python
# -*- coding: utf-8 -*-

import os
import sys
from cStringIO import StringIO
from argparse import ArgumentParser

from pptx import Presentation
from pptx.enum.shapes import MSO_SHAPE
from pptx.dml.color import RGBColor
from pptx.util import Inches, Pt


def is_md(src, f):
    """Returns true if file exists and has a Markdown extension"""
    return os.path.isfile(os.path.join(src, f)) and f.endswith('.md')


def get_pages(src):
    """Returns a list of Markdown filenames in the source directory."""
    return [os.path.join(src, f) for f in os.listdir(src) if is_md(src, f)]


def build_deck(pages):
    """Returns the actual PowerPoint deck."""
    pres = Presentation()
    blank_slidelayout = pres.slide_layouts[6]

    for page in pages:
        slide = pres.slides.add_slide(blank_slidelayout)

        left = top = Inches(0)
        width = Inches(10)
        height = Inches(5.63)

        shape = slide.shapes.add_shape(MSO_SHAPE.RECTANGLE, left, top, width,
                                       height)
        shape.fill.solid()
        shape.fill.fore_color.rgb = RGBColor(0, 0, 0)

        textbox = slide.shapes.add_textbox(left, top, width, height)
        textframe = textbox.text_frame

        para = textframe.paragraphs[0]
        run = para.add_run()
        run.font.name = "Consolas"
        run.font.size = Pt(7)
        run.font.color.rgb = RGBColor(255, 255, 255)

        f = open(page)
        run.text = f.read()
        f.close()

    return pres


if __name__ == '__main__':
    parser = ArgumentParser(
        description='My dorky colleague created a presentation using vimdeck.\
                     The very last thing he would want to see is it as a real\
                     PowerPoint deck. So here you go...',
        epilog='This thing returns binary data, so you probably want to pipe\
                the output to a file with a .pptx extension. Oh, and be sure\
                to "pip install python-pptx" first.')
    parser.add_argument('source', type=str, nargs=1, help='Path to source\
        directory containing vimdeck markdown files.')
    args = parser.parse_args()

    source = os.path.realpath(args.source[0])
    deck = build_deck(get_pages(source))

    output = StringIO()
    deck.save(output)
    sys.stdout.write(output.getvalue())
    output.close()
    sys.exit()

Or grab the Gist. The only Python dependency is pptx. pip install python-pptx will do just fine. For me, this was really just a fun exercise to gain a better understanding of the pptx library. (note: this example was tested against python-pptx==0.5.7)

Python performance: string formatting with .format vs %

2013-08-30T10:12:00-07:00

Python 2.6 introduced the str.format() method for formatting strings which provides a much more flexible alternative to the older modulo (%) based string formatting. But which one performs better? Let's test it out by repeating a simple string format a million times with each method and timing the execution.¹

str.format vs. (%) modulo

from timeit import timeit

def test_modulo():
    'Don\'t %s, I\'m the %s.' % ('worry', 'Doctor')

def test_format():
    'Don\'t {0}, I\'m the {1}.'.format('worry', 'Doctor')

timeit(stmt=test_modulo, number=1000000)
# 0.31221508979797363

timeit(stmt=test_format, number=1000000)
# 0.5489029884338379

Hmmm, the modulo operator is almost twice as fast as str.format() in these simple examples.

In cases where you don't require the flexibility of str.format() and simply want sheer perfomance... Modulo has got to be the way to go... But according to PEP-3101, str.format() is intended to replace the modulo operator... Let's hope that doesn't happen anytime soon.

Template strings

For completeness, Python also supports string templates (introduced in Python 2.4). Let's put that to the test.

from string import Template

s = Template('Don\'t $what, I\'m the $who.')

def test_template():
    s.substitute(what='worry', who='Doctor')

timeit(stmt=test_template, number=1000000)
# 6.010946035385132

Ouch!

Update: 5/12/2017:

Python 3.6 introduced f-strings which provide a new method of string formatting. Let's see how f-strings perform...

from timeit import timeit

def test_fstring():
    a, b = 'worry', 'Doctor'
    f'Don\'t {a}, I\'m the {b}.'

timeit(stmt=test_fstring, number=1000000)
# 0.276932206004858

Well, there we have it... f-strings combine the elegance of .format(), yet is slightly faster than the modulo operator. Big thanks to @StopSpazzing for pointing this out.

Tests were performed on a MacBook Pro (2.4 Ghz Intel Core i7) and Python 2.7.2. With Spotify playing the Melvins in the background. ↩

Want Heroku Apache (PHP), but getting Node.js stack?

2013-08-29T15:42:00-07:00

I've been using Heroku to host a bunch of static sites using the Apache/PHP hack. All I need is an index.php file in the root of my application to hint to Heroku that I want the PHP (Apache) buildpack... and then I disable PHP altogether since I really just want Apache to serve a static site. But some of my recent attempts at creating these stacks have results in Heroku giving me a Node.js stack instead... WTF?!?

Turns out that some of the newer node-based tools for managing static assets require a "package.json" file, which makes Heroku think you want a Node.js stack.

Quick solution

Specify the buildpack when creating the app.

$ heroku create myapp --buildpack https://github.com/heroku/heroku-buildpack-php

Or specify the buildpack as a config parameter.

$ heroku create myapp
$ heroku config:set BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-php

This will force Heroku to use the buildpack you want.

With the rapid evolution of front end tools for managing your Javascript and CSS assets, we're seeing Node-based tools crop up all over the place... In fact, Node Package Manager (npm) is quickly becoming the de facto standard for installing most new web tools. This is a great thing, however, we're really beginning to blur the lines between front and back end technologies. Since Heroku attempts to introspect an app to determine which stack to create, the existence of package.json creates confusion. And because the PHP stack even isn't officially supported, the Node stack wins. We just need to be more explicit these days.

Tomatohater.com now a Pelican/GitHub site

2013-06-08T00:36:00-07:00

This site is now completely static, generated by Pelican and hosted on GitHub Pages. Ridiculously simple and elgant.

The evolution of tomatohater.com:

Dates                    Technology                     Hosting
-----------------------  -----------------------------  ---------------
Original (2008-2010)     django-basic-blog              Amazon EC2
May 2010 - Dec 2011      django-mingus                  Amazon EC2
Dec 2011 - Mar 2012      Wordpress (custom)             Heroku
Mar 2012 - Jun 2013      Wordpress (custom)             Bluehost
Jun 2013 -               Static, generated by Pelican   GitHub Pages

You can check out the source code at https://github.com/tomatohater/tomatohater.github.io/tree/source

Faster! Faster! Accelerate your business with blazing prototypes

2012-07-19T00:54:00-07:00

Slides for my presentation Faster! Faster! Accelerate your business with blazing prototypes at OSCON 2012.

I'm speaking at OSCON 2012

2012-05-11T04:11:00-07:00

I'll be heading to Portland, OR in July since my talk entitled "Faster! Faster! Accelerate your business with blazing prototypes" was accepted for OSCON 2012 (Open Source Convention).

The general idea is three fold:

Why do businesses still choose COTS solutions? I'll attempt to answer that question and provide some weaponry to help get past that unfortunate reality.
Then I'll reveal my survey results of a number of common development frameworks and assess their fitness for "rapid prototyping". We'll attempt to identify which tools can bring a concept in to real working code in the least amount of time and with the least amount of pain.
Finally, we'll walk through a simple prototyping example using one of these frameworks.

If you haven't already, get registered for the conference. Use code **OS12FOS** an get 20% off registration.

Hope to see you there!

Migrating a Django app to Heroku

2012-01-23T01:48:00-08:00

As a Python and Django kind of guy, I had always been jealous of the Ruby on Rails folks. This has nothing at all to do with the framework itself. No, no, no.... Django all the way. It was the Heroku cloud application platform that had me longing.

Yes, I could run my Django application on Google App Engine, but that requires all sorts of hackery and my app ended up an abomination of the original... too unnatural for my tastes.

I sensed a small shift in the Earth's rotation on Sept 28, 2011. This is when Heroku added support for Python/Django on their platform. I needed to give it a test drive and I was amazed how simple it was. The following is an account of what I did to port one of my existing Django sites to Heroku...

Initial setup

Heroku already provides a decent quick start guide for Python. That's a great place to begin. Check out the Prerequisites and Local Workstation Setup which will get you up and running quickly. It helps if you're already familiar with Git, virtualenv and pip. If you're not, then now is an excellent time to learn!

First things first. Assuming your Django project is already in Git, change directory to the project root. Then...

$ heroku auth:login
... output omitted ...
$ heroku create --stack cedar
... output omitted ...

Believe it or not, we're almost there!

Database config

Now, my existing app is running a MySQL database. I'm going to use the built-in Postgres DB on Heroku. So I need to update my Django settings...

# Database config
DATABASES = {
  'default': {
    'ENGINE': 'django.db.backends.postgresql_psycopg2',
  }
}

Note, that I didn't supply any database info or credentials. Heroku auromatically injects this info into your settings.py file. I also need to dump out my current database which I can later import into my new app (this could also be done with fixtures).

$ python myapp/manage.py dumpdata > db.sql

I'm already storing my Python dependencies in a requirements.txt file. If you're not, you'll need to create this file at the project root.

$ pip freeze > requirements.txt
$ cat requirements.txt
Django==1.3
feedparser==5.1
gunicorn==0.12.2
lxml==2.3.3
psycopg2==2.4.2
python-dateutil==1.5
python-sunlightapi==1.1.0

Don't forget to commit your changes! Then push to Heroku.

$ git commit -a -m 'Mods to run on Heroku.'
$ git push heroku master

That second line above is really something to be admired. Not only does it push the code to Heroku's repository, but then it triggers a real deployment. Heroku automatically copies the files to the stack, installs all the dependencies (via requirements.txt), detects that this is a Django application, and runs the application with "runserver". Poof! Done.

Well, not quite done. But really that's bulk of the it and my app is in fact running. You can confirm this by using the "heroku ps" command.

Running your app

Now I need to do all the normal Django setup stuff, like syncdb and loaddata...

$ heroku run python myapp/manage.py syncdb
$ heroku run python myapp/manage.py loaddata < db.sql

If all goes well, I should be able to hit my site with a web browser at the wacky hostname provided by Heroku when I created the stack. Herou provides a shortcut...

$ heroku open

Final thoughts

We'll that was pretty darn easy. A few other things to note if you're trying this yourself.

You'll need to serve static media from somewhere. I used django.contrib.staticfiles in this example, but that's probably not idea for production. Though the output does get cached... so it's also not too bad.
You don't want to use the built-in Django runserver. I prefer gunicorn and it's easy to configure that!
Enjoy yourself. This is cool stuff!

Dear Congress...

2012-01-19T02:20:00-08:00

Dear Representative Van Hollen, Senator Cardin, and Senator Mikulski:

The "Stop Online Piracy Act" (H.R. 3261) and the "Preventing Real Online Threats to Economic Creativity and Theft of Intellectual Property Act of 2011" (S.968) are intended to solve a worthy problem, yet the methods recommended by these bills I find to be completely offensive.

I greatly value the protection of intellectual property, yet I place the values of freedom and free speech even higher. We must not enact a bill with power to dismantle these very freedoms.

Please find another way. Punish the offender, not the messenger. I have confidence that you will find an acceptable alternative.

Please understand very clearly that your vote for these bills in their current forms will be countered with a vote against you in your next election.

Sincerely,

Drew Engelson, Cabin John, MD

Custom Django management commands on Heroku

2012-01-17T02:51:00-08:00

Running Django management commands is easy on Heroku. For example, to syncdb you simply execute:

$ heroku run python your_app/manage.py syncdb

Easy enough. But you may find that running a custom management command to be a little trickier. You might run into something like this:

$ heroku run python your_app/manage.py your_custom_command
Running python your_app/manage.py your_custom_command attached to terminal... up, run.2
Unknown command: 'your_custom_command'
Type 'manage.py help' for usage.

Ouch! And you wouldn't be alone here:

The solution

This really just comes down to Python not finding your app. You just need to adjust your Python path to include your home directory.

On Heroku, your home directory is generally "/app". You should confirm this by running:

$ heroku run env | grep HOME
HOME=/app

A simple way to adjust your Python path within your Heroku environment (and not mucking with your app) is by setting the PYTHONPATH env variable as follows:

$ heroku config:add PYTHONPATH=/app

To confirm it is set correctly, run:

$ heroku run env | grep PYTHONPATH
PYTHONPATH=/app

Now you can run your custom management command. This also allows you to run these as cron (scheduled) tasks:

$ heroku run python your_app/manage.py your_custom_command
Success!

I hope this post will save you some headbanging. Unless, of course, it's to the Melvins.

Why Django?

2011-11-21T16:06:00-08:00

I have been a heavy Django developer, architect, and evangelist since about 2006 when Nowell Strite and I first saw a presentation by Adrian and Jacob at OSCON. We brought Django back to PBS where it quickly became our standard development platform.

I know why I use Django...

I'm conducting some informal research for a project and I want to hear from you. Why you do use Django? Please leave a comment here or tweet me @handofdoom.

Try to limit your responses to 5 words or less.

Faetus: An FTP interface to Amazon S3 file storage.

2011-11-10T06:35:00-08:00

What?

Faetus is an FTP server that translates FTP commands into Amazon S3 API calls providing an FTP interface on top of Amazon S3 storage.

Why?

Amazon's S3 API is awesome and there are plenty of excellent libraries that make this very simple. However, sometimes you don't have control over a system, and when that system knows how to talk FTP but not S3, Faetus is your solution. Read the blog post for more info.

Download

Get the source code from GitHub.

Known issues

Some FTP clients fail with a socket error when writing a file
Connections occasionally lost when in passive mode

Credit where credit is due

This project wouldn't have been possible without extensive use of pyftpdlib and the work of Chmouel Boudjnah's ftp-cloudfs from which Faetus heavily borrows. Thanks!

pyawschart

2011-11-10T06:35:00-08:00

What?

pyawschart is a Python library for generating charts for Amazon Web Services based on Amazon CloudWatch data.

Why?

Amazon's cloud services provide infrastructure metrics data via it's CloudWatch data API. This library aims to render this data more accessible by creating powerful (and pretty) data visualizations.

Download

Get the source code from GitHub.

Usage

import boto
from pyawschart import RANGES
from pyawschart.rds import CPUUtilizationChart

# Connect with boto
conn = boto.connect_cloudwatch(AWS_ACCESS_KEY AWS_SECRET_ACCCES_KEY)
chart = CPUUtilizationChart(conn, 'my-rds-instance', RANGES['hour'])

# Display chart url
print chart.get_url()

# Download chart image
chart.download('/tmp/my-rds-instance-cpu-hourly.png')

Known issues

| Please see the GitHub project for known issues and to submit bugs and other feedback:

https://github.com/tomatohater/pyawschart/issues

Credit where credit is due

This project makes extensive use of boto and Python Google Chart. Thanks!

django-simplegravatar v0.2

2011-11-10T06:33:00-08:00

What?

A simpler Django template tag to add Gravatar support to your Django projects. This is based on my 2008 post entitled Implementing Gravatar in Django.

Installation

To install this app, simply:

pip install django-simplegravatar

or drop the 'simplegravatar' folder somewhere on your PYTHONPATH
Add 'simplegravatar' to your projects INSTALLED_APPS list in

settings.py

Configuration

There are a few optional settings that can be added to your Django settings file to affect the global behavior of simplegravatar.

| **SIMPLEGRAVATAR_SIZE** (default: 80)

Pixel width and height of Gravatar (they are all square)

| **SIMPLEGRAVATAR_RATING** (default: \g\)

g, pg, r, x

| **SIMPLEGRAVATAR_DEFAULT** (default: \\)

| Default image if no Gravatar exists for email. Should be a full image
  URL to your custom image, or one of the Gravatar built-in defaults...

-   404: do not load any image if none is associated with the email
    hash, instead return an HTTP 404 (File Not Found) response
-   mm: (mystery-man) a simple, cartoon-style silhouetted outline of a
    person (does not vary by email hash)
-   identicon: a geometric pattern based on an email hash
-   monsterid: a generated 'monster' with different colors, faces, etc
-   wavatar: generated faces with differing features and backgrounds
-   retro: awesome generated, 8-bit arcade-style pixelated faces

| **SIMPLEGRAVATAR_SECURE** (default: False)

Use https?

For more information about these default options see:

http://en.gravatar.com/site/implement/images/

USAGE (this goes in your templates)

{% load simplegravatar %}
{% show_gravatar "email@address.com" %}

You may optionally pass a size into this template tag:

{% show_gravatar "email@address.com" 48 %}

If you require a secure image (instead of using SETTINGS):

{% show_gravatar_secure "email@address.com" %}

Enjoy!

Concatenating strings in Django templates

2011-09-22T15:43:00-07:00

You might think that joining two strings in a Django template would have been simple and straightforward. In modern versions of Django, we can achieve it using the add filter.

{{ "Mary had a little"|add:" lamb." }}

However, in earlier versions of Django [TODO: identify specific version], the add filter only works with numbers. Using it on a string has no effect.

So how to do it?

It looks like we're left with the stringformat filter. This feels quite dirty to me, but it works.

{{ "Mary had a little"|stringformat:"s lamb." }}

Enjoy!

Recovering from Amazon cloud outage

2011-04-21T15:41:00-07:00

Amazon Web Services (AWS) have been integral to the successes of nearly all recent project launches at PBS. All of our core applications are deployed out on AWS EC2 servers and RDS database instances. While we have experienced an occasional component failure, these have been infrequent. When failures have occurred, we have typically been able to leverage the agility of the cloud to quickly and easily work through them.

AWS East is down

April 21st was quite another story! Early that morning, AWS began experiencing connectivity issues affecting Elastic Block Store (EBS) volumes in the Northern Virginia region (us-east-1), and hence any EC2 or RDS instances that depend on EBS. Oy vey!

The outage was heavily covered in the press:

What did this mean for PBS? Well it took out our main portal site (PBS.org) and many core services (Merlin API, COVE video API, TV Schedules API, mobile apps for iPad and iPhone, and more) for a while. Ouch! While we try to leverage multiple availability zones where possible, we are entirely in Amazon's East Coast data centers.

Recovering from AWS outage

Since the outage affected only EBS-based EC2 and RDS services in the East region, a path to workaround the outage seems obvious:

Avoid EBS (at least for now)
Go West, young man!

This is exactly what we did this morning. We relaunched some applications from backups on temporarily EBS-less servers. And we migrated some RDS database servers to the West coast region (us-west-1). Once this was accomplished, our public facing systems were back online.

Looking forward

Now what? How can we depend on AWS in the future? Should we migrate our services elsewhere? There is a universal truth that applies here: sh*t happens

In practicality, being in the cloud has provided such an improvement in our overall stability and ability to manage our infrastructure with minimal resources that I can no longer imagine life without it.

So the real question is, "How can we reduce our exposure to this in the future?"

Any ideas?

mod_wsgi and the HTTP Authorization header

2011-04-07T15:39:00-07:00

If one is building a WSGI application on mod_wsgi and one wants this WSGI application to handle HTTP authentication, it is likely that one will need access to the "Authorization" HTTP header at some point. Had one read the documentation, one would have already understood that mod_wsgi does not pass this header to your WSGI application by default. This is probably a very good thing.

To tell mod_wsgi to pass the Authorization header through to your application, just set the mod_wsgi configuration directive WSGIPassAuthorization to "On".

# enable authorization headers
WSGIPassAuthorization On

# set up wsgi app
WSGIApplicationGroup myapp-1
WSGIImportScript /projects/myapp/myapp.wsgi application-group=myapp-1 process-group=myapp-1
WSGIScriptAlias /myapp /projects/myapp/myapp.wsgi

| Note to self (and others who may have wasted time on such things):

Read The Docs!

AWS Identity and Access Management (IAM) with Python

2011-01-22T15:37:00-08:00

With all the AWS services that are now available, our opportunities in the cloud are virtually unlimited. But using any of these services requires access to your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and unfortunately, these keys provides complete access to the kingdom. This may not be a problem for some, but for large enterprises, granular access control is a necessity.

Up until recently, we would have been out of luck. But fortunately Amazon released Identity and Access Management (IAM) which makes flexible access control possible. And boto makes it easy in Python.

Take this example

Say your company uses Amazon S3 store your company's image assets in a variety of S3 buckets. If you needed to grant a third party the ability to upload new images to your S3 account, they will need a set of keys. You wouldn't want to give them your main keys since not only would they gain access to all of your S3 buckets, but also your EC2 instances, RDS databases, etc. Not a good situation.

Here's where IAM comes in. With IAM, you can create a user for this specific purpose which would have it's own unique AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY key pair. Additionally, you can apply an IAM policy to restrict what this user can do... a specific S3 bucket, in this case.

Let's give it a try

In this example, we'll create a user called melvins and grant it access to an S3 bucket called houdini.

import boto

# Connect to IAM with boto
iam = boto.connect_iam(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

# Create user
user_response = iam.create_user('melvins')

# Limit access with IAM
policypolicy_json = '''{
    "Statement":[{
        "Sid":"RandomStringIdentifier",
        "Action":"s3:*",
        "Effect":"Allow",
        "Resource":"arn:aws:s3:::houdini/*"
    }]
}'''

iam.put_user_policy('melvins', 'allow_access_houdini', policy_json)

# Generate new access key pair for 'melvins'
key_response = iam.create_access_key('melvins')

We now have a new set of AWS keys with access limited to a single S3 bucket. We could have just as easily further limited it to specific action (i.e. S3:PutObject), or to other services (i.e. EC2:*).

The objects we created with boto provide us with all sorts of nifty info.

>>> import simplejson as json
>>>
>>> user = user_response.create_user_response.create_user_result.user
>>> print json.dumps(user, indent=4)
{
    "path": "/",
    "create_date": "2011-01-22T20:02:27.900Z",
    "user_id": "AIDAXXXXXXXXXXXXXKXBW",
    "arn": "arn:aws:iam::46XXXXXXXX90:user/melvins",
    "user_name": "melvins"
}
>>>
>>> key = key_response.create_access_key_response.create_access_key_result.access_key
>>> print json.dumps(key, indent=4)
{
    "status": "Active",
    "user_name": "melvins",
    "create_date": "2011-01-22T20:16:17.189Z",
    "secret_access_key": "vDskXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2cob",
    "access_key_id": "AKIAXXXXXXXXXXXXQ5DA"
}
>>>
>>> # Provide new keys to user
>>> print 'AWS_ACCESS_KEY_ID', key.access_key_id
AWS_ACCESS_KEY_ID AKIAXXXXXXXXXXXXQ5DA
>>>
>>> print 'AWS_SECRET_ACCESS_KEY', key.secret_access_key
AWS_SECRET_ACCESS_KEY vDskXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2cob

Using IAM groups

Suppose your have three trusted employees (Buzzo, Lorax and Dale) who require full access to your AWS account. You could give them the master key pair. But what if Lorax decides to leave the company? You would have to change your master key pair, redistribute to the remaining employees, and hope there was nothing depending on the old ones. I would grow tired of this very quickly.

A better solution is to grant each employee his own key pair (as described above). But rather than managing their policies individually, you could add them to a group and apply a group policy.

import boto

# Connect to IAM with boto
iam = boto.connect_iam(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

# Create group
iam.create_group('admins')

# Apply full access IAM policy
policy_json = '''{
    "Statement":[{
        "Effect":"Allow",
        "Action":"*",
        "Resource":"*"
    }
]}'''
iam.put_group_policy('admins', 'allow_all', policy_json)

#Create users
iam.create_user('buzzo')
iam.create_user('lorax')
iam.create_user('dale')

# Generate access keys
iam.create_access_key('buzzo')
iam.create_access_key('lorax')
iam.create_access_key('dale')

# Add users to group
iam.add_user_to_group('admins', 'buzzo')
iam.add_user_to_group('admins', 'lorax')
iam.add_user_to_group('admins', 'dale')

So when Lorax takes off, it's just a simple matter.

iam.remove_user_from_group('admins', 'lorax')iam.delete_user('lorax')

The moral of the story is: Quit distributing your master access keys. Use IAM. I doubt you want to learn this lesson the hard way!

Version note: boto added IAM support in 2.0b3.

pyawschart - v0.2 released

2011-01-10T15:35:00-08:00

I have just pushed the source code for pyawschart - v0.2 out to GitHub. This project was created a few months back and I have been using it for personal (Proboscis) and professional (PBS) projects since then. I've just decided to open source it for the betterment of the AWS and Python communities.

The goal was quite simple... Use boto to pipe Amazon CloudWatch data into Python Google Chart and hope for some really cool data visualizations.

The results are pretty neat:

Features

pyawschart - v0.2 supports all of the metrics that Amazon makes available for Elastic Compute Cloud (EC2), Relation Database Service (RDS), Elastic Block Store (EBS), and Elastic Load Balancer (ELB).

Example code

Get EC2 cpu utilization chart for the past hour.

import boto
from pyawschart import RANGES
from pyawschart.ec2 import CPUUtilizationChart

# Connect with botoconn = boto.connect_cloudwatch(AWS_ACCESS_KEY AWS_SECRET_ACCCES_KEY)
chart = CPUUtilizationChart(conn, 'i-6d3efXXX', RANGES['hour'])

# Display chart url
print chart.get_url()

# Download chart image
chart.download('/tmp/i-6d3efXXX-cpu-hour.png')

Get RDS database connections chart for the past day.

import boto
from pyawschart import RANGES
from pyawschart.rds import DatabaseConnectionsChart

# Connect with botoconn = boto.connect_cloudwatch(AWS_ACCESS_KEY AWS_SECRET_ACCCES_KEY)
chart = DatabaseConnectionsChart(conn, 'my-rds-instance', RANGES['day'])

# Display chart url
print chart.get_url()

# Download chart image
chart.download('/tmp/my-ec2-instance-connections-day.png')

Feedback

While the library works well for most of the use cases I require, I would welcome feedback on your use cases. Use it... Abuse it.. then submit comments on this entry, or post issues/requests on GitHub.

URLs

Project page: http://tomatohater.com/pyawschart/
Source code: http://github.com/tomatohater/pyawschart
Issue tracker: http://github.com/tomatohater/pyawschart/issues

New PBS.org launched!

2010-10-25T15:32:00-07:00

Today is a big day. Just after midnight we pulled the trigger and launched the new re-envisioned PBS.org website. This is a huge step for PBS on a number of fronts; mostly having to do bringing PBS to the forefront of new media.

But for me, as you might imagine, the most interesting aspect is the technology architecture underlying it all.

A win for Django and Python

There is no doubt that the new PBS.org site probably ranks as one of the most highly trafficked Django sites in existence today. I think there is a lot to be said about how Django may be scaled if an application is properly designed. The tools are all there for the taking. I will write a future post that details the full architecture, but here is a short list of technologies involved in the stack: Python, Django, MySQL, Apache, Memcached, Celery, Piston, Amazon Web Services

And thanks for cloud computing...

You may have noticed Amazon Web Services mentioned above. I slipped it in at the end of the list, but in reality, it is truly the enabling technology that makes all else possible. The new PBS.org site sits on a cluster of Amazon EC2 instances and is powered by a few Relational Database Service (RDS) instances. In my opinion, RDS is already very cool, and getting much cooler.

Putting the applications in the cloud lets us be very agile on the infrastructure side. In fact, just prior to launch, we decided to put the cache on a bigger machine and that took all of 10 minutes. This kind of stuff is just not possible on a traditional infrastructure. Thank you, Amazon.

And big thanks to the entire team of smart and hardworking individuals that made this site launch a success. Hats off to you all!

CloudFront object invalidation in Python

2010-10-22T15:31:00-07:00

Amazon Web Services enables the management of all AWS services through some pretty powerful REST APIs. CloudFront is Amazon's content delivery network offering which allows you to serve files out of S3 buckets with much better performance. S3 files are cached at edge servers so that users may download them more quickly. The service plays nice with cache-control and expiration headers, but what happens when you need to delete or update some cached files from the edge immediately?

CloudFront object invalidation is a recent addition... here's how to do it in Python.

AWS REST authentication

One of the first things you'll need to do for each request is generate an authentication signature based on your AWS credentials. So let's define a function for that.

import base64
import datetime
import hmac
import hashlib

def get_aws_auth():
    date = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    auth = 'AWS %s:%s' % (
        AWS_ACCESS_KEY,
        base64.b64encode(
            hmac.new(
                AWS_SECRET_ACCESS_KEY.encode('utf-8'),
                date.encode('utf-8'),
                hashlib.sha1
            ).digest()
        )
    )
    return date, auth

print get_aws_auth()('Thu, 22 Oct 2010 06:08:48 GMT', 'AWS 0PN5J17HBGZHT7JJ3X82:4cP0hCJsdCxTJ1jPXo7+e/YSu0g=')

We can now use get_aws_auth() to sign the following REST requests. Next, we'll create an invalidation request.

Invalidating CloudFront objects

The invalidation handler expects a POST request with an xml payload containing a list of file paths to be expired.

import urllib2

host = 'cloudfront.amazonaws.com'
path = '/2010-08-01/distribution/%s/invalidation' % AWS_CF_DISTRIBUTION_ID

# create authentication signature
date, auth = get_aws_auth()

files = ['/file-01.mp4', '/file-02.mp4', '/file-03.mp4']

# create xml data to post
data = ['<?xml version="1.0" encoding="UTF-8"?>']
data.append('<InvalidationBatch>')
data.append('%s' % ''.join(['<Path>/%s</Path>' % f for f in files]))
data.append('<CallerReference>%s</CallerReference>' % date)
data.append('</InvalidationBatch>')
data = ''.join(data)

# define required http headers
headers = {
    'Host': host,
    'Authorization' : auth,
    'Date': date,
    'Content-Type': 'text/xml',
    'Content-Length': len(data),
}

# make REST request, capture 201 (success) response
request_url = 'https://%s%s' % (host, path)
req = urllib2.Request(request_url, data, headers=headers)
try:
    response = urllib2.urlopen(req)
except urllib2.HTTPError, e:
    print "HTTP Error 201: Created"

Of course, you probably also want to check the status of your invalidation request

Retrieving invalidation request status

Invalidation requests can take up to 15 minutes. Hit the GET invalidation API to check on the status of your recent requests.

import urllib2
from xml.dom import minidom

host = 'cloudfront.amazonaws.com'
path = '/2010-08-01/distribution/%s/invalidation?MaxItems=2' % AWS_CF_DISTRIBUTION_ID

# create authentication signature
date, auth = get_aws_auth()

# define required http headers
headers = {
    'Host': host,
    'Authorization': auth,
    'Date': date,
}

# send REST request and print results
request_url = 'https://%s%s' % (host, path)
req = urllib2.Request(request_url, headers=headers)
response = urllib2.urlopen(req)
xml = response.read()
print minidom.parseString(xml).toprettyxml(indent=' '*4)
<?xml version="1.0" ?>
<InvalidationList xmlns="http://cloudfront.amazonaws.com/doc/2010-08-01/">
    <Marker/>
    <NextMarker>
        I1Z3T4GFYRIAZU
    </NextMarker>
    <MaxItems>
        2
    </MaxItems>
    <IsTruncated>
        true
    </IsTruncated>
    <InvalidationSummary>
        <Id>
            I2HD8ADHMH7R
        </Id>
        <Status>
            InProgress
        </Status>
    </InvalidationSummary>
    <InvalidationSummary>
        <Id>
            IEFHBD78AAQN
        </Id>
        <Status>
            Completed
        </Status>
    </InvalidationSummary>
</InvalidationList>

The easy way: Use boto

As of version 2.0b3, the kick-ass boto library (Python interface to Amazon Web Services) has added support for invalidating CloudFront objects.

from boto.cloudfront import CloudFrontConnection

files = ['/file-01.mp4', '/file-02.mp4', '/file-03.mp4']

conn = CloudFrontConnection(AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY)
print conn.create_invalidation_request(AWS_CF_DISTRIBUTION_ID, files)
<boto.cloudfront.invalidation.InvalidationBatch object at 0x1012b6390>

So short and sweet.

Pretty print xml (and json) in Python

2010-10-20T15:29:00-07:00

So that I don't forget... again.

from xml.dom import minidom

# from a string
minidom.parseString(xml_string).toprettyxml(indent=' '*4)

# or from a file
minidom.parse(open(xml_file)).toprettyxml(indent=' '*4)

And for the record...

import simplejson as json

json.dumps(json_obj, indent=4)

The NPR Radio by Livio

2010-10-07T15:28:00-07:00

I was at Micro Center today and came across this neat looking device called The NPR Radio by Livio. My first reaction was, "Wow! This is so cool. You can listen to NPR anytime, anywhere! Why aren't we selling The PBS Television yet?" or something along those lines.

But upon thinking about this for a few moments, it occurred to me that my regular radio already gets NPR, costs less than $199, and it doesn't even require an internet connection.

Private repos on BitBucket

2010-10-02T15:27:00-07:00

Well now they've done it... With the acquisition of BitBucket by Atlassian, we can expect BitBucket to see tremendous gains in usability and developer support. They now offer unlimited public and private repositories for up to 5 users for free.

Will users flock to Mercurial leaving Git behind? I say, "Hello hg!"

Private clouds for developers

2010-09-23T15:24:00-07:00

At PBS we have been launching most new applications up in Amazon's cloud platform utilizing EC2 instances, EBS and S3 storage, etc. The flexibility and agility that this Infrastructure as a Service (IaaS) is truly game changing, if not frightening (to IT departments).

The ease with which I am able spin up a server or 30 still gives me the chills even though I've been doing this for years by now. In fact, it is so easy that in addition to launching production applications in the cloud, we've been providing developers each their own dedicated EC2 servers for the applications they are building, modifying, or testing. The benefits of doing so are easy to see:

Development environment is identical in all aspects to production environment.
Deployment systems (such as Fabric) can be used to push to QA, staging and production servers as easily as to the developers instance.
Instances can be easily cloned if something goes wrong. Made a mistake on configuring the environment? Who cares? Blow it away and start over.

But is it too easy?

For each active application, we may have a dozen or so developers, each with their own EC2 instance. As more applications and developers come on board, you can foresee how we might end up with multiplicative growth in the number of development servers. Let's take 10 developers each working on 3 projects... that's 30 development instances for 3 projects. And that's not counting QA, staging or production instances. Since a throw-away development instance costs just as much as a mission critical production instance... the cost balance feels out of whack to me.

Is there a way to bring down the cost of development instances while maintaining the above-mentioned benefits? What about hosting development servers in a private AWS compatible cloud, such as those fronted by Eucalyptus? There may be potential here and we are prototyping this out. But I suspect the underlying hardware required to run such a private cloud would be as costly (and less flexible) than using dedicated AWS EC2 instances.

Amazon's new micro instances should provide some relief. Any other ideas?

Future of Faetus...

2010-07-21T15:21:00-07:00

In my recent post entitled, Faetus v0.5 released!, I described how and why I built a Python-based FTP server that reads and writes to Amazon S3. This project was born out of necessity because I am using a 3rd-party file management application (which didn't understand the S3 API) but needed the files to end up on S3. The system did talk FTP, so I wrote an FTP server that does just that.

As it turns out, the file manager DOES now support the S3 API. Therefore my immediate need for an intermediary FTP interface is gone. While I think it is still a very cool idea, I no longer have an urgent need for it. I will continue to fix some of the major bugs with it, but most of my tinkering will cease. Please let me know if you are interested in using Faetus and I'm happy to help you out if I can.

Faetus v0.5 released!

2010-07-15T15:20:00-07:00

Faetus is an FTP server that translates FTP commands into Amazon S3 API calls providing an FTP interface on top of Amazon S3 storage.

This project came out of a specific need I have at PBS. I need to put files on Amazon S3 storage, but the 3rd-party file management system doesn't know how to speak to S3's APIs. What the system does understand is FTP. So I decided to write an FTP server that utilizes S3 storage as opposed to local filesystems.

The server is based on pyftpdlib, a Python library that implements most, if not the full RFC spec for. Especially nice is the fact that the the library implements the filesystem layer in an abstract fashion. I basically needed to replace all the filesystem calls with S3 API calls. I used boto to handle all the S3 API calls and I quickly had a working prototype.

After playing with my prototype for a few hours, I stumbled upon Chmouel's ftp-cloudfs... who basically did the exact thing I had just done, but for Rackspace CloudFiles. For consistency's sake, I rolled my code back, forked ftp-cloudfs, gave the project a questionable new name, and added my S3 code back in. La voila!

Faetus project page
Source code on GitHub

GoogleCL is serious life automation

2010-06-21T15:19:00-07:00

Google released GoogleCL last Friday they describe as "Command line tools for the Google Data APIs". I took a few minutes to play with it today and an awed by the possibilities it opens up. Yes, APIs have existed for all these services, so in theory, it's nothing new... But really, who wants to write command-line wrappers for the APIs, or manage all the different 3rd party packages for doing so.

It's basically a bunch of Python scripts that are all imported into one executable file called "google". All the API calls are full abstracted allowing you do run commands like:

$ google contact list
$ google calendar add "Get up at 5:30am Thursday"

GoogleCL currently supports Blogger, Calendar, Contacts, Docs, Picasa, YouTube... though I'm sure others will be added shortly.

Installing is pretty much a piece of cake. All you really need is a modern version of the gdata library (mine was outdated) and the googlecl package itself.

$ pip install gdata googlecl
$ pip install

I see some serious life automation happening in the near future.

django-openlike v0.1

2010-05-20T15:17:00-07:00

| Source code available at GitHub:

http://github.com/tomatohater/django-openlike

Release notes (v0.1):

# django-openlike
Django template tag for easily implementing and extending OpenLike protocol.

See http://www.openlike.org/ for more info.

FEATURES:
    - Templatetags for displaying OpenLike widget

USAGE:
    - Basic: {% openlike %}
    - Less basic: {% openlike "http://domain.com/" "Page title" %}

TODO:
    - Documentation.
    - Add support for extended sources.
    - Configuration via Django admin

AUTHOR'S NOTE:

This app is most definitely overkill given the simplicity of OpenLike and the ease of pure JavaScript implementation. But it was a fun project to fully support a third-party service in Django.

My slides from RightScale MeetUp 2010

2010-05-19T15:15:00-07:00

This is my deck from RightScale MeetUp 2010 before it was incorporated into RightScale's template. The full MeetUp presentation is up on RightScale's website.

At the MeetUp, I met a ton of really smart people with interesting problems to solve... many with the same problems I'm working on as well.

Talking points: Django v. Drupal

2010-05-13T15:13:00-07:00

At PBS, I frequently find myself with a proposal to build a web application with Drupal. We're primarily a Django shop, so you can imagine my bias against Drupal. Yet these proposals continue and I routinely find myself pushing Django instead.

To date, we have attempted exactly four Drupal sites at PBS. Three of them eventually made it to production. All three experienced significant delivery, quality assurance and/or scalability problems. These apps were built by experienced development shops, even some who had proven track records and delivered exceptional (non-Drupal) applications in the past.

I am sure that it is possible to build a solid, highly scalable Drupal application. There are many examples out in the world. So what's the deal?

In a recent post, Drupal v. Django, Nick Sergeant provides some excellent talking points on the matter. I agree with every one of his points. Some of my favorites are echoed below:

Django is a lower level development framework (than Drupal) that encourages rapid development and cleanly written & organized code.

With Django, we're able to construct custom functionality at a more rapid pace than Drupal.

Drupal funnels data into chunks of content referred to as "nodes", which developers must often force their content into. The content should define the data structure, not the other way around.

Because Drupal stores so much configuration and definitions of functionality in the database, it becomes incredibly difficult to version control the software.

Drupal's default admin interface is riddled with UI and workflow problems (one of the primary focuses of the upcoming Drupal 7 release).

Django's default admin interface is designed to encourage quick and easy content management, and allows developers to quickly design custom admin interfaces and workflows.

I don't mean to add fuel the Django versus Drupal fires, so let's just rename this post to "Why you shouldn't introduce Drupal into a Django-centric environment." I'm just looking to add some sanity to my life.

Tomatohater.com is Mingus-ified!

2010-05-11T15:12:00-07:00

While django-basic-blog is perfectly sufficient to power this site, I was interested in testing how Monty Lounge extended it. You see, django-mingus isn't really an application itself; there's really not much code involved. It is more like a recipe for combining a number of existing reusable Django applications that make up a solid blogging platform. The article The apps that power Django-Mingus describes the 28 apps that have been nicely imaplemented into a single project.

My experience setting my Django-mingus blog was ridiculously simple. My only issues were related to the pip install script...

BitBucket was down tonight...
One updated dependency had had trouble finding the version specified ...
One dependency was broken
I had to change it to use git+http:// since my web host blocked traffic on the port

Other than that, I didn't have to do much customization. I treated my site design just like a new Mingus theme and attempted to make all my changes in CSS. I did have to rearrange some of the HTML to support my layout. I then did a bunch of clean-up in the Django templates to get consistent page titles and page themes.

I'm only using a few of the apps included with Mingus... I'll be digging into the rest to see whether I'm missing out.

Headed to RightScale User Meetup 2010

2010-04-20T15:09:00-07:00

Tomorrow I'll be presenting my experiences using RightScale and Amazon Web Services to drastically change the way PBS web applications are designed and hosted.

Our foray into cloud computing is really still experimental, but it has already proven to be transformative on a number of fronts... we are now able to provide much more access and control to developers, while supporting a wider variety of technologies, all the while achieving higher availability. A win-win-win situation if you ask me.

Jumping into Git

2010-04-14T15:08:00-07:00

So far, it has been relatively painless... We've imported a few SVN repositories, maintained the full history, and development continues. But we're still using Git as though it were Subversion.

To make the most of Git will involve rethinking our development and release practices. So far, Vincent Driessen's A successful Git branching model seems quite logical and is the likely candidate for adoption. Let's see where that takes us. Thanks, Vincent!

Python dict from two lists

2009-04-05T15:01:00-07:00

key_list = ['x', 'y', 'z']
value_list = [0, 1, 2]
combined_dict = dict(zip(key_list, value_list))
print combined_dict{'x': 0, 'y': 1, 'z': 2}

Deploying with Fabric and Subversion

2009-02-03T14:58:00-08:00

Here's a sample fabfile for deploying to production and staging systems directly from Subversion.

def production():
    config.fab_hosts = ['server1.example.com', 'server2.example.com']
    config.app_path = '/path/to/app/'

 def staging():
    config.fab_hosts = ['staging.example.com']
    config.app_path = '/path/to/app/'

 def svn_update():
    "Updates the repository."
    sudo("cd $(app_path); svn up")

 def reboot():
    "Reboot Apache2 server."
    sudo("/usr/sbin/apachectl restart")

 def update():
    require('fab_hosts', provided_by=[production])
    invoke(svn_update)

 def deploy():
    require('fab_hosts', provided_by=[production])
    invoke(svn_update)
    invoke(reboot)

Now, deploying is easy. After committing my changes to Subversion, I simply instantiate:

$ fab production deploy

This is equivalent to:

$ fab production update reboot

Implementing Gravatar in Django

2008-08-16T14:55:00-07:00

Update (January 31, 2011) I just released django-simplegravatar, a pluggable Django template tag based heavily on this post.

This blog is written in Python on Django* and utilizes basic.remarks instead of FreeComment for the comments engine. To add some personalization to the comments, I thought avatars would be a nice touch and decided to implement Gravatar, a fantastic system for centrally managing one's avatar across multiple sites. Gravatar basically associates an image with an e-mail address. Here's mine:

The general idea is to recognize a commenter's e-mail address and automatically display their Gravatar (if they have one). Many web applications have Gravatar plugins such as Wordpress, Joomla, MovableType, etc. I came across a Django project called django-gravatar which attempts to provide this functionality, but since it relies on a User object, commenters would have to create a User account. In general, most commenters are not authenticated and would not have a User object in Django. Clearly, django-gravatar is not for us. No matter... implementing Gravatar couldn't be more simple.

All that is required to display a user's Gravatar is their e-mail address. The URL to the Gravatar image simply takes a hashed e-mail address to identify the user, and a few other optional parameters to control the size and what to do if there us no Gravatar associated with their e-mail address. Here's how to create a custom template tag to make this easy.

First, we need to create the template tag Python code. In a file called myproject/templatetags/mytags.py, the following logic creates a flexible custom tag.

import urllib, hashlib
from django import template

register = template.Library()

@register.inclusion_tag('templatetags/gravatar.html')
def show_gravatar(email, size=48):
    default = "http://www.mysite.com/media/images/no-avatar.gif"
    url = "http://www.gravatar.com/avatar.php?"
    url += urllib.urlencode({
        'gravatar_id': hashlib.md5(email).hexdigest(),
        'default': default,
        'size': str(size)
    })
    return {'gravatar': {'url': url, 'size': size}}

Notice that the e-mail address gets md5 encoded, and that the whole querystring is url encoded. Passing in the size parameter enables this tag to be used for Gravatars of varying sizes. The default URL specifies what to display if the commenter does not have a Gravatar.

Next, we need to create the template that defines the HTML output for the Gravatar image. In my case, myproject/templates/templatetags/gravatar.html.

<div class="gravatar">
    <img src="{{ gravatar.url }}"
        width="{{ gravatar.size }}"
        height="{{ gravatar.size }}" border="0" /></div>

Got it? Pretty simple, huh?

Finally, we need to insert this custom tag into your template where the Gravatar should display. In my case, this is the post_detail.html template that controls my individual post page and comments. Here's the code snippit:

{% load mytags %}
{% if remark_list %}
    <div id="comments">
        <a name="comments"></a>
        <h3>{{ remark_list|length }} comment{{ remark_list|pluralize }}</h3>

        {% for remark in remark_list %}
            <div class="comment">
                {% show_gravatar remark.person_email 48 %}
                <div class="comment_header">
                    {{ remark.person_name }} says...
                </div>
                <div class="comment_body">
                    {{ remark.remark|urlizetrunc:"60" }}
                </div>
                <div class="comment_footer">
                    Posted {{ remark.submit_date|date:"F j, Y, P" }}
                </div>
            </div>
        {% endfor %}
    </div>
{% endif %}

There are two significant pieces of this code:

"{% load mytags %}" makes the custom template available to this template. Remember we called the file mytags.py?
"{% show_gravatar remark.person_email 48 %}" instantiates the tag and passes the commenter's e-mail address and the desired Gravatar image size.

You can use CSS to style this up to your heart's desire. Keep in mind that you will always get a result back from Gravatar even if the commenter does not have a Gravatar. We specified a default image for these cases. If no default was specified, Gravatar will return its own default image (a blue Gravatar logo).

We could extend this to support the remaining Gravatar options, such as ratings, but I'll leave that up to you.

* In case you missed it, this is sarcastic reference to Ruby on Rails.

UPDATE (5/11/2010): The first paragraph of this article no longer accurately describes this blog's software architecture. It no longer uses basic.remarks, but was converted to Django Mingus and uses Disqus for comments (which has built-in Gravatar support).

Multi-blog MovableType... a bad idea?

2008-07-31T01:16:00-07:00

We have been running a single install of MovableType 3.21 to run approximately 33 different blogs. This setup has been running for a few years now, and though it has not been without it's difficulties, overall the system was stable. However, for the past few months, some of the blogs have been experiencing increasingly horrendous rebuild times. Additionally, since comment posting triggers a rebuild, the system times-out before the commenter sees any response to their browser resulting in a user experience similar to sticking one's hand in a meat grinder.

I'm sure this is merely a result of numerous factors compounding one another:

Old version of MT
Very large number of posts (?)
Inefficiently built templates
Too many (and poorly written) plugins installed
Underpowered server

However, it seems to me that the MT's architecture also deserves its share of the blame. Many of the installed plugins are only used by one (or few) blog, yet the 32 other blogs suffer as a result since they all must process the plugin, even if there is nothing to do. I don't think this is any different in MT 4.1 (or 4.2). Anyone?

In the end, we decided to split the blogs off onto their own MT installs and reinstall only the plugins required for each blog (no simple task I must say). So now we've created for ourselves a maintenance nightmare, but hey, at least MT-CustomFields won't break some of my blogs anymore.

Tomatohater

Long time, no post

I'm speaking at PGCon 2017 (The PostgreSQL Conference)

[Meetup] Building Smarter Offices

Huge DC Hosts React DC June Meetup

Anti-semitic NYTimes Crossword?

The shape of the puzzle

One Down

NY Times responds

My opinion

What next?

I'm speaking at ModevCon 2014

Generating a PowerPoint from text files

Python performance: string formatting with .format vs %

str.format vs. (%) modulo

Template strings

Want Heroku Apache (PHP), but getting Node.js stack?

Quick solution

Tomatohater.com now a Pelican/GitHub site

The evolution of tomatohater.com:

Faster! Faster! Accelerate your business with blazing prototypes

I'm speaking at OSCON 2012

Migrating a Django app to Heroku

Initial setup

Database config

Running your app

Final thoughts

Dear Congress...

Custom Django management commands on Heroku

The solution

Why Django?

I know why I use Django...

Faetus: An FTP interface to Amazon S3 file storage.

What?

Why?

Download

Known issues

Credit where credit is due

pyawschart

What?

Why?

Download

Usage

Known issues

Credit where credit is due

django-simplegravatar v0.2

What?

Installation

Configuration

USAGE (this goes in your templates)

Concatenating strings in Django templates

So how to do it?

Recovering from Amazon cloud outage

AWS East is down

Recovering from AWS outage

Looking forward

Links

mod_wsgi and the HTTP Authorization header

AWS Identity and Access Management (IAM) with Python

Take this example

Let's give it a try

Using IAM groups

Links

pyawschart - v0.2 released

Features

Example code

Feedback

URLs

New PBS.org launched!

A win for Django and Python

And thanks for cloud computing...

CloudFront object invalidation in Python

AWS REST authentication

Invalidating CloudFront objects

Retrieving invalidation request status

The easy way: Use boto

Pretty print xml (and json) in Python

The NPR Radio by Livio

Private repos on BitBucket

Private clouds for developers