Pythonic Perambulations

Embedding Matplotlib Animations in IPython Notebooks

2013-05-12T19:00:00-07:00

I've spent a lot of time on this blog working with matplotlib animations (see the basic tutorial here, as well as my examples of animating a quantum system, an optical illusion, the Lorenz system in 3D, and recreating Super Mario). Up until now, I've not have not combined the animations with IPython notebooks. The problem is that so far the integration of IPython with matplotlib is entirely static, while animations are by their nature dynamic. There are some efforts in the IPython and matplotlib development communities to remedy this, but it's still not an ideal setup.

I had an idea the other day about how one might get around this limitation in the case of animations. By creating a function which saves an animation and embeds the binary data into an HTML string, you can fairly easily create automatically-embedded animations within a notebook.

The Animation Display Function

As usual, we'll start by enabling the pylab inline mode to make the notebook play well with matplotlib.

In [1]:

%pylab inline

Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.kernel.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.

Now we'll create a function that will save an animation and embed it in an html string. Note that this will require ffmpeg or mencoder to be installed on your system. For reasons entirely beyond my limited understanding of video encoding details, this also requires using the libx264 encoding for the resulting mp4 to be properly embedded into HTML5.

In [2]:

from tempfile import NamedTemporaryFile

VIDEO_TAG = """<video controls>
 <source src="data:video/x-m4v;base64,{0}" type="video/mp4">
 Your browser does not support the video tag.
</video>"""

def anim_to_html(anim):
    if not hasattr(anim, '_encoded_video'):
        with NamedTemporaryFile(suffix='.mp4') as f:
            anim.save(f.name, fps=20, extra_args=['-vcodec', 'libx264'])
            video = open(f.name, "rb").read()
        anim._encoded_video = video.encode("base64")
    
    return VIDEO_TAG.format(anim._encoded_video)

With this HTML function in place, we can use IPython's HTML display tools to create a function which will show the video inline:

In [3]:

from IPython.display import HTML

def display_animation(anim):
    plt.close(anim._fig)
    return HTML(anim_to_html(anim))

Example of Embedding an Animation

The result looks something like this -- we'll use a basic animation example taken from my earlier Matplotlib Animation Tutorial post:

In [4]:

from matplotlib import animation

# First set up the figure, the axis, and the plot element we want to animate
fig = plt.figure()
ax = plt.axes(xlim=(0, 2), ylim=(-2, 2))
line, = ax.plot([], [], lw=2)

# initialization function: plot the background of each frame
def init():
    line.set_data([], [])
    return line,

# animation function.  This is called sequentially
def animate(i):
    x = np.linspace(0, 2, 1000)
    y = np.sin(2 * np.pi * (x - 0.01 * i))
    line.set_data(x, y)
    return line,

# call the animator.  blit=True means only re-draw the parts that have changed.
anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=100, interval=20, blit=True)

# call our new function to display the animation
display_animation(anim)

Out[4]:

Your browser does not support the video tag.

Making the Embedding Automatic

We can go a step further and use IPython's display hooks to automatically represent animation objects with the correct HTML. We'll simply set the _repr_html_ member of the animation base class to our HTML converter function:

In [5]:

animation.Animation._repr_html_ = anim_to_html

Now simply creating an animation will lead to it being automatically embedded in the notebook, without any further function calls:

In [6]:

animation.FuncAnimation(fig, animate, init_func=init,
                        frames=100, interval=20, blit=True)

Out[6]:

Your browser does not support the video tag.

So simple! I hope you'll find this little hack useful!

This post was created entirely in IPython notebook. Download the raw notebook here, or see a static view on nbviewer.

Migrating from Octopress to Pelican

2013-05-07T17:00:00-07:00

After nine months on Octopress, I've decided to move on.

I should start by saying that Octopress is a great platform for static blogging: it's powerful, flexible, well-supported, well-integrated with GitHub pages, and has tools and plugins to do just about anything you might imagine. There's only one problem:

It's written in Ruby.

Now I don't have anything against Ruby per se. However, it was starting to seem a bit awkward that a blog called Pythonic Perambulations was built with Ruby, especially given the availability of so many excellent Python-based static site generators (Hyde, Nikola, and Pelican in particular).

Additionally, a few things with Octopress were starting to become difficult: first, I wanted a way to easily insert IPython notebooks into posts. Sure, I developed a hackish solution to notebooks in Octopress which had worked well enough for a while, but a cleaner method would have involved digging into the Ruby source code and writing a full-fledged Octopress extension, based on nbconvert. This would have involved a fair bit of effort to learn Ruby and figure out how to best interface it with the Python nbconvert code. Second, Ruby has so many strange and difficult pieces: GemFiles, RVM, rake... and I never took the time to really understand the real purpose of all of them (self-reflection: what parts of Python would seem strange and difficult if I hadn't been using them for so many years?). The black-box nature of the process, at least in my own case, was starting to bother me.

But the kicker was this: In January I got a new computer, and after a reasonable amount of effort was unable to successfully build my blog. I've been writing my posts exclusively on my old laptop which I somehow managed to successfully set up last August. But that laptop now has a sorely outdated Ubuntu distro that I couldn't upgrade for fear of losing the ability to update my blog. Needless to say, this was not the most effective setup.

It was time to switch my blog engine to Python.

Choosing a Python Static Generator

I started asking around, and found that there were three solid contenders for a Python-based platform to replace Octopress: Hyde, Nikola, and Pelican. I gave Hyde a test-run a few weeks ago in re-making my website, and I really like it: it's clean, straightforward, powerful, and easy to use. The documentation is a bit lacking, though, and I think it would take a fair bit more effort at this point to build a more complicated site with Hyde.

Nikola and Pelican both seem to be well-loved by their users, but I had to choose one. I went with Pelican for one simple reason: it has more GitHub forks. I'm sure this is entirely unfair to Nikola and all the contributors who have poured their energy into the project, but I had to choose one way or another. I'm pleased to say that Pelican has not been a disappointment: I've found it to be flexible and powerful. It has an active developer-base, and makes available a wide array of themes and plugins. For the few extra pieces I needed, I found the plugin and theming API to be well-documented and straightforward to use.

Migrating to Pelican from Octopress

I won't attempt to write a one-size-fits-all guide to migrating to Pelican from Octopress: there are too many possibile combinations of formats, plugins, themes, etc. But I will walk through my own process in some detail, in hopes that it might help others who find themselves in a similar predicament.

I had several goals when doing this migration:

I wanted, as much as possible, to maintain the look and feel of the blog. I like the default Octopress theme: it's simple, clean, compact, and includes all the aspects I need for a good blog.
I wanted, as much as possible, to leave the source of my posts unmodified: luckily Pelican supports writing posts in markdown and allows easy insertion of custom plugins, so this was relatively easy to accommodate.
I wanted to maintain the history of Disqus comments for each page, as well as the Twitter and Google Pages tools.
I wanted, as much as possible, to maintain the same URLs for all content, including posts, notebooks, images, and videos.
I wanted a clean way to insert html-formatted IPython notebooks into blog posts. Nearly half my posts are written as notebooks, and the old way of including them was becoming much too cumbersome.

I was able to suitably address all these goals with Pelican in a few evenings' effort. Some of it was already built-in to the Pelican settings architecture, some required customization of themes and extensions, and some required writing some brand new plugins. I'll summarize these aspects below:

Blog theme

As I mentioned, I wanted to keep the look and feel of the blog consistent. Luckily, someone had gone before me and created an octopress Pelican theme which did most of the heavy lifting. I contributed a few additional features, including the ability to specify Disqus tags and maintain comment history, to add Twitter, Google Plus, and Facebook links in the sidebar and footer, to add a custom site search box which appears in the upper right of the navigation panel, as well as a few other tweaks. The result is what you see here: nearly identical to the old layout, with all the bells and whistles included.

Octopress Markdown to Pelican Markdown

Octopress has a few plugins which add some syntactic sugar to the markdown language. These are tags specified in Liquid-style syntax:

{% tag arg1 arg2 ... %}

I have made extensive use of these in my octopress posts, primarily to insert videos, images, and code blocks from file. In order to accommodate this in Pelican, I wrote a Pelican plugin which wraps a custom Markdown preprocessor written via the Markdown extension API which can correctly interpret these types of tags. The tags ported from octopress thus far are:

The Image Tag

The image tag allows insertion of an image into the post with a specified size and position:

{% img [position] /url/to/img.png [width] [height] [title] [alt] %}

Here is an example of the result of the image tag:

The Video Tag

The video tag allows embedding of an HTML5/Flash-compatible video into the post:

{% video /url/to/video.mp4 [width] [height] [/url/to/poster.png] %}

Here is an example of the output of the video tag:

(see this post for a description of this video).

The Code Include Tag

The include_code tag allows the insertion of formatted lines from a file into the post, with a title and a link to the source file:

{% include_code filename.py [lang:python] [title] %}

Here is an example of the output of the code include tag:

Hello World hello_world.py download

import sys
import os
print("hello_world")

For more information on using these tags, refer to the module doc-strings.

Maintaining Disqus Comment Threads

Static blogs are fast, lightweight, and easy to deploy. A disadvantage, though, is the inability to natively include dynamic elements such as comment threads. Disqus is a third-party service that skirts this disadvantage very seamlessly. All it takes is to add a small javascript snippet with some identifiers in the appropriate place on your page, and Disqus takes care of the rest. To keep the comment history on each page required assuring that the site identifier and page identifiers remained the same between blog versions. This part happens within the theme, and my Disqus PR to the Pelican Octopress theme made this work correctly.

Maintaining the URL structure

By default, Octopress stores posts with a structure looking like blog/YYYY/MM/DD/article-slug/. The Pelican default is different, but easy enough to change. In the pelicanconf.py settings file, this corresponds to the following:

ARTICLE_URL = 'blog/{date:%Y}/{date:%m}/{date:%d}/{slug}/'
ARTICLE_SAVE_AS = 'blog/{date:%Y}/{date:%m}/{date:%d}/{slug}/index.html'

Next, at the top of the markdown file for each article, the metadata needs to be slightly modified from the form used by Octopress -- here is the actual metadata used in the document that generates this page:

Title: Migrating from Octopress to Pelican
date: 2013-05-07 17:00
slug: migrating-from-octopress-to-pelican

Additionally, the static elements of the blog (images, videos, IPython notebooks, code snippets, etc.) must be put within the correct directory structure. These static files should be put in paths which are specified via the STATIC_PATHS setting:

STATIC_PATHS = ['images', 'figures', 'downloads']

Pelican presented a challenge here: as of the time of this writing, Pelican has a hard-coded 'static' subdirectory where these static paths are saved. I submitted a pull request to Pelican that replaces this hard-coded setting with a configurable path: because the change conflicts with a bigger refactoring of the code which is ongoing, the PR will not be merged. But until this new refactoring is finished, I'll be using my own branch of Pelican to make this blog, and specify the correct static paths.

Inserting Notebooks

The ability to seamlessly insert IPython notebooks into posts was one of the biggest drivers of my switch to Pelican. Pelican has an ipython notebook plugin available, but I wasn't completely happy with it. The plugin implements a reader which treats the notebooks themselves as the source of a post, leading to the requirement to insert blog metadata into the notebook itself. This is a suitable solution, but for my own purposes I much prefer a solution in which the content of a notebook could be inserted into a stand-alone post, such that the notebook and the blog metadata are completely separate.

To accomplish this, I added a submodule to my liquid_tags Pelican plugin which allows the insertion of notebooks using the following syntax:

{% notebook path/to/notebook.ipynb [cells[i:j]] %}

This inserts the notebook at the given location in the post, optionally selecting a given range of notebook cells with standard Python list slicing syntax.

The formatting of notebooks requires some special CSS styles which must be inserted into the header of each page where notebooks are shown. Rather than requiring the theme to be customized to support notebooks, I decided on a solution where an EXTRA_HEADER setting is used to specify html and CSS which should be added to the header of the main page. The notebook plugin saves the required header to a file called _nb_header.html within the main source directory. To insert the appropriate formatting, we add the following lines to the settings file, pelicanconf.py:

EXTRA_HEADER = open('_nb_header.html').read().decode('utf-8')

In the theme file, within the header tag, we add the following:

 {% if EXTRA_HEADER %}
 {{ EXTRA_HEADER }}
 {% endif %}

Here is the result: a short notebook inserted directly into the post:

This Is An IPython Notebook

Here is some code:

In [1]:

import numpy
print numpy.random.random(10)

[ 0.25463203  0.55637185  0.02743774  0.57380221  0.52378531  0.95099357
  0.70975568  0.19575853  0.589227    0.06959599]

Here is some math:

$$e^{i\pi} + 1 = 0$$

With all those things in place, the blog was ready to be built. The result is right in front of you: you're reading it. If you'd like to see the source from which this blog built, it's available at http://www.github.com/jakevdp/PythonicPerambulations. Feel free to adapt the configurations and theme to suit your own needs.

I'm glad to be working purely in Python from now on!

Benchmarking Nearest Neighbor Searches in Python

2013-04-29T08:56:00-07:00

I recently submitted a scikit-learn pull request containing a brand new ball tree and kd-tree for fast nearest neighbor searches in python. In this post I want to highlight-ipynb some of the features of the new ball tree and kd-tree code that's part of this pull request, compare it to what's available in the scipy.spatial.cKDTree implementation, and run a few benchmarks showing the performance of these methods on various data sets.

My first-ever open source contribution was a C++ Ball Tree code, with a SWIG python wrapper, that I submitted to scikit-learn. A Ball Tree is a data structure that can be used for fast high-dimensional nearest-neighbor searches: I'd written it for some work I was doing on nonlinear dimensionality reduction of astronomical data (work that eventually led to these two papers), and thought that it might find a good home in the scikit-learn project, which Gael and others had just begun to bring out of hibernation.

After a short time, it became clear that the C++ code was not performing as well as it could be. I spent a bit of time writing a Cython adaptation of the Ball Tree, which is what currently resides in the sklearn.neighbors module. Though this implementation is fairly fast, it still has several weaknesses:

It only works with a Minkowski distance metric (of which Euclidean is a special case). In general, a ball tree can be written to handle any true metric (i.e. one which obeys the triangle inequality).
It implements only the single-tree approach, not the potentially faster dual-tree approach in which a ball tree is constructed for both the training and query sets.
It implements only nearest-neighbors queries, and not any of the other tasks that a ball tree can help optimize: e.g. kernel density estimation, N-point correlation function calculations, and other so-called Generalized N-body Problems.

I had started running into these limits when creating astronomical data analysis examples for astroML, the Python library for Astronomy and Machine Learning Python that I released last fall. I'd been thinking about it for a while, and finally decided it was time to invest the effort into updating and enhancing the Ball Tree. It took me longer than I planned (in fact, some of my first posts on this blog last August came out of the benchmarking experiments aimed at this task), but just a couple weeks ago I finally got things working and submitted a pull request to scikit-learn with the new code.

Features of the New Ball Tree and KD Tree

The new code is actually more than simply a new ball tree: it's written as a generic N dimensional binary search tree, with specific methods added to implement a ball tree and a kd-tree on top of the same core functionality. The new trees have a lot of very interesting and powerful features:

The ball tree works with any of the following distance metrics, which match those found in the module scipy.spatial.distance:

['euclidean', 'minkowski', 'manhattan', 'chebyshev', 'seuclidean', 'mahalanobis', 'wminkowski', 'hamming', 'canberra', 'braycurtis', 'matching', 'jaccard', 'dice', 'kulsinski', 'rogerstanimoto', 'russellrao', 'sokalmichener', 'sokalsneath', 'haversine']

Alternatively, the user can specify a callable Python function to act as the distance metric. While this will be quite a bit slower than using one of the optimized metrics above, it adds nice flexibility.

The kd-tree works with only the first four of the above metrics. This limitation is primarily because the distance bounds are less efficiently calculated for metrics which are not axis-aligned.
Both the ball tree and kd-tree implement k-neighbor and bounded neighbor searches, and can use either a single tree or dual tree approach, with either a breadth-first or depth-first tree traversal. Naive nearest neighbor searches scale as $\mathcal{O}[N^2]$; the tree-based methods here scale as $\mathcal{O}[N \log N]$.
Both the ball tree and kd-tree have their memory pre-allocated entirely by numpy: this not only leads to code that's easier to debug and maintain (no memory errors!), but means that either data structure can be serialized using Python's pickle module. This is a very important feature in some contexts, most notably when estimators are being sent between multiple machines in a parallel computing framework.
Both the ball tree and kd-tree implement fast kernel density estimation (KDE), which can be used within any of the valid distance metrics. The supported kernels are

['gaussian', 'tophat', 'epanechnikov', 'exponential', 'linear', 'cosine']

the combination of these kernel options with the distance metric options above leads to an extremely large number of effective kernel forms. Naive KDE scales as $\mathcal{O}[N^2]$; the tree-based methods here scale as $\mathcal{O}[N \log N]$.

Both the ball tree and kd-tree implement fast 2-point correlation functions. A correlation function is a statistical measure of the distribution of data (related to the Fourier power spectrum of the density distribution). Naive 2-point correlation calculations scale as $\mathcal{O}[N^2]$; the tree-based methods here scale as $\mathcal{O}[N \log N]$.

Comparison with cKDTree

As mentioned above, there is another nearest neighbor tree available in the SciPy: scipy.spatial.cKDTree. There are a number of things which distinguish the cKDTree from the new kd-tree described here:

like the new kd-tree, cKDTree implements only the first four of the metrics listed above.
Unlike the new ball tree and kd-tree, cKDTree uses explicit dynamic memory allocation at the construction phase. This means that the trained tree object cannot be pickled, and must be re-constructed in place of being serialized.
Because of the flexibility gained through the use of dynamic node allocation, cKDTree can implement a more sophisticated building methods: it uses the "sliding midpoint rule" to ensure that nodes do not become too long and thin. One side-effect of this, however, is that for certain distributions of points, you can end up with a large proliferation of the number of nodes, which may lead to a huge memory footprint (even memory errors in some cases) and potentially inefficient searches.
The cKDTree builds its nodes covering the entire $N$-dimensional data space. this leads to relatively efficient build times because node bounds do not need to be recomputed at each level. However, the resulting tree is not as compact as it could be, which potentially leads to slower query times. The new ball tree and kd tree code shrinks nodes to only cover the part of the volume which contains points.

With these distinctions, I thought it would be interesting to do some benchmarks and get a detailed comparison of the performance of the three trees. Note that the cKDTree has just recently been re-written and extended, and is much faster than its previous incarnation. For that reason, I've run these benchmarks with the current bleeding-edge scipy.

Preparing the Benchmarks

But enough words. Here we'll create some scripts to run these benchmarks. There are several variables that will affect the computation time for a neighbors query:

The number of points $N$: for a brute-force search, the query will scale as $\mathcal{O}[N^2]$ . Tree methods usually bring this down to $\mathcal{O}[N \log N]$ .
The dimension of the data, $D$ : both brute-force and tree-based methods will scale approximately as $\mathcal{O}[D]$ . For high dimensions, however, the curse of dimensionality can make this scaling much worse.
The desired number of neighbors, $k$ : $k$ does not affect build time, but affects query time in a way that is difficult to quantify
The tree leaf size, leaf_size: The leaf size of a tree roughly specifies the number of points at which the tree switches to brute-force, and encodes the tradeoff between the cost of accessing a node, and the cost of computing the distance function.
The structure of the data: though data structure and distribution do not affect brute-force queries, they can have a large effect on the query times of tree-based methods.
Single/Dual tree query: A single-tree query searches for neighbors of one point at a time. A dual tree query builds a tree on both sets of points, and traverses both trees at the same time. This can lead to significant speedups in some cases.
Breadth-first vs Depth-first search: This determines how the nodes are traversed. In practice, it seems not to make a significant difference, so it won't be explored here.
The chosen metric: some metrics are slower to compute than others. The metric may also affect the structure of the data, the geometry of the tree, and thus the query and build times.

In reality, query times depend on all seven of these variables in a fairly complicated way. For that reason, I'm going to show several rounds of benchmarks where these variables are modified while holding the others constant. We'll do all our tests here with the most common Euclidean distance metric, though others could be substituted if desired.

We'll start by doing some imports to get our IPython notebook ready for the benchmarks. Note that at present, you'll have to install scikit-learn off my development branch for this to work. In the future, the new KDTree and BallTree will be part of a scikit-learn release.

In [1]:

%pylab inline

Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.

In [2]:

import numpy as np
from scipy.spatial import cKDTree
from sklearn.neighbors import KDTree, BallTree

Data Sets

For spatial tree benchmarks, it's important to use various realistic data sets. In practice, data rarely looks like a uniform distribution, so running benchmarks on such a distribution will not lead to accurate expectations of the algorithm performance.

For this reason, we'll test three datasets side-by-side: a uniform distribution of points, a set of pixel values from images of hand-written digits, and a set of flux observations from astronomical spectra.

In [3]:

# Uniform random distribution
uniform_N = np.random.random((10000, 4))
uniform_D = np.random.random((1797, 128))

In [4]:

# Digits distribution
from sklearn.datasets import load_digits
digits = load_digits()

print digits.images.shape

(1797, 8, 8)

In [5]:

# We need more than 1797 digits, so let's stack the central
# regions of the images to inflate the dataset.
digits_N = np.vstack([digits.images[:, 2:4, 2:4],
                      digits.images[:, 2:4, 4:6],
                      digits.images[:, 4:6, 2:4],
                      digits.images[:, 4:6, 4:6],
                      digits.images[:, 4:6, 5:7],
                      digits.images[:, 5:7, 4:6]])
digits_N = digits_N.reshape((-1, 4))[:10000]

# For the dimensionality test, we need up to 128 dimesnions, so
# we'll combine some of the images.
digits_D = np.hstack((digits.data,
                      np.vstack((digits.data[:1000], digits.data[1000:]))))
# The edge pixels are all basically zero.  For the dimensionality tests
# to be reasonable, we want the low-dimension case to probe interir pixels
digits_D = np.hstack([digits_D[:, 28:], digits_D[:, :28]])

In [6]:

# The spectra can be downloaded with astroML: see http://www.astroML.org
from astroML.datasets import fetch_sdss_corrected_spectra
spectra = fetch_sdss_corrected_spectra()['spectra']
spectra.shape

Out[6]:

(4000, 1000)

In [7]:

# Take sections of spectra and stack them to reach N=10000 samples
spectra_N = np.vstack([spectra[:, 500:504],
                       spectra[:, 504:508],
                       spectra[:2000, 508:512]])
# Take a central region of the spectra for the dimensionality study
spectra_D = spectra[:1797, 400:528]

In [8]:

print uniform_N.shape, uniform_D.shape
print digits_N.shape, digits_D.shape
print spectra_N.shape, spectra_D.shape

(10000, 4) (1797, 128)
(10000, 4) (1797, 128)
(10000, 4) (1797, 128)

We now have three datasets with similar sizes. Just for the sake of visualization, let's visualize two dimensions from each as a scatter-plot:

In [9]:

titles = ['Uniform', 'Digits', 'Spectra']
datasets_D = [uniform_D, digits_D, spectra_D]
datasets_N = [uniform_N, digits_N, spectra_N]

fig, ax = plt.subplots(1, 3, figsize=(12, 3.5))

for axi, title, dataset in zip(ax, titles, datasets_D):
    axi.plot(dataset[:, 1], dataset[:, 2], '.k')
    axi.set_title(title, size=14)

We can see how different the structure is between these three sets. The uniform data is randomly and densely distributed throughout the space. The digits data actually comprise discrete values between 0 and 16, and more-or-less fill certain regions of the parameter space. The spectra display strongly-correlated values, such that they occupy a very small fraction of the total parameter volume.

Benchmarking Scripts

Now we'll create some scripts that will help us to run the benchmarks. Don't worry about these details for now -- you can simply scroll down past these and get to the plots.

In [10]:

from time import time

def average_time(executable, *args, **kwargs):
    """Compute the average time over N runs"""
    N = 5
    t = 0
    for i in range(N):
        t0 = time()
        res = executable(*args, **kwargs)
        t1 = time()
        t += (t1 - t0)
    return res, t * 1. / N

In [11]:

TREE_DICT = dict(cKDTree=cKDTree, KDTree=KDTree, BallTree=BallTree)
colors = dict(cKDTree='black', KDTree='red', BallTree='blue', brute='gray', gaussian_kde='black')

def bench_knn_query(tree_name, X, N, D, leaf_size, k,
                    build_args=None, query_args=None):
    """Run benchmarks for the k-nearest neighbors query"""
    Tree = TREE_DICT[tree_name]
    
    if build_args is None:
        build_args = {}
        
    if query_args is None:
        query_args = {}
        
    NDLk = np.broadcast(N, D, leaf_size, k)
        
    t_build = np.zeros(NDLk.size)
    t_query = np.zeros(NDLk.size)
    
    for i, (N, D, leaf_size, k) in enumerate(NDLk):
        XND = X[:N, :D]
        
        if tree_name == 'cKDTree':
            build_args['leafsize'] = leaf_size
        else:
            build_args['leaf_size'] = leaf_size
        
        tree, t_build[i] = average_time(Tree, XND, **build_args)
        res, t_query[i] = average_time(tree.query, XND, k, **query_args)
        
    return t_build, t_query

In [12]:

def plot_scaling(data, estimate_brute=False, suptitle='', **kwargs):
    """Plot the scaling comparisons for different tree types"""
    # Find the iterable key
    iterables = [key for (key, val) in kwargs.iteritems() if hasattr(val, '__len__')]
    if len(iterables) != 1:
        raise ValueError("A single iterable argument must be specified")
    x_key = iterables[0]
    x = kwargs[x_key]
    
    # Set some defaults
    if 'N' not in kwargs:
        kwargs['N'] = data.shape[0]
    if 'D' not in kwargs:
        kwargs['D'] = data.shape[1]
    if 'leaf_size' not in kwargs:
        kwargs['leaf_size'] = 15
    if 'k' not in kwargs:
        kwargs['k'] = 5
    
    fig, ax = plt.subplots(1, 2, figsize=(10, 4),
                           subplot_kw=dict(yscale='log', xscale='log'))
        
    for tree_name in ['cKDTree', 'KDTree', 'BallTree']:
        t_build, t_query = bench_knn_query(tree_name, data, **kwargs)
        ax[0].plot(x, t_build, color=colors[tree_name], label=tree_name)
        ax[1].plot(x, t_query, color=colors[tree_name], label=tree_name)
        
        if tree_name != 'cKDTree':
            t_build, t_query = bench_knn_query(tree_name, data,
                                               query_args=dict(breadth_first=True, dualtree=True),
                                               **kwargs)
            ax[0].plot(x, t_build, color=colors[tree_name], linestyle='--')
            ax[1].plot(x, t_query, color=colors[tree_name], linestyle='--')
            
    if estimate_brute:
        Nmin = np.min(kwargs['N'])
        Dmin = np.min(kwargs['D'])
        kmin = np.min(kwargs['k'])
        
        # get a baseline brute force time by setting the leaf size large,
        # ensuring a brute force calculation over the data
        _, t0 = bench_knn_query('KDTree', data, N=Nmin, D=Dmin, leaf_size=2 * Nmin, k=kmin)
        
        # use the theoretical scaling: O[N^2 D]
        if x_key == 'N':
            exponent = 2
        elif x_key == 'D':
            exponent = 1
        else:
            exponent = 0
            
        t_brute = t0 * (np.array(x, dtype=float) / np.min(x)) ** exponent
        ax[1].plot(x, t_brute, color=colors['brute'], label='brute force (est.)')
            
    for axi in ax:
        axi.grid(True)
        axi.set_xlabel(x_key)
        axi.set_ylabel('time (s)')
        axi.legend(loc='upper left')
        axi.set_xlim(np.min(x), np.max(x))
        
    info_str = ', '.join([key + '={' + key + '}' for key in ['N', 'D', 'k'] if key != x_key])
    ax[0].set_title('Tree Build Time ({0})'.format(info_str.format(**kwargs)))
    ax[1].set_title('Tree Query Time ({0})'.format(info_str.format(**kwargs)))
    
    if suptitle:
        fig.suptitle(suptitle, size=16)
    
    return fig, ax

Benchmark Plots

Now that all the code is in place, we can run the benchmarks. For all the plots, we'll show the build time and query time side-by-side. Note the scales on the graphs below: overall, the build times are usually a factor of 10-100 faster than the query times, so the differences in build times are rarely worth worrying about.

A note about legends: we'll show single-tree approaches as a solid line, and we'll show dual-tree approaches as dashed lines. In addition, where it's relevant, we'll estimate the brute force scaling for ease of comparison.

Scaling with Leaf Size

We will start by exploring the scaling with the leaf_size parameter: recall that the leaf size controls the minimum number of points in a given node, and effectively adjusts the tradeoff between the cost of node traversal and the cost of a brute-force distance estimate.

In [13]:

leaf_size = 2 ** np.arange(10)
for title, dataset in zip(titles, datasets_N):
    fig, ax = plot_scaling(dataset, N=2000, leaf_size=leaf_size, suptitle=title)

Note that with larger leaf size, the build time decreases: this is because fewer nodes need to be built. For the query times, we see a distinct minimum. For very small leaf sizes, the query slows down because the algorithm must access many nodes to complete the query. For very large leaf sizes, the query slows down because there are too many pairwise distance computations. If we were to use a less efficient metric function, the balance between these would change and a larger leaf size would be warranted. This benchmark motivates our setting the leaf size to 15 for the remaining tests.

Scaling with Number of Neighbors

Here we'll plot the scaling with the number of neighbors $k$. This should not effect the build time, because $k$ does not enter there. It will, however, affect the query time:

In [14]:

k = 2 ** np.arange(1, 10)
for title, dataset in zip(titles, datasets_N):
    fig, ax = plot_scaling(dataset, N=4000, k=k, suptitle=title,
                           estimate_brute=True)

Naively you might expect linear scaling with $k$, but for large $k$ that is not the case. Because a priority queue of the nearest neighbors must be maintained, the scaling is super-linear for large $k$.

We also see that brute force has no dependence on $k$ (all distances must be computed in any case). This means that if $k$ is very large, a brute force approach will win out (though the exact value for which this is true depends on $N$, $D$, the structure of the data, and all the other factors mentioned above).

Note that although the cKDTree build time is a factor of ~3 faster than the others, the absolute time difference is less than two milliseconds: a difference which is orders of magnitude smaller than the query time. This is due to the shortcut mentioned above: the cKDTree doesn't take the time to shrink the bounds of each node.

Scaling with the Number of Points

This is where things get interesting: the scaling with the number of points $N$ :

In [15]:

N = (10 ** np.linspace(2, 4, 10)).astype(int)
for title, dataset in zip(titles, datasets_N):
    plot_scaling(dataset, N=N, estimate_brute=True, suptitle=title)

We have set d = 4 and k = 5 in each case for ease of comparison. Examining the graphs, we see some common traits: all the tree algorithms seem to be scaling as approximately $\mathcal{O}[N\log N]$, and both kd-trees are beating the ball tree. Somewhat surprisingly, the dual tree approaches are slower than the single-tree approaches. For 10,000 points, the speedup over brute force is around a factor of 50, and this speedup will get larger as $N$ further increases.

Additionally, the comparison of datasets is interesting. Even for this low dimensionality, the tree methods tend to be slightly faster for structured data than for uniform data. Surprisingly, the cKDTree performance gets worse for highly structured data. I believe this is due to the use of the sliding midpoint rule: it works well for evenly distributed data, but for highly structured data can lead to situations where there are many very sparsely-populated nodes.

Scaling with the Dimension

As a final benchmark, we'll plot the scaling with dimension.

In [16]:

D = 2 ** np.arange(8)
for title, dataset in zip(titles, datasets_D):
    plot_scaling(dataset, D=D, estimate_brute=True, suptitle=title)

As we increase the dimension, we see something interesting. For more broadly-distributed data (uniform and digits), the dual-tree approach begins to out-perform the single-tree approach, by as much as a factor of 2. In bottom-right panel, we again see a strong effect of the cKDTree's shortcut in construction: because it builds nodes which span the entire volume of parameter space, most of these nodes are quite empty, especially as the dimension is increased. This leads to queries which are quite a bit slower for sparse data in high dimensions, and overwhelms by a factor of 100 any computational savings at construction.

Conclusion

In a lot of ways, the plots here are their own conclusion. But in general, this exercise convinces me that the new Ball Tree and KD Tree in scikit-learn are at the very least equal to the scipy implementation, and in some cases much better:

All three trees scale in the expected way with the number and dimension of the data
All three trees beat brute force by orders of magnitude in all but the most extreme circumstances.
The cKDTree seems to be less optimal for highly-structured data, which is the kind of data that is generally of interest.
The cKDTree has the further disadvantage of using dynamically allocated nodes, which cannot be serialized. The pre-allocation of memory for the new ball tree and kd tree solves this problem.

On top of this, the new ball tree and kd tree have several other advantages, including more flexibility in traversal methods, more available metrics, and more availale query types (e.g. KDE and 2-point correlation).

One thing that still puzzles me is the fact that the dual tree approaches don't offer much of an improvement over single tree. The literature on the subject would make me expect otherwise (FastLab, for example, quotes near-linear-time queries for dual tree approaches), so perhaps there's some efficiency I've missed.

In a later post, I plan to go into more detail and explore and benchmark some of the new functionalities added: the kernel density estimation and 2-point correlation function methods. Until then, I hope you've found this post interesting, and I hope you find this new code useful!

This post was written entirely in the IPython notebook. You can download this notebook, or see a static view here.

Code Golf in Python: Sudoku

2013-04-15T16:00:00-07:00

Edit: based on suggestions from readers, the best solution is down to 162 characters! Read to the end to see how

A highlight-ipynb of PyCon each year for me is working on the little coding challenges offered by companies in the expo center. I love testing my Python prowess against the problems they pose (and being rewarded with a branded mug or T-shirt!) This year, several of the challenges involved what's become known as code golf: writing a solution with minimal keystrokes.

By way of example, take a look at this function definition:

In [1]:

def S(p):i=p.find('0');return[(s for v in
set(`5**18`)-{(i-j)%9*(i/9^j/9)*(i/27^j/27|i%9/3^j%9/3)or
p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:])),[p]][i<0]

This is a valid function definition (in Python 2.7) which executes a particular task. I'll give more information on the workings of this script later on, but for now I'll leave it to the reader to ponder over what it might do.

Given the level of obfuscation involved, you might wonder what the point is: you'd never want to write "real" code in this style, so why spend the time doing it? I'd argue that it's useful for more than just upping your geek cred: good Python code golf must utilize many quirks of the Python language in seeking brevity above all else. Learning to utilize these quirks can lead to a much deeper understanding of the Python language.

I thought about putting together a list of tricks that can help lead to short programs, but the problem is there are so many of them (and there are other pages out there which do this adequately enough). Instead, I decided to simply work through a step-by-step example of creating a code golf solution to a fun little problem: solving Sudoku.

You've probably seen Sudoku: it's a puzzle consisting of a 9x9 grid of numbers, with some spaces left blank. The grid must be filled so that each row, column, and 3x3 box contains the numbers 1-9. It's a generalization of the Latin Squares first studied by Leonhard Euler nearly 300 years ago.

The reason I chose to use Sudoku here is simple: not only is today Euler's birthday, but Sudoku is how I first learned Python. My first year of graduate school, my research advisor recommended that I learn Python for the project I was working on. Sudoku had just become popular in the US at the time, and I decided to learn Python by writing a Sudoku solver. I did it over my winter break, and the rest (so it's said) is history.

Note that this is by no means a new subject: you can read about Sudoku in Python in several places, and there are even a few code golf solutions floating around out there. In particular, you should take a look at this solution, which is the shortest solver I've seen, and from which I borrowed a few of the tricks used below.

Here we'll pose the problem in a slightly different way, which will give us the chance to develop a brand new short algorithm.

The Problem

Every code golf challenge must start with a well-defined problem. Here is ours:

Write a function S(p) which, given a valid Sudoku puzzle, returns an iterator over all solutions of the puzzle.

The puzzle will be in the form of a length-81 string of digits, with '0' denoting an empty grid space. The solved puzzles should also be length-81 strings, with the zeros replaced by solved values.

For example, a valid S(p) may produce the following results:

puz="027800061000030008910005420500016030000970200070000096700000080006027000030480007"
for s in S(puz):
    print(s)

327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657
327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657

puz = 81*'0'  # empty puzzle
print(next(S(puz)))

132598476598476132476132985319825764825764319764913258981257643647389521253641897

Notice that the function S() cannot simply return a list of valid solutions: if it did, then the empty puzzle example would need to produce all ~$10^{22}$ valid sudoku grids before the first solution could be accessed! Instead, it must make use of Python's extremely useful generator syntax. If you've never used generators and generator expressions in your Python code, stop reading this right now and go learn about them: they're one of the most unique and powerful features of the Python language.

As you'll see below, my best solution is ~~176~~ 162 characters, and is the code snippet I showed above:

In [2]:

def S(p):i=p.find('0');return[(s for v in
set(`5**18`)-{(i-j)%9*(i/9^j/9)*(i/27^j/27|i%9/3^j%9/3)or
p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:])),[p]][i<0]

It's rather unenlightening in itself, so below I'll explain the steps I took to arrive at it, in hopes that you can learn from my thought process. Though this is the best solution I was able to come up with, I don't know whether or not a better one might be out there. If you can beat it, please post your solution in the blog comment thread!

Step 1: Focus on Correct Code

A code golf script must be more than simply short: it must be correct. For this reason, I generally start by simply writing correct code, and not for the moment worrying about brevity.

In the case of Sudoku, there are many rules and rubriks that can be used to create an efficient solver (read about some of them here). Using these, it is possible to solve most (all?) Sudoku puzzles without resorting to guess-and-check approaches. To implement this strategy, one approach might be to enumerate the sets of possible values for each grid space, and apply these rules to eliminate values until only a single possibility remains within each space.

Unfortunately, this is not a very suitable approach for code golf: the number of rules required to accomplish this is very large. Instead, we'll make use of the minimal amount of rules, and write a guess-and-check based solver.

Here's a first attempt, focusing on the algorithm rather than on brevity. We'll start by defining a test puzzle with four solutions, and write a small function that can test our solver:

In [3]:

puz = "027800061000030008910005420500016030000970200070000096700000080006027000030480007"

def test(S):
    # solve an empty puzzle
    print(next(S(81*'0')))
    print('')

    # find all four solutions of puz
    for s in S(puz):
        print(s)

In [4]:

# Write functions that, given an index 0 <= i < 81,
# return the indices of grid spaces in the same row,
# column, and box as entry i
def row_indices(i):
    start = i - i % 9
    return range(start, start + 9)

def col_indices(i):
    start = i % 9
    return range(start, start + 81, 9)

def box_indices(i):
    start = 27 * (i // 27) + 3 * ((i % 9) // 3)
    return [i for j in range(3) for i in range(start + 9 * j, start + 9 * j + 3)]

# compute and store the full set of connected indices for each i
connected = [(set.union(set(box_indices(i)),
                       set(row_indices(i)),
                       set(col_indices(i)))
              - set([i]))
             for i in range(81)]

# S(p) will recursively find solutions and "yield" them
def S(p):
    # First, find the number of empty squares and the number of
    # possible values within each square
    L = []
    for i in range(81):
        if p[i] == '0':
            vals = set('123456789') - set(p[n] for n in connected[i])
            if len(vals) == 0:
                return
            else:
                L.append((len(vals), i, vals))
    
    # if all squares are solved, then yield the current solution
    if len(L) == 0 and '0' not in p:
        yield p
        
    # otherwise, take the index with the smallest number of possibilities,
    # and recursively call S() for each possible value.
    else:
        N, i, vals = min(L)
        for val in vals:
            for s in S(p[:i] + val + p[i + 1:]):
                yield s

test(S)

132598476598476132476132985351249768789361254624857319943785621817624593265913847

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

This is the test output we expect: it quickly finds not only the four solutions of the test puzzle, but a solution derived from a completely empty puzzle. This is by no means a complete test suite, but it gives us good reason to believe that the code is correct.

Step 2: Simplify the Algorithm

For me, the biggest hurdle to writing concise programs was letting go of the compulsion to write clear and efficient code. In my research, the two most important aspects of code are its scalability and its readibility. I need my code to work on extremely large datasets, and I need a collaborator to be able to use my code to reproduce or extend my results. Code that doesn't meet these requirements is hardly worth writing. Code golf, though, is different: it's often an exercise in sacrificing efficiency and readability at the altar of brevity.

For the Sudoku problem, we can start in two obvious places.

We can condense the computation of the connected indices by using a nested list comprehension. List comprehensions are a way of shortening a loop to a single statement. In this case, the resulting algorithm is slightly less efficient, a bit less readable, but saves a lot of typing.
Rather than finding the grid space with the fewest possibilities to recursively guess at a solution, we simply choose any unknown grid space. This can be much less efficient, but saves a lot of typing.

Applying these two ideas leads to the following:

In [5]:

# store the full set of connected indices for each i
connected = [set([j for j in range(81)
              if (i%9==j%9) or (i//9==j//9)
                 or (i//27==j//27 and i%9//3==j%9//3)])
             for i in range(81)]
def S(p):
    # find any grid space without a known value
    i = p.find('0')
    
    # if no entry is zero, then yield the current solution
    if i < 0:
        yield p
        
    # otherwise, take this index and recursively call S()
    # for each possible value.
    else:
        for val in set('123456789') - set(p[n] for n in connected[i]):
            for s in S(p[:i] + val + p[i + 1:]):
                yield s
test(S)

132598476598476132476132985319825764825764319764913258981257643253641897647389521

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

This is good, but we can go further by moving the connected list definition into the S() function. Again, this is less efficient than computing the sets once beforehand, but it saves some typing:

In [6]:

def S(p):
    i = p.find('0')
    if i < 0:
        yield p
    else:
        for v in set('123456789')-set(p[j] for j in range(81)
                                      if (i%9==j%9) or (i//9==j//9)
                                      or (i//27==j//27 and i%9//3==j%9//3)):
            for s in S(p[:i]+v+p[i+1:]):
                yield s
test(S)

132598476598476132476132985319825764825764319764913258981257643253641897647389521

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

We can go a little further by using a set comprehension for the loop over possible values. Set comprehensions are like list comprehensions or generator expressions, but are denoted with curly brackets: {}.

We'll also use a trick here based on the way Python implements boolean logic. When you execute something like

(A or B)

you might expect the result to be either True or False. Instead, Python does something a bit clever. If the result is False, it returns A (which, naturally, evaluates to False). If the result is True, it returns A if A evaluates to True, and B otherwise. We can use this fact to remove the if statement completely from the set comprehension. We'll end up with some extra values within the second set, but the set difference conveniently removes these.

In [7]:

def S(p):
    i = p.find('0')
    if i < 0:
        yield p
    else:
        for v in set('123456789')-{(i%9!=j%9)and(i//9!=j//9)
                                   and(i//27!=j//27or i%9//3!=j%9//3)
                                   or p[j]for j in range(81)}:
            for s in S(p[:i]+v+p[i+1:]):
                yield s
test(S)

132598476598476132476132985319825764825764319764913258981257643253641897647389521

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

Step 3: Combining Expressions

Now we have the basics of the algorithm. We can keep shrinking the implementation by combining the two loops into a single generator expression. It's important that we use a generator expression (surrounded by ()) rather than a list comprehension (surrounded by []), because otherwise all possible solutions would need to be computed in order to return a single one!

For clarity, we'll create a temporary explicit container for the generator, which we can remove later. The result of combining the loops looks like this:

In [8]:

def S(p):
    i = p.find('0')
    if i < 0:
        yield p
    else:
        g = (s for v in set('123456789')
             - {(i%9!=j%9)and(i//9!=j//9)
                and(i//27!=j//27or i%9//3!=j%9//3)
                or p[j]for j in range(81)}
             for s in S(p[:i]+v+p[i+1:]))
        for s in g:
            yield s
test(S)

132598476598476132476132985319825764825764319764913258981257643253641897647389521

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

We can further combine the if-else statement into the generator expression to save some more room: if there are no zeros in p, we'll just loop over [p] instead of looping over the generator.

In [9]:

def S(p):
    i = p.find('0')
    g = (s for v in set('123456789')
         -{(i%9!=j%9)and(i//9!=j//9)and(i//27!=j//27or i%9//3!=j%9//3)
         or p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:]))
    for s in (g if i>=0 else[p]):  # parentheses here for clarity
        yield s
test(S)

132598476598476132476132985319825764825764319764913258981257643253641897647389521

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

Step 4: Sweating the Details

We've condensed the script about as much as we can now, but there are still some tiny changes we can make that will save a few characters here or there. This step is the difference between a code golf amateur and a true code golf pro. Some of the tricks I apply here would not have been obvious to me had I not come across this solution, so I don't think I can call myself a pro just yet!

First of all, we can shorten the definition of the full set of nine digits. Observe:

In [10]:

print(set('123456789'))
print(set(str(5**18)))

set(['1', '3', '2', '5', '4', '7', '6', '9', '8'])
set(['1', '3', '2', '5', '4', '7', '6', '9', '8'])

One character shorter! We're making progress.

Next, we can use compact bitwise operators to test whether square i and square j are related. Our previous expression was

(i%9!=j%9)and(i//9!=j//9)and(i//27!=j//27or i%9//3!=j%9//3)

we can equivalently write

(i-j)%9*(i//9^j//9)*(i//27^j//27|i%9//3^j%9//3)

which saves about 12 more characters.

Further, observe that the variable i, which denotes the index of the first zero in the puzzle string, will be -1 if the string has no zeros. The bitwise inverse of -1 is zero, so ~i will evaluate to False only if there are no zeros in the puzzle. This saves a couple more characters. The result is:

In [11]:

def S(p):
    i = p.find('0')
    g = (s for v in set(str(5**18))
         -{(i-j)%9*(i//9^j//9)*(i//27^j//27|i%9//3^j%9//3)
         or p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:]))
    for s in g if~i else[p]:
        yield s
test(S)

132598476598476132476132985319825764825764319764913258981257643253641897647389521

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

Finally, though it's standard to use four spaces for an indentation, Python will also recognize one-space indentations, which save white space characters. At the same time, we'll remove other unnecessary spaces, and move the definition of g into the statement where it's used. To make things easier to parse, we'll replace a required white-space with a line break (between or and p). Because this break falls between two parentheses, the lack of indentation is still parseable.

In [12]:

def S(p):
 i=p.find('0')
 for s in(s for v in set(str(5**18))-{(i-j)%9*(i//9^j//9)*(i//27^j//27|i%9//3^j%9//3)or
p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:]))if~i else[p]:
  yield s
test(S)

132598476598476132476132985319825764825764319764913258981257643253641897647389521

327894561465132978918765423589216734643978215172543896794651382856327149231489657
327894561465132978918765423589216734643978215271543896794651382856327149132489657
327894561645132978918765423589216734463978215172543896794651382856327149231489657
327894561645132978918765423589216734463978215271543896794651382856327149132489657

We've gotten our solution down to 182 characters! As far as I can tell, this is the best we can do in Python versions less than 3.2. Python 3.3, however, added the "yield from" statement, which can help us further shorten this. In a generator definition, writing

yield from G

is (for our purposes, anyway) essentially equivalent to writing

for g in G:
    yield g

so it fits the bill exactly. As a bonus, the removal of nested indentation allows us to write things on a single line, using the ; character in place of a new line:

In [ ]:

def S(p):i=p.find('0');yield from(s
for v in set(str(5**18))-{(i-j)%9*(i//9^j//9)*(i//27^j//27|i%9//3^j%9//3)or
p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:]))if~i else[p]

Using this new syntactic sugar buys us another twelve characters. We're down to 176 characters: not yet tweetable, but I think it's pretty good! Once again, if you see any further abbreviations that can be made, please let me know in the blog comments.

Another Approach

The other shortest sudoku script I've seen is this one, dating back eight years or so and coming in at 185 characters (see the source, and note that due to the change in integer division syntax, the python 3 version, here, is six characters longer than the python 2 version):

In [ ]:

def r(a):i=a.find('0');~i or exit(a);[m
in[(i-j)%9*(i//9^j//9)*(i//27^j//27|i%9//3^j%9//3)or a[j]for
j in range(81)]or r(a[:i]+m+a[i+1:])for m in'%d'%5**18]
from sys import*;r(argv[1])

This script has a slightly different purpose: it's meant to take an argument in the command line and output one answer. For this reason, a direct comparison of the two solutions is somewhat misleading. Taking away the command-line call brings the count down to 174 characters (note the from sys import* is still required for the exit() call). On the other hand, this script only finds a single solution, and does it in a clever but unorthodox way: in order to break out of the recursion efficiently, it returns the solution as an exit code. This works in the sense that the answer prints to the screen, but means that the script is only useful as a stand-alone application.

Regardless of judgments about which solution "won" this round of code golf, I hope you agree with me that this is a valuable exercise. To me, the end goal of code golf is not simply a concise program: it's the pursuit of a deeper knowledge of the ins and outs of the Python language itself.

Update

Several commenters on the blog and on reddit have suggested improvements to the algorithm. First of all, the conditional of the form

(genexp if~i else[p])

can be made one character shorter by using the fact that boolean variables are interpreted as either 1 or zero:

([p],genexp)[i<0]

Also, it was pointed out that the yield from can be replaced by a simple return in this case, because yield is not used anywhere in the function. So the shortest version of the function becomes this:

In [13]:

def S(p):i=p.find('0');return[(s for v in
set(str(5**18))-{(i-j)%9*(i//9^j//9)*(i//27^j//27|i%9//3^j%9//3)or
p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:])),[p]][i<0]

This is 171 characters!

But there's more. Now that the yield from is unnecessary, we can move to python 2.x and change all the Python 3-style integer division operators (//) to Python 2-style (/). This saves six more characters:

In [14]:

def S(p):i=p.find('0');return[(s for v in
set(str(5**18))-{(i-j)%9*(i/9^j/9)*(i/27^j/27|i%9/3^j%9/3)or
p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:])),[p]][i<0]

165 characters, but note that this requires Python 2.7.

There's one more thing we can add, as noted by a commenter below. In Python 2.x, back-ticks can be used as a shorthand for string representation (this is a feature removed in Python 3.x). Thus:

In [15]:

print(str(5**18))
print(`5**18`)

3814697265625
3814697265625

A problem, though, is that in 32-bit architectures, 5**18 is a long integer, so that the string representation is '3814697265625L' (note the L appended at the end). This would lead to incorrect solutions. But as long as we're assured that we're on a 64-bit platform, we can use this to save three more characters:

In [16]:

def S(p):i=p.find('0');return[(s for v in
set(`5**18`)-{(i-j)%9*(i/9^j/9)*(i/27^j/27|i%9/3^j%9/3)or
p[j]for j in range(81)}for s in S(p[:i]+v+p[i+1:])),[p]][i<0]

That brings our best to 162 characters, though it requires Python 2.7 and a 64-bit system. Thanks to all commenters who suggested these improvements!

This post was written in the IPython notebook. The raw notebook can be downloaded here. See also nbviewer for an online static view.

Matplotlib and the Future of Visualization in Python

2013-03-23T08:31:00-07:00

Last week, I had the privilege of attending and speaking at the PyCon and PyData conferences in Santa Clara, CA. As usual, there were some amazing and inspiring talks throughout: I would highly recommend browsing through the videos as they are put up on pyvideo.

One thing I spent a lot of time thinking, talking, and learning about during these two conferences was the topic of data visualization in Python. Data visualization seemed to be everywhere: PyData had two tutorials on matplotlib (the second given by yours truly), as well as a talk about NodeBox OpenGL and a keynote by Fernando Perez about IPython, including the notebook and the nice interactive data-visualization it allows. Pycon had a tutorial on network visualization, a talk on generating art in Python, and a talk on visualizing Github.

The latter of these I found to be a bit abstract -- more discussion of visual design principles than particular tools or results -- but contained one interesting nugget: though D3 is the current state-of-the-art for publication-quality online, interactive visualization, practitioners often use simpler tools like matplotlib or ggplot for their initial data exploration. This was a thought echoed by Lynn Cherny in her presentation about NodeBox OpenGL: it has some really nice features which allow the creation of beautiful, flexible, and interactive graphics within Python. But if you just want to scatter-plot x versus y to see what your data looks like, matplotlib might be a better option.

I found Lynn’s talk incredibly interesting, and not just because of the good-natured heckling I received from the stage! Her main point was that in the case of interactive & animated visualization, matplotlib is relatively limited. She introduced us to a project called NodeBox OpenGL, which is a cross-platform graphics creation package that has some nice Python bindings, and can create some absolutely beautiful graphics. This is one example of a physics flow visualization, taken from the above website:

Despite Lynn’s admission that, with regards to the limitations of matplotlib, I had taken a bit of the wind out of her sails with my interactive matplotlib tutorial the day before (in which many of the examples I used will be familiar to readers of this blog), I think her point on the limitations of matplotlib is very well-put. Though it remains my tool of choice for data visualization and the creation of publication-quality scientific graphics, matplotlib is also a decade-old platform, and sometimes shows its age.

Matplotlib’s History

Matplotlib is a multi-platform data visualization tool built upon the Numpy and Scipy framework. It was conceived by John Hunter in 2002, originally as a patch to IPython to enable interactive MatLab-style plotting via gnuplot from the IPython command-line. Fernando Perez was, at the time, scrambling to finish his PhD, and let John know he wouldn’t have time to review the patch for several months. John took this as a cue to set out on his own, and the matplotlib package was born, with version 0.1 released in 2003. It received an early boost when it was adopted as the plotting package of choice of the Space Telescope Science Institute, which financially supported matplotlib’s development and led to greatly expanded capabilities.

One of matplotlib’s most important features is its ability to play well with many operating systems and graphics backends. John Hunter highlighted this fact in a keynote talk he gave last summer, shortly before his sudden and tragic passing (video link): “We didn’t try to be the best in the beginning, we just tried to be there...” and fill-in the features as needed. In this talk, John outlined the reasons he thinks matplotlib succeeded in outlasting the dozens of competing packages:

it could be used on any operating system via its array of backends
it had a familiar interface: one similar to MatLab
it had a coherent vision: to do 2D graphics, and do them well
it found early institutional support, from astronomers at STScI and JPL
it had an outspoken advocate in Hunter himself, who enthusiastically promoted the project within the Python world

Matplotlib’s Challenge

This cross-platform, everything-to-everyone approach has been one of the great strengths of matplotlib. It has led to a large user-base, which in turn has led to an active developer base and matplotlib’s powerful tools and ubiquity within the scientific Python world. But as the world of graphics visualization has changed, this strength has started to become matplotlib’s main weakness. The hooks into multiple backends work well for static images, but can be cumbersome and unpredictable for more dynamic, interactive plots (for example, the animation toolkit still does not work with the MacOSX backend, and this is unlikely to improve any time soon).

Contrast this with one of the other great success stories in the scientific Python world, the IPython Notebook. It brings powerful, interactive computing tools to the fingertips of the user, and works well across all platforms. The IPython team, rather than spending time creating backend hooks for all possible graphics toolkits, built the notebook to work in the browser, effectively outsourcing the cross-platform compatibility problem to browser providers. This decision, I believe, is a fundamental piece which enabled IPython notebook to become so widely adopted during its short existence. The developers have been freed to spend their time implementing features, rather than struggling with cross-platform compatibility. Drawing from this lesson, I would venture to predict that whichever graphics package is the community standard five years from now will have adopted this approach as well.

The matplotlib developers, of course, know this very well. In his SciPy 2012 keynote mentioned above, John Hunter talked about lessons learned from ten years of growing matplotlib, the challenges matplotlib faces, and the path toward the future. Prominently mentioned among the challenges was the fact that users have come to expect dynamic, interactive, client-side graphics rendering seen in popular tools like Protovis and D3. Further, they’ve come to value and desire graphical tools which fit naturally in the research flow enabled by the IPython notebook (just look at the spontaneous applause Fernando received when showing off the IPython notebook/D3 integration in his PyCon Canada keynote

I would add one more challenge to that: even simple, static 2D visualization concepts have come a long way since gnuplot and the advent of matplotlib: in particular, The Grammar of Graphics has helped evolve views on best practices for exploratory data visualization. The ggplot integraion in R is one feature that users cite as a clear advantage over Python. Yes, matplotlib is powerful enough to allow implemention of some of these ideas, but its plotting commands remain rather verbose, and its no-frills, default output looks much more like Excel circa 1993 than ggplot circa 2013. (for a quick look at Grammar of Graphics in a Python context, see Peter Wang’s Bokeh talk from Scipy 2012).

The Future of Visualization in Python

Looking toward the future, these are significant challenges for matplotlib. I have no doubt that with a healthy effort from the development community, along with a good dose of vision and leadership, matplotlib could adapt and remain the leading visualization package in Python. But there are challengers: NodeBox OpenGL solves some of the interactivity problems by providing a powerful interface on a single, universally available graphics backend. Packages like Chaco and MayaVi push the boundaries in interaction, extensibility, and 3D capabilities. But all three of these options are still married to the old server-side paradigm of tools like matplotlib and gnuplot rather than the client-side paradigm of tools like IPython notebook, Protovis, and D3.

A more exciting option right now, in my view, is Bokeh, a project of Peter Wang, Hugo Shi, and others at Continuum Analytics. Bokeh is an effort to create a ggplot-inspired graphics package in Python which can produce beautiful, dynamic data visualizations in the web browser. Though Bokeh is young and still missing a lot of features, I think it’s well-poised to address the challenges mentioned above. In particular, it’s explicitly built around the ideas of Grammar of Graphics. It is being designed toward a client-side, in-browser javascript backend to enable the sharing of interactive graphics, a la D3 and Protovis. And comparing to matplotlib’s success story, Bokeh displays many parallels:

Just as matplotlib achieves cross-platform ubiquity using the old model of multiple backends, Bokeh achieves cross-platform ubiquity through IPython’s new model of in-browser, client-side rendering.
Just as matplotlib uses a syntax familiar to MatLab users, Bokeh uses a syntax familiar to R/ggplot users
Just as matplotlib had a coherent vision of focusing on 2D cross-platform graphics, Bokeh has a coherent vision of building a ggplot-inspired, in-browser interactive visualization tool
Just as matplotlib found institutional support from STScI and JPL, Bokeh has institutional support from Continuum Analytics and the recent $3 million DARPA XDATA grant.
Just as matplotlib had John Hunter’s vision and enthusiastic advocacy, Bokeh has the same from Peter Wang. Anyone who has met Peter will know that once you get him talking about projects he’s excited about, it’s hard to make him stop!

Above all that, Bokeh, like matplotlib, is entirely open-sourced. Now, I should make clear that Bokeh still has a long way to go. Its installation instructions & examples are still a bit incomplete and opaque. It currently provides no way of outputting PNG or PDF versions of the graphics it produces. Many of its goals still lie more firmly in the realm of vision than in the realm of implementation. But for the reasons I gave above, I think it’s a project to keep watching.

And where does that leave matplotlib? I would not, by any means, discount it just yet. Still, as John Hunter noted last summer, it faces some significant challenges, particularly in the area of client-rendered, dynamic visualizations. Any core matplotlib developers reading this should go back and re-watch John’s SciPy keynote: it was his last public outline of his vision for the project he started and led over the course of a decade. An IPython notebook-compatible client-side matplotlib viewer along the lines of the ideas John mentioned at the end of his talk would be the killer app that would, in all likelihood, allow matlotlib to maintain its position as the de facto standard visualization package for the Scientific Python community.

And all that being said, regardless of what the future brings, you can be assured that in the meantime I and many others will still be doing all our daily work and research using matplotlib. Despite its weaknesses and the challenges it faces, matplotlib is a powerful tool, and I don’t anticipate it withering away any time soon.

Animating the Lorenz System in 3D

2013-02-16T08:05:00-08:00

One of the things I really enjoy about Python is how easy it makes it to solve interesting problems and visualize those solutions in a compelling way. I've done several posts on creating animations using matplotlib's relatively new animation toolkit: (some examples are a chaotic double pendulum, the collisions of particles in a box, the time-evolution of a quantum-mechanical wavefunction, and even a scene from the classic video game, Super Mario Bros.).

Recently, a reader commented asking whether I might do a 3D animation example. Matplotlib has a decent 3D toolkit called mplot3D, and though I haven't previously seen it used in conjunction with the animation tools, there's nothing fundamental that prevents it.

At the commenter's suggestion, I decided to try this out with a simple example of a chaotic system: the Lorenz equations.

Solving the Lorenz System

The Lorenz Equations are a system of three coupled, first-order, nonlinear differential equations which describe the trajectory of a particle through time. The system was originally derived by Lorenz as a model of atmospheric convection, but the deceptive simplicity of the equations have made them an often-used example in fields beyond atmospheric physics.

The equations describe the evolution of the spatial variables $x$, $y$, and $z$, given the governing parameters $\sigma$, $\beta$, and $\rho$, through the specification of the time-derivatives of the spatial variables:

${\rm d}x/{\rm d}t = \sigma(y - x)$

${\rm d}y/{\rm d}t = x(\rho - z) - y$

${\rm d}z/{\rm d}t = xy - \beta z$

The resulting dynamics are entirely deterministic giving a starting point $(x_0, y_0, z_0)$ and a time interval $t$. Though it looks straightforward, for certain choices of the parameters $(\sigma, \rho, \beta)$, the trajectories become chaotic, and the resulting trajectories display some surprising properties.

Though no general analytic solution exists for this system, the solutions can be computed numerically. Python makes this sort of problem very easy to solve: one can simply use Scipy's interface to ODEPACK, an optimized Fortran package for solving ordinary differential equations. Here's how the problem can be set up:

import numpy as np
from scipy import integrate

# Note: t0 is required for the odeint function, though it's not used here.
def lorentz_deriv((x, y, z), t0, sigma=10., beta=8./3, rho=28.0):
    """Compute the time-derivative of a Lorenz system."""
    return [sigma * (y - x), x * (rho - z) - y, x * y - beta * z]

x0 = [1, 1, 1]  # starting vector
t = np.linspace(0, 3, 1000)  # one thousand time steps
x_t = integrate.odeint(lorentz_deriv, x0, t)

That's all there is to it!

Visualizing the results

Now that we've computed these results, we can use matplotlib's animation and 3D plotting toolkits to visualize the trajectories of several particles. Because I've described the animation tools in-depth in a previous post, I will skip that discussion here and jump straight into the code:

Lorenz System lorentz_animation.py download

import numpy as np
from scipy import integrate

from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.colors import cnames
from matplotlib import animation

N_trajectories = 20


def lorentz_deriv((x, y, z), t0, sigma=10., beta=8./3, rho=28.0):
    """Compute the time-derivative of a Lorentz system."""
    return [sigma * (y - x), x * (rho - z) - y, x * y - beta * z]


# Choose random starting points, uniformly distributed from -15 to 15
np.random.seed(1)
x0 = -15 + 30 * np.random.random((N_trajectories, 3))

# Solve for the trajectories
t = np.linspace(0, 4, 1000)
x_t = np.asarray([integrate.odeint(lorentz_deriv, x0i, t)
                  for x0i in x0])

# Set up figure & 3D axis for animation
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1], projection='3d')
ax.axis('off')

# choose a different color for each trajectory
colors = plt.cm.jet(np.linspace(0, 1, N_trajectories))

# set up lines and points
lines = sum([ax.plot([], [], [], '-', c=c)
             for c in colors], [])
pts = sum([ax.plot([], [], [], 'o', c=c)
           for c in colors], [])

# prepare the axes limits
ax.set_xlim((-25, 25))
ax.set_ylim((-35, 35))
ax.set_zlim((5, 55))

# set point-of-view: specified by (altitude degrees, azimuth degrees)
ax.view_init(30, 0)

# initialization function: plot the background of each frame
def init():
    for line, pt in zip(lines, pts):
        line.set_data([], [])
        line.set_3d_properties([])

        pt.set_data([], [])
        pt.set_3d_properties([])
    return lines + pts

# animation function.  This will be called sequentially with the frame number
def animate(i):
    # we'll step two time-steps per frame.  This leads to nice results.
    i = (2 * i) % x_t.shape[1]

    for line, pt, xi in zip(lines, pts, x_t):
        x, y, z = xi[:i].T
        line.set_data(x, y)
        line.set_3d_properties(z)

        pt.set_data(x[-1:], y[-1:])
        pt.set_3d_properties(z[-1:])

    ax.view_init(30, 0.3 * i)
    fig.canvas.draw()
    return lines + pts

# instantiate the animator.
anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=500, interval=30, blit=True)

# Save as mp4. This requires mplayer or ffmpeg to be installed
#anim.save('lorentz_attractor.mp4', fps=15, extra_args=['-vcodec', 'libx264'])

plt.show()

The resulting animation looks something like this:

Notice that there are two locations in the space that seem to draw-in all paths: these are the so-called "Lorenz attractors", and have some interesting properties which you can read about elsewhere. The qualitative characteristics of these Lorenz attractors vary in somewhat surprising ways as the parameters $(\sigma, \rho, \beta)$ are changed. If you are so inclined, you may wish to download the above code and play with these values to see what the results look like.

I hope that this brief exercise has shown you the power and flexibility of Python for understanding and visualizing a large array of problems, and perhaps given you the inspiration to explore similar problems.

Happy coding!

Setting Up a Mac for Python Development

2013-02-02T11:01:00-08:00

A few weeks ago, after years of using Linux exclusively for all my computing, I started a research fellowship in a new department and found a brand new Macbook Pro on my desk. Naturally, my first instinct was to set up the system for efficient Python development. In order to help others who might find themself in a similar situation, I took some notes on the process, and I'll summarize what I learned below.

First, a disclaimer: I can't promise that these suggestions are the best or most effective way to proceed. I'm by no stretch of the imagination a Mac expert: the last Apple product I used regularly was the trusty Macintosh Classic my parents bought when I was in middle school. I primarily used it for all-day marathons of RoboSport and Civilization, with occasional breaks to teach myself programming in Hypercard. But I digress.

Before moving on to the summary of what I learned, I should note that all of the following was done on OSX 10.8: there will likely be differences between OSX versions. I've done my best to note all the relevant details, and I hope you will find this helpful!

Accessing the Terminal

Being from a Linux background, I was interested in setting up a Linux-like work environment, doing nearly everything from the terminal. Fortunately, OSX is built on unix, with a terminal integrated into the operating system.

To open a terminal, open the finder, click "Applications" and search for "Terminal". To make it easier to access in the future, I dragged the icon down to the dock, the collection of icons usually found on the bottom or side of the screen. Clicking the icon will open a familiar bash terminal that can be used to explore your Mac in a more user-friendly way.

Setting Up MacPorts

To set up the rest of the system components, I opted to use MacPorts, which is a package management system similar to apt or yum on ubuntu and debian systems. There are probably alternatives to MacPorts, but I found it very intuitive, quick, and powerful.

You can download MacPorts for free on the MacPorts website. You'll have to install it for the correct OSX version -- to check which OSX version you're running, click "about this computer" under the apple icon at the top-left of the desktop. Unfortunately, I was following the slightly outdated MacPorts guide and got a few errors:

Error: No Xcode installation was found.
Error: Please install Xcode and/or run xcode-select to specify its location.
Error:
Warning: xcodebuild exists but failed to execute
Warning: Xcode does not appear to be installed; most ports will likely fail to build.

This indicates that Xcode is not yet installed. Xcode is a a collection of developer tools for the Mac, and it can be freely downloaded at the Apple App store. You'll need to create an Apple account to access it, and then make sure you have a fast internet connection: the download is about 1.6GB. Once it's downloaded, find the XCode icon in the Applications menu and click to install.

With this done, I tried installing MacPorts again, but still got an error:

Error: Unable to open port: can't read "build.cmd": Failed to locate 'make' in path: '/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin' or at its MacPorts configuration time location, did you move it?

This indicates that the command-line tools are not installed by default. To fix this, run Xcode, select Xcode->preferences from the menu bar, click downloads, select "command-line tools", and click install. You'll also need Xorg tools, which I installed through XQuartz.

Installing Python

Now that MacPorts is installed, it's very straightforward to install several versions of Python and other programs. MacPorts allows access to a standard repository of programs and packages, which can be explored, downloaded, and installed using the port command.

First of all, run

sudo port selfupdate

which updates the MacPorts base to the latest release. Another useful thing to know about is the MacPorts search command. For example, to see all the available packages which mention "python", use

port search python

This will list all the python versions available. I installed both Python 2.7 and Python 3.3

sudo port install python27
sudo port install python33

If you want python on the command-line to point to a particular version, this can be specified with the select command:

sudo port select --set python python27

You can now check that typing python --version in the terminal returns version 2.7.

Installing Numpy, Scipy, etc.

My scientific development in Python relies on several packages: particularly numpy, scipy, matplotlib, ipython, cython, scikits-learn, virtualenv, nose, pep8, and pip. Here is the series of commands to set these up for Python 2.7:

sudo port install py27-numpy
sudo port install py27-scipy
sudo port install py27-matplotlib

sudo port install py27-tornado
sudo port install py27-zmq  # zmq & tornado needed for notebook/parallel
sudo port install py27-ipython
sudo port select --set ipython ipython27

sudo port install py27-cython
sudo port select --set python cython27

sudo port install py27-scikits-learn

sudo port install py27-virtualenv
sudo port install virtualenv_select
sudo port select --set virtualenv virtualenv27

sudo port install py27-nose-testconfig
sudo port select --set nosetests nosetests27

sudo port install py27-pep8
sudo port install pep8_select
sudo port select --set pep8 pep827

sudo port install py27-pip

Unfortunately, there seems to not yet be a port select command for pip. This bug has been reported and is noted here.

Setting up a Virtual Environment

For Python development, I find it vital to make a good use of virtual environments. Virtual environments, enabled by the virtualenv package, allow you to install several different versions of various python packages, such that the installations are mostly independent. I generally keep stable released versions of packages in the system-wide python install, and use these environments to develop the packages. That way, I can test the compilation/installation of a new feature in scipy or scikit-learn without breaking my tried-and-true system installation.

Here we'll set up a virtual environment called default in a PyEnv subdirectory, and then install numpy in that environment using pip.

mkdir ~/PyEnv
cd ~/PyEnv
virtualenv default
source default/bin/activate
pip install numpy

Other Programs to Install

There are several other things I found helpful to install. First, the g95 Fortran compiler for building scipy and other packages which require fortran:

sudo port install g95

Also, we'll install and configure the tool every open source developer needs:

sudo port install git
git config --global user.name "John Doe"
git config --global user.email john@doe.com

Another essential is a good text editor. There are several good open source options for this.

Textmate is a Mac native text editor which has many nice features, works nicely on mac, and is fairly clean and nice to use:

sudo port install textmate2
mate tmp.txt

Vim is another popular text editor: there is both a command-line version and a GUI version available:

sudo port install vim
sudo port install MacVim
vim tmp.txt

Emacs is my text editor of choice, and like Vim there is both a command-line version and a GUI version:

sudo port install emacs
sudo port install emacs-app
emacs tmp.txt

There are several other GUI emacs versions available as well (e.g. xemacs and emacs-mac-app): I found that I liked emacs-app the best. Unfortunately, it lives in the "Applications" folder, and there doesn't seem to be a way to configure the emacs GUI to work the same way as the default emacs behavior on linux (see the related discussion thread here). I ended up putting the emacs GUI in the dock next to the terminal, and I access it from there.

Final Thoughts

I've had this setup on my new Macbook for about two weeks now, and it seems to be working well for my daily python programming tasks. I hope that this post will be useful to someone out there. I'm still learning as well -- if you have any pro tips for me or for other readers, feel free to leave them in the comments below!

Happy hacking.