Pixel Ping: A node.js Stats Tracker

Since the day we launched, ProPublica has encouraged people to republish our stories for free. We even license our stories under Creative Commons (CC). However, in the past we've had trouble knowing precisely which stories had been republished where, and we had no way of knowing how many people were reading our stories on sites that republished them under our CC license.

Shortly after the redesign of our site, we started working on a system that would help us solve this problem. When we found out that Jeremy Ashkenas, a developer at DocumentCloud, was working on a similar problem, we joined forces, and finished work on a lightweight stats tracker, which we are open sourcing today.

World, meet Pixel Ping.

Pixel Ping is a minimalist pixel tracker -- written in CoffeeScript and compiled into JavaScript -- that runs on top of a tremendously fast server-side JavaScript framework called node.js. Pixel Ping tracks each request for a small 1x1 image and periodically flushes aggregated totals of those requests to an endpoint (in our case, a MySQL database).

Pixel Ping is not a particularly new idea, but other solutions (Google Analytics, Omniture, etc) are designed for a different environment and purpose, and can require a fair amount of resources up front. Also, one of the requirements in our use case is that our pixel tracker count only what’s necessary, and not create any privacy worries on the part of sites that republish our material. So our internal implementation of Pixel Ping just tracks page impressions. It doesn’t know anything about individual users, and it doesn’t set or read any cookies.

Pixel Ping is damn fast: an ApacheBench test averaged 3ms per request for 1,000 requests (yadda-yadda, local requests, grain of salt, etc.), and it also only uses around 8 megabytes of memory while it is running.

Along with Pixel Ping we're open sourcing a simple backend for Pixel Ping, written in Ruby on Rails, called Pixel Pong, which we use at ProPublica to provide a reporting interface on the collected data.

The Setup

Pixel Ping Only

Before you install Pixel Ping you'll need node.js, and npm. Once you've installed both, you can run:

npm install pixel-ping

and then run:

pixel-ping path/to/config.json

where config.json looks something like this:

{
    "host": "127.0.0.1",
    "port": "9187",
    "interval": 600,
    "endpoint": "https://www.example.com/save_hits"
  }

You can test that it works by going to

https://127.0.0.1:9187/pixel.gif?key=test

in your browser.

Pixel Ping with Pixel Pong

The easiest way to get up and running with Pixel Ping is to install Pixel Pong with

git clone https://github.com/propublica/pixel-pong.git

then install DocumentCloud's jammit with:

cd pixel-pong; rake gems:install

You'll also need a database.yml in the config directory then run

rake db:create; rake db:migrate

Then install Pixel Ping with:

rake pixel:install

and then add a file in the config directory called config/pixel-ping.development.json with the following:

{
    "port": "9187",
    "host": "127.0.0.1",
    "interval": 5,
    "endpoint": "https://127.0.0.1:3000/stats.json",
    "secret": "one very good secret"
  }

Once you're all set you can start the Rails app with:

script/server

then in a new shell go to the Pixel Pong directory and start Pixel Ping with:

rake pixel:start

Now if you go to the Pixel Ping tracking gif url:

https://127.0.0.1:9187/pixel.gif?key=test☃https://www.example.com

a couple of times you'll see hits logged every second in the reporting interface.

Pixel Pong accepts flushes from Pixel Ping that are delimited by a ☃, in this case the title would be "test" and the url would be "https://www.example.com".

If you point your web browser to:

https://127.0.0.1:9187/pixel.js

You'll see a small bit of javascript that you can include on your own pages. It will insert the pixel and form the request properly so you won't have to find the ☃ on your keyboard.

Both DocumentCloud and ProPublica have been running Pixel Ping instances in production for over a month without problems, but as with any open source project we expect there will be bugs. If you find any you can use the issue trackers (Pixel Ping and Pixel Pong) or send an email to [email protected].

Follow ProPublica