Mathieu Leplatre

Serve your map layers with a usual Web hosting service

2012-05-05T00:00:00+02:00

Someone asked me about serving map tiles from a basic Web host. I agreed to reply with a blog post, since it completes the stories I've been telling in my last two talks.

How to serve your map layers (tiles) with the simplest Apache or nginx ?

Get the MBTiles file

We start from a tiles package (MBTiles). Depending on where your layers come from, you can either choose :

To publish your Tilemill map

Design your map in Tilemill, even your own OpenStreetMap style customization, and export it as MBTiles ! MapBox hosting is the prefered solution, but you can still host and serve your exported MBTiles file yourself !

To mirror a WMS server

Again, using landez, you can build a MBTiles file from a WMS source (orthophoto...), and then serve those layers yourself as tiles (at the speed of light !).

To mirror tiles services

Using landez, you can gather tiles from a tiles service, and package them in a .mbtiles file locally.

If you plan on mirroring public tile services, do not forget to add attributions and make sure it respects the terms of service. Most services restrict bulk downloads [1].

Extract files on disk

Using mbutil, we can extract the .mbtiles file into a destination folder.

Unfortunately, the pypi mirror is quite old, we'll install the last development version.

wget https://github.com/mapbox/mbutil/zipball/master -O mbutil.zip
unzip mbutil
cd mapbox-mbutil*
sudo python setup.py install

Done. Now extract. (Note that the ``DEST`` folder must not exist) :

mb-util --scheme=osm FILE.mbtiles /path/to/DEST/

If your MBTiles has an interaction layer (UTFGrid), both .png and .json files will be expanded in folders.

Just push the folder to your hosting, and you're done !

Cache headers

If you the master on board, tweak the cache headers :

With Apache :

ExpiresActive On
ExpiresDefault "access plus 7 days"
Alias /DEST /path/to/DEST/;

With nginx :

server {
    location /DEST {
        expires 7d;
        alias /path/to/DEST/;
    }
}

Boost with subdomains

Browsers limit parallel downloads on the same domain. If you can declare subdomains (a.yourserver.org, b.yourserver.org, ...), it will speed-up tiles download.

Use it in your mapping library

With Leaflet for example :

var map = new L.Map('map');
map.addLayer(new L.TileLayer('http://{s}.yourserver.org/DEST/{z}/{x}/{y}.png'));
map.setView(new L.LatLng(43.60, 1.45), 14)

Or Modestmaps :

var provider = new MM.TemplatedLayer('http://{s}.yourserver.org/DEST/{z}/{x}/{y}.png');
var map = new MM.Map('map', provider);
map.setCenter({lat: 43.60, lon: 1.45}).setZoom(14);

It will also work with interaction layers if you use Wax :)

[1]	MapBox strictly forbids proxying and further distribution of their tiles. Bulk downloading from OpenStreetMap.org is strongly discouraged too. And mass downloads cost money on Cloudmade.

Simple and funky Web map printing

2012-05-02T00:00:00+02:00

Strangely, users still insist in having Web page print capabilities, mostly to share, export or archive what they see. Even if relevant permalinks are often acceptable, we can't always dissuade them from printing :)

And when it comes to Web maps, printing can be a nightmare ! Even though most of the time, the needs involve a nice landscape PDF file with the map, a legend and the company logo. For this, designing CSS print stylesheets and using the browser print shortcut is not always helpful.

I present here a simple solution [1] based on headless Webkit screenshots, permalinks, SVG templates (WYSIWYG), and PDF conversion.

What You See Is What You Print

In the following (short) video, we can see :

a map with dynamic content (GeoJSON), bound to a form for attribute filtering and a legend refreshed upon data ranges modifications ;
a Print button that delivers the current view as PDF ;
a landscape printout in which the map view, the legend, the filter form values were nicely inserted.

Your browser does not support the video tag.

(BTW, small boo-boo in last screen: 'montain' instead of 'mountain')

Kids, you can do this at home

Here is how we did it :

a Web page with a "stateful" permalink (i.e. restore the map and page state using anchors, location hash, etc. ;). Backbone.js & co. are meant for this ;
A color scale built client-side from the resulting dataset, using Chroma.js, to colorize the map items and populate the legend entries ;
django-screamshot, a Web page screenshot application, relying on CasperJS. Spooky! ;
a SVG landscape A4 document, edited with Inkscape, as a Django template in which we placed simple tags ({{ filter.age_min }}) for texts, the {% base64capture %} tag for the map screenshot, and a couple of arithmetics tags to recreate a nice vectorial legend using the color scale entries;
a Django view that receives the current page context (posted in JS), renders the SVG (will thus perform the screenshot), and converts it to PDF ;

Quite straightforward, a couple of hours to put together, relatively easy to deploy, obviously meet most users needs... these hacks are our happiness !

If you want to know more about some missing parts, feel free to ask ! I could release stuff or just post some snippets...

[1]	from now on, I shall precise : even if it can cover most needs, it won't be adequate in all situations.

Des cartes d'un autre monde, la suite

2012-04-18T00:00:00+02:00

Bravo aux organisateurs !

L'édition 2012 des rencontres Django-fr a mis la barre très haut ! Ce fut un plaisir de retrouver, ou rencontrer, autant de gens sympathiques et intéressants dans un cadre aussi agréable :)

Comme l'a souligné Olivier [1], cela va bien au delà de la techno qui fédère la communauté ; les membres partagent aussi un esprit, une vision, des approches, qui transpassent l'outil ! Agilité, pragmatisme, KISS, DRY, PEP20...

Au menu, les problématiques de scaling étaient prédominantes, Django propulse des sites à gros volume, comme Liberation.fr, 20minutes.fr, Mozilla, Autolib, représentés pendant ces rencontres, mais aussi Instagram, Lanyrd, Disqus... autant d'expériences à partager! Ce fut aussi un honneur d'accueillir deux invités nord-américains, python Lords chez Heroku.

Cette année, je présentais une approche à contre-courant pour publier des données cartographiques sur le Web :

“ Comment publier des données cartographiques, aussi simplement qu'on publie une image ? Il existe un monde où Django expose lui-même des cartes interactives, sans déployer l'artillerie habituelle ! Une approche à contre-courant se prêtant parfaitement à la mise en valeur quasi-immédiate d'informations geographiques, comme celles libérées par votre ville ! ”

Pour résumer : éloignons-nous de l'OGC (WMS, WFS,...) quand il s'agit juste de publier une carte sur une page Web.

Comment rafraîchir les cartes ?

Je reprends une citation remontée par twidi :

“ There are only two hard things in Computer Science: cache invalidation and naming things ” -- Phil Karlton

Selon la fréquence de rafraichissement de vos données, plusieurs stratégies sont envisageables. Surtout qu'il est rare que l'ensemble de la carte nécessite d'être actualisé.

En générant les tuiles en temps réel

Comme l'expliquait Young Yahn [2], générer des tuiles en temps réel demande d'avoir une architecture assez trappue et cela s'avère stressant à administrer.

Cependant c'est possible avec des outils comme tilelive ou renderd (Apache mod_tile).

Avec une tâche planifiée

Il faut trouver le compromis entre le temps de fabrication de la carte et la fréquence de rafraichissement des données source.

S'il s'agit de minutes, rafraichir la carte toutes les heures semble envisageable. La plupart du temps, une fois par jour suffira.

Il existe plusieurs outils, comme landez, pour regénérer à intervales réguliers votre carte issue de Tilemill.

À la seconde

Pour suivre une flotte de bateaux ou de véhicules, il faut que les éléments soient déplacés en temps réel sur la carte.

Il y a plusieurs outils très efficaces qui se reposent sur les Websockets. J'avais fait l'application Livetitude, disponible en ligne, qui permet d'éditer à plusieurs une carte de marqueurs, grace à Pusher.

La bibliothèque Sharejs, issue du projet défunt Google Wave, permettrait d'aller plus loin en faisant de l'édition collaborative d'attributs de géométries.

En fonction de filtres ou formulaires

Pour redessiner la carte en fonction de filtres, sur des attributs par exemple, l'utilisation du format GeoJSON s'avère assez efficace.

Votre serveur reçoit le formulaire, construit le jeu de données, et renvoie les résultats (Features = geometries + attributs).

Cette approche peut s'avérer délicate selon la taille des jeux de données. Plusieurs ruses existent afin de limiter le volume (ex: généralisation progressive selon la zone affichée)

Carte à échelle unique

Parfois, pour certaines cartes, une seule vue suffit ! Nul besoin de zoomer, puisque le phénomène intervient à une échelle en particulier !

Pensez aux cartes des journaux ! Et vous serez séduits par l'excellent Kartograph, qui permet de publier des cartes sublimes facilement. Le SVG est manipulable en Javascript, et permet d'ajouter des évènements sur les zones.

Martin Dewulf a publié une jolie carte interactive à partir de données ouvertes. Le résultat est très convaincaint, et sort de l'ordinaire.

Requiem pour les trolls

La citation au début de la présentation, issue de #whereconf, était volontairement provocatrice. Mais de nombreux acteurs du Web et de la cartographie rejoignent cette idée. Par exemple, entre temps, Sean Gillies a réitéré :

“ How many MapBox and CartoDB like products would there be today if the Open Source GIS community hadn't gone on a decade long WxS wander? “ -- @sgillies, 2012

Orienté communication

En 12 minutes, c'est très difficile de présenter tous les aspects, inconvénients et avantages d'une approche à contre-courant !

J'ai présenté le besoin plus simple de la cartographie sur le Web :

j'ai des données à caractère géographique ;
je veux les afficher sur une page Web avec une carte interactive.

Le cas le plus simple, mais en même temps le plus répandu !

Pas toujours d'alternatives à l'OGC

Dans certains contextes, les protocoles OGC sont indispensables :

interroperabilité entre systèmes hétérogènes sans médiation préalable (serveurs externes, logiciels propriétaires, etc.);
catalogage sémantique et syndication des jeux de données (INSPIRE)
construction d'une plateforme IDS

OpenLayers est la seule bibliothèque javascript qui a les reins assez solides pour s'intégrer dans ces environnements.

Savoir oublier le Web

Il faut savoir juger la pertinence d'une application Web. Dans certaines situations, le Web n'est pas la seule solution pour faire du client-server en multi-utilisateurs.

L'année dernière par exemple, nous avons développé une application collaborative pour manipuler des tronçons routiers. Nous avons choisi C++/Qt, avec PostGIS et des webservices JSON, parce que c'est ce qui se prêtait le mieux au besoin ! La même chose en ExtJS aurait été catastrophique !

[1]

oloynet

[2]	Rendering the World, FOSS4G NA, 2012

Render your TileMill stylesheets with Landez

2012-03-22T00:00:00+01:00

TileMill is an amazing tool to design your map, and publish it. With landez, you can easily render it using python, or do whatever comes with the API !

From TileMill to Landez

Use Tilemill to design your map, and export the Mapnik XML stylesheet :

Then simply use landez with stylefile argument :

import logging
from landez import MBTilesBuilder

logging.basicConfig(level=logging.DEBUG)

mb = MBTilesBuilder(stylefile="Toulouse-Voirie.xml",
                    filepath="toulouse-voirie.mbtiles")
mb.add_coverage(bbox=[1.39, 43.56, 1.50, 43.64],
                zoomlevels=range(10, 18))
mb.run()

In the above example, a MBTiles file toulouse-voirie.mbtiles will be created with all rendered tiles. (Note: This won't render UTF-Grid tiles, since TileMill does not expose this part in the XML stylesheet.)

If you don't have Mapnik2 installed, you might encounter rendering errors like : AssertionError: Cannot render tiles without mapnik !.

Installation of Mapnik 2

Mapnik2 packages on Debian/Ubuntu

In Ubuntu Precise (12.04) or Debian Wheezy (7.0), it's a piece of cake, the package is available in the repos

sudo apt-get install python-mapnik2

In Ubuntu Maverick (10.10), Natty (11.04), Oneiric (11.10), it's quite easy, there is a PPA, from MapBox

sudo apt-add-repository ppa:developmentseed/mapbox
sudo update
sudo apt-get install python-mapnik2

Mapnik2 and python bindings from sources

Welcome in the quicksands of installing Mapnik2 python bindings from sources !

Christian Spanring wrote a quick tutorial to install it from sources on Ubuntu 10.04.

It might be a bit tricky to tweak this tutorial for your distribution. Hopefully, our colleague Mathieu has prepared a minitage's "minilay" for it, just follow the few steps to compile the whole stack.

Announcing Subtivals, realtime subtitles for film festivals

2012-03-17T00:00:00+01:00

That's how it started...

Subtitles and captions for the deaf and hard-of-hearing during film festivals are rarely a priority, at least in France. Thus, most film copies are not subtitled (except foreign movies shown in their original version).

At the beginning of last year, my friend Lilian was promoting his project of improving and facilitating the projection of subtitles during film festivals. His activity consists in superimposing subtitles or projecting them on a separate display below the screen.

In this way, subtitling dissociates from the movie reel, unlike DVDs for example. An operator is thence in charge of keeping captions synchroneous. It is sometimes called "virtual subtitling"; it is cheaper and easier than subtitled hard copies.

Since there was no suitable Open Source tools to fulfill these precise needs, Arnaud and I gave him a hand. We developed Subtivals, a Free Software with simplicity and usuability for the technical operator in mind.

Main features

Subtivals has gained many features on the way, we released the version 1.0 last month, after almost a year of development, driven by Lilian's experience.

Among most notable features :

Support of Advanced SubStation Alpha subtitles (ASS, *.ass) format
Control Play / Pause / Delay / Speed
Switch between several modes : timecode based, semi-automatic or fully manual
Support for subtitles without timecodes (fixes duration automatically)
SSA styles (italic, positions, colors)
Customize and override styles (color and font size)

In the small world of subtitles projection, where most tools are either very expensive, or very archaic, Subtivals is a revolution !

Hall of fame

Subtivals has proven its efficiency for several months now ! It was used successfully in many film festivals (Festival de cinéma de Douarnenez, Festival Intergalactique de Brest, Festival de Cinéma d'Amérique Latine de Biarritz, Festival International du film d'Amiens, Festival Zoom Arrière, 6e édition, ... and soon : Cinélatino de Toulouse, Résistances à Foix)

Its semi-manual mode also allows surtitling at theaters (or opera, ballets, ...).

Installation

Subtivals runs on GNU/Linux, Windows, Mac OS X and has no other external dependencies.

On Ubuntu, install it easily using our PPA :

sudo add-apt-repository ppa:mathieu.leplatre/subtivals
sudo apt-get update && sudo apt-get install subtivals

Mac OS X and Windows installers are for sale, contact us !

Contribute

Subtivals is written in C++/Qt and released under the GNU General Public License . It is available in English, French, Spanish and Catalan.

If you feel like contributing, testing, translating... join us on Github !

Django Handlebars.js integration

2012-03-17T00:00:00+01:00

In order to write Handlebars.js templates in Django templates, I was gonna copy and paste for the second time Miguel Araujo's verbatim snippet. But since one of the Django weakness is the lack of reusable applications, I thought I would package one instead :)

The two existing django applications [1] that integrate Handlebars.js are somehow bloated, they both kind of compile or render javascript templates on server-side (sic).

Oppositely, my django-templatetag-handlebars is very simple, you write your Handlebars template inside your django template. Django will preserve nicely {} tags, but still render {% %} tags.

For example, with this in your template :

{% tplhandlebars "tpl-infos" %}
    {{total}} {% trans "result(s)." %}
    <p>{% trans "Min" %}: {{min}}</p>
    <p>{% trans "Max" %}: {{max}}</p>
{% endtplhandlebars %}

The following block with end-up in your page :

<script id="tpl-infos" type="text/x-handlebars-template">
    {{total}} result(s).
    <p>Min: {{min}}</p>
    <p>Max: {{max}}</p>
<script>

Render it, client-side, as usual using Handlebars API :

var properties = {
    total: 10,
    min: 4,
    max: 5
};

var template = Handlebars.compile($('#tpl-infos').html()),
    rendered = template(properties);

Your rendered string is ready, and waiting to be inserted in your DOM :)

10 result(s).
<p>Min: 4</p>
<p>Max: 5</p>

Download and more info.

[1]	Both named django-handlebars, by Sergii Iavorskyi and by Chris Vigelius.

landez : introducing new features of our tiles toolbox

2012-02-24T12:45:00+01:00

landez started as a very small toolbox to build MBTiles files specifying bounding boxes and zoom levels. We have been using it for several GIS projects at Makina Corpus, and can tell it's reliable !

Landez is pure python and follows the KISS principle. It has optional requirements on PIL and mapnik for compositing, tile arranging or rendering.

Recently, we've added many extra cool features, which deserve highlight !

Simple WMS support

With landez, you can store your WMS layers into MBTiles files ! It will request the WMS images and save them into tiles on disk ! You can then enjoy the power of MBTiles files : transport, speed, ...

mb = MBTilesBuilder(wms_server="http://yourserver.com/geoserver/wms",
                    wms_layers=["ign:departements"],
                    wms_options=dict(format="image/png",
                                     transparent=True),
                    filepath="dest.mbtiles")
mb.add_coverage(bbox=([-0.9853,43.6435.1126,44.0639]),
                zoomlevels=range(18))
mb.run()

Tiles compositing

This is the killer feature ! With landez, you can now merge multiple sources of tiles (URL, WMS, MBTiles, Mapnik stylesheet) together !

For example, build a new MBTiles file by blending tiles of another on top of OpenStreetMap tiles :

mb = MBTilesBuilder(remote=True,
                    filepath="merged.mbtiles")
overlay = TilesManager(mbtiles_file="carto.mbtiles")
mb.add_layer(overlay)
mb.run()

Simply make a composite a WMS layer with OpenStreetMap using transparency ! You might find this useful for compositing satellite image with street maps :

mb = MBTilesBuilder(wms_server="http://yourserver.com/geoserver/wms",
                    wms_layers=["img:orthophoto"],
                    filepath="wms_osm.mbtiles")
overlay = TilesManager(remote=True)
mb.add_layer(overlay, 0.4)  # 40%
mb.run()

Arrange tiles into single images

This feature can be very useful for printing tiled maps or have a quick overview of your compositing results !

Refer to any source of tiles, like you would do with MBTilesBuilder, add layers if you like and export the image !

ie = ImageExporter(tiles_url='http://server/tile/{z}/{x}/{y}.png')
...
...
ie.export_image(bbox=(-180.0, -90.0, 180.0, 90.0),
                zoomlevel=3,
                imagepath="image.png")

MBTiles content reading

landez can now read MBTiles content !

We could proudly add it to the list of implementations for the MBTiles spec and UTF-Grid spec !

Use MBTiles files like any tile source :

mb = MBTilesBuilder(mbtiles_file="yourfile.mbtiles")

...extract single image or UTF-Grid tiles :

from landez.reader import MBTilesReader

mbreader = MBTilesReader("yourfile.mbtiles")

# Metadata
print mbreader.metadata()
# Zoom levels
print mbreader.zoomlevels()
# Image tile
with open('tile.png', 'wb') as out:
    out.write(reader.tile(z, x, y))
# UTF-Grid tile
print reader.grid(z, x, y, 'callback')

Next steps...

The code has grown quickly and deserves a good refactoring, which is being done in a separate branch on GitHub ! The goal is to keep the same simple API, better modularity, increase tests coverage...

If you are wiling to participate, feel welcome !

TileMill on your Web server behind a reverse proxy

2012-01-31T11:12:00+01:00

In the last version (0.9), TileMill has an additional dedicated process to serve the tiles. We had to change some bits of our server configuration.

Instead of documenting the new configuration in our internal Wiki, I prefered to share here a few technical lines (quite rough though).

We run TileMill inside a virtual machine on a server with reverse proxy rules.

Reverse Proxy configuration

Assuming your reach your TileMill virtual machine at tilemill.sillywalk.loc, with its two processes running (ports 20008 for tiles, 20009 for the application), your Apache reverse proxy configuration will be :

<VirtualHost *:80>
    ServerName tilemill.yourdomain.com

    ProxyPreserveHost On
    RewriteEngine on

    # Serve the tiles as /tiles/
    RewriteCond %{REQUEST_URI} ^(/tiles.*)$
    RewriteRule ^/tiles(.*) http://tilemill.sillywalk.loc:20008$1 [L,P]
    ProxyPassReverse /tiles http://tilemill.sillywalk.loc:20008/

    # Serve the application on /
    ProxyPass / http://tilemill.sillywalk.loc:20009/
    ProxyPassReverse / http://tilemill.sillywalk.loc:20009
</VirtualHost>

TileMill configuration

Now that tiles and application ports are reachable respectively on http://tilemill.yourdomain.com/tiles and http://tilemill.yourdomain.com/, just tell TileMill to serve its pages accordingly in its configuration, /etc/tilemill/tilemill.config :

{
  "files": "/usr/share/mapbox",
  "server": true,
  "coreUrl": "tilemill.yourdomain.com:80",
  "tileUrl": "tilemill.yourdomain.com:80/tiles",
  "port": 20009,
  "listenHost": "0.0.0.0"
}

Restart it...

sudo service tilemill restart

Done !

Merkopolo : a simple yet powerful starter kit for your Qt/C++ GIS application

2012-01-26T11:30:00+01:00

Recently, while the whole world looks completely hyped up with Web applications, we chose to design and develop a desktop software in Qt/C++.

Obviously, the choice was measured and justified ! We had to build a specific GIS application with complex interactions and huge amounts of data, for a limited number of users.

Quickly, we spotted Merkaartor, one of the official OpenStreetMap editors, for its UI components and object model. And since we started to code, we never regretted this choice !

C++ brings the power, Qt offers cross-platform and the compassion towards developers, and Merkaartor a lovely GIS flavour !

We contributed to Merkaartor to give gits components a little bit of genericity, and released Merkopolo, a Qt project skeleton to handle dependencies and inclusion of base components.

Here is what you immediately get once compiled :

Now you can start coding serious stuff on top, with the Merkaartor components stack :

Complete feature model with free attributes (tags)
Custom drawing styles
A variety of layers types (Tiles, WMS, Spatialite, GeoTIFF, GDAL...)
Base classes for mouse interactions on map objects
A projection system (libproj)
And even draw geometries from PostGIS database !

Merkopolo is available on Gitorious.

Access a JSON webservice with Qt C++

2011-12-16T17:00:00+01:00

Original post at Makina Corpus

Webservices are everywhere ! There are relevant in many situations, and accessing them from your Qt C++ application is not an heresy.

I will present here a very simple way to retrieve a JSON from a GET request.

HTTP Requests

Using QNetworkAccessManager is a piece of cake :

QNetworkAccessManager networkManager;

QUrl url("http://gdata.youtube.com/feeds/api/standardfeeds/most_popular?v=2&alt=json");
QNetworkRequest request;
request.setUrl(url);

QNetworkReply* currentReply = networkManager.get(request);  // GET

But, note that a slightly more generic approach would be to build the QUrl from a parameters list :

QUrl url("http://gdata.youtube.com/feeds/api/standardfeeds/");
QString method = "most_popular";
url.setPath(QString("%1%2").arg(url.path()).arg(method));

QMap<QString, QVariant> params;
params["alt"] = "json";
params["v"] = "2";

foreach(QString param, params.keys()) {
    url.addQueryItem(param, params[param].toString());
}

Parsing JSON

Get yourself a slot to parse the QNetworkReply :

connect(&networkManager, SIGNAL(finished(QNetworkReply*)), this, SLOT(onResult(QNetworkReply*)));

void YourClass::onResult(QNetworkReply* reply)
{
    if (m_currentReply->error() != QNetworkReply::NoError)
        return;  // ...only in a blog post

    QString data = (QString) reply->readAll();

    QScriptEngine engine;
    QScriptValue result = engine.evaluate(data);

    /*
      Google YouTube JSON looks like this :

      {
        "version": "1.0",
        "encoding": "UTF-8",
        "feed": {
          ..
          ..
          "entry": [{
            "title": {
                "$t": "Nickelback- When We Stand Together"
            },
            "content": {
                "type": "application/x-shockwave-flash",
                "src": "http://www.youtube.com/v/76vdvdll0Y?version=3&f=standard&app=youtube_gdata"
            },
            "yt$statistics": {
                "favoriteCount": "29182",
                "viewCount": "41513706"
            },
            ...
            ...
          },
          ...
          ...
          ]
        }
      }
    */

    // Now parse this JSON according to your needs !
    QScriptValue entries = result.property("feed").property("entry");
    QScriptValueIterator it(entries);
    while (it.hasNext()) {
        it.next();
        QScriptValue entry = it.value();

        QString link = entry.property("content").property("src").toString();
        int viewCount = entry.property("yt$statistics").property("viewCount").toInteger();

        // Do something with those...
    }
}

That's it :)

If you want more complexity, and don't mind adding extra-dependencies, check out Tomasz Siekierda's QtWebService !

Remove django form field validation errors manually

2011-12-06T09:00:00+01:00

Original post at Makina Corpus

Sometimes I look for something which seems so simple and stupid that I can't imagine it does not exist. It makes me wonder why and who is the fool. Worse, I can't be sure about my search keywords to prove me anything.

I just wanted to delete, reset or remove the validation errors of a single form field, within a django view, without overriding the form or field class.

A one-liner

aform.errors['afield'] = aform['afield'].error_class()

That's it folks !

This will not affect other fields errors or non-field errors ;
This will reuse nicely the field error class (ErrorDict or ErrorList) ;
You cannot set aform.errors['afield'] = None or your form full_clean() will be performed again !
Obviously, the ideal approach is to override your form clean() properly.

An equivalent of Django's select_related for ManyToMany and OneToMany relationships

2011-12-05T11:09:00+01:00

Original post at Makina Corpus

Using an ORM simplifies and reduces greatly the amount of code to interact with databases. Nevertheless, it can easily hide database design defects or become a source of serious performance issues.

A Common Pitfall

With Django, the most classic problem occurs while accessing objects relations attributes inside a loop. That's why QuerySet's method select_related() exists : it will join specified relations so that access to their attributes does not hit the database. Refer to Django's documentation for more information !

One-To-Many and Many-To-Many Relationships

select_related() is not able to follow One-To-Many (1-n) and Many-To-Many (n-n) relationships. The Django team is currently working on prefetch_related(). But before we can enjoy this future feature, we can implement an equivalent in python.

With these models :

class Pizza(models.Model):
    name = models.CharField(max_length=50)

class Restaurant(models.Model):
    name = models.CharField(max_length=50)
    pizzas = models.ManyToManyField(Pizza, through='PizzaRestaurant')

class PizzaRestaurant(models.Model):
    pizza = models.ForeignKey(Pizza)
    restaurant = models.ForeignKey(Restaurant)
    price = models.FloatField()

This loop will generate 1 + N queries :

for restaurant in Restaurant.objects.all():
    for pizza in restaurant.pizzas.all():
        print pizza.name

Whereas this one will only generate 2 queries :

# Store relationships in a dict
byrestaurant = {}
for pr in PizzaRestaurant.objects.select_related('restaurant', 'pizza').all():
    byrestaurant.setdefault(pr.restaurant.id, []).append(pr.pizza)
# Use stored lists
for restaurant in Restaurant.objects.all():
    for pizza in byrestaurant[restaurant.id]:
        print pizza.name

According to the amount of N, doing that trick in views can boost your pages !

This is not perfect and elegant, but if it allows you to downsize the number of queries from several thousands to fifteen, like it did on Memopol2, you can think twice.

Des cartes collaboratives avec Livetitude

2011-11-23T14:00:00+01:00

Article original publié chez Makina Corpus

Cela fait plusieurs semaines que je voulais présenter ma petite application de partage de cartes, dont je me sers déjà comme alternative aux marqueurs de Google Maps.

En bref

Livetitude est un outil Web pour créer des cartes de manière collaborative en temps réel (à la manière d'un pad).

Fonctionnant sur des terminaux mobiles, Livetitude permet également de partager la position des collaborateurs de la carte, d'exporter les données au format GeoJSON ou de publier la carte sur une page Web.

Sous le capot

J'ai pris du plaisir à hacker cette application, elle tire profit d'outils très simples mais très efficaces :

Leaflet pour afficher les cartes ;
Pusher (Websockets) pour la collaboration en temps réel ;
CouchDB pour stocker les données ;
Flask pour servir les pages ;
Heroku pour héberger l'application.

Bien entendu, le code source est libre et disponible sur le GitHub de Makina Corpus.

Utilisation

Une instance est déployée en ligne, dans le cloud d'Heroku, dont vous pouvez vous servir, pour une utilisation en bon père de famille :)

Les marqueurs peuvent contenir du texte ou de l'HTML, et aucune donnée de localisation des visiteurs n'est stockée.

Comment publier vos données existantes ?

Si vous souhaitez publier et visualiser vos marqueurs sur une carte de Livetitude, il suffit de poster (POST) les coordonnées de vos points sur l'URL http://server/<CARTE>/add.

Par exemple, avec une petite fonction python :

import httplib, urllib

SERVER = "server"  # e.g. vivid-warrior-6693.herokuapp.com

def publish(mapname, coords, data):
    url = "/%s/add" % mapname
    request = {
       'coords': coords,
       'data': data,
       'classid': 5  # (=color)
    }
    params = urllib.urlencode(request)
    headers = {"Content-type": "application/x-www-form-urlencoded",
               "Accept": "text/plain"}
    conn = httplib.HTTPConnection(SERVER)
    conn.request("POST", url, params, headers)
    response = conn.getresponse()
    print response.status, response.reason, response.read()
    conn.close()


publish("PasLoin", "10.1,54.9", "Super resto!")
publish("PasLoin", "19.4,65.1", "Bon mojito")

Les points sont alors visibles en ligne sur http://server/PasLoin ou disponible en GeoJSON sur http://server/PasLoin/points.

Contribuer

Livetitude est une application très simple, à l'état de preuve de concept. Mais le code source est très réduit et donc très rapide à prendre en main !

Toutes vos suggestions ou contributions sont les bienvenues !

Git : annuler proprement un commit après un push

2011-11-03T14:15:00+01:00

Ce qu'il faut éviter

Pour annuler des commits, il existe la commande git reset.

git reset --hard HEAD~1
HEAD is now at 444b1cf Rhoo

Celle-ci est pertinente tant que les commits n'ont pas été poussés. Git vous retiendra au push d'ailleurs :

git push
To /tmp/repo
 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to '/tmp/repo'

En effet, à partir du moment où un commit existe sur le serveur, il est potentiellement utilisé par des collaborateurs (mergé, à la base d'une branche, etc.). On pourrait faire le sale et forcer le push :

git push -f
Total 0 (delta 0), reused 0 (delta 0)
To /tmp/repo
 + b67c343...444b1cf master -> master (forced update)

Mais il y a beaucoup mieux !

Ce qu'il faut faire

Annuler un commit, c'est finalement appliquer l'inverse de son diff !

On peut rediriger le diff des commits à annuler vers la commande patch --reverse :)

git diff HEAD^ | patch --reverse

Pour faire plus simple, il y a git revert !

Par exemple pour annuler les trois derniers commits :

git revert HEAD~3

Ou pour annuler un commit en particulier :

git revert 444b1cff

Il suffit alors de pousser proprement le commit obtenu sur le serveur. Les éventuels collaborateurs qui avaient basé leur travail sur les commits annulés devront gérer les conflits au moment venu...

Two major Unity design failures

2011-10-15T00:00:00+02:00

A great advantage of global menus is the ease of pointing them with the mouse.

According to Fitts' law, the time to point a target is function of its distance and size. By sticking the menus on the top part of the screen, the vertical dimension vanishes, since the user can just throw his mouse on top to reach them.

Unfortunately, two major design problems in Unity prevents from completely turning global menus to account.

Window buttons dead edges

The user can not throw his mouse to the top-left corner to reach the close button, since the edges are not clickable.

A bug was registered on Launchpad a couple of minutes before I wrote this post.

Menus not always shown

The problem is being discussed on Launchpad.

In my opinion, the current implementation is a terrible design :

Lack of afordance : There is no obvious hint that a menu is available for the current application.
Fitt's law fail : The user can not throw his mouse to the desired menu since it becomes visible only when he reaches it.

Fortunately, I am gesture and keyboard user (easystroke, Alt+F10, Alt+F4)

Une démo squelettique de python Flask CouchDB

2011-10-11T11:30:00+02:00

Avec Flask et Couchdb (e.g. Flask-CouchDB), on peut faire rapidement des trucs amusants, voire très utiles !

Voici un squelette d'application, fonctionnel, qui stocke et récupère des objets crées à partir de posts HTTP.

import simplejson
from flask import Flask, g, request
from couchdb.design import ViewDefinition
import flaskext.couchdb


app = Flask(__name__)

"""
CouchDB permanent view
"""
docs_by_author = ViewDefinition('docs', 'byauthor',
                                'function(doc) { emit(doc.author, doc); }')

"""
Retrieve docs
"""
@app.route("/<author_id>/docs")
def docs(author_id):
    docs = []
    for row in docs_by_author(g.couch)[author_id]:
        docs.append(row.value)
    return simplejson.dumps(docs)

"""
Add doc
"""
@app.route("/<author_id>/add", methods=['POST'])
def add_doc(author_id):
    try:
        # Build doc with posted values
        doc = { 'author': author_id }
        doc.update(request.form)
        # Insert into database
        g.couch.save(doc)
        state = True
    except Exception, e:
        state = False
    return simplejson.dumps({'ok': state})

"""
Flask main
"""
if __name__ == "__main__":
    app.config.update(
        DEBUG = True,
        COUCHDB_SERVER = 'http://localhost:5984/',
        COUCHDB_DATABASE = 'docsdemo'
    )
    manager = flaskext.couchdb.CouchDBManager()
    manager.setup(app)
    manager.add_viewdef(docs_by_author)  # Install the view
    manager.sync(app)
    app.run(host='0.0.0.0', port=5000)

J'ai déposé ce snippet sur Gist si besoin.

On peut attaquer l'application avec curl :

$ curl -d "title=Globalia&year=2004" http://0.0.0.0:5000/jc.rufin/add
{"ok": true}
$ curl -d "title=Red%20Brazil&contest=goncourt" http://0.0.0.0:5000/jc.rufin/add
{"ok": true}

$ curl http://0.0.0.0:5000/jc.rufin/docs
[{"title": ["Globalia"], "year": ["2004"], "author": "jc.rufin", "_rev": "1-3195...fbc8", "_id": "dec81d...1733c"},
{"title": ["Red Brazil"], "contest": ["goncourt"], "author": "jc.rufin", "_rev": "1-7b15...a9a2", "_id": "dec81dc...17c0c"}]

N'oubliez pas de colorier les cases à votre guise, sinon ce squelette ne sert à rien, le JSON étant déjà la langue maternelle de CouchDB.

Mes premiers pas avec Heroku

2011-10-10T10:30:00+02:00

J'ai pris le temps de tester la plateforme Heroku, qui fait pas mal de bruit en ce moment.

L'idée est simple : le développeur pousse son code dans une branche Git pour déployer son application. Ils se chargent du reste.

La documentation pour déployer du python est bien faite.

heroku et virtualenv

Pour commencer, j'ai installé l'outil heroku en ruby (sic), dans un virtualenv

sudo aptitude install rubygems

virtualenv --no-site-packages env
source env/bin/activate

Merci à Bruno, qui nous a expliqué récemment comment faire cohabiter les gems ruby et virtualenv

export GEM_HOME="$VIRTUAL_ENV/gems"
export GEM_PATH=""
export PATH=$PATH:$GEM_HOME/bin

(À ajouter au hook dans ~/.virtualenvs/postactivate pour plus tard)

Pour terminer, il suffit de poser le gem

gem install heroku

Et j'ai bien heroku cloisonné dans le virtualenv.

(env)src$ which heroku
/home/mathieu/path/env/gems/bin/heroku

(env)src$ heroku help
Usage: heroku COMMAND [--app APP] [command-specific-options]

heroku et la Configuration de l'Application

Pour apprivoiser la plateforme, j'ai utilisé le micro-framework Flask, suggéré dans le tutorial python. C'est ultra-simple, ultra-léger, ultra-tout.

Afin de gérer ma configuration, j'ai créé une classe Settings qui utilise les variables d'environment:

# settings.py
import os

class Settings(object):
    DEBUG = bool(os.environ.get("DEBUG"))
    TESTING = bool(os.environ.get("TESTING"))
    PORT = int(os.environ.get("PORT", 5000))
    HOST = os.environ.get("HOST", '0.0.0.0')

Que je branche dans l'application :

# app.py
from flask import Flask

from settings import Settings


app = Flask(__name__)
settings = Settings()

#...

if __name__ == "__main__":
    # ...
    app.config.from_object(settings)

Ensuite grâce au client heroku, je peux contrôler à distance la configuration de mon application, qui est redémarrée à chaque changement :

(env)src$ heroku config
PATH              => bin:/usr/local/bin:/usr/bin:/bin
PYTHONUNBUFFERED  => true

(env)src$ heroku config:add DEBUG=True
Adding config vars:
  DEBUG => True
Restarting app... done, v19.

(env)src$ heroku config
DEBUG             => True
PATH              => bin:/usr/local/bin:/usr/bin:/bin
PYTHONUNBUFFERED  => true

(env)src$ heroku config:remove DEBUG
Removing DEBUG and restarting app... done, v20.

Je peux revenir en arrière quand un changement de config a posé problème :

(env)src$ heroku releases
Rel   Change                          By                    When
----  ----------------------          ----------            ----------
v20   Config remove DEBUG             your@mail.com         25 seconds ago
v19   Config add DEBUG                your@mail.com         1 minute ago


(env)src$ heroku rollback v19
Rolled back to v19

Avec Git rebase, vos arbres poussent droit

2011-09-16T17:37:00+02:00

Le problème

Par défaut, un git pull est équivalent à git fetch et git merge.

Les merge, c'est bien pour les branches, mais pour le tronc, c'est pénible : ça pollue l'historique et ça zig-zag sévère !

La solution

En réalité, quand on travaille sur le tronc, ce qu'on veut c'est faire git fetch et git rebase. C'est à dire, au lieu de ça :

      A-----B-----C master
     /             \
D---E---F---G---H---I origin/master

on veut ça :

                  A---B---C master
                 /
D---E---F---G---H origin/master

Autrement dit, un git pull --rebase ! Pour le faire par défaut :

git config --global branch.autosetuprebase always

Et ensuite, au cas-où, pour le désactiver ponctuellement, utiliser git pull --no-rebase.

Maintenant, le tronc, il est tout propre !

Les conséquences ?

Lors d'un git pull, il faudra résoudre chaque commit conflictuel indépendamment (perso, je préfère).

Les êtres humains voudront utiliser meld. Il suffit de l'installer, et lors d'un conflit, de lancer git mergetool.

Une fois que tous les conflits sont résolus, terminer l'opération, avec git rebase --continue, et pousser vos prouesses à vos amis avec git push.

A Virtual Local Server Room for you Developper

2011-09-02T09:12:00+02:00

This how-to will help you setting-up a very powerful development environment on your workstation, using virtual machines all in a virtual local network.

But Why ?

You work on various projects, with specific requirements, with various technologies, diverse operating systems or cpu architectures, usually with a couple of services like databases or Web servers.

You would prefer not taking this whole family with you each time you start your machine.

Probably, you also want to upgrade your workstation to the last unstable eye-candy OS without compromising your projects dependencies and without reinstalling all this stuff, or you may want to restore the environment you had when you were working on this famous project a year ago.

You want your colleague to give you a hand, and would like to give him your whole project environment and dependencies quickly and effortlessly ? Your sysadmin put KVM on servers and you would like to push your local instances on production in a blink ?

The magic medicine exists, and is freely available :)

At the end, you are promised to enjoy :

a GUI to manage your Virtual Machines (VM)
a local domain to access your VMs by their name
a fully integrated set of machines, accessible from each others
movable and shareable virtual machines with automatic network configuration

We chose the Linux Kernel-based Virtualization System (KVM), since it is maintained along with the Linux kernel, and is thus fully integrated in the OS. However, this how-to is mainly networking oriented and would be useful for any virtualization system.

A Strict Minimum

A strict minimum is to install kvm and a set of commands to control it : libvirt. As a human being, you may want a GUI : virt-manager.

sudo apt-get install kvm libvirt-bin virt-manager

Virtual Machines Network

Make sure your VM networking is set to NAT. This will allow your VM to access your host network (LAN, Internet, etc.)

During your VM operating system installation, or after login into it, setup your VM network interface as automatic DHCP.

By default KVM creates a virtual network (virnet). Inspect its setup using ifconfig or virt-manager in Connection Details > Virtual Networks. Your VM and your host are thus accessible within this virtual network (probably 172.16.23.0).

A Local Network Domain

In order to access your VM by their name, we run a DNS daemon on main host. We chose bind :

sudo aptitude install bind9

Add a new master zone, for example sillywalk.loc in the file /etc/bind/named.conf.local :

zone "sillywalk.loc" {
    type master;
    file "/etc/bind/db.sillywalk.loc";
};

Define your DNS entries :

Set the name server authority (SOA) to ns.sillywalk.loc (nameserver)
Define a serial number like date+number (YYYYmmdd##)
Associate ns.sillywalk.loc to the IP of your KVM virtual network (see above paragraph)
Define an alias gw.sillywalk.loc so that you can refer to your host as gw (gateway) instead of ns.
Define a couple of entries for your VM (e.g. myvm1, myvm2)

For bind, it would look like this (started from an existing file like ``/etc/bind/db.empty``) :

;/etc/bind/db.sillywalk.loc
;
; BIND data file for local loopback interface
;
$TTL    604800
@   IN  SOA ns.sillywalk.loc. root.ns.sillywalk.loc. (
         2011080301     ; Serial
             604800     ; Refresh
              86400     ; Retry
            2419200     ; Expire
             604800 )   ; Negative Cache TTL
;
@   IN  NS      ns.sillywalk.loc.
@   IN  A       172.16.23.1
ns  IN  A       172.16.23.1
gw  IN  CNAME   ns.sillywalk.loc.

myvm1       IN  A   172.16.23.11
myvm2       IN  A   172.16.23.12

Use your DNS

Use your Network Manager to set the search domain to sillywalk.loc and to add your local DNS server (127.0.0.1) in front of the other(s).

Apply and your /etc/resolv.conf could then look like this :

# Generated by NetworkManager
search sillywalk.loc          # search domain
nameserver 127.0.0.1          # your local DNS server
nameserver 192.168.1.254      # your FAI/company DNS
nameserver 8.8.8.8            # Google public DNS

Test it !

Even if your VM are not running, you can at least test the name resolving and the default search domain :

~$ ping myvm1.sillywalk.loc
PING myvm1.sillywalk.loc (172.16.23.11) 56(84) bytes of data.
# (Ctrl+C)

~$ ping myvm2
PING myvm2.sillywalk.loc (172.16.23.12) 56(84) bytes of data.
# (Ctrl+C)

Dynamic Configuration

In order to make sure your VM always obtains the same IP adress when it boots, we setup a DHCP daemon on host.

We chose ISC DHCP server :

sudo aptitude install isc-dhcp-server

In the configuration file /etc/dhcp/dhcpd.conf, we specify :

a domain name (sillywalk.loc)
the name server to be configured on clients (ns.sillywalk.loc)
the subnet and mask (matching the KVM virtual network)
an IP range (e.g. from 172.16.23.10 to 172.16.23.100)
the default gateway to be configured on clients (ns.sillywalk.loc)
... and two entries for myvm1 and myvm2 with their Mac addresses.

# /etc/dhcp/dhcpd.conf

option domain-name "sillywalk.loc";
option domain-name-servers ns.sillywalk.loc;

subnet 172.16.23.0 netmask 255.255.255.0 {
  range 172.16.23.10 172.16.23.100;
  option broadcast-address 172.16.23.255;
  option routers gw.sillywalk.loc;
}

# Entries

host myvm1 {
  hardware ethernet 52:54:00:55:d1:80;
  fixed-address myvm1.sillywalk.loc;
}

host myvm2 {
  hardware ethernet 52:54:00:55:e1:66;
  fixed-address myvm2.sillywalk.loc;
}

Test it !

Log you in on the VM.

Configure its hostname (e.g. myvm1, myvm2)

root@myvm1:~# cat /etc/hostname
myvm1
root@myvm1:~# cat /etc/hosts
127.0.0.1   localhost
127.0.1.1   myvm1.sillywalk.loc myvm1

Make sure your VM network is set to DHCP automatic configuration

root@myvm1:~# cat /etc/network/interfaces
...
# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp

Reboot it (or restart networking)
Check that it caught the right network configuration (IP, domain, and nameserver)

root@myvm1:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 52:54:00:55:d1:80
          inet addr:172.16.23.11  Bcast:172.16.23.255  Mask:255.255.255.0
          ...

root@myvm1:~# cat /etc/resolv.conf
domain sillywalk.loc
search sillywalk.loc
nameserver 172.16.23.1

Note

While your host is booting, the DHCP daemon usually starts before the KVM service, failing then at accessing the virtual network interface (virbr1, 172.16.23.0), not yet mounted.

A simple solution is to manually restart your DHCP daemon, once your machine's booted :

sudo /etc/init.d/isc-dhcp-server restart

Checklist to add a new VM

Get its Mac address (with virt-manager : Virtual machine details > Virtual network interface > Mac Address)
Add it to your DHCP configuration (/etc/dhcp/dhcpd.conf)
Add an IP for this entry in your DNS zone (/etc/bind/db.sillywalk.loc) and increment the serial.
Restart DHCP service and reload DNS configuration

sudo /etc/init.d/isc-dhcp-server restart
sudo /etc/init.d/bind9 reload

If you do that all day, you'll quickly find it relevant to write a script...

Note on cloning

Cloning your VM with virt-manager is a piece-of-cake.

However, during cloning, KVM assigns a new Mac address to the clone. For debian-based virtual machines (+Ubuntu), log you in on the clone, and reinitialize network interfaces names :

sudo rm /etc/udev/rules.d/70-persistent-net.rules
sudo reboot

Conclusion

Your virtual machines can :

access your network (LAN, Internet) and your host (at ns.sillywalk.loc)
be accessed at user@hostname (from host or from other VMs)
be moved to any host set up likewise (since VM networking is fully automatic)
be cloned easily

Read again "But Why ?" and enjoy your new life !

A very Simple and Stupid plugin system in python

2011-09-02T00:00:00+02:00

Two convenience functions for listing and importing python modules :

# utils.py

import os

def plugins_list(plugins_dirs):
    """ List all python modules in specified plugins folders """
    for path in plugins_dirs.split(os.pathsep):
        for filename in os.listdir(path):
            name, ext = os.path.splitext(filename)
            if ext.endswith(".py"):
                yield name


def import_plugins(plugins_dirs, env):
    """ Import modules into specified environment (symbol table) """
    for p in plugins_list(plugins_dirs):
        m = __import__(p, env)
        env[p] = m

And now use import_plugins() wherever you need to use them !

# yourapp.py

import os
from utils import import_plugins

plugins_dirs = "plugins/:module/plugins/"
sys.path.extend(plugins_dirs.split(os.pathsep))

import_plugins(plugins_dirs, globals())

Note that in order to list all sub-classes of a specific one, you can use this recursive function.

That's all folks !

It is very simple and very stupid, but useful :) You might now want to have a look at serious stuff like Yapsy or PkgResouces.

Le piège des QThread

2011-09-01T11:09:00+02:00

Article original publié chez Makina Corpus

Il y a de nombreux billets de blogs, posts sur des forums, tutoriaux, pages Wiki et autres, mais au final, à part le fameux "You're doing it wrong", qui peut paraître obscure au premier abord, je n'ai pas trouvé de résumé de l'attrape-nigaud que je vais illustrer ici.

Le piège

Naturellement, quand on veut faire une thread, on a envie d'hériter de l'objet QThread. C'est ce qu'on fait avec le module threading de python (en Java aussi il me semble).

Voici ce qu'on écrit naturellement : Objet, la classe qui file l'ordre et Worker, une classe qui bosse dur en arrière plan. On connecte les signaux et on démarre !

import sys
from PyQt4.QtCore import *
from PyQt4.QtGui  import QApplication

class Object(QObject):
    def emitSignal(self):
        self.emit(SIGNAL("aSignal()"))

class Worker(QThread):
    def aSlot(self):
        self.thread().sleep(1)
        print "Slot is executed in thread : ", self.thread().currentThreadId()

if __name__ == "__main__":
    app = QApplication(sys.argv)
    print "Main application thread is : ", app.thread().currentThreadId()

    worker = Worker()
    obj = Object()
    QObject.connect(obj, SIGNAL("aSignal()"), worker.aSlot)

    worker.start()
    obj.emitSignal()

    print "Done."
    app.exec_()

Ici, comme le slot aSlot() est défini dans la classe Worker, qui hérite de QThread, on pense naturellement qu'il va être exécuté en arrière-plan. Que nenni!

Main application thread is :  140068661352224
# (... wait 1 sec ...)
Slot is executed in thread :  140068661352224
Done.

La solution

Un secret ? Les QThread ne sont pas des threads. Elles enrobent l'execution d'une thread.

L'appartenance (affinité) d'un objet à une thread détermine le type de connexion utilisé par défaut, et par conséquent le comportement lors de l'execution des slots.

Ce qu'il faut écrire : Worker n'est plus une QThread, on force son affinité dans une thread avec moveToThread().

class Object(QObject):
    def emitSignal(self):
        self.emit(SIGNAL("aSignal()"))

class Worker(QObject):
    def aSlot(self):
        self.thread().sleep(1)
        print "Slot is executed in thread : ", self.thread().currentThreadId()

if __name__ == "__main__":
    app = QApplication(sys.argv)
    print "Main application thread is : ", app.thread().currentThreadId()

    worker = Worker()
    obj = Object()

    thread = QThread()
    worker.moveToThread(thread)
    QObject.connect(obj, SIGNAL("aSignal()"), worker.aSlot)

    thread.start()
    obj.emitSignal()

    print "Done."
    app.exec_()

Désormais, l'exécution est bien asynchrone, comme on le souhaitait.

Main application thread is :  139961882056480
Done.
# (... wait 1 sec ...)
Slot is executed in thread :  139961512900352

Tout simplement ! Si j'avais lu mon article avant, je n'aurais pas perdu autant de temps à lire toutes ces docs ambiguës sur le Net.

Sources:

Explications complètes
Thread Basics (attention au piège)

PostGIS data in C++ using GDAL and Qt

2011-08-23T10:25:00+02:00

Original post at Makina Corpus

I did not find any ready-to-use snippets on the Web on this matter, so if you are lucky enough, you'll find this one.

The objective is to read GIS geometries from a PostGIS database and manipulate them in C++. I use Qt here, but it is not really a prerequisite, it just helps a lot. Well, actually, it saves lives.

Database Connection

m_db = QSqlDatabase::addDatabase("QPSQL");
m_db.setHostName("host");
m_db.setDatabaseName("dbname");
m_db.setUserName("user");
m_db.setPassword("pass");

Do not close the database at the end of each query.

m_db.close();
m_db = QSqlDatabase();  // reinitialize for real

Shut it down like this in your class' destructor or you may have errors like QSqlDatabasePrivate::removeDatabase: connection 'qt_sql_default_connection' is still in use, all queries will cease to work

Records Reading

QSqlQueryModel* model = new QSqlQueryModel();

model->setQuery("SELECT id, ST_AsBinary(the_geom) AS the_geom "
                "FROM table");

int numRows = model->rowCount();

for (int i=0; i<numRows; ++i) {
    // Read fields
    qlonglong id = record(i).value("id").toLongLong();
    QByteArray wkb = record(i).value("the_geom").toByteArray();

    // Process !
    processRecord(id, wkb);
}

QByteArray uses implicit sharing and can be passed as argument without being copied.

Geometries Parsing

In this part, we rely on GDAL (Geospatial Data Abstraction Library) OGRSpatialReference.

It provides an API to access geometries coordinates etc.

#include "ogrsf_frmts.h" // GDAL
...
...

void Class::processRecord(qlonglong id, QByteArray wkb)
{
    OGRSpatialReference osr;
    OGRGeometry *geom = NULL;

    // Parse WKB
    OGRErr err = OGRGeometryFactory::createFromWkb((unsigned char*)wkb.constData(), &osr, &geom);
    if (err != OGRERR_NONE){
        // process error, like emit signal
    }

    // Analyse geometry by type and process them as you wish
    OGRwkbGeometryType type = wkbFlatten(geom->getGeometryType());
    switch(type) {
        case wkbLineString: {
            OGRLineString *poRing = (OGRLineString*)geom;

            // Access line string nodes for example :
            int numNode = poRing->getNumPoints();
            OGRPoint p;
            for(int i = 0;  i < numNode;  i++) {
                poRing->getPoint(i, &p);
                qDebug() << p.getX() << p.getY();
            }
            break;
        }
        case wkbMultiLineString:
        {
            OGRGeometryCollection  *poCol = (OGRGeometryCollection*) geom;
            int numCol = poCol->getNumGeometries();
            for(int i=0; i<numCol; i++) {
                // Access line length for example :
                qDebug() << poCol->getGeometryRef(i)->get_Length();
            }
            break;
        }
        default:
            // process error, like emit signal
    }

    // Clean-up
    OGRGeometryFactory::destroyGeometry(geom);
}

In this snippet, I only process linestrings, but all geometry types are available. Consider writing a recursive function for geometry collections and so forth...

Hope this helped !

Django, gestion des settings d'application simplifiée

2011-07-29T18:30:00+02:00

Je reprends ici la méthode présentée par Nicolas et je la couple avec mon petit EasyDict pour alléger l'utilisation ! Comme ses snippets sont drôles, je ne les change pas !

Paramètres par défault de l'application

On a juste un constructeur à ajouter par rapport à ce qu'avait présenté NiKo (avec EasyDict installé).

# apps/my_app/__init__.py
from django.conf import settings
from easydict import EasyDict

app_settings = EasyDict(dict({
    'FOO': 42,
    'ENABLE_CHUCK_NORRIZ_MODE': False,
}, **getattr(settings, 'MY_APP_CONFIG', {})))

Surcharge dans le projet

# settings.py
MY_APP_CONFIG = {
    'ENABLE_CHUCK_NORRIZ_MODE': True,
}

Utilisation !

EasyDict transforme les clés du dict en attributs, on accède aux settings en toute simplicité !

# foo/bar.py
from my_app import app_settings

print app_settings.FOO # 42

# apps/my_app/utils.py
from . import app_settings

if app_settings.ENABLE_CHUCK_NORRIZ_MODE:
    print 'Chuck Norriz is watching you'
else:
    print 'Dance dance, little lamb'

Carte des vélos avec Leaflet

2011-05-30T15:25:00+02:00

Article original publié chez Makina Corpus

Les bookmarks, un peu comme les cahiers de recettes, c'est bien de les remplir mais encore faut-il trouver les bons au moment adéquate ! Même quand il s'agit d'outils, de bibliothèques et de services Web, il faut trouver l'occasion de les tester avant le grand soir ! Et si on veut en faire un article de blog, alors là, il faut en plus donner envie d'y goûter :)

Ici, je prends plein d'ingrédients trouvés au bord des chemins :

Je secoue bien fort ! (sans oublier de saupoudrer de jquery) et j'obtiens une carte interactive des stations vélos de Toulouse !

La liste des stations

Sur le site http://velonow.info, je récupère un fichier XML qui contient la liste statique des stations de vélo et leurs identifiants.

C'est l'occasion d'utiliser pyquery pour le transformer en GeoJSON. Gawel nous l'avait présenté aux djangocongs, il s'agit du portage de l'API de JQuery en python !

from pyquery import PyQuery as pq

d = pq(url='http://server.com/file.xml')

for m in map(lambda e: pq(e), d.find('marker')):
    pt = geojson.Point([m.attr('lng'),
                        m.attr('lat')])
    ...
    ...

Je trouve ça génial d'avoir la même syntaxe de manipulation du DOM en python et en javascript ! Et pour faire du webscrapping, c'est top !

Affichage de la carte

Cloudmade a créé leaflet qui rejoint Tile5 et Polymaps en tant que challenger d'Openlayers !

C'est une bibliothèque légère, jolie, fluide, optimisée pour les mobiles, et même compatible Internet Explorer !

Pour afficher une carte centrée sur la localisation du visiteur de la page, il suffit de faire ça :

var map = new L.Map('map');

var cloudmadeUrl = 'http://{s}.tile.cloudmade.com/BC9A493B41014CAABB98F0471D759707/997/256/{z}/{x}/{y}.png',
    cloudmade = new L.TileLayer(cloudmadeUrl);

map.addLayer(cloudmade);

map.locateAndSetView();

Pour l'instant, Leaflet ne gère pas les couches au format GeoJSON, en attendant la prochaine release, nous allons ajouter les points des stations en 2 coups de cuillère à pot :

$.getJSON(url, function(data) {
    $.each(data.features, function(i, f) {
        var cc = f.geometry.coordinates;
        var marker = new L.Marker(new L.LatLng(cc[1], cc[0]));
        map.addLayer(marker);
    });
});

URL reverse en Javascript avec django

2011-05-27T15:25:00+02:00

Article original publié chez Makina Corpus

Un aspect fondamental de la philosophie django consiste à éviter toute sorte de redondance : Don't Repeat Yourself.

La tentation est souvent trop belle, et respecter les fondamentaux s'avère parfois difficile ! C'est le cas de la réécriture d'URL en Javascript.

Imaginons l'URL suivante définie dans urls.py

url(r'^/plop/(?P<x>\d)/(?P<y>\d)$', plopview, name="plop")

Pour utiliser cette URL en Javascript avec des paramètres variables, on peut imaginer plusieurs approches.

J'aime pas

var generic = "{% url 'plop' 0 0 %}";
generic.replace('0/0', x+'/'+y);

Pas DRY ! à cause des /.

"{% url 'plop' 0 0 %}" + '../../' + x + '/' + y;

Pas DRY non plus !

On peut aussi changer le pattern pour éviter les /.

url(r'^/plop/(?P<x>[\d]|x)/(?P<y>[\d]|y)$', , name="plop")

var generic = "{% url 'toto' 'x' 'y' %}";
generic.replace('x', x).replace('y', y);

C'est mieux, mais pas DRY ! à cause des x, y.

On pourrait aussi imaginer une vue django qui ferait le reverse(). Mais cela multiplierait les aller-retours serveur, ce qui n'est pas toujours recommandé...

La bonne

Il existe une application pour ça ! django-js-utils

Elle se charge de générer un fichier Javascript (settings.URLS_JS_GENERATED_FILE) grâce à une commande de gestion

python manage.js js_urls

Ensuite on utilise explicitement le fichier généré

<script type="text/javascript" src="{{ MEDIA_URL }}/js/dutils.js"></script>
<script type="text/javascript" src="{{ MEDIA_URL }}/js/dutils.conf.urls.js"></script>

Et on fait du vrai DRY !

dutils.urls.resolve('plop', { 'x': x, 'y': y })

Gagné ! \o/

Un inconvénient à noter tout de même : la liste de l'ensemble des URLs de l'application est accessible au public. Mais j'ai pas mieux ma pauvre dame !

Django et Jenkins

2011-04-28T17:25:00+02:00

Article original publié chez Makina Corpus

Lors des Recontres Django 2011, Nicolas Perriault a présenté les principes de l'intégration continue avec Django et Jenkins.

Le diaporama, disponible en ligne, suffit amplement pour démarrer !

Mais pour qu'un projet django soit testé facilement, il doit se déployer et se lancer facilement ! C'est certes l'occasion de peaufiner l'automatisation, mais c'est loin d'être trivial quand il y a du SIG, du celery ... Je vais tenter de partager mes notes dans ce billet.

Le minimum requis

Pour l'installation de Jenkins, rien de plus simple (sur debian)

sudo aptitude install jenkins

Mais il va falloir lui donner de quoi télécharger votre code sur git et parfois compiler les librairies python nécessaires

sudo aptitude install git-core
sudo aptitude install python-dev build-essential python-virtualenv

Les plugins indispensables :

covertura
Violations
GIT
Green Balls
Continuous Integration Game

Organisation du projet Django

Définition des dépendances globales dans requirements.txt
```
Django>=1.3
south
```
Définition des dépendances liées aux tests dans requirements-testing.txt
```
django-jenkins
```

Ajout d'un fichier pylint.rc pour régler les niveaux d'alerte PEP-8

[MESSAGES CONTROL]
disable=E1101,E1103,C0111,I0011,I0012,W0704,W0142,W0212,W0232,W0613,W0702,R0201
...
...

Modèle de settings de tests dans project/test_settings.py

from default_settings import *
DEBUG = True
TEMPLATE_DEBUG = DEBUG

INSTALLED_APPS += (
    'django_jenkins',
)

PYLINT_RCFILE = os.path.join(PROJECT_ROOT_PATH, '..', 'conf', 'pylint.rc')

Configuration du job Jenkins

Les informations de la présentation de Nicolas suffisent pour démarrer.

J'ai noté cependant qu'il fallait lancer manage.py depuis un répertoire parent au projet pour que l'exploration du code source fonctionne.

Pour profiter de la magie des ingrédients précédents, nous aurons donc juste à ajouter un bloc script shell, qui installe les dépendances listées, pose les settings de test et migre la base (avec South):

#!/bin/bash -ex
virtualenv --quiet ve
source ./ve/bin/activate
pip install -E ./ve -r $WORKSPACE/requirements.txt
pip install -E ./ve -r $WORKSPACE/requirements-testing.txt
cp $WORKSPACE/project/test_settings.py $WORKSPACE/project/local_settings.py
python $WORKSPACE/project/manage.py syncdb --noinput
python $WORKSPACE/project/manage.py migrate
deactivate

et celui-ci pour lancer les tests proprements dits :

#!/bin/bash -ex
virtualenv --quiet ve
source ./ve/bin/activate
python $WORKSPACE/project/manage.py jenkins yourapps
deactivate

Pour un projet SIG

Il faut installer certaines librairies SIG sur le serveur Jenkins.

sudo aptitude install libproj0 libgeos-c1

Si le besoin de cloisonner ces librairies pour chaque projet se fait ressentir, il faut utiliser des outils comme minitage.

Spatialite au lieu de PostGIS comme base de tests

sudo aptitude install python-sqlite libspatialite2 sqlite3

Script d'initialisation

wget http://www.gaia-gis.it/spatialite/init_spatialite-2.3.zip -O /tmp/init_spatialite-2.3.zip
cd /usr/local/lib/
sudo unzip /tmp/init_spatialite-2.3.zip

avec dans test_settings.py

DATABASES = {
'default': {
    'ENGINE': 'django.contrib.gis.db.backends.spatialite',
    ...
    ...

SPATIALITE_SQL=os.path.join('usr', 'local', 'lib', 'init_spatialite-2.3.sql')

Si pysqlite n'a pas été compilé avec les extensions C (Erreur: The pysqlite library does not support C extension loading.) il va falloir le recompiler !

sudo aptitude install libsqlite3-dev
wget http://pysqlite.googlecode.com/files/pysqlite-2.6.3.tar.gz
tar -zxvf pysqlite-2.6.3.tar.gz
cd pysqlite-2.6.3
sed -i s/define=SQLITE_OMIT_LOAD_EXTENSION/#define=SQLITE_OMIT_LOAD_EXTENSION/g setup.cfg

source ./ve/bin/activate
python setup.py install

Pour un projet Celery

Kombu au lieu de RabbitMQ comme gestionnaire de messages

requirements-testing.txt

kombu
djkombu

test_settings.py

INSTALLED_APPS += (
    'djkombu',
)
CARROT_BACKEND = "django"

Pour désactiver la parallélisation lors des tests

CELERY_ALWAYS_EAGER = True

landez : fabriquer facilement des fichiers MBTiles en python

2011-04-21T12:04:00+02:00

Une nouvelle fois, Makina Corpus se rapproche du projet MapBox, avec une contribution sur la librairie mbutil, qui permet de fabriquer des fichiers MBTiles.

Nous l'utilisons dans landez, un outil qui permet de créer des fichiers MBTiles à partir de sites de tuiles externes ou de feuilles de styles Mapnik.

Son utilisation est fort simple !

Pour un service de tuiles externe :

from landez import MBTilesBuilder

# downloads from Cloudmade by default, be careful with terms of usage  !
mb = MBTilesBuilder(remote=True, filepath="dest.mbtiles")

mb.add_coverage(bbox=(-90.0, -180.0, 180.0, 90.0),
                zoomlevels=[0, 1])
mb.run()

Avec une feuille de style locale :

from landez import MBTilesBuilder

mb = MBTilesBuilder(stylefile="yourstyle.xml", filepath="dest.mbtiles")
mb.add_coverage(bbox=(-90.0, -180.0, 180.0, 90.0),
                zoomlevels=[0, 1])
mb.run()

Une branche experimentale a été initiée pour travailler sur le multiprocessing afin de paralléliser au maximum la fabrication des tuiles en amont. Si cela vous intéresse, soyez les bienvenus !

Deploy Django behind a reverse proxy

2011-04-19T00:00:00+02:00

By default, Django will assume that your root URL is the root (/) of your domain.

Using a reverse proxy, we can run multiple django instances on the same server, using the same domain. (http://server.org/site1/, http://server.org/site2/, ...)

Many redirects of your application will then be broken (most notable is validation of login form). You can fix that by forcing the root URL in your settings

FORCE_SCRIPT_NAME = '/site1'

If you use Sentry, you'll also have to set

SENTRY_URL_PREFIX = '/site1'

Ajouter un sélecteur de couches (layer switcher) à polymaps

2011-03-09T12:03:00+01:00

Je fais partie de ceux qui sont persuadés que polymaps est tout à fait mature ! Certes, il ne fournit pas autant de fonctionnalités et de connecteurs qu'OpenLayers, mais il ne semble pas que ce soit son objectif ! Il est léger et personnalisable à souhait !

Pour le prouver, nous allons créer ici un sélecteur de couches pour polymaps.

Javascript n'est pas mon langage de prédilection, encore moins pour faire de la programmation orientée objets.

Voici ce que j'ai réussi à comprendre de l'héritage et la portée dans le modèle objet de polymaps :

(function(po) {
  po.classname = function(args) {
   var self = {},      // new class or inheritance
        member;         // member variable

    function privatemethod () {
        // [...]
    }

    self.classmethod = function (args) {
        // [...]
        return self;  // allows to chain method calls
    }

    return self;
  };
})(org.polymaps);

Pour faire ce sélecteur de couches, nous aurons besoin d'une classe disposant :

de variables membres qui stockent la liste de couches disponibles (layers) et la couche actuelle (current)
d'une méthode de classe qui bascule d'une couche à l'autre

self.switchto = function (name) {
    var l = layers[name];  // find layer by name
    if (l.map()) {
       l.visible(true);   // if already loaded, make it visible
    }
    else {
        map.add(l);        // else load it
    }
    if (current) current.visible(false);  // hide current
    current = l;
}

d'une méthode qui crée l'interface avec les radio buttons et qui les relie à la méthode précedente

self.container = function (elt) {
    var list = document.createElement('div');
    // [...]
    // For each layer, create a <input>
    for (name in layers) {
        var input = document.createElement('input');
        input.setAttribute('type', 'radio');
        input.setAttribute('value', name);
        // [...]
        // Link onChange event on radio
        input.onchange = function () {
            self.switchto(this.getAttribute('value'));
        };
        // [...]
        list.appendChild(input);
    }
    // [...]
    elt.appendChild(list);
    return self;
}

Maintenant il suffit de l'utiliser ! Voici un exemple simple avec deux couches :

// Create a normal map
map = po.mcmap()
        .container(document.getElementById("map").appendChild(po.svg("svg")))
        .add(po.interact());

// Define the layers
var layers = {
    "layer1" :
    po.image()
      .url(po.url("http://server1/{Z}/{X}/{Y}.png"))
      .id('l1'),

    "layer2" :
    po.image()
      .url(po.url("http://server2/{Z}/{X}/{Y}.png"))
      .id('l2'),
};

// Add the default one
map.add(layers["layer1"]);

// Create the switcher
po.switcher(map, layers, {title : 'Fond de carte'})
  .container(document.getElementById("layerswitcher"));

Et voilà ! Nous avons notre sélecteur de couches, avec un code html tout simple (div, input, label), facile à styler en CSS, contrairement au gros pavé généré par le *LayerSwitcher* d'OpenLayers.

Pour accéder au code complet et l'améliorer : "Fork me on GitHub" !

Article original publié chez Makina Corpus

Afficher des pictogrammes avec polymaps

2011-02-28T17:02:00+01:00

En reprenant l'exemple de l'épisode précédent avec polymaps, nous allons maintenant afficher des pictogrammes sur les points.

On conserve la couche GeoJSON en utilisant un callback pour l'évènement load.

map.add(po.geoJson()
          .url('collecteurs.json')
          .on("load", load));

Dans la fonction load(), nous allons remplacer les cercles dessinés par défaut par des pictogrammes en manipulant les éléments de la page (DOM). Nous utilisons ici la variable n$, qui provient du miniscript nns.js livré dans l'archive polymaps et qui facilite la manipulation du DOM (le vénérable jquery ferait aussi l'affaire)

function load(e) {
    var ICONSIZE = 16;
    // Parcourir les features de la carte
    for (var i = 0; i < e.features.length; i++) {
        var circle = n$(e.features[i].element);
        var root = circle.parent();
        var attributes = e.features[i].data.properties;

        // Ajouter et positionner le pictogramme
        // (à partir de la position du cercle)
        img = root.add("svg:image")
             .attr('width', ICONSIZE)
             .attr('height', ICONSIZE)
             .attr("transform", circle.attr('transform')
                            + ' translate(-'+(ICONSIZE/2)+','
                                       + '-'+(ICONSIZE/2)+')');
        // Enlever le cercle original
        root.remove(circle);
        // Définir le chemin du pictogramme à utiliser
        // en fonction de l'attribut
        img.attr('xlink:href', attributes.Etat == 'Actif' ?
                               'actif.svg' : 'inactif.svg');
    }
}

Ici, nous avons utilisé des pictogrammes SVG, mais le même code fonctionne avec des images PNG ou JPG...

Afficher les données de Paris OpenData avec polymaps

2011-02-24T13:02:00+01:00

En ouvrant l'accès à un catalogue de données diverses (Équipements, bâti, arbres d'alignement, arrêtés municipaux, ...) l'initiative ParisData, l'Open Data de la capitale, nous donne l'occasion de manipuler des données georéférencées. Notre objectif ici sera de les publier sur une page Web grâce à un outil simple et léger : polymaps.

Transformation

Le système de projection utilisé pour certaines données du catalogue est la Lambert Conformal Conic (NTF, EPSG 9802)

Dans la mesure où nous voulons déployer quelquechose de très simple, nous n'avons pas l'intention de sortir la grosse artillerie habituelle (Serveur WMS, Mapserver, QGIS MapServer, ...), nous allons plutôt utiliser un fichier GeoJSON, en longitudes/latitudes WGS84 (EPSG:4326).

Pour cela, la bibliothèque GDAL nous offre tous les outils adéquates:

Ouvrir le fichier shape (ESRI Shapefile) fourni par ParisData
Reprojetter en EPSG:4326
Choisir les données (attributaires) que nous allons conserver
Exporter en GeoJSON

Pour notre exemple, nous avons choisi les emplacements des points de collecte de verre. Parmis les champs fournis, nous choisissons de ne conserver que leur état (Lb_Etat_E) et le nom de leur emplacement (Emplacemnt).

Comme python est notre language préféré, et que c'est toujours un plaisir de le montrer en action, voici la petite procédure qui fait tout ça :

# python gdal
from osgeo import ogr
from osgeo import osr
...
...
# Ouvrir le répertoire contenant les shape
source = ogr.Open(INPUT_FOLDER)

# La projection de sortie
spatialRef = osr.SpatialReference()
spatialRef.ImportFromEPSG(4326)

# Le fichier de sortie
driver = ogr.GetDriverByName('GeoJSON')
shape = driver.CreateDataSource(OUTPUT_FILE)

# Parcourir les couches
for layer in source:
    # Reprojection originale -> destination
    originalSpatialRef = layer.GetSpatialRef()
    coordTransform = osr.CoordinateTransformation(originalSpatialRef,
                                                  spatialRef)
    # Choix des champs des données
    properties = ogr.FeatureDefn()
    properties.AddFieldDefn(ogr.FieldDefn('Etat'))
    properties.AddFieldDefn(ogr.FieldDefn('Emplacement'))

    # Créer la nouvelle couche GeoJSON
    newLayer = shape.CreateLayer(layer.GetName(), spatialRef)

    # Parcourir les features
    for feature in layer:
        # Créer la nouvelle feature
        newFeature = ogr.Feature(properties)
        # Remplir les champs choisis
        newFeature.SetField('Etat', feature.GetField('Lb_Etat_E'))
        newFeature.SetField('Emplacement', feature.GetField('Emplacemnt'))
        # Reprojetter la feature
        geometry = feature.GetGeometryRef()
        geometry.Transform(coordTransform)
        # Sauvegarder
        newFeature.SetGeometry(geometry)
        newLayer.CreateFeature(newFeature)
        newFeature.Destroy()

Nous obtenons en sortie un fichier GeoJSON avec les points en lat / long et les données 'Etat' et 'Emplacement'.

...
{ "type": "Feature", "properties": { "Etat": "Actif", "Emplacement": "37 CHATEAU D'EAU ANGLE BOUCHARDON" }, "geometry": { "type": "Point", "coordinates": [ 2.358920, 48.871154 ] } },
{ "type": "Feature", "properties": { "Etat": "Actif", "Emplacement": "13 place de la Nation" }, "geometry": { "type": "Point", "coordinates": [ 2.398154, 48.848723 ] } },
...

Le fichier pèse 174Ko, mais lorsqu'Apache le servira il pèsera 20Ko (grâce à la compression gzip !)

Affichage

Nous choisissons d'afficher ces données dans une page avec polymaps. Il s'agit d'un composant Javascript permettant de créer des cartes interactives.

Les critères de comparaison avec OpenLayers (OL) sont:

la légèreté (~30Ko, 10Ko en gzip!)
la rapiditité d'exécution
l'utilisation de GeoJSON et SVG (flexibilité et styles)

Cependant, la couverture fonctionnelle n'est absolument pas comparable. Mais pour afficher une carte avec des points, c'est largement suffisant !

On commence par un fond de carte: Cloudmade, dont les tuiles sont dessinées à partir d'OpenStreetMap:

map.add(po.image()
        .url(po.url("http://{S}tile.cloudmade.com"
                  + "/1a1b06b230af4efdbb989ea99e9841af"
                  + "/998/256/{Z}/{X}/{Y}.png")
        .hosts(["a.", "b.", "c.", ""])));

On ajoute ensuite nos données GeoJSON:

map.add(po.geoJson()
        .url('collecteurs.json')

Polymaps facilite la personnalisation du dessin en fonction des données. Ici, nous affichons en vert les collecteurs à l'état "Actif" et en rouge les autres. De même nous mettons leur "Emplacement" en tooltip (svg:title, Firefox 4, Chrome, Opera 11).

map.add(po.geoJson()
        .url('collecteurs.json')
        .on("load", po.stylist()
                      .attr('fill',
                            function(d) {
                                 return d.properties.Etat == 'Actif' ?
                                        'green' : 'red';
                            }))
                      .title(function(d) {
                                 return d.properties.Emplacement
                            }));

Accéder à la page de démonstration

Conclusion

Finalement, l'étape la plus compliquée était de reprojetter les données. On regrettera donc que l'initiative ParisData les ait publié sous cette forme exotique.

Comme le soulignait David : Publieurs de données, concentrez vous sur la qualité, pas la taille, les développeurs vous remercieront !

À noter également que nous avons choisi une approche privilégiant la légèreté. Or, plusieurs sources de données de ParisData sont volumineuses et ne pourraient pas être affichées en GeoJSON sans mettre à genoux le navigateur. Nous serions alors contraints de servir les données sous forme de tuiles...

SQLAlchemy, a brave new World

2011-02-22T15:02:00+01:00

SQLAlchemy becomes an essential technology for any python developper interacting with relational databases. As a Django developper, I have sat on my laurels for long, being completely satisfied with the Django ORM. It was time to explore its challenger.

First, before it sounds like I crush my favorite framework : when Django ORM was developped, there was no SQLAlchemy, or almost no good python ORM at all.

Here are some of the things you might want to know first:

Django ORM documentation is clean and well organized
Dango ORM was not really meant to be used outside django apps
Django ORM has limitations when it comes to Model Inheritance
Django ORM does not manage models migrations without extra stuff like South
Django ORM support for multiple databases was introduced in version 1.2
Django ORM does not always manage connection pooling (e.g. with Oracle)
SQLAlchemy is light and framework independant
SQLAlchemy is stripped to a minimum set of (clean and well-implemented) features
SQLAlchemy is very flexible and supports mapping of objects using declarations or metadata
SQLAlchemy documentation is bloated (API reference is mixed-in with long explanation and use cases)
SQLAlchemy requires a better knowledge of advanced python mechanisms and architecture

In order to explore SQLAlchemy (SA) I created pyfspot (sources are on github): a very small application to manage the database of the F-Spot photo manager. It is not supposed to save lives, but that will at least be:

a pretext for me to dive into the API
a small and useful tool
an example of SA in action for any developper interested
a base for a full F-Spot management application (</dreamer>)

I discovered a few concrete things:

Django inspectdb equivalent ?
- sqlautocode (unfortunately I could not use it)
Django fixtures equivalent ?
- fixture (demo)
Django model.DoesNotExist ?

# get() does not throw exception
tag = session.query(Tag).get(5)
# filter().one() does ...
try:
    tag = session.query(Photo).filter_by(id=1337).one()
except sqlalchemy.orm.exc.NoResultFound:
    # d'oh!
    pass

Invert condition like Django exclude() ?

session.query(Photo).filter(~Photo.base_uri.endswith('/'))

Escape LIKE condition ?

Photo.base_uri.like('%\\%%', escape="\\")

Intersections of many-to-many ?

# A tag
tag = session.query(Tag).filter_by(name='foo').one()
# A set of photos
photoset = session.query(Photo).filter(~Photo.base_uri.endswith('/'))
# intersect() won't work
photoset.intersect(tag.photos)
AttributeError: 'InstrumentedList' object has no attribute 'c'
# Use any()
photoset.filter(Photo.tags.any(id=tag.id))

Well, those were my first steps. As expected, it did not feel so well to relearn how to walk. But at least I am now ready to get my bearings in SQLAlchemy's world.

Original post at Makina Corpus

Javascript Beautifier in command-line (and in Geany editor)

2011-02-01T10:20:00+01:00

Install a Javascript engine (like Mozilla Rhino)

sudo aptitude install rhino

Get the beautifier script (put it somewhere like ~/.bin)

wget http://jsbeautifier.org/beautify.js

Add the following at the end of beautify.js

print( js_beautify( readFile( arguments[0] )));

Create a shell script that will call it (like ~/.bin/beautifyjs)

#!/bin/sh
java -cp /usr/share/java/js.jar org.mozilla.javascript.tools.shell.Main ~/.bin/beautify.js $*

Make sure to set it executable

chmod +x ~/.bin/beautifyjs

Use it from command-line

At least to check that it works !

~/.bin/beautifyjs /your/file.js

Or in Geany Editor

Open a Javascript file

Open menu Build > Define Build Commands

Create a new entry (like beautify)
In command, enter the following
~/.bin/beautifyjs %f > /tmp/tmpfile.js && geany /tmp/tmpfile.js
In working directory, enter %d

Python UTF-8 print fails when redirecting stdout

2011-01-26T11:25:00+01:00

Consider the following piece of code:

# -*- coding: utf-8 -*-
print u"Վարդանաշեն"

Running this in a terminal works:

$ python test.py
Վարդանաշեն

Redirecting standard output to a file fails:

$ python test.py > file
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    print u"Վարդանաշեն"
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-9: ordinal not in range(128)

Explanations are available on Python official wiki: default encoding has to be forced.

With an environment variable:

$ PYTHONIOENCODING='utf_8'
$ export PYTHONIOENCODING
$ python test.py > file
$

With source modification:

import sys
import codecs
import locale
sys.stdout = codecs.getwriter(locale.getpreferredencoding())(sys.stdout)

Grep with context lines above and below

2010-09-30T10:25:00+02:00

I was about to write a script to implement exactly what already exists natively in GNU grep, a blasphemy I got saved from by my workmates.

grep --line-number --colour=AUTO --before-context 5 --after-context 5 PATTERN FILENAME

Python lazy hasattr()

2010-09-30T10:20:00+02:00

Python hasattr() evaluates the specified attribute, which may not be desired !

class Attr(object):
    def __get__(self, obs, cls=None):
        print "evaluated"
        return 0

class ClassA(object):
    a = Attr()

    @property
    def b(self):
        print "evaluated"
        return 0

>>> c = ClassA()
>>> c.a
evaluated
0
>>> c.b
evaluated
0

Now note that hasattr() evaluates the lazy attribute !

>>> hasattr(c, 'a')
evaluated
True
>>> hasattr(c, 'b')
evaluated
True

Let us fix that !

def lazyhasattr(obj, name):
    return any(name in d for d in (obj.__dict__,
                                   obj.__class__.__dict__))

>>> c = ClassA()
>>> lazyhasattr(c, 'a')
True
>>> lazyhasattr(c, 'b')
True

Python check arguments types

2010-06-10T11:25:00+02:00

Decorators help us wrap some routines at function invocation. Here I show a small example that raises TypeError exceptions when given args have unexpected type. Note that it is not pythonic to type check.

This recipe is quite old, as its first pieces appear in PEP-0318 in 2003. A module exists too but it looks neglected...

The (heretic) decorator itself !

def accepts(*argstypes, **kwargstypes):
    def wrapper(func):
        def wrapped(*args, **kwargs):
            if len(args) > len(argstypes):
                raise TypeError("%s() takes at most %s non-keyword arguments (%s given)" % (func.__name__, len(argstypes), len(args)))
            argspairs = zip(args, argstypes)
            for k,v in kwargs.items():
                if k not in kwargstypes:
                    raise TypeError("Unexpected keyword argument '%s' for %s()" % (k, func.__name__))
                argspairs.append((v, kwargstypes[k]))
            for param, expected in argspairs:
                if param is not None and not isinstance(param, expected):
                    raise TypeError("Parameter '%s' is not %s" % (param, expected.__name__))
            return func(*args, **kwargs)
        return wrapped
    return wrapper

Let us decorate !

>>> @accepts(str, arg2=int)
... def f(arg1, arg2=None):
...     pass
...

See it in action...

>>> f('foo')
>>> f('foo', arg2=3)
>>> f()
TypeError: f() takes at least 1 argument (0 given)
>>> f('foo', 'bar')
TypeError: f() takes at most 1 non-keyword arguments (2 given)
>>> f('foo', arg2='bar')
TypeError: Parameter 'bar' is not int
>>> f('foo', arg3='bar')
TypeError: Unexpected keyword argument 'arg3' for f()

Or with classes...

>>> class A(object):
...     pass
...
>>> class B(object):
...     @accepts(object, (str, unicode))
...     def f(self, s):
...         pass
...
...     @accepts(object, A)
...     def g(self, a):
...         pass

>>>
>>> b = B()
>>> b.f(u'foo')
>>> b.f('foo')
>>> b.g(A())
>>> b.g(B())
TypeError: Parameter '<__main__.B object at 0x902466c>' is not A

The same can be applied to return :)

def returns(rtype):
    def wrapper(f):
        def wrapped(*args, **kwargs):
            result = f(*args, **kwargs)
            if not isinstance(result, rtype):
                raise TypeError("return value %r does not match %s" % (result,rtype))
            return result
        return wrapped
    return wrapper

>>> @accepts(str, arg2=int)
... @returns(int)
... def f(arg1, arg2=None):
...     return 0
...

..but kids, don't do this at home :-)

Filesystem Watch with PyQt4

2009-08-14T13:37:00+02:00

I decided to write a generic program that can watch a folder or some files and run a command when changes occur.

Most of the work is done by PyQt4's QFileSystemWatcher.

qfsw = QtCore.QFileSystemWatcher()
qfsw.addPaths([path1, file2 ... ])

QtCore.QObject.connect(qfsw,QtCore.SIGNAL("directoryChanged(QString)"),function)
QtCore.QObject.connect(qfsw,QtCore.SIGNAL("fileChanged(QString)"),function)

Here is the script

# !/usr/bin/env python
#
# Runs a command when a file system change occurs in specified list of paths.
#
# (c) Copyright 2008, Mathieu Leplatre,
#
# This software may be used and distributed according to the terms
# of the GNU Public License, incorporated herein by reference.

import sys, os
from PyQt4 import QtCore, QtGui
import signal
from optparse import OptionParser

# Parse command-line options
usage = """Usage: %prog [options] COMMAND PATHS
Run COMMAND when a file system change occurs in specified list of PATHS.

COMMAND can contain '%s' to refer changed file or directory."""

parser = OptionParser(usage, version="%prog 1.0")
(options, args) = parser.parse_args()
if len(args) < 2:
    parser.print_help()
    sys.exit(2)

# Command
command = args[0]
# List of paths
paths = args[1:]


def onFileSystemChanged(path):
    """
    Callback when file or folder change
    @param path : Changed path
    @type  path : string
    """
    global command
    # Execute command replacing %s with path:
    if '%s' in command:
        command = command % path
    # Run as different process
    print "Run '%s'..." % command
    if os.fork() == 0:
        os.system(command)
        sys.exit(0)


def main():
    app = QtGui.QApplication(sys.argv)

    # Set up file system watcher
    qfsw = QtCore.QFileSystemWatcher()
    qfsw.addPaths(paths)
    QtCore.QObject.connect(qfsw,QtCore.SIGNAL("directoryChanged(QString)"),onFileSystemChanged)
    QtCore.QObject.connect(qfsw,QtCore.SIGNAL("fileChanged(QString)"),onFileSystemChanged)

    # Allow program to be interrupted with Ctrl+C
    signal.signal(signal.SIGINT, signal.SIG_DFL)

    # Qt Main loop
    sys.exit(app.exec_())


if __name__ == "__main__":
    main()

Text extents with Python Cairo

2009-08-10T10:20:00+02:00

I needed this and could not find it. So I share it here (and even better if Google'd index it!)

def textwidth(text, fontsize=14):
    try:
        import cairo
    except Exception, e:
        return len(str) * fontsize
    surface = cairo.SVGSurface('undefined.svg', 1280, 200)
    cr = cairo.Context(surface)
    cr.select_font_face('Arial', cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_BOLD)
    cr.set_font_size(fontsize)
    xbearing, ybearing, width, height, xadvance, yadvance = cr.text_extents(text)
    return width

Colored Output in Console with Python

2008-12-31T13:37:00+01:00

Playing around with ANSI in a color capable terminal.

#!/usr/bin/env python
import sys

BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE = range(8)

#following from Python cookbook, #475186
def has_colours(stream):
    if not hasattr(stream, "isatty"):
        return False
    if not stream.isatty():
        return False # auto color only on TTYs
    try:
        import curses
        curses.setupterm()
        return curses.tigetnum("colors") > 2
    except:
        # guess false in case of error
        return False
has_colours = has_colours(sys.stdout)


def printout(text, colour=WHITE):
        if has_colours:
                seq = "\x1b[1;%dm" % (30+colour) + text + "\x1b[0m"
                sys.stdout.write(seq)
        else:
                sys.stdout.write(text)

A simple demo:

<code python>
#
# Test
#
printout("[debug]   ", GREEN)
print("in green")
printout("[warning] ", YELLOW)
print("in yellow")
printout("[error]   ", RED)
print("in red")

Automatic blog publishing using Python and XML-RPC

2008-12-01T13:37:00+01:00

The following piece of code collects links in a Pligg database (but could be any kind of source like RSS feed...) and automatically builds and publishes entries in a blog using XML-RPC.

A small database class :

class DatabaseAPI(object):
    def connect(self, host="localhost", port=3306, user="root", passwd="root123", db="mysql"):
        import MySQLdb
        self.conn = MySQLdb.connect(host, user, passwd, db)

    def disconnect(self):
        self.conn.close()

    def fetchall(self, sqlquery):
        cursor = self.conn.cursor()
        cursor.execute(sqlquery)
        return cursor.fetchall()

A Blog publishing class :

class BlogAPI(object):
    def __init__(self, urlapi, username, password):
        import xmlrpclib
        self.xmlrpclib = xmlrpclib
        try:
            self.server = xmlrpclib.ServerProxy( urlapi )
        except:
            raise Exception( "Could not connect to %s" % url )
        self.username = username
        self.password = password

    def newPost(self, title, description, blogid = '1', publish = True ):
        if description == "":
            description = "<em>empty entry</em>"
        post = {}
        post['title'] = title
        post['description'] = description
        try:
            r = self.server.metaWeblog.newPost( blogid, self.username, self.password, post, publish )
            return r
        except self.xmlrpclib.Fault, fault:
            raise Exception( fault.faultString )

Build the SQL query : all entries of current week

import datetime
dt = datetime.timedelta(weeks=-1)
today = datetime.date.today()
agelimit = today + dt
stmt = """
SELECT link_url, link_url_title
FROM pligg_links
WHERE link_status = 'published' AND link_published_date > "%s"
ORDER BY link_published_date DESC""" % agelimit.strftime("%Y%m%d000000")

Put everything together :

db = DatabaseAPI()
db.connect(user="user", passwd="pass", db="name")
entries = db.fetchall(stmt)
db.disconnect()

body = """
<ul>"""
for entry in entries:
    body += """
    <li><a href="%s">%s</a></li>""" % (entry[0], entry[1])
body += """
</ul>"""

blog = BlogAPI("http://yourblog/xmlrpc/", "user", "pass")
title = "Links week #%s" % today.strftime("%W")
blog.newPost(title, body)

Another success story about Ruby On Rails working with Apache

2008-08-01T09:12:00+02:00

This procedure does not guarantee anything and should be considered approximate. However this should be fairly enough for someone familiar with Apache environments.

Ruby environment

We install Ruby via apt-get

sudo apt-get install ruby libzlib-ruby rdoc irb

At this point, you can run Ruby scripts like you did with Python or Perl.

Gems

Gems are like perl's CPAN or PHP's PEAR. We install it from source in order to enjoy the whole Gems repository (instead of being limited to packaged gems only).

wget "http://rubyforge.org/frs/download.php/38646/rubygems-x.x.x.tgz"
tar -xvzf rubygems-x.x.x.tgz
rm rubygems-x.x.x.tgz
cd rubygems-x.x.x
sudo ruby setup.rb
cd ..
rm -r rubygems-1.2.0
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem

(DO NOT use sudo ruby rubygemsx.x.x/setup.rb)

sudo gem update --system

We will install additionnal applications... those depend on your needs. The Gems are compiled on the fly, therefore development packages are usually required along.

Ruby on Rails

sudo gem install rails --include-dependencies

Mysql support

sudo apt-get install libmysql-ruby libmysqlclient15-dev
sudo gem install mysql

ImageMagick support

sudo apt-get install librmagick-ruby1.8 libmagick9-dev
sudo gem install rmagick

Integrate with Apache

sudo apt-get install apache2-prefork-dev

Enable additionnal modules

a2enmod rewrite
a2enmod suexec
a2enmod include

I might have missed some.. Some tutorials recommend Fast-CGI.

Install Phusion Passenger (mod_rails)

Usually, RubyOnRails has its own web server (Mongrel) on port 3000. It is also quite common to have a cluster of processes with load balancing and Apache proxy...

But you may want to do something very simple that just integrates within Apache. Here comes Phusion Passenger!

sudo gem install passenger
sudo passenger-install-apache2-module

(following the instructions, or look at the [[http://www.modrails.com/documentation/Users%20guide.html|user guide]]).

At the end, the wizard tells you to add some lines in httpd.conf. I recommend the following method instead, which splits those lines into a module that you can enable / disable.

Create two files : * /etc/apache2/mods-available/mod_rails.load

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.0.2/ext/apache2/mod_passenger.so

/etc/apache2/mods-available/mod_rails.conf

PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.0.2
PassengerRuby /usr/bin/ruby1.8

Enable this new module

sudo a2enmod mod_rails

Create your VirtualHost

The DocumentRoot must point to the public folder of your Ruby On Rails application.

If your Ruby application is alone, your apache site will be something like this

<virtualhost *:80>
    ServerName yourapp
    DocumentRoot /var/rails/yourapp/public/
    ErrorLog /var/rails/yourapp/log/apache.log

    <directory /var/rails/yourapp/public>
       Options ExecCGI FollowSymLinks
       AddHandler cgi-script .cgi
       AllowOverride all
       Order allow,deny
       Allow from all
    </directory>
</virtualhost>

If you want it a subfolder of your current DocumentRoot, look at this.
The simplest for me was to setup a sub-domain. Don't forget to update your DNS information.

Relax ! Restart Apache and that's it !

Apply Debian patches step-by-step

2008-08-01T09:12:00+02:00

I thought it may be relevant to share the history of commands I used to apply a patch and submit it to launchpad.

0) Get the tools

You'll need a working PGP key

Install the necessary tools

sudo apt-get install devscripts dpatch fakeroot dh-make

(I might have forgotten some...) :)

1) Get the files

Get the last package source, from the package page : http://packages.ubuntu.com/hardy/exaile

dget -x http://archive.ubuntu.com/ubuntu/pool/universe/e/exaile/exaile_0.2.11.1-0ubuntu3.dsc

Get the patch file

wget http://launchpadlibrarian.net/9466876/gui_track_filter.patch

Step in the package code
```
cd exaile-0.2.11.1/
```

2) Apply the patch

Have a look at the list of patches in the debian/patches folder

ls debian/patches/

00list
01_fix_makefile_for_pysupport.dpatch
02_files-on-share-dir.dpatch
03_fix_mmkeys.dpatch

Create yours respecting dpatch filenames pattern (without extension)
```
dpatch-edit-patch 04_fix_search_exit
```
You now have a shell in the /tmp folder.

Apply the patch file

patch -p0 < $HOME/Desktop/gui_track_filter.patch

Exit the shell

3) Describe your changes

Add your dpatch name in the 00list file
```
nano debian/patches/00list
```
Modify Debian changelog using
```
dch -i
```
Your name and signing information will be automatically added.
Look at previous descriptions and respect their structure.
Include launchpad bug number with (LP: #number) string
```
exaile (0.2.11.1-0ubuntu4) hardy; urgency=low
```

debian/patches/04_fix_search_exit.dpatch

debian/patches/00list:
- Fix clean search terms on exit (LP: #95860)

-- Mathieu Leplatre <xxxx@gmail.com>  Tue, 27 May 2008 10:45:42 -0300

4) Create debdiff

Create the dsc and diff files using
```
debuild -S
cd ..
```
Step in the package code

Create the debdiff file using

debdiff exaile_0.2.11.1-0ubuntu3.dsc exaile_0.2.11.1-0ubuntu4.dsc > exaile_0.2.11.1-0ubuntu4.debdiff

Have a look at it, it should include all modifications of previous steps

diff -Nru exaile-0.2.11.1/debian/changelog exaile-0.2.11.1/debian/changelog
--- exaile-0.2.11.1/debian/changelog        2008-05-27 10:54:56.000000000 -0300
+++ exaile-0.2.11.1/debian/changelog        2008-05-27 10:54:56.000000000 -0300
@@ -1,3 +1,11 @@
+exaile (0.2.11.1-0ubuntu4) hardy; urgency=low
+
+  * debian/patches/04_fix_search_exit.dpatch,
+    debian/patches/00list:
+    - Fix clean search terms on exit (LP: #95860)
+
+ -- Mathieu Leplatre <xxxx@gmail.com>  Tue, 27 May 2008 10:45:42 -0300
+
 exaile (0.2.11.1-0ubuntu3) hardy; urgency=low

   * debian/patches/03_fix_mmkeys.dpatch,
diff -Nru /tmp/RIBRnUlXkn/exaile-0.2.11.1/debian/patches/00list /tmp/XQpuhOBOst/exaile-0.2.11.1/debian/patches/00l
ist
--- exaile-0.2.11.1/debian/patches/00list   2008-05-27 10:54:56.000000000 -0300
+++ exaile-0.2.11.1/debian/patches/00list   2008-05-27 10:54:56.000000000 -0300
@@ -1,3 +1,5 @@
 01_fix_makefile_for_pysupport
 02_files-on-share-dir
 03_fix_mmkeys
+04_fix_search_exit
+
diff -Nru /tmp/RIBRnUlXkn/exaile-0.2.11.1/debian/patches/04_fix_search_exit.dpatch /tmp/XQpuhOBOst/exaile-0.2.11.1
/debian/patches/04_fix_search_exit.dpatch
--- exaile-0.2.11.1/debian/patches/04_fix_search_exit.dpatch        1969-12-31 21:00:00.000000000 -0300
+++ exaile-0.2.11.1/debian/patches/04_fix_search_exit.dpatch        2008-05-27 10:54:56.000000000 -0300
@@ -0,0 +1,27 @@
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## 04_fix_search_exit.dpatch by Mathieu Leplatre <xxxx@gmail.com>
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: Patch to clean search terms on exit
+
+@DPATCH@
+diff -urNad exaile-0.2.11.1~/xl/gui/main.py exaile-0.2.11.1/xl/gui/main.py
+--- exaile-0.2.11.1~/xl/gui/main.py        2007-11-07 13:12:52.000000000 -0300
++++ exaile-0.2.11.1/xl/gui/main.py 2008-05-27 10:37:36.000000000 -0300
+@@ -1659,8 +1659,16 @@
+         queuefile = xl.path.get_config('queued.save')
+         if os.path.isfile(queuefile):
+             os.unlink(queuefile)
++
+
+         if self.player.current: self.player.current.stop()
++
++        # Clear the search filter so that the entire playlist is saved
++        self.tracks_filter.set_text('')
++        try:
++            self.on_search()
++        except:  # In case we're quitting before the playlist loaded
++            pass
+
+         for i in range(self.playlists_nb.get_n_pages()):
+             page = self.playlists_nb.get_nth_page(i)

Static build of Cairo and librsvg

2008-07-01T11:25:00+02:00

Why ?

Convert SVG files to PDF or PNG, with full Unicode support (right-to-left languages), transparency, gradients, PDF image compression, ...
Cairo and librsvg are the best in town.
Cairo and librsvg are very modern libraries which became famous only in the past 3 years. Thus, GNU/Linux distributions do not always have recent versions and full capabilities.
A static build does all the bindings to libraries at compile time, which hence removes specific versions dependencies. (this method has many drawbacks but can help sometimes)

The program : svgconvert.c

This program is a merge of Carl Worth's svg2pdf and svg2png

/*
* Copyright © 2005 Red Hat, Inc.
* Copyright © 2006 Red Hat, Inc.
* Copyright © 2007 Red Hat, Inc.
*
* Permission is hereby granted, free of charge, to any person
* obtaining a copy of this software and associated documentation
* files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use, copy,
* modify, merge, publish, distribute, sublicense, and/or sell copies
* of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*
* Authors: Kristian Høgsberg <krh@redhat.com>
* Carl Worth <cworth@redhat.com>
* Behdad Esfahbod <besfahbo@redhat.com>
* Mathieu Leplatre <contact@mathieu-leplatre.info>
*/

#include <stdio.h>
#include <stdlib.h>
#include <glib/gprintf.h>
#include <librsvg/rsvg.h>
#include <librsvg/rsvg-cairo.h>

#include <cairo-pdf.h>

#define FAIL(msg) \
do { fprintf (stderr, "FAIL: %s\n", msg); exit (-1); } while (0)

#define PIXELS_PER_POINT 1

#define PDF 0
#define PNG 1

int main (int argc, char *argv[])
{
    GError *error = NULL;
    RsvgHandle *handle;
    RsvgDimensionData dim;
    double width, height;
    const char *filename = argv[1];
    const char *output_filename = argv[2];
    cairo_surface_t *surface;
    cairo_t *cr;
    cairo_status_t status;
    int mode;

    if (argc != 3)
        FAIL ("usage: svgconvert input_file.svg output_file");

    mode = PDF;
    if (g_str_has_suffix (g_ascii_strdown (output_filename, -1), ".png")) {
        mode = PNG;
    }

    g_type_init ();

    rsvg_set_default_dpi (72.0);
    handle = rsvg_handle_new_from_file (filename, &error);
    if (error != NULL)
        FAIL (error->message);

    rsvg_handle_get_dimensions (handle, &dim);
    width = dim.width;
    height = dim.height;

    if (mode == PNG) {
        surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32, width, height);
    }
    else {
        surface = cairo_pdf_surface_create (output_filename, width, height);
    }

    cr = cairo_create (surface);

    rsvg_handle_render_cairo (handle, cr);

    status = cairo_status (cr);
    if (status)
        FAIL (cairo_status_to_string (status));

    if (mode == PNG) {
        cairo_surface_write_to_png (surface, output_filename);
    }

    cairo_destroy (cr);
    cairo_surface_destroy (surface);

    return 0;
}

Build with shared librairies

You would just do :

gcc `pkg-config --cflags --libs librsvg-2.0 cairo-pdf` -o svgconvert svgconvert.c

which creates a binary of 9.0K.

Build with static librairies

First install the development versions of the packages, to make sure you have all /usr/lib/*.a mentioned below.
Use this Makefile, which creates a binary of 5.9M. It was tested on Ubuntu 8.04 which comes with Gnome 2.22, librsvg 2.22 and Cairo 1.6.0.

ALL=svgconvert

MYCFLAGS=`pkg-config --cflags librsvg-2.0 cairo-pdf`
LDFLAGS=`pkg-config --libs librsvg-2.0 cairo-pdf freetype2 fontconfig pango pangoft2 pangocairo  cairo-ft libthai datrie libgsf-1 gnome-vfs-2.0 libcroco-0.6 libpcre pixman-1 libpng libxml-2.0`
MYLDFLAGS=$(LDFLAGS) /usr/lib/libgio-2.0.a  /usr/lib/libglib-2.0.a /usr/lib/libselinux.a /usr/lib/libexpat.a /usr/lib/libfreetype.a /usr/lib/libbz2.a /usr/lib/libjpeg.a /usr/lib/libtiff.a /usr/lib/libbz2.a /usr/lib/libz.a /usr/lib/libm.a

all: $(ALL)

%: %.c
    $(CC) $^ -pthread $(MYCFLAGS) -static $(MYLDFLAGS) -o $@

clean:
    rm -f $(ALL) *.o

To check if pkg-config knows about a specific library :

$ pkg-config --list-all | grep vfs
gnome-vfs-sharp-2.0           GnomeVfs - GnomeVfs
gnome-vfs-2.0                 gnome-vfs - The GNOME virtual file-system libraries
gnome-vfsmm-2.6               gnome-vfsmm - C++ wrapper for gnome-vfs
gnome-vfs-module-2.0          gnome-vfs-module - The GNOME virtual file-system module include info

To check if a library has a specific symbol, use the nm command :

$ nm /usr/lib/libexpat.a | grep XML_SetStart
000001c0 T XML_SetStartCdataSectionHandler
00000240 T XML_SetStartDoctypeDeclHandler
00000150 T XML_SetStartElementHandler
000002a0 T XML_SetStartNamespaceDeclHandler

Download

References

http://cairographics.org
http://librsvg.sourceforge.net
Thanks for the precious help of Carl Worth on #cairo at irc.freenode.net, Zugzwang and nvteighen on http://ubuntuforums.org