dadadi blog

To content | To menu | To search


Entries feed - Comments feed


Thursday, July 28 2011

Screencast de l'utilisation de syj.

Je viens de réaliser une capture vidéo de l'utilisation de Show Your Journey, afin de montrer les possibilités de l'outil:

Sunday, April 10 2011

More playing with srtm data: altiph

After importing srtm data into postgis, it was time to query and use that data.

route-altitude-profile can do that, but this time, I needed a php tool. So, I wrote altiphp library.

When given longitude and latitude, altiphp queries the database and returns altitude. Actually, it queries the altitude for nearest available points, and performs a bilinear interpolation (I took the idea from route-altitude-profile). When given an array of points, altiphp returns an array of altitude.

altiphp can add points to give a more accurate route profile. For example, if you have two points at the top of two mountains, altiphp knows how to add points between them each one separated of about 90m from the next one (90m is srtm resolution). You get then an accurate route profile going down to the valley and then climbing back to the other mountain. This behavior is of course optional.

Altiphp also integrates with gisconverter.php. When gisconverter is available, altiphp can handle geometries directly.

If you have not imported srtm data in a postgis database, you can use non-postgis mode. In this mode, altiphp downloads data files form usgs server, optionally stores them for future retrieval, and parses them to extracts data. This works fine, gets the same results as postgis mode, but parsing the file is really slow and each file, once parsed, takes about 300Mo in memory. So, using this method is not a good idea for a website, even a small one, but this can be helpful in some cases, for example for a seldom used local application or for testing.

altiphp needs php 5.3, is licensed under modified bsd license and is available at github.

Sunday, April 3 2011

srtm2postgis improved

In 2000, NASA Space Shuttle Endeavour had a 11 days mission named SRTM. It was about acquiring topographic information of earth surface, from 56°S to 60°N. Data has been released in public domain. It's one of the main sources for free global topographic data.

I want to play with these data, and the first part of the game is putting data into a postgis database. Fortunately, this had already been done as part of Route altitude profiles SRTM. It's a 2008 Google Summer of Code project by Sjors Provoost. Srtm2postgis is a small utility that downloads srtm data, and loads it into postgis.

Download urls for SRTM had changed, so original srtm2postgis does not work as-is. Fortunately, a fork with updated urls is available on github.

So, I started tickering with srtm2postgis. I fixed a few minor bugs. For example, srtm data is split into 14546 files. 14540 of them have all upper case name, and 6 have lower letters name. Those 6 files could not be downloaded. I also fixed some typos in the README, and a few failing tests, but overall, the script was running smoothly.

Then I spent some time trying to improve import time. During its summer of code, Sjors Provoost reported importing data at one tile a minute speed. A 75 tiles import (44°N to 50°N and 3°E to 7°W) took me 80 minutes. This about the same as Sjors Provoost result.

To connect to postgres, srtm2postgis uses pygresql, but also uses psycopg because of its copy_from method. But copy_from method was ultimately not used, and a psql subprocess was launched to run the copy instead. A comment explained that psycopg copy_from method was hanging. I tried to use it and it was not hanging. As the comment was nearly 3 years old, I assume this is a now fixed bug. I'm also glad this was clearly documented. Otherwise, this workaround would have been impossible to understand. With this modification, my 75 tiles import was only 49 minutes. I really don't understand why this improves performances so much, but I'm happy about it.

Then, I totally remove pygresql, and only used psycopg. This allows me, first to open only one connection, then, to run everything in a single transaction, and commit everything at the end. Import was now 42 minutes.

After improving database-related performances, I tried to speed up things in other parts of the code. SRTM data is read by gdal library. It returns a 1200 * 1200 arrays of numpy.int16. Converting int16 to string is something really slow. On a 1200 * 1200 grid, many points have the same altitude. So, I use a cache to store string representations of numpy.int16 altitudes. This resulted in a major performance improvement. My 75 tile import is now 28 minutes.

I also use string buffer instead of a temporary file to store data to copy, but this does not improve performance significantly. I also tried to remove table primary key constraint and recreate it after import. I thought it would improve copy speed. But actually, it damaged performances a lot.

Eventually, I run VACUUM ANALYZE after import. This takes 10 minutes, but this will probably improve future performances.

I could improve import time from 80 to 28 minutes (38 minutes with VACUUM ANALYZE), with is nearly 3 times faster!!! My updated srtm2postgis version is available on github.

Thursday, November 25 2010

syj is repaired

syj was broken for about ten days. The hardware had multiple failure: RAM wich could not be replaced and hard drive. So, we moved it to another server, still in cr@ns. Backups stopped working a few days before syj stopped working. Then, some paths have been lost (about hundred), a probably a few  accounts also. Sorry for the trouble, and many thanks to Jocelyn from cr@ns for helping me restoring this service.

syj est réparé

syj était cassé depuis une dizaine de jours. En fait, la machine qui l'hébergeait était tombé en panne (RAM irremplaçable plus disque dur). Du coup, il a été déplacé sur une autre machine, toujours au cr@ns. Les sauvegardes sont tombées en panne quelques jours avant le site, ce qui fait que plusieurs chemins ont été perdus (une centaine environ) et probablement des comptes utilisateurs aussi. Désolé pour ce dérangement, et merci à Jocelyn du cr@ns de m'avoir aidé à remettre le site en état.

Tuesday, September 21 2010

import dans OpenStreetMap des points d'eau de la ville de Paris

Ce week-end, en plus de finaliser Show Your Journey, j'ai réalisé pour OpenStreetMap, l'import des points d'eau recensés par Eau de Paris.

Il y a environ un an, j'avais constaté que le site de la régie Eau de Paris fournissait des documents recensant plusieurs centaines de points d'eau. J'avais alors demandé à la régie l'autorisation de les importer dans OpenStreetMap. La régie avait donné son accord, mais ne disposait pas des données brutes, ce qui a empêché l'import. Au cours de l'été dernier, en retournant sur le site, j'ai constaté qu'il y avait maintenant une carte numérique des fontaines de Paris. J'ai alors recontacté la régie qui m'a confirmé que les données pouvaient être importées dans OpenStreetMap.

Je me suis attelé à l'import de ces données. En en discutant sur la mailing liste francophone, nous avons constaté que les données comportaient plusieurs erreurs, et un contributeur a proposé l'algorithme suivant:

Pour chaque point :

  • si il y a un point déjà indiqué à moins de 50 mètres, ne pas importer
  • si il n'y a pas de point à moins de 100 mètres, importer en ajoutant "FIXME=position approximative"
  • si le point d'eau le plus proche est entre 50 et 100 mètres, voir au cas par cas, peut-être grâce à la carte dont j'ai donné le lien

Après plusieurs essais infructueux, j'ai réalisé l'import lors des changeset 5819089, 5821219, et 5819089.

J'ai mis en place une carte des points d'eau qui permet de visualiser les points d'eau potable en France.

Au final, il manque beaucoup de points d'eau sur la carte, et il y en quelques uns qui sont probablement mal placés. Néanmoins, je pense qu'une bonne partie des 320 points d'eau ajoutés est correctement placée.

Cet données sont particulièrement intéressantes pour les randonneurs, les touristes, ou toutes les personnes qui passent plusieurs heures dans Paris en extérieur. J'espère que ces données, aujourd'hui incomplètes et parfois incorrectes, vont être améliorées prochainement, et j'espère aussi que beaucoup d'autres points potables vont être répertoriés en banlieue et dans toute la France.

Monday, September 20 2010

syj: a website to share routes

I spent a lot of times this summer writting a website to share routes. It's now done. That does not mean I won't improve it. That means it's stable, so I can make it public. That also means it has all the functionalities I need for my own usage.

Project name is syj. It means Show Your Journey. With syj, you can create and publish routes. So you can organize hiking, show clever bicycle rides, or any route you want. You can also get your routes as gpx or kml files.You can, if you have created an account, manage routes you have created: modify or delete them. Finally, you can duplicate a route before modifying it. This is really useful when you want many similar routes.

Syj is available at

OpenStreetMap is used for background map. Website is hosted by Cr@ns. That organization already hosts many project related to OpenStreetMap.

Website sources are available under AGPL license. For this project, I also wrote many standalone libraries available on github. Those libraries are available under modified bsd license.

I hope you'll find syj as useful as I do.

syj: site de partage d'itinéraire

J'ai passé une partie de l'été à écrire un site de partage d'itinéraires. Il est maintenant fini. Cela ne veut pas dire que je ne vais pas continuer à l'améliorer. Cela veut dire d'une part, qu'il est suffisamment stable pour que je le rende publique. D'autre part, qu'il comporte toutes les fonctionnalités que j'estimais indispensables pour mon usage.

Le projet s'appelle syj, ce qui veut dire: Show Your Journey. Il permet donc de créer et de publier des itinéraires. Cela peut servir à préparer des randonnées, à montrer des itinéraires cyclistes malins, ou n'importe quoi d'autre. On peut également exporter les itinéraires sous forme de fichiers gpx ou kml. On peut aussi, si on crée un compte, gérer les trajets qu'on a créés, c'est à dire en récupérer la liste, les modifier ou les supprimer. Enfin, on peut dupliquer un itinéraire avant de le modifier, ce qui est très utile si l'on veut plusieurs variantes d'un itinéraire.

Syj est disponible à l'adresse

Le fond de carte utilisé est celui d'OpenStreetMap. Le site est hébergé par le Cr@ns, qui héberge déjà plusieurs projets liés à OpenStreetMap.

Les sources du site sont disponibles sous licence AGPL. J'ai également écrit, pour ce projet, plusieurs librairies indépendantes qui sont disponibles sur github. Ces librairies sont sous licence bsd modifiée.

J'espère que syj vous sera autant utile qu'à moi.

Thursday, May 6 2010

OpenLayers class to draw and modify a path

To display a map on a webpage, you can often use directly the api of a commercial provider (such as cloudmade Web Maps Lite or google maps api). The other solution is to use OpenLayers, a free javascript library to display embed maps in a web document.

With OpenLayers, it's quite easy to draw a path on the map. But unfortunately, once a point has been added to the path, it's not possible to modify or delete it. This is possible with web maps lite, as can be seen in this distance calculator. I really like the ergonomy of the web maps lite path creator, and wanted to have something similar for OpenLayers.

So, I wrote ModifiablePath, a class to improve OpenLayers path creator. So, when creating a path with this handler, you can move, delete or even add new points. I've put a demo online, and code is available at github.