Mathieu Leplatrehttps://blog.mathieu-leplatre.info/2021-10-18T00:00:00+02:00Design for the Long Term2021-10-18T00:00:00+02:002021-10-18T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2021-10-18:/design-for-the-long-term.html<p>I really enjoyed reading the debates around choosing <a class="reference external" href="http://boringtechnology.club">boring</a> or <a class="reference external" href="https://lucjan.medium.com/choose-exciting-technology-e735bba08acc">exciting</a> technology.
In my opinion, there is another interesting point, beyond new or old, that is about longevity, or durability.</p>
<p>Some software can be deliberately ephemeral, like a mobile app for a specific event. But usually we want software to …</p><p>I really enjoyed reading the debates around choosing <a class="reference external" href="http://boringtechnology.club">boring</a> or <a class="reference external" href="https://lucjan.medium.com/choose-exciting-technology-e735bba08acc">exciting</a> technology.
In my opinion, there is another interesting point, beyond new or old, that is about longevity, or durability.</p>
<p>Some software can be deliberately ephemeral, like a mobile app for a specific event. But usually we want software to last as long as possible, like a Web API with millions of clients.</p>
<p>As I outlined in <a class="reference external" href="https://blog.mathieu-leplatre.info/about-maintenance-mode.html">a previous post</a>, long term maintenance takes all sorts of efforts. And logically, applications with the least maintenance cost are more likely to last longer.</p>
<p>So, what makes software durable? How can we take long term maintenance into consideration during the design phase?</p>
<p><strong>tl;dr</strong>: <em>Think of your software architecture as a multi-generational building. Simplicity and adaptability are key to guarantee long term survival in a constantly changing environment. Designing for the long term requires experience and judgement, so focus on values and people more than technologies.</em></p>
<div class="section" id="typical-lifecycle">
<h2>Typical Lifecycle</h2>
<p>From my experience, the typical phases of software engineering are:</p>
<ol class="arabic simple">
<li>Define the needs and requirements</li>
<li>Assemble a team</li>
<li>Design, develop and deploy</li>
<li>Validate conformity</li>
<li>Dismantle the team</li>
<li>Run and maintain forever</li>
</ol>
<p>The last phase is the critical one: you go from a top priority project, backed by a whole team with daily activities, to a low priority project, with a few people involved only sporadically. And it can last years.</p>
<p>What if we would build software with this critical <em>last phase</em> in mind?</p>
<blockquote>
Engineering is 99% managing legacy, 1% deciding on what legacy you want to have in the future. ─ <a class="reference external" href="https://twitter.com/rakyll/status/1440067121522692104">@rakyll, 20 sept. 2021</a></blockquote>
</div>
<div class="section" id="elegance-vs-adaptability">
<h2>Elegance vs. Adaptability</h2>
<p>A CTO recently reminded his teams <a class="footnote-reference" href="#preply" id="footnote-reference-1">[1]</a> that they should «<em>build boxes, not arches</em>». He uses this metaphor to remind programmers that <strong>elegance in their design should not delay the production of value</strong>. Arches are more elegant than blocks, but they show value only when the top key stone is put. Unlike arches, blocks can be easily rearranged and replaced over time, helping adaptability and market fit.</p>
<p>I could reuse this metaphor, and say that blocks are more likely to last over time than arches.</p>
<p>In the book <a class="reference external" href="https://en.wikipedia.org/wiki/How_Buildings_Learn">How buildings learn</a>, Stewart Brand shows that buildings which were built quickly to look impressive have a terrible maintenance record.</p>
<p>Engineers who want exciting technology in their curriculum are like architects who want beautiful buildings in their portfolio. Do they really care how practical it will be over the years? Is it built to last or to shine?</p>
<div class="figure align-center">
<img alt="Notre-Dame Cathedral reconstruction" src="https://blog.mathieu-leplatre.info/images/long-term-notre-dame.jpg" />
<p class="caption">Notre-Dame Cathedral reconstruction © habibeh Madjdabadi architecture Studio</p>
</div>
<p>As Joel famously wrote 21 years ago, «<em>rewriting the code from scratch is the single worst strategic mistake that any software company can make</em>»: <a class="footnote-reference" href="#fjoel" id="footnote-reference-2">[2]</a></p>
<blockquote>
Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand.</blockquote>
<p>Do we want to build, demolish, and rebuild? Or should we design software like a building where engineers and operators have to live for several generations?</p>
<p>When aiming for eternal design, you must focus on an architecture where it is viable to «<em>fail fast, fail small, fail often, learn, and reiterate</em>».</p>
</div>
<div class="section" id="pace-layered-design">
<h2>Pace-layered Design</h2>
<p>How do you balance stability and adaptability with innovation?</p>
<p>I would advise you to spend 30' to watch <a class="reference external" href="https://www.youtube.com/watch?v=ZSaWdp833YM">how buildings are built for change</a>. The parallel with software is very inspiring. You'll think of your application code as a structure with <strong>solid, simple, slow-moving foundations</strong>, whose upper blocks can be replaced and remodeled, fostering innovation. You want to be able to repaint bedrooms or replace windows without jeopardizing life in it!</p>
<div class="figure align-center">
<img alt="“Pace Layers” diagram from Stewart Brand’s book “The Clock of the Long Now”" src="https://blog.mathieu-leplatre.info/images/long-term-pace-layering.jpg" />
<p class="caption">Everlasting, slow, and powerful core layers, versus ephemeral, fast, and weak skin layers</p>
</div>
<p>The firm Gartner applied this concept of <a class="reference external" href="https://en.wikipedia.org/wiki/Shearing_layers">shearing layers</a> to software, where each application layer obeys to a different strategy:</p>
<ul class="simple">
<li><strong>Powerful lower parts move at slow pace</strong>: Change is risky and expensive, strict and only incremental (APIs, databases, ...)</li>
<li><strong>Middle parts adapt to business</strong>: Change occurs more often, following the company's business processes and specifities (custom integrations, SaaS, cloud...)</li>
<li><strong>Top parts move fast</strong>: Change requires less governance and planning, encouraging experimentation (proof-of-concepts, frontend, ...)</li>
</ul>
<p>Uncle Bob's also advocates the same ideas in his book about <a class="reference external" href="https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html">Clean Architecture</a>. Breaking the myth that restarting from scratch is a solution, he expands familiar principles like <a class="reference external" href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> and <a class="reference external" href="https://en.wikipedia.org/wiki/Separation_of_concerns">SoC</a> to system architecture.</p>
</div>
<div class="section" id="wear-and-tear-parts">
<h2>Wear and Tear Parts</h2>
<p>Compare a classic reflux valve with the <a class="reference external" href="https://www.youtube.com/watch?v=suIAo0EYwOE">valve designed by Nikola Tesla</a>. One has at least two moving parts, a spring and a «stopper», the other one has none, which gives it an almost infinite lifetime.</p>
<div class="figure align-center">
<img alt="Tesla Valve | The complete physics - https://www.youtube.com/watch?v=suIAo0EYwOE" src="https://blog.mathieu-leplatre.info/images/long-term-tesla-valve.jpg" />
</div>
<p>When your application handles a request, <strong>what moving parts are involved</strong>?</p>
<p>Compare the moving parts of a statically generated website, with one powered by Wordpress, relying on a MySQL database, a Web server running PHP, with extensions and system libraries, an Admin UI in React, a Varnish cache... With the former, you write your articles in Markdown, run a command once to generate the HTML pages, and simply serve the static content to millions of readers, without any moving part. No monitoring, no security patches, and resilient to system upgrades. In twenty years, even if the script that generates the files doesn't run anymore, the website will still be online.</p>
<p>Take MDN for example. It used to be powered by a <a class="reference external" href="https://github.com/mdn/kuma">Django application</a> with thousands lines of code. Instead of this complex wiki application, the team replaced it with Markdown files in a <a class="reference external" href="https://github.com/mdn/content">Github repo</a>, and some commands to publish the HTML website.</p>
<p>Similarly, <a class="reference external" href="https://youtu.be/vUCr1oTtaKA?t=965">Tim showed us</a> how they fixed the BBC News app scaling issues by prerendering content into static files on Amazon S3, instead of relying directly on BBC Web services.</p>
<p>It's not all or nothing, just make sure to <strong>think about how much resources, machines and people, will be required to operate this service years after the original team built it</strong>.</p>
</div>
<div class="section" id="conviviality">
<h2>Conviviality</h2>
<p>In 1973, Ivan Illich wrote <a class="reference external" href="https://en.wikipedia.org/wiki/Tools_for_Conviviality">Tools for Conviviality</a>. This book contains radical ideas about progress, emancipation, and technology. Illich explains how it is crucial to be able to share, understand, repair, and modify our tools in order to reach independence over the long term.
It is not a coincidence if the book had an immense influence of the first computer hackers <a class="footnote-reference" href="#illich" id="footnote-reference-3">[3]</a>.</p>
<p>Which of the appliances that you own are repairable?</p>
<p>The <a class="reference external" href="https://www.ifixit.com/Manifesto">iFixit Manifesto</a> says it clearly: «<em>If you can't fix it, you don't own it</em>».</p>
<p>Not so long ago, if your <a class="reference external" href="https://en.wikipedia.org/wiki/Citro%C3%ABn_2CV">Citroën 2CV</a>'s belt broke in the middle of nowhere, you could use your <a class="reference external" href="https://en.wikipedia.org/wiki/Fully_fashioned_stockings">nylon stockings</a> as a temporary spare part and make it home.</p>
<div class="figure align-center">
<img alt="Citroën Ami 6" src="https://blog.mathieu-leplatre.info/images/long-term-ami-6.jpg" />
<p class="caption">One of the old cars of my colleague Florian, long term Gecko hacker :)</p>
</div>
<p>If you build your whole application (or business!) on the unique features of a cloud solution (Spanner, Firebase, ...) or on the latest trendy all-in-one framework, you obtain the opposite of conviviality! <strong>Avoid vendor lock-in!</strong> You will need to use their specific screwdrivers to repair it, you won't find standard spare parts, and years later planned obsolescence will threaten you. Avoid «easy» (no effort), and go for «simple» (no complexity).</p>
<p><strong>Simple is understandable. Simple scales. Simple always wins over the long term</strong>. <a class="reference external" href="https://www.youtube.com/watch?v=OuF9weSkS68">Think more like ARM, and less like Intel :)</a></p>
</div>
<div class="section" id="beware-of-over-engineering">
<h2>Beware of Over-engineering</h2>
<p>Most intelligent people produce <strong>intelligent designs</strong>. The reality is that their creations eventually become extremely fragile, costly to maintain, or they just <strong>don't survive the company's successive turnovers</strong>. Because most of us aren't super smart, are sometimes sleepy, often very lazy, or with scars from the past.</p>
<p>I recently fell upon <a class="reference external" href="https://github.com/taskcluster/taskcluster/blob/5a25a717/infrastructure/tooling/src/build/tasks/taskcluster-proxy.js#L65-L79">this example</a>, where a piece of JavaScript will produce a static <tt class="docutils literal">Dockerfile</tt>. What's wrong with static files? It's not because you <em>can</em> do it, that you <em>have</em> to do it!</p>
<p>Discriminating intelligence is crucial.</p>
<p>In the past, we would build Web services as big monoliths. <a class="reference external" href="https://en.wikipedia.org/wiki/Microservices">Microservices</a> emerged in early 2010s, mainly as a response to the needs of continuous deployment, following the <a class="reference external" href="https://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>. After a decade of hype, where engineers would build all sort of applications using dozens of tiny services, in the late 2010s, monolithic applications <a class="reference external" href="https://www.craigkerstiens.com/2019/03/13/give-me-back-my-monolith/">became</a> <a class="reference external" href="https://shopify.engineering/shopify-monolith">acceptable</a> <a class="reference external" href="https://www.bennadel.com/blog/3944-why-ive-been-merging-microservices-back-into-the-monolith-at-invision.htm">again</a>. And as people with discerning abilities <a class="reference external" href="https://martinfowler.com/bliki/MonolithFirst.html">sensed it</a> early, you should just «<em>start with a monolithic approach and move to microservices if needed</em>».</p>
</div>
<div class="section" id="skin-in-the-game">
<h2>Skin in the Game</h2>
<p>Like factories and towns who release their waste in the river, assuming others downstream will handle it, some engineers and managers consider themselves as <em>builders</em>, and will let future <em>fixers</em> handle maintenance over time.</p>
<p>Taleb explores this mistake in his book about <a class="reference external" href="https://en.wikipedia.org/wiki/Antifragile_(book)#Skin_in_the_game">Antifragility</a>:</p>
<blockquote>
Every captain goes down with every ship</blockquote>
<p>If a engineer (or a manager) collects benefits when she/he is right and let others pay the price when she/he is wrong, the result will be fragile.</p>
<p>Like UX designers who don't use their app, or architects who don't spend time in their buildings, software developers who never have experienced the romance of maintenance are less likely to produce long lasting solutions.</p>
<p>Engineers should experience being maintainers.</p>
<div class="figure align-center">
<img alt="A minor upgrade of React Router breaking the entire app." src="https://blog.mathieu-leplatre.info/images/long-term-minor-upgrade.jpg" />
<p class="caption">A patch version upgrade of React Router breaking the entire app.</p>
</div>
<p>You need first-hand experience, having done mistakes, and having paid the consequences, or your ships will keep drowning.</p>
<p>Experience helps you adjust priorities in life. You grow up. Age doesn't really matter. You may have missed the joy of upgrading Redmine on your personal server 15 years ago, if you had to deal with upgrades of React Router in your SPA recently, you know what this is all about. You paid the price of your bad draw. You now have less patience to deal with fragile stuff. You aim to work less, and that's extremely beneficial when it comes to designing solid and durable stuff.</p>
<p>I think that it's also one of the succesful ideas behind <em>Dev Ops</em>, when developers are also accoutable for running their stuff in production.</p>
</div>
<div class="section" id="technology-pick">
<h2>Technology Pick</h2>
<p>Which technology will stand the test of time?</p>
<div class="figure align-center">
<img alt="Design of the BIC pen" src="https://blog.mathieu-leplatre.info/images/long-term-bic-pen.jpg" />
<p class="caption"><a class="reference external" href="https://twitter.com/fxn/status/1443249162653511687">@fxn, 29 sept. 2021</a></p>
</div>
<p>I like one of the first answers to this tweet, arguing that «<em>paper didn't change much, that's why!</em>».
In the world of software, and technology in general, the environment evolves constantly. Some top-notch innovations of yesterday become mainstream building blocks of tomorrow. Others fall under the pressure of the heartless natural selection of their ecosystems.</p>
<p>There are <a class="reference external" href="https://opensource.com/article/18/3/pick-right-technology">rational criteria</a> to maximize your chances, but life always remains full of surprises :) Naturally, technologies capable of embracing and coexisting nicely with other ones have a better chance to survive! Rust, for example, can be executed from almost any language, from Python to Javascript, and can itself execute external libraries written in C++, which makes it a great tool for gradual rewrites.</p>
<p>We can argue that you should <strong>listen to your old guard</strong>. They've seen trends pass by, their sweat shirt may not be cool anymore, but they've washed it a thousand times and it <a class="reference external" href="https://blog.mathieu-leplatre.info/tips-for-your-makefile-with-python.html">still does the job nicely</a>. <strong>They have a lot less free time than young intrepid engineers, and are therefore less likely to reinvent the wheel</strong>.</p>
<p>Successful technologies with a low rate of evolution can be a good bet. As <a class="reference external" href="https://nicolas.perriault.net/">Nico</a> said, a <a class="reference external" href="https://fr.wikipedia.org/wiki/Gibson_Les_Paul">Gibson Les Paul</a> doesn't need any enhancement, it's simple, built with quality, and maintenance will only consist in replacing the strings from time to time. That's also one of the reasons why I enjoyed working with <a class="reference external" href="https://trypyramid.com/">Pyramid</a>, instead of <a class="reference external" href="https://www.djangoproject.com/">Django</a>.</p>
<p>The <a class="reference external" href="https://en.wikipedia.org/wiki/Lindy_effect">Lindy effect</a> gives an interesting perspective too: the future life expectancy of some non-perishable things, like a technology or an idea, is proportional to their current age. Obviously, other forces like the <a class="reference external" href="https://en.wikipedia.org/wiki/Network_effect">network effects</a> come into play. Do you realize that the usage of jQuery only <a class="reference external" href="https://twitter.com/mikesherov/status/1443679254009483273">started to decline</a> at the end of 2020?</p>
<p>Besides, you cannot ignore recruiting. Perl enthusiasts can argue that their ecosystem is fantastic because any library upgrade is almost guaranteed to be backward compatible, nonetheless it will be very hard to recruit talents. Of course, good engineers can learn anything. Even though it's not always that simple. I read that sometimes new hires refused to learn Elm <a class="footnote-reference" href="#elm" id="footnote-reference-4">[4]</a> because they wouldn't see the value for their curriculum!</p>
<p>Anyway, there is no magic rule of thumb here, but I would recommend to <strong>get the people part right because it matters more</strong>. Any technology in the hands of reasonable craftsmen will always perform better over time than great technologies under disastrous management.</p>
</div>
<div class="section" id="architecture-legacy">
<h2>Architecture Legacy</h2>
<div class="figure align-center">
<img alt="Tsunami stones: do not build any homes below this point" src="https://blog.mathieu-leplatre.info/images/long-term-tsunami.jpg" />
<p class="caption">Centuries old <em>tsunami stone</em> in Japan, warning descendants to not build any homes below this point because of great tsunamis</p>
</div>
<p>In order to help future engineers understand your choices, and their context, a good practice consists in documenting <em>architecturally significant</em> decisions.</p>
<p>The idea is to answer potential <em>Why?</em> questions, and focus on «design decisions that are costly to change» <a class="footnote-reference" href="#folzzio" id="footnote-reference-5">[5]</a> <a class="footnote-reference" href="#fcognitect" id="footnote-reference-6">[6]</a>.</p>
<p>The documents must be relatively short to read, but must at least describe the context, what options were considered, with their pros and cons, and what are the consequences of the decision. And must of course be kept in the repo along with the source code.</p>
<p>Check out <a class="reference external" href="https://github.com/joelparkerhenderson/architecture-decision-record#adr-example-templates">the various templates</a>, and <a class="reference external" href="https://github.com/joelparkerhenderson/architecture-decision-record/blob/6c4f515/examples/index.md">examples</a>, think long term, and <strong>start writing architecture decision records</strong>!</p>
<hr class="docutils" />
<p>Many thanks to everyone that participated in these extremely valuable conversations about this topic lately! And a special one to <a class="reference external" href="https://twitter.com/mostlygeek">Benson</a> who introduced me to the work of Stewart Brand.</p>
<p>Please share your feedback and ideas, and as usual don't hesitate to correct me if I'm wrong!</p>
<p>While I was writing this article, Justin wrote <a class="reference external" href="https://www.simplethread.com/20-things-ive-learned-in-my-20-years-as-a-software-engineer/">20 Things I’ve Learned in my 20 Years as a Software Engineer</a> and I think it's is a nice complementary read of this one :)</p>
<table class="docutils footnote" frame="void" id="preply" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td><a class="reference external" href="https://medium.com/preply-engineering/do-you-want-to-be-right-or-successful-52a2cd0a220b">https://medium.com/preply-engineering/do-you-want-to-be-right-or-successful-52a2cd0a220b</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="fjoel" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td><a class="reference external" href="https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/">https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="illich" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td><a class="reference external" href="http://conviviality.ouvaton.org/spip.php?article39">http://conviviality.ouvaton.org/spip.php?article39</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="elm" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-4">[4]</a></td><td><a class="reference external" href="https://discourse.elm-lang.org/t/reasons-that-people-were-forced-to-move-from-elm-to-something-else/6390/18">https://discourse.elm-lang.org/t/reasons-that-people-were-forced-to-move-from-elm-to-something-else/6390/18</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="folzzio" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-5">[5]</a></td><td><a class="reference external" href="https://medium.com/olzzio/from-architectural-decisions-to-design-decisions-f05f6d57032b">https://medium.com/olzzio/from-architectural-decisions-to-design-decisions-f05f6d57032b</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="fcognitect" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-6">[6]</a></td><td><a class="reference external" href="https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions">https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions</a></td></tr>
</tbody>
</table>
</div>
About Maintenance Mode2021-08-04T00:00:00+02:002021-08-04T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2021-08-04:/about-maintenance-mode.html<p>You may be familiar with this situation: your team has been working on a project for a while, and now that it seems to do the job, it is not really justified for so many engineers to spend time on it.</p>
<p>Management agreed that the project should be switched to …</p><p>You may be familiar with this situation: your team has been working on a project for a while, and now that it seems to do the job, it is not really justified for so many engineers to spend time on it.</p>
<p>Management agreed that the project should be switched to «maintenance mode». Even if the software remains critical and runs in production, the team's priorities have changed, and costs must be kept as low as possible.</p>
<p>From my experience, there is often some misunderstanding about turning a software into maintenance mode. If you continue to implement new features or spend time on refactoring internals, you are going to clash with your managers. If you don't touch the source at all and don't invest any engineering time at all, the software will soon qualify as «abandonware».</p>
<p>So, if the software still runs in production and is switched to maintenance, what should you expect to be done?</p>
<div class="figure align-center">
<img alt="NASA Engine Maintenance Training CC-BY-NC https://www.flickr.com/photos/nasa2explore/49243482926/" src="https://blog.mathieu-leplatre.info/images/maintenance-nasa.jpg" />
<p class="caption">NASA Engine Maintenance Training <a class="reference external" href="https://www.flickr.com/photos/nasa2explore/49243482926/">CC-NC-ND</a></p>
</div>
<div class="section" id="monitor-production">
<h2>Monitor production</h2>
<p>I remember one situation where a team used the URL of the origin server for our service instead of the CDN. And went live. We suddenly received a massive amount of traffic, which caused the database to lag a lot, the application was then losing connections and crashing, users were seeing 503 error pages and reporting issues. Our first strategy had been to increase the database resources, which increased the costs of the service significantly.</p>
<p>Monitoring often seems like an obvious one. But it's not only about the server resources, it's mostly about making sure the software is just ticking over, handling load transparently, and not bothering you at night.</p>
<p>Make sure you have ways to monitor:</p>
<ul class="simple">
<li><strong>Load</strong>: amount of traffic or activity</li>
<li><strong>Responsiveness</strong>: latency or lag</li>
<li><strong>Robustness</strong>: number of crash reports</li>
<li><strong>Stability</strong>: production issues tickets</li>
<li><strong>Cost</strong>: bills of cloud services</li>
</ul>
<p>If these metrics become explicit and public, it gives everyone concrete and measurable goals to put the application in its best conditions for the long run.</p>
</div>
<div class="section" id="optimize-costs">
<h2>Optimize costs</h2>
<p>Sometimes, a new system comes as a replacement, and the old one is maintained for legacy clients. Usually, the old system even becomes read-only. Nevertheless, it still relies on a database, some Web workers, executing domain specific code, with the sole purpose of serving static data for legacy clients! Some JSON files on a CDN could do the job!</p>
<p>If you know that you are going for the long run, investing in optimization can be worth it. Even tiny parts.</p>
<p>Of course, it is imperative to evaluate how much you can save per month/year before investing efforts! Beware of rabbit holes, always timebox your work! Start with low hanging fruits, like slow queries, endpoints that serve static data, ...</p>
<p>If your infrastructure allows it, you could also leverage some of the auto-scaling features of your cloud provider. For example, at Mozilla, we clearly have pattern of loads depending of time and day of the week. Setting up rules to scale up/out or in/down your resources can help you save money. Combined with proper monitoring and metrics, it can truly be rewarding.</p>
</div>
<div class="section" id="fix-bugs">
<h2>Fix bugs</h2>
<p>No magic here, apart from the fact that some bug reports can actually be missing features in disguise.</p>
<p>Being able to fix bugs implicitly means that team members have a development setup on their machine. Ideally, running the software or its tests suite locally should not take several days.</p>
<p>Note that sometimes it can also be relevant to ignore some bugs, if their impact is minimal, their reproduction too complex, and the fix too fragile. They become «known bugs». Make sure you document them properly though.</p>
<p>Counterintuitively, fixing bugs and polishing rough edges can be entertaining. Some people are suited for maintenance (fixers!), others prefer innovation (builders!), but having experienced both perspectives is precious.</p>
</div>
<div class="section" id="apply-security-patches">
<h2>Apply security patches</h2>
<p>This one is interesting, because it means that you should keep your libraries and dependencies up-to-date.</p>
<p>It is highly recommended to automate this part with tools like <a class="reference external" href="https://dependabot.com/">Dependabot</a>, <a class="reference external" href="https://renovatebot.com">Renovate</a> or <a class="reference external" href="https://snyk.io/">Snyk</a>.</p>
<p>In our React application, keeping libraries up-to-date came out to be very costly. Indeed, the maintainers of the libraries were shipping security fixes in major releases with breaking changes, obliging us to rewrite some big parts of our app in order to keep up and benefit from security fixes.</p>
<p>In my opinion, it is very healthy for the team (and the project) to frequently touch the code and upgrade dependencies. It forces everyone to cherish the project, and most importantly: redeploy it regularly.</p>
</div>
<div class="section" id="redeploy-regularly">
<h2>Redeploy regularly</h2>
<p>This one is extremely important: the team should be as confident as possible when redeploying the software.</p>
<p>In a perfect world, you have a setup with continuous deployment, at least on stage. Everytime a new version is tagged, the application is rolled out on some server.</p>
<p>If you wait too long between releases and deployments, you take the risk that some deployment recipe breaks, or becomes out of date, relying on missing resources or permissions.</p>
<div class="figure align-center">
<img alt="Aviation Machinist conducts maintenance on an afterburner CC-BY-NC https://www.flickr.com/photos/compacflt/51319755467/" src="https://blog.mathieu-leplatre.info/images/maintenance-plane.jpg" />
<p class="caption">Aviation Machinist conducts maintenance on an afterburner <a class="reference external" href="https://www.flickr.com/photos/compacflt/51319755467/">CC-BY-NC</a></p>
</div>
</div>
<div class="section" id="sustain-infrastructure">
<h2>Sustain Infrastructure</h2>
<p>If your application uses some cloud provider's services, you will also have to keep afloat with upgrades and decommissions. If your application is not compatible with the only versions available, some code ought to be rewritten.</p>
<p>For example, Amazon regularly rolls out new versions of PostgreSQL, and cojointly shutdowns old versions.</p>
<p>Exactly like for your software libraries, part of a Kubernetes cluster lifecycle involves performing periodic upgrades to the latest version, in order to apply the latest security releases. Automation is possible but can also give you surprises!</p>
<p>On top of that, your company can also decide to migrate its whole infrastructure to a different cloud provider. That may require some code to be rewritten (eg. Amazon S3 versus Google Cloud Storage) and very likely critical parts.</p>
</div>
<div class="section" id="manage-knowledge">
<h2>Manage Knowledge</h2>
<p>When the whole team is working on the project daily, knowledge flows and is globally available. In maintenance mode, a couple of people are involved sporadically, and knowledge about procedures or technical details will evaporate quickly.</p>
<p>Plus, it is unfortunately very likely that, along the years, the members of the original team will have left the company.</p>
<p>An efficient way to make sure the project documentation is up-to-date is to give new hires the responsibility to update it as their first assignment :) For example, they will follow the procedure to setup their machine for development, and fix every step in docs where they got stuck.</p>
<p>In addition, exactly like planes or cars have their maintenance logs, it could be useful to keep a single document where every intervention in production is described.</p>
<p>When things turn sour, take the time to write down a <em>post-mortem</em>, that breaks down the timeline of events, the steps of troubleshooting, the lessons learnt, the improvements to be made, etc. This will become highly valuable for the future maintainers.</p>
<div class="figure align-center">
<img alt="https://classicprogrammerpaintings.com/post/143947399671/developers-look-for-documentation-in-legacy" src="https://blog.mathieu-leplatre.info/images/maintenance-classic-programmer.jpg" />
<p class="caption">Developers look for documentation in legacy system - Jean-François Millet, 1857 - Oil on canvas (by <a class="reference external" href="https://classicprogrammerpaintings.com/post/143947399671/developers-look-for-documentation-in-legacy">classic programmer paintings</a>)</p>
</div>
</div>
<div class="section" id="handle-open-source-contributions">
<h2>Handle Open source Contributions</h2>
<p>Imagine the following situation: a company develops a software for a Web API, releases it as open source, and builds a community around it. After some time, the company's strategy (or goals) changes and the API is switched to maintenance mode, with lowest risks possible. The community continues to submit contributions for new features and risky refactors.</p>
<p>What should the company do?</p>
<p>Ignore the contributions and kill the community?</p>
<p>Maintain its own fork with bug and security fixes only?</p>
<p>Make sure every new feature is behind a config flag?</p>
<p>Keep upgrading their API in production to the latest version?</p>
<p>I don't think there is a simple answer to this one. It truly depends on multiple factors, like the size of the community, the criticality of the API, the quality of contributions, etc.</p>
<p>Refusing pull requests is often very hard, but keep in mind that saying «no» can save everybody a lot of trouble.</p>
</div>
<div class="section" id="assess-risks">
<h2>Assess Risks</h2>
<p>You may not have enough resources to complete all of the above successfully. Maintenance of complex software is hard. And shit happens. Think of your software as an old building open to the public, it is your responsability to report any potential danger that you see.</p>
<p>In toxic environments, engineers will sometimes blame each other for having failed to comply with certain expectations. In order to avoid that, some will work triple to reach what they see as <em>perfection</em>. Or when a top-down decision is made, they will disagree and disapprove of their management, complaining that «they have no idea how reckless this is».</p>
<p>Of course, there are really bad managers out there, but I believe that the most common mistake is to keep your risk analysis for yourself.</p>
<p>No matter what the current situation is, and how far it is from being ideal, write down all potential risks explicitly and share them with the team.</p>
<p>The <em>Risk Assessment</em> exercice consists in:</p>
<ol class="arabic simple">
<li>Identifying all potential catastrophic scenarios, incidents or deteriorations, in terms of stability, security, team motivation, whatever!</li>
<li>Evaluating likelihood, severity, and impact of each identified risk</li>
<li>Deciding which ones to ignore and why, and which ones that will have to be taken care of.</li>
</ol>
<p>By making this list explicit and public, your long-term maintenance strategy, whatever it is, is supported by a proper evaluation and awareness of risks.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>If executives think that switching a project to <em>maintenance mode</em> will save a lot of money and effort, you now have some arguments to contrast their idea.</p>
<p>Maintenance cost is probably less than investing in new features, but it is definitely not zero.</p>
<p>Shutting down a service is also an option, leaving consumers in despair.</p>
<p>With the amount of connected devices that depend on closed-source Web APIs out there, I believe that long-term maintenance is going to become a major concern in the next years...</p>
<hr class="docutils" />
<p>Many thanks to <a class="reference external" href="https://nl.linkedin.com/in/nicolas-metaye-27766633">Nico</a>, <a class="reference external" href="https://www.linkedin.com/in/johnwhitlock">John</a>, <a class="reference external" href="https://www.linkedin.com/in/mostlygeek">Benson</a>, <a class="reference external" href="http://stephenhood.com">Stephen</a>, <a class="reference external" href="https://www.linkedin.com/in/smarnach/">Sven</a>, and <a class="reference external" href="http://areskibelaid.com/">Areski</a> for your precious feedback and suggestions!</p>
<p>As usual, please don't hesitate to share your feedback or thoughts, I would be super happy to have a chat and/or integrate your contributions in the article!</p>
</div>
Poucave, an observation standpoint for our services2021-05-11T00:00:00+02:002021-05-11T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2021-05-11:/poucave-an-observation-standpoint-for-our-services.html<p>Most of us are relatively familiar with system monitoring: we monitor RAM, CPU, or disk usage over time and receive alerts when some thresholds are reached.</p>
<p>But the quality of a whole service is rarely defined by the health of a single isolated server. Rather, it is the interactions between …</p><p>Most of us are relatively familiar with system monitoring: we monitor RAM, CPU, or disk usage over time and receive alerts when some thresholds are reached.</p>
<p>But the quality of a whole service is rarely defined by the health of a single isolated server. Rather, it is the interactions between the parts that is often the source of problems.</p>
<p>What about the consistency between the CDN cache and the origins? The last run of a scheduled task? Imminent expiration of your SSL certificates? The amount of clients side errors? The latency of critical HTTP endpoints? The number of pending pull-requests on your repositories?</p>
<p>Poucave is a small web app that executes a series of domain specific checks for the parts and their interactions that make up our service.</p>
<div class="section" id="where-did-we-start-from">
<h2>Where did we start from?</h2>
<p>Centuries ago, telescopes helped our ancestors to distinguish a single shining dot from a whole constellation. Yes, that's how far we started from!</p>
<p>We used to consider our service healthy as long as operating system resources were still available and no HTTP 5XX error was served. This idea was implemented by introducing a <tt class="docutils literal">/__heartbeat__</tt> endpoint on each server, that executes some internal health checks and returns a <tt class="docutils literal">200 Ok</tt> if everything is fine and a <tt class="docutils literal">500 Internal Server Error</tt> if something is wrong. We made this <a class="reference external" href="https://github.com/mozilla-services/dockerflow">a requirement</a> for every deployed application, and were polling this HTTP endpoint from an external alerting service. Operators are paged if it keeps failing for a while.</p>
<p>This served us well. But our service was integrated in a complex ecosystem, and making sure that it was 100% reliable took more than that. We had a CDN, push notifications, some scheduled tasks, data synchronization on clients, etc. Our heartbeat endpoints could be all green, but still, we sometimes had users reporting issues or inconsistencies.</p>
<p>Enhancing the capacities of our heartbeat endpoint would break the separation of concerns. Apart from not being very robust, a system shouldn't be in charge of checking external components that are more related to the way it's deployed than its own functional scope. Concretely, verifying a database connection from the server hearbeart is fine, but not querying the clients Telemetry data.</p>
<p>Discovering issues on your system because some users complain is never a great feeling. And less when subsequent investigations and troubleshooting are taking time and require a lot of knownledge about all sub-systems and their interactions.</p>
<p>We needed to improve our ability to see. See the constellation of systems, instead of just a single one.</p>
<img alt="Early depiction of a ‘Dutch telescope’ from the “Emblemata of zinne-werck” (Middelburg, 1624)" class="align-center" src="https://blog.mathieu-leplatre.info/images/poucave-emblemata-1624.jpg" />
</div>
<div class="section" id="what-do-we-need">
<h2>What do we need?</h2>
<p>We wanted to identify the root cause of incidents as fast as possible. For this, we needed to:</p>
<ul class="simple">
<li>see our service as whole, with all its sub-parts, including clients;</li>
<li>document each part and interactions;</li>
<li>monitor and be alerted on sub-part failures;</li>
<li>track service reliability over time;</li>
<li>capitalize knownledge about troubleshooting and past issue resolutions.</li>
</ul>
<p>Being able to see the problem as a whole would give more insights about the fix. We needed a solution to help us identify the root cause faster, and reassure our users.</p>
<p>In order to include the clients behaviour, we could rely on the company real-time Telemetry platform. A couple of years ago, we unified the way our clients would report <a class="reference external" href="https://searchfox.org/mozilla-central/rev/0bcf81557b89e7757c44e25bb4bc7f4cb8619dc9/services/common/uptake-telemetry.js">uptake Telemetry</a> (success, up-to-date, network-error, certificate-error, etc.), and that is super useful to write generic checks about error rates for multiple different components.</p>
<p>Since some parts of our service are only accesible via our VPN, like the signature infrastructure, an external ping service cannot reach them. This new tool would serve as a bridge between the two.</p>
<p>Not every check has to necessarily be monitored by an alerting service. Some can just be indications for operators, or even never fail and just mash-up information from the underlying sub-parts.</p>
</div>
<div class="section" id="our-solution">
<h2>Our solution</h2>
<p>No rocket science. The same idea as our heartbeat endpoint but from an independant service, and infinitely extensible.</p>
<p>A simple and stupid HTTP API, no backend, that starts from a configuration file where checks are listed:</p>
<div class="highlight"><pre><span></span><span class="k">[checks.a-project.a-check]</span>
<span class="n">description</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"Heartbeat of the public read-only instance."</span>
<span class="n">module</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"checks.core.heartbeat"</span>
<span class="n">params</span><span class="p">.</span><span class="n">url</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"https://firefox.settings.services.mozilla.com/v1/__heartbeat__"</span>
</pre></div>
<p>... and that exposes each execution via a dedicated endpoint <tt class="docutils literal">GET <span class="pre">/checks/{a-project}/{a-check}</span></tt>.</p>
<p>The checks source code for our service becomes a repository of knowledge on how to operate and troubleshoot it.
I decided to use Python, so that almost anybody could read or write the checks implementation.</p>
<p>I chose a simple async framework, <a class="reference external" href="https://docs.aiohttp.org">aiohttp</a>, since most of the checks will pull information from external sources and do very little computation (I/0 bound). Nothing fancy really.</p>
<p>And to follow the long tail of funny project names at Mozilla, I called it <em>Poucave</em> (/pu.kav/), french slang for «snitch».</p>
<div class="figure align-center">
<img alt="Example of diagram with overview" src="https://blog.mathieu-leplatre.info/images/poucave-overview.png" />
<p class="caption">Example of live diagram. The SVG file is part of configuration.</p>
</div>
</div>
<div class="section" id="examples">
<h2>Examples</h2>
<p>Just to present of few of <a class="reference external" href="https://github.com/mozilla-services/poucave/tree/main/checks/">the checks</a> that were implemented:</p>
<ul class="simple">
<li><tt class="docutils literal">checks.core.latency</tt> (<em>generic</em>): fails if the specified URL does not respond under a certain number of milliseconds.</li>
<li><tt class="docutils literal">checks.core.maintenance</tt> (<em>generic</em>): takes a list of Github repositories as input, and fails in any of them has pull-requests who didn't receive activity in the last X days.</li>
<li><tt class="docutils literal">checks.core.deployed_version</tt> (<em>generic</em>): fails while the deployed version does not match the latest tagged version on the specified Github repository.</li>
</ul>
<p>And then, we have more domain specific checks for <a class="reference external" href="https://remotesettings.readthedocs.io">Remote Settings</a>, like:</p>
<ul class="simple">
<li><tt class="docutils literal">checks.remotesettings.push_timestamp</tt>: fails if the timestamp of the published data does not match the one of our Push service</li>
<li><tt class="docutils literal">checks.remotesettings.certification_expiration</tt>: fails if our certificates will expire soon</li>
<li><tt class="docutils literal">checks.remotesettings.uptake_max_age</tt>: fails if the 75th percentile of clients receive only data after X seconds</li>
<li><tt class="docutils literal">checks.remotesettings.uptake_error_rate</tt>: fails if the proportion of errors among reported statuses in Telemetry is above a certain threshold</li>
</ul>
<p>Adding new checks is a piece of cake, even from your own packages, as long as they are available in the <tt class="docutils literal">PYTHONPATH</tt>.</p>
<div class="figure align-center">
<img alt="Example of check details" src="https://blog.mathieu-leplatre.info/images/poucave-check-details.png" />
<p class="caption">Example of check details.</p>
</div>
</div>
<div class="section" id="check-history">
<h2>Check History</h2>
<p>If an issue in our service led twice to the same root cause, we would consider implementing a check for it.</p>
<p>This allowed us to consolidate the reliability of our service over time, and also build a sort of memory for it.</p>
<p>In order to facilitate troubleshooting, we added the ability to link each check with past issues from our Bugtracker.
This way, when a check turns red, we can immediately access the history of its possible past failures, and read the related conversations and resolutions.</p>
<p>For checks that return discrete values, like latency or age, we also wanted to track variations over time. Because this monitoring service had to remain simple, stupid, and reliable, I wanted to avoid introducing a storage dependency.</p>
<p>All our applications output their logs to stdout as JSON. These application logs are then parsed and ingested elsewhere, so that we can plot them in Grafana. I decided to reuse that. Each check execution is logged, and can be presented in charts on dashboards.</p>
<p>It was sometimes annoying to open Grafana just to take a look at a check recent behaviour. So we added the ability to see the history of a check directly in the UI. We came up with something super simple, the server pulls the last entries from the log database, returns them as JSON, and the UI plots them in a basic chart. For more advanced charts and querying, Grafana will always be better of course.</p>
<div class="figure align-center">
<img alt="Check history with linked bugs and graph of past values" src="https://blog.mathieu-leplatre.info/images/poucave-check-history.png" />
<p class="caption">Check history with linked bugs and graph of past values.</p>
</div>
</div>
<div class="section" id="after-a-few-months">
<h2>After a few months...</h2>
<p>This checks platform has served us very well!</p>
<p>Only a small fraction of checks are monitored, and only the crucial ones wake our SREs at night.</p>
<p>We still have false positives, notably on Telemetry. With help from our data science team, our queries and normalizations could certainly be improved.</p>
<p>We had one major false negative. Fortunately the issue was raised by another system. We improved the check and now feel better.</p>
<p>To conclude, I know with certainty that this little healthcheck application, with the live diagram, has changed the way we see and understand our services. Compared to linear problem solving, system thinking is complex. Seeing things as a whole and being able to understand interdependance and causal loops is really helpful. Everyone can now see all the moving pieces in one place, and can be reasonably reassured that the service is working well if checks are green, which also makes customer care easier for us.</p>
<p>In addition to this success, several teams expressed their interest in adding their own checks or running their instance for their service :) If you too are interested in using it in your organization, go on! Nothing is hard-coded and adding your own checks and SVG diagram is fairly easy! Don't hesitate to get in touch of course.</p>
</div>
Tips for your Makefile with Python2021-01-22T00:00:00+01:002021-01-22T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2021-01-22:/tips-for-your-makefile-with-python.html<p>Recently, while I was migrating old repos from TravisCI to Github Actions, I realized that several of them had wobbly Makefiles.</p>
<p>I know that Makefiles are not super elegant, and that intrepid youngsters regularly come up with alternatives. But I find them super handful and powerful! Especially when they are …</p><p>Recently, while I was migrating old repos from TravisCI to Github Actions, I realized that several of them had wobbly Makefiles.</p>
<p>I know that Makefiles are not super elegant, and that intrepid youngsters regularly come up with alternatives. But I find them super handful and powerful! Especially when they are well structured.</p>
<div class="section" id="tips">
<h2>Tips</h2>
<div class="section" id="the-basics">
<h3>The basics</h3>
<p>Everything is based on dependencies and timestamps: if a dependency's timestamp is more recent than the target, then the rule is executed.</p>
<p>For Python projects, the chain looks like this:</p>
<ul class="simple">
<li>Python → Virtualenv → Install packages → Run task (tests, lint)</li>
</ul>
<p>Which, in a <tt class="docutils literal">Makefile</tt> simply looks like this:</p>
<div class="highlight"><pre><span></span><span class="nf">.venv/bin/python</span><span class="o">:</span>
<span class="w"> </span>python3<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span>.venv
<span class="nf">.venv/.install.stamp</span><span class="o">:</span><span class="w"> </span>.<span class="n">venv</span>/<span class="n">bin</span>/<span class="n">python</span> <span class="n">requirements</span>.<span class="n">txt</span>
<span class="w"> </span>.venv/bin/python<span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements.txt
<span class="w"> </span>touch<span class="w"> </span>.venv/.install.stamp
<span class="nf">test</span><span class="o">:</span><span class="w"> </span>.<span class="n">venv</span>/.<span class="n">install</span>.<span class="n">stamp</span>
<span class="w"> </span>.venv/bin/python<span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span>tests/
</pre></div>
<p>Now, when you run <tt class="docutils literal">make test</tt> from a recently cloned repo, the whole chain is executed. But otherwise, the Python packages are installed only if your requirements file has changed since your last installation.</p>
</div>
<div class="section" id="use-variables">
<h3>Use variables</h3>
<p>In order to ease readability of dependencies, I find that using variables helps:</p>
<div class="highlight"><pre><span></span><span class="nv">VENV</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span>.venv
<span class="nv">INSTALL_STAMP</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/.install.stamp
<span class="nv">PYTHON</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/python
<span class="nf">$(PYTHON)</span><span class="o">:</span>
<span class="w"> </span>python3<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="nf">$(INSTALL_STAMP)</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">PYTHON</span><span class="k">)</span> <span class="n">requirements</span>.<span class="n">txt</span>
<span class="w"> </span><span class="k">$(</span>PYTHON<span class="k">)</span><span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements.txt
<span class="w"> </span>touch<span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span>
<span class="nf">test</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>PYTHON<span class="k">)</span><span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span>./tests/
</pre></div>
</div>
<div class="section" id="environment-variables-with-default">
<h3>Environment variables with default</h3>
<p>For example, instead of hardcoding the name of your virtualenv folder, you can read it from the current shell environment and use a default value:</p>
<div class="highlight"><pre><span></span><span class="nv">VENV</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>shell<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$$</span><span class="o">{</span>VIRTUAL_ENV-.venv<span class="o">}</span><span class="k">)</span>
</pre></div>
<p>Basically, <tt class="docutils literal">echo <span class="pre">${VAR-val}</span></tt> will show the content of <tt class="docutils literal">$VAR</tt> and defaults to <tt class="docutils literal">val</tt> if undefined (and we double the <tt class="docutils literal">$</tt> for escaping).</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p><tt class="docutils literal">make</tt> allows you to pass variables and environment values from the command-line, but I always find it quite confusing to distinguish the two. I recommend to only use environment variables, and pass them as usual from command-line:</p>
<div class="highlight"><pre><span></span><span class="nv">LOG_FORMAT</span><span class="o">=</span>json<span class="w"> </span>make<span class="w"> </span><span class="nb">test</span>
</pre></div>
<p>or:</p>
<div class="last"><div class="highlight"><pre><span></span><span class="nb">export</span><span class="w"> </span><span class="nv">LOG_FORMAT</span><span class="o">=</span>json
make<span class="w"> </span><span class="nb">test</span>
</pre></div>
</div></div>
</div>
<div class="section" id="check-if-a-command-is-available">
<h3>Check if a command is available</h3>
<p>It's nice to give a little hint about a missing prerequisite. Most of the time there will be an official system package to be installed for the rest of the Makefile to be executed smoothly.</p>
<div class="highlight"><pre><span></span><span class="nv">PY3</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>shell<span class="w"> </span><span class="nb">command</span><span class="w"> </span>-v<span class="w"> </span>python3<span class="w"> </span><span class="m">2</span>><span class="w"> </span>/dev/null<span class="k">)</span>
<span class="nf">$(PYTHON)</span><span class="o">:</span>
<span class="w"> </span>@if<span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="k">$(</span>PY3<span class="k">)</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"python3 could not be found. See https://docs.python.org/3/"</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">2</span><span class="p">;</span><span class="w"> </span><span class="k">fi</span>
<span class="w"> </span>python3<span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>
</pre></div>
<p><tt class="docutils literal">command <span class="pre">-v</span></tt> is roughly the equivalent of <tt class="docutils literal">which</tt>, but built-in your shell. It returns the executable path or nothing if not found.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">The <tt class="docutils literal">@</tt> prefix will prevent the underlying command to be shown in the output log.</p>
</div>
</div>
<div class="section" id="list-available-targets">
<h3>List available targets</h3>
<p>When running <tt class="docutils literal">make</tt> the <tt class="docutils literal">all</tt> target is implicitly called. We can tweak it and show some help:</p>
<div class="highlight"><pre><span></span><span class="nv">.DEFAULT_GOAL</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">help</span>
<span class="nf">help</span><span class="o">:</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"Please use 'make <target>' where <target> is one of"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">""</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" install install packages and prepare environment"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" format reformat code"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" lint run the code linters"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" test run all the tests"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" clean remove *.pyc files and __pycache__ directory"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">""</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"Check the Makefile to know exactly what each target is doing."</span>
</pre></div>
<p><a class="reference external" href="https://larlet.fr/david/">david</a> suggests to produce the above help summary using <a class="reference external" href="https://www.thapaliya.com/en/writings/well-documented-makefiles/">this trick</a>, that relies on <tt class="docutils literal">awk</tt> and comments on targets:</p>
<div class="highlight"><pre><span></span><span class="nv">.DEFAULT_GOAL</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">help</span>
<span class="nf">help</span><span class="o">:</span><span class="w"> </span><span class="c">## Display this help</span>
<span class="w"> </span>@awk<span class="w"> </span><span class="s1">'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m\033[0m\n\nTargets:\n"} /^[a-zA-Z_-]+:.*?##/ { printf "\033[36m%-10s\033[0m %s\n", $$1, $$2 }'</span><span class="w"> </span><span class="k">$(</span>MAKEFILE_LIST<span class="k">)</span>
<span class="nf">deps</span><span class="o">:</span><span class="w"> </span><span class="c">## Check dependencies</span>
<span class="w"> </span>...
<span class="nf">clean</span><span class="o">:</span><span class="w"> </span><span class="c">## Cleanup the project folders</span>
<span class="w"> </span>...
<span class="nf">build</span><span class="o">:</span><span class="w"> </span><span class="n">clean</span> <span class="n">deps</span> <span class="c">## Build the project</span>
<span class="w"> </span>...
</pre></div>
</div>
<div class="section" id="do-you-think-it-s-phony">
<h3>Do you think it's PHONY?</h3>
<p>By default, <em>Make</em> assumes that the target of a rule is a file. If you have targets that do not produce files on disk (eg. <tt class="docutils literal">make test</tt> or <tt class="docutils literal">make clean</tt>) then mark them as <tt class="docutils literal">.PHONY</tt> (<em>fake</em> in English).</p>
<p>Phony targets are never up-to-date and will always run when invoked, and even if there is a matching file on disk (eg. a file called <tt class="docutils literal">clean</tt>).</p>
<div class="highlight"><pre><span></span><span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">clean</span> <span class="n">test</span>
<span class="nf">clean</span><span class="o">:</span>
<span class="w"> </span>find<span class="w"> </span>.<span class="w"> </span>-type<span class="w"> </span>d<span class="w"> </span>-name<span class="w"> </span><span class="s2">"__pycache__"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="o">{}</span><span class="p">;</span>
<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="nf">test</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>PYTHON<span class="k">)</span><span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span>./tests/
</pre></div>
<p><strong>edit</strong>: Instead of maintaining a list of phony targets on top, <a class="reference external" href="http://agopian.info/">magopian</a> and <a class="reference external" href="https://yohanboniface.me/">ybon</a> recommend to put it along each rule:</p>
<div class="highlight"><pre><span></span><span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">clean</span>
<span class="nf">clean</span><span class="o">:</span>
<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">test</span>
<span class="nf">test</span><span class="o">:</span><span class="w"> </span>...
</pre></div>
</div>
<div class="section" id="multiple-targets">
<h3>Multiple targets</h3>
<p>While I was reading about the multiple PHONY lines, I learned that any target can be repeated multiple times, their dependencies are just «combined»:</p>
<div class="highlight"><pre><span></span><span class="nf">$(INSTALL_STAMP)</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">PYTHON</span><span class="k">)</span> <span class="n">requirements</span>/<span class="n">dev</span>.<span class="n">txt</span>
<span class="w"> </span><span class="k">$(</span>PYTHON<span class="k">)</span><span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements/dev.txt
<span class="w"> </span>touch<span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span>
<span class="nf">$(INSTALL_STAMP)</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">PYTHON</span><span class="k">)</span> <span class="n">requirements</span>/<span class="n">app</span>.<span class="n">txt</span>
<span class="w"> </span><span class="k">$(</span>PYTHON<span class="k">)</span><span class="w"> </span>-m<span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>-r<span class="w"> </span>requirements/app.txt
<span class="w"> </span>touch<span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span>
</pre></div>
<p>Here, we won't reinstall all application's packages when just a <tt class="docutils literal">dev</tt> package has changed.</p>
</div>
</div>
<div class="section" id="full-example-with-poetry">
<h2>Full Example with Poetry</h2>
<p>I gathered most of the above tips in a full working example with Poetry (<a class="reference external" href="https://github.com/mozilla-services/poucave/pull/752">original source</a>):</p>
<div class="highlight"><pre><span></span><span class="nv">NAME</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span>superproject
<span class="nv">INSTALL_STAMP</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span>.install.stamp
<span class="nv">POETRY</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>shell<span class="w"> </span><span class="nb">command</span><span class="w"> </span>-v<span class="w"> </span>poetry<span class="w"> </span><span class="m">2</span>><span class="w"> </span>/dev/null<span class="k">)</span>
<span class="nv">.DEFAULT_GOAL</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nb">help</span>
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">help</span>
<span class="nf">help</span><span class="o">:</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"Please use 'make <target>' where <target> is one of"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">""</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" install install packages and prepare environment"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" clean remove all temporary files"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" lint run the code linters"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" format reformat code"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">" test run all the tests"</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">""</span>
<span class="w"> </span>@echo<span class="w"> </span><span class="s2">"Check the Makefile to know exactly what each target is doing."</span>
<span class="nf">install</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="nf">$(INSTALL_STAMP)</span><span class="o">:</span><span class="w"> </span><span class="n">pyproject</span>.<span class="n">toml</span> <span class="n">poetry</span>.<span class="n">lock</span>
<span class="w"> </span>@if<span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Poetry could not be found. See https://python-poetry.org/docs/"</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">2</span><span class="p">;</span><span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>install
<span class="w"> </span>touch<span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">clean</span>
<span class="nf">clean</span><span class="o">:</span>
<span class="w"> </span>find<span class="w"> </span>.<span class="w"> </span>-type<span class="w"> </span>d<span class="w"> </span>-name<span class="w"> </span><span class="s2">"__pycache__"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="o">{}</span><span class="p">;</span>
<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span><span class="w"> </span>.coverage<span class="w"> </span>.mypy_cache
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">lint</span>
<span class="nf">lint</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>isort<span class="w"> </span>--profile<span class="o">=</span>black<span class="w"> </span>--lines-after-imports<span class="o">=</span><span class="m">2</span><span class="w"> </span>--check-only<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>black<span class="w"> </span>--check<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>--diff
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>flake8<span class="w"> </span>--ignore<span class="o">=</span>W503,E501<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>mypy<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>--ignore-missing-imports
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>bandit<span class="w"> </span>-r<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>-s<span class="w"> </span>B608
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">format</span>
<span class="nf">format</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>isort<span class="w"> </span>--profile<span class="o">=</span>black<span class="w"> </span>--lines-after-imports<span class="o">=</span><span class="m">2</span><span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>black<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">test</span>
<span class="nf">test</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>POETRY<span class="k">)</span><span class="w"> </span>run<span class="w"> </span>pytest<span class="w"> </span>./tests/<span class="w"> </span>--cov-report<span class="w"> </span>term-missing<span class="w"> </span>--cov-fail-under<span class="w"> </span><span class="m">100</span><span class="w"> </span>--cov<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
</pre></div>
<p>With that Makefile, anyone with <tt class="docutils literal">make</tt> and <tt class="docutils literal">poetry</tt> installed can hack on your project :)</p>
<div class="section" id="multiple-python-versions">
<h3>Multiple Python versions</h3>
<p><tt class="docutils literal">make test</tt> will run the tests with the default Python version.</p>
<p>In order to pick another Python version, to run the tests for example, simply rely on Poetry's features:</p>
<pre class="literal-block">
poetry env use 2.7
make test
</pre>
</div>
<div class="section" id="full-example-with-virtualenv">
<h3>Full Example with Virtualenv</h3>
<p>The equivalent with <tt class="docutils literal">virtualenv</tt>, which depends on <tt class="docutils literal">python3</tt> being available, and explicitly manages the creation of the <tt class="docutils literal">.venv</tt> folder.</p>
<div class="highlight"><pre><span></span><span class="nv">NAME</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span>superproject
<span class="nv">VENV</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>shell<span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="nv">$$</span><span class="o">{</span>VIRTUAL_ENV-.venv<span class="o">}</span><span class="k">)</span>
<span class="nv">PY3</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>shell<span class="w"> </span><span class="nb">command</span><span class="w"> </span>-v<span class="w"> </span>python3<span class="w"> </span><span class="m">2</span>><span class="w"> </span>/dev/null<span class="k">)</span>
<span class="nv">PYTHON</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/python
<span class="nv">INSTALL_STAMP</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/.install.stamp
<span class="nf">$(PYTHON)</span><span class="o">:</span>
<span class="w"> </span>@if<span class="w"> </span><span class="o">[</span><span class="w"> </span>-z<span class="w"> </span><span class="k">$(</span>PY3<span class="k">)</span><span class="w"> </span><span class="o">]</span><span class="p">;</span><span class="w"> </span><span class="k">then</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="s2">"Python 3 could not be found."</span><span class="p">;</span><span class="w"> </span><span class="nb">exit</span><span class="w"> </span><span class="m">2</span><span class="p">;</span><span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="k">$(</span>PY3<span class="k">)</span><span class="w"> </span>-m<span class="w"> </span>venv<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="nf">install</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="nf">$(INSTALL_STAMP)</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">PYTHON</span><span class="k">)</span> <span class="n">requirements</span>.<span class="n">txt</span> <span class="n">constraints</span>.<span class="n">txt</span>
<span class="w"> </span><span class="k">$(</span>PIP_INSTALL<span class="k">)</span><span class="w"> </span>-Ur<span class="w"> </span>requirements.txt<span class="w"> </span>-c<span class="w"> </span>constraints.txt
<span class="w"> </span>touch<span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">clean</span>
<span class="nf">clean</span><span class="o">:</span>
<span class="w"> </span>find<span class="w"> </span>.<span class="w"> </span>-type<span class="w"> </span>d<span class="w"> </span>-name<span class="w"> </span><span class="s2">"__pycache__"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>xargs<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="o">{}</span><span class="p">;</span>
<span class="w"> </span>rm<span class="w"> </span>-rf<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span><span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span><span class="w"> </span>.coverage<span class="w"> </span>.mypy_cache
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">lint</span>
<span class="nf">lint</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/isort<span class="w"> </span>--profile<span class="o">=</span>black<span class="w"> </span>--lines-after-imports<span class="o">=</span><span class="m">2</span><span class="w"> </span>--check-only<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>--virtual-env<span class="o">=</span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/black<span class="w"> </span>--check<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>--diff
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/flake8<span class="w"> </span>--ignore<span class="o">=</span>W503,E501<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/mypy<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>--ignore-missing-imports
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/bandit<span class="w"> </span>-r<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>-s<span class="w"> </span>B608
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">format</span>
<span class="nf">format</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/isort<span class="w"> </span>--profile<span class="o">=</span>black<span class="w"> </span>--lines-after-imports<span class="o">=</span><span class="m">2</span><span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span><span class="w"> </span>--virtual-env<span class="o">=</span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/black<span class="w"> </span>./tests/<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span><span class="w"> </span><span class="n">test</span>
<span class="nf">test</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>PYTHON<span class="k">)</span><span class="w"> </span>-m<span class="w"> </span>pytest<span class="w"> </span>./tests/<span class="w"> </span>--cov-report<span class="w"> </span>term-missing<span class="w"> </span>--cov-fail-under<span class="w"> </span><span class="m">100</span><span class="w"> </span>--cov<span class="w"> </span><span class="k">$(</span>NAME<span class="k">)</span>
</pre></div>
</div>
</div>
<div class="section" id="see-also">
<h2>See Also</h2>
<ul class="simple">
<li><a class="reference external" href="https://tech.davis-hansson.com/p/make/">Your Makefiles are wrong</a></li>
</ul>
</div>
The History of Firefox Remote Settings2020-10-19T00:00:00+02:002020-10-19T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2020-10-19:/the-history-of-firefox-remote-settings.html<p>When I started to write the first lines of this article, it was the last day of Ethan in our team. His departure marked the end of an era that I'll try to tell you about here.</p>
<div class="section" id="prehistory-2014">
<h2>Prehistory (~2014)</h2>
<p>While I was working at <a class="reference external" href="https://makina-corpus.com">Makina Corpus</a>, we were regularly coming …</p></div><p>When I started to write the first lines of this article, it was the last day of Ethan in our team. His departure marked the end of an era that I'll try to tell you about here.</p>
<div class="section" id="prehistory-2014">
<h2>Prehistory (~2014)</h2>
<p>While I was working at <a class="reference external" href="https://makina-corpus.com">Makina Corpus</a>, we were regularly coming up with the same kind of architectures: a database, a REST API, and frontend/mobile stuff. Other teams were also starting to consider the idea of «headless CMS».</p>
<p>I wanted to build a reusable backend, where you define your models in JSON and retrieve/store records.
<a class="reference external" href="https://blog.notmyidea.org/">Alexis</a> and I wrote down some pseudo code at a DjangoCon and we started the <a class="reference external" href="https://github.com/spiral-project/daybed/">Daybed project</a>, using <a class="reference external" href="https://trypyramid.com/">Pyramid</a>. It became functional and I even talked about it at <a class="reference external" href="https://archive.fosdem.org/2015/schedule/event/daybed/">FOSDEM</a>.</p>
<p>Alexis, <a class="reference external" href="https://devhub.io/developer/Natim">Rémy</a>, and <a class="reference external" href="https://ziade.org/">Tarek</a> were building Web APIs in the Cloud Services team at Mozilla, and there were needs to offer remote storage features to <a class="reference external" href="https://en.wikipedia.org/wiki/Firefox_OS">Firefox OS</a> app developers.</p>
<p>I was hired onto their team and we were about to roll out our first app: a «reading list» service (à-la Pocket).</p>
</div>
<div class="section" id="neolithic-2015-2016">
<h2>Neolithic (2015-2016)</h2>
<p>The team had accumulated a lot of experience with Web APIs, and one thing was certain: we shouldn't start each project from scratch. Even though we had a secret plan (a reusable service), we couldn't justify starting with that.</p>
<p>We gathered a lot of good practices and a handful tooling in one library: <a class="reference external" href="https://cliquet.readthedocs.io/en/latest/">Cliquet</a>. We hated the word «framework», but at least we wanted a standard way of building services (cf. <a class="reference external" href="https://mozilla-services.github.io/servicedenuages.fr/en/why-cliquet">introduction blog post</a> or <a class="reference external" href="https://mozilla-services.github.io/cliquet/talks/2015.07.pybcn/">talk at PyBCN</a>).</p>
<img alt="" src="images/remote-settings-history-cliquet.png" />
<p><em>Cliquet</em> had the notion of pluggable storage backends. And, because we had this vision of reusable services, we built a proof-of-concept where a service built with <em>Cliquet</em> would be storing its data in another service built with <em>Cliquet</em> 🤯 <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p>
<p>Eventually, this storage service became <a class="reference external" href="https://github.com/Kinto/">Kinto</a>, and <em>Cliquet</em> became <tt class="docutils literal">kinto.core</tt>.</p>
<p><a class="reference external" href="https://nicolas.perriault.net/">n1k0</a> and <a class="reference external" href="https://mathieu.agopian.info/">Magopian</a> joined the team, and built a nice offline-first SDK for Kinto. Plus a generic UI to CRUD records of any accessible Kinto server (kind of remote phpmyadmin). In order to generate forms from collections JSON schema, they released <a class="reference external" href="https://github.com/rjsf-team/react-jsonschema-form/">react-jsonschema-form</a> (which now has like 9k stars on Github!).</p>
<p>This was an exciting time! Kinto was on <a class="reference external" href="https://news.ycombinator.com/item?id=10994736">Hacker</a> <a class="reference external" href="https://news.ycombinator.com/item?id=10733164">news</a>, and the number of Github stars was going up!</p>
<p>At this time, there were many other challengers in the field of offline-first apps and backends as a service, and we were pretty close to them. <a class="reference external" href="https://michielbdejong.com/">Michiel de Jong</a>, the spec author of <a class="reference external" href="https://remotestorage.io/">Remote Storage</a>, joined Mozilla and was briefly in our team. <a class="reference external" href="https://arandomurl.com/">Dale Harvey</a> was already at Mozilla and was doing <a class="reference external" href="https://pouchdb.com/">PouchDB</a>. <a class="reference external" href="https://github.com/xbill82/">Luca Marchesini</a>, a Fullstack Fest friend, was also working on <a class="reference external" href="https://kuzzle.io/">Kuzzle</a>.</p>
<p>But internally in the company — above Tarek (our manager) — we were not able to defend our vision very well. Mozilla didn't seem ready to explore the dangerous grounds of hosting user data, at least not beyond <em>Firefox Sync</em>. And meanwhile, <em>Kinto</em> was still being mentioned as the Mozilla response to Facebook Parse and Google Firebase (completely inaccurate of course, because Kinto was never an official Mozilla project).</p>
<p>Luckily, shortly after, the <a class="reference external" href="https://wiki.mozilla.org/Firefox/Go_Faster">*Go Faster* initiative</a> was started, with the goal of shipping changes to Firefox faster than the current release cycle. If our system could synchronize user data between devices, then it could obviously do a uni-directional synchronization of read-only data from one admin to millions of clients. That's how we started to use <em>Kinto</em> to publish the malicious addons blocklist, the certificates revocation list, and some assets for Firefox Android, independently from release trains.</p>
<p>The Kinto client SDK was now in Firefox, and hundreds of millions of clients were pulling data from our servers at <tt class="docutils literal"><span class="pre">https://firefox.settings.services.mozilla.com</span></tt>.</p>
</div>
<div class="section" id="antiquity-2016-2018">
<h2>Antiquity (2016-2018)</h2>
<p>We were contributing to Gecko in the <tt class="docutils literal"><span class="pre">mozilla-central</span></tt> repo, and that was not so common for a team in the Cloud Services org. Mercurial, old-style JS, C++, XPCOM... we were not comfortable at all.</p>
<p><a class="reference external" href="http://betacantrips.com">Ethan</a> joined the team. He implemented the <tt class="docutils literal">chrome.storage</tt> Web Extension API using <em>Kinto</em>, and suddenly we were storing addons preferences of users (carefully encrypted on the client side of course).</p>
<p><em>Kinto</em> now had <a class="reference external" href="https://mozilla-services.github.io/servicedenuages.fr/en/kinto-at-mozilla">several use-cases</a> within Firefox. However, clearly, we still hadn't reached any critical mass. It was just another arrow on diagrams.</p>
<p>It was a boring and frustrating phase. Pushing the required configuration changes for each new use case. A lot of back and forth with the Ops for root objects manipulations. Responsibilities were limited.</p>
<p>We knew we could do much more, any piece of configuration or JSON in the repository could be migrated to our system! We could use it for internationalization! But we were not good door-to-door salesmen. Feedback, positive or negative, was almost nonexistent.</p>
<p>The revolutionary ideals of our initial team were fading away. We had to admit that we failed at promoting our vision of a user-facing storage backend (see <em>Thoughts</em> section). Self hosting, decentralized Web, Tor hidden services, were conversations at night around a drink, not in meeting rooms anymore. I'm pretty sure Mozilla even had a few projects running on Google Firebase at this time.</p>
<p>At the end of that period, most members of the original team had left, disillusioned. Ethan and I were the last «Kinto guys» at Mozilla.</p>
<div class="figure">
<img alt="" src="images/remote-settings-history-3-contributors.png" />
<p class="caption">Fortunately we had dependabot enabled on our repos so we don't look too lonely.</p>
</div>
</div>
<div class="section" id="modern-age-2018-2020">
<h2>Modern Age (2018-2020)</h2>
<p>In December 2016, we made a list of all the sources where <em>Firefox</em> would pull data from. No spoiler, it was too long.</p>
<p>Our goal was to unify several of these communication channels into one, and to start demanding a good reason to justify rolling out a new update source.</p>
<p>For most use-cases in Firefox, the <em>Kinto</em> API was too low level, and doing too much. As an example, the certificate configuration for signature verification <a class="reference external" href="https://searchfox.org/mozilla-esr60/rev/02b4ae79b24aae2346b1338e2bf095a571192061/services/common/blocklist-clients.js#439-474">was unnecessarily complex</a>. And there was no official cookbook.</p>
<p>Our new manager <a class="reference external" href="https://mostlygeek.com/">Benson</a> was defending the idea of <em>Remote Settings</em>. Two methods: <tt class="docutils literal">.get()</tt> and <tt class="docutils literal"><span class="pre">.on("sync",</span> <span class="pre">...)</span></tt>.
New collections of records had to be self-service, because we were not going to be able to continue the manual onboarding.</p>
<p>We had all the pieces in place to achieve that, it was mostly about improving developer experience. We refactored a number of things, added OAuth support, and Ethan integrated our clients and server with the <a class="reference external" href="https://github.com/mozilla-services/megaphone">broadcast service</a> for push notifications. We morphed the existing client integrations into a high level generic API, available for any component in Firefox. And on the server side, the original Kinto Admin had everything in its DNA to become the <em>Remote Settings Admin UI</em>.</p>
<img alt="" src="images/remote-settings-history-kinto-admin.png" />
<p>In December 2018, I presented <em>Remote Settings</em> in front of hundreds of Firefox developers! This felt energizing and great! On the other hand, I knew I was reaping the rewards of our team mates who had left.</p>
<img alt="" src="images/remote-settings-history-sf-evergreen.png" />
<p>We merged with the <em>SHIELD</em> team and were now part of the <em>Product Delivery</em> team. Now responsible of making off-train changes reliable and safe (ie. updates without reinstall).</p>
<p>Since we were only pulling read-only data from the server, the whole offline bi-directional sync code was overkill and slightly inefficient. For the sake of simplicity, I got rid of the <em>Kinto offline</em> library in Firefox and replaced it with plain Gecko specific IndexedDB code. Way easier to reason about, especially for a future code reader who wouldn't have the back story.</p>
<p>More and more critical components of Firefox, desktop or mobile, are now relying on <em>Remote Settings</em>. A/B testing, experiments metadata, user messages, features recommandations rules, list of password breaches for <a class="reference external" href="https://monitor.firefox.com/">Firefox Monitor</a>, password fields detection rules, localization packages, search partners configuration... All of them using our two methods API :)</p>
<p>We could not have been more successful. <em>Remote Settings</em> had become <em>THE</em> standard way to ship data in Firefox outside release trains! Some of us sometimes still call it <em>Kinto</em>.</p>
<p>In Summer 2020, there are approximately fifty collections on the server. The reliability of the pipeline is my main responibility, and to have a better overview of the whole thing, I mainly rely on two things: a unified client Telemetry, and a monitoring tool: <a class="reference external" href="https://github.com/mozilla-services/poucave/">Poucave</a>, that I'll present in another post.</p>
</div>
<div class="section" id="futurism-2021">
<h2>Futurism (2021-)</h2>
<p>Futurewise, the next obvious part is the Rust client, in order to have a unified experience across platforms, using a <a class="reference external" href="/leveraging-rust-in-python-and-javascript.html">~single code base</a> :)</p>
<p>There was an initiative recently to unify our experimentation solutions between desktop, mobile, websites, and other products. During their study, they legitimately asked why use <em>Remote Settings</em> to ship experiments metadata, and not JSON files behind a CDN.</p>
<p>The question is legitimate, and we could defend our pipeline because we have a validated security workflow, with VPN access, permissions management, some review and signoff features, push notifications, diff-based synchronization, content signature verification...</p>
<p>Nevertheless, at the same time, more and more use-cases are server-to-server or have automated publication from scripts. And some even built their own UI.</p>
<p>Maybe our <em>Remote Settings</em> service could be achieved with just something like a Git repo, static data, and a CDN in front.</p>
<p>Clients pull blobs and content signatures from a URL, each use-case being responsible of parsing these binaries as JSON or using them as plain resources or files. Permissions setup could be achieved using protected branches and submodules in the repo. No more UI to maintain, the reviewing process would just happen upfront, before merging the pull-request. No more backend and database with records and fields, just a few scheduled tasks or commit hooks.</p>
<p>A dumb pipe of static data, with complex workflows outside the system itself. A little bit <a class="reference external" href="/publish-your-pelican-blog-on-github-pages-via-travis-ci.html">like this blog</a> actually.</p>
<p>Why not!</p>
</div>
<div class="section" id="thoughts">
<h2>Thoughts</h2>
<p>I hope this long article helped you understand how a long term project can evolve and mutate. I thought it would be interesting to see it from within a company like Mozilla, often mentioned in headlines.</p>
<div class="section" id="what-made-the-success">
<h3>What made the success?</h3>
<p><em>Remote Settings</em> became a critical part of Firefox. Clearly, since it is leveraging only a subset of <em>Kinto</em>, the success can be largely attributed to the vision and efforts of our early team. We're still friends and I'm super proud of what we accomplished! Big up!</p>
<p>We have the very early days in memory, when Tarek had managed to free some of our time so that we could build prototypes and demos. Later, Benson took over and embraced our vision. Both were advertising the idea of a reusable service within the company while we were busy coding!</p>
<p>The patience of stakeholders who held our hand in order to land patches in Firefox massively contributed to the success. Special thanks to <a class="reference external" href="https://github.com/gijsk">Gijs</a>, <a class="reference external" href="https://github.com/linacambridge">Lina</a>, <a class="reference external" href="http://blog.queze.net/">Florian</a>, <a class="reference external" href="https://github.com/Standard8">Standard8</a>...</p>
<p>I believe it is also important to highlight how important our first «customers» were to the growth and adoption of <em>Remote Settings</em>. The security/crypto teams and the <a class="reference external" href="https://wiki.mozilla.org/Firefox/Activity_Stream">Activity Stream</a> (now UJET) deserve much credit (<a class="reference external" href="https://github.com/computerist">mgoodwin</a>, <a class="reference external" href="https://github.com/ncloudioj">Nanj</a>, <a class="reference external" href="https://insufficient.coffee/about/">JCJ</a>, <a class="reference external" href="https://github.com/rlr">Ricky</a>, <a class="reference external" href="https://github.com/piatra">Andrei</a>...). Their advocacy for the solution and working closely with us to make it better made a big difference.</p>
<p>Another factor is that we were consistently protecting the functional scope of the system. It is a data pipe: publish data on one side, reach the target audience reliably. It took a lot of effort to say «no», and keep the complexity trend downwards, instead of building dedicated features for specific use-cases.</p>
<p>These previous two combined — motivated early adopters and product focus — were crucial in adoption. We were able to provide a good customer service because we had participated in the early integrations and implementations for other teams. When new use-cases were joining us, we <em>almost</em> had everything covered already.</p>
<p>During our company meetings, we were making sure we would spend more time with people, and the least in front of computers. I don't cross the world to do what I can do from home! We had joyful moments and this happiness certainly contributed to the success :) Through the numerous use-cases, I got to know many teams in the company, and always worked in a fantastic atmosphere. I am happy to help, they seem happy with the solution, it is great!</p>
</div>
<div class="section" id="what-about-kinto">
<h3>What about Kinto?</h3>
<p>I don't want this article or section to sound like a retrospective of <em>Kinto</em> itself, but I think there are a few important things to underline.</p>
<p>The <a class="reference external" href="https://github.com/Kinto/">Kinto organization</a> is relatively quiet, but we have users, and <a class="reference external" href="https://dstaley.com/">Dylan</a> is dedicating an amazing amount of energy on the project! A lot of engineers at Mozilla are going to benefit from his efforts to migrate the Admin UI to Bootstrap 4 (3 years after the PR was started 🤣)! Kudos!</p>
<p>Nevertheless, we cannot deny that the <em>Kinto</em> community is not as flourishing as it used to be ;) And let's be clear: from the Mozilla standpoint, <em>Kinto</em> is just an implementation detail.</p>
<p>If I win the lottery and leave (the positive version of the bus factor idea), there is no guarantee that the next pair of eyes taking a fresh look at the <em>Remote Settings</em> architecture or the <em>Kinto</em> code base will decide to keep it. In other words, as long as <em>Kinto</em> is used in <em>Remote Settings</em>, the project will be maintained and be taken care of. Mozilla will continue to invest in <em>Kinto</em> as long as it has value and makes sense.</p>
</div>
<div class="section" id="what-if-i-had-to-do-it-over-again">
<h3>What if I had to do it over again?</h3>
<p>If someone would have come to us 5 years ago with the needs of a solution to update parts of Firefox without reinstall, what would we have done?</p>
<p>If we wouldn't have been desperate to «sell» Kinto internally, would we have used a database as-a-service with a CRUD API?</p>
<p>Alexis started <a class="reference external" href="http://getpelican.com/">Pelican</a> — a static blog generator — around 2011, 4 years before we were working on the first use-cases. I find it extremely ironic that the solution described in the <a class="reference external" href="#Futurism">Futurism</a> section would basically consist in applying the same principles to publish read-only data 🙃 <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a></p>
<p>The current approach also has a downside that I realized only very recently. Before, if someone wanted to contribute a new password recipe in Firefox, they would just have to add a line in the <tt class="docutils literal">.json</tt> and get their patch approved. Now the source of truth is the <em>Remote Settings</em> server. Adding a recipe means opening a ticket to request it, and a stakeholder to connect on the VPN and add it. If the source of truth was the repo, with jobs to publish data online for live updates, it would probably make more sense. Plus, it would save forks or the Thunderbird team to run <a class="reference external" href="https://thunderbird-settings.thunderbird.net/">their own Remote Settings instance</a>...</p>
<p>In 2015, JSON was everywhere and a pretty natural choice. But we had to complement it with a notion of attachments for heavy content. Today, I would probably consider going full binary for everything. JSON was problematic for content signatures, since there are many ways to serialize it (Unicode strings and float numbers, <a class="reference external" href="https://github.com/gibson042/canonicaljson-spec/issues/5">the worse</a>!). Something like <a class="reference external" href="https://en.wikipedia.org/wiki/CBOR">CBOR</a> may have helped.</p>
<p>Generally speaking — and here comes the self-flagellation part — I think that we could have done a better job if we would have studied more past research. As <a class="reference external" href="https://jlongster.com/How-I-Became-Better-Programmer">James Long wrote</a>, «<em>If you're excited about an idea, it's super tempting to sit down and immediately get going. But you shouldn't do that until you've done some cursory research about how people have solved it before.</em>». In our case, we could have considered using <a class="reference external" href="https://www.dotconferences.com/2019/12/james-long-crdts-for-mortals">CRDTs</a> to sync data, <a class="reference external" href="https://en.wikipedia.org/wiki/Merkle_tree">Merkle trees</a> for content signatures of partial diffs, or all the things available from the video games industry to update assets etc... and this is true for specifications too. I now think we made a big mistake when we decided to deviate from the Remote Storage spec while Michiel was in our team. Same with our custom Canonical JSON.</p>
<p>With regards to <em>Kinto</em> itself, it shows that the idea itself does not have much value (<a class="reference external" href="/releasing-software-ideas.html">I wrote about that already</a>). <em>Kinto</em> was a very good idea, and we could develop it to a certain point. We were experienced with code, but our limits became the limits of the project: we were very unfit to pitch ideas and do product marketing.</p>
<p>On the front of data sovereignity, great minds like Sir Tim Berners-Lee <a class="reference external" href="https://solidproject.org">are working on it</a>, and the need of a reusable backend for Web developers hasn't disappeared. <a class="reference external" href="https://appwrite.io/">Appwrite</a> and <a class="reference external" href="https://kuzzle.io/">kuzzle</a> (👏) seemed to be relevant open source solutions in 2020! I don't know about other projects like <a class="reference external" href="https://backendless.com/">https://backendless.com/</a> or <a class="reference external" href="http://hood.ie/">http://hood.ie/</a> though. Among the last challengers, <a class="reference external" href="http://postgrest.org/">PostgREST</a> and <a class="reference external" href="https://hasura.io/">Hasura</a> seem to be doing great.</p>
<p>But hey, in the precise context of <em>Remote Settings</em>, these solutions may not have fit anyway!</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Rémy reapplied a similar concept to build <a class="reference external" href="https://wiki.mozilla.org/Firefox_OS/Syncto">SyncTo</a>, a bridge from <em>Kinto</em> to <em>Firefox Sync</em> for <em>Firefox OS</em> developers.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>Ethan and n1k0 say that we had considered this idea. We can't recall why we discarded it. Maybe because we were too obscessed about selling Kinto internally? Or limitations of our signing infrastructure behind the VPN?</td></tr>
</tbody>
</table>
</div>
</div>
Leveraging Rust in Python and JavaScript2020-09-08T00:00:00+02:002020-09-08T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2020-09-08:/leveraging-rust-in-python-and-javascript.html<p>I had several opportunities to hack with Rust, but so far, besides <a class="reference external" href="https://github.com/mozilla/classify-client/">this very high loaded Web service</a> that runs in production, it was either on prototypes or on stuff that could have been implemented with any other language.</p>
<p>Recently, we had a concrete use-case where Rust would be beneficial …</p><p>I had several opportunities to hack with Rust, but so far, besides <a class="reference external" href="https://github.com/mozilla/classify-client/">this very high loaded Web service</a> that runs in production, it was either on prototypes or on stuff that could have been implemented with any other language.</p>
<p>Recently, we had a concrete use-case where Rust would be beneficial: share an implementation of a Canonical JSON serialization between the server in Python, and the clients in JavaScript, Swift, and Kotlin.</p>
<p>In order to guarantee the integrity of data between the server and the clients, we use content signatures. Canonical JSON is just a variant of JSON where each value has a single, unambiguous serialized form. Having a predictable JSON serialization is essential to get repeatable hashes of encoded data and be able to verify digital signatures. Sharing the same code accross server and clients makes it more robust, especially when it comes to handling funny corner cases of floats or unicode.</p>
<p>The <tt class="docutils literal">canonical_json</tt> <a class="reference external" href="https://crates.io/crates/canonical_json">crate</a> implements the serialization, using <a class="reference external" href="https://serde.rs/">Serde</a> and following <a class="reference external" href="https://github.com/gibson042/canonicaljson-spec">a spec</a>. It is fairly simple to use:</p>
<div class="highlight"><pre><span></span><span class="k">use</span><span class="w"> </span><span class="n">serde_json</span>::<span class="n">json</span><span class="p">;</span>
<span class="k">use</span><span class="w"> </span><span class="n">canonical_json</span>::<span class="n">ser</span>::<span class="n">to_string</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">to_string</span><span class="p">(</span><span class="o">&</span><span class="n">json</span><span class="o">!</span><span class="p">(</span><span class="s">"we ❤ Rust"</span><span class="p">)));</span>
<span class="p">}</span>
<span class="c1">// "we \u2665 Rust"</span>
</pre></div>
<div class="section" id="python-bindings">
<h2>Python bindings</h2>
<p>Our first goal is to be able to call this Rust library from Python. And it should be transparent, run on Linux and Mac OS, as any other library:</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">import</span> <span class="nn">canonicaljson</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">canonicaljson</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s2">"héo"</span><span class="p">:</span> <span class="mi">42</span><span class="p">})</span>
<span class="s1">'{"h</span><span class="se">\\</span><span class="s1">u00e9o":42}'</span>
</pre></div>
<p>Python is just a language, and has several implementations, Jython in Java, Pypy in Python, RustPython in Python... But what interests us here is the most common one: CPython, in C.</p>
<p>The Rust code must be compiled as a shared library (<tt class="docutils literal">.so</tt> file), Python must <a class="reference external" href="https://docs.python.org/3/library/ctypes.html#loading-shared-libraries">load it</a> and then call the exported symbol (<tt class="docutils literal"><span class="pre">canonical_json::ser::to_string()</span></tt>).</p>
<p>Since one side handles Python objects (eg. <tt class="docutils literal">dict</tt>) and the other side expects a Rust data type (cf. <tt class="docutils literal"><span class="pre">json!()</span></tt>), the whole challenge here will be to translate Python values in memory and pass them to Rust. Fortunately, in this modest use-case, we don't have to handle mutability or complex lifetimes, and the serializer just gives back a string in return. However, unlike most documented use-cases, the passed data is not «structured»: the input data can be any Python serializable object, and the destination in Rust is not a domain specific custom type, but the generic <tt class="docutils literal"><span class="pre">serde_json::json::Value</span></tt>.</p>
<p>Using <a class="reference external" href="https://github.com/PyO3/PyO3">PyO3</a>, it is quite straightforward to start. The main principle consists in starting a library crate, that imports both <tt class="docutils literal">pyo3</tt> and <tt class="docutils literal">canonical_json</tt> dependencies. The Rust function will be exposed in the Python module using a high-level macros:</p>
<div class="highlight"><pre><span></span><span class="cp">#[pymodule]</span>
<span class="k">fn</span> <span class="nf">canonicaljson</span><span class="p">(</span><span class="n">_py</span>: <span class="nc">Python</span><span class="p">,</span><span class="w"> </span><span class="n">m</span>: <span class="kp">&</span><span class="nc">PyModule</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">m</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="s">"__version__"</span><span class="p">,</span><span class="w"> </span><span class="fm">env!</span><span class="p">(</span><span class="s">"CARGO_PKG_VERSION"</span><span class="p">))</span><span class="o">?</span><span class="p">;</span>
<span class="w"> </span><span class="n">m</span><span class="p">.</span><span class="n">add_wrapped</span><span class="p">(</span><span class="n">wrap_pyfunction</span><span class="o">!</span><span class="p">(</span><span class="n">dumps</span><span class="p">))</span><span class="o">?</span><span class="p">;</span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span>
<span class="p">}</span>
<span class="cp">#[pyfunction]</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">dumps</span><span class="p">(</span><span class="n">py</span>: <span class="nc">Python</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span>: <span class="nc">PyObject</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">PyResult</span><span class="o"><</span><span class="n">PyObject</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Convert the Python object to a Serde value</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">python_to_serde</span><span class="p">(</span><span class="n">py</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">obj</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// Call Canonical JSON serializer</span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">to_string</span><span class="p">(</span><span class="o">&</span><span class="n">v</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">s</span><span class="p">.</span><span class="n">to_object</span><span class="p">(</span><span class="n">py</span><span class="p">)),</span>
<span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">PyErr</span>::<span class="n">new</span>::<span class="o"><</span><span class="n">PyTypeError</span><span class="p">,</span><span class="w"> </span><span class="n">_</span><span class="o">></span><span class="p">(</span><span class="fm">format!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="n">e</span><span class="p">))),</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">python_to_serde</span><span class="p">(</span><span class="n">py</span>: <span class="nc">Python</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span>: <span class="kp">&</span><span class="nc">PyObject</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="n">serde_json</span>::<span class="n">Value</span><span class="p">,</span><span class="w"> </span><span class="n">PyCanonicalJSONError</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// ... See full implementation</span>
<span class="w"> </span><span class="c1">// https://github.com/mozilla-services/python-canonicaljson-rs/blob/62599b24/src/lib.rs#L87-L167</span>
<span class="p">}</span>
</pre></div>
<p>In order to convert a generic <tt class="docutils literal">PyObject</tt> into the generic <tt class="docutils literal"><span class="pre">serde_json::Value</span></tt>, we will first try to <a class="reference external" href="https://docs.rs/pyo3/0.11.1/pyo3/conversion/trait.FromPyObject.html#tymethod.extract">extract</a> the Rust equivalents of Python basic types (<tt class="docutils literal">String</tt>, <tt class="docutils literal">bool</tt>, <tt class="docutils literal">u64</tt>, ...) from this Python object reference, and simply <a class="reference external" href="https://docs.serde.rs/serde_json/value/fn.to_value.html">instantiate Serde values</a>. For other types, we try to <a class="reference external" href="https://docs.rs/pyo3/0.11.1/pyo3/struct.PyObject.html#method.cast_as">cast the reference</a> to Python object types (<tt class="docutils literal">PyDict</tt>, <tt class="docutils literal">PyList</tt>, <tt class="docutils literal">PyTuple</tt>, ...) in order to recursively convert them. The code was mostly inspired <a class="reference external" href="https://github.com/mre/hyperjson/">by Matthias Endler's hyperjson</a>. See <a class="reference external" href="https://github.com/mozilla-services/python-canonicaljson-rs/blob/62599b24/src/lib.rs#L87-L167">full implementation</a>.</p>
<p>Using <a class="reference external" href="https://github.com/PyO3/maturin">maturin</a>, the above library crate can be built and published as a wheel on Pypi. Wheels save consumers from compiling the Rust part when installing the Python package, and Maturin takes care of packaging metadata etc.</p>
<div class="highlight"><pre><span></span><span class="c1"># pyproject.toml</span>
<span class="k">[build-system]</span>
<span class="n">requires</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="s">"maturin"</span><span class="p">]</span>
<span class="n">build-backend</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"maturin"</span>
<span class="k">[package.metadata.maturin]</span>
<span class="n">classifier</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s">"Intended Audience :: Developers"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"Programming Language :: Python"</span><span class="p">,</span>
<span class="w"> </span><span class="s">"Programming Language :: Rust"</span><span class="p">,</span>
<span class="p">]</span>
</pre></div>
<p><tt class="docutils literal">maturin build</tt> and <tt class="docutils literal">maturin publish</tt> just worked as expected.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">To be honest I haven't battle tested the multiplatform part extensively since my dev box and our servers run Linux.</p>
</div>
</div>
<div class="section" id="javascript-webassembly">
<h2>JavaScript & WebAssembly</h2>
<p>WebAssembly (or Wasm) is a binary format that a virtual machine can execute directly, without having to parse and compile the source code.</p>
<p>In the browser, a WebAssembly module is loaded as a Web page resource, and can be called transparently from JavaScript code.</p>
<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">canonicaljson</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="k">import</span><span class="p">(</span><span class="s2">"./node_modules/canonicaljson-wasm/canonicaljson_wasm.js"</span><span class="p">);</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">str</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">canonicaljson</span><span class="p">.</span><span class="nx">stringify</span><span class="p">({</span><span class="s2">"héo"</span><span class="o">:</span><span class="w"> </span><span class="mf">42</span><span class="p">});</span>
</pre></div>
<p>To achieve this, instead of compiling Rust to binary code that can only be executed by a specific operating system or processor, we will compile it to this universal binary format, using <tt class="docutils literal"><span class="pre">wasm-bindgen</span></tt>.</p>
<p>In order to expose our <tt class="docutils literal">canonical_json</tt> crate to Wasm, like for Python, we will have to create a library crate and to bind passed types. This binding crate will rely on <tt class="docutils literal"><span class="pre">wasm-bindgen</span></tt> and its <tt class="docutils literal"><span class="pre">serde-serialize</span></tt> feature, which does everything we need. Exposing functions and passing arbitrary data from JavaScript to Rust is relatively easy and well documented.</p>
<p>In our example, the main code of the wrapping crate can look like this:</p>
<div class="highlight"><pre><span></span><span class="k">use</span><span class="w"> </span><span class="n">wasm_bindgen</span>::<span class="n">prelude</span>::<span class="o">*</span><span class="p">;</span>
<span class="k">use</span><span class="w"> </span><span class="n">canonical_json</span>::<span class="n">ser</span>::<span class="n">to_string</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">cj_to_string</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">err_to_str</span><span class="p">(</span><span class="n">x</span>: <span class="nc">impl</span><span class="w"> </span><span class="n">std</span>::<span class="n">fmt</span>::<span class="n">Display</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">JsValue</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">JsValue</span>::<span class="n">from_str</span><span class="p">(</span><span class="o">&</span><span class="n">x</span><span class="p">.</span><span class="n">to_string</span><span class="p">())</span>
<span class="p">}</span>
<span class="cp">#[wasm_bindgen]</span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">stringify</span><span class="p">(</span><span class="n">val</span>: <span class="kp">&</span><span class="nc">JsValue</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">JsValue</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">serde_value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">val</span><span class="p">.</span><span class="n">into_serde</span><span class="p">().</span><span class="n">map_err</span><span class="p">(</span><span class="n">err_to_str</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span>
<span class="w"> </span><span class="n">JsValue</span>::<span class="n">from_str</span><span class="p">(</span><span class="o">&</span><span class="n">cj_to_string</span><span class="p">(</span><span class="o">&</span><span class="n">serde_value</span><span class="p">).</span><span class="n">unwrap</span><span class="p">())</span>
<span class="p">}</span>
</pre></div>
<p>We build this crate using <a class="reference external" href="https://github.com/rustwasm/wasm-pack">wasm-pack</a>. It will generate the expected <tt class="docutils literal">.js</tt> module.</p>
<p>I followed this <a class="reference external" href="https://developer.mozilla.org/en-US/docs/WebAssembly/Rust_to_wasm">tutorial on MDN</a> to tie everything up in an <a class="reference external" href="https://leplatrem.github.io/canonicaljson-wasm/">ugly demo page</a> using Webpack.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>There's something super exciting in knowing that the same Rust code, robust and performant, can now be used both from Python and JavaScript. Kotlin and Swift should be similarly straightforward.</p>
<p>Shipping bug fixes will now consist in releasing a new version of the serializer and bumping the dependency in the binding repos!</p>
<p>Even if our use-case was relatively modest, there is a lot of repetitive boiler plate code between the original library and the binding crates. And that's why the Firefox Sync team started the <a class="reference external" href="https://github.com/rfk/uniffi-rs">uniffi-rs</a> prototype: define your types and exposed interfaces in an <a class="reference external" href="https://en.wikipedia.org/wiki/IDL_specification_language">IDL file</a>, and it will take care of all the boilerplate and piping. Unfortunately it does not support the loose type <tt class="docutils literal">Any</tt> yet, that was necessary for the input of our serializer.</p>
<p>If the binding code remains trivial and featherweight, this idea of using Rust to share a codebase between several targets is a massive win!</p>
</div>
Python good practices in early 20202020-04-01T00:00:00+02:002020-04-01T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2020-04-01:/python-good-practices-in-early-2020.html<p>A great part of my job at Mozilla consists in maintaining the ecosystem of <a class="reference external" href="https://remote-settings.readthedocs.io">Firefox Remote Settings</a>, which is already a few years old.</p>
<p>But recently I had the chance to spin up a new Python project (<a class="reference external" href="https://github.com/mozilla-services/poucave/">Poucave</a>), and that was a good opportunity to look at recent trends that …</p><p>A great part of my job at Mozilla consists in maintaining the ecosystem of <a class="reference external" href="https://remote-settings.readthedocs.io">Firefox Remote Settings</a>, which is already a few years old.</p>
<p>But recently I had the chance to spin up a new Python project (<a class="reference external" href="https://github.com/mozilla-services/poucave/">Poucave</a>), and that was a good opportunity to look at recent trends that I missed :) This article goes through some of the choices we made, knowing that almost everything is obviously debatable. By the way, depending on the vigour of the Python community, please note that the information may not age well and could be outdated when you read it :)</p>
<div class="section" id="environment">
<h2>Environment</h2>
<p>Since we publish the app as a Docker container, we don't have to support multiple Python environments (eg. with <a class="reference external" href="https://tox.readthedocs.io">tox</a>). On the other hand, contributors may want to use <a class="reference external" href="https://github.com/pyenv/pyenv">Pyenv</a> to overcome the limitations of their operating system.</p>
<p>I like to keep tooling minimalist, and I can't explain why, but I also enjoy limiting the list of configuration files in the project root folder.</p>
<p>Probably because of my age, I'm familiar with <tt class="docutils literal">make</tt>. It's quite universal and popular. And using a single <a class="reference external" href="https://github.com/mozilla-services/poucave/blob/master/Makefile">Makefile</a> with the appropriate dependencies between targets, we can create the environment and run the application or the tests, by running only one make command.</p>
<p>I was used to Virtualenv, Pip, and requirements files. Common practice consists in having <a class="reference external" href="https://github.com/mozilla-services/poucave/tree/v1.19.0/requirements">a folder with a requirements file by environment</a>, and a constraints file for reproducible builds. We also setup <a class="reference external" href="https://app.dependabot.com/">Dependabot</a> on the repo to make sure our dependencies are kept up to date.</p>
<p>Now the cool kids use <a class="reference external" href="https://github.com/pipxproject/pipx">Pipx</a>, <a class="reference external" href="https://pipenv.pypa.io">Pipenv</a>, <a class="reference external" href="https://flit.readthedocs.io">Flit</a>, or <a class="reference external" href="https://python-poetry.org">Poetry</a>! Even if Poetry seemed to stand out, the debate was still virulent when the project was started, especially with regards to production installs and Docker integration. Therefore I didn't make any decision and remained conservative. I'd be happy to <a class="reference external" href="https://github.com/mozilla-services/poucave/issues/400">reconsider that choice</a>.</p>
<pre class="literal-block">
requirements/constraints.txt
requirements/default.txt
requirements/dev.txt
</pre>
<pre class="literal-block">
# constraints.txt
chardet==3.0.4 \
--hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \
--hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691
...
...
</pre>
<pre class="literal-block">
# default.txt
-c ./constraints.txt
aiohttp==3.6.2 \
--hash=sha256:1e984191d1ec186881ffaed4581092ba04f7c61582a177b187d3a2f07ed9719e \
--hash=sha256:50aaad128e6ac62e7bf7bd1f0c0a24bc968a0c0590a726d5a955af193544bcec \
...
...
</pre>
<p>The <tt class="docutils literal">Makefile</tt> would look like this:</p>
<div class="highlight"><pre><span></span><span class="nv">SOURCE</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span>poucave
<span class="nv">VENV</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span>.venv
<span class="nv">PYTHON</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/python3
<span class="nv">INSTALL_STAMP</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/.install.stamp
<span class="nf">install</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="nf">$(INSTALL_STAMP)</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">PYTHON</span><span class="k">)</span> <span class="n">requirements</span>/<span class="n">default</span>.<span class="n">txt</span> <span class="n">requirements</span>/<span class="n">constraints</span>.<span class="n">txt</span>
<span class="w"> </span><span class="k">$(</span>PIP_INSTALL<span class="k">)</span><span class="w"> </span>-Ur<span class="w"> </span>requirements/default.txt<span class="w"> </span>-c<span class="w"> </span>requirements/constraints.txt
<span class="w"> </span>touch<span class="w"> </span><span class="k">$(</span>INSTALL_STAMP<span class="k">)</span>
<span class="nf">$(PYTHON)</span><span class="o">:</span>
<span class="w"> </span>virtualenv<span class="w"> </span>--python<span class="o">=</span>python3<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="nf">serve</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>:
<span class="w"> </span><span class="nv">PYTHONPATH</span><span class="o">=</span>.<span class="w"> </span><span class="k">$(</span>PYTHON<span class="k">)</span><span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span>
</pre></div>
<p>When running <tt class="docutils literal">make serve</tt>, the virtualenv is created if missing, and the latest dependencies are installed only if outdated...</p>
<p><strong>update</strong> As you can see we don't even bother «activating» the virtualenv. Therefore we don't really need tools like <a class="reference external" href="https://virtualenvwrapper.readthedocs.io">virtualenvwrapper</a> to switch between environments. That being said, I really enjoy having the <a class="reference external" href="https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/virtualenvwrapper">Oh My Zsh plugin for it</a> that automatically activates a related virtualenv when I jump in a folder that contains a <tt class="docutils literal">.venv</tt> folder :) Thanks <a class="reference external" href="https://twitter.com/Exirel/status/1245677611138899970">Florian</a> for the feedback ;)</p>
<p>The CircleCI configuration file is as simple as:</p>
<div class="highlight"><pre><span></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2</span>
<span class="nt">jobs</span><span class="p">:</span>
<span class="w"> </span><span class="nt">test</span><span class="p">:</span>
<span class="w"> </span><span class="nt">docker</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">circleci/python:3.8</span>
<span class="w"> </span><span class="nt">steps</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">checkout</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">run</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Code lint</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">make lint</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">run</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Test</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">make tests</span>
</pre></div>
<p>You can also see how, using an <a class="reference external" href="https://github.com/mozilla-services/poucave/blob/9a102272071ade6ce1b7200707c0fbadc72a5cc1/Dockerfile#L34">ENTRYPOINT</a>, we can <a class="reference external" href="https://github.com/mozilla-services/poucave/blob/9a102272071ade6ce1b7200707c0fbadc72a5cc1/.circleci/config.yml#L34-L40">execute the tests from within the container</a> on Circle CI.</p>
<p>We also have a setup that <a class="reference external" href="https://github.com/mozilla-services/poucave/blob/9a102272071ade6ce1b7200707c0fbadc72a5cc1/.circleci/config.yml#L48-L67">publishes our Docker container to https://hub.docker.com</a> automatically.</p>
</div>
<div class="section" id="code-quality">
<h2>Code quality</h2>
<p>Running <a class="reference external" href="https://black.readthedocs.io">black</a> to format the code is now a no-brainer. We added <a class="reference external" href="https://github.com/timothycrosley/isort">isort</a> to sort and organize imports automatically too.</p>
<p>The working combination in one <tt class="docutils literal">Makefile</tt> target is:</p>
<div class="highlight"><pre><span></span><span class="nf">format</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/isort<span class="w"> </span>--line-width<span class="o">=</span><span class="m">88</span><span class="w"> </span>--lines-after-imports<span class="o">=</span><span class="m">2</span><span class="w"> </span>-rc<span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span><span class="w"> </span>--virtual-env<span class="o">=</span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/black<span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span>
</pre></div>
<p>Again, to avoid having an extra configuration file for <em>isort</em> we used CLI arguments :)</p>
<p>Since we want to verify code linting on the CI, we also have this <tt class="docutils literal">lint</tt> target, that additionnally runs <a class="reference external" href="https://pypi.org/project/flake8/">flake8</a> to detect unused imports or variables, and runs <a class="reference external" href="http://mypy-lang.org/">mypy</a> for type checking.</p>
<div class="highlight"><pre><span></span><span class="nf">lint</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/isort<span class="w"> </span>--line-width<span class="o">=</span><span class="m">88</span><span class="w"> </span>--check-only<span class="w"> </span>--lines-after-imports<span class="o">=</span><span class="m">2</span><span class="w"> </span>-rc<span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span><span class="w"> </span>--virtual-env<span class="o">=</span><span class="k">$(</span>VENV<span class="k">)</span>
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/black<span class="w"> </span>--check<span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span><span class="w"> </span>--diff
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/flake8<span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span><span class="w"> </span>--ignore<span class="o">=</span>W503,E501
<span class="w"> </span><span class="k">$(</span>VENV<span class="k">)</span>/bin/mypy<span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span><span class="w"> </span>--ignore-missing-imports
</pre></div>
<p>By the way, using type checking in your Python project is now pretty straightforward and enjoyable :)</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Optional</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">params</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="k">return</span> <span class="n">params</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span> <span class="k">if</span> <span class="n">params</span> <span class="k">else</span> <span class="p">[]</span>
</pre></div>
<p>Some plugins to guarantee the quality of your contributions exist for your favorite editor. And a commit-hook can also do the job:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span><span class="s2">"make format"</span><span class="w"> </span>><span class="w"> </span>.git/hooks/pre-commit
</pre></div>
<p>Check out <a class="reference external" href="https://pre-commit.com">pre-commit</a> or Rehan's <a class="reference external" href="https://github.com/rehandalal/therapist">therapist</a> for advanced commit hooks.</p>
<p>Note that there are complementary linting tools out there:</p>
<ul class="simple">
<li><a class="reference external" href="https://pypi.org/project/flake8-docstrings/">flake8-docstrings</a> or <a class="reference external" href="https://github.com/terrencepreilly/darglint">darglint</a> to validate your docstrings</li>
<li><a class="reference external" href="https://github.com/wemake-services/wemake-python-styleguide#what-we-are-about">wemake-python-styleguide</a> for a very strict Python linter</li>
<li><a class="reference external" href="https://bandit.readthedocs.io/en/latest/">bandit</a> to find common security issues</li>
</ul>
</div>
<div class="section" id="tests">
<h2>Tests</h2>
<p>There's almost no debate about <a class="reference external" href="https://pytest.readthedocs.io">pytest</a> nowadays. To me, the most appealing feature is the <a class="reference external" href="https://docs.pytest.org/en/latest/fixture.html">fixtures decorator</a>, to keep your tests <a class="reference external" href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>. It enables you to use dependency injection, object factories, connection setup, config changes...</p>
<div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">api_client</span><span class="p">():</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">APIClient</span><span class="p">()</span>
<span class="n">client</span><span class="o">.</span><span class="n">authenticate</span><span class="p">()</span>
<span class="k">yield</span> <span class="n">client</span>
<span class="n">client</span><span class="o">.</span><span class="n">logout</span><span class="p">()</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">mock_responses</span><span class="p">():</span>
<span class="k">with</span> <span class="n">responses</span><span class="o">.</span><span class="n">RequestsMock</span><span class="p">()</span> <span class="k">as</span> <span class="n">rsps</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">rsps</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">make_response</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">_make_response</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="k">return</span> <span class="p">{</span><span class="s2">"name"</span><span class="p">:</span> <span class="n">name</span><span class="p">}</span>
<span class="k">return</span> <span class="n">_make_response</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">test_api_get_gives_name</span><span class="p">(</span><span class="n">api_client</span><span class="p">,</span> <span class="n">mock_responses</span><span class="p">,</span> <span class="n">make_response</span><span class="p">):</span>
<span class="n">mock_responses</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">responses</span><span class="o">.</span><span class="n">GET</span><span class="p">,</span> <span class="s2">"/"</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">make_response</span><span class="p">(</span><span class="s2">"test"</span><span class="p">))</span>
<span class="n">resp</span> <span class="o">=</span> <span class="k">await</span> <span class="n">api_client</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">resp</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="s2">"test"</span>
</pre></div>
<p>The <a class="reference external" href="https://docs.pytest.org/en/latest/example/parametrize.html">parametrize feature</a> is also cool:</p>
<div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">parametrize</span><span class="p">(</span>
<span class="p">(</span><span class="s2">"n"</span><span class="p">,</span> <span class="s2">"expected"</span><span class="p">),</span> <span class="p">[</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span>
<span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">xfail</span><span class="p">((</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">)),</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">xfail</span><span class="p">(</span><span class="n">reason</span><span class="o">=</span><span class="s2">"some bug"</span><span class="p">)((</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">)),</span>
<span class="n">pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">skipif</span><span class="p">(</span><span class="s2">"sys.version_info >= (3,0)"</span><span class="p">)((</span><span class="mi">10</span><span class="p">,</span> <span class="mi">11</span><span class="p">)),</span>
<span class="p">]</span>
<span class="p">)</span>
<span class="k">def</span> <span class="nf">test_increment</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">expected</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">n</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">==</span> <span class="n">expected</span>
</pre></div>
<p>As usual, I like to have make the CI fail when code coverage isn't 100%. So <a class="reference external" href="https://github.com/pytest-dev/pytest-cov">pytest-cov</a> comes to the rescue:</p>
<div class="highlight"><pre><span></span><span class="nf">tests</span><span class="o">:</span><span class="w"> </span><span class="k">$(</span><span class="nv">INSTALL_STAMP</span><span class="k">)</span>
<span class="w"> </span><span class="nv">PYTHONPATH</span><span class="o">=</span>.<span class="w"> </span>.venv/bin/pytest<span class="w"> </span>tests<span class="w"> </span>--cov-report<span class="w"> </span>term-missing<span class="w"> </span>--cov-fail-under<span class="w"> </span><span class="m">100</span><span class="w"> </span>--cov<span class="w"> </span><span class="k">$(</span>SOURCE<span class="k">)</span>
</pre></div>
<p>Among the handy pytest extensions, I would mention:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/pytest-dev/pytest-mock/">pytest-mock</a> that provides <tt class="docutils literal">unittest.mock.patch</tt> as a <tt class="docutils literal">mocker</tt> fixture</li>
<li><a class="reference external" href="https://github.com/ionelmc/pytest-benchmark/">pytest-benchmark</a> that provides a benchmark fixture to measure execution performance</li>
<li><a class="reference external" href="https://github.com/joeyespo/pytest-watch">pytest-watch</a> for TDD</li>
</ul>
</div>
<div class="section" id="executing-and-configuring">
<h2>Executing and configuring</h2>
<p>In order to execute the package directly from the command-line (eg. <tt class="docutils literal">python poucave</tt>), use the <tt class="docutils literal">poucave/__main__.py</tt> file:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">poucave.app</span> <span class="kn">import</span> <span class="n">main</span>
<span class="n">main</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
</pre></div>
<p>The most appreciated libraries for advanced CLI parameters seem to be <a class="reference external" href="https://click.palletsprojects.com">Click</a> (declarative) and <a class="reference external" href="https://github.com/google/python-fire">Fire</a> (automatic).</p>
<p>For the Docker container, at Mozilla we follow our <a class="reference external" href="https://github.com/mozilla-services/Dockerflow">Dockerflow conventions</a>. This helps our operations team to treat all containers the same way, regardless of the implementation language etc.</p>
<p>A good take away for any application deployment is to manage configuration through environment variables (recommended in <a class="reference external" href="https://12factor.net/config">12factor</a> too).</p>
<p>We centralize all configuration values in a dedicated module <tt class="docutils literal">config.py</tt>, that reads variables from env.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="n">DEFAULT_TTL</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"DEFAULT_TTL"</span><span class="p">,</span> <span class="mi">60</span><span class="p">))</span>
<span class="n">LOG_LEVEL</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"LOG_LEVEL"</span><span class="p">,</span> <span class="s2">"INFO"</span><span class="p">)</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span>
<span class="n">LOGGING</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"version"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">"handlers"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"console"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"level"</span><span class="p">:</span> <span class="n">LOG_LEVEL</span><span class="p">,</span>
<span class="o">...</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>And then simply use it everywhere in the app:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">config</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">argv</span><span class="p">):</span>
<span class="n">logging</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">dictConfig</span><span class="p">(</span><span class="n">config</span><span class="o">.</span><span class="n">LOGGING</span><span class="p">)</span>
<span class="n">run</span><span class="p">(</span><span class="n">ttl</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">DEFAULT_TTL</span><span class="p">)</span>
</pre></div>
<p>During tests, config values are changed using <tt class="docutils literal">mock</tt>:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">mock</span>
<span class="k">def</span> <span class="nf">test_diagram_path</span><span class="p">():</span>
<span class="k">with</span> <span class="n">mock</span><span class="o">.</span><span class="n">patch</span><span class="o">.</span><span class="n">object</span><span class="p">(</span><span class="n">config</span><span class="p">,</span> <span class="s2">"DEFAULT_TTL"</span><span class="p">,</span> <span class="s2">"some.svg"</span><span class="p">):</span>
<span class="n">main</span><span class="p">()</span>
<span class="o">...</span>
</pre></div>
<p>But environment can be changed too using the built-in <tt class="docutils literal">monkeypatch</tt> fixture:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_lower_ttl</span><span class="p">(</span><span class="n">monkeypatch</span><span class="p">):</span>
<span class="n">monkeypatch</span><span class="o">.</span><span class="n">setenv</span><span class="p">(</span><span class="s2">"DEFAULT_TTL"</span><span class="p">,</span> <span class="s2">"10"</span><span class="p">)</span>
<span class="n">main</span><span class="p">()</span>
</pre></div>
<p>If you want to allow reading configuration from a file (<tt class="docutils literal">.env</tt> or <tt class="docutils literal">.ini</tt>), or have complex default values, or type casting, you can use <a class="reference external" href="https://github.com/henriquebastos/python-decouple">python-decouple</a> and read configuration values through the provided helper:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">decouple</span> <span class="kn">import</span> <span class="n">config</span>
<span class="n">DEBUG</span> <span class="o">=</span> <span class="n">config</span><span class="p">(</span><span class="s2">"DEBUG"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">cast</span><span class="o">=</span><span class="nb">bool</span><span class="p">)</span>
<span class="n">HEADERS</span> <span class="o">=</span> <span class="n">config</span><span class="p">(</span><span class="s2">"HEADERS"</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">"</span><span class="si">{}</span><span class="s2">"</span><span class="p">,</span> <span class="n">cast</span><span class="o">=</span><span class="k">lambda</span> <span class="n">v</span><span class="p">:</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">v</span><span class="p">))</span>
</pre></div>
</div>
<div class="section" id="a-web-app">
<h2>A Web app</h2>
<p>The project consisted in a minimalist API. There are plenty of candidates, but I wanted something ultra simple and leveraging <tt class="docutils literal">async</tt>/<tt class="docutils literal">await</tt>.</p>
<p><a class="reference external" href="https://github.com/huge-success/sanic">Sanic</a> and <a class="reference external" href="https://fastapi.tiangolo.com">FastAPI</a> seemed to stand out, but since my project needed an async HTTP client too, I decided to go with <a class="reference external" href="https://docs.aiohttp.org/en/stable/web.html">aiohttp</a> which provides both server and client stuff. <a class="reference external" href="https://www.python-httpx.org">httpx</a> used in <em>Sanic</em> could have been a good choice too.</p>
<p>The server code looks familiar:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">aiohttp</span> <span class="kn">import</span> <span class="n">web</span>
<span class="n">routes</span> <span class="o">=</span> <span class="n">web</span><span class="o">.</span><span class="n">RouteTableDef</span><span class="p">()</span>
<span class="nd">@routes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">hello</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">body</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"hello"</span><span class="p">:</span> <span class="s2">"poucave"</span><span class="p">}</span>
<span class="k">return</span> <span class="n">web</span><span class="o">.</span><span class="n">json_response</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">init_app</span><span class="p">(</span><span class="n">argv</span><span class="p">):</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">web</span><span class="o">.</span><span class="n">Application</span><span class="p">()</span>
<span class="n">app</span><span class="o">.</span><span class="n">add_routes</span><span class="p">(</span><span class="n">routes</span><span class="p">)</span>
<span class="k">return</span> <span class="n">app</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">argv</span><span class="p">):</span>
<span class="n">web</span><span class="o">.</span><span class="n">run_app</span><span class="p">(</span><span class="n">init_app</span><span class="p">(</span><span class="n">argv</span><span class="p">))</span>
</pre></div>
<p>And to centralize the HTTP client parameters within the app, we have this wrapper:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">asynccontextmanager</span>
<span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">AsyncGenerator</span>
<span class="kn">import</span> <span class="nn">aiohttp</span>
<span class="nd">@asynccontextmanager</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">ClientSession</span><span class="p">()</span> <span class="o">-></span> <span class="n">AsyncGenerator</span><span class="p">[</span><span class="n">aiohttp</span><span class="o">.</span><span class="n">ClientSession</span><span class="p">,</span> <span class="kc">None</span><span class="p">]:</span>
<span class="n">timeout</span> <span class="o">=</span> <span class="n">aiohttp</span><span class="o">.</span><span class="n">ClientTimeout</span><span class="p">(</span><span class="n">total</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">REQUESTS_TIMEOUT_SECONDS</span><span class="p">)</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"User-Agent"</span><span class="p">:</span> <span class="s2">"poucave"</span><span class="p">,</span> <span class="o">**</span><span class="n">config</span><span class="o">.</span><span class="n">DEFAULT_REQUESTS_HEADERS</span><span class="p">}</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">aiohttp</span><span class="o">.</span><span class="n">ClientSession</span><span class="p">(</span><span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="n">timeout</span><span class="p">)</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">session</span>
</pre></div>
<p>And we use the <a class="reference external" href="https://github.com/litl/backoff/">backoff</a> library to manage retries:</p>
<div class="highlight"><pre><span></span><span class="n">retry_decorator</span> <span class="o">=</span> <span class="n">backoff</span><span class="o">.</span><span class="n">on_exception</span><span class="p">(</span>
<span class="n">backoff</span><span class="o">.</span><span class="n">expo</span><span class="p">,</span>
<span class="p">(</span><span class="n">aiohttp</span><span class="o">.</span><span class="n">ClientError</span><span class="p">,</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">TimeoutError</span><span class="p">),</span>
<span class="n">max_tries</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">REQUESTS_MAX_RETRIES</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="c1"># + 1 because REtries.</span>
<span class="p">)</span>
<span class="nd">@retry_decorator</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">fetch_json</span><span class="p">(</span><span class="n">url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="o">-></span> <span class="nb">object</span><span class="p">:</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">ClientSession</span><span class="p">()</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</pre></div>
<p>In order to mock HTTP requests and responses in this setup, we use the <tt class="docutils literal">aiohttp_client</tt> fixture from <a class="reference external" href="https://github.com/aio-libs/pytest-aiohttp/">pytest-aiohttp</a> for the application part, and <a class="reference external" href="https://github.com/pnuckowski/aioresponses/">aioresponses</a> for the responses part:</p>
<div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">cli</span><span class="p">(</span><span class="n">aiohttp_client</span><span class="p">):</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">init_app</span><span class="p">()</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">aiohttp_client</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">mock_aioresponses</span><span class="p">(</span><span class="n">cli</span><span class="p">):</span>
<span class="n">test_server</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"http://</span><span class="si">{</span><span class="n">cli</span><span class="o">.</span><span class="n">host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">cli</span><span class="o">.</span><span class="n">port</span><span class="si">}</span><span class="s2">"</span>
<span class="k">with</span> <span class="n">aioresponses</span><span class="p">(</span><span class="n">passthrough</span><span class="o">=</span><span class="p">[</span><span class="n">test_server</span><span class="p">])</span> <span class="k">as</span> <span class="n">m</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">m</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">test_api_root_url</span><span class="p">(</span><span class="n">cli</span><span class="p">):</span>
<span class="n">data</span> <span class="o">=</span> <span class="k">await</span> <span class="n">cli</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">data</span><span class="p">[</span><span class="s2">"app"</span><span class="p">]</span> <span class="o">==</span> <span class="s2">"poucave"</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">test_api_fetches_info_from_source</span><span class="p">(</span><span class="n">cli</span><span class="p">,</span> <span class="n">mock_aioresponses</span><span class="p">):</span>
<span class="n">mock_aioresponses</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">config</span><span class="o">.</span><span class="n">SOURCE_URI</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="p">{</span><span class="s2">"success"</span><span class="p">:</span> <span class="kc">True</span><span class="p">})</span>
<span class="n">data</span> <span class="o">=</span> <span class="k">await</span> <span class="n">cli</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"/check-source"</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">data</span><span class="p">[</span><span class="s2">"success"</span><span class="p">]</span>
</pre></div>
</div>
<div class="section" id="misc">
<h2>Misc</h2>
<p>Some libraries and tools worth checking out:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/crsmithdev/arrow/">Arrow</a> for better dates & times for Python</li>
<li><a class="reference external" href="https://github.com/samuelcolvin/pydantic">Pydantic</a> for data parsing and validation</li>
<li><a class="reference external" href="https://www.attrs.org">attrs</a> for a smart alternative to named tuples</li>
<li><a class="reference external" href="https://github.com/cgarciae/pypeln">Pypeln</a> for concurrent async pipelines</li>
<li><a class="reference external" href="https://github.com/hawkowl/towncrier">towncrier</a> to automate CHANGELOG entries</li>
<li><a class="reference external" href="https://www.uvicorn.org">uvicorn</a> for a performant ASGI server</li>
</ul>
<p><strong>update</strong>: Disclaimer: I haven't used all of them. I just saw them in several projects :)</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>I hope you found this article interesting! And most importantly, that you'll have the opportunity to leverage all these tools in your projects :)</p>
<p>If you think something in this article is utterly wrong, please shout out!</p>
<p>Thanks <a class="reference external" href="https://github.com/areski">Areski</a> and <a class="reference external" href="https://github.com/glasserc">Ethan</a> for your early feedback!</p>
</div>
JavaScript return await2019-11-07T00:00:00+01:002019-11-07T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2019-11-07:/javascript-return-await.html<p>A very short article about one of the recent bugs I carelessly designed :)</p>
<p>I wrote the code below and had certain expectations: no matter what happens, catch the error and return a fallback value.</p>
<div class="highlight"><pre><span></span><span class="k">async</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">dl</span><span class="p">(</span><span class="nx">uri</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">resp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">uri</span><span class="p">);</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span>
<span class="w"> </span><span class="k">return …</span></pre></div><p>A very short article about one of the recent bugs I carelessly designed :)</p>
<p>I wrote the code below and had certain expectations: no matter what happens, catch the error and return a fallback value.</p>
<div class="highlight"><pre><span></span><span class="k">async</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">dl</span><span class="p">(</span><span class="nx">uri</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">resp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">uri</span><span class="p">);</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">data</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="nx">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">warn</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">{</span><span class="nx">success</span><span class="o">:</span><span class="w"> </span><span class="kc">false</span><span class="p">};</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Since I find it a bit verbose, I rewrote it this way:</p>
<div class="highlight"><pre><span></span><span class="k">async</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">dl</span><span class="p">(</span><span class="nx">uri</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">resp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">uri</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="nx">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">warn</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">{</span><span class="nx">success</span><span class="o">:</span><span class="w"> </span><span class="kc">false</span><span class="p">};</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Before <em>ESLint</em> could complain about <a class="reference external" href="https://eslint.org/docs/rules/no-return-await">no-return-await</a>), I removed this ugly redundant use of <tt class="docutils literal">await</tt> on a <tt class="docutils literal">return</tt> value. The function returns a promise after all!</p>
<div class="highlight"><pre><span></span><span class="k">async</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">dl</span><span class="p">(</span><span class="nx">uri</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">try</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">const</span><span class="w"> </span><span class="nx">resp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">uri</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="nx">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">warn</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="p">{</span><span class="nx">success</span><span class="o">:</span><span class="w"> </span><span class="kc">false</span><span class="p">};</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>Nice and short!</p>
<p>Maybe you did, but at the time I didn't notice that I had just changed the behaviour of the function. Now, when a JSON parsing error occurs, it will be thrown at the caller instead of returning the fallback value! That's not what I intended and now the code can be misinterpreted :)</p>
<p>ESLint is smart enough to take the <tt class="docutils literal">try {} catch () {}</tt> into account, my inner voice wasn't.</p>
<p>Conclusion: use your <a class="reference external" href="https://blog.mathieu-leplatre.info/your-tests-as-your-specs.html">tests to specify</a> the behaviour of your code!</p>
Check content hash between server and client2019-04-29T00:00:00+02:002019-04-29T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2019-04-29:/check-content-hash-between-server-and-client.html<p>In order to make sure that your remote content was fetched successfully by your client,
we can use a bit of cryptography.</p>
<p>A simple way is to compute a hash on the server, and let the client compare the
hash for the content that was downloaded.</p>
<div class="section" id="what-s-a-hash">
<h2>What's a hash?</h2>
<p>Given …</p></div><p>In order to make sure that your remote content was fetched successfully by your client,
we can use a bit of cryptography.</p>
<p>A simple way is to compute a hash on the server, and let the client compare the
hash for the content that was downloaded.</p>
<div class="section" id="what-s-a-hash">
<h2>What's a hash?</h2>
<p>Given two pieces of data, a cryptographic hash function will return two different (fixed-length) values.</p>
<p>The hash function should never return the same result for two different inputs, a.k.a «collision».</p>
<p>In this article, we'll use SHA-256 (Secure Hash Algorithm, with a 256 bits output). Given any content, the function
will return 256 bits (or 32 bytes). Each byte can be represented in hexadecimal (2 characters, from <tt class="docutils literal">00</tt> to <tt class="docutils literal">FF</tt>),
and thus becomes a 64 characters string (a.k.a. «hex digest»).</p>
<p>I really enjoyed watching <a class="reference external" href="https://www.youtube.com/watch?v=S9JGmA5_unY">How secure is 256 bit security?</a> by 3Blue1Brown. Note that in 2017, Google presented a <a class="reference external" href="https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html">practical technique to break SHA1</a> (used in Git for example).</p>
</div>
<div class="section" id="on-the-server">
<h2>On the server</h2>
<p>In Python (as with most languages) it is straightforward:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">hashlib</span>
<span class="n">content</span> <span class="o">=</span> <span class="s2">"Get up, stand up, don't give up the fight"</span> <span class="c1"># or ``file.read()``</span>
<span class="n">hasher</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">sha256</span><span class="p">()</span>
<span class="n">hasher</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
<span class="nb">hash</span> <span class="o">=</span> <span class="n">hasher</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span>
<span class="c1"># "04cb9657d1a1a34ccd4f30252a061c36e45b2a5afff86e4c91fa778fa70400eb"</span>
</pre></div>
<p>Ideally you would deliver this string to the client somewhere in your application data,
or in the HTTP response headers etc.</p>
</div>
<div class="section" id="on-the-client">
<h2>On the client</h2>
<p>In JavaScript, we can leverage the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest">Crypto API</a>
to compute the SHA-256 of some content.</p>
<p>First, obtain the bytes array of the content.</p>
<p>For a <tt class="docutils literal">String</tt>:</p>
<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">text</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"Get up, stand up, don't give up the fight"</span><span class="p">;</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">encoder</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">TextEncoder</span><span class="p">();</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">encoder</span><span class="p">.</span><span class="nx">encode</span><span class="p">(</span><span class="nx">text</span><span class="p">);</span>
</pre></div>
<p>Or for a URL:</p>
<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">resp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">fetch</span><span class="p">(</span><span class="nx">url</span><span class="p">);</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">buffer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">resp</span><span class="p">.</span><span class="nx">arrayBuffer</span><span class="p">();</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">bytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nb">Uint8Array</span><span class="p">(</span><span class="nx">buffer</span><span class="p">);</span>
</pre></div>
<p>And then compute the hash:</p>
<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">hashBuffer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">await</span><span class="w"> </span><span class="nx">crypto</span><span class="p">.</span><span class="nx">subtle</span><span class="p">.</span><span class="nx">digest</span><span class="p">(</span><span class="s2">"SHA-256"</span><span class="p">,</span><span class="w"> </span><span class="nx">bytes</span><span class="p">);</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">hashBytes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nb">Uint8Array</span><span class="p">(</span><span class="nx">hashBuffer</span><span class="p">);</span>
<span class="c1">// hex digest of bytes</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">hash</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Array</span><span class="p">.</span><span class="kr">from</span><span class="p">(</span><span class="nx">hashBytes</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">b</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="nx">b</span><span class="p">.</span><span class="nx">toString</span><span class="p">(</span><span class="mf">16</span><span class="p">).</span><span class="nx">padStart</span><span class="p">(</span><span class="mf">2</span><span class="p">,</span><span class="w"> </span><span class="s2">"0"</span><span class="p">))</span>
<span class="w"> </span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="s2">""</span><span class="p">);</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">hash</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nx">serverHash</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">throw</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="ne">Error</span><span class="p">(</span><span class="s2">"Bad content"</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="going-further">
<h2>Going further</h2>
<p>This hash verification is pretty solid to make sure that your data was downloaded and
fetched successfully. However, it does not guarantee authenticity, since anybody
can compute the SHA-256 function result without having any specific private key.</p>
<p>In order to prevent <a class="reference external" href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack">man-in-the-middle attacks</a>,
where someone could alter the content and deliver the modified hash values to the client,
you should use signatures. In this model, the server computes a hash using a private key, and the client
verifies the hash using a public key.</p>
<p>Usually, we use <a class="reference external" href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm">Elliptic Curve DSA</a> for that.</p>
</div>
Handling requests timeout in Python2019-04-18T00:00:00+02:002019-04-18T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2019-04-18:/handling-requests-timeout-in-python.html<p>Being optimistic is sometimes a disadvantage. When we make calls to an API, we usually test it under ideal conditions. For example, we make sure the client behaves as expected against a real HTTP server that runs locally, in our CI or devbox.</p>
<p>Let's be honest, we rarely test the …</p><p>Being optimistic is sometimes a disadvantage. When we make calls to an API, we usually test it under ideal conditions. For example, we make sure the client behaves as expected against a real HTTP server that runs locally, in our CI or devbox.</p>
<p>Let's be honest, we rarely test the consequences of a faulty server in our client code. Shit happens in production, when the service is overloaded or the network becomes unreliable and flaky. Within an architecture based on micro-services, this can lead to a chain reaction that can come tumbling down like a house of cards.</p>
<p>In this article, I will show you the basics to handle HTTP requests timeout in Python, using:</p>
<ul class="simple">
<li>the popular <a class="reference external" href="https://python-requests.org">requests</a> library</li>
<li><a class="reference external" href="https://github.com/litl/backoff/">backoff</a>, a handful retry library</li>
<li><a class="reference external" href="https://github.com/shopify/toxiproxy">toxiproxy</a>, a proxy to simulate network chaos</li>
</ul>
<div class="section" id="timeouts-in-requests">
<h2>Timeouts in <tt class="docutils literal">requests</tt></h2>
<p>We all use <tt class="docutils literal">requests</tt>. But «<em>what is the default timeout for your HTTP calls?</em>» may ask your ops on duty.</p>
<p>Don't feel bad, I didn't know either. <tt class="docutils literal">requests</tt> takes it from <tt class="docutils literal">urllib3</tt> which itself take it from the standard <tt class="docutils literal">socket</tt> module, which... does not define it, and <a class="reference external" href="https://github.com/python/cpython/blob/3eca28c61363a03b81b9fb12775490d6e42d8ecf/Modules/socketmodule.c#L6553-L6557">seems to be none</a>.</p>
<p>Best way to make sure you know: make it configurable.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">from</span> <span class="nn">requests.adapters</span> <span class="kn">import</span> <span class="n">TimeoutSauce</span>
<span class="n">REQUESTS_TIMEOUT_SECONDS</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"REQUESTS_TIMEOUT_SECONDS"</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span>
<span class="k">class</span> <span class="nc">CustomTimeout</span><span class="p">(</span><span class="n">TimeoutSauce</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">if</span> <span class="n">kwargs</span><span class="p">[</span><span class="s2">"connect"</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">kwargs</span><span class="p">[</span><span class="s2">"connect"</span><span class="p">]</span> <span class="o">=</span> <span class="n">REQUESTS_TIMEOUT_SECONDS</span>
<span class="k">if</span> <span class="n">kwargs</span><span class="p">[</span><span class="s2">"read"</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">kwargs</span><span class="p">[</span><span class="s2">"read"</span><span class="p">]</span> <span class="o">=</span> <span class="n">REQUESTS_TIMEOUT_SECONDS</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="c1"># Set it globally, instead of specifying ``timeout=..`` kwarg on each call.</span>
<span class="n">requests</span><span class="o">.</span><span class="n">adapters</span><span class="o">.</span><span class="n">TimeoutSauce</span> <span class="o">=</span> <span class="n">CustomTimeout</span>
</pre></div>
<p>Now, any request failing to connect or read data after <tt class="docutils literal">REQUESTS_TIMEOUT_SECONDS</tt> will raise <tt class="docutils literal">requests.exceptions.ConnectTimeout</tt> and <tt class="docutils literal">requests.exceptions.ReadTimeout</tt> errors. These two can be caught under <tt class="docutils literal">requests.exceptions.Timeout</tt>.</p>
</div>
<div class="section" id="retry-failing-requests">
<h2>Retry failing requests</h2>
<p>The same way we urge on hiting the refresh button but some page does not load, you may want your program to retry some failing requests before crashing completely.</p>
<p>By default, <tt class="docutils literal">requests</tt> will retry 0 times. You can specify it using <tt class="docutils literal">max_retries</tt>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">REQUESTS_MAX_RETRIES</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"REQUESTS_MAX_RETRIES"</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
<span class="n">adapter</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">adapters</span><span class="o">.</span><span class="n">HTTPAdapter</span><span class="p">(</span><span class="n">max_retries</span><span class="o">=</span><span class="n">REQUESTS_MAX_RETRIES</span><span class="p">)</span>
<span class="n">session</span><span class="o">.</span><span class="n">mount</span><span class="p">(</span><span class="s1">'https://'</span><span class="p">,</span> <span class="n">adapter</span><span class="p">)</span>
</pre></div>
<p>This approach has some limitations: it will only retry failing connections or data read. If the requests made it to the server but got 503 in return (from a reverse proxy, load balancer, or whatever) then it won't retry it.</p>
<p>That's why I truely recommend the <a class="reference external" href="https://github.com/litl/backoff/">backoff</a> library, which makes it super easy to retry any failing block of code using decorators. It has many cool features, it has several strategies to introduce delays betweens retries, can introduce <a class="reference external" href="https://en.wikipedia.org/wiki/Jitter">jitter</a>, execute callbacks on success or errors etc.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">backoff</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">REQUESTS_MAX_RETRIES</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">"REQUESTS_MAX_RETRIES"</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span>
<span class="k">class</span> <span class="nc">ServerError</span><span class="p">(</span><span class="n">requests</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">HTTPError</span><span class="p">):</span>
<span class="k">pass</span>
<span class="c1"># Re-usable decorator with exponential wait.</span>
<span class="n">retry_timeout</span> <span class="o">=</span> <span class="n">backoff</span><span class="o">.</span><span class="n">on_exception</span><span class="p">(</span>
<span class="n">wait_gen</span><span class="o">=</span><span class="n">backoff</span><span class="o">.</span><span class="n">expo</span><span class="p">,</span>
<span class="n">exception</span><span class="o">=</span><span class="p">(</span>
<span class="n">ServerError</span><span class="p">,</span>
<span class="n">requests</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">Timeout</span><span class="p">,</span>
<span class="n">requests</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">ConnectionError</span>
<span class="p">),</span>
<span class="n">max_tries</span><span class="o">=</span><span class="n">REQUESTS_MAX_RETRIES</span><span class="p">,</span>
<span class="p">)</span>
<span class="nd">@retry_timeout</span>
<span class="k">def</span> <span class="nf">fetch_server_info</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">resp</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">SERVER_URL</span><span class="p">)</span>
<span class="k">if</span> <span class="n">resp</span><span class="o">.</span><span class="n">status_code</span> <span class="o">>=</span> <span class="mi">500</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">ServerError</span><span class="p">(</span><span class="s2">"Boom!"</span><span class="p">,</span> <span class="n">response</span><span class="o">=</span><span class="n">resp</span><span class="p">)</span>
<span class="k">return</span> <span class="n">resp</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
</pre></div>
</div>
<div class="section" id="simulate-bad-network-conditions">
<h2>Simulate Bad Network Conditions</h2>
<p>They are several solutions out there to simulate faulty connections and timeouts. I remember that Tarek was working on <a class="reference external" href="https://github.com/community-libs/vaurien">Vaurien</a> a few years back, Netflix has <a class="reference external" href="https://github.com/Netflix/chaosmonkey#readme">Chaos Monkey</a>, and Shopify offers <a class="reference external" href="https://github.com/shopify/toxiproxy">toxiproxy</a>.</p>
<p>I found the latter convenient enough to get started and do what I had in mind. They all sit between your server and your client, and can receive commands to start or stop manipulating the pipe between the client and the upstream server.</p>
<p>On a recent Ubuntu, <tt class="docutils literal">toxiproxy</tt> is available out of the box:</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>toxiproxy<span class="w"> </span>toxiproxy-cli
</pre></div>
<p>The service runs in the background, and its configuration is done using the CLI tool. For example, we'll run a proxy to our local API that is running on <a class="reference external" href="http://localhost:8888">http://localhost:8888</a>:</p>
<div class="highlight"><pre><span></span>toxiproxy-cli<span class="w"> </span>create<span class="w"> </span>fantastic_api_dev<span class="w"> </span>-l<span class="w"> </span>localhost:22222<span class="w"> </span>-u<span class="w"> </span>localhost:8888
</pre></div>
<p>Then we'll add a 5 seconds latency:</p>
<div class="highlight"><pre><span></span>toxiproxy-cli<span class="w"> </span>toxic<span class="w"> </span>add<span class="w"> </span>fantastic_api_dev<span class="w"> </span>--toxicName<span class="w"> </span>latency_downstream<span class="w"> </span>-t<span class="w"> </span>latency<span class="w"> </span>-a<span class="w"> </span><span class="nv">latency</span><span class="o">=</span><span class="m">5000</span>
</pre></div>
<p>Accessing our service at <a class="reference external" href="http://localhost:22222">http://localhost:22222</a> will now take a lot longer than usual. Check out the list of available <a class="reference external" href="https://github.com/shopify/toxiproxy#toxics">toxics</a> for more fun :)</p>
<p>To remove an existing one, just do:</p>
<div class="highlight"><pre><span></span>toxiproxy-cli<span class="w"> </span>toxic<span class="w"> </span>delete<span class="w"> </span>fantastic_api_dev<span class="w"> </span>--toxicName<span class="w"> </span>latency_downstream
</pre></div>
<p>The whole idea of such a service is to be able to introduce some network hazards in your integration tests. Basically, it consists in using the <a class="reference external" href="https://github.com/douglas/toxiproxy-python">Python client library of toxiproxy</a>:</p>
<div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>toxiproxy-python
</pre></div>
<p>And setup the toxics in your tests setup:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">toxiproxy</span> <span class="kn">import</span> <span class="n">Toxiproxy</span>
<span class="n">toxiserver</span> <span class="o">=</span> <span class="n">Toxiproxy</span><span class="p">()</span>
<span class="n">toxiserver</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">"fantastic_api_dev"</span><span class="p">,</span> <span class="n">upstream</span><span class="o">=</span><span class="s2">"localhost:8888"</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">LatencyTest</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">setUpClass</span><span class="p">(</span><span class="bp">cls</span><span class="p">):</span>
<span class="bp">cls</span><span class="o">.</span><span class="n">proxy</span> <span class="o">=</span> <span class="n">toxiserver</span><span class="o">.</span><span class="n">get_proxy</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">"fantastic_api_dev"</span><span class="p">)</span>
<span class="bp">cls</span><span class="o">.</span><span class="n">proxy</span><span class="o">.</span><span class="n">add_toxic</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">"latency_downstream"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="s2">"latency"</span><span class="p">,</span> <span class="n">attributes</span><span class="o">=</span><span class="p">{</span><span class="s2">"latency"</span><span class="p">:</span> <span class="mi">500</span><span class="p">})</span>
<span class="bp">cls</span><span class="o">.</span><span class="n">proxy_url</span> <span class="o">=</span> <span class="s2">"http://"</span> <span class="o">+</span> <span class="bp">cls</span><span class="o">.</span><span class="n">proxy</span><span class="o">.</span><span class="n">listen</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">tearDownClass</span><span class="p">(</span><span class="bp">cls</span><span class="p">):</span>
<span class="bp">cls</span><span class="o">.</span><span class="n">proxy</span><span class="o">.</span><span class="n">destroy_toxic</span><span class="p">(</span><span class="s2">"latency_downstream"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_client_raises_error</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">APIClient</span><span class="p">(</span><span class="n">server</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">proxy_url</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">assertRaises</span><span class="p">():</span>
<span class="n">client</span><span class="o">.</span><span class="n">fetch_user_info</span><span class="p">()</span>
</pre></div>
<div class="figure align-center">
<img alt="" src="/images/quiet-monkey.gif" />
</div>
<p>See also:</p>
<ul class="simple">
<li>Peter's <a class="reference external" href="https://www.peterbe.com/plog/best-practice-with-retries-with-requests">Best practice with retries with requests</a></li>
<li>In <a class="reference external" href="https://github.com/kennethreitz/requests3#feature-support">requests 3</a> timeouts are required</li>
</ul>
</div>
Degrees of Philosophy2018-03-24T20:25:00+01:002018-03-24T20:25:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2018-03-24:/degrees-of-philosophy.html<p>A while ago, I was enjoying the <a class="reference external" href="http://www.hackernewsletter.com/">Hackernewsletter</a>, and I got amused by how two items of that edition caught my attention :)</p>
<p>The first one was <a class="reference external" href="https://github.com/kennethreitz/requests-html">request-html</a>, which binds the <a class="reference external" href="https://github.com/kennethreitz/requests/">requests</a> Python library to a real browser. It can be pleasant for scraping websites, end-to-end testing of Web applications or …</p><p>A while ago, I was enjoying the <a class="reference external" href="http://www.hackernewsletter.com/">Hackernewsletter</a>, and I got amused by how two items of that edition caught my attention :)</p>
<p>The first one was <a class="reference external" href="https://github.com/kennethreitz/requests-html">request-html</a>, which binds the <a class="reference external" href="https://github.com/kennethreitz/requests/">requests</a> Python library to a real browser. It can be pleasant for scraping websites, end-to-end testing of Web applications or turning websites into APIs. It reminded me <a class="reference external" href="https://github.com/makinacorpus/spynner/">Spynner</a>, that <a class="reference external" href="https://github.com/kiorky">Kiorky</a> wrote like 8 years ago. And obviously <a class="reference external" href="http://casperjs.org/">our esteemed CasperJS</a> <3</p>
<p>The second one was <a class="reference external" href="https://www.sixdegreesofwikipedia.com/">6 degrees of Wikipedia</a>, which builds a graph database of links between articles. It's absolutely fascinating! You can see how many «navigation paths» exist between two notions for example. By the way did you know that there are 2306 ways to navigate <a class="reference external" href="https://www.sixdegreesofwikipedia.com/?source=Erlang%20%28programming%20language%29&target=Barbra%20Streisand">from Barbara Streisand to the Erlang programming language</a> in 4 steps? I'm sure you'll be able to make use of that at your next family dinner.</p>
<p>It reminded me <a class="reference external" href="https://www.youtube.com/watch?v=mu4lJpFQUvU">a funny Youtube video</a> where the guys play some «Wikipedia navigation» games. Like both players open a random article, the winner is the first one to reach the opponent's article by only following links from one page to another. Or both players open a random article, and the first one to reach the «Philosophy» page wins.</p>
<p>While being bored at the airport, I dediced to go for a <a class="reference external" href="filename}../Personal/joy_micro_hacks.rst">microhack</a>!
I will write a small script to figure out how many hops it takes in average to go from any article to the Philosphy page. And following the <strong>first link</strong> of the article like in the Youtube video...</p>
<pre class="code bash literal-block">
pipenv<span class="w"> </span>install<span class="w"> </span>requests-html<span class="w"> </span>--python<span class="o">=</span>python3.6
</pre>
<p>Let's reach the body of a Wikipedia random page first:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">requests_html</span> <span class="kn">import</span> <span class="n">HTMLSession</span>
<span class="n">root</span> <span class="o">=</span> <span class="s1">'https://fr.wikipedia.org'</span>
<span class="n">random_page</span> <span class="o">=</span> <span class="s1">'/wiki/Sp%C3%A9cial:Page_au_hasard'</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">HTMLSession</span><span class="p">()</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">root</span> <span class="o">+</span> <span class="n">random_page</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">html</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s2">"body"</span><span class="p">,</span> <span class="n">first</span><span class="o">=</span><span class="kc">True</span><span class="p">))</span>
</pre></div>
<p>Ok, it works and the API seems to be made for humans ;) Let's find out what would be the right CSS selector to pick the first link of the article:</p>
<img alt="Searching first link with Firefox Devtools" src="/images/wikipedia-first-link.png" />
<p>Something like <cite>#mw-content-text a</cite>?</p>
<p>But that sometimes picks the summary section on the right side. Be more hackish and precise:</p>
<div class="highlight"><pre><span></span><span class="p">#</span><span class="nn">mw-content-text</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="p">.</span><span class="nc">mw-parser-output</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="nt">p</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="nt">a</span><span class="o">[</span><span class="nt">href</span><span class="o">^=</span><span class="s1">'/wiki'</span><span class="o">],</span>
<span class="p">#</span><span class="nn">mw-content-text</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="p">.</span><span class="nc">mw-parser-output</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="nt">ul</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="nt">li</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="nt">a</span><span class="o">[</span><span class="nt">href</span><span class="o">^=</span><span class="s1">'/wiki'</span><span class="o">]</span>
</pre></div>
<p>This code will be probably broken by the time you read that article, but I don't care! The joy of microhacks, not the joy of scraping, remember.</p>
<p>Now let's iterate until the link points to the Philosophy page:</p>
<div class="highlight"><pre><span></span><span class="n">SELECTOR</span> <span class="o">=</span> <span class="p">(</span><span class="s2">"#mw-content-text > .mw-parser-output > p > a[href^='/wiki'],"</span>
<span class="s2">"#mw-content-text > .mw-parser-output > ul > li > a[href^='/wiki']"</span><span class="p">)</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">random_page</span>
<span class="k">while</span> <span class="s2">"first link is not philosophy"</span><span class="p">:</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">root</span> <span class="o">+</span> <span class="n">url</span><span class="p">)</span>
<span class="n">link</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">html</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">SELECTOR</span><span class="p">,</span> <span class="n">first</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">link</span><span class="o">.</span><span class="n">attrs</span><span class="p">[</span><span class="s1">'title'</span><span class="p">]</span>
<span class="nb">print</span><span class="p">(</span><span class="n">title</span><span class="p">)</span>
<span class="k">if</span> <span class="n">title</span> <span class="o">==</span> <span class="s1">'Philosophie'</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">link</span><span class="o">.</span><span class="n">attrs</span><span class="p">[</span><span class="s1">'href'</span><span class="p">]</span>
</pre></div>
<p>Cool it works!</p>
<pre class="literal-block">
$ pipenv run python dop.py
Jeu vidéo
Jeu électronique
Jeu
Psychisme
Conscience
Philosophie
</pre>
<p>Now let's repeat that for a hundred random pages, cache already visited links and compute average of degrees.</p>
<div class="highlight"><pre><span></span><span class="n">results</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">cache</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">while</span> <span class="nb">len</span><span class="p">(</span><span class="n">results</span><span class="p">)</span> <span class="o"><</span> <span class="mi">100</span><span class="p">:</span>
<span class="n">degrees</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">random_page</span>
<span class="k">while</span> <span class="s2">"first link is not philosophy"</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">degrees</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">url</span> <span class="ow">in</span> <span class="n">cache</span><span class="p">:</span>
<span class="n">link</span> <span class="o">=</span> <span class="n">cache</span><span class="p">[</span><span class="n">url</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">r</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">root</span> <span class="o">+</span> <span class="n">url</span><span class="p">)</span>
<span class="n">link</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">html</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">SELECTOR</span><span class="p">,</span> <span class="n">first</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">if</span> <span class="n">link</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">break</span> <span class="c1"># bad selector</span>
<span class="n">cache</span><span class="p">[</span><span class="n">url</span><span class="p">]</span> <span class="o">=</span> <span class="n">link</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">link</span><span class="o">.</span><span class="n">attrs</span><span class="p">[</span><span class="s1">'title'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">title</span> <span class="ow">in</span> <span class="n">degrees</span><span class="p">:</span>
<span class="k">break</span> <span class="c1"># loops</span>
<span class="n">degrees</span> <span class="o">+=</span> <span class="p">[</span><span class="n">title</span><span class="p">]</span>
<span class="k">if</span> <span class="n">title</span> <span class="o">==</span> <span class="s1">'Philosophie'</span><span class="p">:</span>
<span class="n">results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">degrees</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">degrees</span><span class="p">),</span> <span class="s2">" - "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">degrees</span><span class="p">))</span>
<span class="k">break</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">link</span><span class="o">.</span><span class="n">attrs</span><span class="p">[</span><span class="s1">'href'</span><span class="p">]</span>
<span class="nb">print</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="nb">sum</span><span class="p">(</span><span class="n">results</span><span class="p">))</span> <span class="o">/</span> <span class="nb">len</span><span class="p">(</span><span class="n">results</span><span class="p">),</span> <span class="nb">min</span><span class="p">(</span><span class="n">results</span><span class="p">),</span> <span class="nb">max</span><span class="p">(</span><span class="n">results</span><span class="p">))</span>
</pre></div>
<p>I can now tell you that if you keep clicking on the <strong>first link</strong> of a random article you'll reach the <em>Philosophy</em> page with 13.45 steps in average. We could also have run a <a class="reference external" href="https://blog.mathieu-leplatre.info/some-python-3-asyncio-snippets.html">few workers in parallel</a>.</p>
<p>Pretty useful huh!?</p>
<p>It was that or hang around in gear shops where I'd never buy anything anyway...</p>
Placeholder for empty lists in CSS2018-01-15T00:00:00+01:002018-01-15T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2018-01-15:/placeholder-for-empty-lists-in-css.html<p>Call me a weirdo but one of my guilty pleasures is to implement minimalist apps with vanilla JavaScript.</p>
<p>Especially now that we can do ES6, async/await and <a class="reference external" href="https://twitter.com/FirefoxNightly/status/951382754125545473">even modules</a>, it's a bliss! No npm, no browserify, no transpilation, just those three files <tt class="docutils literal">index.html</tt>, <tt class="docutils literal">style.css</tt> and <tt class="docutils literal">script.js …</tt></p><p>Call me a weirdo but one of my guilty pleasures is to implement minimalist apps with vanilla JavaScript.</p>
<p>Especially now that we can do ES6, async/await and <a class="reference external" href="https://twitter.com/FirefoxNightly/status/951382754125545473">even modules</a>, it's a bliss! No npm, no browserify, no transpilation, just those three files <tt class="docutils literal">index.html</tt>, <tt class="docutils literal">style.css</tt> and <tt class="docutils literal">script.js</tt>— like we used to do 15 years ago — pure sweetness :)</p>
<p>Today I needed to show a <em>«Empty»</em> placeholder when a <tt class="docutils literal"><ul></tt> list is empty (ie. no <tt class="docutils literal"><li></tt> child). My first attempt consisted in checking if the list had more than one child, more or less like so:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">ul</span><span class="p">></span>
Empty
<span class="p"></</span><span class="nt">ul</span><span class="p">></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">list</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s2">"ul"</span><span class="p">);</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">list</span><span class="p">.</span><span class="nx">childNodes</span><span class="p">.</span><span class="nx">length</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mf">1</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">list</span><span class="p">.</span><span class="nx">textContent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">""</span><span class="p">;</span><span class="w"> </span><span class="c1">// Remove placeholder.</span>
<span class="p">}</span>
<span class="nx">list</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">item</span><span class="p">);</span>
</pre></div>
<p>That's disturbing, bad and ugly. Mainly because the <em>Empty</em> text node is also a child node, but especially because of this <tt class="docutils literal">if</tt> statement in the application code. Like <a class="reference external" href="https://youtu.be/qrYt4bbEUrU?t=15m33s">Linus said at TED</a>, «[…] good code is when the special case goes away and becomes the normal case» :)</p>
<p>Like it or not, but CSS can really help to keep the HTML and JS code to bare minimum. After a little research, I discovered that the <tt class="docutils literal">:empty</tt> selector works on any tag! And look, the special case goes away!</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">ul</span><span class="p">></</span><span class="nt">ul</span><span class="p">></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="nt">ul</span><span class="p">:</span><span class="nd">empty</span><span class="p">::</span><span class="nd">after</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">content</span><span class="p">:</span><span class="w"> </span><span class="s2">"Empty"</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="kd">const</span><span class="w"> </span><span class="nx">list</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s2">"ul"</span><span class="p">);</span>
<span class="nx">list</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">item</span><span class="p">);</span>
</pre></div>
<p>Yes, that's better!</p>
<p>In order to keep the CSS a little more decoupled from content, we can even do:</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">ul</span> <span class="na">placeholder</span><span class="o">=</span><span class="s">"Empty"</span><span class="p">></</span><span class="nt">ul</span><span class="p">></span>
</pre></div>
<div class="highlight"><pre><span></span><span class="nt">ul</span><span class="p">:</span><span class="nd">empty</span><span class="p">::</span><span class="nd">after</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">content</span><span class="p">:</span><span class="w"> </span><span class="nb">attr</span><span class="p">(</span><span class="n">placeholder</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>Some folks achieve amazing stuff using simple but powerful CSS rules! I often use <a class="reference external" href="https://css-tricks.com/functional-css-tabs-revisited/">these tabs</a> for example :)</p>
The Joy of Microhacks2017-10-18T00:00:00+02:002017-10-18T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2017-10-18:/the-joy-of-microhacks.html<p>Last September, I had the chance to attend the awesome <a class="reference external" href="http://2017.fullstackfest.com/">Fullstack Fest conference</a>, that takes place just accross my neighbourhood.</p>
<p>Like the year before, the content quality was very high. The last talk really resonated in me, and I felt like discoursing on its topic!</p>
<div class="section" id="making-a-lot-of-things-ben-foxall">
<h2>Making a lot of things …</h2></div><p>Last September, I had the chance to attend the awesome <a class="reference external" href="http://2017.fullstackfest.com/">Fullstack Fest conference</a>, that takes place just accross my neighbourhood.</p>
<p>Like the year before, the content quality was very high. The last talk really resonated in me, and I felt like discoursing on its topic!</p>
<div class="section" id="making-a-lot-of-things-ben-foxall">
<h2>Making a lot of things — Ben Foxall</h2>
<p>When Ben appeared on stage, I realized we already met :) It took me a while to recall that he organized the button hack contest in 2016 — in which <a class="reference external" href="https://twitter.com/pusher/status/774256560554016768">I won a funny BB8 robot</a> with an hour of coding in JS :)</p>
<p>In his conference, Ben presents some ideas about explorative coding, where you don't plan anything long term. By getting rid of planning, you keep your expectations low, it is easier to get started, and most importantly, you can focus on the fun part: hacking.</p>
<div class="align-center">
<iframe width="560" height="315" src="https://www.youtube.com/embed/Pve8JoaTNqE" frameborder="0" allowfullscreen></iframe>
</div></div>
<div class="section" id="my-relation-with-software-ideas">
<h2>My relation with software ideas</h2>
<p>As I <a class="reference external" href="https://blog.mathieu-leplatre.info/releasing-software-ideas.html">wrote 3 years ago</a>, software ideas that constantly pop up in my mind become a kind of curse. I can't do everything, and I have trouble to admit it and let it go.</p>
<p>At the time of that article, I had decided to focus on 2 projects. They both became very successful, since one is now <a class="reference external" href="http://subtivals.org">used worldwide</a> by cinemas, theaters, cultural centers, schools, tv channels, production companies etc., and the other one <a class="reference external" href="https://blog.mathieu-leplatre.info/a-year-at-mozilla.html">brought me to work with awesome people</a>.</p>
<p>But if I look back, I realize that having invested so much personal time to grow those two side projects also had terrible consequences — that I won't detail here.</p>
<p>I am too easily absorbed by software ideas and plans. I quickly feel accountable for things that should remain hobbies. Consequently, I sometimes end up neglecting my own needs.</p>
<div class="figure align-center">
<img alt="" src="/images/microhacks-smeagol.jpg" style="width: 70%;" />
<p class="caption">Me in front of my latest project idea :)</p>
</div>
<p>Anyway, I now have a lot more free time that I had 3 years ago. Still, I am truely certain that I don't want to have side projects anymore. Nor do I want to maintain open source libraries as an individual. And I also shut down the mutualized server I was running for self-hosting email, friends websites and stuff a while ago.</p>
<p>Finding the right balance, health, happiness, peace and harmony is hard enough! Especially with passion for your job.</p>
<p>It sounds obvious, but if I open my laptop at night or on a rainy Sunday, <strong>I wish it would be just fun</strong>. Fun only: «otium», not «negotium» <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p>
</div>
<div class="section" id="micro-hacks-to-the-rescue">
<h2>Micro hacks to the rescue</h2>
<p>As the legends state, great ideas come up when you are in your bath, under an apple tree, or drawing on napkins in a bar. Basically when you are free from «obligations» :)</p>
<p>In my case, there are many tasks that are natural and painless at work, but that tend to become burden and clear away the fun at home as a hobby. Most notably triaging bug reports, hosting services, monitoring, designing user experience, defining road maps, fighting feature bloat, packaging and describing installation procedures, writing documentation, providing user support, having tests, supporting cross browser/platform, etc.</p>
<p>Also, like Ben stated in his conf, starting and finding time is hard. From a simple concept, I easily get lost in plans and features ideas, while riding my bike or under the shower. Then, I tell myself «<em>this project will absorb all my evenings for the next 6 months, ...naaa careful, forget it...</em>».</p>
<div class="figure align-center">
<img alt="" src="/images/microhacks-lifetime.png" />
<p class="caption">Lifetime of a microhack (right)</p>
</div>
<p>The main idea of micro-hacks is that you restrict the idea to the bare minimum, in order to <strong>only keep the fun part</strong>: implementing a software idea. Ideally in a couple of hours!</p>
<p>You set a tiny goal, you code, you enjoy the result. Done.</p>
<p>Basically, you get rid of the boring/not fun parts that consume your free time and keep you away from having new ideas :)</p>
</div>
<div class="section" id="kill-it">
<h2>Kill it!</h2>
<p>When an experiment or its result is fun, we want to share it.</p>
<p>At that point, the first reflex is to put it on Github. And that's when it is very tempting to make it look good, add a nice README, explain its goal, let the bug tracker open, tweet the link, and why not mentioning it in your resume after all!</p>
<p>But how do you feel if someone critizes the unoptimized way you handle mouse events? Or notices the lack of coding standards? Or if the code crashes with Firefox Nightly? And if someone wants to try it but can't install it on CentOS 4? Or if someone suggests that with Google authentication this project could be a life saver? What if your tiny piece of software ends up on Hackernews? What if (again) you'll think of turning it into a startup?</p>
<p>Micro hacks, we said: Code. Enjoy. <strong>Done</strong>. Live in peace.</p>
<div class="figure align-center">
<img alt="" src="/images/microhacks-theblob.jpg" style="width: 70%;" />
</div>
</div>
<div class="section" id="immortalize-your-hack">
<h2>Immortalize your hack</h2>
<p>Sometimes the hack works in some specific conditions or setup (some connected Arduino, trial account on some API, a process in a <tt class="docutils literal">screen</tt> on your friend's server...). Also sometimes, it only works on some specific software version, some local fork, or relies on some symlinks hacks deep in your devbox OS. And that's perfectly fine!</p>
<p>Because it will require too much efforts to reproduce the same setup in a few days or weeks, and because you want to capture that moment where it just worked, just <strong>screenshot or screencast</strong> whatever you obtained!</p>
<p>That is usually the most efficient way to share the fun! Plus, it will still be readable in 5 years if you mentioned it in your resume or among the side-projects on your website.</p>
<div class="figure align-center">
<img alt="kinto-telegram-wall micro hack" src="/images/microhacks-kinto-telegram.jpg" />
<p class="caption"><a class="reference external" href="https://github.com/leplatrem/kinto-telegram-wall">A microhack</a> involving a Telegram bot with live updates on a Web page using Kinto.</p>
</div>
</div>
<div class="section" id="absence-without-leave">
<h2>Absence without leave</h2>
<p>Artists who share stuff on <a class="reference external" href="https://dribbble.com/">Dribbble</a> won't give you the source files. But since the hacker culture is about sharing, and since the fun was about coding, publishing the source code online still makes sense of course.</p>
<p>But in order to get rid of any kind of sense of accountability, the README could only contain:</p>
<ul class="simple">
<li>the capture</li>
<li>a disclaimer («<em>I won't maintain this</em>»)</li>
<li>a public domain / CC0 licence</li>
</ul>
<p>That way, you can safely share the link around, for those who wonder how it was done.</p>
<p>Also, using source control gives you the ability to go back in time. It can be very rewarding when expanding around an idea or learning a new technology — like <a class="reference external" href="https://youtu.be/Pve8JoaTNqE?t=21m36s">Ben and his WebGL hacks</a> — because we tend to forget how it used to look in the first place or how cool it felt when it first worked :)</p>
<div class="figure align-center">
<img alt="disclaimer example" src="/images/microhacks-disclaimer.png" />
<p class="caption">Example of a screencast along with <a class="reference external" href="https://github.com/almet/web2mp3">a disclaimer by @almet</a></p>
</div>
</div>
<div class="section" id="self-promotion">
<h2>Self promotion?</h2>
<p>It can be hard to be proud of our achievements. Probably because there is always something more impressive elsewhere. Or maybe because after too much time spent on something it is not so impressive anymore.</p>
<p>But I think it is important to show other developers — especially beginners — that everyone can be happy with tiny achievements.</p>
<p>Sharing the coolness of bringing an idea into life is always interesting to someone. Adding some quick description of how unexpected issues were overcome makes it even more pleasant. As an example, the most popular article of this blog is a 2011 post about reverting Git commits (possibly one of the shortest ones too).</p>
<p>Besides, when applying for a job, I tend to think that someone writing about learning and overcoming obstacles has a lot more value than some expert maintaining a very popular open source project during weekends, even if it takes 1000x less time and efforts.</p>
<div class="align-center">
<video src="/images/subtivals-remote.webm" width="500" controls>
<p>Your browser does not support the video element </p>
</video>
<p class="caption">A microhack in Subtivals to project subtitles on remote devices using Websockets.</p>
</div><table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>See also (in French): <a class="reference external" href="http://www.brehault.net/textes/otium-et-negotium-dans-lindustrie-du-logiciel/">Otium et negotium dans l'industrie du logiciel</a></td></tr>
</tbody>
</table>
</div>
Some Python 3 asyncio snippets2017-08-31T00:00:00+02:002017-08-31T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2017-08-31:/some-python-3-asyncio-snippets.html<p>Until recently, I had never taken the chance to get my hands dirty with <a class="reference external" href="https://docs.python.org/3/library/asyncio.html#module-asyncio">asyncio</a>. But now that our production stacks run Python 3.6, there is no false excuse.</p>
<p>I had done a lot of JavaScript before — with promises and <tt class="docutils literal">async</tt>/<tt class="docutils literal">await</tt> syntax — but still, diving into the <a class="reference external" href="http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/">many …</a></p><p>Until recently, I had never taken the chance to get my hands dirty with <a class="reference external" href="https://docs.python.org/3/library/asyncio.html#module-asyncio">asyncio</a>. But now that our production stacks run Python 3.6, there is no false excuse.</p>
<p>I had done a lot of JavaScript before — with promises and <tt class="docutils literal">async</tt>/<tt class="docutils literal">await</tt> syntax — but still, diving into the <a class="reference external" href="http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/">many primitives of asyncio</a> was far from immediate. Task versus future? Wrapped coroutine? Concurrent futures?</p>
<p>This article gathers some notes and snippets I wish I had up my sleeve before starting.</p>
<p>If you see something that bugs you, please use the comments!</p>
<img alt="" class="align-center" src="/images/async_juggling.jpg" />
<div class="section" id="about-the-concepts">
<h2>About the concepts</h2>
<p>A <strong>coroutine</strong> is basically an async function. But hey, it is also used to describe the object returned an by async function when you don't <tt class="docutils literal">await</tt> it.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">async</span> <span class="k">def</span> <span class="nf">poll</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="o">...</span> <span class="k">return</span> <span class="n">n</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">poll</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="o"><</span><span class="n">coroutine</span> <span class="nb">object</span> <span class="n">poll</span> <span class="n">at</span> <span class="mh">0x7f0e37cfbe08</span><span class="o">></span>
</pre></div>
<p>Unlike promises in JavaScript, calling a coroutine does not start its code running. You have to schedule it manually, or simply <tt class="docutils literal">await</tt> it.</p>
<p>You can only use the <tt class="docutils literal">await</tt> keyword from an async function. And there will always be a blocking call to <tt class="docutils literal">loop.run_until_complete()</tt> somewhere in the code.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="o">...</span> <span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">poll</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="o">...</span> <span class="nb">print</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="n">main</span><span class="p">())</span>
<span class="mi">1</span>
</pre></div>
<p>A <strong>future</strong> is basically like a JavaScript promise — a placeholder for an initially unknown value. A <strong>task</strong> wraps a coroutine and its execution is scheduled in the event loop, and provides the future's interface. But you never instanciate it yourself. The API and documentation about futures and tasks are sometimes confusing about their distinction, so I just considered them synonyms so far.</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">asyncio</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">poll</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="o"><</span><span class="n">Task</span> <span class="n">pending</span> <span class="n">coro</span><span class="o">=<</span><span class="n">poll</span><span class="p">()</span> <span class="n">running</span> <span class="n">at</span> <span class="o"><</span><span class="n">stdin</span><span class="o">></span><span class="p">:</span><span class="mi">1</span><span class="o">>></span>
</pre></div>
<p>The future/task object can be passed around. It has a state, a result and a potential exception. And you can await it:</p>
<div class="highlight"><pre><span></span><span class="n">future</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">poll</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="n">result</span> <span class="o">=</span> <span class="k">await</span> <span class="n">future</span>
</pre></div>
</div>
<div class="section" id="run-coroutines-in-parallel">
<h2>Run coroutines in parallel</h2>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">long_task</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
<span class="n">inputs</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"aa"</span><span class="p">]</span>
<span class="n">futures</span> <span class="o">=</span> <span class="p">[</span><span class="n">long_task</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">inputs</span><span class="p">]</span>
<span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">futures</span><span class="p">)</span>
<span class="k">for</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">results</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
<p>As a human™ I would have excepted the <tt class="docutils literal">asyncio.wait()</tt> function to run futures in parallel and return their results, but it doesn't exactly do that. It returns two lists of futures and you have to unwrap its value with <tt class="docutils literal">result()</tt>. And careful, the signature is not the same (a list of futures versus futures in <tt class="docutils literal">*args</tt>).</p>
<div class="highlight"><pre><span></span><span class="n">futures</span> <span class="o">=</span> <span class="p">[</span><span class="n">long_task</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">inputs</span><span class="p">]</span>
<span class="n">done</span><span class="p">,</span> <span class="n">pending</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span><span class="n">futures</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">[</span><span class="n">future</span><span class="o">.</span><span class="n">result</span><span class="p">()</span> <span class="k">for</span> <span class="n">future</span> <span class="ow">in</span> <span class="n">done</span><span class="p">]</span>
</pre></div>
</div>
<div class="section" id="run-blocking-code-in-parallel">
<h2>Run blocking code in parallel</h2>
<p>Blocking code can be executed accross a pool of threads or processes using <a class="reference external" href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor">executors</a>.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">concurrent.futures</span>
<span class="k">def</span> <span class="nf">long_task</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
<span class="n">executor</span> <span class="o">=</span> <span class="n">concurrent</span><span class="o">.</span><span class="n">futures</span><span class="o">.</span><span class="n">ThreadPoolExecutor</span><span class="p">(</span><span class="n">max_workers</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
<span class="n">inputs</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"aa"</span><span class="p">]</span>
<span class="n">futures</span> <span class="o">=</span> <span class="p">[</span><span class="n">loop</span><span class="o">.</span><span class="n">run_in_executor</span><span class="p">(</span><span class="n">executor</span><span class="p">,</span> <span class="n">long_task</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">inputs</span><span class="p">]</span>
<span class="n">results</span> <span class="o">=</span> <span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">futures</span><span class="p">)</span>
<span class="k">for</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">results</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="asynchronous-stream-from-file-like-objects">
<h2>Asynchronous stream from file-like objects</h2>
<p>Reading from a file or standard input like <tt class="docutils literal">sys.stdin</tt> is blocking. In order to treat them as asynchronuous streams of data, we leverage <tt class="docutils literal">asyncio.StreamReader()</tt> and expose them as <a class="reference external" href="https://www.python.org/dev/peps/pep-0525/">async generators</a>:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">stream_as_generator</span><span class="p">(</span><span class="n">loop</span><span class="p">,</span> <span class="n">stream</span><span class="p">):</span>
<span class="n">reader</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">StreamReader</span><span class="p">(</span><span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">)</span>
<span class="n">reader_protocol</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">StreamReaderProtocol</span><span class="p">(</span><span class="n">reader</span><span class="p">)</span>
<span class="k">await</span> <span class="n">loop</span><span class="o">.</span><span class="n">connect_read_pipe</span><span class="p">(</span><span class="k">lambda</span><span class="p">:</span> <span class="n">reader_protocol</span><span class="p">,</span> <span class="n">stream</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">line</span> <span class="o">=</span> <span class="k">await</span> <span class="n">reader</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line</span><span class="p">:</span> <span class="c1"># EOF.</span>
<span class="k">break</span>
<span class="k">yield</span> <span class="n">line</span>
</pre></div>
<p>The generator is awaited with an <tt class="docutils literal">async for</tt>:</p>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">stream_as_generator</span><span class="p">(</span><span class="n">loop</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="process-data-stream-by-chunk-asynchronously">
<h2>Process data stream by chunk asynchronously</h2>
<div class="highlight"><pre><span></span><span class="k">async</span> <span class="n">parse_urls</span><span class="p">():</span>
<span class="k">async</span> <span class="k">for</span> <span class="n">u</span> <span class="ow">in</span> <span class="n">read_stuff</span><span class="p">():</span>
<span class="k">yield</span> <span class="n">u</span>
<span class="k">async</span> <span class="n">download</span><span class="p">(</span><span class="n">urls</span><span class="p">):</span>
<span class="k">async</span> <span class="k">for</span> <span class="n">response</span> <span class="ow">in</span> <span class="n">download</span><span class="p">(</span><span class="n">url</span><span class="p">):</span>
<span class="k">while</span> <span class="s2">"chunks to read"</span><span class="p">:</span>
<span class="n">chunk</span> <span class="o">=</span> <span class="k">await</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">1024</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">chunk</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">yield</span> <span class="n">chunk</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">split_lines</span><span class="p">(</span><span class="n">stream</span><span class="p">):</span>
<span class="n">leftover</span> <span class="o">=</span> <span class="s1">''</span>
<span class="k">async</span> <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span>
<span class="n">chunk_str</span> <span class="o">=</span> <span class="n">leftover</span> <span class="o">+</span> <span class="n">chunk_str</span>
<span class="n">chunk_str</span> <span class="o">=</span> <span class="n">chunk_str</span><span class="o">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
<span class="n">leftover</span> <span class="o">=</span> <span class="n">lines</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
<span class="k">if</span> <span class="n">lines</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">lines</span>
<span class="n">urls_generator</span> <span class="o">=</span> <span class="n">parse_urls</span><span class="p">()</span>
<span class="n">data_generator</span> <span class="o">=</span> <span class="n">download</span><span class="p">(</span><span class="n">urls_generator</span><span class="p">)</span>
<span class="k">async</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">split_lines</span><span class="p">(</span><span class="n">data_generator</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="mock-aiohttp-responses">
<h2>Mock aiohttp responses</h2>
<p>Suppose the following sample code using <a class="reference external" href="http://aiohttp.readthedocs.io/">aiohttp</a>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">aiohttp</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">get_username</span><span class="p">(</span><span class="n">loop</span><span class="p">):</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">aiohttp</span><span class="o">.</span><span class="n">ClientSession</span><span class="p">(</span><span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">)</span> <span class="k">as</span> <span class="n">session</span><span class="p">:</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">SERVER_URL</span><span class="si">}</span><span class="s2">/profile"</span><span class="p">)</span> <span class="k">as</span> <span class="n">response</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="k">await</span> <span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
<span class="k">return</span> <span class="n">data</span><span class="p">[</span><span class="s2">"user"</span><span class="p">]</span>
</pre></div>
<p>We can test it using the amazing <a class="reference external" href="https://asynctest.readthedocs.io/">asynctest</a> and <a class="reference external" href="https://github.com/pnuckowski/aioresponses/">aioresponses</a> libraries:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">asynctest</span>
<span class="kn">from</span> <span class="nn">aioresponses</span> <span class="kn">import</span> <span class="n">aioresponses</span>
<span class="k">class</span> <span class="nc">Test</span><span class="p">(</span><span class="n">asynctest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="n">remote_content</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"/profile"</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">"user"</span><span class="p">:</span> <span class="s2">"Ada"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">mocked</span> <span class="o">=</span> <span class="n">aioresponses</span><span class="p">()</span>
<span class="n">mocked</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">for</span> <span class="n">url</span><span class="p">,</span> <span class="n">payload</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">remote_content</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">mocked</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">SERVER_URL</span> <span class="o">+</span> <span class="n">url</span><span class="p">,</span> <span class="n">payload</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">addCleanup</span><span class="p">(</span><span class="n">mocked</span><span class="o">.</span><span class="n">stop</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">test_get_username</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">u</span> <span class="o">=</span> <span class="k">await</span> <span class="n">get_username</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">u</span> <span class="o">==</span> <span class="s2">"Ada"</span>
</pre></div>
</div>
<div class="section" id="consume-queue-in-batches">
<h2>Consume queue in batches</h2>
<p>A producer feeds items into a queue, and consumers builds batches from them. When it takes too much time to fill a batch, it proceeds with the current one.</p>
<p>By marking the tasks as done in the queue, we can await the queue and know when everything is processed.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">async_timeout</span>
<span class="k">def</span> <span class="nf">markdone</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="n">n</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Returns a callback that will mark `n` queue items done."""</span>
<span class="k">def</span> <span class="nf">done</span><span class="p">(</span><span class="n">task</span><span class="p">):</span>
<span class="p">[</span><span class="n">queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">)]</span>
<span class="k">return</span> <span class="n">task</span><span class="o">.</span><span class="n">result</span><span class="p">()</span> <span class="c1"># will raise exception if failed.</span>
<span class="k">return</span> <span class="n">done</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">consume</span><span class="p">(</span><span class="n">loop</span><span class="p">,</span> <span class="n">queue</span><span class="p">,</span> <span class="n">executor</span><span class="p">):</span>
<span class="k">while</span> <span class="s1">'consumer is not cancelled'</span><span class="p">:</span>
<span class="n">batch</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="n">async_timeout</span><span class="o">.</span><span class="n">timeout</span><span class="p">(</span><span class="n">WAIT_TIMEOUT</span><span class="p">):</span>
<span class="k">while</span> <span class="nb">len</span><span class="p">(</span><span class="n">batch</span><span class="p">)</span> <span class="o"><</span> <span class="n">BATCH_SIZE</span><span class="p">:</span>
<span class="c1"># Wait for new items.</span>
<span class="n">item</span> <span class="o">=</span> <span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="n">batch</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">record</span><span class="p">)</span>
<span class="k">except</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">TimeoutError</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">if</span> <span class="n">batch</span><span class="p">:</span>
<span class="n">task</span> <span class="o">=</span> <span class="n">loop</span><span class="o">.</span><span class="n">run_in_executor</span><span class="p">(</span><span class="n">executor</span><span class="p">,</span> <span class="n">long_sync_task</span><span class="p">,</span> <span class="n">batch</span><span class="p">)</span>
<span class="n">task</span><span class="o">.</span><span class="n">add_done_callback</span><span class="p">(</span><span class="n">markdone</span><span class="p">(</span><span class="n">queue</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">batch</span><span class="p">)))</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">loop</span><span class="p">):</span>
<span class="n">executor</span> <span class="o">=</span> <span class="n">concurrent</span><span class="o">.</span><span class="n">futures</span><span class="o">.</span><span class="n">ThreadPoolExecutor</span><span class="p">(</span><span class="n">max_workers</span><span class="o">=</span><span class="n">NB_THREADS</span><span class="p">)</span>
<span class="n">queue</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
<span class="c1"># Schedule the consumer</span>
<span class="n">consumer_coro</span> <span class="o">=</span> <span class="n">consume</span><span class="p">(</span><span class="n">loop</span><span class="p">,</span> <span class="n">queue</span><span class="p">,</span> <span class="n">executor</span><span class="p">)</span>
<span class="n">consumer</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">consumer_coro</span><span class="p">)</span>
<span class="c1"># Run the producer and wait for completion</span>
<span class="k">await</span> <span class="n">produce</span><span class="p">(</span><span class="n">loop</span><span class="p">,</span> <span class="n">queue</span><span class="p">)</span>
<span class="c1"># Wait until the consumer is done consuming everything.</span>
<span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="c1"># The consumer is still awaiting for the producer, cancel it.</span>
<span class="n">consumer</span><span class="o">.</span><span class="n">cancel</span><span class="p">()</span>
</pre></div>
</div>
FullstackFest 20162016-09-16T00:00:00+02:002016-09-16T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2016-09-16:/fullstackfest-2016<p class="first last">Summary and feedback from Fullstack Fest 2016 in Barcelona</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">This article was originally posted on <a class="reference external" href="https://mozilla-services.github.io/servicedenuages.fr/en/fullstackfest-2016">servicedenuages.fr</a></p>
</div>
<p>I had the chance to go the Fullstack Fest 2016 and really enjoyed it — thanks Mozilla!</p>
<p>This blog post is an attempt to give you a summary and some feedback of what happened during this 4+1 days conference!</p>
<div class="section" id="huge-applause">
<h2>Huge applause!</h2>
<p>The <a class="reference external" href="http://www.codegram.com">Codegram team</a> did really really well! They raised the bar for tech conferences; all the feedback I heard was extremely positive! Plus, Barcelona is a pleasant destination :)</p>
<p>Former <em>RubyCon</em> and <em>FutureJS</em> events were merged into one <em>fullstack</em> event, 2 days of backend talks, 1 gap day with social activites, and 2 days of frontend talks.</p>
<blockquote>
Problems of Today, ideas from the future. — FullstackFest motto</blockquote>
<p>I must congratulate the organizers for the choice of speakers and topics. It's a huge challenge to captivate 500 persons during 5 days, and they did really well! And also a big up for the impressive performance of the captioners!</p>
<img alt="CC-BY @fullstackfest" class="align-center" src="https://blog.mathieu-leplatre.info/images/fullstackfest-venue.jpg" />
</div>
<div class="section" id="trends">
<h2>Trends</h2>
<ul class="simple">
<li>Docker</li>
<li>Amazon Lambda</li>
<li>GraphQL</li>
<li>Criticism of microservices</li>
<li>Offline / high latency / decentralized Web</li>
<li>Functional programming</li>
<li>Reactive programming</li>
<li>Immutability</li>
<li>Machine learning</li>
<li>Virtual reality</li>
<li>Physical Web</li>
</ul>
<img alt="CC-BY @fullstackfest" class="align-center" src="https://blog.mathieu-leplatre.info/images/fullstackfest-trends.jpg" />
</div>
<div class="section" id="one-sentence-summaries">
<h2>One sentence summaries</h2>
<p><em>The videos are already online! In order to help you choose which one to watch, here is the shortest summary I could come up with for each talk, videos are linked on authors names:</em></p>
<ul class="simple">
<li>Hardware is amazing, software is crap. Use parallelism and keep your data under your control — <a class="reference external" href="https://www.youtube.com/watch?v=itKFrXghGuA&index=2&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Joe Armstrong</a>, Erlang's inventor.</li>
<li>Scale using «pods»: deploy several isolated stacks of your multitenant app — <a class="reference external" href="https://www.youtube.com/watch?v=7UyDK2bDjc4&index=3&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Simon Eskildsen</a>, Shopify</li>
<li>Theorical knowledge can help when designing distributed systems with high availability — <a class="reference external" href="https://www.youtube.com/watch?v=bUlpp8_Mevk&index=4&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Ines Sombra</a>, Fastly</li>
<li>Improve efficiency and security using a unikernel: a kernel whose components are built specifically for your app — <a class="reference external" href="https://www.youtube.com/watch?v=XNu2lze6jS0&index=5&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Amir Chaudhry</a>, Docker Inc</li>
<li>Instead of making X faster, do less of X — <a class="reference external" href="https://www.youtube.com/watch?v=b9H9AtbxpPM&index=6&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Tobias Pfeiffer</a>, Bitcrowd</li>
<li>Docker containers are only as secure as your best practices — <a class="reference external" href="https://www.youtube.com/watch?v=oANurUSaOFs&index=7&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Ben Hall</a>, <a class="reference external" href="http://katacoda.com">http://katacoda.com</a></li>
<li>Elixir provides high level abstractions that make Erlang awesome — <a class="reference external" href="https://www.youtube.com/watch?v=Ba3aCm3A0o8&index=8&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Saša Jurić</a>, Aircloak</li>
<li>Unison is an experiment to come up with a new programming paradigm focused on distributed computing and persistence — <a class="reference external" href="https://www.youtube.com/watch?v=f6yA3t0dO-k&index=10&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Paul Chiusano</a>, Stage N</li>
<li>HTTP2 reduces latency. It's here, use it with a one liner in your server configuration — <a class="reference external" href="https://www.youtube.com/watch?v=CThgMRXS8w8&index=11&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Ole Michaelis</a>, Jimdo</li>
<li>Scale by publishing pre-rendered content to S3 when your data source changes — <a class="reference external" href="https://www.youtube.com/watch?v=vUCr1oTtaKA&index=12&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Tim Perry</a>, BBC</li>
<li>Use Amazon lambda as a very cheap and ultra-efficient scaling solution — <a class="reference external" href="https://www.youtube.com/watch?v=9IrFIobZUEA&index=13&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Austen Collins</a>, serverless.com</li>
<li>Processing natural language can be very easy with simple tricks — <a class="reference external" href="https://www.youtube.com/watch?v=vRb-El8hC-U&index=14&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Duretti Hirpa</a>, Slack</li>
<li>It is our responsability to decentralize access and assure durability of our data. Internet of hash = bittorrent + git — <a class="reference external" href="https://www.youtube.com/watch?v=jONZtXMu03w&index=15&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Juan Benet</a>, ipfs.io</li>
<li>Enable GraphQL on your API and rely on smart clients to do the hard work of querying and caching — <a class="reference external" href="https://www.youtube.com/watch?v=eD7kLFGOgVw&index=16&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Marc-Andre Giroux</a>, Shopify</li>
<li>Some insights about recommendation engines and machine learning techniques — <a class="reference external" href="https://www.youtube.com/watch?v=SRnM_P_ygqI&index=17&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Brian Sam-Bodden</a>, Integrallis</li>
</ul>
<img alt="CC-BY @fullstackfest" class="align-center" src="https://blog.mathieu-leplatre.info/images/fullstackfest-talks.jpg" />
<ul class="simple">
<li>Immutable UIs make it simpler to improve durability, utility and beauty in your software architecture — <a class="reference external" href="https://www.youtube.com/watch?v=pLvrZPSzHxo&index=18&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Lee Byron</a>, Facebook, author of Immutable.js</li>
<li>Elm applications have no runtime errors — <a class="reference external" href="https://www.youtube.com/watch?v=rDQ22Yg3Fms&index=19&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Jack Franklin</a>, Songkick</li>
<li>With new versions of Ecmascript, asynchronuous programs will look more and more synchronuous — <a class="reference external" href="https://www.youtube.com/watch?v=3pKNRgResq0&index=20&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Jafar Husain</a>, Netflix, author of Falcor</li>
<li>PostCSS and CSS modules are really helpful for building a libray of UI components — <a class="reference external" href="https://www.youtube.com/watch?v=j8eBXGPl_5E&index=21&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">David Wells</a>, Serverless</li>
<li>At last we have proper grids with CSS4 — <a class="reference external" href="https://www.youtube.com/watch?v=axVw1Zduqn0&index=22&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Jen Kramer</a>, Hardvard University</li>
<li>Evolution of Web apps which now have views, business logic and persistence — <a class="reference external" href="https://www.youtube.com/watch?v=wtURpqTgtUs&index=23&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Luca Marchesini</a>, Kuzzle</li>
<li>Om Next leverages Clojure to bring conventions for data immutability, data specifications and data routing — <a class="reference external" href="https://www.youtube.com/watch?v=Zb18iPjDgwM&index=24&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">António Nuno Monteiro</a>, Technische Universität Dresden</li>
<li>The OpenBCI project allows everyone to study brainwaves using a homemade helmet — <a class="reference external" href="https://www.youtube.com/watch?v=CSfUr3m0-w8&index=25&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Alex Castillo</a>, Netflix</li>
<li>Flow based programming via CycleJS allows you to design and debug your apps via a high level of abstraction — <a class="reference external" href="https://www.youtube.com/watch?v=R-GzJgEccEQ&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc&index=27">André Staltz</a>, Futurice</li>
<li>CSP channels in Javascript is the best tool for dealing with streams of data — <a class="reference external" href="https://www.youtube.com/watch?v=r7yWWxdP_nc&index=28&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Vincenzo Chianese</a>, Apiary</li>
<li>With Service workers we won't have to wait for the server response when submitting data — <a class="reference external" href="https://www.youtube.com/watch?v=xs_QRqGZ8xQ&index=29&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Andrew Dunkman</a>, Harvest</li>
<li>Build your apps as if there were be accessed by Martians via a high-latency connection — <a class="reference external" href="https://www.youtube.com/watch?v=7rlEidtXlZg&index=30&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Slobodan Stojanovic</a>, Cloud Horizon</li>
<li>WebAssembly allows native apps to be run by the browser VM in the same stack context as JavaScript — <a class="reference external" href="https://www.youtube.com/watch?v=vmzz17JGPHI&index=31&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Ben Smith</a>, Google</li>
<li>Render a 2D UI with React within a 3D WebGL scene — <a class="reference external" href="https://www.youtube.com/watch?v=DfPPlakRvow&index=32&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Massimiliano Mantione</a>, HyperFair</li>
<li>Virtual reality in the browser is here! — <a class="reference external" href="https://www.youtube.com/watch?v=Ciqucr_Ww9s&index=33&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Liv Erickson</a>, Microsoft</li>
<li>The physical Web is built on top of the open Web and will change our lives — <a class="reference external" href="https://www.youtube.com/watch?v=gV72mCdomo4&index=34&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">Scott Jenson</a>, Google</li>
</ul>
</div>
<div class="section" id="some-thoughts">
<h2>Some thoughts</h2>
<p>The challenges of Web content durability and decentralization are very concerning. The Web of today has too many points of failure, Web engineers are responsible for making it better. For example, 500 attendees cannot collaborate on a document without going through servers outside the room! My great-great-grandchildren might not be able to find a picture of me on the Web! <a class="reference external" href="https://ipfs.io/#why">But read this page</a> and you'll realize this is not <em>only</em> about the future!</p>
<blockquote>
You are cyberwizards! — Juan Benet</blockquote>
<p>Both frontend and backend talks seem to agree on the fact that REST has strong limitations for high latency connections like on the mobile Web. GraphQL and Falcor are good candidates today, and can be complementary to our usual REST/CRUD endpoints.</p>
<p>Google and Microsoft are following Amazon and its <a class="reference external" href="https://aws.amazon.com/lambda/details/">Lambda</a> to provide a platform of nanoservices for which you don't pay when idle, and that scale on trafic peaks. A whole app can be decomposed into functions and designed using internal events. Orchestration of those tiny pieces of code is another story (<em>for a CRUD endpoint, each verb is bound to a function</em>), but <a class="reference external" href="http://blog.serverless.com/defining-serverless/">Serverless</a> wants to tackle that.</p>
<img alt="CC-BY @fullstackfest" class="align-center" src="https://blog.mathieu-leplatre.info/images/fullstackfest-vr.jpg" />
<p>Functional programming is in the air. Pure (stateless) functions scale because they can be executed anywere without context. Without side effects, it also becomes obvious to test them. They give backend developers some good pattern for highly available services, and provide the necessary constraints for immutability to frontend developers. I believe that if functional programming is not adopted as much as it could, it is not because of the paradigm and concepts, but because of the disconcerting syntax!</p>
<blockquote>
There is no architecture nirvana! — Lee Byron</blockquote>
<p>Angular/TypeScript was almost absent from the conference, but we could see a trend around type inference/checking. It really improves developer experience and reduces runtime errors in the end. With projects like Elm, developers feel safe while coding: the compiler is smart and polite, refactoring is fun! The concept of <tt class="docutils literal">null</tt> is replaced with a notion of <em>maybe <type></em> which makes apps robust.</p>
<p>Everybody seemed to agree that JavaScript transpilation will last for a long time. The language will evolve, and we shouldn't hope for feature freeze. Plus, it really helps developers focus on modern stacks, leveraging modern features like <a class="reference external" href="https://github.com/tc39/proposal-cancelable-promises">cancelable promises</a>. And nowadays it is rather easy to build your app for legacy versions of JavaScript like ES5. But hey, it is a bit absurd that modern browsers have to download big bundles of code full of polyfills! It would be a lot nicer if they were delivered to legacy browsers only — <cite><!--[if IE 10]></cite> oh yeah :) — or even better: shipped as something like a browser addon :)</p>
<p>The old times where we could inspect the source code of a page — and learn from it — is far! It is sad, but we may have to accept it because WebAssembly goes a lot further and ships code that looks closely to assembler! It's far from the Web we know, but it's still the open Web! Note that it is very different from what we saw in the past with plugins like Java applets, now a WebAssembly function can be called from a usual JS script!</p>
<p>The physical Web is awesome and I'm really excited about this revolution, especially how it could bring a peer-to-peer local Web, using <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Bluetooth_API">Web bluetooth</a> for example. Scott told me that the Mozilla approach with <a class="reference external" href="https://hacks.mozilla.org/2016/09/flyweb-pure-web-cross-device-interaction/">FlyWeb</a> is slightly different because it goes through Wifi, but both teams in contact. The physical Web is an open project, and Google is pushing it strongly. I'm bothered by the <em>URL redirections</em> strategy though because it puts the redirection intermediary in a very strong position. I believe that Google wants to be that intermediary.</p>
</div>
<div class="section" id="personal-feedback">
<h2>Personal feedback</h2>
<p>I never attended a conference that was so close to my flat! 3min walking! It was kind of weird to meet so many prestigious IT talents in my neighbourhood! ...and so many floppy disks :)</p>
<img alt="CC-BY @fullstackfest" class="align-center" src="https://blog.mathieu-leplatre.info/images/fullstackfest-floppy.jpg" />
<p>The venue was great, and very well setup for the show (<a class="reference external" href="https://www.youtube.com/watch?v=vxMASndC3k4&index=1&list=PLe9psSNJBf76DOOKMkDpyo_A5PfZk7JWc">see the introductory video clip!</a>). The coffee was not so good though.</p>
<p>I really enjoyed the fact that there was only <strong>one track</strong>. I never had to divide myself between two interesting talks, and could just sit and enjoy :)</p>
<p>Almost every topic resonated with the challenges we face at work. We are one of the few teams at Mozilla that do both backend and frontend, and that's awesome to feel in the same boat as the whole room of attendees. For example, we implemented <a class="reference external" href="https://amo2kinto-lambda.readthedocs.io">something on Amazon lambda</a> recently, we are well aware of the frontend architecture challenges through our <a class="reference external" href="https://github.com/Kinto/kinto-admin">Kinto-admin</a>, Ethan has a functional programming background in Haskell, and Nico and Mathieu <a class="reference external" href="https://github.com/n1k0/kinto-elm-experiments">are playing with Elm</a>. That's why I strongly regretted that my teammates were not with me!</p>
<p>Also, I had the opportunity to meet Luca and Anthony from the Kuzzle team! We had a lot of fun together and our conversations about generic reusable backends were super insightful — see <a class="reference external" href="https://mail.mozilla.org/pipermail/kinto/2016-September/000197.html">my post on kinto ML</a>.</p>
<p>Last, but not least, I would like to thank Pusher.com for organizing a hack contest using their API! They had a physical button in their booth, and I made a kind-of video booth that records the webcam when the button is pressed. I could not have won this BB8 droid without <a class="reference external" href="https://hacks.mozilla.org/2016/04/record-almost-everything-in-the-browser-with-mediarecorder/">Soledad's insights on MozillaHacks</a>, and I thank her again :)</p>
<img alt="Me holding the prize proudly!" class="align-center" src="https://blog.mathieu-leplatre.info/images/fullstackfest-contest.jpg" />
</div>
<div class="section" id="some-links">
<h2>Some links</h2>
<ul class="simple">
<li><a class="reference external" href="https://www.kontena.io">https://www.kontena.io</a>: open source container platform. The features are really really attractive!</li>
<li><a class="reference external" href="http://toxiproxy.io">http://toxiproxy.io</a>: a proxy to simulate network and system conditions</li>
<li><a class="reference external" href="https://openresty.org">https://openresty.org</a>: REST via Lua in Nginx</li>
<li><a class="reference external" href="https://en.wikipedia.org/wiki/Gossip_protocol">Gossip/epidemic protocols</a> to invalidate cache</li>
<li>Challenge unikernel security with the <a class="reference external" href="http://amirchaudhry.com/bitcoin-pinata">Bitcoin piñata</a></li>
<li>Solve deterministic problems with randomness using <a class="reference external" href="https://en.wikipedia.org/wiki/Monte_Carlo_method">the Monte Carlo method</a></li>
<li>Decision tree learning with <a class="reference external" href="https://en.wikipedia.org/wiki/Random_forest">random forests</a></li>
<li>Attack systems with <a class="reference external" href="https://en.wikipedia.org/wiki/Fork_bomb">fork bombs</a></li>
<li>The $10 million australian #censusfail <a class="reference external" href="http://eftm.com.au/2016/08/how-two-uni-students-built-a-better-census-site-in-just-54-hours-for-500-30752">rebuilt for $500 using AWS Lambda</a></li>
<li>Clean natural language with <a class="reference external" href="https://stanfordnlp.github.io/CoreNLP/">CoreNLP</a></li>
<li><a class="reference external" href="https://howdy.ai">https://howdy.ai</a>: trainable bot for Slack</li>
<li><a class="reference external" href="https://openbazaar.org">https://openbazaar.org</a>: A free market for all, no fees, no restrictions</li>
<li><a class="reference external" href="http://www.apollostack.com">http://www.apollostack.com</a>: everything you need to start using GraphQL</li>
<li><a class="reference external" href="https://www.ted.com/talks/barry_schwartz_on_the_paradox_of_choice?language=en">The paradox of choice</a></li>
<li><a class="reference external" href="http://davidwells.io/19-open-source-react-component-libraries-to-use-in-your-next-project/">19 Open Source React Component Libraries to use in your next project</a></li>
<li><a class="reference external" href="https://github.com/team-gryff/react-monocle">react-monocle</a>: A developer tool to visualize a React application's component hierarchy</li>
<li><a class="reference external" href="https://hyperdev.com">https://hyperdev.com</a>: A fullstack JSBin</li>
<li><a class="reference external" href="http://ux.mulesoft.com">http://ux.mulesoft.com</a>: MuleSoft components library</li>
</ul>
</div>
A year at Mozilla2016-05-05T14:10:00+02:002016-05-05T14:10:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2016-05-05:/a-year-at-mozilla.html<p>When I started the draft of this article, I had spent a year at Mozilla. Now it
has been a year and a half, but the post title would be less cool.</p>
<p><strong>tdlr; An amazing experience so far, a lot happened already. Working remotely is
hard. Being pragmatic and satisfy …</strong></p><p>When I started the draft of this article, I had spent a year at Mozilla. Now it
has been a year and a half, but the post title would be less cool.</p>
<p><strong>tdlr; An amazing experience so far, a lot happened already. Working remotely is
hard. Being pragmatic and satisfy everyone expectations is impossible.</strong></p>
<div class="section" id="the-transition">
<h2>The transition</h2>
<div class="section" id="barcelona">
<h3>Barcelona</h3>
<p>The city is very pleasant, the weather is amazing — it never rains basically.</p>
<p>Being far from close friends is very hard. But I knew moving to Barcelona was
not going to be a problem in itself since I speak Spanish and a little
bit of Catalan.</p>
<p>I really enjoy the Mediterrean culture and gastronomy, and had a lot of
interesting conversations and fascinating debates about politics, especially
with regards to the Catalan independance.</p>
<p>Although the new mayor is making a lot of great infrastructure improvements
for pushing the bicycles forward, it is still a very noisy and contaminated
environment. Car drivers are still in velvet.</p>
<img alt="" src="images/barcelona-giants.jpg" />
</div>
<div class="section" id="working-remotely">
<h3>Working remotely</h3>
<p>Leaving the collocial office of Makina Corpus in Toulouse and working as
a full remotee was definitely the most impacting life change.</p>
<img alt="" src="images/twitter-quote-remote.png" />
<p>From previous experencies as a freelancer, I was aware of my inabilty to
keep a discipline when working from home.</p>
<p>Thanks to <a class="reference external" href="https://twitter.com/areskib">Areski</a> I <a class="reference external" href="http://www.betahaus.es/mathieu-leplatre/">landed at Betahaus</a>,
probably the best place to work from! This building is full of awesome people
from all around the world, working in very distinct areas, from journalism
to wood craving through cartoonists and photographs.</p>
<p>Still, interacting on a daily basis with remote colleagues through video
calls and IRC can sometimes be very frustrating. Especially when the Internet connection
is flaky, when proprietary software suck badly on Linux (<em>I'm looking at you Vidyo</em>)
or when a cacophonous horde of motorcycles hijacks your microphone.</p>
<p>Timezones are also challenging :) Since Mozilla employees are spread around the world,
it's not rare to extend the work day of a few hours in order to get in touch
with colleagues! I can't complain, everything has been very flexible so far.</p>
<p>Favorably, Mozilla offers us the possibility to travel and meet regularly. We make
our best to organize small workshops several times per quarter, in addition to the
gatherings of the whole company twice a year. Those are super fun and extremely
productive!</p>
<img alt="" src="images/orlando-all-hands.jpg" />
</div>
<div class="section" id="maintaining-former-projects">
<h3>Maintaining former projects</h3>
<p>As part of the transition, I had to withdraw from the several open-source projects
<a class="reference external" href="http://makina-corpus.com/blog/metier/2013/geotrek-histoire-dun-projet-libre">I maintained while at Makina Corpus</a>.</p>
<p>Obviously, giving up on something you had cherished for years is delicate. Emotions
come in the way.</p>
<p>Some projects were not receiving the love and attention they deserved and that made
me sad. But I realized I couldn't continue to watch them all and suffered from not
being able to provide thorough answers on issues and questions.</p>
<p>I begged for help or support several times along the year, and eventually I left them :|
That's perhaps part of the open-source darwinism.</p>
<p>By the way, Janis' <a class="reference external" href="https://jazzband.co">Jazzband project</a> is awesome! Think about it
instead of using the Github organization of your current company...</p>
</div>
<div class="section" id="chocolate-factory">
<h3>Chocolate factory</h3>
<p>Working for an organization that has a good image everywhere around the globe
is quite rewarding of course (...a typical reaction: <em>«Oh really! I use it at home,
Firefox is the best search engine to send emails on Facebook»</em>).</p>
<p>I remember very well the feeling of pride when I first came to the Mozilla
office in Paris. Impressed by its stairs covered with a giant fox logo, I felt like
I wasn't up to the job!</p>
<img alt="CC-BY-SA Robert Kaiser https://www.flickr.com/photos/kairo_at/22208348283/" src="images/mozilla-paris-stairs.jpg" />
<p>The welcoming workmates there helped lowering the pressure quickly.
By the way, special thanks to the teams in charge of the different
offices, they are amazing! Being so spoiled sometimes makes me feel disconcerted!</p>
<p>After a year, I somehow got used to the idea. But I must admit that I still
feel some pride when my 3 years old kid shouts out <em>«Daddy's wooooork!»</em> as soon as he spots
the browser logo :)</p>
<p>Still, the popular <em>impostor syndrome</em> is on watch: my coworkers are the best engineers
in the world! A lot of them have their name in the W3C standards, and some have
their own Wikipedia page!</p>
<!-- `Eric <https://blog.mozilla.org/blog/2015/12/16/announcing-mozilla-fellow-eric-rescorla/>`_ co-designed TLS, -->
<!-- `Chris <https://en.wikipedia.org/wiki/Chris_Montgomery>`_ created Ogg/Vorbis, -->
<!-- `Tim <http://people.xiph.org/~tterribe/>`_lead the Theora project, `Jack <https://en.wikipedia.org/wiki/Jack_Moffitt>`_ -->
<!-- co-authored Icecast, `Ian's tools <http://www.ianbicking.org/blog/2014/02/saying-goodbye-to-python.html>`_ -->
<!-- are used by millions of Python developers, `Ben <http://www.groovie.org/>`_ created the Pylons/Pyramid -->
<!-- framework -->
</div>
<div class="section" id="trolling-the-giants">
<h3>Trolling the giants</h3>
<p>If you think Mozilla is just the editor of a declining browser, please take
some time to read <a class="reference external" href="https://blog.mozilla.org/blog/2015/12/08/the-power-of-mozilla/">the power of Mozilla</a>.</p>
<p>There are a lot of expectations towards the organization. Lots of affection, concern, hope,
and disappointment obviously. This leads to very strong criticism and attacks, which
sometimes seem rather unfair. When you look at the overall landscape, who is the enemy?</p>
<img alt="" src="images/twitter-quote-reaction.png" />
<p>Now that I am within the organization I understand what is at stake. Especially
when considering the difference of size with the competitors (~100x). It does not
mean I agree with every decision, but I grasp why concessions are necessary.</p>
</div>
</div>
<div class="section" id="storage-team">
<h2>Storage team</h2>
<p>I could not have joined a more desirable team. Our history takes its roots a long
time ago. We've known each others for years! <a class="reference external" href="http://ziade.org">Tarek</a>'s books about Python were major
pieces while I was learning, <a class="reference external" href="https://notmyidea.org">Alex</a> was a trainee at Makina Corpus when I arrived
there 6 years ago, <a class="reference external" href="https://nicolas.perriault.net/">N1k0</a> was one of the organizers of the fantastic "DjangoCong" 2012
edition, <a class="reference external" href="https://github.com/Natim">Natim</a> and <a class="reference external" href="http://mathieu.agopian.info">magopian</a> were awesome comrades during DjangoCon Toulouse four years ago...</p>
<p>We are all aligned on the vision we have for decentralizing the Web, we work in the
same timezone, we all speak French and we share the same taste for exquisite
home-made food and beverages :)</p>
<p>I shall also mention <a class="reference external" href="https://github.com/michielbdejong">Michiel</a> and <a class="reference external" href="https://github.com/glasserc">Ethan</a>
who joined the team recently!</p>
<p>By the way, check out <a class="reference external" href="http://www.servicedenuages.fr/">our blog</a>, in French and English.</p>
<img alt="" src="images/london-team-selfie.jpg" />
<div class="section" id="organization">
<h3>Organization</h3>
<p>Since we mostly work on backend stuff, we don't suffer too much from release agenda
and marketing communication. The server can usually be upgraded transparently
and our priorities reassessed depending of the present meteorology.</p>
<p>We have precise goals for the current quarter, but we are mostly self-organized.
The open-source aspect of our work is an important property but the main purpose
is still to serve internal needs. Everything we do to make Kinto a community project
for example requires a lot of extra efforts and willpower.</p>
<p>Nevertheless, shipping features on the server-side is usually a lot easier than what
other teams handle on the client-side: we treat with one operating system, deploy one main version,
can introspect the server remotely and monitor internal details without too much pain.</p>
<p>We don't do much of agile ceremonials. Maybe because we work remotely. Also maybe because
we haven't designated anyone as a proper product owner.</p>
<img alt="" src="images/douarnenez-meeting.jpg" />
</div>
<div class="section" id="our-first-project">
<h3>Our first project</h3>
<p>When I joined the team, <a class="reference external" href="https://www.mozilla.org/en-US/firefox/hello/">Firefox Hello</a>
had just been put in production, and the team was responsible for the server part:
<a class="reference external" href="https://github.com/mozilla-services/loop-server">Loop server</a>.
The server handles Websockets and handshakes using NodeJS and Redis.
Fixing bugs and implementing minor features was possibly the best opportunity
for me to dig into this project.</p>
<p>One of the first goals that came up was to build <a class="reference external" href="http://www.servicedenuages.fr/en/service-de-nuages">a readinglist service</a>, a sort
of alternative to Pocket. We succeeded in setting up a stack, deploying and scaling
it quite rapidly, working sometimes late at night. Since we wanted to capitalize on all the efforts we had done
(API design, scaling, ...) we created <a class="reference external" href="http://www.servicedenuages.fr/en/why-cliquet">a toolkit called Cliquet</a>.
I could talk about it at <a class="reference external" href="http://mozilla-services.github.io/cliquet/talks/2015.07.pybcn/">PyBCN</a>
and <a class="reference external" href="http://mozilla-services.github.io/cliquet/talks/2015.10.pyconfr/">Pycon-FR</a>.</p>
<p>We even hacked on <a class="reference external" href="https://github.com/mozilla/readability">readability</a>, a JavaScript library
that transforms a webpage into plain text. Maybe you executed it already if you used the <a class="reference external" href="https://support.mozilla.org/en-US/kb/firefox-reader-view-clutter-free-web-pages">Firefox Reader Mode</a>.
Before joining our team officially, N1k0 made a <a class="reference external" href="https://github.com/mozilla-services/readinglist-client">full-Web version of a readinglist app</a> :)</p>
<p>Unfortunately, the readinglist project was shutdown at the last moment. That was hard to take,
and we had quite a hard time recovering from this disappointment.</p>
<img alt="" src="images/big-chair.jpg" />
</div>
<div class="section" id="booting-kinto">
<h3>Booting Kinto</h3>
<p>We revamped the readinglist server code and saw an opportunity to bring back our idea of <a class="reference external" href="http://www.servicedenuages.fr/en/generic-storage-ecosystem">a generic remote storage</a>.
Our first concrete use-case was a prototype
for a payment and subscription service, which became a pretext to implement a fine-grained
permissions API into our newly born Kinto.</p>
<pre class="literal-block">
> On Fri 26 December 2014, Mathieu wrote:
> [...]
> Whether Daybed will become a true Mozilla project or not is totally
> unclear.
>
</pre>
<p>Daybed never became a Mozilla project, but its concepts were all re-implemented
in Kinto!</p>
<p>Last summer, Firefox OS had no way to synchronize bookmarks and browsing history
accross devices. We built <a class="reference external" href="https://github.com/Kinto/kinto.js">kinto.js</a>, a JavaScript
client for Kinto, focused on offline capabilities and records synchronization.
Personally, I learned a lot from this project. N1k0 was leading us towards modern
and elegant patterns in JavaScript, and we could get our hands dirty with IndexedDB.</p>
<p>As a side project I could spend some time on an offline-first Web app, <a class="reference external" href="http://github.com/leplatrem/Routina/">Routina</a>,
for which we did <a class="reference external" href="https://github.com/Kinto/kinto-react-boilerplate/">a Kinto+React boilerplate</a>.</p>
<img alt="" src="images/orlando-kinto-meeting.jpg" />
<p>We saw an opportunity to spread Kinto within the organization with the <a class="reference external" href="http://www.servicedenuages.fr/en/summer-perspectives">Go Faster initiative</a>, which consists in shipping
settings, content and features quicker that the 6-weeks release cycle.</p>
<p>We even <a class="reference external" href="https://github.com/mozilla-services/servicedenuages.fr/blob/7a3edb700d46b41207a8df97b6341f184818732c/content/2016.02.localisation-service.md">did some experiments</a>
to have the <a class="reference external" href="https://kinto-ota.dev.mozaws.net/attachments/demo.mp4">UI translations shipped asynchronously</a>!</p>
<p>Kinto was <a class="reference external" href="https://news.ycombinator.com/item?id=10733164">featured</a>
<a class="reference external" href="https://news.ycombinator.com/item?id=10994736">twice</a> on Hackernews, and referred as an alternative
to Parse and Firebase! This was both exciting and stressful, especially because it was my
first experience of heavy exposure — unlike Tarek, N1k0 or Alexis who had lead notorious
open source projects before, and already knew the downsides ;)</p>
<p>The funny part was someone saying <em>«Google has Firebase, Apple has Cloudkit,
Facebook has Parse, now Mozilla has Kinto»</em>: how do we reply that we are only a few
folks hacking cool stuff and that top-management is probably not even aware of it?</p>
<img alt="" src="images/london-laptop-pocmo.jpg" />
</div>
<div class="section" id="coming-next">
<h3>Coming next</h3>
<p>In 2016, we now have Kinto instances <a class="reference external" href="https://firefox.settings.services.mozilla.com">running in production</a>,
and about to be used by millions of users to <a class="reference external" href="http://www.servicedenuages.fr/kinto-file-attachment">deliver assets of Firefox mobile</a>,
or <a class="reference external" href="http://www.servicedenuages.fr/en/data-signature">revoke SSL certificates</a>.
It was a great opportunity to grow the eco-system, with <a class="reference external" href="https://github.com/Kinto/kinto-admin/">a Web admin UI</a>
for example!</p>
<pre class="literal-block">
> On Fri 26 December 2014, Mathieu wrote:
> [...]
> I'm not likely to hack the C++ codebase of Firefox/Gecko, even though I
> could and would if it was necessary.
>
</pre>
<p>Well, here you go! In the last few months, I've spent a good proportion of my time
contributing to Gecko. We want to replace the existing system based on a big XML file
downloaded every day with a sweet diff-based synchronization of JSON (<a class="reference external" href="http://ziade.org/2016/02/22/kinto-work-week-recap/">more info...</a>).</p>
<p>On top of that, Kinto could become the key component for the <a class="reference external" href="https://github.com/Kinto/storage-sync">storage.sync API</a>
implementation in Gecko, which would bring the <a class="reference external" href="http://arewewebextensionsyet.com">Web Extension compability table</a> close
to 100%. As Michiel (founder of <a class="reference external" href="https://remotestorage.io/">remoteStorage</a>) said: this is really exciting because it would offer a decentralized
remote storage with encryption support to everybody — <em>contrary to the one offered by Google by the way</em>.</p>
<p>One of the next challenges will presumably be related to horizontal scaling for
massive usage of writable instances. We could also imagine integrating
<a class="reference external" href="https://github.com/Kinto/kinto-webpush/">WebPush notifications</a>
which landed in Firefox 44 and Chrome 42.</p>
<p>Whether Kinto will replace the existing Sync server or not, it is completely uncertain
and would require quite a big amount of work. Although that could make sense :)</p>
<p>Mozilla has a lot of exciting projects in the pipeline, like <a class="reference external" href="https://github.com/servo/servo/">Servo</a>
or <a class="reference external" href="https://github.com/mozilla/tofino">Tofino</a>, which will gain visibility in the
coming months, and could benefit from having a synchronized remote storage!</p>
<img alt="" src="images/internet-for-good.jpg" style="width: 100%;" />
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Thanks Ethan, Areski and Remy for proofreading this article and giving
me early feedback!</p>
</div>
</div>
</div>
Your tests as your specs2015-04-17T00:00:00+02:002015-04-17T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2015-04-17:/your-tests-as-your-specs.html<p>At last, with all this surrounding pressure, you finally decided to write tests.
Now, you always setup TravisCI to run when code change is submitted. You now feel
confident when pull-requests come in, and soon your test suite code coverage will
reach 100%.</p>
<p>Let me be difficult to please here …</p><p>At last, with all this surrounding pressure, you finally decided to write tests.
Now, you always setup TravisCI to run when code change is submitted. You now feel
confident when pull-requests come in, and soon your test suite code coverage will
reach 100%.</p>
<p>Let me be difficult to please here...</p>
<img alt="" class="align-center" src="/images/tests-specs-nitpick.jpg" />
<p>You can reach clean code heaven with one final little step: <strong>transform the tests
suites into specifications</strong>!</p>
<div class="section" id="bad-pattern">
<h2>Bad pattern</h2>
<p>In many Open Source projects, a great attention is paid to the quality of the
code itself. It is quite often that some code is merged without the tests being
exhaustively reviewed.</p>
<p>In other words, it is not rare that the quality of the tests is not reflecting
the quality of the application code.</p>
<p>The most frequent pattern is that all tests are implemented in the same test case <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">GameTest</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_update</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">game</span> <span class="o">=</span> <span class="n">Game</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">game</span><span class="o">.</span><span class="n">played</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
<span class="n">game</span><span class="o">.</span><span class="n">score</span> <span class="o">=</span> <span class="mi">4</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">game</span><span class="o">.</span><span class="n">played</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span>
<span class="n">game</span><span class="o">.</span><span class="n">player</span> <span class="o">=</span> <span class="s1">'Jean-Louis'</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">unicode</span><span class="p">(</span><span class="n">game</span><span class="p">),</span> <span class="s1">'Jean-Louis (score: 4)'</span><span class="p">)</span>
<span class="c1"># See bug #2780</span>
<span class="n">game</span><span class="o">.</span><span class="n">score</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">unicode</span><span class="p">(</span><span class="n">game</span><span class="p">),</span> <span class="s1">'Jean-Louis'</span><span class="p">)</span>
</pre></div>
<p>Writing tests like this has many drawbacks:</p>
<ul class="simple">
<li>One has to read the entire test to understand the expected behaviour of the code;</li>
<li>If the test fails, it will be hard to find out what part of the code has failed precisely;</li>
<li>Refactoring the code will probably mean to rewrite the entire test since instructions are
inter-dependant;</li>
<li>The <tt class="docutils literal">TestCase</tt> and <tt class="docutils literal">setUp()</tt> notions are underused.</li>
</ul>
</div>
<div class="section" id="better-pattern">
<h2>Better pattern</h2>
<p>A simple way to improve the quality of the tests is to see them as specifications.
After all, it makes sense, you add some code for a reason! The tests are not only
here to prevent regressions, but also to explicit the expected behaviour of the application!</p>
<p>I believe that many projects would take great benefits if following this approach <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>.</p>
<div class="highlight"><pre><span></span><span class="c1"># test_game.py</span>
<span class="k">class</span> <span class="nc">InitializationTest</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">game</span> <span class="o">=</span> <span class="n">Game</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">test_default_played_status_is_false</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">game</span><span class="o">.</span><span class="n">played</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">UpdateTest</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">game</span> <span class="o">=</span> <span class="n">Game</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">game</span><span class="o">.</span><span class="n">player</span> <span class="o">=</span> <span class="s1">'Jean-Louis'</span>
<span class="bp">self</span><span class="o">.</span><span class="n">game</span><span class="o">.</span><span class="n">score</span> <span class="o">=</span> <span class="mi">4</span>
<span class="k">def</span> <span class="nf">test_played_status_is_true_if_score_is_set</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">game</span><span class="o">.</span><span class="n">played</span><span class="p">,</span> <span class="kc">True</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_string_representation_is_player_with_score_if_played</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">unicode</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">game</span><span class="p">),</span> <span class="s1">'Jean-Louis (score: 4)'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_string_representation_is_only_player_if_not_played</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># See bug #2780</span>
<span class="bp">self</span><span class="o">.</span><span class="n">game</span><span class="o">.</span><span class="n">score</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">unicode</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">game</span><span class="p">),</span> <span class="s1">'Jean-Louis'</span><span class="p">)</span>
</pre></div>
<p>Writing tests like this has many advantages:</p>
<ul class="simple">
<li>Each case isolates a situation;</li>
<li>Each test is an aspect of the specification;</li>
<li>Each test is independant;</li>
<li>The testing vocabulary is honored: we <em>setup</em> a <em>test case</em>;</li>
<li>If a test fails, it is straightforward to understand what part of the spec
was violated;</li>
<li>Tests that were written when fixing bugs will explicit the expected behaviour
for edge cases.</li>
</ul>
</div>
<div class="section" id="reporting">
<h2>Reporting</h2>
<p>Now that we have explicited the specs, we will want to read them properly.</p>
<p>One of things I like in JavaScript is <a class="reference external" href="http://mochajs.org">Mocha</a>. Appart
from the nice API and the very rich feature set, its default test reporter is
great, tt is colourful and structurally invites you to write tests as specs.</p>
<img alt="" class="align-center" src="/images/tests-specs-mocha.png" />
<p>In our project, we were using <a class="reference external" href="http://nose.readthedocs.org">nose</a>, so I
decided to write a reporter that would produce the same output as Mocha.</p>
<p>You can install and use it this way:</p>
<pre class="literal-block">
$ pip install nose-mocha-reporter
$ nosetests --with-mocha-reporter yourpackage/
</pre>
<p>It will produce the following output:</p>
<img alt="" class="align-center" src="/images/tests-specs-nose-reporter.png" />
<p>It takes the tests suites and extract the names as readable strings:</p>
<ul class="simple">
<li><tt class="docutils literal">tests/core/test_game.py</tt> → <tt class="docutils literal">CORE GAME</tt></li>
<li><tt class="docutils literal">class InitializationTest(TestCase)</tt> → <tt class="docutils literal">Initialization</tt></li>
<li><tt class="docutils literal">def test_played_status_is_true_if_score_is_set</tt> → <tt class="docutils literal">Played status is true if score is set</tt></li>
</ul>
<p>It also mesures the execution time of each test in order highlight the slow ones.</p>
<p>To conclude, this reporter has a pretty modest objective: remind you that <strong>the tests
you write should be read as specifications</strong> <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a>!</p>
</div>
<div class="section" id="special-thanks">
<h2>Special thanks!</h2>
<p>I'm very grateful to <a class="reference external" href="http://antoine.cezar.fr/">Antoine</a> and
<a class="reference external" href="http://alexmarandon.com/">Alex</a> that showed me the light on
this. Since they might not be conscious of the influence they had on me,
I jump on the occasion to thank them loudly :)</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>See <a class="reference external" href="https://github.com/makinacorpus/Geotrek/blob/v0.33.4/geotrek/trekking/tests/test_models.py#L71-L99">a good example</a> that I wrote in the past</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>For example, <a class="reference external" href="https://github.com/mozilla-services/cliquet/blob/1.7.0/cliquet/tests/resource/test_record.py">see this code</a> I wrote later on.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>To be honest, I haven't worked much with <a class="reference external" href="http://pytest.org">pytest</a>
(<em>I probably should</em>), and I don't know its eco-system: there might
something similar...</td></tr>
</tbody>
</table>
</div>
Subtivals, remote display ?2015-03-26T00:00:00+01:002015-03-26T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2015-03-26:/subtivals-remote-display.html<p>Recently I was asked to think about a new feature for Subtivals : allowing
spectators to read subtitles on their mobile or tablet.</p>
<div class="section" id="usual-setup">
<h2>Usual setup</h2>
<p>In a typical installation, Subtivals overlays the subtitles on top of the
main image coming from the cinema projector. It allows to add captioning on
any …</p></div><p>Recently I was asked to think about a new feature for Subtivals : allowing
spectators to read subtitles on their mobile or tablet.</p>
<div class="section" id="usual-setup">
<h2>Usual setup</h2>
<p>In a typical installation, Subtivals overlays the subtitles on top of the
main image coming from the cinema projector. It allows to add captioning on
any movie, especially when the physical copy (35mm or DCP) does not carry them.
That's why most of Subtivals users are international film festivals, cultural centers or
film libraries.</p>
<img alt="" class="align-center" src="/images/subtivals-remote-cinema.png" style="width: 500px;" />
<p>Another common use-case is live captionning, where subtitles are projected on
a small screen below the scene. It allows to add captioning to concerts, theater,
operas, conference talks, live shows... Surprisingly there was no tool for that,
and most people were using Poweroint slides before discovering Subtivals.</p>
<img alt="" class="align-center" src="/images/subtivals-remote-opera.png" style="width: 500px;" />
</div>
<div class="section" id="use-case">
<h2>Use-case</h2>
<p>Overlaying subtitles requires a powerful projector (~4000 Lumens for white/yellow
and ~5000 Lumens for the colors used in hard of hearing captioning).</p>
<p>When the scene is very large or the atmosphere very bright, such as open air events,
it can be hard to obtain sharp subtitles on a screen, even with a powerful longthrow projector.</p>
<img alt="" class="align-center" src="/images/subtivals-remote-stadium.png" style="width: 300px;" />
<p>Also, when the venue is very big, such as operas, subtitles have to be projected
with a very big text size for the spectators seated in the last rows. And
those in the front raws won't read easily either.</p>
<img alt="" class="align-center" src="/images/subtivals-remote-theater.png" style="width: 300px;" />
<p>Having remote display of subtitles for these situations makes sense. Subtitles
should be shown synchronously on various screens.</p>
</div>
<div class="section" id="solution">
<h2>Solution</h2>
<p>We can imagine:</p>
<ul class="simple">
<li>Adding several small LCD screens in the venue (like every 20 meters) instead of
one big projected image;</li>
</ul>
<img alt="" class="align-center" src="/images/subtivals-remote-backseat.png" style="width: 300px;" />
<ul class="simple">
<li>Showing subtitles on the spectator mobile or tablet;</li>
</ul>
<img alt="" class="align-center" src="/images/subtivals-remote-tablet.png" style="width: 300px;" />
<ul class="simple">
<li>Adding small screens (or tablets) on the backseat, like it is already done in
some operas;</li>
</ul>
<img alt="" class="align-center" src="/images/subtivals-remote-backseat.jpg" style="width: 300px;" />
</div>
<div class="section" id="technology">
<h2>Technology</h2>
<p>I have of course no intention to implement native mobile applications, relying
on a specific communication protocol and struggling with obscure restricted
application stores.</p>
<p>It has to be universal, fully open and follow well-known standards, it will
thus be built on a Web stack.</p>
<p>We now have WebSockets, and Qt provides everything to use them! The remote display
will then be a simple JavaScript Web page, receiving subtitles in real-time from
the WebSocket server.</p>
<p>Funny hacks ahead :)</p>
</div>
<div class="section" id="prototype">
<h2>Prototype</h2>
<p>I could build a very simple prototype in a few hours, with a minimalist JavaScript
code for the display page and some idiot code for the Node.js server. The code
is <a class="reference external" href="https://github.com/Subtivals/live.subtivals.org">online</a>.</p>
<p>The Qt code in Subtivals in charge of push the subtitles in real time is
<a class="reference external" href="https://github.com/traxtech/subtivals/pull/252">also very small</a>.</p>
<p>It looks already fun and promising :)</p>
<video src="/images/subtivals-remote.webm" width="500" controls>
<p>Your browser does not support the video element </p>
</video></div>
<div class="section" id="strategy-1-global">
<h2>Strategy #1: Global</h2>
<p>We deploy a central server that is used by default in the application.</p>
<img alt="" class="align-center" src="/images/subtivals-remote-server.png" style="width: 500px;" />
<p><strong>Pros</strong></p>
<ul class="simple">
<li>Operators have no setup, it's just clic and play, like it always has
been with Subtivals;</li>
<li>Spectators have no setup either, they just open a Web page;</li>
<li>An opportunity to generate some regular income if we choose to charge for the
service;</li>
</ul>
<p><strong>Cons</strong></p>
<ul class="simple">
<li>It relies on an Internet connection;</li>
<li>The server has a regular cost, even if unused;</li>
<li>The server should scale to support several thousands spectators and potentially
several simultaneous projections in the world;</li>
<li>Some minimal cryptography has to be introduced to prevent attackers from
sending messages to the audience;</li>
<li>It implies some stressful responsabilities 24/7 for events happening all over the
globe;</li>
</ul>
</div>
<div class="section" id="strategy-2-local-server">
<h2>Strategy #2: local server</h2>
<p>A variant of the first one, a server is deployed locally. A local wifi can
be setup in case Internet is not available.</p>
<img alt="" class="align-center" src="/images/subtivals-remote-localserver.png" style="width: 500px;" />
<p><strong>Pros</strong></p>
<ul class="simple">
<li>It does not rely on Internet, and can be fully autonomous on site;</li>
<li>The server and client code can be a lot simpler, since the pairing of Subtivals
with the clients is done locally only;</li>
<li>Operators are responsible for their installation, no alert email for us
at 2AM in July;</li>
</ul>
<p><strong>Cons</strong></p>
<ul class="simple">
<li>It will remain an obscure feature for most Subtivals users;</li>
<li>It requires some networking skills to wire-up the whole installation;</li>
<li>This may imply packaging of Web server stacks for Windows and Mac OS X,
which is truely not one of my passions;</li>
</ul>
</div>
<div class="section" id="strategy-3-subtivals-as-server">
<h2>Strategy #3: Subtivals as server</h2>
<p>We get rid of the local Web server, and the Subtivals software itself acts as
a server, pushing subtitles itself to the audience.</p>
<img alt="" class="align-center" src="/images/subtivals-remote-localwifi.png" style="width: 500px;" />
<p><strong>Pros</strong></p>
<ul class="simple">
<li>Nothing extra to be installed, just networking/firewall setup;</li>
<li>It paves the way to the implementation of a Subtivals <a class="reference external" href="https://github.com/traxtech/subtivals/issues/145">remote control</a>;</li>
<li>The operator can track the number of spectators;</li>
</ul>
<p><strong>Cons</strong></p>
<ul class="simple">
<li>It could affect Subtivals stability;</li>
<li>The code base would grow;</li>
<li>I expect to receive emails complaining about the application not being
reachable, all because of firewall setup etc.</li>
</ul>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>We might go for the second strategy at first. We will provide some assistance
to the festival organizers for setting up the stack. It will allow us to get started
with a very small effort on the code implementation.</p>
<p>Later, we can undertake the first strategy. By the way, if someone is interested
in implementing and running such a Web service 24/7, charging users or not,
please contact me :)</p>
<p>The third strategy scares me, but the remote control idea is awesome! It means
that an operator could control the subtitles projection from her smartphone,
seating among the audience instead of from the booth!</p>
</div>
Big life changes2014-12-26T14:10:00+01:002014-12-26T14:10:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2014-12-26:/big-life-changes.html<p>So far I haven't written anything truely personal on this blog, but I was
asked a lot of questions recently, and thought the answers might be worth a post :)</p>
<p>First of all, <strong>thank you very much all of you</strong> for the massive amount of
<a class="reference external" href="https://twitter.com/leplatrem/status/542993732367556608">positive</a> <a class="reference external" href="https://www.linkedin.com/pulse/activities/mathieu-leplatre%2B0_1DkGqDxItMdaF_sJjg5h8J?trk=mp-reader-h">feedback</a>
regarding my new position …</p><p>So far I haven't written anything truely personal on this blog, but I was
asked a lot of questions recently, and thought the answers might be worth a post :)</p>
<p>First of all, <strong>thank you very much all of you</strong> for the massive amount of
<a class="reference external" href="https://twitter.com/leplatrem/status/542993732367556608">positive</a> <a class="reference external" href="https://www.linkedin.com/pulse/activities/mathieu-leplatre%2B0_1DkGqDxItMdaF_sJjg5h8J?trk=mp-reader-h">feedback</a>
regarding my new position! Being taken on in the Cloud Services team at Mozilla is definitely very exciting!</p>
<p>But that is not the only upcoming change: we moved from Toulouse to Barcelona!
Combining several big life changes in a very short period can be stimulating, but
it's undoubtedly anxiety producing!</p>
<div class="section" id="about-to-change">
<h2>About to change</h2>
<div class="section" id="where-i-live">
<h3>Where I live...</h3>
<p>I arrived in Toulouse 10 years ago, but feel like it was yesterday. Even
though it was not continuous because I lived in Middle-East and South America meanwhile,
Toulouse became my world, with close friends and a lot of memories.</p>
<p>Barcelona is the hometown of my partner, and the kids already speak Catalan.</p>
<p>I believe I am prepared for this, ready to forget about many little details,
like french bakeries and fresh tap water, but I can already forsee many positive
aspects of living in a sunny coastal metropolis :)</p>
<img alt="by MorBCN" src="/images/barcelona.jpg" />
</div>
<div class="section" id="who-i-work-for">
<h3>Who I work for...</h3>
<p>More than 4 years ago, I left Météo France to join <a class="reference external" href="http://makina-corpus.com">Makina Corpus</a>,
a small company dedicated to Free and Open Source Software.</p>
<p>As a developer, it feels like I was born there! I discovered the interactions
with communities, the art and subtleties of maintening software in the open.</p>
<p>I could really take advantage of the freedom we had within the teams. The atmosphere
was geeky and acquisition of knowledge was permanent! Being professional, about
code quality, method, approach and taste for sharing, all comes from there.</p>
<p>It was hence hard to leave and tear that apart, since I certainly could have worked remotely from Barcelona. On top of that, <a class="reference external" href="http://geotrek.fr">the main product</a> I was working on is
truely becoming a reference.</p>
<p>But the opportunity to join Mozilla represents a true aknowledgement and progression
in the world of Open Source that one can't let pass by :) <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a></p>
</div>
<div class="section" id="how-i-work">
<h3>How I work...</h3>
<p>Mozilla has no offices in Barcelona yet, so I will be working remotely from
a coworking space. Working at home is not my cup of tea: I'm addicted to cycling in
the morning and self-discipline isn't one of my strong points anyway.</p>
<p>There are many (many) coworking spaces, and I was told there's a few other Mozillians
down there... <em>to be continued</em></p>
<p>The Cloud Services team is spread around the world, and interacting with people
on several timezones (from San Francisco to Melbourne) will be completely new to me!
This represents a couple of challenges, but I feel confident since the interactions
are mainly focused on software, with the usual tools like IRC, Trello, Github or Bugzilla.</p>
</div>
<div class="section" id="what-i-build">
<h3>What I build...</h3>
<p>At Meteo-France or Makina Corpus, the fields of application were almost always
very specific, and the number of simultaneous users was generally low. Most of
the time, the challenges were about delivering some complex features, in a
rather short period.</p>
<p>At Mozilla, scale and performance is the difficult task. I was used to take advantage
of cache and optimizations techniques to handle long complex CPU-consuming processes,
but now I will have to learn how to handle millions of small simple requests.</p>
<p>Likewise, authentication, cryptography and security aspects have never been crucial in my past
experiences. But when you join an organization whose objective is to build solutions
for a sustainable Open Web, data privacy is the key and nothing should be missed!</p>
</div>
</div>
<div class="section" id="sweep-curiosity-away">
<h2>Sweep curiosity away</h2>
<p>This part is a tentative to answer most questions I received so far :)</p>
<img alt="" src="/images/firefox-wear-pride.jpg" />
<div class="section" id="what-was-the-recruiting-process">
<h3>What was the recruiting process ?</h3>
<p>For almost two years, we have been hacking on <a class="reference external" href="http://daybed.readthedocs.org">Daybed</a> with
Alexis and Remy, two members of the Cloud services team. We met several times,
at various Python or Django conferences, and we let us understand that if
a position would be available at Mozilla, we would be mutually delighted to work together!</p>
<p>At the end of the summer, we discussed it again, and the official recruitment
process started! After several interviews, I received a proposition :)</p>
</div>
<div class="section" id="oh-so-will-you-work-on-the-browser">
<h3>Oh, so will you work on the browser ?</h3>
<p>Yes and no!</p>
<p>Yes, because most of the services deployed by the Cloud services team
provide features to the Firefox browser (e.g. <em>Firefox Sync</em>, <em>Firefox Hello</em>, ...).</p>
<p>No, because I'm not likely to hack the C++ codebase of Firefox/Gecko, even though I
could and would if it was necessary.</p>
</div>
<div class="section" id="what-will-do-in-the-cloud-services-team">
<h3>What will do in the Cloud services team ?</h3>
<p>The team is in charge of various Open Source applications and libraries.</p>
<p>We basically build Web APIs that are used within Mozilla products. For the most famous:</p>
<ul class="simple">
<li>Sync</li>
<li>Firefox Accounts</li>
<li>Location service</li>
<li>Marketplace</li>
<li>... <a class="reference external" href="https://wiki.mozilla.org/CloudServices">and more</a> !</li>
</ul>
<p>Personnaly, I will participate at the maintenance of Firefox Hello services,
and start some new projects in regards to remote data storage, like bringing cloud features
to the <a class="reference external" href="https://support.mozilla.org/en-US/kb/how-to-use-reader-mode">reader mode</a>.</p>
<p>Indirectly, I will of course bring some energy to the several underlying projects,
such as <a class="reference external" href="http://cornice.readthedocs.org">Cornice</a> or <a class="reference external" href="http://circus.readthedocs.org">Circus</a>.</p>
</div>
<div class="section" id="how-does-mozilla-generate-revenue-to-pay-you">
<h3>How does Mozilla generate revenue to pay you ?</h3>
<p>There are several entities: the Mozilla Project, founded in 1998 to resurrect
Netscape; the Mozilla Foundation, a non-profit organization founded
in 2003, and the Mozilla Corporation, a subsidiary founded in 2005, that handles
the goals of the former.</p>
<p>Basically, I am an employee of the Mozilla Corporation, which <a class="reference external" href="https://www.mozilla.org/en-US/foundation/annualreport/2013/faq/">earns money from the search feature in Firefox</a>, thinly
augmented with donations and grants.</p>
<p>The key is how the money is spent, and this is what makes the Mozilla Corporation <strong>unique</strong>.
Just read the <a class="reference external" href="https://www.mozilla.org/en-US/mission/">Mozilla Manifesto</a>: there is no other company in the world whose aim is to maintain the Web as a universal resource for the whole humanity!</p>
</div>
<div class="section" id="will-you-still-use-chromium">
<h3>Will you still use Chromium ?</h3>
<p>I have been using Firefox since the 0.8 version, in early 2004. Before that I wasn't
particularly attached to any browser on the market, even if Netscape and Mozilla
were the ones I used most extensively at the University. From that date, I
installed hundreds of Firefox, in many places of the world :)</p>
<img alt="" src="/images/firefox-qatar-2006.jpg" />
<p>I remember that I started to use Chromium to get the web version of Tweetdeck, in 2011.
A few months later I was working on a JavaScript project with offline capabilities,
bound to WebSQL, thus not supported by Firefox. Chromium hadn't become my main browser yet.
The next year, we started a cartography application with huge vectorial datasets.
Chromium was like ten times faster at rendering massive amounts of SVG. And with time,
I became addicted to the Chrome dev tools, from profiling to inspecting pseudo-elements.
Firefox was not my main browser anymore.</p>
<p>I've started to use it back for daily stuff a few months ago, to decontaminate
myself before the big day :) And I must admit that I like it so far!
The developper tools have improved greatly, and it does not feel sluggish at all!</p>
</div>
<div class="section" id="what-does-this-mean-for-daybed">
<h3>What does this mean for Daybed ?</h3>
<p>Most Daybed contributors are now working together in the same team, and this is
very exciting! Whether Daybed will become a true Mozilla project or not is totally
unclear.</p>
<p>But it does not really matter, since the most important thing is the experience we've built
together as a team on this project!</p>
<p>Daybed is still like a lab, where we can experiment stuff and build our vision of
remote data storage.</p>
<p><em>Stay tuned!</em></p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Hey come on! Founders of pip, virtualenv, Pylons, circus, pelican, ... are my new team mates!</td></tr>
</tbody>
</table>
</div>
</div>
Use Docker to ease database schema migrations2014-11-06T00:00:00+01:002014-11-06T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2014-11-06:/use-docker-to-ease-database-schema-migrations.html<p><a class="reference external" href="https://www.docker.com/">Docker</a> is very versatile, and can fulfill many (<em>many</em>) different
use-cases. You've probably heard of it to put applications in production.</p>
<p>Even though I wrote quite a few <em>Dockerfiles</em> to package and ship Web apps,
what has convinced me most is running database backends during development :)</p>
<p>For example, when you …</p><p><a class="reference external" href="https://www.docker.com/">Docker</a> is very versatile, and can fulfill many (<em>many</em>) different
use-cases. You've probably heard of it to put applications in production.</p>
<p>Even though I wrote quite a few <em>Dockerfiles</em> to package and ship Web apps,
what has convinced me most is running database backends during development :)</p>
<p>For example, when you work with relational databases, and develop features in different
branches, it can be very painful to reset your divergent schemas each time you switch branch.
(<em>like with Django South, or 1.7</em>).</p>
<p>Likewise, if you want to replay data imports, migrations, on different database schemas,
or with various PostgreSQL versions etc.</p>
<p>In this article, I will show how to use the <tt class="docutils literal">commit</tt> feature of <em>Docker</em> to
solve these issues.</p>
<div class="section" id="run-postgresql-in-docker">
<h2>Run PostgreSQL in Docker</h2>
<p>First, you should choose an image in the <a class="reference external" href="https://registry.hub.docker.com/">Docker index</a>.</p>
<p>Once you've chosen the one that fits your needs, pull an image is as easy as:</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>docker<span class="w"> </span>pull<span class="w"> </span>helmi03/docker-postgis
</pre></div>
<p>Then you can run it as a deamon, on <tt class="docutils literal">localhost:5432</tt>:</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>-d<span class="w"> </span>-p<span class="w"> </span><span class="m">5432</span>:5432<span class="w"> </span>helmi03/docker-postgis
</pre></div>
<p>Congratulations, you have a PostGIS container running. Log-in as superuser with <em>docker/docker</em>.</p>
<p>You can see its identifier with <tt class="docutils literal">sudo docker ps</tt>, and stop it using <tt class="docutils literal">sudo docker stop <ID></tt>.</p>
</div>
<div class="section" id="commit">
<h2>Commit</h2>
<p>The killer feature of <em>Docker</em> is its incremental filesystem, which allows to
tag and commit the states of containers.</p>
<p>For example, once you've created an empty database in your container, you will
save its state :</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>docker<span class="w"> </span>commit<span class="w"> </span><ID><span class="w"> </span>postgis-empty
</pre></div>
<p>Then, for example, you will initialize your database tables for one of your applications,
create users, etc. You commit !</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>docker<span class="w"> </span>commit<span class="w"> </span><ID><span class="w"> </span>geotrek-0.28-empty
</pre></div>
<p>Loaded data into the database ? Commit !</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>docker<span class="w"> </span>commit<span class="w"> </span><ID><span class="w"> </span>geotrek-0.28-demo
</pre></div>
<p>Now, that you've performed a series of commits, you can stop and re-run the container
at a previous state! <em>Docker</em> is the git of virtual machines!</p>
<p>See the list of your tags with <tt class="docutils literal">sudo docker images</tt>:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>sudo<span class="w"> </span>docker<span class="w"> </span>images
REPOSITORY<span class="w"> </span>TAG<span class="w"> </span>IMAGE<span class="w"> </span>ID<span class="w"> </span>CREATED<span class="w"> </span>VIRTUAL<span class="w"> </span>SIZE
geotrek-0.28-demo<span class="w"> </span>latest<span class="w"> </span>48f51d78273a<span class="w"> </span><span class="m">2</span><span class="w"> </span>weeks<span class="w"> </span>ago<span class="w"> </span><span class="m">1</span>.236<span class="w"> </span>GB
geotrek-0.28-empty<span class="w"> </span>latest<span class="w"> </span>3985183cd01a<span class="w"> </span><span class="m">4</span><span class="w"> </span>weeks<span class="w"> </span>ago<span class="w"> </span><span class="m">1</span>.208<span class="w"> </span>GB
postgis-empty<span class="w"> </span>latest<span class="w"> </span>24ec864dc058<span class="w"> </span><span class="m">4</span><span class="w"> </span>months<span class="w"> </span>ago<span class="w"> </span><span class="m">844</span>.4<span class="w"> </span>MB
helmi03/docker-postgis<span class="w"> </span>latest<span class="w"> </span>f62d8f0fb8af<span class="w"> </span><span class="m">9</span><span class="w"> </span>months<span class="w"> </span>ago<span class="w"> </span><span class="m">746</span>.2<span class="w"> </span>MB
</pre></div>
</div>
<div class="section" id="checkout">
<h2>Checkout</h2>
<p>Checkout a previous state is very simple, it's like the first time you started
the image from Docker index, except that you now specify the name of your tag!</p>
<p>First, stop the current instance:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>sudo<span class="w"> </span>docker<span class="w"> </span>ps
CONTAINER<span class="w"> </span>ID<span class="w"> </span>IMAGE<span class="w"> </span>COMMAND<span class="w"> </span>CREATED<span class="w"> </span>STATUS<span class="w"> </span>PORTS
7e1e44ce36d8<span class="w"> </span>geotrek-0.28-demo:latest<span class="w"> </span><span class="s2">"/start.sh"</span><span class="w"> </span><span class="m">6</span><span class="w"> </span>hours<span class="w"> </span>ago<span class="w"> </span>Up<span class="w"> </span><span class="m">6</span><span class="w"> </span>hours<span class="w"> </span><span class="m">0</span>.0.0.0:5432->5432/tcp
$<span class="w"> </span>sudo<span class="w"> </span>docker<span class="w"> </span>stop<span class="w"> </span>7e1e44ce36d8
</pre></div>
<p>Re-run at a previous state:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>sudo<span class="w"> </span>docker<span class="w"> </span>run<span class="w"> </span>-d<span class="w"> </span>-p<span class="w"> </span><span class="m">5432</span>:5432<span class="w"> </span>postgis-empty
</pre></div>
<p>You will quickly figure out that you can:</p>
<ul class="simple">
<li>Run different versions of your containers (or PostgreSQL servers) on different ports</li>
<li>Restore the state of your database for your <em>master</em> branch at one fell swoop!</li>
<li>Replay migrations scripts</li>
<li>Run your application on customer database with no configuration change</li>
<li>...</li>
<li>Have the same approach with <em>CouchDB</em>, <em>Redis</em>, <em>ElasticSearch</em>, ...</li>
</ul>
<p>To be honest, it's like git, it changed my way of working and I can't go
without it anymore...</p>
</div>
Good practices for Open Source projects2014-08-19T00:00:00+02:002014-08-19T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2014-08-19:/good-practices-for-open-source-projects.html<p>In this article, I gathered some good practices for open source projects.</p>
<p>I made my best to keep it short and not too verbose. If you want to discuss,
contribute or enhance this article, <a class="reference external" href="https://github.com/leplatrem/blog.mathieu-leplatre.info/blob/master/content/Dev/opensource_project_good_practices.rst">simply use Github</a> !</p>
<div class="section" id="documentation">
<h2>Documentation</h2>
<ul class="simple">
<li>A README file is the bare minimum</li>
<li>More documentation content is to …</li></ul></div><p>In this article, I gathered some good practices for open source projects.</p>
<p>I made my best to keep it short and not too verbose. If you want to discuss,
contribute or enhance this article, <a class="reference external" href="https://github.com/leplatrem/blog.mathieu-leplatre.info/blob/master/content/Dev/opensource_project_good_practices.rst">simply use Github</a> !</p>
<div class="section" id="documentation">
<h2>Documentation</h2>
<ul class="simple">
<li>A README file is the bare minimum</li>
<li>More documentation content is to be stored in a <tt class="docutils literal">docs/</tt> folder</li>
<li>Use the fabulous <a class="reference external" href="http://rtfd.org">Read the Docs</a> service</li>
<li>Mention the license and copyright holder</li>
<li>Explain <strong>why</strong> this project was created</li>
<li>Mention existing related projects</li>
<li>How to <strong>install</strong> and <strong>use</strong> it ?</li>
<li>Frequently asked questions (<em>time saver</em>)</li>
<li>Link to the list of contributors</li>
<li>Provide details about design and architecture</li>
<li>Give instructions to developers on how to contribute</li>
<li>Mention what you expect from contributors (<em>Definition-of-Done, checklist,
conventions...</em>)</li>
</ul>
</div>
<div class="section" id="source-code">
<h2>Source code</h2>
<ul class="simple">
<li>Use <strong>English</strong> everywhere</li>
<li>Use a decentralized versioning system (<em>Git</em>, <em>mercurial</em>, ...)</li>
<li>Unit test your code. <em>Software without tests is broken by design</em></li>
<li>Setup continuous integration (with <a class="reference external" href="http://travis-ci.org">TravisCI</a>)</li>
<li>Follow the programming language conventions (<a class="reference external" href="https://flake8.readthedocs.org">PEP8</a>, <a class="reference external" href="http://www.jslint.com">JSLint</a>, ...)</li>
<li>Follow the framework conventions, and list explicitly the infringements you
decided to make in the documentation</li>
<li>Be a professional coder, by following <a class="reference external" href="http://www.amazon.com/The-Clean-Coder-Professional-Programmers/dp/0137081073">Uncle Bob recommendations</a></li>
<li>Use a translation system, like <em>gettext</em>, and keep English as the default language</li>
<li>Enable collaborative translation of your app with <a class="reference external" href="https://www.transifex.com">Transifex</a></li>
<li>Provide simple commands to install or run tests (in a <em>Makefile</em> for example)</li>
<li>Test your project features (<em>functional tests</em>)</li>
<li>Make sure the tests self-describe your project specifications</li>
</ul>
</div>
<div class="section" id="releases">
<h2>Releases</h2>
<ul class="simple">
<li>Use semantic versionning (<em>main version, features change, bug fixes</em>)</li>
<li>Keep a list of changes by version (<em>Changelog</em>)</li>
<li>Follow some workflow for your changelog (Include it in docs, update it in the merge commit of pull-requests, <a class="reference external" href="http://tech.novapost.fr/changelog-howto-en.html">read more recommendations</a>...)</li>
<li>Create a tag for each release (<tt class="docutils literal">vX.Y.Z</tt>)</li>
<li>Create a branch for each version (<a class="reference external" href="http://fle.github.io/an-efficient-git-workflow-for-midlong-term-projects.html">recommended workflow</a>)</li>
<li>Publish your release (<em>backup copy</em>) in a repository (<a class="reference external" href="https://pypi.python.org">PyPi</a>, <a class="reference external" href="https://www.npmjs.org">NPM</a>)</li>
<li>Automate your release process (<em>Makefile</em>, <a class="reference external" href="http://zestreleaser.readthedocs.org">Zest releaser</a>, <a class="reference external" href="https://www.npmjs.org">npm</a>)</li>
<li><em>Release often, release early</em></li>
<li>Communicate about new versions (<em>tweet</em>, <a class="reference external" href="https://www.openhub.net">OpenHub</a>, <a class="reference external" href="http://freecode.com">Freecode</a>, ...)</li>
</ul>
</div>
<div class="section" id="community">
<h2>Community</h2>
<ul class="simple">
<li>Make sure users can interact somewhere about your project (<a class="reference external" href="https://www.uservoice.com">UserVoice</a>, <em>Google groups</em>, ...)</li>
<li>Setup alerts on <a class="reference external" href="http://stackoverflow.com">Stackoverflow</a> to help users and promote your project</li>
<li>Make your best to find at least one valuable contributor, and give him
permissions on the repository</li>
<li>Be clear about the project <em>functional perimeter</em></li>
<li>Reject every addition of feature that introduces complexity or twists the project
main use cases</li>
<li>Look for a successor as soon as the motivation goes down</li>
</ul>
</div>
<div class="section" id="history">
<h2>History</h2>
<ul class="simple">
<li>History of commits is as valuable as comments in the source code</li>
<li>Mention the ticket number in the commit messages (e.g. <tt class="docutils literal">Update specs (ref #123)</tt>)</li>
<li>Respect the commits messages formatting of your community (e.g. Drupal's prefixes like <tt class="docutils literal">CHG</tt>, <tt class="docutils literal">DOC</tt>...)</li>
</ul>
<div class="section" id="workflow">
<h3>Workflow</h3>
<ul class="simple">
<li>Keep the <em>master</em> branch stable</li>
<li>Create a dedicated branch with an explicit name for each feature (e.g. <tt class="docutils literal">187_add_drop_down_menu</tt>)</li>
<li>Use pull-requests</li>
<li>Even if you are the owner, and alone in your project, use pull-requests (<em>allows code review, triggers integration tests, history is clearer, ...</em>)</li>
<li>Ideally, who merges the pull-request is not its author</li>
<li>Ideally merge branches with the no fast-forward option (<tt class="docutils literal">git merge <span class="pre">--no-ff</span></tt>)</li>
<li>Create a <em>develop</em> branch in case of major refactoring, or follow a proper <em>Git workflow</em>
to ease merging</li>
</ul>
</div>
</div>
<div class="section" id="github">
<h2>Github</h2>
<ul class="simple">
<li>Use Github</li>
<li>Use Github issues (<em>as much as you can, for everything</em>)</li>
<li>Even if you are alone, use Github issues as a TODO list</li>
<li>Create some standard labels (<em>help-needed</em>, <em>docs</em>, <em>duplicate</em> ...)</li>
<li>Use the fantastic <a class="reference external" href="https://github.com/blog/1375%0A-task-lists-in-gfm-issues-pulls-comments">checklists</a> feature in issues and pull-requests descriptions</li>
<li>Take advantage of the milestones, for the next version, or for a general roadmap (e.g. <em>Soon</em>, <em>Later</em>, <em>Final 1.0</em>, ...)</li>
<li>Categorize the labels, renaming them with a convention (e.g. <em>priority: low</em>, <em>priority: high</em>, ...)</li>
<li>Copy the changelog parts into the dedicated releases page</li>
<li>Use the <a class="reference external" href="https://github.com/blog/1184-contributing-guidelines">contributing</a> feature</li>
<li>Use the <a class="reference external" href="https://pages.github.com">gh-pages</a> feature to demo your project</li>
<li>Configure the main branch in the settings</li>
</ul>
</div>
Edit Django geometries fields with Leaflet2014-03-26T00:00:00+01:002014-03-26T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2014-03-26:/edit-django-geometries-fields-with-leaflet.html<p>After the <a class="reference external" href="https://blog.mathieu-leplatre.info/geodjango-maps-with-leaflet.html">previous article</a>
regarding Django, Leaflet and GeoJSON, I wanted to highlight
the simplicity of GeoDjango geometries creation and edition with <a class="reference external" href="https://github.com/makinacorpus/django-leaflet">django-leaflet</a>.</p>
<p>It relies on <a class="reference external" href="https://github.com/Leaflet/Leaflet.draw">Leaflet.draw</a> for user interactions.</p>
<img alt="" src="/images/leaflet-form-widget.png" />
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field"><th class="field-name">note:</th><td class="field-body">Until recently, it was restricted to Django 1.6+, but <a class="reference external" href="https://github.com/makinacorpus/django-leaflet/pull/63">Gaël contributed</a>
a backport that allows Django 1 …</td></tr></tbody></table><p>After the <a class="reference external" href="https://blog.mathieu-leplatre.info/geodjango-maps-with-leaflet.html">previous article</a>
regarding Django, Leaflet and GeoJSON, I wanted to highlight
the simplicity of GeoDjango geometries creation and edition with <a class="reference external" href="https://github.com/makinacorpus/django-leaflet">django-leaflet</a>.</p>
<p>It relies on <a class="reference external" href="https://github.com/Leaflet/Leaflet.draw">Leaflet.draw</a> for user interactions.</p>
<img alt="" src="/images/leaflet-form-widget.png" />
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field"><th class="field-name">note:</th><td class="field-body">Until recently, it was restricted to Django 1.6+, but <a class="reference external" href="https://github.com/makinacorpus/django-leaflet/pull/63">Gaël contributed</a>
a backport that allows Django 1.4.2+ users to have them too !</td>
</tr>
</tbody>
</table>
<div class="section" id="in-django-adminsite">
<h2>In Django Adminsite</h2>
<div class="section" id="given-a-geodjango-model-as-usual">
<h3>Given a GeoDjango model as usual</h3>
<div class="highlight"><pre><span></span><span class="c1"># models.py</span>
<span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">django.contrib.gis.db</span> <span class="kn">import</span> <span class="n">models</span> <span class="k">as</span> <span class="n">gismodels</span>
<span class="k">class</span> <span class="nc">MushroomSpot</span><span class="p">(</span><span class="n">gismodels</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">geom</span> <span class="o">=</span> <span class="n">gismodels</span><span class="o">.</span><span class="n">PolygonField</span><span class="p">()</span>
<span class="n">objects</span> <span class="o">=</span> <span class="n">gismodels</span><span class="o">.</span><span class="n">GeoManager</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">__unicode__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span>
</pre></div>
</div>
<div class="section" id="register-using-leaflet-geoadmin">
<h3>Register using Leaflet GeoAdmin</h3>
<p>It's as simple as :</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.contrib</span> <span class="kn">import</span> <span class="n">admin</span>
<span class="kn">from</span> <span class="nn">leaflet.admin</span> <span class="kn">import</span> <span class="n">LeafletGeoAdmin</span>
<span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">MushroomSpot</span>
<span class="n">admin</span><span class="o">.</span><span class="n">site</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">MushroomSpot</span><span class="p">,</span> <span class="n">LeafletGeoAdmin</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="in-forms-views">
<h2>In forms views</h2>
<div class="section" id="edition-view-usign-class-based-view">
<h3>Edition view usign Class-Based View</h3>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django</span> <span class="kn">import</span> <span class="n">forms</span>
<span class="kn">from</span> <span class="nn">django.views.generic</span> <span class="kn">import</span> <span class="n">UpdateView</span>
<span class="kn">from</span> <span class="nn">leaflet.forms.widgets</span> <span class="kn">import</span> <span class="n">LeafletWidget</span>
<span class="kn">from</span> <span class="nn">.models</span> <span class="kn">import</span> <span class="n">MushroomSpot</span>
<span class="k">class</span> <span class="nc">MushroomSpotForm</span><span class="p">(</span><span class="n">forms</span><span class="o">.</span><span class="n">ModelForm</span><span class="p">):</span>
<span class="k">class</span> <span class="nc">Meta</span><span class="p">:</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">MushroomSpot</span>
<span class="n">fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'geom'</span><span class="p">)</span>
<span class="n">widgets</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'geom'</span><span class="p">:</span> <span class="n">LeafletWidget</span><span class="p">()}</span>
<span class="k">class</span> <span class="nc">EditMushroomSpot</span><span class="p">(</span><span class="n">UpdateView</span><span class="p">):</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">MushroomSpot</span>
<span class="n">form_class</span> <span class="o">=</span> <span class="n">MushroomSpotForm</span>
<span class="n">template_name</span> <span class="o">=</span> <span class="s1">'form.html'</span>
</pre></div>
</div>
<div class="section" id="form-template-with-leaflet-tags">
<h3>Form template with Leaflet tags</h3>
<div class="highlight"><pre><span></span>{% load leaflet_tags %}
<span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
{% leaflet_js plugins="forms" %}
{% leaflet_css plugins="forms" %}
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">h1</span><span class="p">></span>Edit {{ object }}<span class="p"></</span><span class="nt">h1</span><span class="p">></span>
<span class="p"><</span><span class="nt">form</span> <span class="na">method</span><span class="o">=</span><span class="s">"POST"</span><span class="p">></span>
{{ form }}
{% csrf_token %}
<span class="p"><</span><span class="nt">input</span> <span class="na">type</span><span class="o">=</span><span class="s">"submit"</span><span class="p">/></span>
<span class="p"></</span><span class="nt">form</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
</div>
</div>
<div class="section" id="going-further">
<h2>Going further...</h2>
<p>The Django form widget has <a class="reference external" href="https://github.com/makinacorpus/django-leaflet/blob/master/leaflet/forms/widgets.py">a couple of options</a>, that can tweak some aspects of
the map (size, read-only, ...).</p>
<p>But some advanced usage might require specific interactions or behaviour, beyond
Django <a class="reference external" href="https://docs.djangoproject.com/en/1.6/ref/forms/fields/#creating-custom-fields">field</a> and <a class="reference external" href="https://docs.djangoproject.com/en/1.6/ref/forms/widgets/#base-widget-classes">widgets</a> customizations.</p>
<div class="section" id="custom-field-javascript-component">
<h3>Custom field JavaScript component</h3>
<p>The frontend field component behaviour and initialization is also pluggable, and
can be used to add extra controls, layers or whatever.</p>
<div class="highlight"><pre><span></span><span class="nx">Custom</span><span class="p">.</span><span class="nx">GeometryField</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">GeometryField</span><span class="p">.</span><span class="nx">extend</span><span class="p">({</span>
<span class="w"> </span><span class="nx">addTo</span><span class="o">:</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">map</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">GeometryField</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">addTo</span><span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="nx">map</span><span class="p">);</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">filecontrol</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">filecontrol</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">Control</span><span class="p">.</span><span class="nx">fileLayerLoad</span><span class="p">();</span>
<span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">addControl</span><span class="p">(</span><span class="nx">filecontrol</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">});</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">CustomLeafletWidget</span><span class="p">(</span><span class="n">LeafletWidget</span><span class="p">):</span>
<span class="n">geometry_field_class</span> <span class="o">=</span> <span class="s1">'Custom.GeometryField'</span>
</pre></div>
</div>
<div class="section" id="custom-de-serialization-of-form-field-value">
<h3>Custom de/serialization of form field value</h3>
<p>The Javascript component for de/serializing fields value is pluggable, can be used to override
the way the geometries are sent to the form.</p>
<div class="highlight"><pre><span></span><span class="nx">Custom</span><span class="p">.</span><span class="nx">FieldStore</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">FieldStore</span><span class="p">({</span>
<span class="w"> </span><span class="nx">save</span><span class="o">:</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">layer</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">this</span><span class="p">.</span><span class="nx">formfield</span><span class="p">.</span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="s2">"latlngs"</span><span class="o">:</span><span class="w"> </span><span class="nx">layer</span><span class="p">.</span><span class="nx">getLatLngs</span><span class="p">()};</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">});</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">CustomLeafletWidget</span><span class="p">(</span><span class="n">LeafletWidget</span><span class="p">):</span>
<span class="n">field_store_class</span> <span class="o">=</span> <span class="s1">'Custom.FieldStore'</span>
</pre></div>
</div>
</div>
<div class="section" id="help-us-improve-django-leaflet">
<h2>Help us improve django-leaflet !</h2>
<p>We built <em>django-leaflet</em> at <a class="reference external" href="http://makinacorpus.com">Makina Corpus</a>
for some our Webmapping projects. It is used in production and gives
us satisfaction in most use-cases.</p>
<p>If our initial design does not match your needs, please tell us what you
think !</p>
<p>For example, personnally, I would like to remove the <cite><script></cite> tag in the <a class="reference external" href="https://github.com/makinacorpus/django-leaflet/blob/0.13.0/leaflet/templates/leaflet/_leaflet_map.html">map template</a>, and pass configuration entries through the DOM instead...</p>
<p>...your turn !</p>
</div>
Releasing software ideas2014-02-17T21:25:00+01:002014-02-17T21:25:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2014-02-17:/releasing-software-ideas.html<p>A couple of weeks ago, I had a very interesting reading :
<a class="reference external" href="http://danielsolisblog.blogspot.fr/2013/07/what-if-someone-steals-your-idea.html">What if someone steals your idea ?</a> (translated into French <a class="reference external" href="http://www.framablog.org/index.php/post/2013/08/29/voler-votre-idee">by Framasoft</a>).</p>
<p>The author states that his boardgame concepts take a lot of value from development, time and efforts,
and very few from the original idea itself.</p>
<p>The parallel with …</p><p>A couple of weeks ago, I had a very interesting reading :
<a class="reference external" href="http://danielsolisblog.blogspot.fr/2013/07/what-if-someone-steals-your-idea.html">What if someone steals your idea ?</a> (translated into French <a class="reference external" href="http://www.framablog.org/index.php/post/2013/08/29/voler-votre-idee">by Framasoft</a>).</p>
<p>The author states that his boardgame concepts take a lot of value from development, time and efforts,
and very few from the original idea itself.</p>
<p>The parallel with software is quite obvious: we can still share our software ideas and concepts,
a lot of time is yet to be invested before they become a usable product, with an identity, a user
community...</p>
<p>Having the ambition to implement all ideas that come into our floroushing minds can be a serious curse!
A more reasonnable attitude would be to stay focused and share the ideas left over! Hence this post...</p>
<img alt="Overbooking, by Xavi Puigg" src="/images/overbooking.jpg" />
<div class="section" id="stay-focused">
<h2>Stay focused</h2>
<p>This year I would like to focus on two personal projects: <a class="reference external" href="http://subtivals.org">Subtivals</a> and
<a class="reference external" href="https://github.com/spiral-project/daybed">Daybed</a>. and I promise myself that I won't invest time on all scatterbrained software ideas that may pop around my head!</p>
<div class="section" id="subtivals-1">
<h3>Subtivals</h3>
<p><em>Subtivals</em> was born from an actual need : to project subtitles during film festivals. It has a very
small user community, so called a <em>niche</em>, but from all over the world !</p>
<p>We receive feedback, questions and orders quite often, which maintains a level
of motivation above the average :)</p>
<p>The code is clean, and the application very robust. I enjoy coding with Qt and C++!
Moreover most ideas can be crash-tested by real users on the field without too
much pressure :)</p>
</div>
<div class="section" id="daybed-1">
<h3>Daybed</h3>
<p><em>Daybed</em> is a lot more abstract and subject to competition: it's a Web API
that brings database as-a-service. You define your data models, and a REST API
comes into life automatically, for validation and storage.</p>
<p>It covers a very wide range of potential use-cases (like many <a class="reference external" href="http://en.wikipedia.org/wiki/Rapid_application_development">Rapid Application Development</a> tools by the way...), which
makes it hard to build its identity, and thus its community!</p>
<p>We made a great progress on the API lately, and there will be a Daybed 1.0 very soon!
But we have to provide a lot of <em>cool-stuff-ization</em> efforts, with small demos
and slick documentation.</p>
</div>
</div>
<div class="section" id="software-ideas-for-free">
<h2>Software ideas for free !</h2>
<p>After the reading of Daniel Solis' article, I thought it would be fun
to share some ideas and concepts that crossed my mind.</p>
<p>But when I read at Jeff Knupp that there were, on Earth, <a class="reference external" href="http://www.jeffknupp.com/blog/2014/01/28/need-a-project-idea-scratch-your-own-itch/">« people who can't think of a good project idea »</a>, I thought it was a serious waste to keep them for me!</p>
<img alt="Share your ideas, by nan palmero" src="/images/share_ideas.jpg" />
<p>The following ideas were collected on my notepad along the previous months. Some of them are
quite old, some them may be really bad, others quite awesome, some of them
are easy to implement, others aren't at all.</p>
<ul class="simple">
<li>A social directory of community-supported farms, where users can compare
deliveries, by season, by region, with pictures and rating ;</li>
<li>A portal for community-supported farms to organize their production and
relationships with members ;</li>
<li>An open database of relationships between politicians and corporations ;</li>
<li>A map of bars and restaurants with sunny terraces, by hour, by city ;</li>
<li>A map of places where bikes were stolen ;</li>
<li>A tool to compute dates of planet alignments ;</li>
<li>A simple, pop-up-less, map bounding box and coordinates converter tool ;</li>
<li>A tool to find the best place to meet for a group of friends at different locations,
so that each of them has the same (walking|driving|flight) distance <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> ;</li>
<li>A self-hosted application to manage wishlists, with ability to
secretly mark stuff as picked or organize shared budget <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> ;</li>
<li><span class="strike">An equivalent of Pelican for photos galleries</span> <a class="reference external" href="http://sigal.saimon.org/">Sigal, by Saimon</a> !
It takes a folder with pictures, and generates a static page with <a class="reference external" href="http://galleria.io/">galleria.js</a> ;</li>
<li><span class="strike">Manual activities search by "ingredient"</span> <a class="reference external" href="http://instructables.com">instructables.com</a> <a class="footnote-reference" href="#footnote-3" id="footnote-reference-3">[3]</a> :
I have some clips, a plastic bottle and a match box, tell me what I can build with that for my kids ;</li>
<li>A directory of pictures of food factories buildings ;</li>
<li>A collaborative interactive timeline with maps to visualize movement, spread, empires... ;</li>
</ul>
<p><strong>Digital cinema</strong></p>
<ul class="simple">
<li>A GUI to edit DCP files, edit projection dates, add subtitles etc. I began to write <a class="reference external" href="https://docs.google.com/document/d/1FVUw70wpLwOp8xj6Uok8WAWah4V1KxYxan7OKHHGPUk/edit?usp=sharing">some specs for this</a>;</li>
<li>A GUI to easily generate KDM certificates, I also <a class="reference external" href="https://docs.google.com/document/d/1XVqpMmwwGuGaCmN_odJmRHihr4aEuwhbXe0r_7D7eEI/edit?usp=sharing">wrote some specs</a>.</li>
</ul>
<p><strong>Tech stuff</strong></p>
<ul class="simple">
<li>GeoJSON model fields in Django in order to get rid of GIS stack for simple stuff ;</li>
<li>A SVG template engine, with specific tags for tables, vertical distributions, sub-templates...
(<em>I made one in PHP almost ten years ago...</em>) ;</li>
<li>System integration monitoring : draw your instances and their relationships, add stupid
sensors on them that keep you informed of broken links or streams ;</li>
<li>Django SQL template, in order to load SQL commands with table and field names substituted
from your models ;</li>
<li>A Web API on PostGIS for GeoJSON "data clips" : it lists all tables and views with geometries,
and let you obtain GeoJSON using simple query parameters ;</li>
<li>A Leaflet plugin that takes a trajectory and a speed, and fires the ongoing position at a determined
frequency ;</li>
</ul>
<p><strong>Daybed apps</strong></p>
<p>Some stuff that can easily be built with <em>Daybed</em> :</p>
<ul class="simple">
<li>A Web forms service (<em>Google Forms</em>) ;</li>
<li>A remote storage for JavaScript or mobile applications ;</li>
<li>A lightweight alternative to <em>FormHub</em>, <em>ODK aggregate</em> or <em>Enketo</em> ;</li>
<li>A JavaScript CMS where users can define their content types and forms ;</li>
<li>A mobile application builder ;</li>
<li>A data wiki or pad ;</li>
<li>...</li>
<li>Daybed mobile app in Qt, equivalent of <em>ODKCollect</em> for Daybed ;</li>
<li>Daybed plugged into <em>geojson.io</em>, instead of using Github gist for storage ;</li>
</ul>
<div class="section" id="your-turn">
<h3>Your turn...</h3>
<p>Now that I shared them, they are not mine anymore, take them if you like!</p>
<p>Many things can happen now :</p>
<ul class="simple">
<li>You will let them rot in the forgotten Web ;</li>
<li>You will show me how bad they were ;</li>
<li>You will point out existing projects ;</li>
<li>You will ask me details ;</li>
<li>You will implement an idea and share the code ;</li>
<li>You will start a company, raise money and build an empire (<em>I wouldn't have anyway</em>) ;</li>
<li>You will sue me for <a class="reference external" href="http://lanyrd.com/2012/dotjs/scbgdz/">your new puppy to have ruined your life</a> :)</li>
<li><strong>Update</strong>: You will propose <a class="reference external" href="https://twitter.com/daks_/status/435893872514695168">more ideas</a> on twitter :)</li>
</ul>
<p>Meanwhile, they'll disappear from my notebook and its underlying <em>todo list</em>,
they won't haunt me anymore, I will be at peace.</p>
<img alt="Kid schedulem by Carissa GoodNCrazy" src="/images/kid_schedule.jpg" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Not the UI of the year but <a class="reference external" href="http://www.mezzoman.com">http://www.mezzoman.com</a> looks like a candidate</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>For example, like <a class="reference external" href="http://mygiftslist.be">http://mygiftslist.be</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-3">[3]</a></td><td>Merci <a class="reference external" href="https://twitter.com/simongeorges/status/435807135302053889">Simon</a></td></tr>
</tbody>
</table>
</div>
</div>
Publish your Pelican blog on Github pages via Travis-CI2014-01-09T12:25:00+01:002014-01-09T12:25:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2014-01-09:/publish-your-pelican-blog-on-github-pages-via-travis-ci.html<p>This blog is powered by Pelican, and until recently I have rendered the HTML pages
on my local machine and published them on a personal server via SSH.</p>
<p>Most of the time, I used to forget pushing the raw files on Github. Therefore automatizing
the publishing process based on commits …</p><p>This blog is powered by Pelican, and until recently I have rendered the HTML pages
on my local machine and published them on a personal server via SSH.</p>
<p>Most of the time, I used to forget pushing the raw files on Github. Therefore automatizing
the publishing process based on commits looked like a good idea :)</p>
<p>The strategy is :</p>
<ul class="simple">
<li>I write, commit and push articles in rST as usual ;</li>
<li>Travis-CI detects changes ;</li>
<li>It builds the HTML pages ;</li>
<li>It pushes the content on the repo's <tt class="docutils literal"><span class="pre">gh-pages</span></tt> branch.</li>
</ul>
<p>Andrea Zonca <a class="reference external" href="http://zonca.github.io/2013/09/automatically-build-pelican-and-publish-to-github-pages.html">wrote something similar</a>
with a very appealing title, but it was not very enlightening to me.
I hope getting it done will be clear enough with this present article.</p>
<div class="section" id="travis-ci-configuration">
<h2>Travis-CI configuration</h2>
<p>Just add a <tt class="docutils literal">.travis.yml</tt> file, like you would do for your unit-tests.</p>
<div class="highlight"><pre><span></span>language: python
branches:
only:
- master
install:
- pip install pelican ghp-import
script:
- make publish github
</pre></div>
<p>In order to avoid the blog to be updated by changes on the <tt class="docutils literal">draft</tt> branch,
only commits of <tt class="docutils literal">master</tt> trigger updates.</p>
</div>
<div class="section" id="travis-authentication-on-github">
<h2>Travis authentication on Github</h2>
<p>Since it will push to the <tt class="docutils literal"><span class="pre">gh-pages</span></tt> branch of your repo, it has to authenticate on
Github. For this, we use a <a class="reference external" href="https://github.com/blog/1270-easier-builds-and-deployments-using-git-over-https-and-oauth">OAuth token</a>,
that can be created and revoked from the <a class="reference external" href="https://github.com/settings/applications">GitHub applications</a> page.</p>
<p>In order to keep it secret with Travis, we use their Ruby application to
encrypt it :</p>
<pre class="literal-block">
sudo apt-get install ruby1.9.1-dev build-essential
sudo gem install travis
travis encrypt GH_TOKEN=your_token
</pre>
<p>A new block will be added to <tt class="docutils literal">.travis.yml</tt>.</p>
<div class="highlight"><pre><span></span>env:
global:
secure: NWjh6sCvmjuX...yWo=
</pre></div>
</div>
<div class="section" id="push-on-github-pages">
<h2>Push on Github pages</h2>
<p>I modified the <tt class="docutils literal">Makefile</tt> provided in Pelican.</p>
<p>It uses <em>ghp-import</em> to build the branch from the output folder and pushes quietly with force via HTTPS using the token variable.</p>
<div class="highlight"><pre><span></span><span class="nf">github</span><span class="o">:</span><span class="w"> </span><span class="n">publish</span>
<span class="w"> </span>ghp-import<span class="w"> </span>-n<span class="w"> </span><span class="k">$(</span>OUTPUTDIR<span class="k">)</span>
<span class="w"> </span>@git<span class="w"> </span>push<span class="w"> </span>-fq<span class="w"> </span>https://<span class="si">${</span><span class="nv">GH_TOKEN</span><span class="si">}</span>@github.com/<span class="k">$(</span>TRAVIS_REPO_SLUG<span class="k">)</span>.git<span class="w"> </span>gh-pages<span class="w"> </span>><span class="w"> </span>/dev/null
</pre></div>
<p>Use leading <tt class="docutils literal">@</tt> to remove command from output, thanks <a class="reference external" href="https://github.com/leplatrem/blog.mathieu-leplatre.info/issues/1">Ryan Peck</a>!</p>
<p>Also disable pull request builds in Travis to prevent the blog being updated by a pull request (thanks <a class="reference external" href="https://github.com/leplatrem/blog.mathieu-leplatre.info/pull/2">Andrew Aitken</a> !).</p>
<img alt="" src="/images/travis-pull-request.png" />
</div>
<div class="section" id="custom-domain">
<h2>Custom domain</h2>
<p>Github expects a <tt class="docutils literal">CNAME</tt> file. Pelican provides a simple way to create it using <em>extra paths</em>,
controlled via settings :</p>
<div class="highlight"><pre><span></span><span class="n">STATIC_PATHS</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'images'</span><span class="p">,</span> <span class="s1">'extra/CNAME'</span><span class="p">]</span>
<span class="n">EXTRA_PATH_METADATA</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'extra/CNAME'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'path'</span><span class="p">:</span> <span class="s1">'CNAME'</span><span class="p">},}</span>
</pre></div>
<p>If any doubt, just have a look the <a class="reference external" href="https://github.com/leplatrem/blog.mathieu-leplatre.info">repository of this blog</a>...</p>
</div>
Django : Create a QuerySet from a list, preserving order2013-11-08T00:00:00+01:002013-11-08T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-11-08:/django-create-a-queryset-from-a-list-preserving-order.html<p>I thought it would be an easy one, but found myself lost with 34 opened tabs
on <em>stackoverflow</em>...</p>
<div class="section" id="the-problem-keep-it-ordered">
<h2>The problem : keep it ordered</h2>
<p>Usually, obtaining a <tt class="docutils literal">QuerySet</tt> from a list is quite simple :</p>
<div class="highlight"><pre><span></span>>>><span class="w"> </span><span class="nv">queryset</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>Theme.objects.filter<span class="o">(</span><span class="nv">pk__in</span><span class="o">=[</span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">10</span><span class="o">])</span>
>>><span class="w"> </span>type<span class="o">(</span>queryset<span class="o">)</span>
<class<span class="w"> </span><span class="s1">'django.db.models.query.QuerySet'</span>>
>>><span class="w"> </span>queryset …</pre></div></div><p>I thought it would be an easy one, but found myself lost with 34 opened tabs
on <em>stackoverflow</em>...</p>
<div class="section" id="the-problem-keep-it-ordered">
<h2>The problem : keep it ordered</h2>
<p>Usually, obtaining a <tt class="docutils literal">QuerySet</tt> from a list is quite simple :</p>
<div class="highlight"><pre><span></span>>>><span class="w"> </span><span class="nv">queryset</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>Theme.objects.filter<span class="o">(</span><span class="nv">pk__in</span><span class="o">=[</span><span class="m">1</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">10</span><span class="o">])</span>
>>><span class="w"> </span>type<span class="o">(</span>queryset<span class="o">)</span>
<class<span class="w"> </span><span class="s1">'django.db.models.query.QuerySet'</span>>
>>><span class="w"> </span>queryset
<span class="o">[</span><Theme:<span class="w"> </span>Fauna>,<span class="w"> </span><Theme:<span class="w"> </span>Flora>,<span class="w"> </span><Theme:<span class="w"> </span>Refuge><span class="o">]</span>
</pre></div>
<p>The problem is that the list order is ignored :</p>
<div class="highlight"><pre><span></span>>>><span class="w"> </span>Theme.objects.filter<span class="o">(</span><span class="nv">pk__in</span><span class="o">=[</span><span class="m">10</span>,<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="m">1</span><span class="o">])</span>
<span class="o">[</span><Theme:<span class="w"> </span>Fauna>,<span class="w"> </span><Theme:<span class="w"> </span>Flora>,<span class="w"> </span><Theme:<span class="w"> </span>Refuge><span class="o">]</span>
</pre></div>
<p>If obtaining a <tt class="docutils literal">QuerySet</tt> is not a requirement, it's <a class="reference external" href="http://stackoverflow.com/a/7361598/141895">rather easy to get a list sorted</a>
according to another :</p>
<div class="highlight"><pre><span></span><span class="n">pks_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">10</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">themes</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">Theme</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">pk__in</span><span class="o">=</span><span class="n">pks_list</span><span class="p">))</span>
<span class="n">themes</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">t</span><span class="p">:</span> <span class="n">pks_list</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">t</span><span class="o">.</span><span class="n">pk</span><span class="p">))</span>
</pre></div>
<p>In my case, I want a <tt class="docutils literal">QuerySet</tt>, a brave lazy one, with proper <tt class="docutils literal">filter()</tt>,
<tt class="docutils literal">exclude()</tt>, <tt class="docutils literal">values()</tt> ...</p>
</div>
<div class="section" id="fallback-to-sql">
<h2>Fallback to SQL</h2>
<p>AFAIK, most database engines ignore order of records, until you specify an
ordering column. In our case, the list is arbitrary, and does not map to any
existing attribute, thus db column.</p>
<p>If you use MySQL (<em>who does?!</em>), there is a <tt class="docutils literal">FIELD()</tt> function that provides
<a class="reference external" href="http://stackoverflow.com/a/3626200/141895">custom input for the sort method</a> :</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">theme</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">FIELD</span><span class="p">(</span><span class="o">`</span><span class="n">id</span><span class="o">`</span><span class="p">,</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">);</span>
</pre></div>
<p>Using the ORM, it gives us (<em>thanks Daniel Roseman</em>)</p>
<div class="highlight"><pre><span></span><span class="n">pk_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">10</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">ordering</span> <span class="o">=</span> <span class="s1">'FIELD(`id`, </span><span class="si">%s</span><span class="s1">)'</span> <span class="o">%</span> <span class="s1">','</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span> <span class="k">for</span> <span class="nb">id</span> <span class="ow">in</span> <span class="n">pk_list</span><span class="p">)</span>
<span class="n">queryset</span> <span class="o">=</span> <span class="n">Theme</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">pk__in</span><span class="o">=</span><span class="p">[</span><span class="n">pk_list</span><span class="p">])</span><span class="o">.</span><span class="n">extra</span><span class="p">(</span>
<span class="n">select</span><span class="o">=</span><span class="p">{</span><span class="s1">'ordering'</span><span class="p">:</span> <span class="n">ordering</span><span class="p">},</span> <span class="n">order_by</span><span class="o">=</span><span class="p">(</span><span class="s1">'ordering'</span><span class="p">,))</span>
</pre></div>
<p>Well, good news it can be <a class="reference external" href="http://stackoverflow.com/questions/1309624/simulating-mysqls-order-by-field-in-postgresql">ported to PostgreSQL</a>.
But if possible, I would prefer native SQL.</p>
<p>And it looks like the magnificient syntax of SQL provides <tt class="docutils literal">ORDER BY CASE WHEN ... END</tt> !</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">theme</span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span>
<span class="w"> </span><span class="k">CASE</span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">id</span><span class="o">=</span><span class="mi">10</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="mi">0</span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">id</span><span class="o">=</span><span class="mi">2</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">id</span><span class="o">=</span><span class="mi">1</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="mi">2</span>
<span class="w"> </span><span class="k">END</span><span class="p">;</span>
</pre></div>
<p>Using the ORM, it gives us :</p>
<div class="highlight"><pre><span></span><span class="n">pk_list</span> <span class="o">=</span> <span class="p">[</span><span class="mi">10</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">clauses</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="s1">'WHEN id=</span><span class="si">%s</span><span class="s1"> THEN </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">pk</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">pk</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">pk_list</span><span class="p">)])</span>
<span class="n">ordering</span> <span class="o">=</span> <span class="s1">'CASE </span><span class="si">%s</span><span class="s1"> END'</span> <span class="o">%</span> <span class="n">clauses</span>
<span class="n">queryset</span> <span class="o">=</span> <span class="n">Theme</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">pk__in</span><span class="o">=</span><span class="n">pk_list</span><span class="p">)</span><span class="o">.</span><span class="n">extra</span><span class="p">(</span>
<span class="n">select</span><span class="o">=</span><span class="p">{</span><span class="s1">'ordering'</span><span class="p">:</span> <span class="n">ordering</span><span class="p">},</span> <span class="n">order_by</span><span class="o">=</span><span class="p">(</span><span class="s1">'ordering'</span><span class="p">,))</span>
</pre></div>
<p>I wonder how it behaves with zillions of records though ;)</p>
<p>One more thing: before Django 1.6, there <a class="reference external" href="https://code.djangoproject.com/ticket/14930">was a bug</a> with calling <tt class="docutils literal">values_list()</tt>
on a queryset ordered by an extra column. Use this :</p>
<div class="highlight"><pre><span></span><span class="n">values</span> <span class="o">=</span> <span class="n">queryset</span><span class="o">.</span><span class="n">values</span><span class="p">(</span><span class="s1">'ordering'</span><span class="p">,</span> <span class="s1">'label'</span><span class="p">)</span>
<span class="n">labels</span> <span class="o">=</span> <span class="p">[</span><span class="n">value</span><span class="p">[</span><span class="s1">'label'</span><span class="p">]</span> <span class="k">for</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">values</span><span class="p">]</span>
</pre></div>
<p>Good luck ! Please share your advices or critics ;)</p>
</div>
Generate office documents with Django2013-09-13T00:00:00+02:002013-09-13T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-09-13:/generate-office-documents-with-django.html<p>In our recent Django projects, we had to create documents (Libre/OpenOffice, Microsoft Office, PDF...),
and therefore created two components :</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/makinacorpus/django-appypod">django-appypod</a>, providing document templates views, built on top of <a class="reference external" href="http://appyframework.org/pod.html">Appy.pod</a> ;</li>
<li><a class="reference external" href="https://github.com/makinacorpus/convertit">convertit</a>, a generic format conversion Web API.</li>
</ul>
<div class="section" id="django-appypod-1">
<h2>django-appypod</h2>
<p><em>Appy</em> is a set of python tools (e.g. framework) by …</p></div><p>In our recent Django projects, we had to create documents (Libre/OpenOffice, Microsoft Office, PDF...),
and therefore created two components :</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/makinacorpus/django-appypod">django-appypod</a>, providing document templates views, built on top of <a class="reference external" href="http://appyframework.org/pod.html">Appy.pod</a> ;</li>
<li><a class="reference external" href="https://github.com/makinacorpus/convertit">convertit</a>, a generic format conversion Web API.</li>
</ul>
<div class="section" id="django-appypod-1">
<h2>django-appypod</h2>
<p><em>Appy</em> is a set of python tools (e.g. framework) by <a class="reference external" href="https://launchpad.net/~gaetan-delannay">Gaetan Delannay</a>, which provides, among other stuff,
a templating engine for OpenDocument files.</p>
<p>One the great advantage is that you edit your templates in LibreOffice, <a class="reference external" href="http://en.wikipedia.org/wiki/WYSIWYG">WYSIWYG</a> !</p>
<img alt="" class="align-center" src="/images/appypod-template.png" style="width: 70%;" />
<p><em>django-appypod</em> is a template view that renders a OpenDocument template for a context.
The exact same way you already do for HTML.</p>
<p>Using class-based generic views :</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.view.generic.detail</span> <span class="kn">import</span> <span class="n">TemplateView</span>
<span class="kn">from</span> <span class="nn">djappypod.response</span> <span class="kn">import</span> <span class="n">OdtTemplateResponse</span>
<span class="k">class</span> <span class="nc">YourDocument</span><span class="p">(</span><span class="n">TemplateView</span><span class="p">):</span>
<span class="n">response_class</span> <span class="o">=</span> <span class="n">OdtTemplateResponse</span>
<span class="n">template_name</span> <span class="o">=</span> <span class="s2">"your/template.odt"</span>
<span class="k">def</span> <span class="nf">get_context_data</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">kwargs</span><span class="p">[</span><span class="s1">'title'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'Simple as hello ;)'</span>
<span class="k">return</span> <span class="n">kwargs</span>
</pre></div>
<p>Using classic functions-based views :</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">your_view</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">context</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'title'</span><span class="p">:</span> <span class="s1">'Simple as hello ;)'</span>
<span class="p">}</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">OdtTemplateResponse</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="s2">"your/template.odt"</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
<span class="n">response</span><span class="o">.</span><span class="n">render</span><span class="p">()</span>
<span class="k">return</span> <span class="n">response</span>
</pre></div>
<img alt="" class="align-center" src="/images/appypod-rendered.png" style="width: 70%;" />
</div>
<div class="section" id="convertit-1">
<h2>ConvertIt</h2>
<p>We often need to serve those document as PDF files, and some users can't
be satisfied with <em>OpenDocument</em> files.</p>
<p><em>Appy</em> can rely on OpenOffice to convert documents to PDF and MS-Word, but we didn't like
the idea of having to install the bunch of binaries along every Django project.
Therefore we created <em>ConvertIt</em>, a Web API that will just be in charge of
format conversion. It can live on a dedicated server, and thus isolate binaries, and
potentially convert from and to any exotic formats, relying on any exotic system binaries.</p>
<p>So far we implemented most office documents conversions (.pdf, .doc, .xls), as well as SVG to PDF and PNG.</p>
<div class="section" id="docker-image">
<h3>Docker image</h3>
<p>If you use Docker, you can get a ConvertIt instance running in one command :</p>
<pre class="literal-block">
sudo docker run -p :6543 makinacorpus/convertit
</pre>
</div>
<div class="section" id="manual-installation">
<h3>Manual installation</h3>
<p>It is a Pyramid project, pretty straightforward :</p>
<div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>convertit
</pre></div>
<p>Plus some conversion binaries (each one is optional):</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>libreoffice<span class="w"> </span>unoconv<span class="w"> </span>inkscape
</pre></div>
<p>To run a development instance :</p>
<div class="highlight"><pre><span></span>pserve<span class="w"> </span>development.ini<span class="w"> </span>--reload
</pre></div>
<p>To run a production instance :</p>
<div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>gunicorn
gunicorn<span class="w"> </span>--paste<span class="w"> </span>production.ini
</pre></div>
</div>
<div class="section" id="usage">
<h3>Usage</h3>
<p>Using GET requests :</p>
<pre class="literal-block">
curl http://convertit/?url=http://server/document.odt&to=application/pdf
HTTP/1.1 302 Found
Content-Disposition: attachement; filename=document.pdf
...
</pre>
<p>Uploading file with POST :</p>
<pre class="literal-block">
curl -F "file=@tiger.svg" http://convertit/?to=image/png
HTTP/1.1 302 Found
Content-Disposition: attachement; filename=tiger.png
...
</pre>
</div>
<div class="section" id="integration-with-django">
<h3>Integration with Django</h3>
<p>If your documents do not require login, a simple and stupid template tag can do it :</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">django</span> <span class="kn">import</span> <span class="n">template</span>
<span class="kn">from</span> <span class="nn">django.core.urlresolvers</span> <span class="kn">import</span> <span class="n">reverse</span><span class="p">,</span> <span class="n">NoReverseMatch</span>
<span class="n">register</span> <span class="o">=</span> <span class="n">template</span><span class="o">.</span><span class="n">Library</span><span class="p">()</span>
<span class="nd">@register</span><span class="o">.</span><span class="n">simple_tag</span>
<span class="k">def</span> <span class="nf">convert_url</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">sourceurl</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="nb">format</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'format'</span><span class="p">,</span> <span class="s1">'application/pdf'</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">sourceurl</span> <span class="o">=</span> <span class="n">reverse</span><span class="p">(</span><span class="n">sourceurl</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">except</span> <span class="n">NoReverseMatch</span><span class="p">:</span>
<span class="k">pass</span>
<span class="n">fullurl</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">build_absolute_uri</span><span class="p">(</span><span class="n">sourceurl</span><span class="p">)</span>
<span class="k">return</span> <span class="s2">"</span><span class="si">%s</span><span class="s2">?url=</span><span class="si">%s</span><span class="s2">&to=</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">settings</span><span class="o">.</span><span class="n">CONVERSION_SERVER</span><span class="p">,</span>
<span class="n">urllib</span><span class="o">.</span><span class="n">quote</span><span class="p">(</span><span class="n">fullurl</span><span class="p">),</span>
<span class="n">urllib</span><span class="o">.</span><span class="n">quote</span><span class="p">(</span><span class="nb">format</span><span class="p">))</span>
</pre></div>
<p>Which you then use in templates:</p>
<pre class="literal-block">
<a href="{% convert_url "app:document" object.pk %}">Download PDF version</a>
</pre>
<p>However, if your view requires authentication, there are several strategies:</p>
<ul class="simple">
<li>Auto-login requests coming from ConvertIt server ;</li>
<li>Add a login required proxy view that download the file and perform a POST query to ConvertIt ;</li>
<li>Setup SSO or any other token mechanism ;</li>
<li>Contribute to ConvertIt to add HTTP authentication (<tt class="docutils literal"><span class="pre">url=http://user:pass@host</span></tt>) ;</li>
</ul>
<p><a class="reference external" href="https://gist.github.com/leplatrem/6552003">I made a snippet</a> for the first option</p>
</div>
</div>
<div class="section" id="in-short">
<h2>In short...</h2>
<ul class="simple">
<li><em>django-appypod</em> is great because templates are WYSIWYG ;</li>
<li><em>ConvertIt</em> is great because it's generic and pluggable ;</li>
<li>There are great together because their deliver both office and PDF formats ;</li>
</ul>
<p>There are alternatives though if PDF is enough for you :</p>
<ul class="simple">
<li><a class="reference external" href="http://weasyprint.org">WeasyPrint</a></li>
<li><a class="reference external" href="http://www.xhtml2pdf.com/">xhtml2pdf</a></li>
</ul>
</div>
GeoDjango maps with Leaflet2013-08-21T00:00:00+02:002013-08-21T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-08-21:/geodjango-maps-with-leaflet.html<p>A short introduction to web mapping with Django, using two very simple
applications: <a class="reference external" href="https://github.com/makinacorpus/django-leaflet">django-leaflet</a> and <a class="reference external" href="https://github.com/makinacorpus/django-geojson">django-geojson</a>, by <a class="reference external" href="http://makinacorpus.com">Makina Corpus</a>.</p>
<p>We will build a map with all major weather stations of the world.</p>
<div class="section" id="weather-stations">
<h2>Weather stations</h2>
<p>Each weather station has an id, a name and a position.</p>
<p>As a GeoDjango model, it …</p></div><p>A short introduction to web mapping with Django, using two very simple
applications: <a class="reference external" href="https://github.com/makinacorpus/django-leaflet">django-leaflet</a> and <a class="reference external" href="https://github.com/makinacorpus/django-geojson">django-geojson</a>, by <a class="reference external" href="http://makinacorpus.com">Makina Corpus</a>.</p>
<p>We will build a map with all major weather stations of the world.</p>
<div class="section" id="weather-stations">
<h2>Weather stations</h2>
<p>Each weather station has an id, a name and a position.</p>
<p>As a GeoDjango model, it becomes :</p>
<div class="highlight"><pre><span></span><span class="c1"># models.py</span>
<span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">django.contrib.gis.db</span> <span class="kn">import</span> <span class="n">models</span> <span class="k">as</span> <span class="n">gismodels</span>
<span class="k">class</span> <span class="nc">WeatherStation</span><span class="p">(</span><span class="n">gismodels</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">wmoid</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">IntegerField</span><span class="p">(</span><span class="n">primary_key</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">256</span><span class="p">)</span>
<span class="n">geom</span> <span class="o">=</span> <span class="n">gismodels</span><span class="o">.</span><span class="n">PointField</span><span class="p">()</span>
<span class="n">objects</span> <span class="o">=</span> <span class="n">gismodels</span><span class="o">.</span><span class="n">GeoManager</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">__unicode__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span>
</pre></div>
<div class="section" id="loading-actual-data">
<h3>Loading actual data</h3>
<p>The <a class="reference external" href="http://www.wmo.int">World Meteorological Organization</a> publishes a list of all major weather stations, in a <a class="reference external" href="ftp://ftp.wmo.ch/wmo-ddbs/VolA_New/">CSV format</a>.</p>
<p>Unfortunately, this format is not very friendly (especially latitudes and longitudes) :</p>
<pre class="literal-block">
StationId StationName Latitude Longitude ...
60351 JIJEL- ACHOUAT 36 48 00N 05 53 00E
...
07630 TOULOUSE BLAGNAC 43 37 16N 01 22 44E
...
</pre>
<p>We will convert coordinates <a class="reference external" href="http://en.wikipedia.org/wiki/Geographic_coordinate_conversion#Conversion_from_DMS_to_Decimal_Degree">from degres minutes seconds to decimal degrees</a>:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">dms2dec</span><span class="p">(</span><span class="n">value</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Degres Minutes Seconds to Decimal degres</span>
<span class="sd"> """</span>
<span class="n">degres</span><span class="p">,</span> <span class="n">minutes</span><span class="p">,</span> <span class="n">seconds</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
<span class="n">seconds</span><span class="p">,</span> <span class="n">direction</span> <span class="o">=</span> <span class="n">seconds</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">seconds</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">dec</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">degres</span><span class="p">)</span> <span class="o">+</span> <span class="nb">float</span><span class="p">(</span><span class="n">minutes</span><span class="p">)</span><span class="o">/</span><span class="mi">60</span> <span class="o">+</span> <span class="nb">float</span><span class="p">(</span><span class="n">seconds</span><span class="p">)</span><span class="o">/</span><span class="mi">3600</span>
<span class="k">if</span> <span class="n">direction</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'S'</span><span class="p">,</span> <span class="s1">'W'</span><span class="p">):</span>
<span class="k">return</span> <span class="o">-</span><span class="n">dec</span>
<span class="k">return</span> <span class="n">dec</span>
</pre></div>
<p>And create an instance of our model for each entry in the CSV :</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">csv</span>
<span class="kn">from</span> <span class="nn">django.contrib.gis.geos</span> <span class="kn">import</span> <span class="n">Point</span>
<span class="kn">from</span> <span class="nn">webmap.models</span> <span class="kn">import</span> <span class="n">WeatherStation</span>
<span class="n">csv_file</span> <span class="o">=</span> <span class="s1">'Pub9volA130819x.flatfile.txt'</span>
<span class="n">reader</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">DictReader</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="n">csv_file</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">),</span> <span class="n">delimiter</span><span class="o">=</span><span class="s2">"</span><span class="se">\t</span><span class="s2">"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">reader</span><span class="p">:</span>
<span class="n">lng</span> <span class="o">=</span> <span class="n">dms2dec</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'Longitude'</span><span class="p">))</span>
<span class="n">lat</span> <span class="o">=</span> <span class="n">dms2dec</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'Latitude'</span><span class="p">))</span>
<span class="n">wmoid</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'StationId'</span><span class="p">))</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s1">'StationName'</span><span class="p">)</span><span class="o">.</span><span class="n">title</span><span class="p">()</span>
<span class="n">WeatherStation</span><span class="p">(</span><span class="n">wmoid</span><span class="o">=</span><span class="n">wmoid</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="n">geom</span><span class="o">=</span><span class="n">Point</span><span class="p">(</span><span class="n">lng</span><span class="p">,</span> <span class="n">lat</span><span class="p">))</span><span class="o">.</span><span class="n">save</span><span class="p">()</span>
</pre></div>
<p>Now, our table is full of records (~ 12000) !</p>
<p>If you open it with graphical tools like QGis, it's stuffed !</p>
<img alt="" class="align-center" src="/images/weather-stations-qgis.png" style="width: 100%;" />
<p>( <em>If you want a script that converts this stations file into GeoJSON</em>, you can use <a class="reference external" href="https://gist.github.com/leplatrem/6294314">this piece of code</a>)</p>
</div>
</div>
<div class="section" id="plot-on-map">
<h2>Plot on map</h2>
<p>With <em>django-leaflet</em>, after having added <tt class="docutils literal">leaflet</tt> to your <tt class="docutils literal">INSTALLED_APPS</tt>,
you can insert maps in templates :</p>
<div class="highlight"><pre><span></span>{% load leaflet_tags %}
<span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
{% leaflet_js %}
{% leaflet_css %}
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">h1</span><span class="p">></span>Weather Stations<span class="p"></</span><span class="nt">h1</span><span class="p">></span>
{% leaflet_map "main" callback="main_map_init" %}
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span><span class="p">></span>
<span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">main_map_init</span><span class="w"> </span><span class="p">(</span><span class="nx">map</span><span class="p">,</span><span class="w"> </span><span class="nx">options</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Use Leaflet API here</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
<p>A blank map shows up, with a basic OpenStreetMap background.</p>
<div class="section" id="vectorial-data">
<h3>Vectorial data</h3>
<p>We now want to place markers for each weather station. For this, we use
<em>django-geojson</em>, which provides a very simple base view :</p>
<div class="highlight"><pre><span></span><span class="c1"># urls.py</span>
<span class="kn">from</span> <span class="nn">djgeojson.views</span> <span class="kn">import</span> <span class="n">GeoJSONLayerView</span>
<span class="kn">from</span> <span class="nn">webmap.models</span> <span class="kn">import</span> <span class="n">WeatherStation</span>
<span class="n">urlpatterns</span> <span class="o">=</span> <span class="n">patterns</span><span class="p">(</span><span class="s1">''</span><span class="p">,</span>
<span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^data.geojson$'</span><span class="p">,</span> <span class="n">GeoJSONLayerView</span><span class="o">.</span><span class="n">as_view</span><span class="p">(</span><span class="n">model</span><span class="o">=</span><span class="n">WeatherStation</span><span class="p">),</span> <span class="n">name</span><span class="o">=</span><span class="s1">'data'</span><span class="p">)</span>
<span class="p">)</span>
</pre></div>
<p>We load this data in Ajax, and add it as map layer, in the initialization function left empty in the above snippet :</p>
<div class="highlight"><pre><span></span><span class="kd">function</span><span class="w"> </span><span class="nx">main_map_init</span><span class="w"> </span><span class="p">(</span><span class="nx">map</span><span class="p">,</span><span class="w"> </span><span class="nx">options</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">dataurl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'{% url "data" %}'</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// Download GeoJSON via Ajax</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">getJSON</span><span class="p">(</span><span class="nx">dataurl</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Add GeoJSON layer</span>
<span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">geoJson</span><span class="p">(</span><span class="nx">data</span><span class="p">).</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span>
<span class="w"> </span><span class="p">});</span>
<span class="p">}</span>
</pre></div>
<p>The map shows up, and get filled with weather stations !</p>
<img alt="" class="align-center" src="/images/weather-stations-leaflet.png" style="width: 100%;" />
</div>
</div>
<div class="section" id="going-further">
<h2>Going further...</h2>
<p>This was a first introduction, but it applies to all kinds of goemetries (lines, polygons, ...).</p>
<p>I <a class="reference external" href="http://github.com/leplatrem/django-leaflet-geojson">published the full project</a> if you want to start from an example.</p>
<p>If you already feel comfortable with Django, there won't be any surprise : have a look at <em>Leaflet</em>, <em>django-leaflet</em> and <em>django-geojson</em> respective documentations in order to get an idea of the customizations you are offered...</p>
<div class="section" id="performance">
<h3>Performance</h3>
<p>A map with more than 12 000 HTML objects is not going to be snappy.</p>
<p>Hopefully, it won't be the case for your first applications !</p>
<p>And fortunately, there are plently of different strategies to draw such an amount of data :</p>
<ul class="simple">
<li>Use <a class="reference external" href="https://github.com/Leaflet/Leaflet.markercluster">marker clusters</a> to reduce the number of elements on the map (<a class="reference external" href="https://github.com/leplatrem/django-leaflet-geojson/blob/master/Pub9volA130819x.geojson">see result here</a>) ;</li>
<li>Draw circles instead of markers and switch to Canvas (see Leaflet documentation) ;</li>
<li>Use <a class="reference external" href="https://github.com/glenrobertson/leaflet-tilelayer-geojson/">tiled geojson</a> ;</li>
<li>Render tiles using Tilemill/Mapnik ;</li>
<li>...</li>
</ul>
<p>It gives us a lot of topics to explore and blog about :)</p>
</div>
</div>
Deploy Django projects using git push2013-08-12T22:00:00+02:002013-08-12T22:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-08-12:/deploy-django-projects-using-git-push.html<p>Deploying stuff in one command is becoming the Holy Grail of development,
with currently ten times more blog articles than Medieval crusades :)</p>
<p>I could not miss the opportunity to write mine !</p>
<p><strong>Disclaimer:</strong> In order to keep this article as clear (and short) as possible,
the first step setup on a …</p><p>Deploying stuff in one command is becoming the Holy Grail of development,
with currently ten times more blog articles than Medieval crusades :)</p>
<p>I could not miss the opportunity to write mine !</p>
<p><strong>Disclaimer:</strong> In order to keep this article as clear (and short) as possible,
the first step setup on a blank server does not use any provisionning system
(<em>another religious matter</em>).</p>
<div class="section" id="prerequesites-easy-dependencies">
<h2>Prerequesites : easy dependencies</h2>
<p>Having a simple <tt class="docutils literal">Makefile</tt> for your application is highly recommended, it will gather all repetitive
commands for setting up dependencies.</p>
<p>Here is a minimalist (working) example, with a project called <tt class="docutils literal">revolution</tt></p>
<pre class="literal-block">
install: bin/python
bin/python:
virtualenv .
bin/python setup.py develop
serve: bin/python
bin/python ./manage.py runserver 8888
deploy: bin/python
bin/python ./manage.py collectstatic --clear --noinput
touch revolution/wsgi.py # trigger reload
clean:
rm -rf bin/ lib/ build/ dist/ *.egg-info/ include/ local/
</pre>
<p>If your project does have any <tt class="docutils literal">setup.py</tt>, just write one <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> or use a <tt class="docutils literal">requirements.txt</tt> file <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>
and replace <tt class="docutils literal">bin/python setup.py develop</tt> in <em>Makefile</em> with <tt class="docutils literal">bin/pip install <span class="pre">-r</span> requirements.txt</tt>.</p>
<p>Now you can then run your deployment commands with <tt class="docutils literal">make deploy</tt>.</p>
</div>
<div class="section" id="first-step-server-setup">
<h2>First step : server setup</h2>
<div class="section" id="prepare-repository">
<h3>Prepare repository</h3>
<p>On your server, create two folders : the repository and deployed app.</p>
<div class="highlight"><pre><span></span>mkdir<span class="w"> </span>-p<span class="w"> </span>/var/git/yourapp.git
mkdir<span class="w"> </span>-p<span class="w"> </span>/var/www/yourapp
</pre></div>
<p>The Git repository will serve as a remote for our code.</p>
<div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>/var/git/yourapp.git
git<span class="w"> </span>init<span class="w"> </span>--bare<span class="w"> </span>.
</pre></div>
<p>Using Git hooks, we will deploy the code being pushed into the
deployed app folder. Create the file <em>hooks/post-receive</em>
with the following content</p>
<pre class="literal-block">
#!/bin/sh
dest=/var/www/yourapp
echo "Deploying into $dest"
GIT_WORK_TREE=$dest git checkout --force
cd $dest
make deploy
</pre>
<p>And make it executable</p>
<pre class="literal-block">
chmod +x /var/git/yourapp.git/hooks/post-receive
</pre>
</div>
<div class="section" id="setup-web-server">
<h3>Setup Web server</h3>
<p>Again, in order to make this straight to the point, I will use <em>Apache's mod_wsgi</em>,
since the configuration is trivial.</p>
<p>Of course, <em>nginx</em>, <em>gunicorn</em>, <em>uwsgi</em> or <em>circus</em> still belong to our prefered stacks but
currently our main point is <em>deploying with git push</em> !</p>
<pre class="literal-block">
sudo apt-get install libapache2-mod-wsgi
</pre>
<p>Create a very simple Apache configuration file in <em>/etc/apache2/sites-available/001-yourapp</em></p>
<pre class="literal-block">
WSGIPythonPath /var/www/yourapp:/var/www/yourapp/lib/python2.6/site-packages
<VirtualHost *:80>
ServerName yourapp.com
ServerAdmin contact@yourapp.com
Alias /static/ /var/www/yourapp/public/static/
<Directory /var/www/yourapp/public/static>
Order deny,allow
Allow from all
</Directory>
WSGIScriptAlias / /var/www/yourapp/revolution/wsgi.py
<Directory /var/www/yourapp/revolution/>
<Files wsgi.py>
Order deny,allow
Allow from all
</Files>
</Directory>
</VirtualHost>
</pre>
<p>And enable it</p>
<pre class="literal-block">
sudo a2ensite 001-yourapp
sudo /etc/init.d/apache2 restart
</pre>
<p>Your site is now up and running...</p>
</div>
</div>
<div class="section" id="next-steps-push-updates">
<h2>Next steps : push updates !</h2>
<p>Now that the application is in production, you will obviously want to push updates !</p>
<p>Comfortably installed at your desk, you just have to push commits to the server,
the same way you already do for your code !</p>
<p>Add the remote (once)</p>
<pre class="literal-block">
$ git remote add production ssh://user@server:/var/git/yourapp.git
</pre>
<p>And push !</p>
<pre class="literal-block">
$ git push production master
...
Counting objects: 862, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (437/437), done.
Writing objects: 100% (817/817), 121.16 KiB, done.
Total 817 (delta 608), reused 452 (delta 332)
remote: Deploying into /var/www/yourapp
...
remote: bin/python setup.py develop
...
...
remote: 345 static files copied.
To server:/var/git/yourapp.git
2fe81f4..76a3fb8 master -> master
</pre>
<p>Your site is up-to-date ! Depending of course of caching policies, but it runs the last version.</p>
<p>Obviously, it is very likely that you will want to push specific branches, but that, you already know!</p>
<img alt="" src="/images/cat_pope.jpg" />
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td><a class="reference external" href="https://docs.djangoproject.com/en/dev/intro/reusable-apps/#packaging-your-app">https://docs.djangoproject.com/en/dev/intro/reusable-apps/#packaging-your-app</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td><a class="reference external" href="http://www.pip-installer.org/en/latest/requirements.html">http://www.pip-installer.org/en/latest/requirements.html</a></td></tr>
</tbody>
</table>
</div>
Embed Daybed forms2013-07-23T14:25:00+02:002013-07-23T14:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-07-23:/embed-daybed-forms.html<p>A brief article to introduce <a class="reference external" href="https://github.com/spiral-project/backbone-daybed">backbone-daybed</a>, a few
helpers to render Web forms for Daybed models.</p>
<img alt="Antikes Sofa Diwan furniert Laden daybed" src="http://upload.wikimedia.org/wikipedia/commons/7/70/Antikes_Sofa_Diwan_furniert_Laden_daybed.jpg" style="width: 512px;" />
<div class="section" id="daybed-lay-down-and-rest">
<h2>Daybed, lay down and REST</h2>
<p><a class="reference external" href="https://github.com/spiral-project/daybed">Daybed</a> is a data validation and storage API, written in Python,
using Pyramid and the fantastic <a class="reference external" href="https://cornice.readthedocs.org/">Cornice</a> addon.</p>
<p>It's a minimalist Web API where you :</p>
<ul class="simple">
<li>define models (schemas)</li>
<li>validate …</li></ul></div><p>A brief article to introduce <a class="reference external" href="https://github.com/spiral-project/backbone-daybed">backbone-daybed</a>, a few
helpers to render Web forms for Daybed models.</p>
<img alt="Antikes Sofa Diwan furniert Laden daybed" src="http://upload.wikimedia.org/wikipedia/commons/7/70/Antikes_Sofa_Diwan_furniert_Laden_daybed.jpg" style="width: 512px;" />
<div class="section" id="daybed-lay-down-and-rest">
<h2>Daybed, lay down and REST</h2>
<p><a class="reference external" href="https://github.com/spiral-project/daybed">Daybed</a> is a data validation and storage API, written in Python,
using Pyramid and the fantastic <a class="reference external" href="https://cornice.readthedocs.org/">Cornice</a> addon.</p>
<p>It's a minimalist Web API where you :</p>
<ul class="simple">
<li>define models (schemas)</li>
<li>validate data and store data</li>
<li>retrieve and update records</li>
</ul>
<p>Key features are:</p>
<ul class="simple">
<li><a class="reference external" href="http://en.wikipedia.org/wiki/Cross-origin_resource_sharing">CORS</a> built-in support</li>
<li>pluggable datastore engines (default is <em>CouchDB</em>)</li>
<li>Geometry fields (maps!)</li>
<li><a class="reference external" href="https://github.com/SPORE/specifications">Spore</a></li>
<li>Access Control (<em>under development</em>)</li>
</ul>
<p>It's a side-project we've been hacking on for a while, and we envision <a class="reference external" href="https://github.com/spiral-project/daybed/wiki/Use-cases">many
applications</a> !
One of them is, since the beginning, a Web forms service !</p>
</div>
<div class="section" id="models-are-yours">
<h2>Models are yours</h2>
<p>In order to create your own models, you can either use the
crude GUI of <a class="reference external" href="http://leplatrem.github.io/daybed-map/">daybed-maps</a>
<strong>or</strong> post a JSON manually on the <a class="reference external" href="http://daybed.lolnet.org">Daybed instance</a> we run for you.</p>
<p>In both cases you will reference your model definition using
the <tt class="docutils literal">ìd</tt> you chose.</p>
<p>Below, we define <tt class="docutils literal"><span class="pre">demo-poll-conf</span></tt> using cUrl in the command-line.</p>
<p>The model will be a stupid poll to ask how many conferences you attended
in the past year.</p>
<div class="highlight"><pre><span></span><span class="c1"># in a terminal...</span>
<span class="nv">definition</span><span class="o">=</span><span class="s1">'{</span>
<span class="s1">"title": "Conferences Poll",</span>
<span class="s1">"description": "How many conferences attended last year ?",</span>
<span class="s1">"fields": [</span>
<span class="s1"> {</span>
<span class="s1"> "name": "total",</span>
<span class="s1"> "type": "int",</span>
<span class="s1"> "description": "How many times ?"</span>
<span class="s1"> }, {</span>
<span class="s1"> "name": "category",</span>
<span class="s1"> "type": "enum",</span>
<span class="s1"> "choices": ["Web", "Strategy", "Technology"],</span>
<span class="s1"> "description": "Mostly in..."</span>
<span class="s1"> }</span>
<span class="s1">]}'</span>
curl<span class="w"> </span>-XPUT<span class="w"> </span>http://daybed.lolnet.org/definitions/demo-poll-conf<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">definition</span><span class="si">}</span><span class="s2">"</span>
</pre></div>
</div>
<div class="section" id="backbone-daybed-simple-and-stupid">
<h2>backbone-daybed, simple and stupid</h2>
<p>Backbone.js is not the <em>next</em> big thing :) <em>#ooold, it's sooo 2011</em> !</p>
<p>But frankly, it has remained simple, very easy to learn and yet quite efficient!
That's why I chose it to demo the power of having storage-as-a-service with Daybed.
Plus, <a class="reference external" href="https://github.com/powmedia/backbone-forms">backbone-forms</a> provided
the right level of abstraction I needed !</p>
<p>For example, here we embed a form in the page for the model we just created,
and start polling the audience !</p>
<p>The few lines of Javascript below render the form and reacts on submission !</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"http://cdnjs.cloudflare.com/ajax/libs/jquery/1.8.3/jquery.min.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"http://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.4.2/underscore-min.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"http://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.0.0/backbone-min.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"http://cdnjs.cloudflare.com/ajax/libs/mustache.js/0.7.0/mustache.min.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"http://cdnjs.cloudflare.com/ajax/libs/backbone-forms/0.12.0/backbone-forms.min.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"https://rawgithub.com/spiral-project/backbone-daybed/1e410a85/backbone-daybed.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span><span class="p">></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">form</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Daybed</span><span class="p">.</span><span class="nx">renderForm</span><span class="p">(</span><span class="s1">'#demo-form-container'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">id</span><span class="o">:</span><span class="w"> </span><span class="s1">'demo-poll-conf'</span><span class="p">});</span>
<span class="w"> </span><span class="nx">form</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'created'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">record</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// plot chart !</span>
<span class="w"> </span><span class="p">})</span>
<span class="p"></</span><span class="nt">script</span><span class="p">></span>
</pre></div>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/jquery/1.8.3/jquery.min.js"></script>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.4.2/underscore-min.js"></script>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.0.0/backbone-min.js"></script>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/mustache.js/0.7.0/mustache.min.js"></script>
<script type="text/javascript" src="http://cdnjs.cloudflare.com/ajax/libs/backbone-forms/0.12.0/backbone-forms.min.js"></script>
<script type="text/javascript" src="https://rawgithub.com/spiral-project/backbone-daybed/1e410a85/backbone-daybed.js"></script>
<script type="text/javascript" src="https://rawgithub.com/nnnick/Chart.js/master/Chart.min.js"></script>
<style>
#demo-form-container {
border: 1px solid #6C0AAB;
margin-bottom: 30px;
display: inline-block;
padding: 10px;
border-radius: 5px;
}
#demo-form-container .field-error {
color: red;
}
#demo-form-container label {
font-weight: bold;
}
#demo-form-container ul {
margin: 0px;
list-style-type: none;
}
#demo-form-container a.btn {
float: right;
text-decoration: none;
background-color: #6C0AAB;
color: white;
border-radius: 3px;
font-size: 16px;
}
</style>
<div id="demo-form-container"></div>
<script type="text/javascript">
$(document).ready(function () {
Daybed.SETTINGS.SERVER = "http://daybed.lolnet.org"; // no trailing slash
var form = Daybed.renderForm('#demo-form-container',
{id: 'demo-poll-conf',
title: 'Conferences poll :',
save: 'Submit',
cancel: null});
// Fetch all the records
var records = new Daybed.ItemList(form.definition);
records.fetch();
// On submission, plot the chart
form.on('created', function (record) {
records.add(record);
// Prepare plot data
var data = {
labels : [],
datasets : [{
fillColor : "#E0E4CC",
strokeColor : "#6C0AAB",
data : []
}]
};
var byCat = {};
records.each(function (r) {
var cat = r.attributes.category;
if (!byCat[cat]) byCat[cat] = [];
byCat[cat].push(r.attributes.total);
});
for(var cat in byCat) {
var sum = _.reduce(byCat[cat], function(memo, num){ return memo + num; }, 0),
val = sum / byCat[cat].length;
data.labels.push(cat);
data.datasets[0].data.push(val);
}
// Render the bar chart
var ctx = $('#demo-form-container').html('<p>Avg. per category</p>'+
'<canvas height="200"/>')
.find('canvas')[0].getContext("2d"),
chart = new Chart(ctx).Bar(data);
});
});
</script><p>The helper downloads the definition JSON, renders fields within an HTML form with
<em>backbone-forms</em>. And in this example specifically, on submission, we fetch all the records,
compute average values by category in order to plot some naive chart using <a class="reference external" href="http://chartjs.org">Chart.js</a>.</p>
<img alt="" src="/images/backbone-daybed-preview.png" style="width: 700px;" />
<p>You can also have a look at the very few lines of the backbone-daybed demo, it's dead easy !
It features a CRUD application : Create, edit and delete records for the model of your choice ! <a class="reference external" href="http://spiral-project.github.io/backbone-daybed/#demo-poll-conf">http://spiral-project.github.io/backbone-daybed/#demo-poll-conf</a> (<em>See URL hash</em>)</p>
</div>
<div class="section" id="shortly">
<h2>Shortly</h2>
<ul class="simple">
<li>Daybed is a generic backend where you define models, validate and post data ;</li>
<li>There are already <a class="reference external" href="https://github.com/spiral-project/daybed/wiki/Use-cases">various working applications</a> built with this storage-as-a-service ;</li>
<li>Most Javascript frameworks will play well natively with Daybed REST API ;</li>
<li>backbone-daybed is just a helper to render Daybed models as forms, .... <a class="reference external" href="https://github.com/spiral-project/backbone-daybed/#readme">and a little bit more</a> !</li>
</ul>
<p>So far, Daybed data is not protected (like a wiki), but access control is currently being implemented :)</p>
<p>Stay tuned !</p>
</div>
Use PostGIS topologies to clean-up road networks2013-07-03T14:25:00+02:002013-07-03T14:25:00+02:00Mathieu Leplatre, Frédéric Bonifastag:blog.mathieu-leplatre.info,2013-07-03:/use-postgis-topologies-to-clean-up-road-networks.html<p>This article gives a few basics to get started with using the PostGIS topology extension.</p>
<p>We will take avandtage of topologies to clean-up a real topological road network, coming from <a class="reference external" href="http://data.grandtoulouse.fr/web/guest/les-donnees/-/opendata/card/12693-filaire-de-voirie">Toulouse OpenData files</a>.</p>
<img alt="" class="align-center" src="/images/toulouse-voirie.png" />
<div class="section" id="topological">
<h2>Topological</h2>
<p>A topology is a general concept, where objects are defined by their relationships
instead of their …</p></div><p>This article gives a few basics to get started with using the PostGIS topology extension.</p>
<p>We will take avandtage of topologies to clean-up a real topological road network, coming from <a class="reference external" href="http://data.grandtoulouse.fr/web/guest/les-donnees/-/opendata/card/12693-filaire-de-voirie">Toulouse OpenData files</a>.</p>
<img alt="" class="align-center" src="/images/toulouse-voirie.png" />
<div class="section" id="topological">
<h2>Topological</h2>
<p>A topology is a general concept, where objects are defined by their relationships
instead of their geometries. Instead of lines, we manipulate edges, vertices and faces :
might remind you the core concepts of graph theory.</p>
<p>A topological road network is supposed to have their lines (edges) connected at single points (nodes).</p>
<p>In this example dataset, <a class="reference external" href="http://wiki.openstreetmap.org/wiki/JOSM/Validator">JOSM validator</a> detects not less than 1643 errors :) Broken connections, crossing lines ...</p>
<img alt="" src="/images/toulouse-voirie-error.png" />
<img alt="" src="/images/toulouse-voirie-error2.png" />
<img alt="" src="/images/toulouse-voirie-error3.png" />
<p><strong>Let's clean this up !</strong></p>
</div>
<div class="section" id="installation">
<h2>Installation</h2>
<p>On Ubuntu 12.04, you just have to install PostGIS :</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>apt-add-repository<span class="w"> </span>-y<span class="w"> </span>ppa:ubuntugis/ppa
sudo<span class="w"> </span>apt-get<span class="w"> </span>update
sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>-y<span class="w"> </span>postgresql<span class="w"> </span>postgis
</pre></div>
<p>The topology extension is installed by default. Just activate it in your database:</p>
<div class="highlight"><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="ss">"roadsdb"</span><span class="p">;</span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">EXTENSION</span><span class="w"> </span><span class="n">postgis</span><span class="p">;</span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">EXTENSION</span><span class="w"> </span><span class="n">postgis_topology</span><span class="p">;</span>
<span class="k">SET</span><span class="w"> </span><span class="n">search_path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">topology</span><span class="p">,</span><span class="k">public</span><span class="p">;</span>
</pre></div>
</div>
<div class="section" id="data-import">
<h2>Data Import</h2>
<p>Load your shapefile (using command-line) like usual :</p>
<div class="highlight"><pre><span></span><span class="nv">schema</span><span class="o">=</span><span class="s2">"public."</span>
<span class="nv">db</span><span class="o">=</span><span class="s2">"roadsdb"</span>
<span class="nv">user</span><span class="o">=</span><span class="s2">"postgres"</span>
<span class="nv">password</span><span class="o">=</span><span class="s2">"postgres"</span>
<span class="nv">host</span><span class="o">=</span><span class="s2">"localhost"</span>
ogr2ogr<span class="w"> </span>-f<span class="w"> </span><span class="s2">"PostgreSQL"</span><span class="w"> </span>PG:<span class="s2">"host=</span><span class="si">${</span><span class="nv">host</span><span class="si">}</span><span class="s2"> user=</span><span class="si">${</span><span class="nv">user</span><span class="si">}</span><span class="s2"> dbname=</span><span class="si">${</span><span class="nv">db</span><span class="si">}</span><span class="s2"> password=</span><span class="si">${</span><span class="nv">password</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span>-a_srs<span class="w"> </span><span class="s2">"EPSG:2154"</span><span class="w"> </span>-nln<span class="w"> </span><span class="si">${</span><span class="nv">schema</span><span class="si">}</span>roads<span class="w"> </span>-nlt<span class="w"> </span>MULTILINESTRING<span class="w"> </span>ROAD_SHAPEFILE.SHP
</pre></div>
<p>Create and associate the PostGIS topology:</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="n">topology</span><span class="p">.</span><span class="n">CreateTopology</span><span class="p">(</span><span class="s1">'roads_topo'</span><span class="p">,</span><span class="w"> </span><span class="mi">2154</span><span class="p">);</span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">topology</span><span class="p">.</span><span class="n">AddTopoGeometryColumn</span><span class="p">(</span><span class="s1">'roads_topo'</span><span class="p">,</span><span class="w"> </span><span class="s1">'public'</span><span class="p">,</span><span class="w"> </span><span class="s1">'roads'</span><span class="p">,</span><span class="w"> </span><span class="s1">'topo_geom'</span><span class="p">,</span><span class="w"> </span><span class="s1">'LINESTRING'</span><span class="p">);</span>
</pre></div>
<p>Convert linestrings to vertices and edges within the topology :</p>
<div class="highlight"><pre><span></span><span class="c1">-- Layer 1, with 1.0 meter tolerance</span>
<span class="k">UPDATE</span><span class="w"> </span><span class="n">roads</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">topo_geom</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">topology</span><span class="p">.</span><span class="n">toTopoGeom</span><span class="p">(</span><span class="n">wkb_geometry</span><span class="p">,</span><span class="w"> </span><span class="s1">'roads_topo'</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="p">);</span>
</pre></div>
<p>From now on, we have a topology, whose imperfections were corrected. It smoothly merged
all <em>dirty</em> junctions, whose defects were at most 1.0 meter wide.</p>
<img alt="" src="/images/toulouse-voirie-clean.png" />
<p>You may enconter insertion problems : the tool fails <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> and aborts the whole transaction.
Use this snippet to skip errors and go on with the next records:</p>
<div class="highlight"><pre><span></span><span class="k">DO</span><span class="w"> </span><span class="err">$$</span><span class="k">DECLARE</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="n">record</span><span class="p">;</span>
<span class="k">BEGIN</span>
<span class="w"> </span><span class="k">FOR</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="k">IN</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">roads</span><span class="w"> </span><span class="n">LOOP</span>
<span class="w"> </span><span class="k">BEGIN</span>
<span class="w"> </span><span class="k">UPDATE</span><span class="w"> </span><span class="n">roads</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">topo_geom</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">topology</span><span class="p">.</span><span class="n">toTopoGeom</span><span class="p">(</span><span class="n">wkb_geometry</span><span class="p">,</span><span class="w"> </span><span class="s1">'roads_topo'</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="p">)</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">ogc_fid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">ogc_fid</span><span class="p">;</span>
<span class="w"> </span><span class="k">EXCEPTION</span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">OTHERS</span><span class="w"> </span><span class="k">THEN</span>
<span class="w"> </span><span class="n">RAISE</span><span class="w"> </span><span class="n">WARNING</span><span class="w"> </span><span class="s1">'Loading of record % failed: %'</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">ogc_fid</span><span class="p">,</span><span class="w"> </span><span class="n">SQLERRM</span><span class="p">;</span>
<span class="w"> </span><span class="k">END</span><span class="p">;</span>
<span class="w"> </span><span class="k">END</span><span class="w"> </span><span class="n">LOOP</span><span class="p">;</span>
<span class="k">END</span><span class="err">$$</span><span class="p">;</span>
</pre></div>
<p>This is rather frustrating to face topological errors at insertion ! You can try with a lower tolerance,
or check that your records have at least valid geometries. <em>Any clarification or help on this would be welcome</em> :)</p>
</div>
<div class="section" id="visualize-and-export">
<h2>Visualize and export</h2>
<p>In order to visualize your topology vertices in QGIS, browse your database tables,
and add the following layers: <tt class="docutils literal">roads_topo.edge_data</tt> and <tt class="docutils literal">roads_topo.node</tt>.</p>
<img alt="" src="/images/toulouse-voirie-topology.png" />
<p>You can also export the resulting geometries into a new table :</p>
<div class="highlight"><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">roads_clean</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="n">ogc_fid</span><span class="p">,</span><span class="w"> </span><span class="n">topo_geom</span><span class="p">::</span><span class="n">geometry</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">roads</span>
<span class="p">);</span>
</pre></div>
<p>Or obtain your lovable Shapefile in return :</p>
<div class="highlight"><pre><span></span><span class="n">ogr2ogr</span><span class="w"> </span><span class="o">-</span><span class="n">f</span><span class="w"> </span><span class="ss">"ESRI Shapefile"</span><span class="w"> </span><span class="n">ROAD_CLEAN</span><span class="p">.</span><span class="n">SHP</span><span class="w"> </span><span class="n">PG</span><span class="p">:</span><span class="ss">"host=${host} user=${user} dbname=${db} password=${password}"</span><span class="w"> </span><span class="o">-</span><span class="k">sql</span><span class="w"> </span><span class="ss">"SELECT topo_geom::geometry FROM roads"</span>
</pre></div>
<p>If, like <a class="reference external" href="http://gis.stackexchange.com/questions/71270/split-line-at-intersection-and-attach-attributes/">Amit</a> you want to split the lines at
intersections and assign original attributes, just join <tt class="docutils literal">roads_topo.edge_data</tt> and on the <tt class="docutils literal">roads</tt> table :</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">lib_off</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">ogc_fid</span><span class="p">,</span><span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="n">geom</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">roads_topo</span><span class="p">.</span><span class="n">edge</span><span class="w"> </span><span class="n">e</span><span class="p">,</span>
<span class="w"> </span><span class="n">roads_topo</span><span class="p">.</span><span class="n">relation</span><span class="w"> </span><span class="n">rel</span><span class="p">,</span>
<span class="w"> </span><span class="n">roads</span><span class="w"> </span><span class="n">r</span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">e</span><span class="p">.</span><span class="n">edge_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rel</span><span class="p">.</span><span class="n">element_id</span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">rel</span><span class="p">.</span><span class="n">topogeo_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">topo_geom</span><span class="p">).</span><span class="n">id</span>
</pre></div>
</div>
<div class="section" id="going-further">
<h2>Going further...</h2>
<p>We could collapse crossing lines and disconnected junctions into a nice and clean network.</p>
<p>Yes ahem, we weren't able to repair <em>every</em> topological error of this dataset using this automatic method.
Some inconsistencies, like the following one, are like 6 meters wide ! They are, by the way, perfectly described in OpenStreetMap :</p>
<img alt="" src="/images/toulouse-voirie-error4.png" />
<p>We could also play with simplifications using <a class="reference external" href="http://strk.keybit.net/blog/2012/04/13/simplifying-a-map-layer-using-postgis-topology/">Sandro Santilli</a>'s <tt class="docutils literal">SimplifyEdgeGeom</tt> <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a> function, it will collapse edges with a higher tolerance ...</p>
<div class="highlight"><pre><span></span><span class="k">SELECT</span><span class="w"> </span><span class="n">SimplifyEdgeGeom</span><span class="p">(</span><span class="s1">'roads_topo'</span><span class="p">,</span><span class="w"> </span><span class="n">edge_id</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">.</span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">roads_topo</span><span class="p">.</span><span class="n">edge</span><span class="p">;</span>
</pre></div>
<p>Don't hesitate to share your thoughts and feedback. Concrete use cases and examples are rare about this!
And as usual, drop a comment if anything is wrong or not clear :)</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td><tt class="docutils literal">SQL/MM Spatial exception</tt>, <tt class="docutils literal">geometry intersects edge</tt>, <tt class="docutils literal">side location conflict</tt>, ...</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td>Just execute the <a class="reference external" href="https://gist.github.com/leplatrem/5729022">function SQL code</a>. It's
just an elegant wrapper around <tt class="docutils literal">ST_ChangeEdgeGeom</tt> and <tt class="docutils literal">ST_Simplify</tt>.</td></tr>
</tbody>
</table>
</div>
News about Subtivals2013-06-07T00:00:00+02:002013-06-07T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-06-07:/news-about-subtivals.html<p>Subtivals was used successfully in movie festivals in Cyprus, and that
makes us happy ! And we regularly receive demands for Subtivals installers
accross the globe ! The great unicode support of Subtivals (<em>provided natively by Qt</em>)
makes it a great tool for movie subtitling on the international scene !</p>
<div class="section" id="a-new-website-subtivals-org">
<h2>A new website …</h2></div><p>Subtivals was used successfully in movie festivals in Cyprus, and that
makes us happy ! And we regularly receive demands for Subtivals installers
accross the globe ! The great unicode support of Subtivals (<em>provided natively by Qt</em>)
makes it a great tool for movie subtitling on the international scene !</p>
<div class="section" id="a-new-website-subtivals-org">
<h2>A new website... subtivals.org !</h2>
<p>We used to host our main project page within a Github README file. We now
have a dedicated website :</p>
<p><a class="reference external" href="http://subtivals.org">http://subtivals.org</a> !</p>
<p>It's still hosted and edited on Github <3, we thank them for that !</p>
<video src="http://mathieu-leplatre.info/media/subtivals/subtivals-1.6-calibration.webm" width="500" preload="auto" autoplay loop>
<p>Your browser does not support the video element </p>
</video><p>( <em>Calibration tool in action</em> )</p>
</div>
<div class="section" id="some-new-features-in-version-1-6">
<h2>Some new features in version 1.6</h2>
<p>We added a shortcut editor, it's a great way to explore existing shortcuts,
and thus prevent reading the documentation ! Plus, if Subtivals users have habits
with former tools like <em>Icareus Screen pro software</em>, they can adjust the settings
and use this great opensource alternative easily ;)</p>
<img alt="" class="align-center" src="/images/subtivals-1.6-shorcuteditor.png" />
<p>Also, Subtivals can now read plain text files. This appears to be useful for
theaters or operas, since timecode notion is irrelevant in those use cases.
We chose a very simple text format:</p>
<pre class="literal-block">
This is one subtitle!
Here comes another,
with two lines!
</pre>
<p>Timecodes are optional, but still supported :</p>
<pre class="literal-block">
00:00:19:13 00:00:23:08
À 24-25 ans, j'avais déjà un film
qui tournait en festival.
</pre>
</div>
<div class="section" id="get-it">
<h2>Get it !</h2>
<p>Subtivals is released under <a class="reference external" href="http://www.gnu.org/copyleft/gpl.html">GPL</a>,
this means you can use, access the source code, modify, package, distribute the software,
as long as it remains GPL.</p>
<p>Like previous versions, we provide packages for Ubuntu. If you already
installed it, it will be updated automatically. Otherwise, it's just simple as :</p>
<pre class="literal-block">
sudo add-apt-repository ppa:mathieu.leplatre/subtivals
sudo apt-get update && sudo apt-get install subtivals
</pre>
<p>Demo versions can be downloaded though for <a class="reference external" href="http://mathieu-leplatre.info/media/subtivals/">Mac OS and Windows</a>.</p>
<p>We still follow <a class="reference external" href="http://gcompris.net/-Download-">Bruno’s approach</a> :
we sell the installers on proprietary operating systems, in order to promote GNU/Linux.</p>
</div>
Drape lines on a DEM with PostGIS2013-04-30T10:25:00+02:002013-04-30T10:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-04-30:/drape-lines-on-a-dem-with-postgis.html<p>This article gives a few SQL commands to drape 2D geometries on a DEM (<em>Digital Elevation Model</em>), in order to obtain 3D geometries.
We use PostGIS 2, and its rasters support especially.</p>
<div class="section" id="load-your-dem">
<h2>Load your DEM</h2>
<p>Assuming you have a DEM compatible with GDAL, you can easily load the raster into …</p></div><p>This article gives a few SQL commands to drape 2D geometries on a DEM (<em>Digital Elevation Model</em>), in order to obtain 3D geometries.
We use PostGIS 2, and its rasters support especially.</p>
<div class="section" id="load-your-dem">
<h2>Load your DEM</h2>
<p>Assuming you have a DEM compatible with GDAL, you can easily load the raster into the database using these commands.</p>
<p><strong>Reprojects</strong> to specified SRID, <strong>crops</strong> to specified extent, and writes output in a file:</p>
<div class="highlight"><pre><span></span>gdalwarp<span class="w"> </span>-t_srs<span class="w"> </span>EPSG:32632<span class="w"> </span>-te<span class="w"> </span><span class="m">289942</span><span class="w"> </span><span class="m">4845809</span><span class="w"> </span><span class="m">400671</span><span class="w"> </span><span class="m">4947295</span><span class="w"> </span>dem_file.geotif<span class="w"> </span>output.bin
</pre></div>
<p>Tiles into 100 pixels squares and <strong>converts to SQL</strong>:</p>
<div class="highlight"><pre><span></span>raster2pgsql<span class="w"> </span>-c<span class="w"> </span>-C<span class="w"> </span>-I<span class="w"> </span>-M<span class="w"> </span>-t<span class="w"> </span>100x100<span class="w"> </span>output.bin<span class="w"> </span>mnt<span class="w"> </span>><span class="w"> </span>output.sql
</pre></div>
<p><strong>Load SQL</strong> into database:</p>
<div class="highlight"><pre><span></span>psql<span class="w"> </span>-d<span class="w"> </span>yourdb<span class="w"> </span><<span class="w"> </span>output.sql
</pre></div>
<img alt="" class="align-center" src="/images/postgis_dem_qgis.jpg" />
</div>
<div class="section" id="drape-geometries">
<h2>Drape geometries</h2>
<p>There are at least 3 strategies to drape your geometries.</p>
<div class="section" id="with-geometry-resolution">
<h3>With geometry resolution</h3>
<p>We obtain one elevation value per point on your line.</p>
<p><strong>Pros</strong>: You keep your original geometry resolution (number of points)</p>
<p><strong>Cons</strong>: You potentially loose a lot of 3D information (think of "hops")</p>
<img alt="" class="align-center" src="/images/postgis_dem_native.png" />
<div class="highlight"><pre><span></span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- From an arbitrary line</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="s1">'SRID=32632;LINESTRING (348595 4889225,352577 4887465,354784 4883841)'</span><span class="p">::</span><span class="n">geometry</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="p">),</span>
<span class="w"> </span><span class="n">points2d</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- Extract its points</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="p">(</span><span class="n">ST_DumpPoints</span><span class="p">(</span><span class="n">geom</span><span class="p">)).</span><span class="n">geom</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">line</span><span class="p">),</span>
<span class="w"> </span><span class="n">cells</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- Get DEM elevation for each</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">geom</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="p">,</span><span class="w"> </span><span class="n">ST_Value</span><span class="p">(</span><span class="n">mnt</span><span class="p">.</span><span class="n">rast</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">val</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">mnt</span><span class="p">,</span><span class="w"> </span><span class="n">points2d</span><span class="w"> </span><span class="n">p</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">ST_Intersects</span><span class="p">(</span><span class="n">mnt</span><span class="p">.</span><span class="n">rast</span><span class="p">,</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">geom</span><span class="p">)),</span>
<span class="w"> </span><span class="c1">-- Instantiate 3D points</span>
<span class="w"> </span><span class="n">points3d</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">ST_SetSRID</span><span class="p">(</span><span class="n">ST_MakePoint</span><span class="p">(</span><span class="n">ST_X</span><span class="p">(</span><span class="n">geom</span><span class="p">),</span><span class="w"> </span><span class="n">ST_Y</span><span class="p">(</span><span class="n">geom</span><span class="p">),</span><span class="w"> </span><span class="n">val</span><span class="p">),</span><span class="w"> </span><span class="mi">32632</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">cells</span><span class="p">)</span>
<span class="c1">-- Build 3D line from 3D points</span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">ST_MakeLine</span><span class="p">(</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">points3d</span><span class="p">;</span>
</pre></div>
<p><strong>Note</strong> by Daniel Gerber: if the line goes outside your DEM, use a left join (<tt class="docutils literal">FROM points2d LEFT OUTER JOIN elevation ON <span class="pre">ST_Intersects(...)</span></tt>) and set default value to 0.0 with <tt class="docutils literal"><span class="pre">coalesce(ST_Value(..),</span> 0.0)</tt>.</p>
</div>
<div class="section" id="with-dem-resolution">
<h3>With DEM resolution</h3>
<p>We obtain one elevation value per cell of your raster.</p>
<p><strong>Pros</strong>: You take full advantage of your DEM</p>
<p><strong>Cons</strong>: You may increase tremendously the resolution of geometries</p>
<img alt="" class="align-center" src="/images/postgis_dem_full.png" />
<div class="highlight"><pre><span></span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- From an arbitrary line</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="s1">'SRID=32632;LINESTRING (348595 4889225,352577 4887465,354784 4883841)'</span><span class="p">::</span><span class="n">geometry</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="p">),</span>
<span class="w"> </span><span class="n">cells</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- Get DEM elevation for each intersected cell</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">ST_Centroid</span><span class="p">((</span><span class="n">ST_Intersection</span><span class="p">(</span><span class="n">mnt</span><span class="p">.</span><span class="n">rast</span><span class="p">,</span><span class="w"> </span><span class="n">line</span><span class="p">.</span><span class="n">geom</span><span class="p">)).</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="p">,</span>
<span class="w"> </span><span class="p">(</span><span class="n">ST_Intersection</span><span class="p">(</span><span class="n">mnt</span><span class="p">.</span><span class="n">rast</span><span class="p">,</span><span class="w"> </span><span class="n">line</span><span class="p">.</span><span class="n">geom</span><span class="p">)).</span><span class="n">val</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">val</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">mnt</span><span class="p">,</span><span class="w"> </span><span class="n">line</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">ST_Intersects</span><span class="p">(</span><span class="n">mnt</span><span class="p">.</span><span class="n">rast</span><span class="p">,</span><span class="w"> </span><span class="n">line</span><span class="p">.</span><span class="n">geom</span><span class="p">)),</span>
<span class="w"> </span><span class="c1">-- Instantiate 3D points, ordered on line</span>
<span class="w"> </span><span class="n">points3d</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">ST_SetSRID</span><span class="p">(</span><span class="n">ST_MakePoint</span><span class="p">(</span><span class="n">ST_X</span><span class="p">(</span><span class="n">cells</span><span class="p">.</span><span class="n">geom</span><span class="p">),</span><span class="w"> </span><span class="n">ST_Y</span><span class="p">(</span><span class="n">cells</span><span class="p">.</span><span class="n">geom</span><span class="p">),</span><span class="w"> </span><span class="n">val</span><span class="p">),</span><span class="w"> </span><span class="mi">32632</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">cells</span><span class="p">,</span><span class="w"> </span><span class="n">line</span>
<span class="w"> </span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">ST_Distance</span><span class="p">(</span><span class="n">ST_StartPoint</span><span class="p">(</span><span class="n">line</span><span class="p">.</span><span class="n">geom</span><span class="p">),</span><span class="w"> </span><span class="n">cells</span><span class="p">.</span><span class="n">geom</span><span class="p">))</span>
<span class="c1">-- Build 3D line from 3D points</span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">ST_MakeLine</span><span class="p">(</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">points3d</span><span class="p">;</span>
</pre></div>
</div>
<div class="section" id="sampling">
<h3>Sampling</h3>
<p>We obtain one elevation value per step of X units (meters).</p>
<p><strong>Pros</strong>: You control the resulting resolution</p>
<p><strong>Cons</strong>: Sometimes hard to find a good balance depending on geometries extents</p>
<img alt="" class="align-center" src="/images/postgis_dem_sampled.png" />
<div class="highlight"><pre><span></span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="n">line</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- From an arbitrary line</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="s1">'SRID=32632;LINESTRING (348595 4889225,352577 4887465,354784 4883841)'</span><span class="p">::</span><span class="n">geometry</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="p">),</span>
<span class="w"> </span><span class="n">linemesure</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- Add a mesure dimension to extract steps</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">ST_AddMeasure</span><span class="p">(</span><span class="n">line</span><span class="p">.</span><span class="n">geom</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">ST_Length</span><span class="p">(</span><span class="n">line</span><span class="p">.</span><span class="n">geom</span><span class="p">))</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">linem</span><span class="p">,</span>
<span class="w"> </span><span class="n">generate_series</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="n">ST_Length</span><span class="p">(</span><span class="n">line</span><span class="p">.</span><span class="n">geom</span><span class="p">)::</span><span class="nb">int</span><span class="p">,</span><span class="w"> </span><span class="mi">50</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">i</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">line</span><span class="p">),</span>
<span class="w"> </span><span class="n">points2d</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">ST_GeometryN</span><span class="p">(</span><span class="n">ST_LocateAlong</span><span class="p">(</span><span class="n">linem</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">),</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">linemesure</span><span class="p">),</span>
<span class="w"> </span><span class="n">cells</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="c1">-- Get DEM elevation for each</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">geom</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="p">,</span><span class="w"> </span><span class="n">ST_Value</span><span class="p">(</span><span class="n">mnt</span><span class="p">.</span><span class="n">rast</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">val</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">mnt</span><span class="p">,</span><span class="w"> </span><span class="n">points2d</span><span class="w"> </span><span class="n">p</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">ST_Intersects</span><span class="p">(</span><span class="n">mnt</span><span class="p">.</span><span class="n">rast</span><span class="p">,</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">geom</span><span class="p">)),</span>
<span class="w"> </span><span class="c1">-- Instantiate 3D points</span>
<span class="w"> </span><span class="n">points3d</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">ST_SetSRID</span><span class="p">(</span><span class="n">ST_MakePoint</span><span class="p">(</span><span class="n">ST_X</span><span class="p">(</span><span class="n">geom</span><span class="p">),</span><span class="w"> </span><span class="n">ST_Y</span><span class="p">(</span><span class="n">geom</span><span class="p">),</span><span class="w"> </span><span class="n">val</span><span class="p">),</span><span class="w"> </span><span class="mi">32632</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">cells</span><span class="p">)</span>
<span class="c1">-- Build 3D line from 3D points</span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">ST_MakeLine</span><span class="p">(</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">points3d</span><span class="p">;</span>
</pre></div>
</div>
<div class="section" id="as-a-postgresql-function">
<h3>As a PostgreSQL function</h3>
<p>You can define a function:</p>
<div class="highlight"><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="k">REPLACE</span><span class="w"> </span><span class="k">FUNCTION</span><span class="w"> </span><span class="n">drape</span><span class="p">(</span><span class="n">line</span><span class="w"> </span><span class="n">geometry</span><span class="p">)</span><span class="w"> </span><span class="k">RETURNS</span><span class="w"> </span><span class="n">geometry</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="err">$$</span>
<span class="k">DECLARE</span>
<span class="w"> </span><span class="n">line3d</span><span class="w"> </span><span class="n">geometry</span><span class="p">;</span>
<span class="k">BEGIN</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="n">ST_MakeLine</span><span class="p">(</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">geom3d</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">points3d</span><span class="p">;</span>
<span class="w"> </span><span class="k">RETURN</span><span class="w"> </span><span class="n">geom3d</span><span class="p">;</span>
<span class="k">END</span><span class="p">;</span>
<span class="err">$$</span><span class="w"> </span><span class="k">LANGUAGE</span><span class="w"> </span><span class="n">plpgsql</span><span class="p">;</span>
</pre></div>
<p>And drape your geometries:</p>
<div class="highlight"><pre><span></span><span class="c1">-- Add a column to your table</span>
<span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">yourtable</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="n">geom_3d</span><span class="w"> </span><span class="n">geometry</span><span class="p">(</span><span class="n">LineStringZ</span><span class="p">,</span><span class="w"> </span><span class="mi">32632</span><span class="p">);</span>
<span class="c1">-- Fill it</span>
<span class="k">UPDATE</span><span class="w"> </span><span class="n">yourtable</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">geom_3d</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">drape</span><span class="p">(</span><span class="n">geom</span><span class="p">);</span>
</pre></div>
</div>
</div>
<div class="section" id="altimetric-profiles">
<h2>Altimetric profiles</h2>
<p>We obtain a basic chart, where you have the distance in abscissa and altitude in ordinate. This SQL query returns 2 columns, <em>x</em> and <em>y</em> axis.</p>
<div class="highlight"><pre><span></span><span class="k">WITH</span><span class="w"> </span><span class="n">points3d</span><span class="w"> </span><span class="k">AS</span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="p">(</span><span class="n">ST_DumpPoints</span><span class="p">(</span><span class="n">geom_3d</span><span class="p">)).</span><span class="n">geom</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">geom</span><span class="p">,</span>
<span class="w"> </span><span class="n">ST_StartPoint</span><span class="p">(</span><span class="n">geom_3d</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">origin</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">yourtable</span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1234</span><span class="p">)</span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">ST_distance</span><span class="p">(</span><span class="n">origin</span><span class="p">,</span><span class="w"> </span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">ST_Z</span><span class="p">(</span><span class="n">geom</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">y</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">points3d</span><span class="p">;</span>
</pre></div>
<p>Of course, you can apply a different strategy at this stage, and get full resolution or sampled altimetric profiles...</p>
<p>Drop a comment if anything is not clear :)</p>
</div>
Test your Leaflet applications with Mocha2013-03-29T00:00:00+01:002013-03-29T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-03-29:/test-your-leaflet-applications-with-mocha.html<p>Pretty much like <a class="reference external" href="https://nicolas.perriault.net/code/2013/why_javascript/">n1k0</a>, I feel like I had learned Javascript three or four times, from the <tt class="docutils literal">alert()</tt> back in 1997 to this article about automatic testing. I must admit that, lately, most of my progress in Javascript comes from using and hacking Leaflet <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>, but I hadn't gone as …</p><p>Pretty much like <a class="reference external" href="https://nicolas.perriault.net/code/2013/why_javascript/">n1k0</a>, I feel like I had learned Javascript three or four times, from the <tt class="docutils literal">alert()</tt> back in 1997 to this article about automatic testing. I must admit that, lately, most of my progress in Javascript comes from using and hacking Leaflet <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>, but I hadn't gone as far as unit testing until now !</p>
<p>These are some notes about me getting started with using <a class="reference external" href="http://visionmedia.github.com/mocha/#browser-support">Mocha</a> and <a class="reference external" href="http://leafletjs.com">Leaflet</a>. If what you read is not clear or simply wrong, please let me know or <a class="reference external" href="https://github.com/leplatrem/blog.mathieu-leplatre.info">fork it directly</a> so that everybody can learn !</p>
<div class="section" id="goals">
<h2>Goals</h2>
<ul class="simple">
<li>Test your Javascript code to prevent regressions, just as you already do with Python ;</li>
<li>Run test suites from command-line, especially for <a class="reference external" href="http://jenkins-ci.org">CI</a> ;</li>
<li>Learn something new and practical !</li>
</ul>
<p>There are many ways to achieve this, you might have spotted <em>QUnit</em> or <em>Jasmine</em>. We also like <em>CasperJS</em>, coupled with <em>resurectio</em>, but this would be more adapted to navigation automation or functional tests.</p>
<p>I chose <em>Mocha</em> since it seems to be well suited for unit tests and command-line usage. And since <a class="reference external" href="https://github.com/Leaflet/Leaflet/issues/1428">there is a pull-request</a> for switching from <em>Jasmine</em> to <em>Mocha</em> in Leaflet core... why not !</p>
</div>
<div class="section" id="first-run-the-suite">
<h2>First, run the suite</h2>
<p>Get your hand on the <tt class="docutils literal">npm</tt> command (comes with <tt class="docutils literal">nodejs</tt> package in Ubuntu)</p>
<p>Create a <tt class="docutils literal">package.json</tt> file with your application description. There are plenty of examples, just make sure you require the right stuff :</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="w"> </span><span class="s2">"name"</span><span class="o">:</span><span class="w"> </span><span class="s2">"yourapp"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"version"</span><span class="o">:</span><span class="w"> </span><span class="s2">"0.0.1"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"description"</span><span class="o">:</span><span class="w"> </span><span class="s2">"your app"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"main"</span><span class="o">:</span><span class="w"> </span><span class="s2">"yourapp.js"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"scripts"</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s2">"test"</span><span class="o">:</span><span class="w"> </span><span class="s2">"make test"</span><span class="p">,</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="s2">"dependencies"</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s2">"leaflet"</span><span class="o">:</span><span class="w"> </span><span class="s2">"*"</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="s2">"devDependencies"</span><span class="o">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s2">"mocha"</span><span class="o">:</span><span class="w"> </span><span class="s2">"*"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>And fetch !</p>
<pre class="literal-block">
npm install
</pre>
<p>Create <tt class="docutils literal">yourapp.js</tt> with simple and stupid stuff :</p>
<div class="highlight"><pre><span></span><span class="nx">L</span><span class="p">.</span><span class="nx">YourApp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">compute</span><span class="o">:</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mf">2</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>And create a test for it in <tt class="docutils literal">test/beginner.js</tt> :</p>
<div class="highlight"><pre><span></span><span class="c1">// Use require only if available (ran from Node)</span>
<span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="ow">typeof</span><span class="w"> </span><span class="nx">require</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s1">'function'</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">assert</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">require</span><span class="p">(</span><span class="s1">'assert'</span><span class="p">),</span>
<span class="w"> </span><span class="nx">L</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">require</span><span class="p">(</span><span class="s1">'leaflet/src/Leaflet'</span><span class="p">);</span>
<span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">YourApp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">require</span><span class="p">(</span><span class="s1">'./../yourapp'</span><span class="p">).</span><span class="nx">YourApp</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Test function call</span>
<span class="nx">describe</span><span class="p">(</span><span class="s1">'compute'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">it</span><span class="p">(</span><span class="s1">'should be ok'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">done</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="mf">2</span><span class="p">,</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">YourApp</span><span class="p">.</span><span class="nx">compute</span><span class="p">());</span>
<span class="w"> </span><span class="nx">done</span><span class="p">();</span>
<span class="w"> </span><span class="p">});</span>
<span class="p">});</span>
</pre></div>
<p>Admire the result !</p>
<pre class="literal-block">
@./node_modules/.bin/mocha
</pre>
</div>
<div class="section" id="make-it-run-in-the-browser-too">
<h2>Make it run in the browser too</h2>
<p>So far we do not rely too much on Leaflet :) But in a real application test, we will quickly need a <tt class="docutils literal">L.Map</tt> instance, along with a DOM most probably.</p>
<p>By turning on the <em>Mocha</em> HTML runner, we can indeed run tests from a web browser. But since the console remains one of our goals, we add <a class="reference external" href="https://github.com/metaskills/mocha-phantomjs/#readme">mocha-phantomjs</a> in the scene !</p>
<p>Install <tt class="docutils literal">phantomjs</tt> and add it to the <tt class="docutils literal">PATH</tt> (the Ubuntu package does that for you). Then modify your <tt class="docutils literal">package.json</tt> to add <tt class="docutils literal"><span class="pre">mocha-phantomjs</span></tt> as a <em>devDependency</em>. Re-run <tt class="docutils literal">npm install</tt> to fetch it.</p>
<p>With <em>mocha-phantomjs</em>, we will be able to run tests from within a browser <strong>and</strong> from the command-line. The entry point will be the following <tt class="docutils literal">test/index.html</tt>:</p>
<div class="highlight"><pre><span></span><span class="cp"><!DOCTYPE html></span>
<span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></span>Mocha<span class="p"></</span><span class="nt">title</span><span class="p">></span>
<span class="p"><</span><span class="nt">meta</span> <span class="na">http-equiv</span><span class="o">=</span><span class="s">"Content-Type"</span> <span class="na">content</span><span class="o">=</span><span class="s">"text/html; charset=UTF-8"</span><span class="p">></span>
<span class="p"><</span><span class="nt">meta</span> <span class="na">name</span><span class="o">=</span><span class="s">"viewport"</span> <span class="na">content</span><span class="o">=</span><span class="s">"width=device-width, initial-scale=1.0"</span><span class="p">></span>
<span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"../node_modules/mocha/mocha.css"</span> <span class="p">/></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">"mocha"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">"map"</span> <span class="na">style</span><span class="o">=</span><span class="s">"display: none; height: 300px"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"../node_modules/mocha/mocha.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"../node_modules/leaflet/debug/leaflet-include.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"../yourapp.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="s1">'map'</span><span class="p">).</span><span class="nx">fitWorld</span><span class="p">();</span>
<span class="w"> </span><span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span><span class="nx">mocha</span><span class="p">.</span><span class="nx">setup</span><span class="p">(</span><span class="s1">'bdd'</span><span class="p">)</</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"begginner.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span>
<span class="w"> </span><span class="p">(</span><span class="nb">window</span><span class="p">.</span><span class="nx">mochaPhantomJS</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="nb">window</span><span class="p">.</span><span class="nx">mocha</span><span class="p">).</span><span class="nx">run</span><span class="p">();</span>
<span class="w"> </span><span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</pre></div>
<p>Open the page locally or run in console with :</p>
<pre class="literal-block">
@./node_modules/mocha-phantomjs/bin/mocha-phantomjs test/index.html
</pre>
<p><em>PhantomJS</em> is installed by default on Travis by the way :)</p>
</div>
<div class="section" id="spying-and-mocking">
<h2>Spying and mocking</h2>
<p>One of the popular tools in JS testing is <a class="reference external" href="http://sinonjs.org">Sinon.js</a>. There are many useful features allowing to spy and mock behaviour of your application components or dependencies (events, AJAX requests, errors, timers, etc.)</p>
<p>For example, let's test that events are thrown as we expect :</p>
<div class="highlight"><pre><span></span><span class="nx">L</span><span class="p">.</span><span class="nx">YourApp</span><span class="p">.</span><span class="nx">snap</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">marker</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">fire</span><span class="p">(</span><span class="s1">'snap'</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>Test event with a <em>spy</em> callback :</p>
<div class="highlight"><pre><span></span><span class="nx">describe</span><span class="p">(</span><span class="s1">'snap'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">it</span><span class="p">(</span><span class="s1">'event is thrown'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">done</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">marker</span><span class="p">([</span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="mf">0</span><span class="p">]),</span>
<span class="w"> </span><span class="nx">callback</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">sinon</span><span class="p">.</span><span class="nx">spy</span><span class="p">();</span>
<span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'snap'</span><span class="p">,</span><span class="w"> </span><span class="nx">callback</span><span class="p">);</span>
<span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">YourApp</span><span class="p">.</span><span class="nx">snap</span><span class="p">(</span><span class="nx">marker</span><span class="p">);</span>
<span class="w"> </span><span class="nx">assert</span><span class="p">.</span><span class="nx">isTrue</span><span class="p">(</span><span class="nx">callback</span><span class="p">.</span><span class="nx">called</span><span class="p">);</span>
<span class="w"> </span><span class="nx">done</span><span class="p">();</span>
<span class="w"> </span><span class="p">});</span>
<span class="p">});</span>
</pre></div>
<p>Faking user inputs is also possible using <a class="reference external" href="https://github.com/tmcw/happen#readme">happen</a> :</p>
<div class="highlight"><pre><span></span><span class="nx">describe</span><span class="p">(</span><span class="s1">'zoom'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">it</span><span class="p">(</span><span class="s1">'zooms-in with double click'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">done</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">getZoom</span><span class="p">());</span>
<span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'zoomend'</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">assert</span><span class="p">.</span><span class="nx">equal</span><span class="p">(</span><span class="mf">1</span><span class="p">,</span><span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">getZoom</span><span class="p">());</span>
<span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">off</span><span class="p">(</span><span class="s1">'zoomend'</span><span class="p">);</span>
<span class="w"> </span><span class="nx">done</span><span class="p">();</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="c1">// Simulate double-click</span>
<span class="w"> </span><span class="nx">happen</span><span class="p">.</span><span class="nx">dblclick</span><span class="p">(</span><span class="nx">map</span><span class="p">.</span><span class="nx">_container</span><span class="p">);</span>
<span class="w"> </span><span class="p">});</span>
<span class="p">});</span>
</pre></div>
</div>
<div class="section" id="real-world-example">
<h2>Real world example</h2>
<p><a class="reference external" href="https://github.com/bbecquet">Benjamin Becquet</a> implemented <a class="reference external" href="https://github.com/bbecquet/Leaflet.PolylineDecorator">some linear referencing utilities</a> for Leaflet. So did we last year at <a class="reference external" href="http://makina-corpus.com">Makina Corpus</a> ! We thus decided to merge our code base in a proper way :)</p>
<p>We both are making our first steps with <em>Mocha</em>, and didn't really started to build up the whole code, but you still can have a look at <a class="reference external" href="https://github.com/makinacorpus/Leaflet.GeometryUtil">the repository</a>, for its Makefile, Travis setup, usage of JSDocs or Chai.js...</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>By the way, <em>Secrets of the Javascript Ninja</em> by John Resig and Bear Bibeault is a wonderful book !</td></tr>
</tbody>
</table>
</div>
Subtivals, ready for Cinelatino !2013-03-15T00:00:00+01:002013-03-15T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-03-15:/subtivals-ready-for-cinelatino.html<p>The <a class="reference external" href="http://www.cinelatino.com.fr">film festival CineLatino in Toulouse</a>
opens its doors today ! And I've just delivered the source package of version Subtivals 1.5.0
to the building robots of Launchpad !</p>
<img alt="" class="align-left" src="/images/symbol-deafness.png" />
<p>During the festival, Subtivals will not simply be used for captioning, it will be the
key tool for the <a class="reference external" href="http://www.cinelatino.com.fr/contenu/accessibilite-pour-les-sourds-2013">projection of …</a></p><p>The <a class="reference external" href="http://www.cinelatino.com.fr">film festival CineLatino in Toulouse</a>
opens its doors today ! And I've just delivered the source package of version Subtivals 1.5.0
to the building robots of Launchpad !</p>
<img alt="" class="align-left" src="/images/symbol-deafness.png" />
<p>During the festival, Subtivals will not simply be used for captioning, it will be the
key tool for the <a class="reference external" href="http://www.cinelatino.com.fr/contenu/accessibilite-pour-les-sourds-2013">projection of two movies specifically</a>,
for which some very rich deaf-specific subtitles have been prepared !</p>
<p>As the same time, we are receiving very positive feedback from users in Turkey,
Poland, Greece... thank you all !</p>
<div class="section" id="now-a-mac-os-x-demo">
<h2>Now a Mac OS X demo</h2>
<p>In addition to the <a class="reference external" href="http://mathieu-leplatre.info/media/subtivals/subtivals-setup-1.4.1-demo.exe">Windows demo version</a>, <a class="reference external" href="http://twitter.com/mallox">mallox</a>
packaged a <a class="reference external" href="http://mathieu-leplatre.info/media/subtivals/subtivals-setup-1.4.0-demo.dmg">Mac OS X automatic installer</a>.</p>
<img alt="" class="align-center" src="/images/subtivals-1.4-macos.png" />
</div>
<div class="section" id="whats-new-in-1-5">
<h2>What’s new in 1.5</h2>
<p>We mainly reworked the subtitle positioning system, in order to support
linespacing control, absolute and relative positions, using <tt class="docutils literal">PlayResX</tt> and <tt class="docutils literal">PlayResY</tt>.</p>
<p>See <a class="reference external" href="https://github.com/traxtech/subtivals/blob/master/debian/changelog">Changelog for more info...</a></p>
</div>
<div class="section" id="get-it">
<h2>Get it !</h2>
<p>Subtivals is released under <a class="reference external" href="http://www.gnu.org/copyleft/gpl.html">GPL</a>,
this means you can use, access the source code, modify, package, distribute the software,
as long as it remains GPL.</p>
<p>Like previous versions, we provide packages for Ubuntu. If you already
installed it, it will be updated automatically. Otherwise, it's just simple as :</p>
<pre class="literal-block">
sudo add-apt-repository ppa:mathieu.leplatre/subtivals
sudo apt-get update && sudo apt-get install subtivals
</pre>
<p>Regarding Mac OS and Windows, we still follow <a class="reference external" href="http://gcompris.net/-Download-">Bruno’s approach</a> :
we sell the installers on proprietary operating systems, in order to promote GNU/Linux.</p>
</div>
Django : Do not forget Do Not Track2013-03-01T00:00:00+01:002013-03-01T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-03-01:/django-do-not-forget-do-not-track.html<p>If you (fooly) Sail© on the Web® without <a class="reference external" href="http://www.ghostery.com/">Ghostery™</a>,
this website is currently tracking you. Sad news, but there are indeed 3 widgets
(Disqus, Piwik and Twitter on the About page) that may collect data about your presence here.</p>
<p>The <a class="reference external" href="http://www.mozilla.org/en-US/dnt/">Do-Not-Track</a> initiative consists in
avoiding user tracking, using an HTTP …</p><p>If you (fooly) Sail© on the Web® without <a class="reference external" href="http://www.ghostery.com/">Ghostery™</a>,
this website is currently tracking you. Sad news, but there are indeed 3 widgets
(Disqus, Piwik and Twitter on the About page) that may collect data about your presence here.</p>
<p>The <a class="reference external" href="http://www.mozilla.org/en-US/dnt/">Do-Not-Track</a> initiative consists in
avoiding user tracking, using an HTTP header, sent by the browser.
It is a voluntary process, and we should honour it when we can ! <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p>
<p>Here is a quick way of respecting privacy in your Django websites.</p>
<p>We use a context processor to spread the word accross all templates.</p>
<div class="highlight"><pre><span></span><span class="c1"># context_processors.py</span>
<span class="k">def</span> <span class="nf">donottrack</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s1">'donottrack'</span><span class="p">:</span> <span class="n">request</span><span class="o">.</span><span class="n">META</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'HTTP_DNT'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'1'</span>
<span class="p">}</span>
</pre></div>
<p>You can then disable spying widgets and tools, such as Google Analytics...</p>
<div class="highlight"><pre><span></span>{% load ganalytics %}
{% if not donottrack %}
{% ganalytics %}
{% endif %}
</pre></div>
<p>...or share buttons !</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">ul</span> <span class="na">class</span><span class="o">=</span><span class="s">"social-buttons cf"</span><span class="p">></span>
{% if donottrack %}
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"http://wikipedia.org/wiki/Do_Not_Track"</span><span class="p">></span>{% trans "Do-Not-Track is set." %}<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span>
{% else %}
<span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"//twitter.com/share"</span> <span class="na">class</span><span class="o">=</span><span class="s">"socialite twitter-share"</span> <span class="na">data-text</span><span class="o">=</span><span class="s">"{{ TITLE }} {{ URL }}"</span><span class="p">></span>
<span class="p"><</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">"vhidden"</span><span class="p">></span>{% trans "Twitter" %}<span class="p"></</span><span class="nt">span</span><span class="p">></</span><span class="nt">a</span><span class="p">></span>
<span class="p"></</span><span class="nt">li</span><span class="p">></span>
{% endif %}
<span class="p"></</span><span class="nt">ul</span><span class="p">></span>
</pre></div>
<p>We now need a middleware to add vary headers (for cache control), since content
depends on this header.</p>
<div class="highlight"><pre><span></span><span class="c1"># middleware.py</span>
<span class="kn">from</span> <span class="nn">django.utils.cache</span> <span class="kn">import</span> <span class="n">patch_vary_headers</span>
<span class="k">class</span> <span class="nc">DoNotTrackMiddleware</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">process_response</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
<span class="n">patch_vary_headers</span><span class="p">(</span><span class="n">response</span><span class="p">,</span> <span class="p">(</span><span class="s1">'DNT'</span><span class="p">,))</span>
<span class="k">return</span> <span class="n">response</span>
</pre></div>
<p>Add those to your <tt class="docutils literal">TEMPLATE_CONTEXT_PROCESSORS</tt> and <tt class="docutils literal">MIDDLEWARE_CLASSES</tt> settings and you're done.</p>
<p><strong>Update</strong> : There are reusable apps doing just that if you prefer : <a class="reference external" href="https://github.com/mozilla/django-dnt">django-dnt</a>,
<a class="reference external" href="https://github.com/benspaulding/django-donottrack/">django-donottrack</a>.</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>I wonder how I could do that with a static blog. Using headers-based rewrite condition ?</td></tr>
</tbody>
</table>
Subtivals 1.4 is out2013-01-08T00:00:00+01:002013-01-08T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2013-01-08:/subtivals-14-is-out.html<p>Since the <a class="reference external" href="/announcing-subtivals-realtime-subtitles-for-film-festivals.html">article introducing Subtivals</a> was
published, a lot of exciting things happened ! The first reward was the
<a class="reference external" href="http://www.cinelatino.com.fr/">festival of Cinelatino</a> in Toulouse, which had a major focus on accessibility this year.
Subtivals was used successfully : subtitles had never been so beautiful and refined,
and the public really acknowledged the …</p><p>Since the <a class="reference external" href="/announcing-subtivals-realtime-subtitles-for-film-festivals.html">article introducing Subtivals</a> was
published, a lot of exciting things happened ! The first reward was the
<a class="reference external" href="http://www.cinelatino.com.fr/">festival of Cinelatino</a> in Toulouse, which had a major focus on accessibility this year.
Subtivals was used successfully : subtitles had never been so beautiful and refined,
and the public really acknowledged the efforts !</p>
<p>Along the year we were in contact with people from Spain, Portugal, Canada,
Lebanon, Greece, Russia, Mexico... and the interest they showed in our tool
really made us cheerful and enthusiasts ! We had the proof that Subtivals
can be used in many contexts, from primary school performances to short-movie
festivals, through cinematheques and movie transcribers...</p>
<img alt="" class="align-center" src="/images/subtivals-arabic.png" />
<div class="section" id="subtivals-a-surtitling-captioning-subtitling-program">
<h2>Subtivals, a surtitling / captioning / subtitling program</h2>
<p>I usually have difficulties at explaining Subtivals goals, since its usage
is very specific to the world of movie projection.</p>
<p>In addition, the activity of projecting subtitles on top of a movie has many names (soft-titling, surtitling,
supertitling, electronic subtitles, virtual subtitles or even <a class="reference external" href="http://en.wikipedia.org/wiki/Closed_captioning">closed
captioning</a>...) making it harder for such
a tool to become visible on the Web.</p>
<p>Subtivals won’t transcribe sound and perform voice recognition ! It is a
projection software, with fancy subtitles and advanced positioning, necessary
to reproduce music, ambiance, dialogs and context for the hard of hearing.</p>
<p>Subtivals is very stable, very easy, very comfortable for the operator,
and particularly bullet-proof : subtitling should never fail in live!
Its particular strength lies in the control of the projection,
with a very good cooperation between automatic and manual modes.</p>
<img alt="" class="align-center" src="/images/subtivals-1.4dev-crop-win32.png" />
</div>
<div class="section" id="whats-new-in-1-4">
<h2>What’s new in 1.4</h2>
<p>Since the version 1.0, we improved a lot of things, and added many features
(see <a class="reference external" href="https://github.com/traxtech/subtivals/blob/master/debian/changelog">changelog</a>).</p>
<p>Among them :</p>
<ul class="simple">
<li>Subrip (SRT) support</li>
<li>Subtitles progress highlighting</li>
<li>Better interactions (shortcuts, scrolling)</li>
<li>Text outlining and fullscreen</li>
<li>Characters per second feedback</li>
<li>ASS absolute positioning support</li>
<li>Styles edition (colors, font, size, margins, alignments)</li>
<li>Better calibration screen</li>
<li>Mac OS version installation package</li>
</ul>
<p>From now, if you have a video mixer, with chroma-key support, then a separate
projector is not necessary for inlaying subtitles !</p>
<img alt="" class="align-center" src="/images/subtivals-chromakey.png" />
</div>
<div class="section" id="get-it">
<h2>Get it !</h2>
<p>Subtivals is released under <a class="reference external" href="http://www.gnu.org/copyleft/gpl.html">GPL</a>,
this means you can use, access the source code, modify, package, distribute the software,
as long as it remains GPL.</p>
<p>Like previous versions, we provide packages for Ubuntu. If you already
installed it, it will be updated automatically. Otherwise, it's just simple as :</p>
<pre class="literal-block">
sudo add-apt-repository ppa:mathieu.leplatre/subtivals
sudo apt-get update && sudo apt-get install subtivals
</pre>
<p>Regarding Mac OS and Windows, we still follow <a class="reference external" href="http://gcompris.net/-Download-">Bruno’s approach</a> :
we sell the installers on proprietary operating systems, in order to promote GNU/Linux.</p>
<p>But from now on, we also distribute a demo version ! <a class="reference external" href="http://mathieu-leplatre.info/media/subtivals/subtivals-setup-1.4.1-demo.exe">Download it
here</a> !</p>
</div>
Cheap debugging of PostgreSQL triggers in Django2012-10-22T12:25:00+02:002012-10-22T12:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-10-22:/cheap-debugging-of-postgresql-triggers-in-django.html<p>Lately, we were hacking on PostgreSQL (PostGIS) triggers, and we quickly felt
like debugging our code... Here is a cheap and quick way of printing out
triggers variables and context through Django.</p>
<div class="section" id="postgresql-server-configuration">
<h2>PostgreSQL server configuration</h2>
<p>In <em>postgresql.conf</em>, adjust the minimum level of notice sent to the client :</p>
<pre class="literal-block">
client_min_messages = log …</pre></div><p>Lately, we were hacking on PostgreSQL (PostGIS) triggers, and we quickly felt
like debugging our code... Here is a cheap and quick way of printing out
triggers variables and context through Django.</p>
<div class="section" id="postgresql-server-configuration">
<h2>PostgreSQL server configuration</h2>
<p>In <em>postgresql.conf</em>, adjust the minimum level of notice sent to the client :</p>
<pre class="literal-block">
client_min_messages = log
</pre>
<p>Note that this does not affect logging verbosity on server.</p>
</div>
<div class="section" id="catch-messages-in-django">
<h2>Catch messages in Django</h2>
<p>For a specific model :</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">connection</span>
<span class="k">def</span> <span class="nf">save</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">before</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">connection</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">notices</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="nb">super</span><span class="p">(</span><span class="n">Model</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">finally</span><span class="p">:</span>
<span class="k">for</span> <span class="n">notice</span> <span class="ow">in</span> <span class="n">connection</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">notices</span><span class="p">[</span><span class="n">before</span><span class="p">:]:</span>
<span class="nb">print</span> <span class="n">notice</span>
</pre></div>
<p>Or globally, using <tt class="docutils literal">post_save</tt> signals <em>(can be verbose)</em>:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">django.db</span> <span class="kn">import</span> <span class="n">connection</span>
<span class="kn">from</span> <span class="nn">django.db.models.signals</span> <span class="kn">import</span> <span class="n">post_save</span>
<span class="k">def</span> <span class="nf">show_notices</span><span class="p">(</span><span class="n">sender</span><span class="p">,</span> <span class="n">instance</span><span class="p">,</span> <span class="n">created</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">for</span> <span class="n">notice</span> <span class="ow">in</span> <span class="n">connection</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">notices</span><span class="p">:</span>
<span class="nb">print</span> <span class="n">notice</span>
<span class="n">post_save</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">show_notices</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="let-your-trigger-be-talkative">
<h2>Let your trigger be talkative</h2>
<p>You can basically print out values, arrays, functions results, records...</p>
<pre class="literal-block">
RAISE LOG '% has geom %', NEW.id, ST_AsEWKT(NEW.geom);
</pre>
<p>Will output something like <tt class="docutils literal">LOG: 3 has geom SRID=4326;POINT(0 0)</tt>.</p>
<pre class="literal-block">
FOR record IN SELECT * FROM table
LOOP
RAISE LOG 'Found %', record;
END LOOP;
</pre>
<p>Will output something like <tt class="docutils literal">LOG: Found (a,b,c)</tt>.</p>
<pre class="literal-block">
intersections_on_new := ARRAY[]::float[];
FOR pk IN SELECT ST_Line_Locate_Point(NEW.geom, (ST_Dump(ST_Intersection(other.geom, NEW.geom))).geom)
LOOP
intersections_on_new := array_append(intersections_on_new, pk);
END LOOP;
RAISE LOG 'Intersects at %', intersections_on_new;
</pre>
<p>Will output something like <tt class="docutils literal">LOG: Intersects at {0.5,0.3}</tt>.</p>
</div>
<div class="section" id="one-more-thing">
<h2>One more thing...</h2>
<p>If you load your triggers source file through Django (like a <tt class="docutils literal">post_migrate</tt> signal or so),
and thus with <em>psycopg2</em>, you might face that nasty internal quirck :</p>
<pre class="literal-block">
postgresql_psycopg2/base.py", line 52, in execute
return self.cursor.execute(query, args)
IndexError: tuple index out of range
</pre>
<p>This is due to <tt class="docutils literal">%</tt> characters, that you have to escape, replacing them with <tt class="docutils literal">%%</tt>.</p>
</div>
Leaflet Tiles in Lambert 93 projection (2154)2012-08-05T00:00:00+02:002012-08-05T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-08-05:/leaflet-tiles-in-lambert-93-projection-2154.html<div class="section" id="what-purpose">
<h2>What purpose ?</h2>
<p>Ideally, unless living in Lapland, we should not be bothered that much about maps projections !
Unfortunately, in practice, there are a few contexts in which you simply
can't avoid braving them. For example, if you must show a raster layer with texts, and don't control its production.</p>
<p>Reprojected …</p></div><div class="section" id="what-purpose">
<h2>What purpose ?</h2>
<p>Ideally, unless living in Lapland, we should not be bothered that much about maps projections !
Unfortunately, in practice, there are a few contexts in which you simply
can't avoid braving them. For example, if you must show a raster layer with texts, and don't control its production.</p>
<p>Reprojected pictures loose sharpness ! Your map may have to support the original
local projection.</p>
<p>Let's see what Leaflet offers on this matter...</p>
</div>
<div class="section" id="configuring-the-map">
<h2>Configuring the map</h2>
<p>My knowledge about local projections machinery is (very) light !</p>
<p>I could demystify a couple of concepts with, among others, Tom Mac Wright's <a class="reference external" href="http://macwright.org/2012/01/27/projections-understanding.html">educational</a>
<a class="reference external" href="http://macwright.org/2012/05/15/how-web-maps-work.html">articles</a> and <a class="reference external" href="http://macwright.org/2012/03/12/project-it-yourself.html">tools</a>, or <a class="reference external" href="http://blog.kartena.se/local-projections-in-a-world-of-spherical-mercator/">Kartena's article on Leaflet</a>.</p>
<p>Nevertheless, the whole thing is pretty straightforward to setup, especially when
you look at <a class="reference external" href="http://blog.thematicmapping.org/2012/07/using-custom-projections-with-tilecache.html">Bjorn Sandvik's working example</a> !</p>
<p>Download Leaflet, Proj4js and Proj4Leaflet, and fasten your seat belt !</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"leaflet.css"</span> <span class="p">/></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/leaflet.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/proj4js-compressed.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/proj4leaflet.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
</pre></div>
<p>Your layer characterics :</p>
<div class="highlight"><pre><span></span><span class="c1">// Your source tile layer extent, expressed in local projection</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">bbox</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mf">700000</span><span class="p">,</span><span class="w"> </span><span class="mf">6325197</span><span class="p">,</span><span class="w"> </span><span class="mf">1060000</span><span class="p">,</span><span class="w"> </span><span class="mf">6617738</span><span class="p">];</span>
<span class="c1">// Maximum resolution in meters per pixel (largest area side / tile size).</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">maxResolution</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">1406.25</span><span class="p">;</span>
<span class="c1">// Scale for each level</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">scale</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">zoom</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mf">1</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="p">(</span><span class="nx">maxResolution</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="nb">Math</span><span class="p">.</span><span class="nx">pow</span><span class="p">(</span><span class="mf">2</span><span class="p">,</span><span class="w"> </span><span class="nx">zoom</span><span class="p">));</span>
<span class="p">};</span>
<span class="c1">// Coordinate to grid transformation matrix</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">transformation</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">Transformation</span><span class="p">(</span><span class="mf">1</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="nx">bbox</span><span class="p">[</span><span class="mf">0</span><span class="p">],</span><span class="w"> </span><span class="o">-</span><span class="mf">1</span><span class="p">,</span><span class="w"> </span><span class="nx">bbox</span><span class="p">[</span><span class="mf">3</span><span class="p">]);</span>
</pre></div>
<p>Assemble in Leaflet :</p>
<div class="highlight"><pre><span></span><span class="c1">// Official Spatial Reference from http://www.spatialreference.org/ref/epsg/2154/</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">crs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">CRS</span><span class="p">.</span><span class="nx">proj4js</span><span class="p">(</span><span class="s1">'EPSG:2154'</span><span class="p">,</span>
<span class="w"> </span><span class="s1">'+proj=lcc +lat_1=49 +lat_2=44 +lat_0=46.5 +lon_0=3 +x_0=700000 +y_0=6600000 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">transformation</span><span class="p">);</span>
<span class="nx">crs</span><span class="p">.</span><span class="nx">scale</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">scale</span><span class="p">;</span><span class="w"> </span><span class="c1">// required by Leaflet 0.4</span>
<span class="c1">// Location of tiles (see next paragraph)</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">ignLayer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">TileLayer</span><span class="p">(</span><span class="s1">'http://localhost:8080/1.0.0/ign/{z}/{x}/{y}.png'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">continuousWorld</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="c1">// very important</span>
<span class="p">});</span>
<span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nb">Map</span><span class="p">(</span><span class="s1">'map'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">crs</span><span class="o">:</span><span class="w"> </span><span class="nx">crs</span><span class="p">,</span>
<span class="w"> </span><span class="nx">scale</span><span class="o">:</span><span class="w"> </span><span class="nx">scale</span><span class="p">,</span>
<span class="w"> </span><span class="nx">continuousWorld</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="c1">// very important</span>
<span class="w"> </span><span class="nx">layers</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="nx">ignLayer</span><span class="p">,]</span>
<span class="w"> </span><span class="c1">// Initial view</span>
<span class="w"> </span><span class="nx">center</span><span class="o">:</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">LatLng</span><span class="p">(</span><span class="mf">44.65</span><span class="p">,</span><span class="w"> </span><span class="mf">6.12</span><span class="p">),</span><span class="w"> </span><span class="c1">// Universal Lat/Lng</span>
<span class="w"> </span><span class="nx">zoom</span><span class="o">:</span><span class="w"> </span><span class="mf">5</span><span class="p">,</span>
<span class="p">});</span>
</pre></div>
<img alt="" class="align-center" src="/images/leaflet-l93.png" />
<p><a class="reference external" href="https://github.com/kartena/Proj4Leaflet/issues/8">kokoni</a> suggests to use
an overlay layer to debug tile numbering :</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">canvasTiles</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">tileLayer</span><span class="p">.</span><span class="nx">canvas</span><span class="p">();</span>
<span class="w"> </span><span class="nx">canvasTiles</span><span class="p">.</span><span class="nx">drawTile</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">canvas</span><span class="p">,</span><span class="w"> </span><span class="nx">tilePoint</span><span class="p">,</span><span class="w"> </span><span class="nx">zoom</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">ctx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">canvas</span><span class="p">.</span><span class="nx">getContext</span><span class="p">(</span><span class="s1">'2d'</span><span class="p">);</span>
<span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nx">strokeStyle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nx">fillStyle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"red"</span><span class="p">;</span>
<span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nx">rect</span><span class="p">(</span><span class="mf">0</span><span class="p">,</span><span class="mf">0</span><span class="p">,</span><span class="w"> </span><span class="mf">256</span><span class="p">,</span><span class="mf">256</span><span class="p">);</span>
<span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nx">stroke</span><span class="p">();</span>
<span class="w"> </span><span class="nx">ctx</span><span class="p">.</span><span class="nx">fillText</span><span class="p">(</span><span class="s1">'('</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">tilePoint</span><span class="p">.</span><span class="nx">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">', '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">tilePoint</span><span class="p">.</span><span class="nx">y</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">')'</span><span class="p">,</span><span class="mf">5</span><span class="p">,</span><span class="mf">10</span><span class="p">);</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">addLayer</span><span class="p">(</span><span class="nx">canvasTiles</span><span class="p">);</span>
</pre></div>
<img alt="" class="align-center" src="/images/leaflet-tiles-overlay.png" />
</div>
<div class="section" id="serve-the-tiles-from-a-wms">
<h2>Serve the tiles from a WMS</h2>
<p>Adapt with the <tt class="docutils literal">bbox</tt> and <tt class="docutils literal">maxesolution</tt>, as in the map, and save this configuration to <em>tilecache.cfg</em>.
It also assumes to match the <tt class="docutils literal">scale</tt> we chose above (divided by 2 at each level) :</p>
<pre class="literal-block">
[ign]
type=WMSLayer
layers=scan100,scan25
url=http://server.carto.fr/wms?
extension=jpg
tms_type=google
srs=EPSG:2154
bbox=700000,6325197,1060000,6617738
maxResolution=1142.7383
[cache]
type=GoogleDisk
base=/tmp/
</pre>
<p>In order to run this configuration, just install :</p>
<pre class="literal-block">
virtualenv .
source bin/activate
pip install TileCache
pip install Paste
</pre>
<p>And either run it locally with <tt class="docutils literal">tilecache_http_server.py</tt> or tile the pyramid once
with <tt class="docutils literal">tilecache_seed.py ign 0 10</tt>.</p>
<p>Finally, adjust the tile url in your map config !</p>
</div>
<div class="section" id="first-impressions">
<h2>First impressions...</h2>
<p>It works !</p>
<p>A couple of things seem implicit, and Leaflet feels a bit fragile with this. Changing
a value somewhere might break an hidden assumption somewhere else.</p>
<p>Also, the documentation can be misleading : it says to set <tt class="docutils literal">continuousWorld</tt> to <em>true</em> for
things that don't represent the world :)</p>
<p>We'll try to push Leaflet limits in the next days at <a class="reference external" href="http://makina-corpus.com">Makina Corpus</a>, especially 0.4 branch,
we'll keep posting if we face anything worth sharing :)</p>
</div>
landez : tiles post-processing2012-05-29T14:45:00+02:002012-05-29T14:45:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-05-29:/landez-tiles-post-processing.html<p>Some weeks ago, I started to refactor <a class="reference external" href="https://github.com/makinacorpus/landez">landez</a> (timidly).
But smart caching and post-processing of WMS maps were expected in my last project, so it gave me
a great boost : Landez 2.0 has landed ! :)</p>
<p>The base code is much clearer, and a few new features came out ! Among them …</p><p>Some weeks ago, I started to refactor <a class="reference external" href="https://github.com/makinacorpus/landez">landez</a> (timidly).
But smart caching and post-processing of WMS maps were expected in my last project, so it gave me
a great boost : Landez 2.0 has landed ! :)</p>
<p>The base code is much clearer, and a few new features came out ! Among them, the ability to
apply image filters to your maps !</p>
<div class="section" id="grayscale-conversion">
<h2>Grayscale conversion</h2>
<p>Not the funniest one of course, but quite handy to highlight map content !</p>
<img alt="" src="/images/landez-grayscale.jpg" />
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">landez</span> <span class="kn">import</span> <span class="n">MBTilesBuilder</span>
<span class="kn">from</span> <span class="nn">landez.filters</span> <span class="kn">import</span> <span class="n">GrayScale</span>
<span class="n">overlay</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">()</span>
<span class="n">overlay</span><span class="o">.</span><span class="n">add_filter</span><span class="p">(</span><span class="n">GrayScale</span><span class="p">())</span>
</pre></div>
</div>
<div class="section" id="color-to-alpha">
<h2>Color to Alpha</h2>
<p>If the tiles you overlay are mainly white, they might make your background layer brighter.
Therefore, adding a filter replacing white by transparent will nicely blend your
top layer without lightening the global result :</p>
<img alt="" src="/images/landez-overlay.jpg" />
<img alt="" src="/images/landez-blend.jpg" />
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">landez</span> <span class="kn">import</span> <span class="n">TilesManager</span><span class="p">,</span> <span class="n">ImageExporter</span>
<span class="kn">from</span> <span class="nn">landez.filters</span> <span class="kn">import</span> <span class="n">ColorToAlpha</span>
<span class="n">overlay</span> <span class="o">=</span> <span class="n">TilesManager</span><span class="p">(</span><span class="n">tiles_url</span><span class="o">=</span><span class="s1">'http://an.osm.mirror.org/</span><span class="si">{z}</span><span class="s1">/</span><span class="si">{x}</span><span class="s1">/</span><span class="si">{y}</span><span class="s1">.png'</span><span class="p">)</span>
<span class="n">overlay</span><span class="o">.</span><span class="n">add_filter</span><span class="p">(</span><span class="n">ColorToAlpha</span><span class="p">(</span><span class="s1">'#ffffff'</span><span class="p">))</span>
<span class="n">orthophoto</span> <span class="o">=</span> <span class="n">ImageExporter</span><span class="p">(</span><span class="n">wms_server</span><span class="o">=</span><span class="s1">'http://server/wms'</span><span class="p">,</span>
<span class="n">wms_layers</span><span class="o">=</span><span class="p">[</span><span class="s1">'orthophoto'</span><span class="p">])</span>
<span class="n">orthophoto</span><span class="o">.</span><span class="n">add_layer</span><span class="p">(</span><span class="n">overlay</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="continuous-integration">
<h2>Continuous Integration</h2>
<p>A <a class="reference external" href="http://travis-ci.org/#!/makinacorpus/landez">Travis job</a> was setup and allows
me to improve the testing strictness :)</p>
<p>The Travis configuration has some kind of magic, just drop one file and enable the hook in your Github repo :</p>
<pre class="literal-block">
language: python
python:
- 2.6
- 2.7
install:
- pip install Pillow
- python setup.py develop
script: python -m landez.tests
</pre>
</div>
<div class="section" id="next-steps">
<h2>Next steps...</h2>
<p>I hope to get the opportunity to develop new post-processing filters, as pretty as those
<a class="reference external" href="http://mapbox.com/blog/tilemill-compositing-operations-preview/">coming-up in MapBox Tilemill</a> !</p>
<p>PerryGeo wrote <a class="reference external" href="https://github.com/perrygeo/python-mbtiles">python-mbtiles</a>
which might be a good candidate for low-level access of MBTiles content.
I like the idea of a common python stack for reading and writing, it is not very clear to
me <a class="reference external" href="https://github.com/mapbox/mbtiles-spec/wiki/Implementations">which one</a> will emerge as the best one though...</p>
</div>
Serve your map layers with a usual Web hosting service2012-05-05T00:00:00+02:002012-05-05T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-05-05:/serve-your-map-layers-with-a-usual-web-hosting-service.html<p>Someone asked me about serving map tiles from a basic Web host. I agreed
to reply with a blog post, since it completes the stories I've been telling
in my last <a class="reference external" href="http://www.slideshare.net/makinacorpus/solutions-alternatives-google-maps-11501753">two</a>
<a class="reference external" href="/des-cartes-dun-autre-monde-la-suite-fr.html">talks</a>.</p>
<p>How to serve your map layers (<em>tiles</em>) with the simplest Apache or nginx ?</p>
<div class="section" id="get-the-mbtiles-file">
<h2>Get the MBTiles file …</h2></div><p>Someone asked me about serving map tiles from a basic Web host. I agreed
to reply with a blog post, since it completes the stories I've been telling
in my last <a class="reference external" href="http://www.slideshare.net/makinacorpus/solutions-alternatives-google-maps-11501753">two</a>
<a class="reference external" href="/des-cartes-dun-autre-monde-la-suite-fr.html">talks</a>.</p>
<p>How to serve your map layers (<em>tiles</em>) with the simplest Apache or nginx ?</p>
<div class="section" id="get-the-mbtiles-file">
<h2>Get the MBTiles file</h2>
<p>We start from a tiles package (<em>MBTiles</em>). Depending on where your layers come
from, you can either choose :</p>
<div class="section" id="to-publish-your-tilemill-map">
<h3>To publish your Tilemill map</h3>
<p>Design your map in Tilemill, even <a class="reference external" href="http://mapbox.com/tilemill/docs/guides/osm-bright-ubuntu-quickstart/">your own OpenStreetMap style customization</a>,
and export it as MBTiles !
<a class="reference external" href="http://mapbox.com/">MapBox hosting</a> is the prefered solution, but you can still host and
serve your exported MBTiles file yourself !</p>
</div>
<div class="section" id="to-mirror-a-wms-server">
<h3>To mirror a WMS server</h3>
<p>Again, using <a class="reference external" href="/landez-introducing-new-features-of-our-tiles-toolbox.html">landez</a>, you can build a MBTiles
file from a WMS source (<em>orthophoto</em>...), and then serve those layers yourself as tiles (at the speed of light !).</p>
</div>
<div class="section" id="to-mirror-tiles-services">
<h3>To mirror tiles services</h3>
<p>Using <a class="reference external" href="/landez-introducing-new-features-of-our-tiles-toolbox.html">landez</a>, you
can gather tiles from a tiles service, and package them in a <tt class="docutils literal">.mbtiles</tt>
file locally.</p>
<p>If you plan on mirroring public tile services, do not forget to add attributions
and <strong>make sure it respects the terms of service</strong>. Most services restrict bulk downloads <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>.</p>
</div>
</div>
<div class="section" id="extract-files-on-disk">
<h2>Extract files on disk</h2>
<p>Using <a class="reference external" href="https://github.com/mapbox/mbutil">mbutil</a>, we can extract the <tt class="docutils literal">.mbtiles</tt>
file into a destination folder.</p>
<p>Unfortunately, the <em>pypi</em> mirror is quite old, we'll install the last development version.</p>
<div class="highlight"><pre><span></span>wget<span class="w"> </span>https://github.com/mapbox/mbutil/zipball/master<span class="w"> </span>-O<span class="w"> </span>mbutil.zip
unzip<span class="w"> </span>mbutil
<span class="nb">cd</span><span class="w"> </span>mapbox-mbutil*
sudo<span class="w"> </span>python<span class="w"> </span>setup.py<span class="w"> </span>install
</pre></div>
<p>Done. Now extract. (<em>Note that the ``DEST`` folder must not exist</em>) :</p>
<div class="highlight"><pre><span></span>mb-util<span class="w"> </span>--scheme<span class="o">=</span>osm<span class="w"> </span>FILE.mbtiles<span class="w"> </span>/path/to/DEST/
</pre></div>
<p>If your MBTiles has an interaction layer (<em>UTFGrid</em>), both <tt class="docutils literal">.png</tt> and <tt class="docutils literal">.json</tt>
files will be expanded in folders.</p>
<p>Just push the folder to your hosting, and you're done !</p>
<div class="section" id="cache-headers">
<h3>Cache headers</h3>
<p>If you the master on board, tweak the cache headers :</p>
<p>With <em>Apache</em> :</p>
<pre class="literal-block">
ExpiresActive On
ExpiresDefault "access plus 7 days"
Alias /DEST /path/to/DEST/;
</pre>
<p>With <em>nginx</em> :</p>
<pre class="literal-block">
server {
location /DEST {
expires 7d;
alias /path/to/DEST/;
}
}
</pre>
</div>
<div class="section" id="boost-with-subdomains">
<h3>Boost with subdomains</h3>
<p>Browsers limit parallel downloads on the same domain. If you can declare
subdomains (<em>a.yourserver.org</em>, <em>b.yourserver.org</em>, ...), it will speed-up
tiles download.</p>
</div>
</div>
<div class="section" id="use-it-in-your-mapping-library">
<h2>Use it in your mapping library</h2>
<p>With <a class="reference external" href="http://leaflet.cloudmade.com">Leaflet</a> for example :</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nb">Map</span><span class="p">(</span><span class="s1">'map'</span><span class="p">);</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">addLayer</span><span class="p">(</span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">TileLayer</span><span class="p">(</span><span class="s1">'http://{s}.yourserver.org/DEST/{z}/{x}/{y}.png'</span><span class="p">));</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">setView</span><span class="p">(</span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">LatLng</span><span class="p">(</span><span class="mf">43.60</span><span class="p">,</span><span class="w"> </span><span class="mf">1.45</span><span class="p">),</span><span class="w"> </span><span class="mf">14</span><span class="p">)</span>
</pre></div>
<p>Or <a class="reference external" href="http://modestmaps.com">Modestmaps</a> :</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">provider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">MM</span><span class="p">.</span><span class="nx">TemplatedLayer</span><span class="p">(</span><span class="s1">'http://{s}.yourserver.org/DEST/{z}/{x}/{y}.png'</span><span class="p">);</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">MM</span><span class="p">.</span><span class="nb">Map</span><span class="p">(</span><span class="s1">'map'</span><span class="p">,</span><span class="w"> </span><span class="nx">provider</span><span class="p">);</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">setCenter</span><span class="p">({</span><span class="nx">lat</span><span class="o">:</span><span class="w"> </span><span class="mf">43.60</span><span class="p">,</span><span class="w"> </span><span class="nx">lon</span><span class="o">:</span><span class="w"> </span><span class="mf">1.45</span><span class="p">}).</span><span class="nx">setZoom</span><span class="p">(</span><span class="mf">14</span><span class="p">);</span>
</pre></div>
<p>It will also work with interaction layers if you use <a class="reference external" href="http://mapbox.com/wax/">Wax</a> :)</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>MapBox <a class="reference external" href="http://mapbox.com/tos/">strictly forbids</a> proxying and further distribution of their tiles.
Bulk downloading from OpenStreetMap.org <a class="reference external" href="http://wiki.openstreetmap.org/wiki/Tile_usage_policy#Bulk_Downloading">is strongly discouraged too</a>.
And mass downloads <a class="reference external" href="http://support.cloudmade.com/answers/offline-maps">cost money on Cloudmade</a>.</td></tr>
</tbody>
</table>
</div>
Simple and funky Web map printing2012-05-02T00:00:00+02:002012-05-02T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-05-02:/simple-and-funky-web-map-printing.html<p>Strangely, users still insist in having Web page print capabilities, mostly to
share, export or archive what they see. Even if relevant permalinks are
often acceptable, we can't always dissuade them from printing :)</p>
<p>And when it comes to Web maps, printing can be a nightmare ! Even though most of the …</p><p>Strangely, users still insist in having Web page print capabilities, mostly to
share, export or archive what they see. Even if relevant permalinks are
often acceptable, we can't always dissuade them from printing :)</p>
<p>And when it comes to Web maps, printing can be a nightmare ! Even though most of the time,
the needs involve a nice landscape PDF file with the map, a legend and the company logo.
For this, designing CSS print stylesheets and using the browser <em>print</em> shortcut is not always
helpful.</p>
<p>I present here a simple solution <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> based on headless Webkit screenshots, permalinks,
SVG templates (<em>WYSIWYG</em>), and PDF conversion.</p>
<div class="section" id="what-you-see-is-what-you-print">
<h2>What You See Is What You Print</h2>
<p>In the following (short) video, we can see :</p>
<ul class="simple">
<li>a map with dynamic content (<em>GeoJSON</em>), bound to a form for attribute filtering and a legend
refreshed upon data ranges modifications ;</li>
<li>a <em>Print</em> button that delivers the current view as <em>PDF</em> ;</li>
<li>a landscape printout in which the map view, the legend, the filter form values were nicely inserted.</li>
</ul>
<video id="webprint" width="560" controls="controls">
<source src="http://mathieu-leplatre.info/media/20120501-print.ogv" type="video/ogg" />
Your browser does not support the video tag.
</video><p>(<em>BTW, small boo-boo in last screen: 'montain' instead of 'mountain'</em>)</p>
</div>
<div class="section" id="kids-you-can-do-this-at-home">
<h2>Kids, you can do this at home</h2>
<p>Here is how we did it :</p>
<ul class="simple">
<li>a Web page with a "stateful" permalink (<em>i.e. restore the map and page state using anchors, location hash, etc. ;</em>).
Backbone.js & co. are meant for this ;</li>
<li>A color scale built client-side from the resulting dataset, using <a class="reference external" href="https://github.com/gka/chroma.js">Chroma.js</a>,
to colorize the map items and populate the legend entries ;</li>
<li><a class="reference external" href="https://github.com/makinacorpus/django-screamshot">django-screamshot</a>, a Web page
screenshot application, relying on <a class="reference external" href="http://casperjs.org/">CasperJS</a>. Spooky! ;</li>
<li>a SVG landscape A4 document, edited with <em>Inkscape</em>, as a Django template in which we placed simple tags (<tt class="docutils literal">{{ filter.age_min }}</tt>) for texts,
the <tt class="docutils literal">{% base64capture %}</tt> tag for the map screenshot, and a couple of arithmetics tags to
recreate a nice vectorial legend using the color scale entries;</li>
<li>a Django view that receives the current page context (posted in <em>JS</em>), renders the SVG (<em>will thus perform the screenshot</em>),
and converts it to PDF ;</li>
</ul>
<p>Quite straightforward, a couple of hours to put together, relatively easy
to deploy, obviously meet most users needs... these hacks are our happiness !</p>
<p>If you want to know more about some missing parts, feel free to ask ! I could release stuff or just post some snippets...</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>from now on, I shall precise : even if it can cover most needs, it won't be adequate in all situations.</td></tr>
</tbody>
</table>
</div>
Des cartes d'un autre monde, la suite2012-04-18T00:00:00+02:002012-04-18T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-04-18:/des-cartes-dun-autre-monde-la-suite-fr.html<div class="section" id="bravo-aux-organisateurs">
<h2>Bravo aux organisateurs !</h2>
<p>L'édition 2012 des rencontres Django-fr a mis la barre très haut ! Ce fut un plaisir
de retrouver, ou rencontrer, autant de gens sympathiques et intéressants dans un cadre
aussi agréable :)</p>
<p>Comme l'a souligné Olivier <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>, cela va bien au delà de la techno qui fédère la communauté …</p></div><div class="section" id="bravo-aux-organisateurs">
<h2>Bravo aux organisateurs !</h2>
<p>L'édition 2012 des rencontres Django-fr a mis la barre très haut ! Ce fut un plaisir
de retrouver, ou rencontrer, autant de gens sympathiques et intéressants dans un cadre
aussi agréable :)</p>
<p>Comme l'a souligné Olivier <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a>, cela va bien au delà de la techno qui fédère la communauté ;
les membres partagent aussi un esprit, une vision, des approches, qui transpassent
l'outil ! Agilité, pragmatisme, KISS, DRY, PEP20...</p>
<p>Au menu, les problématiques de scaling étaient prédominantes, Django propulse des sites à
gros volume, comme <em>Liberation.fr</em>, <em>20minutes.fr</em>, <em>Mozilla</em>, <em>Autolib</em>, représentés
pendant ces rencontres, mais aussi <em>Instagram</em>, <em>Lanyrd</em>, <em>Disqus</em>... autant d'expériences à partager!
Ce fut aussi un honneur d'accueillir deux invités nord-américains, python Lords chez <em>Heroku</em>.</p>
<p>Cette année, je présentais une approche à contre-courant pour publier des données
cartographiques sur le Web :</p>
<blockquote>
“ Comment publier des données cartographiques, aussi simplement qu'on
publie une image ? Il existe un monde où Django expose lui-même des
cartes interactives, sans déployer l'artillerie habituelle !
Une approche à contre-courant se prêtant parfaitement à la mise en
valeur quasi-immédiate d'informations geographiques, comme celles
libérées par votre ville ! ”</blockquote>
<iframe src="http://www.slideshare.net/slideshow/embed_code/12698176" width="560" height="432" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe><p><strong>Pour résumer</strong> : éloignons-nous de l'OGC (WMS, WFS,...) quand il s'agit
juste de publier une carte sur une page Web.</p>
</div>
<div class="section" id="comment-rafraichir-les-cartes">
<h2>Comment rafraîchir les cartes ?</h2>
<p>Je reprends une citation remontée par <em>twidi</em> :</p>
<blockquote>
“ There are only two hard things in Computer Science: cache invalidation and naming things ”
-- Phil Karlton</blockquote>
<p>Selon la fréquence de rafraichissement de vos données, plusieurs stratégies sont
envisageables. Surtout qu'il est rare que l'ensemble de la carte nécessite d'être actualisé.</p>
<div class="section" id="en-generant-les-tuiles-en-temps-reel">
<h3>En générant les tuiles en temps réel</h3>
<p>Comme l'expliquait Young Yahn <a class="footnote-reference" href="#footnote-2" id="footnote-reference-2">[2]</a>, générer des tuiles en temps réel demande
d'avoir une architecture assez trappue et cela s'avère stressant à administrer.</p>
<p>Cependant c'est possible avec des outils comme <em>tilelive</em> ou <em>renderd</em> (Apache <em>mod_tile</em>).</p>
</div>
<div class="section" id="avec-une-tache-planifiee">
<h3>Avec une tâche planifiée</h3>
<p>Il faut trouver le compromis entre le temps de fabrication de la carte et la fréquence
de rafraichissement des données source.</p>
<p>S'il s'agit de minutes, rafraichir la carte toutes les heures semble envisageable. La plupart du
temps, une fois par jour suffira.</p>
<p>Il existe plusieurs outils, comme <a class="reference external" href="/render-your-tilemill-stylesheets-with-landez.html">landez</a>,
pour regénérer à intervales réguliers votre carte issue de <em>Tilemill</em>.</p>
</div>
<div class="section" id="a-la-seconde">
<h3>À la seconde</h3>
<p>Pour suivre une flotte de bateaux ou de véhicules, il faut que les éléments soient
déplacés en temps réel sur la carte.</p>
<p>Il y a plusieurs outils très efficaces qui se reposent sur les Websockets. J'avais
fait l'application <a class="reference external" href="/des-cartes-collaboratives-avec-livetitude-fr.html">Livetitude</a>,
<a class="reference external" href="http://vivid-warrior-6693.herokuapp.com">disponible en ligne</a>, qui permet
d'éditer à plusieurs une carte de marqueurs, grace à <em>Pusher</em>.</p>
<p>La bibliothèque <em>Sharejs</em>, issue du projet défunt Google Wave, permettrait d'aller
plus loin en faisant de l'édition collaborative d'attributs de géométries.</p>
</div>
<div class="section" id="en-fonction-de-filtres-ou-formulaires">
<h3>En fonction de filtres ou formulaires</h3>
<p>Pour redessiner la carte en fonction de filtres, sur des attributs par exemple, l'utilisation
du format GeoJSON s'avère assez efficace.</p>
<p>Votre serveur reçoit le formulaire, construit le jeu de données, et renvoie les
résultats (<em>Features = geometries + attributs</em>).</p>
<p>Cette approche peut s'avérer délicate selon la taille des jeux de données. Plusieurs
ruses existent afin de limiter le volume (ex: généralisation progressive selon la zone affichée)</p>
</div>
</div>
<div class="section" id="carte-a-echelle-unique">
<h2>Carte à échelle unique</h2>
<p>Parfois, pour certaines cartes, une seule vue suffit ! Nul besoin de zoomer, puisque le
phénomène intervient à une échelle en particulier !</p>
<p>Pensez aux cartes des journaux ! Et vous serez séduits par l'excellent <em>Kartograph</em>,
qui permet de publier des cartes sublimes facilement. Le SVG est manipulable en Javascript,
et permet d'ajouter des évènements sur les zones.</p>
<p>Martin Dewulf a publié <a class="reference external" href="http://migrationsmap.net">une jolie carte interactive à partir de données ouvertes</a>.
Le résultat est très convaincaint, et sort de l'ordinaire.</p>
</div>
<div class="section" id="requiem-pour-les-trolls">
<h2>Requiem pour les trolls</h2>
<p>La citation au début de la présentation, issue de <em>#whereconf</em>, était volontairement
provocatrice. Mais de nombreux acteurs du Web et de la cartographie rejoignent
cette idée. Par exemple, entre temps, Sean Gillies a réitéré :</p>
<blockquote>
“ How many MapBox and CartoDB like products would there be today if the
Open Source GIS community hadn't gone on a decade long WxS wander? “
-- @sgillies, 2012</blockquote>
<div class="section" id="oriente-communication">
<h3>Orienté communication</h3>
<p>En 12 minutes, c'est très difficile de présenter tous les aspects, inconvénients
et avantages d'une approche à contre-courant !</p>
<p>J'ai présenté le besoin plus simple de la cartographie sur le Web :</p>
<ul class="simple">
<li>j'ai des données à caractère géographique ;</li>
<li>je veux les afficher sur une page Web avec une carte interactive.</li>
</ul>
<p>Le cas le plus simple, mais en même temps le plus répandu !</p>
</div>
<div class="section" id="pas-toujours-d-alternatives-a-l-ogc">
<h3>Pas toujours d'alternatives à l'OGC</h3>
<p>Dans certains contextes, les protocoles OGC sont indispensables :</p>
<ul class="simple">
<li>interroperabilité entre systèmes hétérogènes sans médiation préalable (<em>serveurs externes, logiciels propriétaires, etc.</em>);</li>
<li>catalogage sémantique et syndication des jeux de données (<em>INSPIRE</em>)</li>
<li>construction d'une plateforme IDS</li>
</ul>
<p><em>OpenLayers</em> est la seule bibliothèque javascript qui a les reins assez solides pour
s'intégrer dans ces environnements.</p>
</div>
<div class="section" id="savoir-oublier-le-web">
<h3>Savoir oublier le Web</h3>
<p>Il faut savoir juger la pertinence d'une application Web. Dans certaines situations,
le Web n'est pas la seule solution pour faire du client-server en multi-utilisateurs.</p>
<p>L'année dernière par exemple, nous avons développé une application collaborative
pour manipuler des tronçons routiers. Nous avons <a class="reference external" href="/merkopolo-a-simple-yet-powerful-starter-kit-for-your-qtc-gis-application.html">choisi C++/Qt</a>,
avec PostGIS et des <a class="reference external" href="/access-a-json-webservice-with-qt-c.html">webservices JSON</a>,
parce que c'est ce qui se prêtait le mieux au besoin ! La même chose en <em>ExtJS</em> aurait
été catastrophique !</p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/7NPQo54NbJ8" frameborder="0" allowfullscreen></iframe><table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td><a class="reference external" href="https://twitter.com/#!/oloynet/status/192295759431995393">oloynet</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="footnote-2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-2">[2]</a></td><td><a class="reference external" href="http://mapbox.com/blog/rendering-the-world/">Rendering the World, FOSS4G NA, 2012</a></td></tr>
</tbody>
</table>
</div>
</div>
Render your TileMill stylesheets with Landez2012-03-22T00:00:00+01:002012-03-22T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-03-22:/render-your-tilemill-stylesheets-with-landez.html<p><a class="reference external" href="http://mapbox.com/tilemill/">TileMill</a> is an amazing tool to design your map, and publish it.
With <a class="reference external" href="http://pypi.python.org/pypi/landez">landez</a>, you can easily render it using python,
or do <a class="reference external" href="/landez-introducing-new-features-of-our-tiles-toolbox.html">whatever comes with the API</a> !</p>
<div class="section" id="from-tilemill-to-landez">
<h2>From TileMill to Landez</h2>
<p>Use Tilemill to design your map, and export the <a class="reference external" href="http://mapnik.org">Mapnik</a> XML stylesheet :</p>
<img alt="" src="/images/tilemill-export-stylesheet.png" />
<p>Then simply use <em>landez</em> with <tt class="docutils literal">stylefile …</tt></p></div><p><a class="reference external" href="http://mapbox.com/tilemill/">TileMill</a> is an amazing tool to design your map, and publish it.
With <a class="reference external" href="http://pypi.python.org/pypi/landez">landez</a>, you can easily render it using python,
or do <a class="reference external" href="/landez-introducing-new-features-of-our-tiles-toolbox.html">whatever comes with the API</a> !</p>
<div class="section" id="from-tilemill-to-landez">
<h2>From TileMill to Landez</h2>
<p>Use Tilemill to design your map, and export the <a class="reference external" href="http://mapnik.org">Mapnik</a> XML stylesheet :</p>
<img alt="" src="/images/tilemill-export-stylesheet.png" />
<p>Then simply use <em>landez</em> with <tt class="docutils literal">stylefile</tt> argument :</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">landez</span> <span class="kn">import</span> <span class="n">MBTilesBuilder</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="n">mb</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">(</span><span class="n">stylefile</span><span class="o">=</span><span class="s2">"Toulouse-Voirie.xml"</span><span class="p">,</span>
<span class="n">filepath</span><span class="o">=</span><span class="s2">"toulouse-voirie.mbtiles"</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">add_coverage</span><span class="p">(</span><span class="n">bbox</span><span class="o">=</span><span class="p">[</span><span class="mf">1.39</span><span class="p">,</span> <span class="mf">43.56</span><span class="p">,</span> <span class="mf">1.50</span><span class="p">,</span> <span class="mf">43.64</span><span class="p">],</span>
<span class="n">zoomlevels</span><span class="o">=</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="mi">18</span><span class="p">))</span>
<span class="n">mb</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
<p>In the above example, a <em>MBTiles</em> file <tt class="docutils literal"><span class="pre">toulouse-voirie.mbtiles</span></tt> will be
created with all rendered tiles. (<strong>Note:</strong> This won't render UTF-Grid tiles,
since TileMill does not expose this part in the XML stylesheet.)</p>
<p>If you don't have Mapnik2 installed, you might encounter rendering errors
like : <tt class="docutils literal">AssertionError: Cannot render tiles without mapnik !</tt>.</p>
</div>
<div class="section" id="installation-of-mapnik-2">
<h2>Installation of Mapnik 2</h2>
<div class="section" id="mapnik2-packages-on-debian-ubuntu">
<h3>Mapnik2 packages on Debian/Ubuntu</h3>
<p>In Ubuntu Precise (12.04) or Debian Wheezy (7.0), it's a piece of cake,
the package is available in the repos</p>
<pre class="literal-block">
sudo apt-get install python-mapnik2
</pre>
<p>In Ubuntu Maverick (10.10), Natty (11.04), Oneiric (11.10), it's quite easy,
there is a PPA, from MapBox</p>
<pre class="literal-block">
sudo apt-add-repository ppa:developmentseed/mapbox
sudo update
sudo apt-get install python-mapnik2
</pre>
</div>
<div class="section" id="mapnik2-and-python-bindings-from-sources">
<h3>Mapnik2 and python bindings from sources</h3>
<p>Welcome in the quicksands of installing Mapnik2 python bindings from sources !</p>
<p><a class="reference external" href="https://github.com/cspanring">Christian Spanring</a> wrote a quick tutorial
to <a class="reference external" href="https://gist.github.com/1314907">install it from sources on Ubuntu 10.04</a>.</p>
<p>It might be a bit tricky to tweak this tutorial for your distribution. Hopefully, our
colleague <a class="reference external" href="https://github.com/kiorky">Mathieu</a> has prepared a <a class="reference external" href="http://minitage.org">minitage's</a> "<em>minilay</em>" for it, <a class="reference external" href="http://pypi.python.org/pypi/mapnik2#minitage">just follow the few steps</a>
to compile the whole stack.</p>
</div>
</div>
Announcing Subtivals, realtime subtitles for film festivals2012-03-17T00:00:00+01:002012-03-17T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-03-17:/announcing-subtivals-realtime-subtitles-for-film-festivals.html<img alt="" src="/images/subtivals-logo.png" />
<div class="section" id="that-s-how-it-started">
<h2>That's how it started...</h2>
<p>Subtitles and captions for the deaf and hard-of-hearing during film festivals
are rarely a priority, at least in France. Thus, most film copies are not
subtitled (except foreign movies shown in their original version).</p>
<p>At the beginning of last year, <a class="reference external" href="http://st2l.fr">my friend Lilian</a> was promoting his …</p></div><img alt="" src="/images/subtivals-logo.png" />
<div class="section" id="that-s-how-it-started">
<h2>That's how it started...</h2>
<p>Subtitles and captions for the deaf and hard-of-hearing during film festivals
are rarely a priority, at least in France. Thus, most film copies are not
subtitled (except foreign movies shown in their original version).</p>
<p>At the beginning of last year, <a class="reference external" href="http://st2l.fr">my friend Lilian</a> was promoting his project
of improving and facilitating the projection of subtitles during film festivals.
His activity consists in superimposing subtitles or projecting them on a
separate display below the screen.</p>
<img alt="" class="align-center" src="/images/subtivals-superimposed.png" />
<p>In this way, subtitling dissociates from the movie reel, unlike DVDs for example.
An operator is thence in charge of keeping captions synchroneous. It is sometimes called
"<em>virtual subtitling</em>"; it is cheaper and easier than subtitled hard copies.</p>
<p>Since there was no suitable Open Source tools to fulfill these precise needs,
<a class="reference external" href="http://gedial.com">Arnaud</a> and I gave him a hand. We developed <a class="reference external" href="https://github.com/traxtech/subtivals">Subtivals</a>,
a Free Software with simplicity and usuability for the technical operator in mind.</p>
<img alt="" class="align-center" src="/images/subtivals-screenshot.png" />
</div>
<div class="section" id="main-features">
<h2>Main features</h2>
<p><em>Subtivals</em> has gained many features on the way, we released the <strong>version 1.0</strong> last month,
after almost a year of development, driven by Lilian's experience.</p>
<p>Among most notable features :</p>
<ul class="simple">
<li>Support of Advanced SubStation Alpha subtitles (ASS, *.ass) format</li>
<li>Control Play / Pause / Delay / Speed</li>
<li>Switch between several modes : timecode based, semi-automatic or fully manual</li>
<li>Support for subtitles without timecodes (fixes duration automatically)</li>
<li>SSA styles (italic, positions, colors)</li>
<li>Customize and override styles (color and font size)</li>
</ul>
<p>In the small world of subtitles projection, where most tools are either
very expensive, or very archaic, <em>Subtivals</em> is a revolution !</p>
</div>
<div class="section" id="hall-of-fame">
<h2>Hall of fame</h2>
<p><em>Subtivals</em> has proven its efficiency for several months now ! It was used
successfully in many film festivals (<a class="reference external" href="http://www.festival-douarnenez.com">Festival de cinéma de Douarnenez</a>,
<a class="reference external" href="http://festival-galactique.infini.fr">Festival Intergalactique de Brest</a>,
<a class="reference external" href="http://www.festivaldebiarritz.com">Festival de Cinéma d'Amérique Latine de Biarritz</a>,
<a class="reference external" href="http://www.filmfestamiens.org">Festival International du film d'Amiens</a>,
<a class="reference external" href="http://www.lacinemathequedetoulouse.com/archives/2012/thematiques">Festival Zoom Arrière, 6e édition</a>,
... and soon :
<a class="reference external" href="http://www.cinelatino.com.fr">Cinélatino de Toulouse</a>,
<a class="reference external" href="http://festival-resistances.fr">Résistances à Foix</a>)</p>
<p>Its semi-manual mode also allows <a class="reference external" href="http://en.wikipedia.org/wiki/Surtitle">surtitling</a> at theaters (or opera, ballets, ...).</p>
</div>
<div class="section" id="installation">
<h2>Installation</h2>
<p><em>Subtivals</em> runs on GNU/Linux, Windows, Mac OS X and has no other external dependencies.</p>
<p>On Ubuntu, install it easily using our PPA :</p>
<pre class="literal-block">
sudo add-apt-repository ppa:mathieu.leplatre/subtivals
sudo apt-get update && sudo apt-get install subtivals
</pre>
<p>Mac OS X and Windows installers are for sale, contact us !</p>
</div>
<div class="section" id="contribute">
<h2>Contribute</h2>
<p><em>Subtivals</em> is written in C++/Qt and released under the <a class="reference external" href="http://www.gnu.org/copyleft/gpl.html">GNU General Public License</a> .
It is available in English, French, Spanish and Catalan.</p>
<p>If you feel like contributing, testing, translating... <a class="reference external" href="https://github.com/traxtech/subtivals">join us on Github</a> !</p>
</div>
Django Handlebars.js integration2012-03-17T00:00:00+01:002012-03-17T00:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-03-17:/django-handlebarsjs-integration.html<p>In order to write <a class="reference external" href="http://handlebarsjs.com/">Handlebars.js</a> templates
in <a class="reference external" href="http://djangoproject.com">Django</a> templates, I was gonna copy and paste for the second time
<a class="reference external" href="https://gist.github.com/893408">Miguel Araujo's verbatim snippet</a>.
But since one of the <em>Django</em> weakness is the lack of reusable applications, I thought
I would package one instead :)</p>
<p>The two existing django applications <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1 …</a></p><p>In order to write <a class="reference external" href="http://handlebarsjs.com/">Handlebars.js</a> templates
in <a class="reference external" href="http://djangoproject.com">Django</a> templates, I was gonna copy and paste for the second time
<a class="reference external" href="https://gist.github.com/893408">Miguel Araujo's verbatim snippet</a>.
But since one of the <em>Django</em> weakness is the lack of reusable applications, I thought
I would package one instead :)</p>
<p>The two existing django applications <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a> that integrate <em>Handlebars.js</em> are somehow
bloated, they both kind of compile or render javascript templates on server-side <em>(sic)</em>.</p>
<p>Oppositely, my <a class="reference external" href="https://github.com/makinacorpus/django-templatetag-handlebars">django-templatetag-handlebars</a> is very simple, you
write your <em>Handlebars</em> template inside your <em>django</em> template. <em>Django</em>
will preserve nicely <tt class="docutils literal">{}</tt> tags, but still render <tt class="docutils literal">{% %}</tt> tags.</p>
<p>For example, with this in your template :</p>
<div class="highlight"><pre><span></span>{% tplhandlebars "tpl-infos" %}
{{total}} {% trans "result(s)." %}
<span class="p"><</span><span class="nt">p</span><span class="p">></span>{% trans "Min" %}: {{min}}<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>{% trans "Max" %}: {{max}}<span class="p"></</span><span class="nt">p</span><span class="p">></span>
{% endtplhandlebars %}
</pre></div>
<p>The following block with end-up in your page :</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">script</span> <span class="na">id</span><span class="o">=</span><span class="s">"tpl-infos"</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/x-handlebars-template"</span><span class="p">></span>
<span class="w"> </span><span class="p">{{</span><span class="nx">total</span><span class="p">}}</span><span class="w"> </span><span class="nx">result</span><span class="p">(</span><span class="nx">s</span><span class="p">).</span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>Min: {{min}}<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>Max: {{max}}<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span>
</pre></div>
<p>Render it, client-side, as usual using <em>Handlebars</em> API :</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">properties</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">total</span><span class="o">:</span><span class="w"> </span><span class="mf">10</span><span class="p">,</span>
<span class="w"> </span><span class="nx">min</span><span class="o">:</span><span class="w"> </span><span class="mf">4</span><span class="p">,</span>
<span class="w"> </span><span class="nx">max</span><span class="o">:</span><span class="w"> </span><span class="mf">5</span>
<span class="p">};</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">template</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Handlebars</span><span class="p">.</span><span class="nx">compile</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="s1">'#tpl-infos'</span><span class="p">).</span><span class="nx">html</span><span class="p">()),</span>
<span class="w"> </span><span class="nx">rendered</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">template</span><span class="p">(</span><span class="nx">properties</span><span class="p">);</span>
</pre></div>
<p>Your rendered string is ready, and waiting to be inserted in your DOM :)</p>
<div class="highlight"><pre><span></span>10 result(s).
<span class="p"><</span><span class="nt">p</span><span class="p">></span>Min: 4<span class="p"></</span><span class="nt">p</span><span class="p">></span>
<span class="p"><</span><span class="nt">p</span><span class="p">></span>Max: 5<span class="p"></</span><span class="nt">p</span><span class="p">></span>
</pre></div>
<p><a class="reference external" href="https://github.com/makinacorpus/django-templatetag-handlebars">Download and more info</a>.</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>Both named <em>django-handlebars</em>, <a class="reference external" href="https://github.com/yavorskiy/django-handlebars">by Sergii Iavorskyi</a> and <a class="reference external" href="https://bitbucket.org/chrisv/django-handlebars">by Chris Vigelius</a>.</td></tr>
</tbody>
</table>
landez : introducing new features of our tiles toolbox2012-02-24T12:45:00+01:002012-02-24T12:45:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-02-24:/landez-introducing-new-features-of-our-tiles-toolbox.html<p><a class="reference external" href="https://github.com/makinacorpus/landez">landez</a> started as a very small toolbox to build MBTiles files
specifying bounding boxes and zoom levels. We have been using it for several GIS projects
at <a class="reference external" href="http://www.makina-corpus.com">Makina Corpus</a>, and can tell it's reliable !</p>
<p>Landez is pure python and follows the <a class="reference external" href="http://en.wikipedia.org/wiki/KISS_principle">KISS principle</a>.
It has optional requirements on <a class="reference external" href="http://pypi.python.org/pypi/PIL">PIL</a> and …</p><p><a class="reference external" href="https://github.com/makinacorpus/landez">landez</a> started as a very small toolbox to build MBTiles files
specifying bounding boxes and zoom levels. We have been using it for several GIS projects
at <a class="reference external" href="http://www.makina-corpus.com">Makina Corpus</a>, and can tell it's reliable !</p>
<p>Landez is pure python and follows the <a class="reference external" href="http://en.wikipedia.org/wiki/KISS_principle">KISS principle</a>.
It has optional requirements on <a class="reference external" href="http://pypi.python.org/pypi/PIL">PIL</a> and <a class="reference external" href="http://pypi.python.org/pypi/mapnik2">mapnik</a>
for compositing, tile arranging or rendering.</p>
<p>Recently, we've added many extra cool features, which deserve highlight !</p>
<div class="section" id="simple-wms-support">
<h2>Simple WMS support</h2>
<p>With landez, you can store your WMS layers into MBTiles files ! It will
request the WMS /images and save them into tiles on disk ! You can then
enjoy the power of MBTiles files : transport, speed, ...</p>
<div class="highlight"><pre><span></span><span class="n">mb</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">(</span><span class="n">wms_server</span><span class="o">=</span><span class="s2">"http://yourserver.com/geoserver/wms"</span><span class="p">,</span>
<span class="n">wms_layers</span><span class="o">=</span><span class="p">[</span><span class="s2">"ign:departements"</span><span class="p">],</span>
<span class="n">wms_options</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="nb">format</span><span class="o">=</span><span class="s2">"image/png"</span><span class="p">,</span>
<span class="n">transparent</span><span class="o">=</span><span class="kc">True</span><span class="p">),</span>
<span class="n">filepath</span><span class="o">=</span><span class="s2">"dest.mbtiles"</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">add_coverage</span><span class="p">(</span><span class="n">bbox</span><span class="o">=</span><span class="p">([</span><span class="o">-</span><span class="mf">0.9853</span><span class="p">,</span><span class="mf">43.6435.1126</span><span class="p">,</span><span class="mf">44.0639</span><span class="p">]),</span>
<span class="n">zoomlevels</span><span class="o">=</span><span class="nb">range</span><span class="p">(</span><span class="mi">18</span><span class="p">))</span>
<span class="n">mb</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
</div>
<div class="section" id="tiles-compositing">
<h2>Tiles compositing</h2>
<p>This is the killer feature ! With landez, you can now merge multiple sources
of tiles (URL, WMS, MBTiles, Mapnik stylesheet) together !</p>
<p>For example, build a new MBTiles file by blending tiles of another on top of OpenStreetMap tiles :</p>
<div class="highlight"><pre><span></span><span class="n">mb</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">(</span><span class="n">remote</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">filepath</span><span class="o">=</span><span class="s2">"merged.mbtiles"</span><span class="p">)</span>
<span class="n">overlay</span> <span class="o">=</span> <span class="n">TilesManager</span><span class="p">(</span><span class="n">mbtiles_file</span><span class="o">=</span><span class="s2">"carto.mbtiles"</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">add_layer</span><span class="p">(</span><span class="n">overlay</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
<p>Simply make a composite a WMS layer with OpenStreetMap using transparency ! You might find this useful
for compositing satellite image with street maps :</p>
<div class="highlight"><pre><span></span><span class="n">mb</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">(</span><span class="n">wms_server</span><span class="o">=</span><span class="s2">"http://yourserver.com/geoserver/wms"</span><span class="p">,</span>
<span class="n">wms_layers</span><span class="o">=</span><span class="p">[</span><span class="s2">"img:orthophoto"</span><span class="p">],</span>
<span class="n">filepath</span><span class="o">=</span><span class="s2">"wms_osm.mbtiles"</span><span class="p">)</span>
<span class="n">overlay</span> <span class="o">=</span> <span class="n">TilesManager</span><span class="p">(</span><span class="n">remote</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">add_layer</span><span class="p">(</span><span class="n">overlay</span><span class="p">,</span> <span class="mf">0.4</span><span class="p">)</span> <span class="c1"># 40%</span>
<span class="n">mb</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
</div>
<div class="section" id="arrange-tiles-into-single-images">
<h2>Arrange tiles into single /images</h2>
<p>This feature can be very useful for printing tiled maps or have a quick overview
of your compositing results !</p>
<p>Refer to any source of tiles, like you would do with <cite>MBTilesBuilder</cite>,
add layers if you like and export the image !</p>
<div class="highlight"><pre><span></span><span class="n">ie</span> <span class="o">=</span> <span class="n">ImageExporter</span><span class="p">(</span><span class="n">tiles_url</span><span class="o">=</span><span class="s1">'http://server/tile/</span><span class="si">{z}</span><span class="s1">/</span><span class="si">{x}</span><span class="s1">/</span><span class="si">{y}</span><span class="s1">.png'</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">...</span>
<span class="n">ie</span><span class="o">.</span><span class="n">export_image</span><span class="p">(</span><span class="n">bbox</span><span class="o">=</span><span class="p">(</span><span class="o">-</span><span class="mf">180.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">90.0</span><span class="p">,</span> <span class="mf">180.0</span><span class="p">,</span> <span class="mf">90.0</span><span class="p">),</span>
<span class="n">zoomlevel</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
<span class="n">imagepath</span><span class="o">=</span><span class="s2">"image.png"</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="mbtiles-content-reading">
<h2>MBTiles content reading</h2>
<p>landez can now read MBTiles content !</p>
<p>We could proudly add it to the list of implementations for the <a class="reference external" href="https://github.com/mapbox/mbtiles-spec/wiki/Implementations">MBTiles spec</a>
and <a class="reference external" href="https://github.com/mapbox/utfgrid-spec/wiki/Implementations">UTF-Grid spec</a> !</p>
<p>Use MBTiles files like any tile source :</p>
<div class="highlight"><pre><span></span><span class="n">mb</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">(</span><span class="n">mbtiles_file</span><span class="o">=</span><span class="s2">"yourfile.mbtiles"</span><span class="p">)</span>
</pre></div>
<p>...extract single image or UTF-Grid tiles :</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">landez.reader</span> <span class="kn">import</span> <span class="n">MBTilesReader</span>
<span class="n">mbreader</span> <span class="o">=</span> <span class="n">MBTilesReader</span><span class="p">(</span><span class="s2">"yourfile.mbtiles"</span><span class="p">)</span>
<span class="c1"># Metadata</span>
<span class="nb">print</span> <span class="n">mbreader</span><span class="o">.</span><span class="n">metadata</span><span class="p">()</span>
<span class="c1"># Zoom levels</span>
<span class="nb">print</span> <span class="n">mbreader</span><span class="o">.</span><span class="n">zoomlevels</span><span class="p">()</span>
<span class="c1"># Image tile</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s1">'tile.png'</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">out</span><span class="p">:</span>
<span class="n">out</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">reader</span><span class="o">.</span><span class="n">tile</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">))</span>
<span class="c1"># UTF-Grid tile</span>
<span class="nb">print</span> <span class="n">reader</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="n">z</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="s1">'callback'</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="next-steps">
<h2>Next steps...</h2>
<p>The code has grown quickly and deserves a good refactoring, which is being done in a separate
branch <a class="reference external" href="https://github.com/makinacorpus/landez">on GitHub</a> ! The goal is to
keep the same simple API, better modularity, increase tests coverage...</p>
<p>If you are wiling to participate, feel welcome !</p>
</div>
TileMill on your Web server behind a reverse proxy2012-01-31T11:12:00+01:002012-01-31T11:12:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-01-31:/tilemill-on-your-web-server-behind-a-reverse-proxy.html<p>In the last version (0.9), <a class="reference external" href="http://mapbox.com/tilemill/">TileMill</a> has an additional dedicated process to
serve the tiles. We had to change some bits of our server configuration.</p>
<p>Instead of documenting the new configuration in our internal Wiki,
I prefered to share here a few technical lines (quite rough though).</p>
<p>We run …</p><p>In the last version (0.9), <a class="reference external" href="http://mapbox.com/tilemill/">TileMill</a> has an additional dedicated process to
serve the tiles. We had to change some bits of our server configuration.</p>
<p>Instead of documenting the new configuration in our internal Wiki,
I prefered to share here a few technical lines (quite rough though).</p>
<p>We run TileMill inside a <a class="reference external" href="/a-virtual-local-server-room-for-you-developper.html">virtual machine</a> on a server with reverse proxy rules.</p>
<div class="section" id="reverse-proxy-configuration">
<h2>Reverse Proxy configuration</h2>
<p>Assuming your reach your TileMill virtual machine at <tt class="docutils literal">tilemill.sillywalk.loc</tt>,
with its two processes running (ports <tt class="docutils literal">20008</tt> for tiles, <tt class="docutils literal">20009</tt> for the application),
your Apache reverse proxy configuration will be :</p>
<div class="highlight"><pre><span></span><span class="nt"><VirtualHost</span><span class="w"> </span><span class="err">*:80</span><span class="nt">></span>
<span class="w"> </span>ServerName<span class="w"> </span>tilemill.yourdomain.com
<span class="w"> </span>ProxyPreserveHost<span class="w"> </span>On
<span class="w"> </span>RewriteEngine<span class="w"> </span>on
<span class="w"> </span>#<span class="w"> </span>Serve<span class="w"> </span>the<span class="w"> </span>tiles<span class="w"> </span>as<span class="w"> </span>/tiles/
<span class="w"> </span>RewriteCond<span class="w"> </span>%{REQUEST_URI}<span class="w"> </span>^(/tiles.*)$
<span class="w"> </span>RewriteRule<span class="w"> </span>^/tiles(.*)<span class="w"> </span>http://tilemill.sillywalk.loc:20008$1<span class="w"> </span>[L,P]
<span class="w"> </span>ProxyPassReverse<span class="w"> </span>/tiles<span class="w"> </span>http://tilemill.sillywalk.loc:20008/
<span class="w"> </span>#<span class="w"> </span>Serve<span class="w"> </span>the<span class="w"> </span>application<span class="w"> </span>on<span class="w"> </span>/
<span class="w"> </span>ProxyPass<span class="w"> </span>/<span class="w"> </span>http://tilemill.sillywalk.loc:20009/
<span class="w"> </span>ProxyPassReverse<span class="w"> </span>/<span class="w"> </span>http://tilemill.sillywalk.loc:20009
<span class="nt"></VirtualHost></span>
</pre></div>
</div>
<div class="section" id="tilemill-configuration">
<h2>TileMill configuration</h2>
<p>Now that tiles and application ports are reachable respectively on <tt class="docutils literal"><span class="pre">http://tilemill.yourdomain.com/tiles</span></tt>
and <tt class="docutils literal"><span class="pre">http://tilemill.yourdomain.com/</span></tt>, just tell TileMill to serve its pages accordingly in its configuration, <tt class="docutils literal">/etc/tilemill/tilemill.config</tt> :</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="w"> </span><span class="s2">"files"</span><span class="o">:</span><span class="w"> </span><span class="s2">"/usr/share/mapbox"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"server"</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"coreUrl"</span><span class="o">:</span><span class="w"> </span><span class="s2">"tilemill.yourdomain.com:80"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"tileUrl"</span><span class="o">:</span><span class="w"> </span><span class="s2">"tilemill.yourdomain.com:80/tiles"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"port"</span><span class="o">:</span><span class="w"> </span><span class="mf">20009</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"listenHost"</span><span class="o">:</span><span class="w"> </span><span class="s2">"0.0.0.0"</span>
<span class="p">}</span>
</pre></div>
<p>Restart it...</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>service<span class="w"> </span>tilemill<span class="w"> </span>restart
</pre></div>
<p>Done !</p>
</div>
Merkopolo : a simple yet powerful starter kit for your Qt/C++ GIS application2012-01-26T11:30:00+01:002012-01-26T11:30:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2012-01-26:/merkopolo-a-simple-yet-powerful-starter-kit-for-your-qtc-gis-application.html<p>Recently, while the whole world looks completely hyped up with Web applications,
we chose to design and develop a desktop software in Qt/C++.</p>
<p>Obviously, the choice was measured and justified ! We had to build a specific GIS application
with complex interactions and huge amounts of data, for a limited …</p><p>Recently, while the whole world looks completely hyped up with Web applications,
we chose to design and develop a desktop software in Qt/C++.</p>
<p>Obviously, the choice was measured and justified ! We had to build a specific GIS application
with complex interactions and huge amounts of data, for a limited number of users.</p>
<p>Quickly, we spotted <a class="reference external" href="http://merkaartor.be/">Merkaartor</a>, one of <a class="reference external" href="http://wiki.openstreetmap.org/wiki/Editing">the official OpenStreetMap editors</a>,
for its UI components and object model. And since we started to code, we never regretted this choice !</p>
<p>C++ brings the power, Qt offers cross-platform and the compassion towards developers, and Merkaartor a lovely GIS flavour !</p>
<p>We contributed to Merkaartor to give gits components a little bit of genericity, and
released <a class="reference external" href="https://gitorious.org/merkopolo/merkopolo">Merkopolo</a>, a Qt project
skeleton to handle dependencies and inclusion of base components.</p>
<p>Here is what you immediately get once compiled :</p>
<img alt="" src="/images/merkopolo-preview.png" />
<p>Now you can start coding serious stuff on top, with the Merkaartor components stack :</p>
<ul class="simple">
<li>Complete feature model with free attributes (tags)</li>
<li>Custom drawing styles</li>
<li>A variety of layers types (Tiles, WMS, Spatialite, GeoTIFF, GDAL...)</li>
<li>Base classes for mouse interactions on map objects</li>
<li>A projection system (<em>libproj</em>)</li>
<li>And even <a class="reference external" href="postgis-data-in-c-using-gdal-and-qt.html">draw geometries from PostGIS database</a> !</li>
</ul>
<p><a class="reference external" href="https://gitorious.org/merkopolo/merkopolo">Merkopolo is available on Gitorious</a>.</p>
Access a JSON webservice with Qt C++2011-12-16T17:00:00+01:002011-12-16T17:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-12-16:/access-a-json-webservice-with-qt-c.html<p><em>Original post at</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Webservices are everywhere ! There are relevant in many situations, and
accessing them from your Qt C++ application is not an heresy.</p>
<p>I will present here a very simple way to retrieve a JSON from a GET request.</p>
<div class="section" id="http-requests">
<h2>HTTP Requests</h2>
<p>Using <a class="reference external" href="http://developer.qt.nokia.com/doc/qt-4.7/qnetworkaccessmanager.html">QNetworkAccessManager</a> is
a piece of …</p></div><p><em>Original post at</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Webservices are everywhere ! There are relevant in many situations, and
accessing them from your Qt C++ application is not an heresy.</p>
<p>I will present here a very simple way to retrieve a JSON from a GET request.</p>
<div class="section" id="http-requests">
<h2>HTTP Requests</h2>
<p>Using <a class="reference external" href="http://developer.qt.nokia.com/doc/qt-4.7/qnetworkaccessmanager.html">QNetworkAccessManager</a> is
a piece of cake :</p>
<div class="highlight"><pre><span></span><span class="n">QNetworkAccessManager</span><span class="w"> </span><span class="n">networkManager</span><span class="p">;</span>
<span class="n">QUrl</span><span class="w"> </span><span class="n">url</span><span class="p">(</span><span class="s">"http://gdata.youtube.com/feeds/api/standardfeeds/most_popular?v=2&alt=json"</span><span class="p">);</span>
<span class="n">QNetworkRequest</span><span class="w"> </span><span class="n">request</span><span class="p">;</span>
<span class="n">request</span><span class="p">.</span><span class="n">setUrl</span><span class="p">(</span><span class="n">url</span><span class="p">);</span>
<span class="n">QNetworkReply</span><span class="o">*</span><span class="w"> </span><span class="n">currentReply</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">networkManager</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">request</span><span class="p">);</span><span class="w"> </span><span class="c1">// GET</span>
</pre></div>
<p>But, note that a slightly more generic approach would be to build the <tt class="docutils literal">QUrl</tt> from a parameters list :</p>
<div class="highlight"><pre><span></span><span class="n">QUrl</span><span class="w"> </span><span class="n">url</span><span class="p">(</span><span class="s">"http://gdata.youtube.com/feeds/api/standardfeeds/"</span><span class="p">);</span>
<span class="n">QString</span><span class="w"> </span><span class="n">method</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"most_popular"</span><span class="p">;</span>
<span class="n">url</span><span class="p">.</span><span class="n">setPath</span><span class="p">(</span><span class="n">QString</span><span class="p">(</span><span class="s">"%1%2"</span><span class="p">).</span><span class="n">arg</span><span class="p">(</span><span class="n">url</span><span class="p">.</span><span class="n">path</span><span class="p">()).</span><span class="n">arg</span><span class="p">(</span><span class="n">method</span><span class="p">));</span>
<span class="n">QMap</span><span class="o"><</span><span class="n">QString</span><span class="p">,</span><span class="w"> </span><span class="n">QVariant</span><span class="o">></span><span class="w"> </span><span class="n">params</span><span class="p">;</span>
<span class="n">params</span><span class="p">[</span><span class="s">"alt"</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"json"</span><span class="p">;</span>
<span class="n">params</span><span class="p">[</span><span class="s">"v"</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"2"</span><span class="p">;</span>
<span class="n">foreach</span><span class="p">(</span><span class="n">QString</span><span class="w"> </span><span class="n">param</span><span class="p">,</span><span class="w"> </span><span class="n">params</span><span class="p">.</span><span class="n">keys</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">url</span><span class="p">.</span><span class="n">addQueryItem</span><span class="p">(</span><span class="n">param</span><span class="p">,</span><span class="w"> </span><span class="n">params</span><span class="p">[</span><span class="n">param</span><span class="p">].</span><span class="n">toString</span><span class="p">());</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="parsing-json">
<h2>Parsing JSON</h2>
<p>Get yourself a <em>slot</em> to parse the <tt class="docutils literal">QNetworkReply</tt> :</p>
<div class="highlight"><pre><span></span><span class="n">connect</span><span class="p">(</span><span class="o">&</span><span class="n">networkManager</span><span class="p">,</span><span class="w"> </span><span class="n">SIGNAL</span><span class="p">(</span><span class="n">finished</span><span class="p">(</span><span class="n">QNetworkReply</span><span class="o">*</span><span class="p">)),</span><span class="w"> </span><span class="k">this</span><span class="p">,</span><span class="w"> </span><span class="n">SLOT</span><span class="p">(</span><span class="n">onResult</span><span class="p">(</span><span class="n">QNetworkReply</span><span class="o">*</span><span class="p">)));</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="kt">void</span><span class="w"> </span><span class="nf">YourClass::onResult</span><span class="p">(</span><span class="n">QNetworkReply</span><span class="o">*</span><span class="w"> </span><span class="n">reply</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">m_currentReply</span><span class="o">-></span><span class="n">error</span><span class="p">()</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">QNetworkReply</span><span class="o">::</span><span class="n">NoError</span><span class="p">)</span>
<span class="w"> </span><span class="k">return</span><span class="p">;</span><span class="w"> </span><span class="c1">// ...only in a blog post</span>
<span class="w"> </span><span class="n">QString</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">QString</span><span class="p">)</span><span class="w"> </span><span class="n">reply</span><span class="o">-></span><span class="n">readAll</span><span class="p">();</span>
<span class="w"> </span><span class="n">QScriptEngine</span><span class="w"> </span><span class="n">engine</span><span class="p">;</span>
<span class="w"> </span><span class="n">QScriptValue</span><span class="w"> </span><span class="n">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">engine</span><span class="p">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">data</span><span class="p">);</span>
<span class="w"> </span><span class="cm">/*</span>
<span class="cm"> Google YouTube JSON looks like this :</span>
<span class="cm"> {</span>
<span class="cm"> "version": "1.0",</span>
<span class="cm"> "encoding": "UTF-8",</span>
<span class="cm"> "feed": {</span>
<span class="cm"> ..</span>
<span class="cm"> ..</span>
<span class="cm"> "entry": [{</span>
<span class="cm"> "title": {</span>
<span class="cm"> "$t": "Nickelback- When We Stand Together"</span>
<span class="cm"> },</span>
<span class="cm"> "content": {</span>
<span class="cm"> "type": "application/x-shockwave-flash",</span>
<span class="cm"> "src": "http://www.youtube.com/v/76vdvdll0Y?version=3&f=standard&app=youtube_gdata"</span>
<span class="cm"> },</span>
<span class="cm"> "yt$statistics": {</span>
<span class="cm"> "favoriteCount": "29182",</span>
<span class="cm"> "viewCount": "41513706"</span>
<span class="cm"> },</span>
<span class="cm"> ...</span>
<span class="cm"> ...</span>
<span class="cm"> },</span>
<span class="cm"> ...</span>
<span class="cm"> ...</span>
<span class="cm"> ]</span>
<span class="cm"> }</span>
<span class="cm"> }</span>
<span class="cm"> */</span>
<span class="w"> </span><span class="c1">// Now parse this JSON according to your needs !</span>
<span class="w"> </span><span class="n">QScriptValue</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">result</span><span class="p">.</span><span class="n">property</span><span class="p">(</span><span class="s">"feed"</span><span class="p">).</span><span class="n">property</span><span class="p">(</span><span class="s">"entry"</span><span class="p">);</span>
<span class="w"> </span><span class="n">QScriptValueIterator</span><span class="w"> </span><span class="n">it</span><span class="p">(</span><span class="n">entries</span><span class="p">);</span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">it</span><span class="p">.</span><span class="n">hasNext</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">it</span><span class="p">.</span><span class="n">next</span><span class="p">();</span>
<span class="w"> </span><span class="n">QScriptValue</span><span class="w"> </span><span class="n">entry</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">it</span><span class="p">.</span><span class="n">value</span><span class="p">();</span>
<span class="w"> </span><span class="n">QString</span><span class="w"> </span><span class="n">link</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">entry</span><span class="p">.</span><span class="n">property</span><span class="p">(</span><span class="s">"content"</span><span class="p">).</span><span class="n">property</span><span class="p">(</span><span class="s">"src"</span><span class="p">).</span><span class="n">toString</span><span class="p">();</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">viewCount</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">entry</span><span class="p">.</span><span class="n">property</span><span class="p">(</span><span class="s">"yt$statistics"</span><span class="p">).</span><span class="n">property</span><span class="p">(</span><span class="s">"viewCount"</span><span class="p">).</span><span class="n">toInteger</span><span class="p">();</span>
<span class="w"> </span><span class="c1">// Do something with those...</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>That's it :)</p>
<p>If you want more complexity, and don't mind adding extra-dependencies, check out Tomasz Siekierda's <a class="reference external" href="http://gitorious.org/qwebservice">QtWebService</a> !</p>
</div>
Remove django form field validation errors manually2011-12-06T09:00:00+01:002011-12-06T09:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-12-06:/remove-django-form-field-valiation-errors-manually.html<p><em>Original post at</em> <a class="reference external" href="http://www.makina-corpus.org">Makina Corpus</a></p>
<p>Sometimes I look for something which seems so simple and stupid that I can't imagine
it does not exist. It makes me wonder why and who is the fool. Worse, I can't be sure about my search keywords to prove me anything.</p>
<p>I just wanted …</p><p><em>Original post at</em> <a class="reference external" href="http://www.makina-corpus.org">Makina Corpus</a></p>
<p>Sometimes I look for something which seems so simple and stupid that I can't imagine
it does not exist. It makes me wonder why and who is the fool. Worse, I can't be sure about my search keywords to prove me anything.</p>
<p>I just wanted to delete, reset or remove the validation errors of a single form field, within a django view, without
overriding the form or field class.</p>
<div class="section" id="a-one-liner">
<h2>A one-liner</h2>
<div class="highlight"><pre><span></span><span class="n">aform</span><span class="o">.</span><span class="n">errors</span><span class="p">[</span><span class="s1">'afield'</span><span class="p">]</span> <span class="o">=</span> <span class="n">aform</span><span class="p">[</span><span class="s1">'afield'</span><span class="p">]</span><span class="o">.</span><span class="n">error_class</span><span class="p">()</span>
</pre></div>
<p><strong>That's it folks !</strong></p>
<ul class="simple">
<li>This will not affect other fields errors or non-field errors ;</li>
<li>This will reuse nicely the field error class (<tt class="docutils literal">ErrorDict</tt> or <tt class="docutils literal">ErrorList</tt>) ;</li>
<li>You cannot set <tt class="docutils literal"><span class="pre">aform.errors['afield']</span> = None</tt> or your form <tt class="docutils literal">full_clean()</tt> will be performed again !</li>
<li>Obviously, the ideal approach is to override your form <tt class="docutils literal">clean()</tt> properly.</li>
</ul>
</div>
An equivalent of Django's select_related for ManyToMany and OneToMany relationships2011-12-05T11:09:00+01:002011-12-05T11:09:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-12-05:/django-selectrelated-manytomany.html<p><em>Original post at</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Using an ORM simplifies and reduces greatly the amount of code to interact with databases.
Nevertheless, it can easily hide database design defects or become a source of serious performance issues.</p>
<div class="section" id="a-common-pitfall">
<h2>A Common Pitfall</h2>
<p>With Django, the most classic problem occurs while accessing objects relations …</p></div><p><em>Original post at</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Using an ORM simplifies and reduces greatly the amount of code to interact with databases.
Nevertheless, it can easily hide database design defects or become a source of serious performance issues.</p>
<div class="section" id="a-common-pitfall">
<h2>A Common Pitfall</h2>
<p>With Django, the most classic problem occurs while accessing objects relations attributes
inside a loop. That's why QuerySet's method <tt class="docutils literal">select_related()</tt> exists :
it will join specified relations so that access to their attributes does not hit the database.
<a class="reference external" href="https://docs.djangoproject.com/en/dev/ref/models/querysets/#select-related">Refer to Django's documentation</a> for more information !</p>
</div>
<div class="section" id="one-to-many-and-many-to-many-relationships">
<h2>One-To-Many and Many-To-Many Relationships</h2>
<p><tt class="docutils literal">select_related()</tt> is not able to follow One-To-Many (<em>1-n</em>) and Many-To-Many (<em>n-n</em>) relationships.
The Django team is currently working on <tt class="docutils literal">prefetch_related()</tt>. But before we can enjoy this
future feature, we can implement an equivalent in python.</p>
<p>With these models :</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pizza</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">50</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Restaurant</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">50</span><span class="p">)</span>
<span class="n">pizzas</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ManyToManyField</span><span class="p">(</span><span class="n">Pizza</span><span class="p">,</span> <span class="n">through</span><span class="o">=</span><span class="s1">'PizzaRestaurant'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">PizzaRestaurant</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">pizza</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">Pizza</span><span class="p">)</span>
<span class="n">restaurant</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">Restaurant</span><span class="p">)</span>
<span class="n">price</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">FloatField</span><span class="p">()</span>
</pre></div>
<p>This loop will generate <em>1 + N</em> queries :</p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">restaurant</span> <span class="ow">in</span> <span class="n">Restaurant</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">all</span><span class="p">():</span>
<span class="k">for</span> <span class="n">pizza</span> <span class="ow">in</span> <span class="n">restaurant</span><span class="o">.</span><span class="n">pizzas</span><span class="o">.</span><span class="n">all</span><span class="p">():</span>
<span class="nb">print</span> <span class="n">pizza</span><span class="o">.</span><span class="n">name</span>
</pre></div>
<p>Whereas this one will <strong>only</strong> generate <em>2</em> queries :</p>
<div class="highlight"><pre><span></span><span class="c1"># Store relationships in a dict</span>
<span class="n">byrestaurant</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">pr</span> <span class="ow">in</span> <span class="n">PizzaRestaurant</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">select_related</span><span class="p">(</span><span class="s1">'restaurant'</span><span class="p">,</span> <span class="s1">'pizza'</span><span class="p">)</span><span class="o">.</span><span class="n">all</span><span class="p">():</span>
<span class="n">byrestaurant</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">pr</span><span class="o">.</span><span class="n">restaurant</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="p">[])</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">pr</span><span class="o">.</span><span class="n">pizza</span><span class="p">)</span>
<span class="c1"># Use stored lists</span>
<span class="k">for</span> <span class="n">restaurant</span> <span class="ow">in</span> <span class="n">Restaurant</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">all</span><span class="p">():</span>
<span class="k">for</span> <span class="n">pizza</span> <span class="ow">in</span> <span class="n">byrestaurant</span><span class="p">[</span><span class="n">restaurant</span><span class="o">.</span><span class="n">id</span><span class="p">]:</span>
<span class="nb">print</span> <span class="n">pizza</span><span class="o">.</span><span class="n">name</span>
</pre></div>
<p>According to the amount of <em>N</em>, doing that trick in views can boost your pages !</p>
<p>This is not perfect and elegant, but if it allows you to downsize the number of queries
from several thousands to fifteen, like <a class="reference external" href="http://gitorious.org/memopol2-0/memopol2-0/merge_requests/18">it did on Memopol2</a>, you can think twice.</p>
</div>
Des cartes collaboratives avec Livetitude2011-11-23T14:00:00+01:002011-11-23T14:00:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-11-23:/des-cartes-collaboratives-avec-livetitude-fr.html<p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Cela fait plusieurs semaines que je voulais présenter ma petite application
de partage de cartes, dont je me sers déjà comme alternative aux marqueurs de Google Maps.</p>
<div class="section" id="en-bref">
<h2>En bref</h2>
<p><a class="reference external" href="https://github.com/makinacorpus/livetitude">Livetitude</a> est un outil Web pour créer des cartes de manière collaborative
<strong>en temps réel …</strong></p></div><p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Cela fait plusieurs semaines que je voulais présenter ma petite application
de partage de cartes, dont je me sers déjà comme alternative aux marqueurs de Google Maps.</p>
<div class="section" id="en-bref">
<h2>En bref</h2>
<p><a class="reference external" href="https://github.com/makinacorpus/livetitude">Livetitude</a> est un outil Web pour créer des cartes de manière collaborative
<strong>en temps réel</strong> (à la manière d'un <a class="reference external" href="http://fr.wikipedia.org/wiki/EtherPad">pad</a>).</p>
<p>Fonctionnant sur des terminaux mobiles, Livetitude permet également de partager la position des collaborateurs de la carte,
d'exporter les données au format GeoJSON ou de publier la carte sur une page Web.</p>
</div>
<div class="section" id="sous-le-capot">
<h2>Sous le capot</h2>
<p>J'ai pris du plaisir à hacker cette application, elle tire profit
d'outils très simples mais très efficaces :</p>
<ul class="simple">
<li><a class="reference external" href="http://http://leaflet.cloudmade.com">Leaflet</a> pour afficher les cartes ;</li>
<li><a class="reference external" href="http://pusher.com">Pusher</a> (Websockets) pour la collaboration en temps réel ;</li>
<li><a class="reference external" href="http://couchdb.apache.org/">CouchDB</a> pour stocker les données ;</li>
<li><a class="reference external" href="http://flask.pocoo.org">Flask</a> pour servir les pages ;</li>
<li><a class="reference external" href="http://www.heroku.com">Heroku</a> pour héberger l'application.</li>
</ul>
<p>Bien entendu, le code source est libre et disponible sur <a class="reference external" href="https://github.com/makinacorpus/livetitude">le GitHub de Makina Corpus</a>.</p>
</div>
<div class="section" id="utilisation">
<h2>Utilisation</h2>
<p>Une instance est <a class="reference external" href="http://vivid-warrior-6693.herokuapp.com/">déployée en ligne</a>, dans le cloud d'Heroku,
dont vous pouvez vous servir, <em>pour une utilisation en bon père de famille</em> :)</p>
<p>Les marqueurs peuvent contenir du texte ou de l'HTML, et aucune donnée de localisation des visiteurs n'est stockée.</p>
<img alt="" src="/images/livetitude-poc.png" />
<div class="section" id="comment-publier-vos-donnees-existantes">
<h3>Comment publier vos données existantes ?</h3>
<p>Si vous souhaitez publier et visualiser vos marqueurs sur une carte de Livetitude,
il suffit de poster (<tt class="docutils literal">POST</tt>) les coordonnées de vos points sur l'URL <tt class="docutils literal"><span class="pre">http://server/<CARTE>/add</span></tt>.</p>
<p>Par exemple, avec une petite fonction python :</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">httplib</span><span class="o">,</span> <span class="nn">urllib</span>
<span class="n">SERVER</span> <span class="o">=</span> <span class="s2">"server"</span> <span class="c1"># e.g. vivid-warrior-6693.herokuapp.com</span>
<span class="k">def</span> <span class="nf">publish</span><span class="p">(</span><span class="n">mapname</span><span class="p">,</span> <span class="n">coords</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
<span class="n">url</span> <span class="o">=</span> <span class="s2">"/</span><span class="si">%s</span><span class="s2">/add"</span> <span class="o">%</span> <span class="n">mapname</span>
<span class="n">request</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'coords'</span><span class="p">:</span> <span class="n">coords</span><span class="p">,</span>
<span class="s1">'data'</span><span class="p">:</span> <span class="n">data</span><span class="p">,</span>
<span class="s1">'classid'</span><span class="p">:</span> <span class="mi">5</span> <span class="c1"># (=color)</span>
<span class="p">}</span>
<span class="n">params</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"Content-type"</span><span class="p">:</span> <span class="s2">"application/x-www-form-urlencoded"</span><span class="p">,</span>
<span class="s2">"Accept"</span><span class="p">:</span> <span class="s2">"text/plain"</span><span class="p">}</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">httplib</span><span class="o">.</span><span class="n">HTTPConnection</span><span class="p">(</span><span class="n">SERVER</span><span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">request</span><span class="p">(</span><span class="s2">"POST"</span><span class="p">,</span> <span class="n">url</span><span class="p">,</span> <span class="n">params</span><span class="p">,</span> <span class="n">headers</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">getresponse</span><span class="p">()</span>
<span class="nb">print</span> <span class="n">response</span><span class="o">.</span><span class="n">status</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">publish</span><span class="p">(</span><span class="s2">"PasLoin"</span><span class="p">,</span> <span class="s2">"10.1,54.9"</span><span class="p">,</span> <span class="s2">"Super resto!"</span><span class="p">)</span>
<span class="n">publish</span><span class="p">(</span><span class="s2">"PasLoin"</span><span class="p">,</span> <span class="s2">"19.4,65.1"</span><span class="p">,</span> <span class="s2">"Bon mojito"</span><span class="p">)</span>
</pre></div>
<p>Les points sont alors visibles en ligne sur <tt class="docutils literal"><span class="pre">http://server/PasLoin</span></tt> ou
disponible en GeoJSON sur <tt class="docutils literal"><span class="pre">http://server/PasLoin/points</span></tt>.</p>
</div>
</div>
<div class="section" id="contribuer">
<h2>Contribuer</h2>
<p>Livetitude est une application très simple, à l'état de preuve de concept. Mais
le code source est très réduit et donc très rapide à prendre en main !</p>
<p>Toutes vos <a class="reference external" href="https://github.com/makinacorpus/livetitude/issues">suggestions ou contributions</a> sont les bienvenues !</p>
</div>
Git : annuler proprement un commit après un push2011-11-03T14:15:00+01:002011-11-03T14:15:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-11-03:/git-annuler-proprement-un-commit-apres-un-push-fr.html<div class="section" id="ce-qu-il-faut-eviter">
<h2>Ce qu'il faut éviter</h2>
<p>Pour annuler des commits, il existe la commande <tt class="docutils literal">git reset</tt>.</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>reset<span class="w"> </span>--hard<span class="w"> </span>HEAD~1
HEAD<span class="w"> </span>is<span class="w"> </span>now<span class="w"> </span>at<span class="w"> </span>444b1cf<span class="w"> </span>Rhoo
</pre></div>
<p>Celle-ci est pertinente tant que les commits n'ont pas été poussés. Git vous retiendra au <tt class="docutils literal">push</tt> d'ailleurs :</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>push
To<span class="w"> </span>/tmp/repo
<span class="w"> </span>!<span class="w"> </span><span class="o">[</span>rejected<span class="o">]</span><span class="w"> </span>master<span class="w"> </span>-><span class="w"> </span>master …</pre></div></div><div class="section" id="ce-qu-il-faut-eviter">
<h2>Ce qu'il faut éviter</h2>
<p>Pour annuler des commits, il existe la commande <tt class="docutils literal">git reset</tt>.</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>reset<span class="w"> </span>--hard<span class="w"> </span>HEAD~1
HEAD<span class="w"> </span>is<span class="w"> </span>now<span class="w"> </span>at<span class="w"> </span>444b1cf<span class="w"> </span>Rhoo
</pre></div>
<p>Celle-ci est pertinente tant que les commits n'ont pas été poussés. Git vous retiendra au <tt class="docutils literal">push</tt> d'ailleurs :</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>push
To<span class="w"> </span>/tmp/repo
<span class="w"> </span>!<span class="w"> </span><span class="o">[</span>rejected<span class="o">]</span><span class="w"> </span>master<span class="w"> </span>-><span class="w"> </span>master<span class="w"> </span><span class="o">(</span>non-fast-forward<span class="o">)</span>
error:<span class="w"> </span>failed<span class="w"> </span>to<span class="w"> </span>push<span class="w"> </span>some<span class="w"> </span>refs<span class="w"> </span>to<span class="w"> </span><span class="s1">'/tmp/repo'</span>
</pre></div>
<p>En effet, à partir du moment où un commit existe sur le serveur, il est potentiellement utilisé
par des collaborateurs (<em>mergé, à la base d'une branche, etc.</em>). On pourrait faire le sale et forcer le push :</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>push<span class="w"> </span>-f
Total<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>delta<span class="w"> </span><span class="m">0</span><span class="o">)</span>,<span class="w"> </span>reused<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="o">(</span>delta<span class="w"> </span><span class="m">0</span><span class="o">)</span>
To<span class="w"> </span>/tmp/repo
<span class="w"> </span>+<span class="w"> </span>b67c343...444b1cf<span class="w"> </span>master<span class="w"> </span>-><span class="w"> </span>master<span class="w"> </span><span class="o">(</span>forced<span class="w"> </span>update<span class="o">)</span>
</pre></div>
<p>Mais il y a beaucoup mieux !</p>
</div>
<div class="section" id="ce-qu-il-faut-faire">
<h2>Ce qu'il faut faire</h2>
<p>Annuler un commit, c'est finalement appliquer l'inverse de son <strong>diff</strong> !</p>
<p>On peut rediriger le diff des commits à annuler vers la commande <tt class="docutils literal">patch <span class="pre">--reverse</span></tt> :)</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>diff<span class="w"> </span>HEAD^<span class="w"> </span><span class="p">|</span><span class="w"> </span>patch<span class="w"> </span>--reverse
</pre></div>
<p>Pour faire plus simple, il y a <tt class="docutils literal">git revert</tt> !</p>
<p>Par exemple pour annuler les trois derniers commits :</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>revert<span class="w"> </span>HEAD~3..HEAD
</pre></div>
<p>Ou pour annuler un commit en particulier :</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>revert<span class="w"> </span>444b1cff
</pre></div>
<p>Il suffit alors de pousser proprement le commit obtenu sur le
serveur. Les éventuels collaborateurs qui avaient basé leur travail sur les commits
annulés devront gérer les conflits au moment venu...</p>
</div>
Two major Unity design failures2011-10-15T00:00:00+02:002011-10-15T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-10-15:/two-major-unity-design-failures.html<p>A great advantage of global menus is the ease of pointing them with the mouse.</p>
<p>According to <a class="reference external" href="http://en.wikipedia.org/wiki/Fitts%27s_law">Fitts' law</a>,
the time to point a target is function of its distance and size. By sticking
the menus on the top part of the screen, the vertical dimension vanishes, since
the user …</p><p>A great advantage of global menus is the ease of pointing them with the mouse.</p>
<p>According to <a class="reference external" href="http://en.wikipedia.org/wiki/Fitts%27s_law">Fitts' law</a>,
the time to point a target is function of its distance and size. By sticking
the menus on the top part of the screen, the vertical dimension vanishes, since
the user can just throw his mouse on top to reach them.</p>
<p>Unfortunately, two major design problems in Unity prevents from completely
turning global menus to account.</p>
<div class="section" id="window-buttons-dead-edges">
<h2>Window buttons dead edges</h2>
<p>The user can not throw his mouse to the top-left corner to reach the close button,
since the edges are not clickable.</p>
<img alt="" src="/images/unity-deadzone.png" />
<p><a class="reference external" href="https://bugs.launchpad.net/ubuntu/+source/unity/+bug/874980">A bug was registered on Launchpad</a> a couple of minutes before I wrote this
post.</p>
</div>
<div class="section" id="menus-not-always-shown">
<h2>Menus not always shown</h2>
<img alt="" src="/images/unity-menu-hidden.gif" />
<p>The problem is <a class="reference external" href="https://bugs.launchpad.net/ubuntu/+source/unity/+bug/701294">being discussed on Launchpad</a>.</p>
<p>In my opinion, the current implementation is a terrible design :</p>
<ul class="simple">
<li>Lack of <em>afordance</em> : There is no obvious hint that a menu is available for the current application.</li>
<li><em>Fitt's law</em> fail : The user can not throw his mouse to the desired menu since it becomes visible only when he reaches it.</li>
</ul>
<p>Fortunately, I am gesture and keyboard user (<a class="reference external" href="apt://easystroke">easystroke</a>, <tt class="docutils literal">Alt+F10</tt>, <tt class="docutils literal">Alt+F4</tt>)</p>
</div>
Une démo squelettique de python Flask CouchDB2011-10-11T11:30:00+02:002011-10-11T11:30:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-10-11:/une-demo-squelettique-de-python-flask-couchdb-fr.html<p>Avec <tt class="docutils literal">Flask</tt> et <tt class="docutils literal">Couchdb</tt> (e.g. <a class="reference external" href="http://packages.python.org/Flask-CouchDB/">Flask-CouchDB</a>),
on peut faire rapidement des trucs amusants, voire très utiles !</p>
<p>Voici un <strong>squelette</strong> d'application, fonctionnel, qui stocke et récupère des objets crées à partir de posts HTTP.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">simplejson</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">request</span>
<span class="kn">from</span> <span class="nn">couchdb.design</span> <span class="kn">import</span> <span class="n">ViewDefinition</span>
<span class="kn">import</span> <span class="nn">flaskext.couchdb …</span></pre></div><p>Avec <tt class="docutils literal">Flask</tt> et <tt class="docutils literal">Couchdb</tt> (e.g. <a class="reference external" href="http://packages.python.org/Flask-CouchDB/">Flask-CouchDB</a>),
on peut faire rapidement des trucs amusants, voire très utiles !</p>
<p>Voici un <strong>squelette</strong> d'application, fonctionnel, qui stocke et récupère des objets crées à partir de posts HTTP.</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">simplejson</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">request</span>
<span class="kn">from</span> <span class="nn">couchdb.design</span> <span class="kn">import</span> <span class="n">ViewDefinition</span>
<span class="kn">import</span> <span class="nn">flaskext.couchdb</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="sd">"""</span>
<span class="sd">CouchDB permanent view</span>
<span class="sd">"""</span>
<span class="n">docs_by_author</span> <span class="o">=</span> <span class="n">ViewDefinition</span><span class="p">(</span><span class="s1">'docs'</span><span class="p">,</span> <span class="s1">'byauthor'</span><span class="p">,</span>
<span class="s1">'function(doc) { emit(doc.author, doc); }'</span><span class="p">)</span>
<span class="sd">"""</span>
<span class="sd">Retrieve docs</span>
<span class="sd">"""</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s2">"/<author_id>/docs"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">docs</span><span class="p">(</span><span class="n">author_id</span><span class="p">):</span>
<span class="n">docs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">docs_by_author</span><span class="p">(</span><span class="n">g</span><span class="o">.</span><span class="n">couch</span><span class="p">)[</span><span class="n">author_id</span><span class="p">]:</span>
<span class="n">docs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">row</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">return</span> <span class="n">simplejson</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">docs</span><span class="p">)</span>
<span class="sd">"""</span>
<span class="sd">Add doc</span>
<span class="sd">"""</span>
<span class="nd">@app</span><span class="o">.</span><span class="n">route</span><span class="p">(</span><span class="s2">"/<author_id>/add"</span><span class="p">,</span> <span class="n">methods</span><span class="o">=</span><span class="p">[</span><span class="s1">'POST'</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">add_doc</span><span class="p">(</span><span class="n">author_id</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Build doc with posted values</span>
<span class="n">doc</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'author'</span><span class="p">:</span> <span class="n">author_id</span> <span class="p">}</span>
<span class="n">doc</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">form</span><span class="p">)</span>
<span class="c1"># Insert into database</span>
<span class="n">g</span><span class="o">.</span><span class="n">couch</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span>
<span class="n">state</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span>
<span class="n">state</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">return</span> <span class="n">simplejson</span><span class="o">.</span><span class="n">dumps</span><span class="p">({</span><span class="s1">'ok'</span><span class="p">:</span> <span class="n">state</span><span class="p">})</span>
<span class="sd">"""</span>
<span class="sd">Flask main</span>
<span class="sd">"""</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">update</span><span class="p">(</span>
<span class="n">DEBUG</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span>
<span class="n">COUCHDB_SERVER</span> <span class="o">=</span> <span class="s1">'http://localhost:5984/'</span><span class="p">,</span>
<span class="n">COUCHDB_DATABASE</span> <span class="o">=</span> <span class="s1">'docsdemo'</span>
<span class="p">)</span>
<span class="n">manager</span> <span class="o">=</span> <span class="n">flaskext</span><span class="o">.</span><span class="n">couchdb</span><span class="o">.</span><span class="n">CouchDBManager</span><span class="p">()</span>
<span class="n">manager</span><span class="o">.</span><span class="n">setup</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="n">manager</span><span class="o">.</span><span class="n">add_viewdef</span><span class="p">(</span><span class="n">docs_by_author</span><span class="p">)</span> <span class="c1"># Install the view</span>
<span class="n">manager</span><span class="o">.</span><span class="n">sync</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="s1">'0.0.0.0'</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">5000</span><span class="p">)</span>
</pre></div>
<p>J'ai déposé ce snippet sur <a class="reference external" href="https://gist.github.com/1277655">Gist</a> si besoin.</p>
<p>On peut attaquer l'application avec <tt class="docutils literal">curl</tt> :</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>curl<span class="w"> </span>-d<span class="w"> </span><span class="s2">"title=Globalia&year=2004"</span><span class="w"> </span>http://0.0.0.0:5000/jc.rufin/add
<span class="o">{</span><span class="s2">"ok"</span>:<span class="w"> </span>true<span class="o">}</span>
$<span class="w"> </span>curl<span class="w"> </span>-d<span class="w"> </span><span class="s2">"title=Red%20Brazil&contest=goncourt"</span><span class="w"> </span>http://0.0.0.0:5000/jc.rufin/add
<span class="o">{</span><span class="s2">"ok"</span>:<span class="w"> </span>true<span class="o">}</span>
$<span class="w"> </span>curl<span class="w"> </span>http://0.0.0.0:5000/jc.rufin/docs
<span class="o">[{</span><span class="s2">"title"</span>:<span class="w"> </span><span class="o">[</span><span class="s2">"Globalia"</span><span class="o">]</span>,<span class="w"> </span><span class="s2">"year"</span>:<span class="w"> </span><span class="o">[</span><span class="s2">"2004"</span><span class="o">]</span>,<span class="w"> </span><span class="s2">"author"</span>:<span class="w"> </span><span class="s2">"jc.rufin"</span>,<span class="w"> </span><span class="s2">"_rev"</span>:<span class="w"> </span><span class="s2">"1-3195...fbc8"</span>,<span class="w"> </span><span class="s2">"_id"</span>:<span class="w"> </span><span class="s2">"dec81d...1733c"</span><span class="o">}</span>,
<span class="o">{</span><span class="s2">"title"</span>:<span class="w"> </span><span class="o">[</span><span class="s2">"Red Brazil"</span><span class="o">]</span>,<span class="w"> </span><span class="s2">"contest"</span>:<span class="w"> </span><span class="o">[</span><span class="s2">"goncourt"</span><span class="o">]</span>,<span class="w"> </span><span class="s2">"author"</span>:<span class="w"> </span><span class="s2">"jc.rufin"</span>,<span class="w"> </span><span class="s2">"_rev"</span>:<span class="w"> </span><span class="s2">"1-7b15...a9a2"</span>,<span class="w"> </span><span class="s2">"_id"</span>:<span class="w"> </span><span class="s2">"dec81dc...17c0c"</span><span class="o">}]</span>
</pre></div>
<p>N'oubliez pas de colorier les cases à votre guise, sinon ce squelette ne sert à rien, le JSON étant déjà la langue maternelle de CouchDB.</p>
Mes premiers pas avec Heroku2011-10-10T10:30:00+02:002011-10-10T10:30:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-10-10:/mes-premiers-pas-avec-heroku-fr.html<p>J'ai pris le temps de tester la plateforme <a class="reference external" href="http://www.heroku.com">Heroku</a>, qui fait pas mal de bruit
en ce moment.</p>
<p>L'idée est simple : le développeur pousse son code dans une branche Git
pour déployer son application. Ils se chargent du reste.</p>
<p>La <a class="reference external" href="http://devcenter.heroku.com/articles/python">documentation pour déployer du python</a> est bien faite.</p>
<div class="section" id="heroku-et-virtualenv">
<h2>heroku et …</h2></div><p>J'ai pris le temps de tester la plateforme <a class="reference external" href="http://www.heroku.com">Heroku</a>, qui fait pas mal de bruit
en ce moment.</p>
<p>L'idée est simple : le développeur pousse son code dans une branche Git
pour déployer son application. Ils se chargent du reste.</p>
<p>La <a class="reference external" href="http://devcenter.heroku.com/articles/python">documentation pour déployer du python</a> est bien faite.</p>
<div class="section" id="heroku-et-virtualenv">
<h2>heroku et virtualenv</h2>
<p>Pour commencer, j'ai installé l'outil <tt class="docutils literal">heroku</tt> en ruby (sic), dans un <em>virtualenv</em></p>
<pre class="literal-block">
sudo aptitude install rubygems
virtualenv --no-site-packages env
source env/bin/activate
</pre>
<p>Merci à Bruno, qui nous a expliqué récemment <a class="reference external" href="http://bruno.im/2011/sep/29/streamline-your-django-workflow/">comment faire cohabiter les gems ruby et virtualenv</a></p>
<pre class="literal-block">
export GEM_HOME="$VIRTUAL_ENV/gems"
export GEM_PATH=""
export PATH=$PATH:$GEM_HOME/bin
</pre>
<p>(À ajouter au hook dans <tt class="docutils literal"><span class="pre">~/.virtualenvs/postactivate</span></tt> pour plus tard)</p>
<p>Pour terminer, il suffit de poser le <em>gem</em></p>
<pre class="literal-block">
gem install heroku
</pre>
<p>Et j'ai bien <tt class="docutils literal">heroku</tt> cloisonné dans le <em>virtualenv</em>.</p>
<div class="highlight"><pre><span></span><span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>which<span class="w"> </span>heroku
/home/mathieu/path/env/gems/bin/heroku
<span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>heroku<span class="w"> </span><span class="nb">help</span>
Usage:<span class="w"> </span>heroku<span class="w"> </span>COMMAND<span class="w"> </span><span class="o">[</span>--app<span class="w"> </span>APP<span class="o">]</span><span class="w"> </span><span class="o">[</span>command-specific-options<span class="o">]</span>
</pre></div>
</div>
<div class="section" id="heroku-et-la-configuration-de-l-application">
<h2>heroku et la Configuration de l'Application</h2>
<p>Pour apprivoiser la plateforme, j'ai utilisé le <em>micro</em>-framework <a class="reference external" href="http://flask.pocoo.org/">Flask</a>,
suggéré dans le tutorial python. C'est ultra-simple, ultra-léger, ultra-tout.</p>
<p>Afin de gérer ma configuration, j'ai créé une classe <tt class="docutils literal">Settings</tt> qui utilise les variables d'environment:</p>
<div class="highlight"><pre><span></span><span class="c1"># settings.py</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">class</span> <span class="nc">Settings</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">DEBUG</span> <span class="o">=</span> <span class="nb">bool</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"DEBUG"</span><span class="p">))</span>
<span class="n">TESTING</span> <span class="o">=</span> <span class="nb">bool</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"TESTING"</span><span class="p">))</span>
<span class="n">PORT</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"PORT"</span><span class="p">,</span> <span class="mi">5000</span><span class="p">))</span>
<span class="n">HOST</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"HOST"</span><span class="p">,</span> <span class="s1">'0.0.0.0'</span><span class="p">)</span>
</pre></div>
<p>Que je branche dans l'application :</p>
<div class="highlight"><pre><span></span><span class="c1"># app.py</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">from</span> <span class="nn">settings</span> <span class="kn">import</span> <span class="n">Settings</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<span class="n">settings</span> <span class="o">=</span> <span class="n">Settings</span><span class="p">()</span>
<span class="c1">#...</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="c1"># ...</span>
<span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">from_object</span><span class="p">(</span><span class="n">settings</span><span class="p">)</span>
</pre></div>
<p>Ensuite grâce au client <tt class="docutils literal">heroku</tt>, je peux contrôler à distance
la configuration de mon application, qui est redémarrée à chaque changement :</p>
<div class="highlight"><pre><span></span><span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>heroku<span class="w"> </span>config
<span class="nv">PATH</span><span class="w"> </span><span class="o">=</span>><span class="w"> </span>bin:/usr/local/bin:/usr/bin:/bin
<span class="nv">PYTHONUNBUFFERED</span><span class="w"> </span><span class="o">=</span>><span class="w"> </span><span class="nb">true</span>
<span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>heroku<span class="w"> </span>config:add<span class="w"> </span><span class="nv">DEBUG</span><span class="o">=</span>True
Adding<span class="w"> </span>config<span class="w"> </span>vars:
<span class="w"> </span><span class="nv">DEBUG</span><span class="w"> </span><span class="o">=</span>><span class="w"> </span>True
Restarting<span class="w"> </span>app...<span class="w"> </span><span class="k">done</span>,<span class="w"> </span>v19.
<span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>heroku<span class="w"> </span>config
<span class="nv">DEBUG</span><span class="w"> </span><span class="o">=</span>><span class="w"> </span>True
<span class="nv">PATH</span><span class="w"> </span><span class="o">=</span>><span class="w"> </span>bin:/usr/local/bin:/usr/bin:/bin
<span class="nv">PYTHONUNBUFFERED</span><span class="w"> </span><span class="o">=</span>><span class="w"> </span><span class="nb">true</span>
<span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>heroku<span class="w"> </span>config:remove<span class="w"> </span>DEBUG
Removing<span class="w"> </span>DEBUG<span class="w"> </span>and<span class="w"> </span>restarting<span class="w"> </span>app...<span class="w"> </span><span class="k">done</span>,<span class="w"> </span>v20.
</pre></div>
<p>Je peux revenir en arrière quand un changement de config a posé problème :</p>
<div class="highlight"><pre><span></span><span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>heroku<span class="w"> </span>releases
Rel<span class="w"> </span>Change<span class="w"> </span>By<span class="w"> </span>When
----<span class="w"> </span>----------------------<span class="w"> </span>----------<span class="w"> </span>----------
v20<span class="w"> </span>Config<span class="w"> </span>remove<span class="w"> </span>DEBUG<span class="w"> </span>your@mail.com<span class="w"> </span><span class="m">25</span><span class="w"> </span>seconds<span class="w"> </span>ago
v19<span class="w"> </span>Config<span class="w"> </span>add<span class="w"> </span>DEBUG<span class="w"> </span>your@mail.com<span class="w"> </span><span class="m">1</span><span class="w"> </span>minute<span class="w"> </span>ago
<span class="o">(</span>env<span class="o">)</span>src$<span class="w"> </span>heroku<span class="w"> </span>rollback<span class="w"> </span>v19
Rolled<span class="w"> </span>back<span class="w"> </span>to<span class="w"> </span>v19
</pre></div>
</div>
Avec Git rebase, vos arbres poussent droit2011-09-16T17:37:00+02:002011-09-16T17:37:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-09-16:/avec-git-rebase-vos-arbres-poussent-droit-fr.html<div class="section" id="le-probleme">
<h2>Le problème</h2>
<p>Par défaut, un <tt class="docutils literal">git pull</tt> est équivalent à <tt class="docutils literal">git fetch</tt> et <tt class="docutils literal">git merge</tt>.</p>
<p>Les <tt class="docutils literal">merge</tt>, c'est bien pour les branches, mais pour le tronc, c'est pénible : ça pollue l'historique et
ça zig-zag sévère !</p>
<img alt="" class="align-center" src="/images/git-merge-mess.png" />
</div>
<div class="section" id="la-solution">
<h2>La solution</h2>
<p>En réalité, quand on travaille sur le tronc, ce qu'on veut c'est faire …</p></div><div class="section" id="le-probleme">
<h2>Le problème</h2>
<p>Par défaut, un <tt class="docutils literal">git pull</tt> est équivalent à <tt class="docutils literal">git fetch</tt> et <tt class="docutils literal">git merge</tt>.</p>
<p>Les <tt class="docutils literal">merge</tt>, c'est bien pour les branches, mais pour le tronc, c'est pénible : ça pollue l'historique et
ça zig-zag sévère !</p>
<img alt="" class="align-center" src="/images/git-merge-mess.png" />
</div>
<div class="section" id="la-solution">
<h2>La solution</h2>
<p>En réalité, quand on travaille sur le tronc, ce qu'on veut c'est faire <tt class="docutils literal">git fetch</tt> et <tt class="docutils literal">git rebase</tt>.
C'est à dire, au lieu de ça :</p>
<pre class="literal-block">
A-----B-----C master
/ \
D---E---F---G---H---I origin/master
</pre>
<p>on veut ça :</p>
<pre class="literal-block">
A---B---C master
/
D---E---F---G---H origin/master
</pre>
<p>Autrement dit, un <tt class="docutils literal">git pull <span class="pre">--rebase</span></tt> ! Pour le faire par défaut :</p>
<div class="highlight"><pre><span></span>git<span class="w"> </span>config<span class="w"> </span>--global<span class="w"> </span>branch.autosetuprebase<span class="w"> </span>always
</pre></div>
<p>Et ensuite, au cas-où, pour le désactiver ponctuellement, utiliser <tt class="docutils literal">git pull <span class="pre">--no-rebase</span></tt>.</p>
<p>Maintenant, le tronc, il est tout propre !</p>
<img alt="" class="align-center" src="/images/git-merge-clean.png" />
</div>
<div class="section" id="les-consequences">
<h2>Les conséquences ?</h2>
<p>Lors d'un <tt class="docutils literal">git pull</tt>, il faudra résoudre chaque <em>commit</em> conflictuel indépendamment (perso, je préfère).</p>
<p>Les êtres humains voudront utiliser <a class="reference external" href="http://meld.sourceforge.net/">meld</a>. Il suffit de l'installer, et lors
d'un conflit, de lancer <tt class="docutils literal">git mergetool</tt>.</p>
<p>Une fois que tous les conflits sont résolus, terminer l'opération, avec <tt class="docutils literal">git rebase <span class="pre">--continue</span></tt>, et pousser vos prouesses à
vos amis avec <tt class="docutils literal">git push</tt>.</p>
</div>
A Virtual Local Server Room for you Developper2011-09-02T09:12:00+02:002011-09-02T09:12:00+02:00Mathieu Leplatre (credits: Anthony Prades)tag:blog.mathieu-leplatre.info,2011-09-02:/a-virtual-local-server-room-for-you-developper.html<p>This how-to will help you setting-up a very powerful development environment
on your workstation, using virtual machines all in a virtual local network.</p>
<div class="section" id="but-why">
<h2>But Why ?</h2>
<p>You work on various projects, with specific requirements, with various
technologies, diverse operating systems or cpu architectures, usually with
a couple of services like databases …</p></div><p>This how-to will help you setting-up a very powerful development environment
on your workstation, using virtual machines all in a virtual local network.</p>
<div class="section" id="but-why">
<h2>But Why ?</h2>
<p>You work on various projects, with specific requirements, with various
technologies, diverse operating systems or cpu architectures, usually with
a couple of services like databases or Web servers.</p>
<p>You would prefer not taking this whole family with you each time you start your machine.</p>
<p>Probably, you also want to upgrade your workstation to the last unstable eye-candy OS
without compromising your projects dependencies and without reinstalling all this stuff,
or you may want to restore the environment you had when you were working on this famous project a year ago.</p>
<p>You want your colleague to give you a hand, and would like to give him your
whole project environment and dependencies quickly and effortlessly ?
Your sysadmin put KVM on servers and you would like to push your local instances on
production in a blink ?</p>
<p>The magic medicine exists, and is freely available :)</p>
<p>At the end, you are promised to enjoy :</p>
<ul class="simple">
<li>a GUI to manage your Virtual Machines (VM)</li>
<li>a local domain to access your VMs by their name</li>
<li>a fully integrated set of machines, accessible from each others</li>
<li>movable and shareable virtual machines with automatic network configuration</li>
</ul>
<p>We chose the Linux Kernel-based Virtualization System (KVM), since it is maintained
along with the Linux kernel, and is thus fully integrated in the OS. However, this
how-to is mainly networking oriented and would be useful for any virtualization system.</p>
</div>
<div class="section" id="a-strict-minimum">
<h2>A Strict Minimum</h2>
<p>A strict minimum is to install <tt class="docutils literal">kvm</tt> and a set of commands to control it : <tt class="docutils literal">libvirt</tt>.
As a human being, you may want a GUI : <tt class="docutils literal"><span class="pre">virt-manager</span></tt>.</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>kvm<span class="w"> </span>libvirt-bin<span class="w"> </span>virt-manager
</pre></div>
<div class="section" id="virtual-machines-network">
<h3>Virtual Machines Network</h3>
<p>Make sure your VM networking is set to <em>NAT</em>. This will allow
your VM to access your host network (LAN, Internet, etc.)</p>
<p>During your VM operating system installation, or after login into it,
setup your VM network interface as automatic DHCP.</p>
<p>By default KVM creates a virtual network (<tt class="docutils literal">virnet</tt>). Inspect its setup
using <tt class="docutils literal">ifconfig</tt> or <tt class="docutils literal"><span class="pre">virt-manager</span></tt> in <em>Connection Details</em> > <em>Virtual Networks</em>.
Your VM and your host are thus accessible within this virtual network (probably <tt class="docutils literal">172.16.23.0</tt>).</p>
</div>
</div>
<div class="section" id="a-local-network-domain">
<h2>A Local Network Domain</h2>
<p>In order to access your VM by their name, we run a DNS daemon on main host. We chose <em>bind</em> :</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>aptitude<span class="w"> </span>install<span class="w"> </span>bind9
</pre></div>
<p>Add a new master zone, for example <tt class="docutils literal">sillywalk.loc</tt> in the file <tt class="docutils literal">/etc/bind/named.conf.local</tt> :</p>
<pre class="literal-block">
zone "sillywalk.loc" {
type master;
file "/etc/bind/db.sillywalk.loc";
};
</pre>
<p>Define your DNS entries :</p>
<ul class="simple">
<li>Set the name server authority (<em>SOA</em>) to <tt class="docutils literal">ns.sillywalk.loc</tt> (<em>nameserver</em>)</li>
<li>Define a serial number like date+number (<tt class="docutils literal">YYYYmmdd##</tt>)</li>
<li>Associate <tt class="docutils literal">ns.sillywalk.loc</tt> to the IP of your KVM virtual network (<em>see above paragraph</em>)</li>
<li>Define an alias <cite>gw.sillywalk.loc</cite> so that you can refer to your host as <cite>gw</cite> (<em>gateway</em>) instead of <cite>ns</cite>.</li>
<li>Define a couple of entries for your VM (e.g. <tt class="docutils literal">myvm1</tt>, <tt class="docutils literal">myvm2</tt>)</li>
</ul>
<p>For <em>bind</em>, it would look like this <em>(started from an existing file like ``/etc/bind/db.empty``)</em> :</p>
<pre class="literal-block">
;/etc/bind/db.sillywalk.loc
;
; BIND data file for local loopback interface
;
$TTL 604800
@ IN SOA ns.sillywalk.loc. root.ns.sillywalk.loc. (
2011080301 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS ns.sillywalk.loc.
@ IN A 172.16.23.1
ns IN A 172.16.23.1
gw IN CNAME ns.sillywalk.loc.
myvm1 IN A 172.16.23.11
myvm2 IN A 172.16.23.12
</pre>
<div class="section" id="use-your-dns">
<h3>Use your DNS</h3>
<p>Use your Network Manager to set the search domain to <tt class="docutils literal">sillywalk.loc</tt> and
to add your local DNS server (<tt class="docutils literal">127.0.0.1</tt>) in front of the other(s).</p>
<p>Apply and your <tt class="docutils literal">/etc/resolv.conf</tt> could then look like this :</p>
<pre class="literal-block">
# Generated by NetworkManager
search sillywalk.loc # search domain
nameserver 127.0.0.1 # your local DNS server
nameserver 192.168.1.254 # your FAI/company DNS
nameserver 8.8.8.8 # Google public DNS
</pre>
</div>
<div class="section" id="test-it">
<h3>Test it !</h3>
<p>Even if your VM are not running, you can at least test the name resolving
and the default search domain :</p>
<div class="highlight"><pre><span></span>~$<span class="w"> </span>ping<span class="w"> </span>myvm1.sillywalk.loc
PING<span class="w"> </span>myvm1.sillywalk.loc<span class="w"> </span><span class="o">(</span><span class="m">172</span>.16.23.11<span class="o">)</span><span class="w"> </span><span class="m">56</span><span class="o">(</span><span class="m">84</span><span class="o">)</span><span class="w"> </span>bytes<span class="w"> </span>of<span class="w"> </span>data.
<span class="c1"># (Ctrl+C)</span>
~$<span class="w"> </span>ping<span class="w"> </span>myvm2
PING<span class="w"> </span>myvm2.sillywalk.loc<span class="w"> </span><span class="o">(</span><span class="m">172</span>.16.23.12<span class="o">)</span><span class="w"> </span><span class="m">56</span><span class="o">(</span><span class="m">84</span><span class="o">)</span><span class="w"> </span>bytes<span class="w"> </span>of<span class="w"> </span>data.
<span class="c1"># (Ctrl+C)</span>
</pre></div>
</div>
</div>
<div class="section" id="dynamic-configuration">
<h2>Dynamic Configuration</h2>
<p>In order to make sure your VM always obtains the same IP adress when it
boots, we setup a DHCP daemon on host.</p>
<p>We chose <em>ISC DHCP server</em> :</p>
<pre class="literal-block">
sudo aptitude install isc-dhcp-server
</pre>
<p>In the configuration file <tt class="docutils literal">/etc/dhcp/dhcpd.conf</tt>, we specify :</p>
<ul class="simple">
<li>a domain name (<tt class="docutils literal">sillywalk.loc</tt>)</li>
<li>the name server to be configured on clients (<tt class="docutils literal">ns.sillywalk.loc</tt>)</li>
<li>the subnet and mask (<em>matching the KVM virtual network</em>)</li>
<li>an IP range (e.g. from <tt class="docutils literal">172.16.23.10</tt> to <tt class="docutils literal">172.16.23.100</tt>)</li>
<li>the default gateway to be configured on clients (<tt class="docutils literal">ns.sillywalk.loc</tt>)</li>
<li>... and two entries for <tt class="docutils literal">myvm1</tt> and <tt class="docutils literal">myvm2</tt> with their Mac addresses.</li>
</ul>
<pre class="literal-block">
# /etc/dhcp/dhcpd.conf
option domain-name "sillywalk.loc";
option domain-name-servers ns.sillywalk.loc;
subnet 172.16.23.0 netmask 255.255.255.0 {
range 172.16.23.10 172.16.23.100;
option broadcast-address 172.16.23.255;
option routers gw.sillywalk.loc;
}
# Entries
host myvm1 {
hardware ethernet 52:54:00:55:d1:80;
fixed-address myvm1.sillywalk.loc;
}
host myvm2 {
hardware ethernet 52:54:00:55:e1:66;
fixed-address myvm2.sillywalk.loc;
}
</pre>
<div class="section" id="test-it-1">
<h3>Test it !</h3>
<p>Log you in on the VM.</p>
<ul class="simple">
<li>Configure its hostname (e.g. <tt class="docutils literal">myvm1</tt>, <tt class="docutils literal">myvm2</tt>)</li>
</ul>
<div class="highlight"><pre><span></span>root@myvm1:~#<span class="w"> </span>cat<span class="w"> </span>/etc/hostname
myvm1
root@myvm1:~#<span class="w"> </span>cat<span class="w"> </span>/etc/hosts
<span class="m">127</span>.0.0.1<span class="w"> </span>localhost
<span class="m">127</span>.0.1.1<span class="w"> </span>myvm1.sillywalk.loc<span class="w"> </span>myvm1
</pre></div>
<ul class="simple">
<li>Make sure your VM network is set to DHCP automatic configuration</li>
</ul>
<div class="highlight"><pre><span></span>root@myvm1:~#<span class="w"> </span>cat<span class="w"> </span>/etc/network/interfaces
...
<span class="c1"># The primary network interface</span>
allow-hotplug<span class="w"> </span>eth0
iface<span class="w"> </span>eth0<span class="w"> </span>inet<span class="w"> </span>dhcp
</pre></div>
<ul class="simple">
<li>Reboot it (or restart networking)</li>
<li>Check that it caught the right network configuration (IP, domain, and nameserver)</li>
</ul>
<div class="highlight"><pre><span></span>root@myvm1:~#<span class="w"> </span>ifconfig
eth0<span class="w"> </span>Link<span class="w"> </span>encap:Ethernet<span class="w"> </span>HWaddr<span class="w"> </span><span class="m">52</span>:54:00:55:d1:80
<span class="w"> </span>inet<span class="w"> </span>addr:172.16.23.11<span class="w"> </span>Bcast:172.16.23.255<span class="w"> </span>Mask:255.255.255.0
<span class="w"> </span>...
</pre></div>
<div class="highlight"><pre><span></span>root@myvm1:~#<span class="w"> </span>cat<span class="w"> </span>/etc/resolv.conf
domain<span class="w"> </span>sillywalk.loc
search<span class="w"> </span>sillywalk.loc
nameserver<span class="w"> </span><span class="m">172</span>.16.23.1
</pre></div>
</div>
<div class="section" id="note">
<h3>Note</h3>
<p>While your host is booting, the DHCP daemon usually starts before the KVM service, failing
then at accessing the virtual network interface (<tt class="docutils literal">virbr1</tt>, <tt class="docutils literal">172.16.23.0</tt>), not yet mounted.</p>
<p>A simple solution is to manually restart your DHCP daemon, once your machine's booted :</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>/etc/init.d/isc-dhcp-server<span class="w"> </span>restart
</pre></div>
</div>
</div>
<div class="section" id="checklist-to-add-a-new-vm">
<h2>Checklist to add a new VM</h2>
<ul class="simple">
<li>Get its Mac address (with <tt class="docutils literal"><span class="pre">virt-manager</span></tt> : <em>Virtual machine details</em> > <em>Virtual network interface</em> > <em>Mac Address</em>)</li>
<li>Add it to your DHCP configuration (<tt class="docutils literal">/etc/dhcp/dhcpd.conf</tt>)</li>
<li>Add an IP for this entry in your DNS zone (<tt class="docutils literal">/etc/bind/db.sillywalk.loc</tt>) and increment the serial.</li>
<li>Restart DHCP service and reload DNS configuration</li>
</ul>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>/etc/init.d/isc-dhcp-server<span class="w"> </span>restart
sudo<span class="w"> </span>/etc/init.d/bind9<span class="w"> </span>reload
</pre></div>
<p>If you do that all day, you'll quickly find it relevant to write a script...</p>
<div class="section" id="note-on-cloning">
<h3>Note on cloning</h3>
<p>Cloning your VM with <tt class="docutils literal"><span class="pre">virt-manager</span></tt> is a piece-of-cake.</p>
<p>However, during cloning, KVM assigns a new Mac address to the clone.
For debian-based virtual machines (+Ubuntu), log you in on the clone, and reinitialize network interfaces names :</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>rm<span class="w"> </span>/etc/udev/rules.d/70-persistent-net.rules
sudo<span class="w"> </span>reboot
</pre></div>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Your virtual machines can :</p>
<ul class="simple">
<li>access your network (LAN, Internet) and your host (at <tt class="docutils literal">ns.sillywalk.loc</tt>)</li>
<li>be accessed at <tt class="docutils literal">user@hostname</tt> (from host or from other VMs)</li>
<li>be moved to any host set up likewise (since VM networking is fully automatic)</li>
<li>be cloned easily</li>
</ul>
<p>Read again "<a class="reference internal" href="#but-why">But Why ?</a>" and enjoy your new life !</p>
</div>
A very Simple and Stupid plugin system in python2011-09-02T00:00:00+02:002011-09-02T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-09-02:/a-very-simple-and-stupid-plugin-system-in-python.html<p>Two convenience functions for listing and importing python modules :</p>
<div class="highlight"><pre><span></span><span class="c1"># utils.py</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">def</span> <span class="nf">plugins_list</span><span class="p">(</span><span class="n">plugins_dirs</span><span class="p">):</span>
<span class="w"> </span><span class="sd">""" List all python modules in specified plugins folders """</span>
<span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">plugins_dirs</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">pathsep</span><span class="p">):</span>
<span class="k">for</span> <span class="n">filename</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
<span class="n">name</span><span class="p">,</span> <span class="n">ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ext</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">".py"</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">name</span>
<span class="k">def …</span></pre></div><p>Two convenience functions for listing and importing python modules :</p>
<div class="highlight"><pre><span></span><span class="c1"># utils.py</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">def</span> <span class="nf">plugins_list</span><span class="p">(</span><span class="n">plugins_dirs</span><span class="p">):</span>
<span class="w"> </span><span class="sd">""" List all python modules in specified plugins folders """</span>
<span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="n">plugins_dirs</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">pathsep</span><span class="p">):</span>
<span class="k">for</span> <span class="n">filename</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
<span class="n">name</span><span class="p">,</span> <span class="n">ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ext</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">".py"</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">name</span>
<span class="k">def</span> <span class="nf">import_plugins</span><span class="p">(</span><span class="n">plugins_dirs</span><span class="p">,</span> <span class="n">env</span><span class="p">):</span>
<span class="w"> </span><span class="sd">""" Import modules into specified environment (symbol table) """</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">plugins_list</span><span class="p">(</span><span class="n">plugins_dirs</span><span class="p">):</span>
<span class="n">m</span> <span class="o">=</span> <span class="nb">__import__</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">env</span><span class="p">)</span>
<span class="n">env</span><span class="p">[</span><span class="n">p</span><span class="p">]</span> <span class="o">=</span> <span class="n">m</span>
</pre></div>
<p>And now use <tt class="docutils literal">import_plugins()</tt> wherever you need to use them !</p>
<div class="highlight"><pre><span></span><span class="c1"># yourapp.py</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">utils</span> <span class="kn">import</span> <span class="n">import_plugins</span>
<span class="n">plugins_dirs</span> <span class="o">=</span> <span class="s2">"plugins/:module/plugins/"</span>
<span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">plugins_dirs</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">pathsep</span><span class="p">))</span>
<span class="n">import_plugins</span><span class="p">(</span><span class="n">plugins_dirs</span><span class="p">,</span> <span class="nb">globals</span><span class="p">())</span>
</pre></div>
<p>Note that in order to list all sub-classes of a specific one, you can use <a class="reference external" href="http://code.activestate.com/recipes/576949/">this
recursive function</a>.</p>
<p>That's all folks !</p>
<p>It is very simple and very stupid, but useful :) You might now want to have
a look at serious stuff like <a class="reference external" href="http://packages.python.org/Yapsy/">Yapsy</a> or
<a class="reference external" href="http://packages.python.org/distribute/pkg_resources.html">PkgResouces</a>.</p>
Le piège des QThread2011-09-01T11:09:00+02:002011-09-01T11:09:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-09-01:/le-piege-des-qthread-fr.html<p><em>Article original publié chez</em> <a class="reference external" href="http://www.makina-corpus.org/blog/le-pi%C3%A8ge-des-qthread">Makina Corpus</a></p>
<p>Il y a de nombreux billets de blogs, posts sur des forums, tutoriaux,
pages Wiki et autres, mais au final, à part le fameux <a class="reference external" href="http://labs.qt.nokia.com/2010/06/17/youre-doing-it-wrong/">"You're doing it wrong"</a>,
qui peut paraître obscure au premier abord, je n'ai pas trouvé de résumé
de l'attrape-nigaud que …</p><p><em>Article original publié chez</em> <a class="reference external" href="http://www.makina-corpus.org/blog/le-pi%C3%A8ge-des-qthread">Makina Corpus</a></p>
<p>Il y a de nombreux billets de blogs, posts sur des forums, tutoriaux,
pages Wiki et autres, mais au final, à part le fameux <a class="reference external" href="http://labs.qt.nokia.com/2010/06/17/youre-doing-it-wrong/">"You're doing it wrong"</a>,
qui peut paraître obscure au premier abord, je n'ai pas trouvé de résumé
de l'attrape-nigaud que je vais illustrer ici.</p>
<div class="section" id="le-piege">
<h2>Le piège</h2>
<p>Naturellement, quand on veut faire une thread, on a envie d'hériter de l'objet
<a class="reference external" href="http://doc.qt.nokia.com/latest/qthread.html">QThread</a>. C'est ce qu'on fait avec le module <tt class="docutils literal">threading</tt> de python (en Java aussi il me semble).</p>
<p>Voici ce qu'on écrit naturellement : <tt class="docutils literal">Objet</tt>, la classe qui file l'ordre et <tt class="docutils literal">Worker</tt>, une classe qui bosse dur en arrière plan. On connecte les signaux et on démarre !</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">PyQt4.QtCore</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">from</span> <span class="nn">PyQt4.QtGui</span> <span class="kn">import</span> <span class="n">QApplication</span>
<span class="k">class</span> <span class="nc">Object</span><span class="p">(</span><span class="n">QObject</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">emitSignal</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"aSignal()"</span><span class="p">))</span>
<span class="k">class</span> <span class="nc">Worker</span><span class="p">(</span><span class="n">QThread</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">aSlot</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">thread</span><span class="p">()</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="nb">print</span> <span class="s2">"Slot is executed in thread : "</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">thread</span><span class="p">()</span><span class="o">.</span><span class="n">currentThreadId</span><span class="p">()</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">QApplication</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
<span class="nb">print</span> <span class="s2">"Main application thread is : "</span><span class="p">,</span> <span class="n">app</span><span class="o">.</span><span class="n">thread</span><span class="p">()</span><span class="o">.</span><span class="n">currentThreadId</span><span class="p">()</span>
<span class="n">worker</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">()</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">Object</span><span class="p">()</span>
<span class="n">QObject</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"aSignal()"</span><span class="p">),</span> <span class="n">worker</span><span class="o">.</span><span class="n">aSlot</span><span class="p">)</span>
<span class="n">worker</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">obj</span><span class="o">.</span><span class="n">emitSignal</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"Done."</span>
<span class="n">app</span><span class="o">.</span><span class="n">exec_</span><span class="p">()</span>
</pre></div>
<p>Ici, comme le slot <tt class="docutils literal">aSlot()</tt> est défini dans la classe <tt class="docutils literal">Worker</tt>, qui hérite de <tt class="docutils literal">QThread</tt>, on
pense naturellement qu'il va être exécuté en arrière-plan. Que nenni!</p>
<div class="highlight"><pre><span></span>Main<span class="w"> </span>application<span class="w"> </span>thread<span class="w"> </span>is<span class="w"> </span>:<span class="w"> </span><span class="m">140068661352224</span>
<span class="c1"># (... wait 1 sec ...)</span>
Slot<span class="w"> </span>is<span class="w"> </span>executed<span class="w"> </span><span class="k">in</span><span class="w"> </span>thread<span class="w"> </span>:<span class="w"> </span><span class="m">140068661352224</span>
Done.
</pre></div>
</div>
<div class="section" id="la-solution">
<h2>La solution</h2>
<p>Un secret ? Les <tt class="docutils literal">QThread</tt> ne sont pas des threads. Elles enrobent l'execution d'une thread.</p>
<p>L'appartenance (affinité) d'un objet à une thread détermine le <a class="reference external" href="http://doc.qt.nokia.com/latest/qt.html#ConnectionType-enum">type de connexion</a> <a class="reference external" href="http://doc.qt.nokia.com/latest/threads-qobject.html#signals-and-slots-across-threads">utilisé par défaut</a>, et par conséquent le comportement lors de l'execution des slots.</p>
<p>Ce qu'il faut écrire : <tt class="docutils literal">Worker</tt> n'est plus une <tt class="docutils literal">QThread</tt>, on force son affinité dans une thread avec <tt class="docutils literal">moveToThread()</tt>.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Object</span><span class="p">(</span><span class="n">QObject</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">emitSignal</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"aSignal()"</span><span class="p">))</span>
<span class="k">class</span> <span class="nc">Worker</span><span class="p">(</span><span class="n">QObject</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">aSlot</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">thread</span><span class="p">()</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="nb">print</span> <span class="s2">"Slot is executed in thread : "</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">thread</span><span class="p">()</span><span class="o">.</span><span class="n">currentThreadId</span><span class="p">()</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">QApplication</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
<span class="nb">print</span> <span class="s2">"Main application thread is : "</span><span class="p">,</span> <span class="n">app</span><span class="o">.</span><span class="n">thread</span><span class="p">()</span><span class="o">.</span><span class="n">currentThreadId</span><span class="p">()</span>
<span class="n">worker</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">()</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">Object</span><span class="p">()</span>
<span class="n">thread</span> <span class="o">=</span> <span class="n">QThread</span><span class="p">()</span>
<span class="n">worker</span><span class="o">.</span><span class="n">moveToThread</span><span class="p">(</span><span class="n">thread</span><span class="p">)</span>
<span class="n">QObject</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"aSignal()"</span><span class="p">),</span> <span class="n">worker</span><span class="o">.</span><span class="n">aSlot</span><span class="p">)</span>
<span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">obj</span><span class="o">.</span><span class="n">emitSignal</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"Done."</span>
<span class="n">app</span><span class="o">.</span><span class="n">exec_</span><span class="p">()</span>
</pre></div>
<p>Désormais, l'exécution est bien asynchrone, comme on le souhaitait.</p>
<div class="highlight"><pre><span></span>Main<span class="w"> </span>application<span class="w"> </span>thread<span class="w"> </span>is<span class="w"> </span>:<span class="w"> </span><span class="m">139961882056480</span>
Done.
<span class="c1"># (... wait 1 sec ...)</span>
Slot<span class="w"> </span>is<span class="w"> </span>executed<span class="w"> </span><span class="k">in</span><span class="w"> </span>thread<span class="w"> </span>:<span class="w"> </span><span class="m">139961512900352</span>
</pre></div>
<p>Tout simplement ! Si j'avais lu mon article avant, je n'aurais pas perdu autant de temps à lire toutes ces docs ambiguës sur le Net.</p>
<p><strong>Sources</strong>:</p>
<ul class="simple">
<li><a class="reference external" href="http://developer.qt.nokia.com/wiki/Threads_Events_QObjects">Explications complètes</a></li>
<li><a class="reference external" href="http://doc.qt.nokia.com/4.7-snapshot/thread-basics.html">Thread Basics</a> <em>(attention au piège)</em></li>
</ul>
</div>
PostGIS data in C++ using GDAL and Qt2011-08-23T10:25:00+02:002011-08-23T10:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-08-23:/postgis-data-in-c-using-gdal-and-qt.html<p><em>Original post at</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>I did not find any ready-to-use snippets on the Web on this matter, so if
you are lucky enough, you'll find this one.</p>
<p>The objective is to read GIS geometries from a PostGIS database and manipulate
them in C++. I use Qt here, but it …</p><p><em>Original post at</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>I did not find any ready-to-use snippets on the Web on this matter, so if
you are lucky enough, you'll find this one.</p>
<p>The objective is to read GIS geometries from a PostGIS database and manipulate
them in C++. I use Qt here, but it is not really a prerequisite, it just
helps a lot. Well, actually, it saves lives.</p>
<div class="section" id="database-connection">
<h2>Database Connection</h2>
<div class="highlight"><pre><span></span><span class="n">m_db</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">QSqlDatabase</span><span class="o">::</span><span class="n">addDatabase</span><span class="p">(</span><span class="s">"QPSQL"</span><span class="p">);</span>
<span class="n">m_db</span><span class="p">.</span><span class="n">setHostName</span><span class="p">(</span><span class="s">"host"</span><span class="p">);</span>
<span class="n">m_db</span><span class="p">.</span><span class="n">setDatabaseName</span><span class="p">(</span><span class="s">"dbname"</span><span class="p">);</span>
<span class="n">m_db</span><span class="p">.</span><span class="n">setUserName</span><span class="p">(</span><span class="s">"user"</span><span class="p">);</span>
<span class="n">m_db</span><span class="p">.</span><span class="n">setPassword</span><span class="p">(</span><span class="s">"pass"</span><span class="p">);</span>
</pre></div>
<p>Do not close the database at the end of each query.</p>
<div class="highlight"><pre><span></span><span class="n">m_db</span><span class="p">.</span><span class="n">close</span><span class="p">();</span>
<span class="n">m_db</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">QSqlDatabase</span><span class="p">();</span><span class="w"> </span><span class="c1">// reinitialize for real</span>
</pre></div>
<p>Shut it down like this in your class' destructor or you may have errors like
<em>QSqlDatabasePrivate::removeDatabase: connection 'qt_sql_default_connection' is still in use, all queries will cease to work</em></p>
</div>
<div class="section" id="records-reading">
<h2>Records Reading</h2>
<div class="highlight"><pre><span></span><span class="n">QSqlQueryModel</span><span class="o">*</span><span class="w"> </span><span class="n">model</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">QSqlQueryModel</span><span class="p">();</span>
<span class="n">model</span><span class="o">-></span><span class="n">setQuery</span><span class="p">(</span><span class="s">"SELECT id, ST_AsBinary(the_geom) AS the_geom "</span>
<span class="w"> </span><span class="s">"FROM table"</span><span class="p">);</span>
<span class="kt">int</span><span class="w"> </span><span class="n">numRows</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">model</span><span class="o">-></span><span class="n">rowCount</span><span class="p">();</span>
<span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o"><</span><span class="n">numRows</span><span class="p">;</span><span class="w"> </span><span class="o">++</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Read fields</span>
<span class="w"> </span><span class="n">qlonglong</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">record</span><span class="p">(</span><span class="n">i</span><span class="p">).</span><span class="n">value</span><span class="p">(</span><span class="s">"id"</span><span class="p">).</span><span class="n">toLongLong</span><span class="p">();</span>
<span class="w"> </span><span class="n">QByteArray</span><span class="w"> </span><span class="n">wkb</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">record</span><span class="p">(</span><span class="n">i</span><span class="p">).</span><span class="n">value</span><span class="p">(</span><span class="s">"the_geom"</span><span class="p">).</span><span class="n">toByteArray</span><span class="p">();</span>
<span class="w"> </span><span class="c1">// Process !</span>
<span class="w"> </span><span class="n">processRecord</span><span class="p">(</span><span class="n">id</span><span class="p">,</span><span class="w"> </span><span class="n">wkb</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>QByteArray uses <a class="reference external" href="http://doc.qt.nokia.com/latest/implicit-sharing.html">implicit sharing</a>
and can be passed as argument without being copied.</p>
</div>
<div class="section" id="geometries-parsing">
<h2>Geometries Parsing</h2>
<p>In this part, we rely on <a class="reference external" href="http://www.gdal.org">GDAL (Geospatial Data Abstraction Library)</a>
<a class="reference external" href="http://www.gdal.org/ogr/osr_tutorial.html">OGRSpatialReference</a>.</p>
<p>It provides an API to access geometries coordinates etc.</p>
<div class="highlight"><pre><span></span><span class="cp">#include</span><span class="w"> </span><span class="cpf">"ogrsf_frmts.h"</span><span class="c1"> // GDAL</span>
<span class="p">...</span>
<span class="p">...</span>
<span class="kt">void</span><span class="w"> </span><span class="n">Class</span><span class="o">::</span><span class="n">processRecord</span><span class="p">(</span><span class="n">qlonglong</span><span class="w"> </span><span class="n">id</span><span class="p">,</span><span class="w"> </span><span class="n">QByteArray</span><span class="w"> </span><span class="n">wkb</span><span class="p">)</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">OGRSpatialReference</span><span class="w"> </span><span class="n">osr</span><span class="p">;</span>
<span class="w"> </span><span class="n">OGRGeometry</span><span class="w"> </span><span class="o">*</span><span class="n">geom</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">NULL</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// Parse WKB</span>
<span class="w"> </span><span class="n">OGRErr</span><span class="w"> </span><span class="n">err</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">OGRGeometryFactory</span><span class="o">::</span><span class="n">createFromWkb</span><span class="p">((</span><span class="kt">unsigned</span><span class="w"> </span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="n">wkb</span><span class="p">.</span><span class="n">constData</span><span class="p">(),</span><span class="w"> </span><span class="o">&</span><span class="n">osr</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">geom</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">err</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">OGRERR_NONE</span><span class="p">){</span>
<span class="w"> </span><span class="c1">// process error, like emit signal</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1">// Analyse geometry by type and process them as you wish</span>
<span class="w"> </span><span class="n">OGRwkbGeometryType</span><span class="w"> </span><span class="n">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">wkbFlatten</span><span class="p">(</span><span class="n">geom</span><span class="o">-></span><span class="n">getGeometryType</span><span class="p">());</span>
<span class="w"> </span><span class="k">switch</span><span class="p">(</span><span class="n">type</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="no">wkbLineString</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">OGRLineString</span><span class="w"> </span><span class="o">*</span><span class="n">poRing</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">OGRLineString</span><span class="o">*</span><span class="p">)</span><span class="n">geom</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// Access line string nodes for example :</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">numNode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">poRing</span><span class="o">-></span><span class="n">getNumPoints</span><span class="p">();</span>
<span class="w"> </span><span class="n">OGRPoint</span><span class="w"> </span><span class="n">p</span><span class="p">;</span>
<span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">numNode</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">poRing</span><span class="o">-></span><span class="n">getPoint</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">p</span><span class="p">);</span>
<span class="w"> </span><span class="n">qDebug</span><span class="p">()</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">getX</span><span class="p">()</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">p</span><span class="p">.</span><span class="n">getY</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="no">wkbMultiLineString</span><span class="p">:</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">OGRGeometryCollection</span><span class="w"> </span><span class="o">*</span><span class="n">poCol</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">OGRGeometryCollection</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="n">geom</span><span class="p">;</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">numCol</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">poCol</span><span class="o">-></span><span class="n">getNumGeometries</span><span class="p">();</span>
<span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o"><</span><span class="n">numCol</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Access line length for example :</span>
<span class="w"> </span><span class="n">qDebug</span><span class="p">()</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="n">poCol</span><span class="o">-></span><span class="n">getGeometryRef</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="o">-></span><span class="n">get_Length</span><span class="p">();</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">default</span><span class="o">:</span>
<span class="w"> </span><span class="c1">// process error, like emit signal</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1">// Clean-up</span>
<span class="w"> </span><span class="n">OGRGeometryFactory</span><span class="o">::</span><span class="n">destroyGeometry</span><span class="p">(</span><span class="n">geom</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
<p>In this snippet, I only process linestrings, but all <a class="reference external" href="http://www.gdal.org/ogr/ogr__core_8h.html#800236a0d460ef66e687b7b65610f12a">geometry types are available</a>.
Consider writing a recursive function for geometry collections and so forth...</p>
<p>Hope this helped !</p>
</div>
Django, gestion des settings d'application simplifiée2011-07-29T18:30:00+02:002011-07-29T18:30:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-07-29:/django-gestion-des-settings-dapplication-simplifiee-fr.html<p>Je reprends ici la méthode présentée par <a class="reference external" href="http://blog.akei.com/post/4575980188/une-autre-facon-de-gerer-ses-settings-dapplication">Nicolas</a> et je la couple avec mon petit <a class="reference external" href="https://github.com/makinacorpus/easydict">EasyDict</a> pour alléger l'utilisation !
Comme ses snippets sont drôles, je ne les change pas !</p>
<div class="section" id="parametres-par-default-de-l-application">
<h2>Paramètres par défault de l'application</h2>
<p>On a juste un constructeur à ajouter par rapport à ce qu'avait présenté NiKo (avec <a class="reference external" href="http://pypi.python.org/pypi/easydict/">EasyDict …</a></p></div><p>Je reprends ici la méthode présentée par <a class="reference external" href="http://blog.akei.com/post/4575980188/une-autre-facon-de-gerer-ses-settings-dapplication">Nicolas</a> et je la couple avec mon petit <a class="reference external" href="https://github.com/makinacorpus/easydict">EasyDict</a> pour alléger l'utilisation !
Comme ses snippets sont drôles, je ne les change pas !</p>
<div class="section" id="parametres-par-default-de-l-application">
<h2>Paramètres par défault de l'application</h2>
<p>On a juste un constructeur à ajouter par rapport à ce qu'avait présenté NiKo (avec <a class="reference external" href="http://pypi.python.org/pypi/easydict/">EasyDict installé</a>).</p>
<div class="highlight"><pre><span></span><span class="c1"># apps/my_app/__init__.py</span>
<span class="kn">from</span> <span class="nn">django.conf</span> <span class="kn">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">easydict</span> <span class="kn">import</span> <span class="n">EasyDict</span>
<span class="n">app_settings</span> <span class="o">=</span> <span class="n">EasyDict</span><span class="p">(</span><span class="nb">dict</span><span class="p">({</span>
<span class="s1">'FOO'</span><span class="p">:</span> <span class="mi">42</span><span class="p">,</span>
<span class="s1">'ENABLE_CHUCK_NORRIZ_MODE'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="p">},</span> <span class="o">**</span><span class="nb">getattr</span><span class="p">(</span><span class="n">settings</span><span class="p">,</span> <span class="s1">'MY_APP_CONFIG'</span><span class="p">,</span> <span class="p">{})))</span>
</pre></div>
</div>
<div class="section" id="surcharge-dans-le-projet">
<h2>Surcharge dans le projet</h2>
<div class="highlight"><pre><span></span><span class="c1"># settings.py</span>
<span class="n">MY_APP_CONFIG</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'ENABLE_CHUCK_NORRIZ_MODE'</span><span class="p">:</span> <span class="kc">True</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="utilisation">
<h2>Utilisation !</h2>
<p>EasyDict transforme les clés du <cite>dict</cite> en attributs, on accède aux settings en toute simplicité !</p>
<div class="highlight"><pre><span></span><span class="c1"># foo/bar.py</span>
<span class="kn">from</span> <span class="nn">my_app</span> <span class="kn">import</span> <span class="n">app_settings</span>
<span class="nb">print</span> <span class="n">app_settings</span><span class="o">.</span><span class="n">FOO</span> <span class="c1"># 42</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># apps/my_app/utils.py</span>
<span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">app_settings</span>
<span class="k">if</span> <span class="n">app_settings</span><span class="o">.</span><span class="n">ENABLE_CHUCK_NORRIZ_MODE</span><span class="p">:</span>
<span class="nb">print</span> <span class="s1">'Chuck Norriz is watching you'</span>
<span class="k">else</span><span class="p">:</span>
<span class="nb">print</span> <span class="s1">'Dance dance, little lamb'</span>
</pre></div>
</div>
Carte des vélos avec Leaflet2011-05-30T15:25:00+02:002011-05-30T15:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-05-30:/carte-des-velos-avec-leaflet-fr.html<p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Les bookmarks, un peu comme les cahiers de recettes, c'est bien de les remplir mais encore faut-il trouver les bons au moment adéquate !
Même quand il s'agit d'outils, de bibliothèques et de services Web, il faut trouver l'occasion de les tester avant le grand …</p><p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Les bookmarks, un peu comme les cahiers de recettes, c'est bien de les remplir mais encore faut-il trouver les bons au moment adéquate !
Même quand il s'agit d'outils, de bibliothèques et de services Web, il faut trouver l'occasion de les tester avant le grand soir !
Et si on veut en faire un article de blog, alors là, il faut en plus donner envie d'y goûter :)</p>
<p>Ici, je prends plein d'ingrédients trouvés au bord des chemins :</p>
<ul class="simple">
<li><a class="reference external" href="http://packages.python.org/pyquery/">pyquery</a></li>
<li><a class="reference external" href="http://leaflet.cloudmade.com">leaflet</a></li>
<li><a class="reference external" href="http://developer.yahoo.com/yql/">Yahoo Query Language</a></li>
<li><a class="reference external" href="http://mustache.github.com/">mustache</a></li>
</ul>
<p>Je secoue bien fort ! (sans oublier de saupoudrer de <a class="reference external" href="http://jquery.com">jquery</a>) et j'obtiens une carte interactive des stations vélos de Toulouse !</p>
<div class="section" id="la-liste-des-stations">
<h2>La liste des stations</h2>
<p>Sur le site <a class="reference external" href="http://velonow.info">http://velonow.info</a>, je récupère un fichier XML qui contient
la liste statique des stations de vélo et leurs identifiants.</p>
<p>C'est l'occasion d'utiliser <a class="reference external" href="http://packages.python.org/pyquery/">pyquery</a> pour le transformer en GeoJSON. <a class="reference external" href="http://www.gawel.org">Gawel</a> nous l'avait présenté aux djangocongs, il s'agit du portage de l'API de JQuery en python !</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pyquery</span> <span class="kn">import</span> <span class="n">PyQuery</span> <span class="k">as</span> <span class="n">pq</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">pq</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="s1">'http://server.com/file.xml'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">e</span><span class="p">:</span> <span class="n">pq</span><span class="p">(</span><span class="n">e</span><span class="p">),</span> <span class="n">d</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'marker'</span><span class="p">)):</span>
<span class="n">pt</span> <span class="o">=</span> <span class="n">geojson</span><span class="o">.</span><span class="n">Point</span><span class="p">([</span><span class="n">m</span><span class="o">.</span><span class="n">attr</span><span class="p">(</span><span class="s1">'lng'</span><span class="p">),</span>
<span class="n">m</span><span class="o">.</span><span class="n">attr</span><span class="p">(</span><span class="s1">'lat'</span><span class="p">)])</span>
<span class="o">...</span>
<span class="o">...</span>
</pre></div>
<p>Je trouve ça génial d'avoir la même syntaxe de manipulation du DOM en python et en javascript ! Et pour faire du webscrapping, c'est top !</p>
</div>
<div class="section" id="affichage-de-la-carte">
<h2>Affichage de la carte</h2>
<p><a class="reference external" href="http://cloudmade.com">Cloudmade</a> a créé <a class="reference external" href="http://leaflet.cloudmade.com">leaflet</a> qui rejoint <a class="reference external" href="http://www.tile5.org">Tile5</a> et <a class="reference external" href="http://polymaps.org">Polymaps</a> en tant que challenger d'Openlayers !</p>
<p>C'est une bibliothèque légère, jolie, fluide, optimisée pour les mobiles,
et même compatible Internet Explorer !</p>
<p>Pour afficher une carte centrée sur la localisation du visiteur de la page, il suffit de faire ça :</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nb">Map</span><span class="p">(</span><span class="s1">'map'</span><span class="p">);</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">cloudmadeUrl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'http://{s}.tile.cloudmade.com/BC9A493B41014CAABB98F0471D759707/997/256/{z}/{x}/{y}.png'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">cloudmade</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">TileLayer</span><span class="p">(</span><span class="nx">cloudmadeUrl</span><span class="p">);</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">addLayer</span><span class="p">(</span><span class="nx">cloudmade</span><span class="p">);</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">locateAndSetView</span><span class="p">();</span>
</pre></div>
<p>Pour l'instant, Leaflet ne gère pas les couches au format GeoJSON, en
attendant <a class="reference external" href="https://github.com/CloudMade/Leaflet/issues/13">la prochaine release</a>,
nous allons ajouter les points des stations en 2 coups de cuillère à pot :</p>
<div class="highlight"><pre><span></span><span class="nx">$</span><span class="p">.</span><span class="nx">getJSON</span><span class="p">(</span><span class="nx">url</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">$</span><span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">features</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">i</span><span class="p">,</span><span class="w"> </span><span class="nx">f</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">cc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">f</span><span class="p">.</span><span class="nx">geometry</span><span class="p">.</span><span class="nx">coordinates</span><span class="p">;</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">Marker</span><span class="p">(</span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">LatLng</span><span class="p">(</span><span class="nx">cc</span><span class="p">[</span><span class="mf">1</span><span class="p">],</span><span class="w"> </span><span class="nx">cc</span><span class="p">[</span><span class="mf">0</span><span class="p">]));</span>
<span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">addLayer</span><span class="p">(</span><span class="nx">marker</span><span class="p">);</span>
<span class="w"> </span><span class="p">});</span>
<span class="p">});</span>
</pre></div>
</div>
<div class="section" id="details-d-une-station-en-popup">
<h2>Détails d'une station en popup</h2>
<p>Les détails d'une station (nombre de vélos, emplacements, libres, occupés) sont disponibles en fournissant un identifiant sur le site de <a class="reference external" href="http://www.velo.toulouse.fr">velo toulouse</a>.
Mais lorsqu'on appelle la page en Ajax, le corps de la réponse XML est vide. Une protection contre la bidouillabilité sûrement.</p>
<p>C'est là que <a class="reference external" href="http://developer.yahoo.com/yql/">Yahoo Query Language</a> nous aide ! On passe par Yahoo pour accèder aux ressources du Web avec <a class="reference external" href="http://developer.yahoo.com/yql/console/">des requêtes similaires aux bases de données</a> !</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">yql</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"select * from xml where url = '"</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">url</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s2">"'"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">yqlurl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'http://query.yahooapis.com/v1/public/yql?q='</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">encodeURIComponent</span><span class="p">(</span><span class="nx">yql</span><span class="p">);</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">yqlurl</span><span class="p">,</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// show data in pop up !</span>
<span class="p">});</span>
</pre></div>
<p>Je fais une petite fonction pour transformer l'XML récupéré en objet :</p>
<div class="highlight"><pre><span></span><span class="kd">function</span><span class="w"> </span><span class="nx">xml2obj</span><span class="p">(</span><span class="nx">xmldata</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">d</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{};</span>
<span class="w"> </span><span class="nx">$</span><span class="p">(</span><span class="nx">xmldata</span><span class="p">).</span><span class="nx">children</span><span class="p">().</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">index</span><span class="p">,</span><span class="w"> </span><span class="nx">value</span><span class="p">){</span>
<span class="w"> </span><span class="nx">d</span><span class="p">[</span><span class="nx">$</span><span class="p">(</span><span class="nx">value</span><span class="p">).</span><span class="nx">get</span><span class="p">(</span><span class="mf">0</span><span class="p">).</span><span class="nx">nodeName</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">$</span><span class="p">(</span><span class="nx">value</span><span class="p">).</span><span class="nx">text</span><span class="p">();</span>
<span class="w"> </span><span class="p">});</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">d</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="nt"><station></span>
<span class="w"> </span><span class="nt"><free></span>12<span class="nt"></free></span>
<span class="w"> </span><span class="nt"><available></span>4<span class="nt"></available></span>
<span class="w"> </span><span class="nt"><total></span>16<span class="nt"></total></span>
<span class="nt"></station></span>
</pre></div>
<p>devient :</p>
<div class="highlight"><pre><span></span><span class="p">{</span>
<span class="w"> </span><span class="nx">free</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mf">12</span><span class="p">,</span>
<span class="w"> </span><span class="nx">available</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="mf">4</span><span class="p">,</span>
<span class="w"> </span><span class="nx">total</span><span class="o">:</span><span class="w"> </span><span class="mf">16</span>
<span class="p">}</span>
</pre></div>
<p>Pour mettre en forme ces informations dans la pop-up, nous allons utiliser <a class="reference external" href="http://mustache.github.com/">mustache</a> !
Conceptuellement, il s'agit tout simplement d'un moteur de template avec <a class="reference external" href="http://mustache.github.com/mustache.5.html">une syntaxe simplifiée</a> ! Il y a une implémentation
dans quasiment tous les languages, dont Javascript.</p>
<p>Cela évite principalement de faire du code javascript pour la mise en forme des données, notamment pour
celles récupérées en JSON via Ajax.</p>
<p>On construit une chaîne avec les fameuses <cite>{{}}</cite> et on fournit un objet pour substituer les valeurs :</p>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">xml2obj</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="nx">xmldata</span><span class="p">).</span><span class="nx">find</span><span class="p">(</span><span class="s1">'station'</span><span class="p">)),</span>
<span class="w"> </span><span class="nx">template</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"<h2>Station #{{ number }}</h2></span>
<span class="s2"> <p>{{ address }}</p> \</span>
<span class="s2"> {{# station }} \</span>
<span class="s2"> <ul> \</span>
<span class="s2"> <li>{{ available }} available</li> \</span>
<span class="s2"> <li>{{ free }} free slots</li> \</span>
<span class="s2"> </ul> \</span>
<span class="s2"> {{/ station }}"</span><span class="p">,</span>
<span class="w"> </span><span class="nx">content</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Mustache</span><span class="p">.</span><span class="nx">to_html</span><span class="p">(</span><span class="nx">template</span><span class="p">,</span><span class="w"> </span><span class="nx">data</span><span class="p">);</span>
<span class="c1">// Show marker popup !</span>
<span class="nx">marker</span><span class="p">.</span><span class="nx">bindPopup</span><span class="p">(</span><span class="nx">content</span><span class="p">).</span><span class="nx">openPopup</span><span class="p">();</span>
</pre></div>
<p>Et voilà !</p>
<img alt="" src="/images/leaflet-velo.png" />
</div>
URL reverse en Javascript avec django2011-05-27T15:25:00+02:002011-05-27T15:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-05-27:/url-reverse-en-javascript-avec-django-fr.html<p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Un aspect fondamental de la philosophie django consiste à éviter toute sorte de redondance : <a class="reference external" href="https://docs.djangoproject.com/en/dev/misc/design-philosophies/">Don't Repeat Yourself</a>.</p>
<p>La tentation est souvent trop belle, et respecter les fondamentaux s'avère parfois difficile ! C'est le cas de la réécriture d'URL en Javascript.</p>
<p>Imaginons l'URL suivante définie dans …</p><p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Un aspect fondamental de la philosophie django consiste à éviter toute sorte de redondance : <a class="reference external" href="https://docs.djangoproject.com/en/dev/misc/design-philosophies/">Don't Repeat Yourself</a>.</p>
<p>La tentation est souvent trop belle, et respecter les fondamentaux s'avère parfois difficile ! C'est le cas de la réécriture d'URL en Javascript.</p>
<p>Imaginons l'URL suivante définie dans <cite>urls.py</cite></p>
<div class="highlight"><pre><span></span><span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^/plop/(?P<x>\d)/(?P<y>\d)$'</span><span class="p">,</span> <span class="n">plopview</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">"plop"</span><span class="p">)</span>
</pre></div>
<p>Pour utiliser cette URL en Javascript avec des paramètres variables, on peut imaginer plusieurs approches.</p>
<div class="section" id="j-aime-pas">
<h2>J'aime pas</h2>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">generic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"{% url 'plop' 0 0 %}"</span><span class="p">;</span>
<span class="nx">generic</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="s1">'0/0'</span><span class="p">,</span><span class="w"> </span><span class="nx">x</span><span class="o">+</span><span class="s1">'/'</span><span class="o">+</span><span class="nx">y</span><span class="p">);</span>
</pre></div>
<p>Pas DRY ! à cause des <cite>/</cite>.</p>
<div class="highlight"><pre><span></span><span class="s2">"{% url 'plop' 0 0 %}"</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">'../../'</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">'/'</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nx">y</span><span class="p">;</span>
</pre></div>
<p>Pas DRY non plus !</p>
<p>On peut aussi changer le pattern pour éviter les <cite>/</cite>.</p>
<div class="highlight"><pre><span></span><span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^/plop/(?P<x>[\d]|x)/(?P<y>[\d]|y)$'</span><span class="p">,</span> <span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">"plop"</span><span class="p">)</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="kd">var</span><span class="w"> </span><span class="nx">generic</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"{% url 'toto' 'x' 'y' %}"</span><span class="p">;</span>
<span class="nx">generic</span><span class="p">.</span><span class="nx">replace</span><span class="p">(</span><span class="s1">'x'</span><span class="p">,</span><span class="w"> </span><span class="nx">x</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="s1">'y'</span><span class="p">,</span><span class="w"> </span><span class="nx">y</span><span class="p">);</span>
</pre></div>
<p>C'est mieux, mais pas DRY ! à cause des <cite>x</cite>, <cite>y</cite>.</p>
<p>On pourrait aussi imaginer une vue django qui ferait le <cite>reverse()</cite>. Mais cela multiplierait les aller-retours serveur, ce qui n'est pas toujours recommandé...</p>
</div>
<div class="section" id="la-bonne">
<h2>La bonne</h2>
<p>Il existe une application pour ça ! <a class="reference external" href="https://github.com/Dimitri-Gnidash/django-js-utils">django-js-utils</a></p>
<p>Elle se charge de générer un fichier Javascript (<cite>settings.URLS_JS_GENERATED_FILE</cite>) grâce à une commande de gestion</p>
<pre class="literal-block">
python manage.js js_urls
</pre>
<p>Ensuite on utilise explicitement le fichier généré</p>
<div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"{{ MEDIA_URL }}/js/dutils.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">type</span><span class="o">=</span><span class="s">"text/javascript"</span> <span class="na">src</span><span class="o">=</span><span class="s">"{{ MEDIA_URL }}/js/dutils.conf.urls.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
</pre></div>
<p>Et on fait du vrai DRY !</p>
<div class="highlight"><pre><span></span><span class="nx">dutils</span><span class="p">.</span><span class="nx">urls</span><span class="p">.</span><span class="nx">resolve</span><span class="p">(</span><span class="s1">'plop'</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s1">'x'</span><span class="o">:</span><span class="w"> </span><span class="nx">x</span><span class="p">,</span><span class="w"> </span><span class="s1">'y'</span><span class="o">:</span><span class="w"> </span><span class="nx">y</span><span class="w"> </span><span class="p">})</span>
</pre></div>
<p>Gagné ! \o/</p>
<p>Un inconvénient à noter tout de même : la liste de l'ensemble des URLs de l'application est accessible au public. Mais j'ai pas mieux ma pauvre dame !</p>
</div>
Django et Jenkins2011-04-28T17:25:00+02:002011-04-28T17:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-04-28:/django-et-jenkins-fr.html<p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Lors des <a class="reference external" href="http://rencontres.django-fr.org/2011/">Recontres Django 2011</a>, <a class="reference external" href="http://www.akei.com">Nicolas Perriault</a> a présenté les principes de l'<a class="reference external" href="http://fr.wikipedia.org/wiki/Int%C3%A9gration_continue">intégration continue</a> avec <a class="reference external" href="http://djangoproject.com">Django</a> et <a class="reference external" href="http://jenkins-ci.org/">Jenkins</a>.</p>
<p>Le diaporama, <a class="reference external" href="http://www.akei.com/presentations/2011-Djangocong/index.html">disponible en ligne</a>, suffit amplement pour démarrer !</p>
<p>Mais pour qu'un projet django soit testé facilement, il doit se déployer et se lancer facilement …</p><p><em>Article original publié chez</em> <a class="reference external" href="http://makina-corpus.org">Makina Corpus</a></p>
<p>Lors des <a class="reference external" href="http://rencontres.django-fr.org/2011/">Recontres Django 2011</a>, <a class="reference external" href="http://www.akei.com">Nicolas Perriault</a> a présenté les principes de l'<a class="reference external" href="http://fr.wikipedia.org/wiki/Int%C3%A9gration_continue">intégration continue</a> avec <a class="reference external" href="http://djangoproject.com">Django</a> et <a class="reference external" href="http://jenkins-ci.org/">Jenkins</a>.</p>
<p>Le diaporama, <a class="reference external" href="http://www.akei.com/presentations/2011-Djangocong/index.html">disponible en ligne</a>, suffit amplement pour démarrer !</p>
<p>Mais pour qu'un projet django soit testé facilement, il doit se déployer et se lancer facilement ! C'est certes l'occasion de peaufiner l'automatisation, mais c'est loin d'être trivial quand il y a du SIG, du <a class="reference external" href="http://celeryproject.org">celery</a> ...
Je vais tenter de partager mes notes dans ce billet.</p>
<div class="section" id="le-minimum-requis">
<h2>Le minimum requis</h2>
<p>Pour l'installation de Jenkins, rien de plus simple (<em>sur debian</em>)</p>
<pre class="literal-block">
sudo aptitude install jenkins
</pre>
<p>Mais il va falloir lui donner de quoi télécharger votre code sur <cite>git</cite> et parfois compiler les librairies python nécessaires</p>
<pre class="literal-block">
sudo aptitude install git-core
sudo aptitude install python-dev build-essential python-virtualenv
</pre>
<p>Les plugins indispensables :</p>
<ul class="simple">
<li>covertura</li>
<li>Violations</li>
<li>GIT</li>
<li>Green Balls</li>
<li>Continuous Integration Game</li>
</ul>
</div>
<div class="section" id="organisation-du-projet-django">
<h2>Organisation du projet Django</h2>
<ul>
<li><p class="first">Définition des dépendances globales dans <cite>requirements.txt</cite></p>
<pre class="literal-block">
Django>=1.3
south
</pre>
</li>
<li><p class="first">Définition des dépendances liées aux tests dans <cite>requirements-testing.txt</cite></p>
<pre class="literal-block">
django-jenkins
</pre>
</li>
<li><p class="first">Ajout d'un fichier <cite>pylint.rc</cite> pour régler les niveaux d'alerte <a class="reference external" href="http://www.python.org/dev/peps/pep-0008/">PEP-8</a></p>
<pre class="literal-block">
[MESSAGES CONTROL]
disable=E1101,E1103,C0111,I0011,I0012,W0704,W0142,W0212,W0232,W0613,W0702,R0201
...
...
</pre>
</li>
<li><p class="first">Modèle de settings de tests dans <cite>project/test_settings.py</cite></p>
</li>
</ul>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">default_settings</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">DEBUG</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">TEMPLATE_DEBUG</span> <span class="o">=</span> <span class="n">DEBUG</span>
<span class="n">INSTALLED_APPS</span> <span class="o">+=</span> <span class="p">(</span>
<span class="s1">'django_jenkins'</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">PYLINT_RCFILE</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">PROJECT_ROOT_PATH</span><span class="p">,</span> <span class="s1">'..'</span><span class="p">,</span> <span class="s1">'conf'</span><span class="p">,</span> <span class="s1">'pylint.rc'</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="configuration-du-job-jenkins">
<h2>Configuration du job Jenkins</h2>
<p>Les informations de la présentation de Nicolas suffisent pour démarrer.</p>
<p>J'ai noté cependant qu'il fallait lancer <cite>manage.py</cite> depuis un répertoire parent au projet pour que l'exploration du code source fonctionne.</p>
<p>Pour profiter de la magie des ingrédients précédents, nous aurons donc juste à ajouter un bloc script shell, qui installe les dépendances listées, pose les settings de test et migre la base (avec <a class="reference external" href="http://south.aeracode.org">South</a>):</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/bash -ex</span>
virtualenv<span class="w"> </span>--quiet<span class="w"> </span>ve
<span class="nb">source</span><span class="w"> </span>./ve/bin/activate
pip<span class="w"> </span>install<span class="w"> </span>-E<span class="w"> </span>./ve<span class="w"> </span>-r<span class="w"> </span><span class="nv">$WORKSPACE</span>/requirements.txt
pip<span class="w"> </span>install<span class="w"> </span>-E<span class="w"> </span>./ve<span class="w"> </span>-r<span class="w"> </span><span class="nv">$WORKSPACE</span>/requirements-testing.txt
cp<span class="w"> </span><span class="nv">$WORKSPACE</span>/project/test_settings.py<span class="w"> </span><span class="nv">$WORKSPACE</span>/project/local_settings.py
python<span class="w"> </span><span class="nv">$WORKSPACE</span>/project/manage.py<span class="w"> </span>syncdb<span class="w"> </span>--noinput
python<span class="w"> </span><span class="nv">$WORKSPACE</span>/project/manage.py<span class="w"> </span>migrate
deactivate
</pre></div>
<p>et celui-ci pour lancer les tests proprements dits :</p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/bash -ex</span>
virtualenv<span class="w"> </span>--quiet<span class="w"> </span>ve
<span class="nb">source</span><span class="w"> </span>./ve/bin/activate
python<span class="w"> </span><span class="nv">$WORKSPACE</span>/project/manage.py<span class="w"> </span>jenkins<span class="w"> </span>yourapps
deactivate
</pre></div>
</div>
<div class="section" id="pour-un-projet-sig">
<h2>Pour un projet SIG</h2>
<p>Il faut installer certaines librairies SIG sur le serveur Jenkins.</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>aptitude<span class="w"> </span>install<span class="w"> </span>libproj0<span class="w"> </span>libgeos-c1
</pre></div>
<p>Si le besoin de cloisonner ces librairies pour chaque projet se fait ressentir, il faut utiliser des outils comme <a class="reference external" href="http://www.minitage.org">minitage</a>.</p>
<div class="section" id="spatialite-au-lieu-de-postgis-comme-base-de-tests">
<h3>Spatialite au lieu de PostGIS comme base de tests</h3>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>aptitude<span class="w"> </span>install<span class="w"> </span>python-sqlite<span class="w"> </span>libspatialite2<span class="w"> </span>sqlite3
</pre></div>
<p>Script d'initialisation</p>
<div class="highlight"><pre><span></span>wget<span class="w"> </span>http://www.gaia-gis.it/spatialite/init_spatialite-2.3.zip<span class="w"> </span>-O<span class="w"> </span>/tmp/init_spatialite-2.3.zip
<span class="nb">cd</span><span class="w"> </span>/usr/local/lib/
sudo<span class="w"> </span>unzip<span class="w"> </span>/tmp/init_spatialite-2.3.zip
</pre></div>
<p>avec dans <cite>test_settings.py</cite></p>
<div class="highlight"><pre><span></span><span class="n">DATABASES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'default'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'ENGINE'</span><span class="p">:</span> <span class="s1">'django.contrib.gis.db.backends.spatialite'</span><span class="p">,</span>
<span class="o">...</span>
<span class="o">...</span>
<span class="n">SPATIALITE_SQL</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">'usr'</span><span class="p">,</span> <span class="s1">'local'</span><span class="p">,</span> <span class="s1">'lib'</span><span class="p">,</span> <span class="s1">'init_spatialite-2.3.sql'</span><span class="p">)</span>
</pre></div>
<p>Si pysqlite n'a pas été compilé avec les extensions C (Erreur: <em>The pysqlite library does not support C extension loading.</em>) il va falloir le recompiler !</p>
<div class="highlight"><pre><span></span>sudo<span class="w"> </span>aptitude<span class="w"> </span>install<span class="w"> </span>libsqlite3-dev
wget<span class="w"> </span>http://pysqlite.googlecode.com/files/pysqlite-2.6.3.tar.gz
tar<span class="w"> </span>-zxvf<span class="w"> </span>pysqlite-2.6.3.tar.gz
<span class="nb">cd</span><span class="w"> </span>pysqlite-2.6.3
sed<span class="w"> </span>-i<span class="w"> </span>s/define<span class="o">=</span>SQLITE_OMIT_LOAD_EXTENSION/#define<span class="o">=</span>SQLITE_OMIT_LOAD_EXTENSION/g<span class="w"> </span>setup.cfg
<span class="nb">source</span><span class="w"> </span>./ve/bin/activate
python<span class="w"> </span>setup.py<span class="w"> </span>install
</pre></div>
</div>
</div>
<div class="section" id="pour-un-projet-celery">
<h2>Pour un projet Celery</h2>
<div class="section" id="kombu-au-lieu-de-rabbitmq-comme-gestionnaire-de-messages">
<h3>Kombu au lieu de RabbitMQ comme gestionnaire de messages</h3>
<p><cite>requirements-testing.txt</cite></p>
<pre class="literal-block">
kombu
djkombu
</pre>
<p><cite>test_settings.py</cite></p>
<div class="highlight"><pre><span></span><span class="n">INSTALLED_APPS</span> <span class="o">+=</span> <span class="p">(</span>
<span class="s1">'djkombu'</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">CARROT_BACKEND</span> <span class="o">=</span> <span class="s2">"django"</span>
</pre></div>
<p>Pour désactiver la parallélisation lors des tests</p>
<div class="highlight"><pre><span></span><span class="n">CELERY_ALWAYS_EAGER</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div>
</div>
</div>
landez : fabriquer facilement des fichiers MBTiles en python2011-04-21T12:04:00+02:002011-04-21T12:04:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-04-21:/landez-fabriquer-facilement-des-fichiers-mbtiles-en-python-fr.html<p><a class="reference external" href="http://www.makina-corpus.org/blog/integration-mbtiles-format-android">Une nouvelle fois</a>,
Makina Corpus se rapproche du projet <a class="reference external" href="http://mapbox.com/">MapBox</a>, avec une contribution
sur la librairie <a class="reference external" href="https://github.com/mapbox/mbutil/contributors">mbutil</a>, qui permet de fabriquer des fichiers MBTiles.</p>
<p>Nous l'utilisons dans <a class="reference external" href="https://github.com/makinacorpus/landez">landez</a>, un outil qui permet
de créer des fichiers MBTiles à partir de sites de tuiles externes ou de feuilles de styles Mapnik …</p><p><a class="reference external" href="http://www.makina-corpus.org/blog/integration-mbtiles-format-android">Une nouvelle fois</a>,
Makina Corpus se rapproche du projet <a class="reference external" href="http://mapbox.com/">MapBox</a>, avec une contribution
sur la librairie <a class="reference external" href="https://github.com/mapbox/mbutil/contributors">mbutil</a>, qui permet de fabriquer des fichiers MBTiles.</p>
<p>Nous l'utilisons dans <a class="reference external" href="https://github.com/makinacorpus/landez">landez</a>, un outil qui permet
de créer des fichiers MBTiles à partir de sites de tuiles externes ou de feuilles de styles Mapnik.</p>
<p>Son utilisation est fort simple !</p>
<p>Pour un service de tuiles externe :</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">landez</span> <span class="kn">import</span> <span class="n">MBTilesBuilder</span>
<span class="c1"># downloads from Cloudmade by default, be careful with terms of usage !</span>
<span class="n">mb</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">(</span><span class="n">remote</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">filepath</span><span class="o">=</span><span class="s2">"dest.mbtiles"</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">add_coverage</span><span class="p">(</span><span class="n">bbox</span><span class="o">=</span><span class="p">(</span><span class="o">-</span><span class="mf">90.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">180.0</span><span class="p">,</span> <span class="mf">180.0</span><span class="p">,</span> <span class="mf">90.0</span><span class="p">),</span>
<span class="n">zoomlevels</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>
<span class="n">mb</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
<p>Avec une feuille de style locale :</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">landez</span> <span class="kn">import</span> <span class="n">MBTilesBuilder</span>
<span class="n">mb</span> <span class="o">=</span> <span class="n">MBTilesBuilder</span><span class="p">(</span><span class="n">stylefile</span><span class="o">=</span><span class="s2">"yourstyle.xml"</span><span class="p">,</span> <span class="n">filepath</span><span class="o">=</span><span class="s2">"dest.mbtiles"</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">add_coverage</span><span class="p">(</span><span class="n">bbox</span><span class="o">=</span><span class="p">(</span><span class="o">-</span><span class="mf">90.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">180.0</span><span class="p">,</span> <span class="mf">180.0</span><span class="p">,</span> <span class="mf">90.0</span><span class="p">),</span>
<span class="n">zoomlevels</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span>
<span class="n">mb</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
<p>Une branche experimentale a été initiée pour travailler sur le multiprocessing
afin de paralléliser au maximum la fabrication des tuiles en amont. Si cela vous intéresse, soyez les bienvenus !</p>
Deploy Django behind a reverse proxy2011-04-19T00:00:00+02:002011-04-19T00:00:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-04-19:/deploy-django-behind-a-reverse-proxy.html<p>By default, Django will assume that your root URL is the root (<cite>/</cite>) of your domain.</p>
<p>Using a <a class="reference external" href="http://en.wikipedia.org/wiki/Reverse_proxy">reverse proxy</a>, we can run multiple django instances on the same server, using the same domain. (<cite>http://server.org/site1/</cite>, <cite>http://server.org/site2/</cite>, ...)</p>
<p>Many redirects of your application will then be broken …</p><p>By default, Django will assume that your root URL is the root (<cite>/</cite>) of your domain.</p>
<p>Using a <a class="reference external" href="http://en.wikipedia.org/wiki/Reverse_proxy">reverse proxy</a>, we can run multiple django instances on the same server, using the same domain. (<cite>http://server.org/site1/</cite>, <cite>http://server.org/site2/</cite>, ...)</p>
<p>Many redirects of your application will then be broken (most notable is validation of login form). You can fix that by forcing the root URL in your settings</p>
<pre class="literal-block">
FORCE_SCRIPT_NAME = '/site1'
</pre>
<p>If you use <a class="reference external" href="https://github.com/dcramer/django-sentry">Sentry</a>, you'll also have to set</p>
<pre class="literal-block">
SENTRY_URL_PREFIX = '/site1'
</pre>
Ajouter un sélecteur de couches (layer switcher) à polymaps2011-03-09T12:03:00+01:002011-03-09T12:03:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-03-09:/ajouter-un-selecteur-de-couches-layer-switcher-a-polymaps-fr.html<p>Je fais partie de ceux qui sont persuadés que <a class="reference external" href="http://polymaps.org/">polymaps</a> est tout à fait mature !
Certes, il ne fournit pas autant de fonctionnalités et de connecteurs qu'OpenLayers,
mais il ne semble pas que ce soit son objectif ! Il est léger et personnalisable à souhait !</p>
<p>Pour le prouver, nous allons créer …</p><p>Je fais partie de ceux qui sont persuadés que <a class="reference external" href="http://polymaps.org/">polymaps</a> est tout à fait mature !
Certes, il ne fournit pas autant de fonctionnalités et de connecteurs qu'OpenLayers,
mais il ne semble pas que ce soit son objectif ! Il est léger et personnalisable à souhait !</p>
<p>Pour le prouver, nous allons créer ici un sélecteur de couches pour polymaps.</p>
<p>Javascript n'est pas mon langage de prédilection, encore moins pour faire
de la programmation orientée objets.</p>
<p>Voici ce que j'ai réussi à comprendre de l'héritage et la portée dans le modèle objet de polymaps :</p>
<div class="highlight"><pre><span></span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">po</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">po</span><span class="p">.</span><span class="nx">classname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">args</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{},</span><span class="w"> </span><span class="c1">// new class or inheritance</span>
<span class="w"> </span><span class="nx">member</span><span class="p">;</span><span class="w"> </span><span class="c1">// member variable</span>
<span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="nx">privatemethod</span><span class="w"> </span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// [...]</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="nx">self</span><span class="p">.</span><span class="nx">classmethod</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">args</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// [...]</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">self</span><span class="p">;</span><span class="w"> </span><span class="c1">// allows to chain method calls</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">self</span><span class="p">;</span>
<span class="w"> </span><span class="p">};</span>
<span class="p">})(</span><span class="nx">org</span><span class="p">.</span><span class="nx">polymaps</span><span class="p">);</span>
</pre></div>
<p>Pour faire ce sélecteur de couches, nous aurons besoin d'une classe disposant :</p>
<ul class="simple">
<li>de variables membres qui stockent la liste de couches disponibles (<tt class="docutils literal">layers</tt>) et la couche actuelle (<tt class="docutils literal">current</tt>)</li>
<li>d'une méthode de classe qui bascule d'une couche à l'autre</li>
</ul>
<div class="highlight"><pre><span></span><span class="nx">self</span><span class="p">.</span><span class="nx">switchto</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">l</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">layers</span><span class="p">[</span><span class="nx">name</span><span class="p">];</span><span class="w"> </span><span class="c1">// find layer by name</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">l</span><span class="p">.</span><span class="nx">map</span><span class="p">())</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">l</span><span class="p">.</span><span class="nx">visible</span><span class="p">(</span><span class="kc">true</span><span class="p">);</span><span class="w"> </span><span class="c1">// if already loaded, make it visible</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">map</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">l</span><span class="p">);</span><span class="w"> </span><span class="c1">// else load it</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">current</span><span class="p">)</span><span class="w"> </span><span class="nx">current</span><span class="p">.</span><span class="nx">visible</span><span class="p">(</span><span class="kc">false</span><span class="p">);</span><span class="w"> </span><span class="c1">// hide current</span>
<span class="w"> </span><span class="nx">current</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">l</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<ul class="simple">
<li>d'une méthode qui crée l'interface avec les radio buttons et qui les relie à la méthode précedente</li>
</ul>
<div class="highlight"><pre><span></span><span class="nx">self</span><span class="p">.</span><span class="nx">container</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">(</span><span class="nx">elt</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">list</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="s1">'div'</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// [...]</span>
<span class="w"> </span><span class="c1">// For each layer, create a <input></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="nx">name</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="nx">layers</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">input</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="s1">'input'</span><span class="p">);</span>
<span class="w"> </span><span class="nx">input</span><span class="p">.</span><span class="nx">setAttribute</span><span class="p">(</span><span class="s1">'type'</span><span class="p">,</span><span class="w"> </span><span class="s1">'radio'</span><span class="p">);</span>
<span class="w"> </span><span class="nx">input</span><span class="p">.</span><span class="nx">setAttribute</span><span class="p">(</span><span class="s1">'value'</span><span class="p">,</span><span class="w"> </span><span class="nx">name</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// [...]</span>
<span class="w"> </span><span class="c1">// Link onChange event on radio</span>
<span class="w"> </span><span class="nx">input</span><span class="p">.</span><span class="nx">onchange</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kd">function</span><span class="w"> </span><span class="p">()</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">self</span><span class="p">.</span><span class="nx">switchto</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">getAttribute</span><span class="p">(</span><span class="s1">'value'</span><span class="p">));</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="c1">// [...]</span>
<span class="w"> </span><span class="nx">list</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">input</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="c1">// [...]</span>
<span class="w"> </span><span class="nx">elt</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">list</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">self</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>Maintenant il suffit de l'utiliser ! Voici un exemple simple avec deux couches :</p>
<div class="highlight"><pre><span></span><span class="c1">// Create a normal map</span>
<span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">po</span><span class="p">.</span><span class="nx">mcmap</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">container</span><span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="s2">"map"</span><span class="p">).</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">svg</span><span class="p">(</span><span class="s2">"svg"</span><span class="p">)))</span>
<span class="w"> </span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">interact</span><span class="p">());</span>
<span class="c1">// Define the layers</span>
<span class="kd">var</span><span class="w"> </span><span class="nx">layers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="s2">"layer1"</span><span class="w"> </span><span class="o">:</span>
<span class="w"> </span><span class="nx">po</span><span class="p">.</span><span class="nx">image</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="s2">"http://server1/{Z}/{X}/{Y}.png"</span><span class="p">))</span>
<span class="w"> </span><span class="p">.</span><span class="nx">id</span><span class="p">(</span><span class="s1">'l1'</span><span class="p">),</span>
<span class="w"> </span><span class="s2">"layer2"</span><span class="w"> </span><span class="o">:</span>
<span class="w"> </span><span class="nx">po</span><span class="p">.</span><span class="nx">image</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="s2">"http://server2/{Z}/{X}/{Y}.png"</span><span class="p">))</span>
<span class="w"> </span><span class="p">.</span><span class="nx">id</span><span class="p">(</span><span class="s1">'l2'</span><span class="p">),</span>
<span class="p">};</span>
<span class="c1">// Add the default one</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">layers</span><span class="p">[</span><span class="s2">"layer1"</span><span class="p">]);</span>
<span class="c1">// Create the switcher</span>
<span class="nx">po</span><span class="p">.</span><span class="nx">switcher</span><span class="p">(</span><span class="nx">map</span><span class="p">,</span><span class="w"> </span><span class="nx">layers</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="nx">title</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s1">'Fond de carte'</span><span class="p">})</span>
<span class="w"> </span><span class="p">.</span><span class="nx">container</span><span class="p">(</span><span class="nb">document</span><span class="p">.</span><span class="nx">getElementById</span><span class="p">(</span><span class="s2">"layerswitcher"</span><span class="p">));</span>
</pre></div>
<p>Et voilà ! Nous avons notre sélecteur de couches, avec un code html
tout simple (<tt class="docutils literal">div</tt>, <tt class="docutils literal">input</tt>, <tt class="docutils literal">label</tt>), facile à styler en CSS,
contrairement au <a class="reference external" href="http://pastebin.com/LQPBv6tZ">gros pavé généré par le *LayerSwitcher* d'OpenLayers</a>.</p>
<img alt="" src="/images/polymaps-switcher.png" />
<p>Pour accéder au code complet et l'améliorer : "<a class="reference external" href="https://github.com/makinacorpus/polymaps-extensions">Fork me on GitHub</a>" !</p>
<p><em>Article original publié chez</em> <a class="reference external" href="http://www.makina-corpus.org/blog/ajouter-un-s%C3%A9lecteur-de-couches-layer-switcher-%C3%A0-polymaps">Makina Corpus</a></p>
Afficher des pictogrammes avec polymaps2011-02-28T17:02:00+01:002011-02-28T17:02:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-02-28:/afficher-des-pictogrammes-avec-polymaps-fr.html<p>En reprenant l'exemple de l'<a class="reference external" href="http://www.makina-corpus.org/blog/afficher-les-donn%C3%A9es-de-paris-opendata-avec-polymaps">épisode précédent avec polymaps</a>, nous allons maintenant afficher des pictogrammes sur les points.</p>
<p>On conserve la couche GeoJSON en utilisant un <a class="reference external" href="http://fr.wikipedia.org/wiki/Fonction_de_rappel">callback</a> pour l'évènement <tt class="docutils literal">load</tt>.</p>
<div class="highlight"><pre><span></span><span class="nx">map</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">geoJson</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="s1">'collecteurs.json'</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"load"</span><span class="p">,</span><span class="w"> </span><span class="nx">load</span><span class="p">));</span>
</pre></div>
<p>Dans la fonction <tt class="docutils literal">load()</tt>, nous allons remplacer les cercles dessinés …</p><p>En reprenant l'exemple de l'<a class="reference external" href="http://www.makina-corpus.org/blog/afficher-les-donn%C3%A9es-de-paris-opendata-avec-polymaps">épisode précédent avec polymaps</a>, nous allons maintenant afficher des pictogrammes sur les points.</p>
<p>On conserve la couche GeoJSON en utilisant un <a class="reference external" href="http://fr.wikipedia.org/wiki/Fonction_de_rappel">callback</a> pour l'évènement <tt class="docutils literal">load</tt>.</p>
<div class="highlight"><pre><span></span><span class="nx">map</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">geoJson</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="s1">'collecteurs.json'</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"load"</span><span class="p">,</span><span class="w"> </span><span class="nx">load</span><span class="p">));</span>
</pre></div>
<p>Dans la fonction <tt class="docutils literal">load()</tt>, nous allons remplacer les cercles dessinés par défaut par des pictogrammes en manipulant les éléments de la page (<a class="reference external" href="http://fr.wikipedia.org/wiki/Document_Object_Model">DOM</a>).
Nous utilisons ici la variable <tt class="docutils literal">n$</tt>, qui provient du miniscript <tt class="docutils literal">nns.js</tt> livré dans l'archive <em>polymaps</em> et qui facilite la manipulation du DOM (le vénérable <a class="reference external" href="http://jquery.com/">jquery</a> ferait aussi l'affaire)</p>
<div class="highlight"><pre><span></span><span class="kd">function</span><span class="w"> </span><span class="nx">load</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">ICONSIZE</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">16</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// Parcourir les features de la carte</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="kd">var</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">0</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="nx">e</span><span class="p">.</span><span class="nx">features</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span><span class="w"> </span><span class="nx">i</span><span class="o">++</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">circle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">n$</span><span class="p">(</span><span class="nx">e</span><span class="p">.</span><span class="nx">features</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">element</span><span class="p">);</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">root</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">circle</span><span class="p">.</span><span class="nx">parent</span><span class="p">();</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="nx">attributes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">e</span><span class="p">.</span><span class="nx">features</span><span class="p">[</span><span class="nx">i</span><span class="p">].</span><span class="nx">data</span><span class="p">.</span><span class="nx">properties</span><span class="p">;</span>
<span class="w"> </span><span class="c1">// Ajouter et positionner le pictogramme</span>
<span class="w"> </span><span class="c1">// (à partir de la position du cercle)</span>
<span class="w"> </span><span class="nx">img</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">root</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="s2">"svg:image"</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'width'</span><span class="p">,</span><span class="w"> </span><span class="nx">ICONSIZE</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'height'</span><span class="p">,</span><span class="w"> </span><span class="nx">ICONSIZE</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s2">"transform"</span><span class="p">,</span><span class="w"> </span><span class="nx">circle</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'transform'</span><span class="p">)</span>
<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">' translate(-'</span><span class="o">+</span><span class="p">(</span><span class="nx">ICONSIZE</span><span class="o">/</span><span class="mf">2</span><span class="p">)</span><span class="o">+</span><span class="s1">','</span>
<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">'-'</span><span class="o">+</span><span class="p">(</span><span class="nx">ICONSIZE</span><span class="o">/</span><span class="mf">2</span><span class="p">)</span><span class="o">+</span><span class="s1">')'</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// Enlever le cercle original</span>
<span class="w"> </span><span class="nx">root</span><span class="p">.</span><span class="nx">remove</span><span class="p">(</span><span class="nx">circle</span><span class="p">);</span>
<span class="w"> </span><span class="c1">// Définir le chemin du pictogramme à utiliser</span>
<span class="w"> </span><span class="c1">// en fonction de l'attribut</span>
<span class="w"> </span><span class="nx">img</span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'xlink:href'</span><span class="p">,</span><span class="w"> </span><span class="nx">attributes</span><span class="p">.</span><span class="nx">Etat</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s1">'Actif'</span><span class="w"> </span><span class="o">?</span>
<span class="w"> </span><span class="s1">'actif.svg'</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s1">'inactif.svg'</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<img alt="" src="/images/polymaps-pictogrammes.jpg" />
<p><em>© City of Paris, ODBL, CloudMade, OpenStreetMap contributors, CCBYSA</em></p>
<p>Ici, nous avons utilisé des pictogrammes SVG, mais le même code fonctionne avec des /images PNG ou JPG...</p>
Afficher les données de Paris OpenData avec polymaps2011-02-24T13:02:00+01:002011-02-24T13:02:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-02-24:/afficher-les-donnees-de-paris-opendata-avec-polymaps-fr.html<p>En ouvrant l'accès à un catalogue de données diverses (Équipements, bâti, arbres d'alignement, arrêtés municipaux, ...)
l'initiative <a class="reference external" href="http://opendata.paris.fr">ParisData</a>, l'Open Data de la capitale, nous donne l'occasion de manipuler des données georéférencées.
Notre objectif ici sera de les publier sur une page Web grâce à un outil simple et léger : <a class="reference external" href="http://polymaps.org">polymaps</a>.</p>
<div class="section" id="transformation">
<h2>Transformation …</h2></div><p>En ouvrant l'accès à un catalogue de données diverses (Équipements, bâti, arbres d'alignement, arrêtés municipaux, ...)
l'initiative <a class="reference external" href="http://opendata.paris.fr">ParisData</a>, l'Open Data de la capitale, nous donne l'occasion de manipuler des données georéférencées.
Notre objectif ici sera de les publier sur une page Web grâce à un outil simple et léger : <a class="reference external" href="http://polymaps.org">polymaps</a>.</p>
<div class="section" id="transformation">
<h2>Transformation</h2>
<p>Le système de projection utilisé pour certaines données du catalogue est la <a class="reference external" href="http://en.wikipedia.org/wiki/Lambert_conformal_conic_projection">Lambert Conformal Conic</a> (NTF, EPSG 9802)</p>
<p>Dans la mesure où nous voulons déployer quelquechose de très simple, nous n'avons pas l'intention de sortir la grosse artillerie habituelle (Serveur WMS, Mapserver, QGIS MapServer, ...), nous allons plutôt utiliser un fichier GeoJSON, en longitudes/latitudes WGS84 (EPSG:4326).</p>
<p>Pour cela, la bibliothèque <a class="reference external" href="http://www.gdal.org">GDAL</a> nous offre tous les outils adéquates:</p>
<ul class="simple">
<li>Ouvrir le fichier shape (ESRI Shapefile) fourni par ParisData</li>
<li>Reprojetter en EPSG:4326</li>
<li>Choisir les données (attributaires) que nous allons conserver</li>
<li>Exporter en GeoJSON</li>
</ul>
<p>Pour notre exemple, nous avons choisi les emplacements des <a class="reference external" href="http://opendata.paris.fr/opendata/jsp/site/Portal.jsp?document_id=57&portlet_id=106">points de collecte de verre</a>.
Parmis les champs fournis, nous choisissons de ne conserver que leur état (<tt class="docutils literal">Lb_Etat_E</tt>) et le nom de leur emplacement (<tt class="docutils literal">Emplacemnt</tt>).</p>
<p>Comme python est notre language préféré, et que c'est toujours un plaisir de le montrer en action, voici la petite procédure qui fait tout ça :</p>
<div class="highlight"><pre><span></span><span class="c1"># python gdal</span>
<span class="kn">from</span> <span class="nn">osgeo</span> <span class="kn">import</span> <span class="n">ogr</span>
<span class="kn">from</span> <span class="nn">osgeo</span> <span class="kn">import</span> <span class="n">osr</span>
<span class="o">...</span>
<span class="o">...</span>
<span class="c1"># Ouvrir le répertoire contenant les shape</span>
<span class="n">source</span> <span class="o">=</span> <span class="n">ogr</span><span class="o">.</span><span class="n">Open</span><span class="p">(</span><span class="n">INPUT_FOLDER</span><span class="p">)</span>
<span class="c1"># La projection de sortie</span>
<span class="n">spatialRef</span> <span class="o">=</span> <span class="n">osr</span><span class="o">.</span><span class="n">SpatialReference</span><span class="p">()</span>
<span class="n">spatialRef</span><span class="o">.</span><span class="n">ImportFromEPSG</span><span class="p">(</span><span class="mi">4326</span><span class="p">)</span>
<span class="c1"># Le fichier de sortie</span>
<span class="n">driver</span> <span class="o">=</span> <span class="n">ogr</span><span class="o">.</span><span class="n">GetDriverByName</span><span class="p">(</span><span class="s1">'GeoJSON'</span><span class="p">)</span>
<span class="n">shape</span> <span class="o">=</span> <span class="n">driver</span><span class="o">.</span><span class="n">CreateDataSource</span><span class="p">(</span><span class="n">OUTPUT_FILE</span><span class="p">)</span>
<span class="c1"># Parcourir les couches</span>
<span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="n">source</span><span class="p">:</span>
<span class="c1"># Reprojection originale -> destination</span>
<span class="n">originalSpatialRef</span> <span class="o">=</span> <span class="n">layer</span><span class="o">.</span><span class="n">GetSpatialRef</span><span class="p">()</span>
<span class="n">coordTransform</span> <span class="o">=</span> <span class="n">osr</span><span class="o">.</span><span class="n">CoordinateTransformation</span><span class="p">(</span><span class="n">originalSpatialRef</span><span class="p">,</span>
<span class="n">spatialRef</span><span class="p">)</span>
<span class="c1"># Choix des champs des données</span>
<span class="n">properties</span> <span class="o">=</span> <span class="n">ogr</span><span class="o">.</span><span class="n">FeatureDefn</span><span class="p">()</span>
<span class="n">properties</span><span class="o">.</span><span class="n">AddFieldDefn</span><span class="p">(</span><span class="n">ogr</span><span class="o">.</span><span class="n">FieldDefn</span><span class="p">(</span><span class="s1">'Etat'</span><span class="p">))</span>
<span class="n">properties</span><span class="o">.</span><span class="n">AddFieldDefn</span><span class="p">(</span><span class="n">ogr</span><span class="o">.</span><span class="n">FieldDefn</span><span class="p">(</span><span class="s1">'Emplacement'</span><span class="p">))</span>
<span class="c1"># Créer la nouvelle couche GeoJSON</span>
<span class="n">newLayer</span> <span class="o">=</span> <span class="n">shape</span><span class="o">.</span><span class="n">CreateLayer</span><span class="p">(</span><span class="n">layer</span><span class="o">.</span><span class="n">GetName</span><span class="p">(),</span> <span class="n">spatialRef</span><span class="p">)</span>
<span class="c1"># Parcourir les features</span>
<span class="k">for</span> <span class="n">feature</span> <span class="ow">in</span> <span class="n">layer</span><span class="p">:</span>
<span class="c1"># Créer la nouvelle feature</span>
<span class="n">newFeature</span> <span class="o">=</span> <span class="n">ogr</span><span class="o">.</span><span class="n">Feature</span><span class="p">(</span><span class="n">properties</span><span class="p">)</span>
<span class="c1"># Remplir les champs choisis</span>
<span class="n">newFeature</span><span class="o">.</span><span class="n">SetField</span><span class="p">(</span><span class="s1">'Etat'</span><span class="p">,</span> <span class="n">feature</span><span class="o">.</span><span class="n">GetField</span><span class="p">(</span><span class="s1">'Lb_Etat_E'</span><span class="p">))</span>
<span class="n">newFeature</span><span class="o">.</span><span class="n">SetField</span><span class="p">(</span><span class="s1">'Emplacement'</span><span class="p">,</span> <span class="n">feature</span><span class="o">.</span><span class="n">GetField</span><span class="p">(</span><span class="s1">'Emplacemnt'</span><span class="p">))</span>
<span class="c1"># Reprojetter la feature</span>
<span class="n">geometry</span> <span class="o">=</span> <span class="n">feature</span><span class="o">.</span><span class="n">GetGeometryRef</span><span class="p">()</span>
<span class="n">geometry</span><span class="o">.</span><span class="n">Transform</span><span class="p">(</span><span class="n">coordTransform</span><span class="p">)</span>
<span class="c1"># Sauvegarder</span>
<span class="n">newFeature</span><span class="o">.</span><span class="n">SetGeometry</span><span class="p">(</span><span class="n">geometry</span><span class="p">)</span>
<span class="n">newLayer</span><span class="o">.</span><span class="n">CreateFeature</span><span class="p">(</span><span class="n">newFeature</span><span class="p">)</span>
<span class="n">newFeature</span><span class="o">.</span><span class="n">Destroy</span><span class="p">()</span>
</pre></div>
<p>Nous obtenons en sortie un fichier GeoJSON avec les points en lat / long et les données 'Etat' et 'Emplacement'.</p>
<div class="highlight"><pre><span></span><span class="p">...</span>
<span class="p">{</span><span class="w"> </span><span class="s2">"type"</span><span class="o">:</span><span class="w"> </span><span class="s2">"Feature"</span><span class="p">,</span><span class="w"> </span><span class="s2">"properties"</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s2">"Etat"</span><span class="o">:</span><span class="w"> </span><span class="s2">"Actif"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Emplacement"</span><span class="o">:</span><span class="w"> </span><span class="s2">"37 CHATEAU D'EAU ANGLE BOUCHARDON"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="s2">"geometry"</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s2">"type"</span><span class="o">:</span><span class="w"> </span><span class="s2">"Point"</span><span class="p">,</span><span class="w"> </span><span class="s2">"coordinates"</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mf">2.358920</span><span class="p">,</span><span class="w"> </span><span class="mf">48.871154</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span>
<span class="p">{</span><span class="w"> </span><span class="s2">"type"</span><span class="o">:</span><span class="w"> </span><span class="s2">"Feature"</span><span class="p">,</span><span class="w"> </span><span class="s2">"properties"</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s2">"Etat"</span><span class="o">:</span><span class="w"> </span><span class="s2">"Actif"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Emplacement"</span><span class="o">:</span><span class="w"> </span><span class="s2">"13 place de la Nation"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="s2">"geometry"</span><span class="o">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s2">"type"</span><span class="o">:</span><span class="w"> </span><span class="s2">"Point"</span><span class="p">,</span><span class="w"> </span><span class="s2">"coordinates"</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="mf">2.398154</span><span class="p">,</span><span class="w"> </span><span class="mf">48.848723</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span>
<span class="p">...</span>
</pre></div>
<p>Le fichier pèse 174Ko, mais lorsqu'Apache le servira il pèsera 20Ko (grâce à la compression gzip !)</p>
</div>
<div class="section" id="affichage">
<h2>Affichage</h2>
<p>Nous choisissons d'afficher ces données dans une page avec <a class="reference external" href="http://polymaps.org">polymaps</a>. Il s'agit
d'un composant Javascript permettant de créer des cartes interactives.</p>
<p>Les critères de comparaison avec OpenLayers (OL) sont:</p>
<ul class="simple">
<li>la légèreté (~30Ko, 10Ko en gzip!)</li>
<li>la rapiditité d'exécution</li>
<li>l'utilisation de GeoJSON et SVG (flexibilité et styles)</li>
</ul>
<p>Cependant, la couverture fonctionnelle n'est absolument pas comparable. Mais pour afficher une carte avec des points, c'est largement suffisant !</p>
<p>On commence par un fond de carte: <a class="reference external" href="http://cloudmade.com/">Cloudmade</a>, dont les tuiles sont dessinées à partir d'<a class="reference external" href="http://www.openstreetmap.org/">OpenStreetMap</a>:</p>
<div class="highlight"><pre><span></span><span class="nx">map</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">image</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="s2">"http://{S}tile.cloudmade.com"</span>
<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s2">"/1a1b06b230af4efdbb989ea99e9841af"</span>
<span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s2">"/998/256/{Z}/{X}/{Y}.png"</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">hosts</span><span class="p">([</span><span class="s2">"a."</span><span class="p">,</span><span class="w"> </span><span class="s2">"b."</span><span class="p">,</span><span class="w"> </span><span class="s2">"c."</span><span class="p">,</span><span class="w"> </span><span class="s2">""</span><span class="p">])));</span>
</pre></div>
<p>On ajoute ensuite nos données GeoJSON:</p>
<div class="highlight"><pre><span></span><span class="nx">map</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">geoJson</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="s1">'collecteurs.json'</span><span class="p">)</span>
</pre></div>
<p>Polymaps facilite la personnalisation du dessin en fonction des données. Ici, nous affichons en vert les collecteurs à l'état "Actif" et en rouge les autres.
De même nous mettons leur "Emplacement" en tooltip (<tt class="docutils literal">svg:title</tt>, Firefox 4, Chrome, Opera 11).</p>
<div class="highlight"><pre><span></span><span class="nx">map</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="nx">po</span><span class="p">.</span><span class="nx">geoJson</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">url</span><span class="p">(</span><span class="s1">'collecteurs.json'</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"load"</span><span class="p">,</span><span class="w"> </span><span class="nx">po</span><span class="p">.</span><span class="nx">stylist</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'fill'</span><span class="p">,</span>
<span class="w"> </span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">properties</span><span class="p">.</span><span class="nx">Etat</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s1">'Actif'</span><span class="w"> </span><span class="o">?</span>
<span class="w"> </span><span class="s1">'green'</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s1">'red'</span><span class="p">;</span>
<span class="w"> </span><span class="p">}))</span>
<span class="w"> </span><span class="p">.</span><span class="nx">title</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">d</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">d</span><span class="p">.</span><span class="nx">properties</span><span class="p">.</span><span class="nx">Emplacement</span>
<span class="w"> </span><span class="p">}));</span>
</pre></div>
<img alt="" src="/images/parisdata-polymaps.jpg" />
<p><a class="reference external" href="http://www.makina-corpus.org/demos/mle/parisdata-polymaps/">Accéder à la page de démonstration</a></p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Finalement, l'étape la plus compliquée était de reprojetter les données.
On regrettera donc que l'initiative ParisData les ait publié sous cette forme exotique.</p>
<p>Comme le soulignait <a class="reference external" href="http://www.biologeek.com/2010/12/ce-nest-pas-la-taille-qui-compte/">David</a> : <em>Publieurs de données, concentrez vous sur la qualité, pas la taille, les développeurs vous remercieront !</em></p>
<p>À noter également que nous avons choisi une approche privilégiant la légèreté. Or, plusieurs sources de données de ParisData sont volumineuses et ne pourraient pas être affichées en GeoJSON sans mettre à genoux le navigateur. Nous serions alors contraints de servir les données sous forme de tuiles...</p>
</div>
SQLAlchemy, a brave new World2011-02-22T15:02:00+01:002011-02-22T15:02:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-02-22:/sqlalchemy-a-brave-new-world.html<p><a class="reference external" href="http://www.sqlalchemy.org/">SQLAlchemy</a> becomes an essential technology for any python developper interacting with relational databases.
As a Django developper, I have sat on my laurels for long, being completely satisfied with the <a class="reference external" href="http://docs.djangoproject.com/en/dev/topics/db/queries/">Django ORM</a>. It was time to explore its challenger.</p>
<p>First, before it sounds like I crush my favorite framework : when …</p><p><a class="reference external" href="http://www.sqlalchemy.org/">SQLAlchemy</a> becomes an essential technology for any python developper interacting with relational databases.
As a Django developper, I have sat on my laurels for long, being completely satisfied with the <a class="reference external" href="http://docs.djangoproject.com/en/dev/topics/db/queries/">Django ORM</a>. It was time to explore its challenger.</p>
<p>First, before it sounds like I crush my favorite framework : when Django ORM was developped, there was no SQLAlchemy, or almost no good python ORM at all.</p>
<p>Here are some of the things you might want to know first:</p>
<ul class="simple">
<li>Django ORM documentation is clean and well organized</li>
<li>Dango ORM was not really meant to be used outside django apps</li>
<li>Django ORM has limitations when it comes to <a class="reference external" href="http://sralab.com/2009/01/14/limitations-of-the-django-orm-10-in-model-inheritance/">Model</a> <a class="reference external" href="http://linfiniti.com/2010/03/django-foreign-key-inheritance-solved/">Inheritance</a></li>
<li>Django ORM does not manage models migrations without extra stuff like <a class="reference external" href="http://south.aeracode.org/">South</a></li>
<li>Django ORM support for multiple databases was introduced <a class="reference external" href="http://docs.djangoproject.com/en/dev/topics/db/multi-db/">in version 1.2</a></li>
<li>Django ORM does not always manage connection pooling (e.g. <a class="reference external" href="http://code.djangoproject.com/ticket/7732">with Oracle</a>)</li>
<li>SQLAlchemy is light and framework independant</li>
<li>SQLAlchemy is stripped to a minimum set of (clean and well-implemented) features</li>
<li>SQLAlchemy is very flexible and supports mapping of objects using declarations or metadata</li>
<li>SQLAlchemy documentation is bloated (API reference is mixed-in with long explanation and use cases)</li>
<li>SQLAlchemy requires a better knowledge of advanced python mechanisms and architecture</li>
</ul>
<p>In order to explore SQLAlchemy (SA) I created <a class="reference external" href="http://pypi.python.org/pypi/pyfspot">pyfspot</a> (<a class="reference external" href="https://github.com/leplatrem/pyfspot">sources are on github</a>): a very small application to manage the database of the F-Spot photo manager.
It is not supposed to save lives, but that will at least be:</p>
<ul class="simple">
<li>a pretext for me to dive into the API</li>
<li>a small and useful tool</li>
<li>an example of SA in action for any developper interested</li>
<li>a base for a full <a class="reference external" href="http://f-spot.org/">F-Spot</a> management application (<tt class="docutils literal"></dreamer></tt>)</li>
</ul>
<p>I discovered a few concrete things:</p>
<ul class="simple">
<li>Django inspectdb equivalent ?<ul>
<li><a class="reference external" href="http://code.google.com/p/sqlautocode/">sqlautocode</a> (unfortunately <a class="reference external" href="http://code.google.com/p/sqlautocode/issues/detail?id=32">I could not use it</a>)</li>
</ul>
</li>
<li>Django fixtures equivalent ?<ul>
<li><a class="reference external" href="http://code.google.com/p/fixture/">fixture</a> (<a class="reference external" href="http://farmdev.com/projects/fixture/using-loadable-fixture.html#an-example-of-loading-data-using-sqlalchemy">demo</a>)</li>
</ul>
</li>
<li>Django <cite>model.DoesNotExist</cite> ?</li>
</ul>
<div class="highlight"><pre><span></span><span class="c1"># get() does not throw exception</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">Tag</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="c1"># filter().one() does ...</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">Photo</span><span class="p">)</span><span class="o">.</span><span class="n">filter_by</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="mi">1337</span><span class="p">)</span><span class="o">.</span><span class="n">one</span><span class="p">()</span>
<span class="k">except</span> <span class="n">sqlalchemy</span><span class="o">.</span><span class="n">orm</span><span class="o">.</span><span class="n">exc</span><span class="o">.</span><span class="n">NoResultFound</span><span class="p">:</span>
<span class="c1"># d'oh!</span>
<span class="k">pass</span>
</pre></div>
<ul class="simple">
<li>Invert condition like Django exclude() ?</li>
</ul>
<div class="highlight"><pre><span></span><span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">Photo</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="o">~</span><span class="n">Photo</span><span class="o">.</span><span class="n">base_uri</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s1">'/'</span><span class="p">))</span>
</pre></div>
<ul class="simple">
<li>Escape LIKE condition ?</li>
</ul>
<div class="highlight"><pre><span></span><span class="n">Photo</span><span class="o">.</span><span class="n">base_uri</span><span class="o">.</span><span class="n">like</span><span class="p">(</span><span class="s1">'%</span><span class="se">\\</span><span class="si">%%</span><span class="s1">'</span><span class="p">,</span> <span class="n">escape</span><span class="o">=</span><span class="s2">"</span><span class="se">\\</span><span class="s2">"</span><span class="p">)</span>
</pre></div>
<ul class="simple">
<li>Intersections of many-to-many ?</li>
</ul>
<div class="highlight"><pre><span></span><span class="c1"># A tag</span>
<span class="n">tag</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">Tag</span><span class="p">)</span><span class="o">.</span><span class="n">filter_by</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'foo'</span><span class="p">)</span><span class="o">.</span><span class="n">one</span><span class="p">()</span>
<span class="c1"># A set of photos</span>
<span class="n">photoset</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">Photo</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="o">~</span><span class="n">Photo</span><span class="o">.</span><span class="n">base_uri</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s1">'/'</span><span class="p">))</span>
<span class="c1"># intersect() won't work</span>
<span class="n">photoset</span><span class="o">.</span><span class="n">intersect</span><span class="p">(</span><span class="n">tag</span><span class="o">.</span><span class="n">photos</span><span class="p">)</span>
<span class="ne">AttributeError</span><span class="p">:</span> <span class="s1">'InstrumentedList'</span> <span class="nb">object</span> <span class="n">has</span> <span class="n">no</span> <span class="n">attribute</span> <span class="s1">'c'</span>
<span class="c1"># Use any()</span>
<span class="n">photoset</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">Photo</span><span class="o">.</span><span class="n">tags</span><span class="o">.</span><span class="n">any</span><span class="p">(</span><span class="nb">id</span><span class="o">=</span><span class="n">tag</span><span class="o">.</span><span class="n">id</span><span class="p">))</span>
</pre></div>
<p>Well, those were my first steps. As expected, it did not feel so well to relearn how to walk. But at least I am now ready to get my bearings in SQLAlchemy's world.</p>
<p><em>Original post at</em> <a class="reference external" href="http://www.makina-corpus.org/blog/sqlalchemy-brave-new-world">Makina Corpus</a></p>
Javascript Beautifier in command-line (and in Geany editor)2011-02-01T10:20:00+01:002011-02-01T10:20:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-02-01:/javascript-beautifier-in-command-line-and-in-geany-editor.html<p>Install a Javascript engine (like Mozilla Rhino)</p>
<pre class="literal-block">
sudo aptitude install rhino
</pre>
<p>Get the beautifier script (put it somewhere like <cite>~/.bin</cite>)</p>
<pre class="literal-block">
wget http://jsbeautifier.org/beautify.js
</pre>
<p>Add the following at the end of <cite>beautify.js</cite></p>
<pre class="literal-block">
print( js_beautify( readFile( arguments[0] )));
</pre>
<p>Create a shell script that will call it (like <cite>~/.bin …</cite></p><p>Install a Javascript engine (like Mozilla Rhino)</p>
<pre class="literal-block">
sudo aptitude install rhino
</pre>
<p>Get the beautifier script (put it somewhere like <cite>~/.bin</cite>)</p>
<pre class="literal-block">
wget http://jsbeautifier.org/beautify.js
</pre>
<p>Add the following at the end of <cite>beautify.js</cite></p>
<pre class="literal-block">
print( js_beautify( readFile( arguments[0] )));
</pre>
<p>Create a shell script that will call it (like <cite>~/.bin/beautifyjs</cite>)</p>
<pre class="literal-block">
#!/bin/sh
java -cp /usr/share/java/js.jar org.mozilla.javascript.tools.shell.Main ~/.bin/beautify.js $*
</pre>
<p>Make sure to set it executable</p>
<pre class="literal-block">
chmod +x ~/.bin/beautifyjs
</pre>
<div class="section" id="use-it-from-command-line">
<h2>Use it from command-line</h2>
<p>At least to check that it works !</p>
<pre class="literal-block">
~/.bin/beautifyjs /your/file.js
</pre>
</div>
<div class="section" id="or-in-geany-editor">
<h2>Or in Geany Editor</h2>
<blockquote>
<ul>
<li><p class="first">Open a Javascript file</p>
</li>
<li><p class="first">Open menu <em>Build</em> > <em>Define Build Commands</em></p>
</li>
<li><p class="first">Create a new entry (like <cite>beautify</cite>)</p>
</li>
<li><p class="first">In command, enter the following</p>
<pre class="literal-block">
~/.bin/beautifyjs %f > /tmp/tmpfile.js && geany /tmp/tmpfile.js
</pre>
</li>
<li><p class="first">In working directory, enter <cite>%d</cite></p>
</li>
</ul>
</blockquote>
</div>
Python UTF-8 print fails when redirecting stdout2011-01-26T11:25:00+01:002011-01-26T11:25:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2011-01-26:/python-utf-8-print-fails-when-redirecting-stdout.html<p>Consider the following piece of code:</p>
<div class="highlight"><pre><span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="nb">print</span> <span class="sa">u</span><span class="s2">"Վարդանաշեն"</span>
</pre></div>
<p>Running this in a terminal works:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>test.py
Վարդանաշեն
</pre></div>
<p>Redirecting standard output to a file <strong>fails</strong>:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>test.py<span class="w"> </span>><span class="w"> </span>file
Traceback<span class="w"> </span><span class="o">(</span>most<span class="w"> </span>recent<span class="w"> </span>call<span class="w"> </span>last<span class="o">)</span>:
<span class="w"> </span>File<span class="w"> </span><span class="s2">"test.py"</span>,<span class="w"> </span>line<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="k">in</span><span class="w"> </span><module>
<span class="w"> </span>print<span class="w"> </span>u<span class="s2">"Վարդանաշեն"</span>
UnicodeEncodeError:<span class="w"> </span><span class="s1">'ascii'</span><span class="w"> </span>codec …</pre></div><p>Consider the following piece of code:</p>
<div class="highlight"><pre><span></span><span class="c1"># -*- coding: utf-8 -*-</span>
<span class="nb">print</span> <span class="sa">u</span><span class="s2">"Վարդանաշեն"</span>
</pre></div>
<p>Running this in a terminal works:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>test.py
Վարդանաշեն
</pre></div>
<p>Redirecting standard output to a file <strong>fails</strong>:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span>python<span class="w"> </span>test.py<span class="w"> </span>><span class="w"> </span>file
Traceback<span class="w"> </span><span class="o">(</span>most<span class="w"> </span>recent<span class="w"> </span>call<span class="w"> </span>last<span class="o">)</span>:
<span class="w"> </span>File<span class="w"> </span><span class="s2">"test.py"</span>,<span class="w"> </span>line<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="k">in</span><span class="w"> </span><module>
<span class="w"> </span>print<span class="w"> </span>u<span class="s2">"Վարդանաշեն"</span>
UnicodeEncodeError:<span class="w"> </span><span class="s1">'ascii'</span><span class="w"> </span>codec<span class="w"> </span>can<span class="err">'</span>t<span class="w"> </span>encode<span class="w"> </span>characters<span class="w"> </span><span class="k">in</span><span class="w"> </span>position<span class="w"> </span><span class="m">0</span>-9:<span class="w"> </span>ordinal<span class="w"> </span>not<span class="w"> </span><span class="k">in</span><span class="w"> </span>range<span class="o">(</span><span class="m">128</span><span class="o">)</span>
</pre></div>
<p><a class="reference external" href="http://wiki.python.org/moin/PrintFails">Explanations are available on Python official wiki</a>: default encoding has to be forced.</p>
<p>With an environment variable:</p>
<div class="highlight"><pre><span></span>$<span class="w"> </span><span class="nv">PYTHONIOENCODING</span><span class="o">=</span><span class="s1">'utf_8'</span>
$<span class="w"> </span><span class="nb">export</span><span class="w"> </span>PYTHONIOENCODING
$<span class="w"> </span>python<span class="w"> </span>test.py<span class="w"> </span>><span class="w"> </span>file
$
</pre></div>
<p>With source modification:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">codecs</span>
<span class="kn">import</span> <span class="nn">locale</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span> <span class="o">=</span> <span class="n">codecs</span><span class="o">.</span><span class="n">getwriter</span><span class="p">(</span><span class="n">locale</span><span class="o">.</span><span class="n">getpreferredencoding</span><span class="p">())(</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
</pre></div>
Grep with context lines above and below2010-09-30T10:25:00+02:002010-09-30T10:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2010-09-30:/grep-with-context-lines-above-and-below.html<p>I was about to write a script to implement exactly what already exists natively in GNU <cite>grep</cite>, a blasphemy I got saved from by my workmates.</p>
<div class="highlight"><pre><span></span>grep<span class="w"> </span>--line-number<span class="w"> </span>--colour<span class="o">=</span>AUTO<span class="w"> </span>--before-context<span class="w"> </span><span class="m">5</span><span class="w"> </span>--after-context<span class="w"> </span><span class="m">5</span><span class="w"> </span>PATTERN<span class="w"> </span>FILENAME
</pre></div>
<img alt="" src="/images/grep-lines.png" />
Python lazy hasattr()2010-09-30T10:20:00+02:002010-09-30T10:20:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2010-09-30:/python-lazy-hasattr.html<p>Python <cite>hasattr()</cite> evaluates the specified attribute, which may not be desired !</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Attr</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obs</span><span class="p">,</span> <span class="bp">cls</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="nb">print</span> <span class="s2">"evaluated"</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="k">class</span> <span class="nc">ClassA</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">Attr</span><span class="p">()</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">b</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span> <span class="s2">"evaluated"</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="o">>>></span> <span class="n">c</span> <span class="o">=</span> <span class="n">ClassA</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">c</span><span class="o">.</span><span class="n">a</span>
<span class="n">evaluated</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">c</span><span class="o">.</span><span class="n">b</span>
<span class="n">evaluated</span>
<span class="mi">0</span>
</pre></div>
<p>Now note that …</p><p>Python <cite>hasattr()</cite> evaluates the specified attribute, which may not be desired !</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Attr</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obs</span><span class="p">,</span> <span class="bp">cls</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="nb">print</span> <span class="s2">"evaluated"</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="k">class</span> <span class="nc">ClassA</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">Attr</span><span class="p">()</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">b</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span> <span class="s2">"evaluated"</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="o">>>></span> <span class="n">c</span> <span class="o">=</span> <span class="n">ClassA</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">c</span><span class="o">.</span><span class="n">a</span>
<span class="n">evaluated</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">c</span><span class="o">.</span><span class="n">b</span>
<span class="n">evaluated</span>
<span class="mi">0</span>
</pre></div>
<p>Now note that <cite>hasattr()</cite> evaluates the lazy attribute !</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">)</span>
<span class="n">evaluated</span>
<span class="kc">True</span>
<span class="o">>>></span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="s1">'b'</span><span class="p">)</span>
<span class="n">evaluated</span>
<span class="kc">True</span>
</pre></div>
<p>Let us fix that !</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">lazyhasattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">any</span><span class="p">(</span><span class="n">name</span> <span class="ow">in</span> <span class="n">d</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="p">(</span><span class="n">obj</span><span class="o">.</span><span class="vm">__dict__</span><span class="p">,</span>
<span class="n">obj</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__dict__</span><span class="p">))</span>
<span class="o">>>></span> <span class="n">c</span> <span class="o">=</span> <span class="n">ClassA</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">lazyhasattr</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">)</span>
<span class="kc">True</span>
<span class="o">>>></span> <span class="n">lazyhasattr</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="s1">'b'</span><span class="p">)</span>
<span class="kc">True</span>
</pre></div>
Python check arguments types2010-06-10T11:25:00+02:002010-06-10T11:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2010-06-10:/python-check-arguments-types.html<p>Decorators help us wrap some routines at function invocation. Here I show a small example that raises <cite>TypeError</cite> exceptions when given args have unexpected type. Note that <strong>it is not pythonic to type check</strong>.</p>
<p>This recipe is quite old, as its first pieces appear in <a class="reference external" href="http://www.python.org/dev/peps/pep-0318/">PEP-0318</a> in 2003. <a class="reference external" href="http://pypi.python.org/pypi/typecheck">A module …</a></p><p>Decorators help us wrap some routines at function invocation. Here I show a small example that raises <cite>TypeError</cite> exceptions when given args have unexpected type. Note that <strong>it is not pythonic to type check</strong>.</p>
<p>This recipe is quite old, as its first pieces appear in <a class="reference external" href="http://www.python.org/dev/peps/pep-0318/">PEP-0318</a> in 2003. <a class="reference external" href="http://pypi.python.org/pypi/typecheck">A module exists too</a> but it looks neglected...</p>
<p>The (heretic) decorator itself !</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">accepts</span><span class="p">(</span><span class="o">*</span><span class="n">argstypes</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargstypes</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrapped</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="o">></span> <span class="nb">len</span><span class="p">(</span><span class="n">argstypes</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">"</span><span class="si">%s</span><span class="s2">() takes at most </span><span class="si">%s</span><span class="s2"> non-keyword arguments (</span><span class="si">%s</span><span class="s2"> given)"</span> <span class="o">%</span> <span class="p">(</span><span class="n">func</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">argstypes</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)))</span>
<span class="n">argspairs</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">argstypes</span><span class="p">)</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span><span class="n">v</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">kwargstypes</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">"Unexpected keyword argument '</span><span class="si">%s</span><span class="s2">' for </span><span class="si">%s</span><span class="s2">()"</span> <span class="o">%</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">func</span><span class="o">.</span><span class="vm">__name__</span><span class="p">))</span>
<span class="n">argspairs</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">v</span><span class="p">,</span> <span class="n">kwargstypes</span><span class="p">[</span><span class="n">k</span><span class="p">]))</span>
<span class="k">for</span> <span class="n">param</span><span class="p">,</span> <span class="n">expected</span> <span class="ow">in</span> <span class="n">argspairs</span><span class="p">:</span>
<span class="k">if</span> <span class="n">param</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">param</span><span class="p">,</span> <span class="n">expected</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">"Parameter '</span><span class="si">%s</span><span class="s2">' is not </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">param</span><span class="p">,</span> <span class="n">expected</span><span class="o">.</span><span class="vm">__name__</span><span class="p">))</span>
<span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">wrapped</span>
<span class="k">return</span> <span class="n">wrapper</span>
</pre></div>
<p>Let us decorate !</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nd">@accepts</span><span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
<span class="o">...</span> <span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="o">...</span> <span class="k">pass</span>
<span class="o">...</span>
</pre></div>
<p>See it in action...</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="n">f</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">f</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">f</span><span class="p">()</span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">f</span><span class="p">()</span> <span class="n">takes</span> <span class="n">at</span> <span class="n">least</span> <span class="mi">1</span> <span class="n">argument</span> <span class="p">(</span><span class="mi">0</span> <span class="n">given</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">f</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">,</span> <span class="s1">'bar'</span><span class="p">)</span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">f</span><span class="p">()</span> <span class="n">takes</span> <span class="n">at</span> <span class="n">most</span> <span class="mi">1</span> <span class="n">non</span><span class="o">-</span><span class="n">keyword</span> <span class="n">arguments</span> <span class="p">(</span><span class="mi">2</span> <span class="n">given</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">f</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="s1">'bar'</span><span class="p">)</span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">Parameter</span> <span class="s1">'bar'</span> <span class="ow">is</span> <span class="ow">not</span> <span class="nb">int</span>
<span class="o">>>></span> <span class="n">f</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">,</span> <span class="n">arg3</span><span class="o">=</span><span class="s1">'bar'</span><span class="p">)</span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">Unexpected</span> <span class="n">keyword</span> <span class="n">argument</span> <span class="s1">'arg3'</span> <span class="k">for</span> <span class="n">f</span><span class="p">()</span>
</pre></div>
<p>Or with classes...</p>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="k">class</span> <span class="nc">A</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="o">...</span> <span class="k">pass</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="k">class</span> <span class="nc">B</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="o">...</span> <span class="nd">@accepts</span><span class="p">(</span><span class="nb">object</span><span class="p">,</span> <span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">unicode</span><span class="p">))</span>
<span class="o">...</span> <span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">s</span><span class="p">):</span>
<span class="o">...</span> <span class="k">pass</span>
<span class="o">...</span>
<span class="o">...</span> <span class="nd">@accepts</span><span class="p">(</span><span class="nb">object</span><span class="p">,</span> <span class="n">A</span><span class="p">)</span>
<span class="o">...</span> <span class="k">def</span> <span class="nf">g</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">a</span><span class="p">):</span>
<span class="o">...</span> <span class="k">pass</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">b</span> <span class="o">=</span> <span class="n">B</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">b</span><span class="o">.</span><span class="n">f</span><span class="p">(</span><span class="sa">u</span><span class="s1">'foo'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">b</span><span class="o">.</span><span class="n">f</span><span class="p">(</span><span class="s1">'foo'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">b</span><span class="o">.</span><span class="n">g</span><span class="p">(</span><span class="n">A</span><span class="p">())</span>
<span class="o">>>></span> <span class="n">b</span><span class="o">.</span><span class="n">g</span><span class="p">(</span><span class="n">B</span><span class="p">())</span>
<span class="ne">TypeError</span><span class="p">:</span> <span class="n">Parameter</span> <span class="s1">'<__main__.B object at 0x902466c>'</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">A</span>
</pre></div>
<p>The same can be applied to <cite>return</cite> :)</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">returns</span><span class="p">(</span><span class="n">rtype</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="n">f</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">wrapped</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">f</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">rtype</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">"return value </span><span class="si">%r</span><span class="s2"> does not match </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">result</span><span class="p">,</span><span class="n">rtype</span><span class="p">))</span>
<span class="k">return</span> <span class="n">result</span>
<span class="k">return</span> <span class="n">wrapped</span>
<span class="k">return</span> <span class="n">wrapper</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="nd">@accepts</span><span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="nb">int</span><span class="p">)</span>
<span class="o">...</span> <span class="nd">@returns</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span>
<span class="o">...</span> <span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="o">...</span> <span class="k">return</span> <span class="mi">0</span>
<span class="o">...</span>
</pre></div>
<p>..but kids, don't do this at home :-)</p>
Filesystem Watch with PyQt42009-08-14T13:37:00+02:002009-08-14T13:37:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2009-08-14:/filesystem-watch-with-pyqt4.html<p>I decided to write a generic program that can watch a folder or some files and run a command when changes occur.</p>
<p>Most of the work is done by PyQt4's <cite>QFileSystemWatcher</cite>.</p>
<div class="highlight"><pre><span></span><span class="n">qfsw</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">QFileSystemWatcher</span><span class="p">()</span>
<span class="n">qfsw</span><span class="o">.</span><span class="n">addPaths</span><span class="p">([</span><span class="n">path1</span><span class="p">,</span> <span class="n">file2</span> <span class="o">...</span> <span class="p">])</span>
<span class="n">QtCore</span><span class="o">.</span><span class="n">QObject</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">qfsw</span><span class="p">,</span><span class="n">QtCore</span><span class="o">.</span><span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"directoryChanged(QString)"</span><span class="p">),</span><span class="n">function</span><span class="p">)</span>
<span class="n">QtCore</span><span class="o">.</span><span class="n">QObject</span><span class="o">.</span><span class="n">connect …</span></pre></div><p>I decided to write a generic program that can watch a folder or some files and run a command when changes occur.</p>
<p>Most of the work is done by PyQt4's <cite>QFileSystemWatcher</cite>.</p>
<div class="highlight"><pre><span></span><span class="n">qfsw</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">QFileSystemWatcher</span><span class="p">()</span>
<span class="n">qfsw</span><span class="o">.</span><span class="n">addPaths</span><span class="p">([</span><span class="n">path1</span><span class="p">,</span> <span class="n">file2</span> <span class="o">...</span> <span class="p">])</span>
<span class="n">QtCore</span><span class="o">.</span><span class="n">QObject</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">qfsw</span><span class="p">,</span><span class="n">QtCore</span><span class="o">.</span><span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"directoryChanged(QString)"</span><span class="p">),</span><span class="n">function</span><span class="p">)</span>
<span class="n">QtCore</span><span class="o">.</span><span class="n">QObject</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">qfsw</span><span class="p">,</span><span class="n">QtCore</span><span class="o">.</span><span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"fileChanged(QString)"</span><span class="p">),</span><span class="n">function</span><span class="p">)</span>
</pre></div>
<p>Here is the script</p>
<div class="highlight"><pre><span></span><span class="c1"># !/usr/bin/env python</span>
<span class="c1">#</span>
<span class="c1"># Runs a command when a file system change occurs in specified list of paths.</span>
<span class="c1">#</span>
<span class="c1"># (c) Copyright 2008, Mathieu Leplatre,</span>
<span class="c1">#</span>
<span class="c1"># This software may be used and distributed according to the terms</span>
<span class="c1"># of the GNU Public License, incorporated herein by reference.</span>
<span class="kn">import</span> <span class="nn">sys</span><span class="o">,</span> <span class="nn">os</span>
<span class="kn">from</span> <span class="nn">PyQt4</span> <span class="kn">import</span> <span class="n">QtCore</span><span class="p">,</span> <span class="n">QtGui</span>
<span class="kn">import</span> <span class="nn">signal</span>
<span class="kn">from</span> <span class="nn">optparse</span> <span class="kn">import</span> <span class="n">OptionParser</span>
<span class="c1"># Parse command-line options</span>
<span class="n">usage</span> <span class="o">=</span> <span class="s2">"""Usage: %prog [options] COMMAND PATHS</span>
<span class="s2">Run COMMAND when a file system change occurs in specified list of PATHS.</span>
<span class="s2">COMMAND can contain '</span><span class="si">%s</span><span class="s2">' to refer changed file or directory."""</span>
<span class="n">parser</span> <span class="o">=</span> <span class="n">OptionParser</span><span class="p">(</span><span class="n">usage</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s2">"%prog 1.0"</span><span class="p">)</span>
<span class="p">(</span><span class="n">options</span><span class="p">,</span> <span class="n">args</span><span class="p">)</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="o"><</span> <span class="mi">2</span><span class="p">:</span>
<span class="n">parser</span><span class="o">.</span><span class="n">print_help</span><span class="p">()</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="c1"># Command</span>
<span class="n">command</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># List of paths</span>
<span class="n">paths</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
<span class="k">def</span> <span class="nf">onFileSystemChanged</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""</span>
<span class="sd"> Callback when file or folder change</span>
<span class="sd"> @param path : Changed path</span>
<span class="sd"> @type path : string</span>
<span class="sd"> """</span>
<span class="k">global</span> <span class="n">command</span>
<span class="c1"># Execute command replacing %s with path:</span>
<span class="k">if</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">'</span> <span class="ow">in</span> <span class="n">command</span><span class="p">:</span>
<span class="n">command</span> <span class="o">=</span> <span class="n">command</span> <span class="o">%</span> <span class="n">path</span>
<span class="c1"># Run as different process</span>
<span class="nb">print</span> <span class="s2">"Run '</span><span class="si">%s</span><span class="s2">'..."</span> <span class="o">%</span> <span class="n">command</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">fork</span><span class="p">()</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">QtGui</span><span class="o">.</span><span class="n">QApplication</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span>
<span class="c1"># Set up file system watcher</span>
<span class="n">qfsw</span> <span class="o">=</span> <span class="n">QtCore</span><span class="o">.</span><span class="n">QFileSystemWatcher</span><span class="p">()</span>
<span class="n">qfsw</span><span class="o">.</span><span class="n">addPaths</span><span class="p">(</span><span class="n">paths</span><span class="p">)</span>
<span class="n">QtCore</span><span class="o">.</span><span class="n">QObject</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">qfsw</span><span class="p">,</span><span class="n">QtCore</span><span class="o">.</span><span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"directoryChanged(QString)"</span><span class="p">),</span><span class="n">onFileSystemChanged</span><span class="p">)</span>
<span class="n">QtCore</span><span class="o">.</span><span class="n">QObject</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">qfsw</span><span class="p">,</span><span class="n">QtCore</span><span class="o">.</span><span class="n">SIGNAL</span><span class="p">(</span><span class="s2">"fileChanged(QString)"</span><span class="p">),</span><span class="n">onFileSystemChanged</span><span class="p">)</span>
<span class="c1"># Allow program to be interrupted with Ctrl+C</span>
<span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="n">signal</span><span class="o">.</span><span class="n">SIGINT</span><span class="p">,</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIG_DFL</span><span class="p">)</span>
<span class="c1"># Qt Main loop</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">exec_</span><span class="p">())</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</pre></div>
Text extents with Python Cairo2009-08-10T10:20:00+02:002009-08-10T10:20:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2009-08-10:/text-extents-with-python-cairo.html<p>I needed this and could not find it. So I share it here (and even better if Google'd index it!)</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">textwidth</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">14</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">cairo</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span> <span class="o">*</span> <span class="n">fontsize</span>
<span class="n">surface</span> <span class="o">=</span> <span class="n">cairo</span><span class="o">.</span><span class="n">SVGSurface</span><span class="p">(</span><span class="s1">'undefined.svg'</span><span class="p">,</span> <span class="mi">1280</span><span class="p">,</span> <span class="mi">200</span><span class="p">)</span>
<span class="n">cr</span> <span class="o">=</span> <span class="n">cairo</span><span class="o">.</span><span class="n">Context</span><span class="p">(</span><span class="n">surface</span><span class="p">)</span>
<span class="n">cr</span><span class="o">.</span><span class="n">select_font_face</span><span class="p">(</span><span class="s1">'Arial'</span><span class="p">,</span> <span class="n">cairo …</span></pre></div><p>I needed this and could not find it. So I share it here (and even better if Google'd index it!)</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">textwidth</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">fontsize</span><span class="o">=</span><span class="mi">14</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">cairo</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span> <span class="o">*</span> <span class="n">fontsize</span>
<span class="n">surface</span> <span class="o">=</span> <span class="n">cairo</span><span class="o">.</span><span class="n">SVGSurface</span><span class="p">(</span><span class="s1">'undefined.svg'</span><span class="p">,</span> <span class="mi">1280</span><span class="p">,</span> <span class="mi">200</span><span class="p">)</span>
<span class="n">cr</span> <span class="o">=</span> <span class="n">cairo</span><span class="o">.</span><span class="n">Context</span><span class="p">(</span><span class="n">surface</span><span class="p">)</span>
<span class="n">cr</span><span class="o">.</span><span class="n">select_font_face</span><span class="p">(</span><span class="s1">'Arial'</span><span class="p">,</span> <span class="n">cairo</span><span class="o">.</span><span class="n">FONT_SLANT_NORMAL</span><span class="p">,</span> <span class="n">cairo</span><span class="o">.</span><span class="n">FONT_WEIGHT_BOLD</span><span class="p">)</span>
<span class="n">cr</span><span class="o">.</span><span class="n">set_font_size</span><span class="p">(</span><span class="n">fontsize</span><span class="p">)</span>
<span class="n">xbearing</span><span class="p">,</span> <span class="n">ybearing</span><span class="p">,</span> <span class="n">width</span><span class="p">,</span> <span class="n">height</span><span class="p">,</span> <span class="n">xadvance</span><span class="p">,</span> <span class="n">yadvance</span> <span class="o">=</span> <span class="n">cr</span><span class="o">.</span><span class="n">text_extents</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="k">return</span> <span class="n">width</span>
</pre></div>
Colored Output in Console with Python2008-12-31T13:37:00+01:002008-12-31T13:37:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2008-12-31:/colored-output-in-console-with-python.html<p>Playing around with ANSI in a color capable terminal.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">BLACK</span><span class="p">,</span> <span class="n">RED</span><span class="p">,</span> <span class="n">GREEN</span><span class="p">,</span> <span class="n">YELLOW</span><span class="p">,</span> <span class="n">BLUE</span><span class="p">,</span> <span class="n">MAGENTA</span><span class="p">,</span> <span class="n">CYAN</span><span class="p">,</span> <span class="n">WHITE</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
<span class="c1">#following from Python cookbook, #475186</span>
<span class="k">def</span> <span class="nf">has_colours</span><span class="p">(</span><span class="n">stream</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="s2">"isatty"</span><span class="p">):</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">stream</span><span class="o">.</span><span class="n">isatty</span><span class="p">():</span>
<span class="k">return</span> <span class="kc">False</span> <span class="c1"># auto color only on …</span></pre></div><p>Playing around with ANSI in a color capable terminal.</p>
<div class="highlight"><pre><span></span><span class="ch">#!/usr/bin/env python</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="n">BLACK</span><span class="p">,</span> <span class="n">RED</span><span class="p">,</span> <span class="n">GREEN</span><span class="p">,</span> <span class="n">YELLOW</span><span class="p">,</span> <span class="n">BLUE</span><span class="p">,</span> <span class="n">MAGENTA</span><span class="p">,</span> <span class="n">CYAN</span><span class="p">,</span> <span class="n">WHITE</span> <span class="o">=</span> <span class="nb">range</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span>
<span class="c1">#following from Python cookbook, #475186</span>
<span class="k">def</span> <span class="nf">has_colours</span><span class="p">(</span><span class="n">stream</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="s2">"isatty"</span><span class="p">):</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">stream</span><span class="o">.</span><span class="n">isatty</span><span class="p">():</span>
<span class="k">return</span> <span class="kc">False</span> <span class="c1"># auto color only on TTYs</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">curses</span>
<span class="n">curses</span><span class="o">.</span><span class="n">setupterm</span><span class="p">()</span>
<span class="k">return</span> <span class="n">curses</span><span class="o">.</span><span class="n">tigetnum</span><span class="p">(</span><span class="s2">"colors"</span><span class="p">)</span> <span class="o">></span> <span class="mi">2</span>
<span class="k">except</span><span class="p">:</span>
<span class="c1"># guess false in case of error</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="n">has_colours</span> <span class="o">=</span> <span class="n">has_colours</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">printout</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">colour</span><span class="o">=</span><span class="n">WHITE</span><span class="p">):</span>
<span class="k">if</span> <span class="n">has_colours</span><span class="p">:</span>
<span class="n">seq</span> <span class="o">=</span> <span class="s2">"</span><span class="se">\x1b</span><span class="s2">[1;</span><span class="si">%d</span><span class="s2">m"</span> <span class="o">%</span> <span class="p">(</span><span class="mi">30</span><span class="o">+</span><span class="n">colour</span><span class="p">)</span> <span class="o">+</span> <span class="n">text</span> <span class="o">+</span> <span class="s2">"</span><span class="se">\x1b</span><span class="s2">[0m"</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">seq</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
</pre></div>
<p>A simple demo:</p>
<div class="highlight"><pre><span></span><span class="o"><</span><span class="n">code</span> <span class="n">python</span><span class="o">></span>
<span class="c1">#</span>
<span class="c1"># Test</span>
<span class="c1">#</span>
<span class="n">printout</span><span class="p">(</span><span class="s2">"[debug] "</span><span class="p">,</span> <span class="n">GREEN</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in green"</span><span class="p">)</span>
<span class="n">printout</span><span class="p">(</span><span class="s2">"[warning] "</span><span class="p">,</span> <span class="n">YELLOW</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in yellow"</span><span class="p">)</span>
<span class="n">printout</span><span class="p">(</span><span class="s2">"[error] "</span><span class="p">,</span> <span class="n">RED</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"in red"</span><span class="p">)</span>
</pre></div>
<img alt="" src="/images/ansi-color.png" />
Automatic blog publishing using Python and XML-RPC2008-12-01T13:37:00+01:002008-12-01T13:37:00+01:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2008-12-01:/automatic-blog-publishing-using-python-and-xml-rpc.html<p>The following piece of code collects links in a Pligg database (but could be any kind of source like RSS feed...) and automatically builds and publishes entries in a blog using XML-RPC.</p>
<p>A small database class :</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DatabaseAPI</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="s2">"localhost"</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">3306</span><span class="p">,</span> <span class="n">user</span><span class="o">=</span><span class="s2">"root"</span><span class="p">,</span> <span class="n">passwd</span><span class="o">=</span><span class="s2">"root123 …</span></pre></div><p>The following piece of code collects links in a Pligg database (but could be any kind of source like RSS feed...) and automatically builds and publishes entries in a blog using XML-RPC.</p>
<p>A small database class :</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">DatabaseAPI</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">connect</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="s2">"localhost"</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">3306</span><span class="p">,</span> <span class="n">user</span><span class="o">=</span><span class="s2">"root"</span><span class="p">,</span> <span class="n">passwd</span><span class="o">=</span><span class="s2">"root123"</span><span class="p">,</span> <span class="n">db</span><span class="o">=</span><span class="s2">"mysql"</span><span class="p">):</span>
<span class="kn">import</span> <span class="nn">MySQLdb</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">user</span><span class="p">,</span> <span class="n">passwd</span><span class="p">,</span> <span class="n">db</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">disconnect</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">fetchall</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sqlquery</span><span class="p">):</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sqlquery</span><span class="p">)</span>
<span class="k">return</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
</pre></div>
<p>A Blog publishing class :</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">BlogAPI</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">urlapi</span><span class="p">,</span> <span class="n">username</span><span class="p">,</span> <span class="n">password</span><span class="p">):</span>
<span class="kn">import</span> <span class="nn">xmlrpclib</span>
<span class="bp">self</span><span class="o">.</span><span class="n">xmlrpclib</span> <span class="o">=</span> <span class="n">xmlrpclib</span>
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">server</span> <span class="o">=</span> <span class="n">xmlrpclib</span><span class="o">.</span><span class="n">ServerProxy</span><span class="p">(</span> <span class="n">urlapi</span> <span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span> <span class="s2">"Could not connect to </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">url</span> <span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">username</span> <span class="o">=</span> <span class="n">username</span>
<span class="bp">self</span><span class="o">.</span><span class="n">password</span> <span class="o">=</span> <span class="n">password</span>
<span class="k">def</span> <span class="nf">newPost</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">title</span><span class="p">,</span> <span class="n">description</span><span class="p">,</span> <span class="n">blogid</span> <span class="o">=</span> <span class="s1">'1'</span><span class="p">,</span> <span class="n">publish</span> <span class="o">=</span> <span class="kc">True</span> <span class="p">):</span>
<span class="k">if</span> <span class="n">description</span> <span class="o">==</span> <span class="s2">""</span><span class="p">:</span>
<span class="n">description</span> <span class="o">=</span> <span class="s2">"<em>empty entry</em>"</span>
<span class="n">post</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">post</span><span class="p">[</span><span class="s1">'title'</span><span class="p">]</span> <span class="o">=</span> <span class="n">title</span>
<span class="n">post</span><span class="p">[</span><span class="s1">'description'</span><span class="p">]</span> <span class="o">=</span> <span class="n">description</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">server</span><span class="o">.</span><span class="n">metaWeblog</span><span class="o">.</span><span class="n">newPost</span><span class="p">(</span> <span class="n">blogid</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">username</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">password</span><span class="p">,</span> <span class="n">post</span><span class="p">,</span> <span class="n">publish</span> <span class="p">)</span>
<span class="k">return</span> <span class="n">r</span>
<span class="k">except</span> <span class="bp">self</span><span class="o">.</span><span class="n">xmlrpclib</span><span class="o">.</span><span class="n">Fault</span><span class="p">,</span> <span class="n">fault</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span> <span class="n">fault</span><span class="o">.</span><span class="n">faultString</span> <span class="p">)</span>
</pre></div>
<p>Build the SQL query : all entries of current week</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">datetime</span>
<span class="n">dt</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">weeks</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">today</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">date</span><span class="o">.</span><span class="n">today</span><span class="p">()</span>
<span class="n">agelimit</span> <span class="o">=</span> <span class="n">today</span> <span class="o">+</span> <span class="n">dt</span>
<span class="n">stmt</span> <span class="o">=</span> <span class="s2">"""</span>
<span class="s2">SELECT link_url, link_url_title</span>
<span class="s2">FROM pligg_links</span>
<span class="s2">WHERE link_status = 'published' AND link_published_date > "</span><span class="si">%s</span><span class="s2">"</span>
<span class="s2">ORDER BY link_published_date DESC"""</span> <span class="o">%</span> <span class="n">agelimit</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"%Y%m</span><span class="si">%d</span><span class="s2">000000"</span><span class="p">)</span>
</pre></div>
<p>Put everything together :</p>
<div class="highlight"><pre><span></span><span class="n">db</span> <span class="o">=</span> <span class="n">DatabaseAPI</span><span class="p">()</span>
<span class="n">db</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">user</span><span class="o">=</span><span class="s2">"user"</span><span class="p">,</span> <span class="n">passwd</span><span class="o">=</span><span class="s2">"pass"</span><span class="p">,</span> <span class="n">db</span><span class="o">=</span><span class="s2">"name"</span><span class="p">)</span>
<span class="n">entries</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">fetchall</span><span class="p">(</span><span class="n">stmt</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">disconnect</span><span class="p">()</span>
<span class="n">body</span> <span class="o">=</span> <span class="s2">"""</span>
<span class="s2"><ul>"""</span>
<span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="n">entries</span><span class="p">:</span>
<span class="n">body</span> <span class="o">+=</span> <span class="s2">"""</span>
<span class="s2"> <li><a href="</span><span class="si">%s</span><span class="s2">"></span><span class="si">%s</span><span class="s2"></a></li>"""</span> <span class="o">%</span> <span class="p">(</span><span class="n">entry</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">entry</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">body</span> <span class="o">+=</span> <span class="s2">"""</span>
<span class="s2"></ul>"""</span>
<span class="n">blog</span> <span class="o">=</span> <span class="n">BlogAPI</span><span class="p">(</span><span class="s2">"http://yourblog/xmlrpc/"</span><span class="p">,</span> <span class="s2">"user"</span><span class="p">,</span> <span class="s2">"pass"</span><span class="p">)</span>
<span class="n">title</span> <span class="o">=</span> <span class="s2">"Links week #</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">today</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"%W"</span><span class="p">)</span>
<span class="n">blog</span><span class="o">.</span><span class="n">newPost</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="n">body</span><span class="p">)</span>
</pre></div>
Another success story about Ruby On Rails working with Apache2008-08-01T09:12:00+02:002008-08-01T09:12:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2008-08-01:/another-success-story-about-ruby-on-rails-working-with-apache.html<p>This procedure does not guarantee anything and should be considered approximate. However this should be fairly enough for someone familiar with Apache environments.</p>
<div class="section" id="ruby-environment">
<h2>Ruby environment</h2>
<p>We install Ruby via apt-get</p>
<pre class="literal-block">
sudo apt-get install ruby libzlib-ruby rdoc irb
</pre>
<p>At this point, you can run Ruby scripts like you did with Python …</p></div><p>This procedure does not guarantee anything and should be considered approximate. However this should be fairly enough for someone familiar with Apache environments.</p>
<div class="section" id="ruby-environment">
<h2>Ruby environment</h2>
<p>We install Ruby via apt-get</p>
<pre class="literal-block">
sudo apt-get install ruby libzlib-ruby rdoc irb
</pre>
<p>At this point, you can run Ruby scripts like you did with Python or Perl.</p>
</div>
<div class="section" id="gems">
<h2>Gems</h2>
<p>Gems are like perl's CPAN or PHP's PEAR. We install it from source in order to enjoy the whole Gems repository (instead of being limited to packaged gems only).</p>
<pre class="literal-block">
wget "http://rubyforge.org/frs/download.php/38646/rubygems-x.x.x.tgz"
tar -xvzf rubygems-x.x.x.tgz
rm rubygems-x.x.x.tgz
cd rubygems-x.x.x
sudo ruby setup.rb
cd ..
rm -r rubygems-1.2.0
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem
</pre>
<p><em>(DO NOT use sudo ruby rubygemsx.x.x/setup.rb)</em></p>
<pre class="literal-block">
sudo gem update --system
</pre>
<p>We will install additionnal applications... those depend on your needs. The Gems are compiled on the fly, therefore development packages are usually required along.</p>
</div>
<div class="section" id="ruby-on-rails">
<h2>Ruby on Rails</h2>
<pre class="literal-block">
sudo gem install rails --include-dependencies
</pre>
</div>
<div class="section" id="mysql-support">
<h2>Mysql support</h2>
<pre class="literal-block">
sudo apt-get install libmysql-ruby libmysqlclient15-dev
sudo gem install mysql
</pre>
</div>
<div class="section" id="imagemagick-support">
<h2>ImageMagick support</h2>
<pre class="literal-block">
sudo apt-get install librmagick-ruby1.8 libmagick9-dev
sudo gem install rmagick
</pre>
</div>
<div class="section" id="integrate-with-apache">
<h2>Integrate with Apache</h2>
<pre class="literal-block">
sudo apt-get install apache2-prefork-dev
</pre>
<p>Enable additionnal modules</p>
<pre class="literal-block">
a2enmod rewrite
a2enmod suexec
a2enmod include
</pre>
<p><em>I might have missed some.</em>. Some tutorials recommend Fast-CGI.</p>
</div>
<div class="section" id="install-phusion-passenger-mod-rails">
<h2>Install Phusion Passenger (mod_rails)</h2>
<p>Usually, RubyOnRails has its own web server (<a class="reference external" href="http://en.wikipedia.org/wiki/Mongrel_(web_server)">Mongrel</a>) on port 3000. It is also quite common to have a cluster of processes with load balancing and Apache proxy...</p>
<p>But you may want to do something very simple that just integrates within Apache. Here comes Phusion Passenger!</p>
<pre class="literal-block">
sudo gem install passenger
sudo passenger-install-apache2-module
</pre>
<p><em>(following the instructions, or look at the [[http://www.modrails.com/documentation/Users%20guide.html|user guide]]).</em></p>
<p>At the end, the wizard tells you to add some lines in <cite>httpd.conf</cite>. I recommend the following method instead, which splits those lines into a module that you can enable / disable.</p>
<p>Create two files :
* /etc/apache2/mods-available/mod_rails.load</p>
<pre class="literal-block">
LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.0.2/ext/apache2/mod_passenger.so
</pre>
<ul>
<li><p class="first">/etc/apache2/mods-available/mod_rails.conf</p>
<pre class="literal-block">
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.0.2
PassengerRuby /usr/bin/ruby1.8
</pre>
</li>
</ul>
<p>Enable this new module</p>
<pre class="literal-block">
sudo a2enmod mod_rails
</pre>
</div>
<div class="section" id="create-your-virtualhost">
<h2>Create your VirtualHost</h2>
<p>The DocumentRoot must point to the public folder of your Ruby On Rails application.</p>
<ul>
<li><p class="first">If your Ruby application is alone, your apache site will be something like this</p>
<pre class="literal-block">
<virtualhost *:80>
ServerName yourapp
DocumentRoot /var/rails/yourapp/public/
ErrorLog /var/rails/yourapp/log/apache.log
<directory /var/rails/yourapp/public>
Options ExecCGI FollowSymLinks
AddHandler cgi-script .cgi
AllowOverride all
Order allow,deny
Allow from all
</directory>
</virtualhost>
</pre>
</li>
<li><p class="first">If you want it a subfolder of your current DocumentRoot, look at this.</p>
</li>
<li><p class="first">The simplest for me was to setup a sub-domain. Don't forget to update your DNS information.</p>
</li>
</ul>
<p>Relax ! Restart Apache and that's it !</p>
</div>
Apply Debian patches step-by-step2008-08-01T09:12:00+02:002008-08-01T09:12:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2008-08-01:/apply-debian-patches-step-by-step.html<p>I thought it may be relevant to share the history of commands I used to apply a patch and submit it to <a class="reference external" href="http://launchpad.net">launchpad</a>.</p>
<div class="section" id="get-the-tools">
<h2>0) Get the tools</h2>
<ul>
<li><p class="first">You'll need a working PGP key</p>
</li>
<li><p class="first">Install the necessary tools</p>
<pre class="literal-block">
sudo apt-get install devscripts dpatch fakeroot dh-make
</pre>
</li>
</ul>
<p><em>(I might have forgotten some...)</em> :)</p>
</div>
<div class="section" id="get-the-files">
<h2>1 …</h2></div><p>I thought it may be relevant to share the history of commands I used to apply a patch and submit it to <a class="reference external" href="http://launchpad.net">launchpad</a>.</p>
<div class="section" id="get-the-tools">
<h2>0) Get the tools</h2>
<ul>
<li><p class="first">You'll need a working PGP key</p>
</li>
<li><p class="first">Install the necessary tools</p>
<pre class="literal-block">
sudo apt-get install devscripts dpatch fakeroot dh-make
</pre>
</li>
</ul>
<p><em>(I might have forgotten some...)</em> :)</p>
</div>
<div class="section" id="get-the-files">
<h2>1) Get the files</h2>
<ul>
<li><p class="first">Get the last package source, from the package page : <a class="reference external" href="http://packages.ubuntu.com/hardy/exaile">http://packages.ubuntu.com/hardy/exaile</a></p>
<pre class="literal-block">
dget -x http://archive.ubuntu.com/ubuntu/pool/universe/e/exaile/exaile_0.2.11.1-0ubuntu3.dsc
</pre>
</li>
<li><p class="first">Get the patch file</p>
<pre class="literal-block">
wget http://launchpadlibrarian.net/9466876/gui_track_filter.patch
</pre>
</li>
<li><p class="first">Step in the package code</p>
<pre class="literal-block">
cd exaile-0.2.11.1/
</pre>
</li>
</ul>
</div>
<div class="section" id="apply-the-patch">
<h2>2) Apply the patch</h2>
<ul>
<li><p class="first">Have a look at the list of patches in the <cite>debian/patches</cite> folder</p>
<pre class="literal-block">
ls debian/patches/
00list
01_fix_makefile_for_pysupport.dpatch
02_files-on-share-dir.dpatch
03_fix_mmkeys.dpatch
</pre>
</li>
<li><p class="first">Create yours respecting dpatch filenames pattern (without extension)</p>
<pre class="literal-block">
dpatch-edit-patch 04_fix_search_exit
</pre>
</li>
<li><p class="first">You now have a shell in the <cite>/tmp</cite> folder.</p>
</li>
<li><p class="first">Apply the patch file</p>
<pre class="literal-block">
patch -p0 < $HOME/Desktop/gui_track_filter.patch
</pre>
</li>
<li><p class="first">Exit the shell</p>
</li>
</ul>
</div>
<div class="section" id="describe-your-changes">
<h2>3) Describe your changes</h2>
<ul>
<li><p class="first">Add your dpatch name in the <cite>00list</cite> file</p>
<pre class="literal-block">
nano debian/patches/00list
</pre>
</li>
<li><p class="first">Modify Debian changelog using</p>
<pre class="literal-block">
dch -i
</pre>
</li>
<li><p class="first">Your name and signing information will be automatically added.</p>
</li>
<li><p class="first">Look at previous descriptions and respect their structure.</p>
</li>
<li><p class="first">Include launchpad bug number with <cite>(LP: #number)</cite> string</p>
<pre class="literal-block">
exaile (0.2.11.1-0ubuntu4) hardy; urgency=low
</pre>
</li>
<li><p class="first"><cite>debian/patches/04_fix_search_exit.dpatch</cite></p>
<pre class="literal-block">
debian/patches/00list:
- Fix clean search terms on exit (LP: #95860)
-- Mathieu Leplatre <xxxx@gmail.com> Tue, 27 May 2008 10:45:42 -0300
</pre>
</li>
</ul>
</div>
<div class="section" id="create-debdiff">
<h2>4) Create debdiff</h2>
<ul>
<li><p class="first">Create the dsc and diff files using</p>
<pre class="literal-block">
debuild -S
cd ..
</pre>
</li>
<li><p class="first">Step in the package code</p>
</li>
<li><p class="first">Create the debdiff file using</p>
<pre class="literal-block">
debdiff exaile_0.2.11.1-0ubuntu3.dsc exaile_0.2.11.1-0ubuntu4.dsc > exaile_0.2.11.1-0ubuntu4.debdiff
</pre>
</li>
<li><p class="first">Have a look at it, it should include all modifications of previous steps</p>
<pre class="literal-block">
diff -Nru exaile-0.2.11.1/debian/changelog exaile-0.2.11.1/debian/changelog
--- exaile-0.2.11.1/debian/changelog 2008-05-27 10:54:56.000000000 -0300
+++ exaile-0.2.11.1/debian/changelog 2008-05-27 10:54:56.000000000 -0300
@@ -1,3 +1,11 @@
+exaile (0.2.11.1-0ubuntu4) hardy; urgency=low
+
+ * debian/patches/04_fix_search_exit.dpatch,
+ debian/patches/00list:
+ - Fix clean search terms on exit (LP: #95860)
+
+ -- Mathieu Leplatre <xxxx@gmail.com> Tue, 27 May 2008 10:45:42 -0300
+
exaile (0.2.11.1-0ubuntu3) hardy; urgency=low
* debian/patches/03_fix_mmkeys.dpatch,
diff -Nru /tmp/RIBRnUlXkn/exaile-0.2.11.1/debian/patches/00list /tmp/XQpuhOBOst/exaile-0.2.11.1/debian/patches/00l
ist
--- exaile-0.2.11.1/debian/patches/00list 2008-05-27 10:54:56.000000000 -0300
+++ exaile-0.2.11.1/debian/patches/00list 2008-05-27 10:54:56.000000000 -0300
@@ -1,3 +1,5 @@
01_fix_makefile_for_pysupport
02_files-on-share-dir
03_fix_mmkeys
+04_fix_search_exit
+
diff -Nru /tmp/RIBRnUlXkn/exaile-0.2.11.1/debian/patches/04_fix_search_exit.dpatch /tmp/XQpuhOBOst/exaile-0.2.11.1
/debian/patches/04_fix_search_exit.dpatch
--- exaile-0.2.11.1/debian/patches/04_fix_search_exit.dpatch 1969-12-31 21:00:00.000000000 -0300
+++ exaile-0.2.11.1/debian/patches/04_fix_search_exit.dpatch 2008-05-27 10:54:56.000000000 -0300
@@ -0,0 +1,27 @@
+#! /bin/sh /usr/share/dpatch/dpatch-run
+## 04_fix_search_exit.dpatch by Mathieu Leplatre <xxxx@gmail.com>
+##
+## All lines beginning with `## DP:' are a description of the patch.
+## DP: Patch to clean search terms on exit
+
+@DPATCH@
+diff -urNad exaile-0.2.11.1~/xl/gui/main.py exaile-0.2.11.1/xl/gui/main.py
+--- exaile-0.2.11.1~/xl/gui/main.py 2007-11-07 13:12:52.000000000 -0300
++++ exaile-0.2.11.1/xl/gui/main.py 2008-05-27 10:37:36.000000000 -0300
+@@ -1659,8 +1659,16 @@
+ queuefile = xl.path.get_config('queued.save')
+ if os.path.isfile(queuefile):
+ os.unlink(queuefile)
++
+
+ if self.player.current: self.player.current.stop()
++
++ # Clear the search filter so that the entire playlist is saved
++ self.tracks_filter.set_text('')
++ try:
++ self.on_search()
++ except: # In case we're quitting before the playlist loaded
++ pass
+
+ for i in range(self.playlists_nb.get_n_pages()):
+ page = self.playlists_nb.get_nth_page(i)
</pre>
</li>
</ul>
</div>
Static build of Cairo and librsvg2008-07-01T11:25:00+02:002008-07-01T11:25:00+02:00Mathieu Leplatretag:blog.mathieu-leplatre.info,2008-07-01:/static-build-of-cairo-and-librsvg.html<img alt="" src="/images/unicode.png" />
<div class="section" id="why">
<h2>Why ?</h2>
<ul class="simple">
<li>Convert SVG files to PDF or PNG, with full Unicode support (right-to-left languages), transparency, gradients, PDF image compression, ...</li>
<li>Cairo and librsvg are the best in town.</li>
<li>Cairo and librsvg are very modern libraries which became famous only in the past 3 years. Thus, GNU/Linux distributions do not always …</li></ul></div><img alt="" src="/images/unicode.png" />
<div class="section" id="why">
<h2>Why ?</h2>
<ul class="simple">
<li>Convert SVG files to PDF or PNG, with full Unicode support (right-to-left languages), transparency, gradients, PDF image compression, ...</li>
<li>Cairo and librsvg are the best in town.</li>
<li>Cairo and librsvg are very modern libraries which became famous only in the past 3 years. Thus, GNU/Linux distributions do not always have recent versions and full capabilities.</li>
<li>A <cite>static build</cite> does all the bindings to libraries at compile time, which hence removes specific versions dependencies. <em>(this method has many drawbacks but can help sometimes)</em></li>
</ul>
</div>
<div class="section" id="the-program-svgconvert-c">
<h2>The program : svgconvert.c</h2>
<blockquote>
<ul class="simple">
<li>This program is a merge of Carl Worth's <a class="reference external" href="http://cgit.freedesktop.org/~cworth/svg2pdf/">svg2pdf</a> and <a class="reference external" href="http://cgit.freedesktop.org/~cworth/svg2png/">svg2png</a></li>
</ul>
</blockquote>
<div class="highlight"><pre><span></span><span class="cm">/*</span>
<span class="cm">* Copyright © 2005 Red Hat, Inc.</span>
<span class="cm">* Copyright © 2006 Red Hat, Inc.</span>
<span class="cm">* Copyright © 2007 Red Hat, Inc.</span>
<span class="cm">*</span>
<span class="cm">* Permission is hereby granted, free of charge, to any person</span>
<span class="cm">* obtaining a copy of this software and associated documentation</span>
<span class="cm">* files (the "Software"), to deal in the Software without</span>
<span class="cm">* restriction, including without limitation the rights to use, copy,</span>
<span class="cm">* modify, merge, publish, distribute, sublicense, and/or sell copies</span>
<span class="cm">* of the Software, and to permit persons to whom the Software is</span>
<span class="cm">* furnished to do so, subject to the following conditions:</span>
<span class="cm">*</span>
<span class="cm">* The above copyright notice and this permission notice shall be</span>
<span class="cm">* included in all copies or substantial portions of the Software.</span>
<span class="cm">*</span>
<span class="cm">* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,</span>
<span class="cm">* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF</span>
<span class="cm">* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND</span>
<span class="cm">* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS</span>
<span class="cm">* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN</span>
<span class="cm">* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN</span>
<span class="cm">* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE</span>
<span class="cm">* SOFTWARE.</span>
<span class="cm">*</span>
<span class="cm">* Authors: Kristian Høgsberg <krh@redhat.com></span>
<span class="cm">* Carl Worth <cworth@redhat.com></span>
<span class="cm">* Behdad Esfahbod <besfahbo@redhat.com></span>
<span class="cm">* Mathieu Leplatre <contact@mathieu-leplatre.info></span>
<span class="cm">*/</span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdio.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><stdlib.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><glib/gprintf.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><librsvg/rsvg.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><librsvg/rsvg-cairo.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><cairo-pdf.h></span>
<span class="cp">#define FAIL(msg) \</span>
<span class="cp">do { fprintf (stderr, "FAIL: %s\n", msg); exit (-1); } while (0)</span>
<span class="cp">#define PIXELS_PER_POINT 1</span>
<span class="cp">#define PDF 0</span>
<span class="cp">#define PNG 1</span>
<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">argc</span><span class="p">,</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">argv</span><span class="p">[])</span>
<span class="p">{</span>
<span class="w"> </span><span class="n">GError</span><span class="w"> </span><span class="o">*</span><span class="n">error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">NULL</span><span class="p">;</span>
<span class="w"> </span><span class="n">RsvgHandle</span><span class="w"> </span><span class="o">*</span><span class="n">handle</span><span class="p">;</span>
<span class="w"> </span><span class="n">RsvgDimensionData</span><span class="w"> </span><span class="n">dim</span><span class="p">;</span>
<span class="w"> </span><span class="kt">double</span><span class="w"> </span><span class="n">width</span><span class="p">,</span><span class="w"> </span><span class="n">height</span><span class="p">;</span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">filename</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">char</span><span class="w"> </span><span class="o">*</span><span class="n">output_filename</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">argv</span><span class="p">[</span><span class="mi">2</span><span class="p">];</span>
<span class="w"> </span><span class="n">cairo_surface_t</span><span class="w"> </span><span class="o">*</span><span class="n">surface</span><span class="p">;</span>
<span class="w"> </span><span class="n">cairo_t</span><span class="w"> </span><span class="o">*</span><span class="n">cr</span><span class="p">;</span>
<span class="w"> </span><span class="n">cairo_status_t</span><span class="w"> </span><span class="n">status</span><span class="p">;</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">mode</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">argc</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">3</span><span class="p">)</span>
<span class="w"> </span><span class="n">FAIL</span><span class="w"> </span><span class="p">(</span><span class="s">"usage: svgconvert input_file.svg output_file"</span><span class="p">);</span>
<span class="w"> </span><span class="n">mode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PDF</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">g_str_has_suffix</span><span class="w"> </span><span class="p">(</span><span class="n">g_ascii_strdown</span><span class="w"> </span><span class="p">(</span><span class="n">output_filename</span><span class="p">,</span><span class="w"> </span><span class="mi">-1</span><span class="p">),</span><span class="w"> </span><span class="s">".png"</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">mode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PNG</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">g_type_init</span><span class="w"> </span><span class="p">();</span>
<span class="w"> </span><span class="n">rsvg_set_default_dpi</span><span class="w"> </span><span class="p">(</span><span class="mf">72.0</span><span class="p">);</span>
<span class="w"> </span><span class="n">handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rsvg_handle_new_from_file</span><span class="w"> </span><span class="p">(</span><span class="n">filename</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">error</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">error</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="nb">NULL</span><span class="p">)</span>
<span class="w"> </span><span class="n">FAIL</span><span class="w"> </span><span class="p">(</span><span class="n">error</span><span class="o">-></span><span class="n">message</span><span class="p">);</span>
<span class="w"> </span><span class="n">rsvg_handle_get_dimensions</span><span class="w"> </span><span class="p">(</span><span class="n">handle</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">dim</span><span class="p">);</span>
<span class="w"> </span><span class="n">width</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dim</span><span class="p">.</span><span class="n">width</span><span class="p">;</span>
<span class="w"> </span><span class="n">height</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dim</span><span class="p">.</span><span class="n">height</span><span class="p">;</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">mode</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">PNG</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">surface</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cairo_image_surface_create</span><span class="w"> </span><span class="p">(</span><span class="n">CAIRO_FORMAT_ARGB32</span><span class="p">,</span><span class="w"> </span><span class="n">width</span><span class="p">,</span><span class="w"> </span><span class="n">height</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">surface</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cairo_pdf_surface_create</span><span class="w"> </span><span class="p">(</span><span class="n">output_filename</span><span class="p">,</span><span class="w"> </span><span class="n">width</span><span class="p">,</span><span class="w"> </span><span class="n">height</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">cr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cairo_create</span><span class="w"> </span><span class="p">(</span><span class="n">surface</span><span class="p">);</span>
<span class="w"> </span><span class="n">rsvg_handle_render_cairo</span><span class="w"> </span><span class="p">(</span><span class="n">handle</span><span class="p">,</span><span class="w"> </span><span class="n">cr</span><span class="p">);</span>
<span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cairo_status</span><span class="w"> </span><span class="p">(</span><span class="n">cr</span><span class="p">);</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">status</span><span class="p">)</span>
<span class="w"> </span><span class="n">FAIL</span><span class="w"> </span><span class="p">(</span><span class="n">cairo_status_to_string</span><span class="w"> </span><span class="p">(</span><span class="n">status</span><span class="p">));</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">mode</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">PNG</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">cairo_surface_write_to_png</span><span class="w"> </span><span class="p">(</span><span class="n">surface</span><span class="p">,</span><span class="w"> </span><span class="n">output_filename</span><span class="p">);</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="n">cairo_destroy</span><span class="w"> </span><span class="p">(</span><span class="n">cr</span><span class="p">);</span>
<span class="w"> </span><span class="n">cairo_surface_destroy</span><span class="w"> </span><span class="p">(</span><span class="n">surface</span><span class="p">);</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="build-with-shared-librairies">
<h2>Build with shared librairies</h2>
<p>You would just do :</p>
<div class="highlight"><pre><span></span>gcc<span class="w"> </span><span class="sb">`</span>pkg-config<span class="w"> </span>--cflags<span class="w"> </span>--libs<span class="w"> </span>librsvg-2.0<span class="w"> </span>cairo-pdf<span class="sb">`</span><span class="w"> </span>-o<span class="w"> </span>svgconvert<span class="w"> </span>svgconvert.c
</pre></div>
<p>which creates a binary of 9.0K.</p>
</div>
<div class="section" id="build-with-static-librairies">
<h2>Build with static librairies</h2>
<ul class="simple">
<li>First install the development versions of the packages, to make sure you have all <strong>/usr/lib/*.a</strong> mentioned below.</li>
<li>Use this Makefile, which creates a binary of 5.9M. It was tested on Ubuntu 8.04 which comes with Gnome 2.22, librsvg 2.22 and Cairo 1.6.0.</li>
</ul>
<div class="highlight"><pre><span></span><span class="nv">ALL</span><span class="o">=</span>svgconvert
<span class="nv">MYCFLAGS</span><span class="o">=</span><span class="sb">`</span>pkg-config<span class="w"> </span>--cflags<span class="w"> </span>librsvg-2.0<span class="w"> </span>cairo-pdf<span class="sb">`</span>
<span class="nv">LDFLAGS</span><span class="o">=</span><span class="sb">`</span>pkg-config<span class="w"> </span>--libs<span class="w"> </span>librsvg-2.0<span class="w"> </span>cairo-pdf<span class="w"> </span>freetype2<span class="w"> </span>fontconfig<span class="w"> </span>pango<span class="w"> </span>pangoft2<span class="w"> </span>pangocairo<span class="w"> </span>cairo-ft<span class="w"> </span>libthai<span class="w"> </span>datrie<span class="w"> </span>libgsf-1<span class="w"> </span>gnome-vfs-2.0<span class="w"> </span>libcroco-0.6<span class="w"> </span>libpcre<span class="w"> </span>pixman-1<span class="w"> </span>libpng<span class="w"> </span>libxml-2.0<span class="sb">`</span>
<span class="nv">MYLDFLAGS</span><span class="o">=</span><span class="k">$(</span>LDFLAGS<span class="k">)</span><span class="w"> </span>/usr/lib/libgio-2.0.a<span class="w"> </span>/usr/lib/libglib-2.0.a<span class="w"> </span>/usr/lib/libselinux.a<span class="w"> </span>/usr/lib/libexpat.a<span class="w"> </span>/usr/lib/libfreetype.a<span class="w"> </span>/usr/lib/libbz2.a<span class="w"> </span>/usr/lib/libjpeg.a<span class="w"> </span>/usr/lib/libtiff.a<span class="w"> </span>/usr/lib/libbz2.a<span class="w"> </span>/usr/lib/libz.a<span class="w"> </span>/usr/lib/libm.a
all:<span class="w"> </span><span class="k">$(</span>ALL<span class="k">)</span>
%:<span class="w"> </span>%.c
<span class="w"> </span><span class="k">$(</span>CC<span class="k">)</span><span class="w"> </span>$^<span class="w"> </span>-pthread<span class="w"> </span><span class="k">$(</span>MYCFLAGS<span class="k">)</span><span class="w"> </span>-static<span class="w"> </span><span class="k">$(</span>MYLDFLAGS<span class="k">)</span><span class="w"> </span>-o<span class="w"> </span><span class="nv">$@</span>
clean:
<span class="w"> </span>rm<span class="w"> </span>-f<span class="w"> </span><span class="k">$(</span>ALL<span class="k">)</span><span class="w"> </span>*.o
</pre></div>
<ul class="simple">
<li>To check if <strong>pkg-config</strong> knows about a specific library :</li>
</ul>
<div class="highlight"><pre><span></span>$<span class="w"> </span>pkg-config<span class="w"> </span>--list-all<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>vfs
gnome-vfs-sharp-2.0<span class="w"> </span>GnomeVfs<span class="w"> </span>-<span class="w"> </span>GnomeVfs
gnome-vfs-2.0<span class="w"> </span>gnome-vfs<span class="w"> </span>-<span class="w"> </span>The<span class="w"> </span>GNOME<span class="w"> </span>virtual<span class="w"> </span>file-system<span class="w"> </span>libraries
gnome-vfsmm-2.6<span class="w"> </span>gnome-vfsmm<span class="w"> </span>-<span class="w"> </span>C++<span class="w"> </span>wrapper<span class="w"> </span><span class="k">for</span><span class="w"> </span>gnome-vfs
gnome-vfs-module-2.0<span class="w"> </span>gnome-vfs-module<span class="w"> </span>-<span class="w"> </span>The<span class="w"> </span>GNOME<span class="w"> </span>virtual<span class="w"> </span>file-system<span class="w"> </span>module<span class="w"> </span>include<span class="w"> </span>info
</pre></div>
<ul class="simple">
<li>To check if a library has a specific symbol, use the <strong>nm</strong> command :</li>
</ul>
<div class="highlight"><pre><span></span>$<span class="w"> </span>nm<span class="w"> </span>/usr/lib/libexpat.a<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>XML_SetStart
000001c0<span class="w"> </span>T<span class="w"> </span>XML_SetStartCdataSectionHandler
<span class="m">00000240</span><span class="w"> </span>T<span class="w"> </span>XML_SetStartDoctypeDeclHandler
<span class="m">00000150</span><span class="w"> </span>T<span class="w"> </span>XML_SetStartElementHandler
000002a0<span class="w"> </span>T<span class="w"> </span>XML_SetStartNamespaceDeclHandler
</pre></div>
</div>
<div class="section" id="download">
<h2>Download</h2>
<ul class="simple">
<li><a class="reference external" href="http://mathieu-leplatre.info/media/svgconvert-src.tar.gz">Source</a></li>
<li><a class="reference external" href="http://mathieu-leplatre.info/media/svgconvert-bin.tar.gz">Binary</a></li>
</ul>
</div>
<div class="section" id="references">
<h2>References</h2>
<ul class="simple">
<li><a class="reference external" href="http://cairographics.org">http://cairographics.org</a></li>
<li><a class="reference external" href="http://librsvg.sourceforge.net">http://librsvg.sourceforge.net</a></li>
<li>Thanks for the precious help of <a class="reference external" href="http://www.cworth.org">Carl Worth</a> on <cite>#cairo</cite> at irc.freenode.net, Zugzwang and nvteighen on <a class="reference external" href="http://ubuntuforums.org">http://ubuntuforums.org</a></li>
</ul>
</div>