Serving live tiles from a QGIS project via TileStache


I’m more then likly way behind the 8 ball here, aren’t all the cool kids doing tiles these days, but regardless I thought it was pretty cool to share. The other day I found TileStache a neat little Python app that can generate, cache, and serve tiles from a list of providers. The normal way is via Mapnik (and others) to render a image, there is also a vector provider which can render vector tiles. Nifty.

A while ago I wrote qgis2img which can generate an image for project, or layers, and export it for you. It serves two roles, one is to benchmark a project and layer render times, the other as a simple export tool. I thought it would be pretty cool to be able to export tiles from it but was I never really up for working on the math and all the logic so I left it. Then I found TileStache.

The best part about TileStache, apart from that it’s Python, is that you can make your own providers for it, and the API is dead easy

class Provider:
    def __init__(self, layer):
        self.layer = layer

    def renderArea(self, width, height, srs, xmin, ymin, xmax, ymax, zoom):
        pass

How easy is that! Just implement one method and you are good to go. So that’s what I did. I created a custom provider that will load a QGIS project and render out images. Thanks to the work done by Martin from Lutra Consulting for the multithreaded rendering in QGIS 2.4 this is a hell of a lot easier then it used to be.

Ignoring some of the setup code to create and load the session the whole export logic is in these few lines

   extents = QgsRectangle(xmin, ymin, xmax, ymax)
   settings.setExtent(extents)
   settings.setOutputSize(QSize(width, height))
   layers = [layer.id() for layer in project.visiblelayers()]
   image, rendertime = qgis2img.render.render_layers(settings, layers, RenderType=QgsMapRendererSequentialJob)

with render_layers defined as

def render_layers(settings, layers, RenderType):
    settings.setLayers(layers)
    job = RenderType(settings)
    job.start()
    job.waitForFinished()
    image = job.renderedImage()
    return image, job.renderingTime()

As this is build on top of my qgis2img tool you can see the full code here

Running it is as simple as installing TileStache, cloneing qgis2img, updating tilestache.cfg, and running the server.

$ pip install TileStache
$ git clone https://github.com/DMS-Aus/qgis2img.git
$ cd qgis2img

In tilestache.cfg you can just change the path to the project to render:

{
  "cache": {
    "name": "Test",
    "path": "/tmp/stache"
    },
  "layers": {
    "qgis":
    {
      "provider": {"class": "qgis2img.tilestache:Provider",
                   "kwargs": {"projectfile": "data/test.qgs"}
                  }
    }
  }
}

Then run the server

$ tilestache-server /path/to/tilestache.cfg

Note: The path to the .cfg file seems to have to be the full path. I had issues with relative paths working.

To view the tiles you can load the preview URL that TileStache provides or you can use it in something like LeafLet

<!DOCTYPE html>
<html>
<head>
    <title>QGIS Tiles WOOT!</title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.css" />
</head>
<body>
    <div id="map" style="position: absolute; top: 0; right: 0; bottom: 0; left: 0;"></div>
    <script src="http://cdn.leafletjs.com/leaflet-0.7.3/leaflet.js"></script>
    <script>

        var map = L.map('map').setView([51.505, -0.09], 5);

        L.tileLayer('http://10.0.0.11:8080/{id}/{z}/{x}/{y}.png', {
            maxZoom: 18,
            id: 'qgis'
        }).addTo(map);

    </script>
</body>
</html>

And the result is live tiles from a QGIS project.

tiles

Winning!

Some Caveats

  • The provider currently doesn’t use metatiles so labels and points will get chopped at tile edge. I have working code for this but haven’t pushed it yet.

  • I don’t kill the QGIS session that I create. Creating a session for each request was really expensive so I just keep it around.

  • I only load the project once so any changes mean starting and stopping the server. Wouldn’t be hard to add a file watcher for this.

  • It’s just me using this at home for fun, I have no idea on how it would scale, or even if it would, but I would be keen to hear feedback on that theory.

Exporting QGIS symbols as images


Ever wanted to export your QGIS symbols as images? Yes. Well here is some Python code that will let you do just that:

from PyQt4.QtCore import QSize
from PyQt4.QtGui import QImage, QPainter

style = QgsStyleV2.defaultStyle()
names = style.symbolNames()
size = QSize(64, 64)

for name in names:
    symbol = style.symbol(name)
    if not symbol.type() == QgsSymbolV2.Marker:
        continue

    image = QImage(size, QImage.Format_ARGB32_Premultiplied)
    image.fill(0) 
    painter = QPainter(image)
    symbol.drawPreviewIcon(painter, size)
    painter.end()
    image.save(r"C:temp{}.png".format(name), "PNG")

Or in 2.6 it’s even easier:

from PyQt4.QtCore import QSize
from PyQt4.QtGui import QImage, QPainter

style = QgsStyleV2.defaultStyle()
names = style.symbolNames()
size = QSize(64, 64)

for name in names:
    symbol = style.symbol(name)
    if not symbol.type() == QgsSymbolV2.Marker:
        continue

    image = symbol.asImage(size)
    image.save(r"C:temp{}.png".format(name), "PNG")

Bam!

symbols

Why? Because we can.

The QGIS Field calculator is dead. Long live the Field calculator bar


Ahhh the good old field calculator, it’s in a better place now. OK not really, it’s still there if you need it, but we can do a little better in 2.4. Introducing the field calculator bar:

Alt Text

oooo fancy.

The field calculator has always bugged me, I think it was just the combination of a few things:

  • It’s modal so it blocks you from doing anything else – this alone is motivation enough in my mind.
  • You have to do the Open – Run – Close – Open – Run – Close dance which isn’t great – annoying to say the least.
  • Did I mention it’s modal – AAAAAAAHHHHHHHHHHH
  • Defaults to creating a new field – which is the edge case
  • You only have All or Selection, which is a bit limiting

Anyway, enough with that. Last night I was having a chat to Nyall about something unrelated, and while looking at Excel I thought about that little bar at the top and how handy that is. You don’t see a field calculator dialog in Excel – well there is one but not for the common case – you just wack in your expression and it does its thing. Why couldn’t we have this for QGIS? I think I said to Nyall “you know this would be pretty cool, I might give it a go”. Couple of hours later and this is it.

fieldcalc

I have expressed in the past, and above, my hate for modal dialogs, so that was the main motivation and the results are much nicer then before.

What do we gain:

  • Not modal – WIN!
  • Don’t have to close anything to see your results
  • See the results as soon as you run Update (All|Filtered)
  • Works on the features in table (All|Filtered)
  • Does one job

The other improvment to the old dialog is what features the bar works on. The bar will update what it sees in the dialog. If you need to update just the selection, simply select Show Selected and run the update. Need to search for something to update? Run a filter and press update. The method has changed from All and Selected to All and Filtered. Just remember if you see it in the attribute table it will be updated.

fieldcalc-filter

The last point is important too, if you need a new field you use the New Field button, then run the update, there is no need to mix the two function into one tool. SRP.

This feature will be in 2.4. If you find any bugs assign them to me at hub.qgis.org and I will try to address them before 2.4 is out.

RIP Field calculator dialog

IntraMaps Roam – A Python QGIS data collection app


For the last couple of months I, though Digital Mapping Solutions, have been working on a tablet friendly data collection application that has been built on Python and QGIS. For those of you who have seen my QMap project I started a year or so ago you can consider this a reincarnation of that project, and that project retired. Most of the code has been reworked and using the QGIS libs gave me full flexibility in layout and workflow.

IntraMaps Roam (or Roam for short) is a standalone, fully bundled, Python application that was created to do data collection with a QGIS backend. The primary use of Roam is in a disconnected setup were one might not have internet connection, however Roam is using QGIS so will support any data format QGIS does. You can can use Roam in a connected environment, if your internet premits, by having WFS and WMS layers, or direct database connections; it’s up to you. Roam forms also allow for custom logic to be added to each form using Python so you can add your own workflow if needed.

Roam Map Window
Roam Data Capture

The binary package comes with a config manager application that can be used to create and manage Roam projects

Config Manager
Config Manager

The release page contains links to the 2.0 installers. The wiki contains all the information to get started. You can also take a look at the FAQ for the common questions.

Roam has been a great exercise in using and bundling QGIS libs with a Python application, which I have never done before but turned out to be pretty easy. Being a fully bundled application means you don’t need to install QGIS, or Python, on the client in order to run the application. Everything is in a nice bundled exe.

As Roam is based on PyQt and QGIS it is under the GPL2 license. Pull requests are welcome.

Currently Roam is only being packaged for Windows, because that was our first priority, however there isn’t a lot of Windows only stuff in the code itself so creating a version for OS X and Linux shouldn’t be too hard for someone with the know how.

Links

Happy mapping!?

Keeping QGIS settings in sync on different machines


Here is a quick tip from a GIS.SE post that I answered the other day.

The topic was keeping the WMS settings in sync over different operating systems and machines. Normally QGIS will store it settings in the registry on Windows and in different locations on Linux and OS X. So then comes the question of how do you keep them in sync if you are using different machines.

Well the answer is simple. QGIS provides --optionspath and --configpath command line options in order to move the .qgis2 and settings files. Using these two options, or just the one depending on what you need, will allow you to store the QGIS settings in a different location. Rather then storing the settings in the registry, or .config and .plist files, it will create a .ini file and save everything there.

All in all this means you can redirect your QGIS settings to a folder on dropbox and tell your QGIS installs to load the settings from a single place keeping everything in sync. When you change a setting it will sync with Dropbox and onto your other machines.

The simple way on Windows to add the --optionspath and --configpath options is to copy the shortcut to QGIS and append it to the end of the Target.

--optionspath "F:mydropboxqgis" --configpath "F:mydropboxqgis"

QGIS IN THE CLOUD!!11! ok not really but still cool

Oh God my plugins! My precious QGIS plugins


tl;dr

The API had to change. We don’t like breaking plugins. It makes us sad. We did it now to save pain in the future. You will like the new version better. Trust me.

What happened to my cheese?

When updating to QGIS 2.0 you might have noticed two things. (Apart from all the new awesome stuff!)

  1. All your settings have been set back to defaults
  2. Some of your plugins are gone, or missing in the installer.

Resetting of settings is caused by QGIS now storing its 2.0 settings in a different folder then we used for 1.8. In 1.8, all your plugins, etc, were stored in the ./qgis folder in your home directory, in 2.0 these are now stored in ./qgis2. The reason will become evident later. All user settings, the UI layout, database connections, etc, are now stored in a QGIS location. In windows this in the registry key HKEY_CURRENT_USER\Software\QGIS\QGIS2. So this explains why your settings are missing when you first load QGIS 2.0.

Why did we have to move the settings location?

Good question.

2.0 is very different from 1.8. There has been a lot of work to make this the best version we have ever made, new features, more bug fixes, a bigger dev team, and a even bigger community. Being the next major release we had to make some calls to remove some of the old code and features that were weighing us down. Features such as the old labelling engine, old symbol engine, the old vector API. Carrying old code and old out dated features into the future can sometimes really hurt a project and they have to be cut loose. Because of the massive amount of changes in 2.0 people needed to be able to run 2.0 and 1.8 on the same machine without causing any issues. If they both store settings in the same location this would have had bad results.

Why move the settings. Part 2

Moving the settings was also a result of having non backwards compatible plugins between 1.x and 2.x. If we kept both plugins in the same folder it just wouldn’t work. You would install a 1.8 version of a plugin, I would update my plugin to 2.0, you then install the same plugin in 2.0, and now you 1.8 version is broken. Fun!. To avoid this we moved all QGIS 2.0 stuff into ./qgis2.

Why did my plugins break anyway. Why not just leave them be.

In 1.x we were using SIP v1. This meant the framework we used, PyQt, felt more like C++ then it did Python. If you are a Python developer then this isn’t a very fun thing to deal with. In SIP v1 you need to tell PyQt, and our QGIS API, what to convert the type to. feature['mycolumn'].toInt() that is pretty gross. In V2 you can just do feature['mycolumn'] and SIP will auto convert the type for you. This makes our API feel more like Python and less like C++. There are other changes when using SIP V2 but you get the idea. Unfortunately SIP v1 and v2 do not work together so we couldn’t make the code backwards compatible. This was also a forced change for us. Once we switch to Python 3 at some stage in the future V2 would be the default and we have to change then. The bright side of this change is most of the time you are removing code. Consider it a good time to go though your code, give it a bit of a polish, and remove anything that is no longer needed.

There was another major API change that needed to happen. Vector API update. In order to allow for multithreading in the future, and we all know everyone is looking forward to that, we needed to change how code can ask for features from a layer. The old method would never work in a multithreaded structure and had to go.

What can I do if I need a plugin?

Having a plugin missing from the plugin installer when you really need it can be a bit of a pain. Plugin authors are working hard to update there plugins. I approve about two a day into the plugin repository. While most plugins might be updated at some stage. There are some things that you can do if you need a plugin update to work with 2.0.

  1. Email the author of the plugin to see where they are at with the update

  2. Email the author and offer your help to update the plugin. Remember a lot of plugins are written by volunteers who just need the plugin to get their work done and wanted to share it with everyone.

  3. If the author has no intention of updating the plugin, or can’t be contacted. You are free to update you local copy and offer it back to the community as the updated copy. If you are going to upload it back to the plugin repository please try to contact the author and seek permission first. I will be watching for copies of plugins to make sure we don’t end up with 10 versions of the same plugin. The GPL allows for you to modify and share your updated version but it’s nice to keep the original author in the loop. If the author no longer wants to maintain the plugin and you are able to then contact me and I will make you the owner of the plugin. Overall be nice not evil, we are all friends here.

  4. If you don’t have, or know someone with, the skills to update the plugin. You can contact a developer to help update the plugin for you. Companies like mine, or Faunalia, or a whole range of other open source devs, can normally be contracted to update a plugin if needed.

Moving forward

We like the new API. It makes the Python side of QGIS much cleaner. There is still more work to do, it’s never ending, but this is a good step. We don’t like breaking plugins, but breaking a few now is better then breaking heaps as the popularity of QGIS continues to grow.

QGIS 2.0: Dealing with Null values in pyqgis


Nothing in life ever comes free. Each time you make a desision there can sometimes be long a lasting effect. This is exectly the case with the new QGIS 2.0 pyqgis API. Changing to the new api was a big step in order make the API clear and more Pythonic. But there was a small detail to fix.

In QGIS 2.0 we can now just do this to read the attribute value for a field:

>>> feature['yourcolumn']
"Hello World"

What would you expect to get if the value was NULL (not empty string). Would you expect to see None? You would think so. Right? However lets just check that.

>>> feature['yourcolumn']
NULL

NULL? What the heck? That isn’t None! What is NULL?

>>> type(NULL)
<class 'PyQt4.QtCore.QPyNullVariant'>

QPyNullVariant WA! WHAAAT! Why didn’t we get None? Turns out by removing QVariant from PyQt it had some impact on methods that expected a NULL QVariant – A QVariant with no value. Passing None didn’t work because those methods needed the type information that QVariant holds, even when NULL.

When using SIP V2, which is what QGIS 2.0 is using, PyQt will auto convert any Null QVariants to QPyNullVariant.

The NULL you see is a variable masking the QPyNullVariant so that the output is nicer and it’s easier to work with. We have also added a bunch of methods to QPyNullVariant in order to make it act as much like None as we can.

If you see a NULL this is how you can deal with it:

>>> feature['yourcolumn']
NULL

>>> if not feature['nullcolumn']:
>>> print "Was null"
Was null

>>> if feature['nullcolumn'] == NULL:
>>> print "Was null"
Was null

NULL will also return False if boolean checked:

>>> value = NULL
>>> if value:
... print "This value wasn't null"
... else:
... print 'This value was null'
This value was null

It will also be equal to any other NULL aka QPyNullVariant variable

>>> value = NULL
>>> value2 = NULL
>>> value == value2
True
>>> value = 100
>>> value2 = NULL
>>> value == value2
False

Does None is NULL work? No.

One way to check if something is Null in Python is to use value is None however this will not work with our NULL type. Overloading the is operator in Python is not supported and there is no way we can support this – trust me I have tried. is is really doing id(object) == id(object) under the hood:

DO NOT do the following if you are checking for null values in pyqgis unless you know the return type is None:

>>> value = NULL
>>> value is None
false