Introduction

For the past couple of weeks I’ve been trying to build a toy dynamic tile server with Golang. “But your highness, there are existing tile server implementations in Python like titiler, and terracotta. Why build a new one?” you ask?

This serves 2 major purposes

1. To learn what goes under the hood behind something like rio_tiler.tile(x, y, z). Believe me there’s more than you would think.

2. Golang ecosystem for geospatial computations is so far behind from where Python’s is, especially to deal with rasters. With libraries like rasterio, titiler, rio-tiler, Pythonistas like me are blessed to have all these tools to build map things. And naturally, someone new would flock to these libraries for good reasons.

Although with the existence of gospatial, most of the vector manipulation and vector tile server etc have been dealt with. Especially tools to do computations for rasters like tiffs and zarr are not readily available. This tile server is my way to create something people can refer to.

Since there are some positive things in Golang which enables writing better webservices than Python, I have a feeling that it’d open some new avenues and unlock ideas on how we can make geospatial data pipelines and services faster. (How? Magick?)

The onion analogy in modern day software development

When computers were built out of punched cards, I imagine it would have been easy to evaluate the state and processes in a machine. It might have been easier to debug as the onion bulb was tender and small and everything was binary. The onion bulb representing how transparent the “software” is.

As time progressed, people started programming on Assembly, and then C was developed which could compile English into assembly code and then machine code. Thus there were peels/layers on the onion bulb and people only saw and interacted with the exterior layer.

Now when someone needs to use an old C library, they write language bindings on their favourite Programming Language which calls C functions, and then there were more layers on the onion bulb.

So what goes under the hood – “rio_tiler.tile(x, y, z)”

An example is how we “typically” work on an old scientific library like GDAL. Each layer is an abstraction over the subsequent layer in this chain

rio-tiler -> rasterio -> GDAL-py -> GDAL-cpp (base library)

My software would wrap rio-tiler. So I’d attach the ‘5th’ peel to the onion bulb.

I don’t want to preach something I’m unqualified for. But abstraction is a necessary evil. If something wasn’t abstracted, it wouldn’t get adopted a lot and the library would server a lesser purpose than it could have with abstractions.

Coming back to konan and why am I working on it?

konan is a dynamic raster tile server written in Golang which roughly behaves like tegola but it generates png tiles unlike tegola which generates MVTs. Here are some of its features

1. User gets a binary to run (more on this later)
2. They can pass link to geotiffs to serve tiles for
3. konan dynamically reads bytes from this geotiff as requested from the map-frontend.

This would serve 2 major purposes:

1. If we can give the user a static binary to run on their platform, it’d be very nifty for one off use cases. (Say you wish to ship a minimal tile-server for your native application, but don’t want to write build scripts to install GDAL).

2. Golang’s concurrency patterns can be used to write performant tile servers.

An anonymous angel lukeroth wrote Go bindings for GDAL here, but unfortunately there are very few references to use this library. JSYK, the bindings are as good as they can be, it’s just people don’t think Golang when they want to work with Geospatial raster data.

This library coupled with some existing pure Python (Well, using “gdal-py”) implementations of “dynamic tile servers”/“tile generators” like gdal2tiles.py, terracotta and lukeroth/gdal/examples/ helped me with enough context to start writing konan.

Why tell us when it is neither ready nor open source?

I’ve written an alpha version of konan, but I’m stuck.

As I was saying, one of the purposes of building konan was to generate “static binaries”. This seemed simple at first:

go build ?

First of all, this won’t work because we’re using CGo indirectly while using lukeroth/gdal. So the environment variable CGO_ENABLED=1 should be set.

Okay, so the following code generates a static binary?

env CGO_ENABLED=1 go build

No, by default Go builds static binaries BUT not when CGo is used. So the command above generates a binary which is dynamically linked to dependencies, so a user would still have to install gdal-bin.

I searched around the internet and found that to generate static binaries for a system:

CGO_ENABLED=1 go build -ldflags ‘-linknode external -extldflags “-static”’

I see complaints from ld along the lines of:

/usr/bin/ld: (.text+0x1a24): undefined reference to `operator delete(void*, unsigned long)’

With the shallow knowledge I have of C/C++ environment, this looks like an issue with linker not finding the definitions when it can read the dependency function declarations. This is happening for dependencies of libgdal.so.

I tried switching my compiler to musl hoping to see some “magik”, but didn’t have any success.

As I run out of my depth here, I’m starting on a journey, to understand how dependencies, source code, and distributions work in C/C++ codebases, eventually to generate a static binary for konan.

It might be practically impossible, but I’ll dig a mine if I have to, to make konan free of dependencies. My next blogpost should be along the lines of “I thank my wife and parents to be by my side as I generated cross-platform binaries for a useless tile server…”. If not, know that I tried.

What’s in the name?

Konan is a character in Naruto whose superpower is to generate infinite paper tiles , thus very worthy of being called the ultimate tile-server :)