OSGeo, GDAL and other GIS fun. Find me on Mastodon , GitHub or on my company Spatialys website
Every domain has its baggage of concepts, which at first sight don't appear to be that terrible, but which are in practice. Let's start with a non-geospatial example. A lot of programming languages have the concept of a "null" pointer. It is available in the C and C++ programming languages in particular, but in Java as well, or Python, etc.. Tony Hoare, null's creator, regrets its invention: "I…
All this story started with this QGIS bug about a crash when a user adds a new field to a GeoJSON file. Who is the culprit between GDAL and QGIS could be debated. Let's look at the GDAL side of things. Let's consider this simple GeoJSON file: { "type": "FeatureCollection", "features": [ { "type": "Feature", "geometry": null, "properties": { "A": "a", "C": "c" }, { "type": "Feature", "geometry":…
(This is a post I started on December 2019 and didn't publish because I felt it was too negative. But recent events show that it is still a current topic, and at least this will document my own preception of things) We have lately stumbled upon a few clumsy attempts of corporations at contributing to open source software. Needless to say this is a bumpy ride not for the faint of heart, from both…
GDAL has now been put under the continuous scrutinity of OSS-Fuzz for more than 4 years. To keep it simple, OSS-Fuzz is a continuous running infrastructure that stresses a software with (not-so-)random data to discover various flaws, and automatically files issues in a dedicated issue tracker, with reproducer test cases and stack traces when available. It is time to use a bit the data accumulated…
Those past days, I've experimented with Docker to be able to produce "official" nightly builds of GDAL Docker Hub is supposed to have an automated build mechanism, but the cloud resources they put behind that feature seem to be insufficient to sustain the demand and builds tend to drag forever. Hence I decided to setup a local cron job to refresh my images and push them. Of course, as there are…
This is the 8th progress report of the GDAL SRS barn effort. As the title implies, a decisive milestone has now been reached, with the "gdalbarn" branches of libgeotiff and GDAL having been now merged in their respective master branch. On the PROJ side, a number of fixes and enhancements have been done: - missing documentation for a few functions, the evolution of cs2cs and the new projinfo…
This is the 7th progress report of the GDAL SRS barn effort. On the PROJ side, things are consolidating up. Tens of "random" fixes have been pushed due to the GDAL autotest suite triggering a number of interesting cases. The C API has also been enhanced to accommodate for the needs of GDAL and libgeotiff. We also have received feedback from an early adopter (a developer of a pyproj binding based…
This is the sixth progress report of the GDAL SRS barn effort. The pace of changes has not yet slow down, with still a significant part of the work being in PROJ, and an initial integration in GDAL. The major news item is that RFC2 , implementing the new capabilities (WKT-2 support, late-binding approach, SQLite database), has now been merged into PROJ master. An initial integration of PROJ master…
This is the fifth progress report of the GDAL SRS barn effort. A lot of activity in PROJ developments has been done this month again. New conversion and transformation methods referenced in the EPSG database have been integrated: Projection methods: Lambert Conic Conformal (2SP Michigan) and Laborde Oblique Mercator "Change of Vertical Unit", "Vertical offset" and other fixes relative to vertical…
This is the fourth progress report of the GDAL SRS barn effort. The in-progess task at the time of the last report was the addition of a method createFromPROJString() that takes a PROJ string and builds ISO-19111 corresponding objects. It has now been completed for ProjectedCRS. Not all arbitrary valid PROJ strings can be currently mapped to ISO-19111 objects, but at least strings expressing CRS…
This is the third progress report of the GDAL SRS barn effort. In the last month, the main task was to continue, and finish, the mapping of all GDAL currently supported projection methods between their different representations as WKT 2, WKT 1 and PROJ string: LCC_2SP, LCC_2SP_Belgium, Modified Azimuthal Equidistant, Guam Projection, Bonne, (Lambert) Cylindrical Equal Area,…
This is the second progress report of the GDAL SRS barn effort. Since the previous report , a number of topics have been addressed: - extension of the class hierarchy to implement BoundCRS (the generalization of the WKT1 TOWGS84 concept. This concept only exists in WKT 2 and has not been modeled in ISO-19111, so I went on my own modelling), TimeCRS, DerivedGeodeticCRS - implementation of the…
Thanks to the support given by the sponsors of the GDAL SRS barn effort, I have been able to kick in the first works in the past weeks. The work up to now has been concentrated on the PROJ front. The first step was to set a foundation of C++ classes that implement the ISO-19111 / OGC Topic 2 "Referencing by coordinates" standard. Actually I have anticipated the future adoption of the 18-005r1 2018…
Those past couple days, I was working on implementing multi-layer transaction support for GeoPackage datasources (for QGIS 3.4). Multi-layer transaction is an advanced functionality of QGIS (you have to enable it in project settings), initially implemented for PostgreSQL connections where several layers can be edited together so as to have atomic modifications when editing them. Modifications are…
Over this summer I have spent 40 days (*) in the guts of the OpenJPEG open-source library (BSD 2-clause licensed) optimizing the decoding speed and memory consumption. The result of this work is now available in the OpenJPEG 2.3.0 release . For those who are not familiar with J PEG-2000 , and they have a lot of excuse given its complexity, this is a standard for image compression, that supports…
In the past weeks, a number of improvements related to access to cloud storage have been committed to GDAL trunk (future GDAL 2.3.0) Cloud based virtual file systems There was already support to access private data in Amazon S3 buckets through the /vsis3/ virtual file system (VFS). Besides a few robustness fixes, a few new capabilities have been added, like creation and deletion of directories…
Recently, I've fixed a bug in the OGR OpenFileGDB driver , the driver made from the reverse engineering the ESRI FileGeoDatabase format, so as to be able to read tables whose section that enumerates and describes fields is located beyond the first 4 GB of the file. This table from the 2016 TIGER database is indeed featuring all linear edges of the USA and is 15 GB large (feature and spatial…
Note for geospatial focused readers: this article has little to do with geo, although it is applied to GDAL, but more with software virtualization, hacks, software archeology and the value of free software. Note for virtualization experts: I'm not one, so please bear with my approximate language and inaccuracies. Travis-CI is a popular continuous integration platform, that can be easily used with…
GDAL offers a method ComputeStatistics() that given a raster band returns the minimum and maximum values of pixels, the mean value and the standard deviation. For those not remembering how to compute mean and standard deviations, the basic formulas for values indexed from 0 to N-1 are : mean = sum(value(i) for i = 0 to N-1) / N std_dev = square root of the mean of the square of the differences of…
On behalf of the GDAL/OGR development team and community, I am pleased to announce the release of GDAL/OGR 2.1.0. GDAL/OGR is a C++ geospatial data access library for raster and vector file formats, databases and web services. It includes bindings for several languages, and a variety of command line tools. The 2.1.0 release is a major new feature release with the following highlights: RFC 26: Add…
While my memories are still fresh, here is a report of this week of code sprinting . First, a big thanks to Olivier Courtin for organizing this event, to all sponsors that brought up the money to make it happen and to the Mozilla Foundation for hosting us in the most scenic coding venue I've ever seen. As expected, I mostly concentrated on GDAL work. My main tasks were related to polishing and…
As a new year is popping up, it is time to take good resolutions. To help you, especially if you are a C/C++ developer, this article gives feedback on efforts made over the last few months to improve the quality of the GDAL/OGR code base, and hopefully its quality for the end user. Enable as many compiler warning options as possible By default, C/C++ compilers enable a few warning categories, but…
It has been often stated that the popular GDAL/OGR C/C++ command line utilities, such as gdal_translate , gdalwarp , ogr2ogr , etc... are mostly examples of how to use the GDAL/OGR API for your own purposes. The truth is that many people just need their functionnality without much additional tuning. Furthermore, over time those utilities have gained a lot of options and tweaks which make it time…
On behalf of the GDAL/OGR development team, I am pleased to announce the release of the GDAL/OGR 1.11.3 and 2.0.1 bug fix versions. Each one contains more than 40 bug fixes since 1.11.2 / 2.0.0. The sources for 1.11.3 are available at: http://download.osgeo.org/gdal/1.11.3/gdal-1.11.3.tar.xz http://download.osgeo.org/gdal/1.11.3/gdal-1.11.3.tar.gz http://download.osgeo.org/gdal/1.11.3/gdal1113.zip…
As the race to increase processor speed has stalled during the last decade, CPU manufacturors have rather increased the number of transistors and parallel execution units within a single CPU to keep up with Moore's law. Hence the multi-cores, hyper-threading technologies. This is all good but contrary to extra MHz, software has to be explicitely designed to deal with those parallelism mechanisms,…