By: Robin Kraft
Robin is a remote sensing professional with an environmental concentrate. He was a co-founder of Global Forest Check out, tracking tropical deforestation in in the vicinity of authentic time working with satellite imagery. He led imagery analytics efforts at Planet, and is a product or service manager at the information science startup Domino Data Lab.
Throughout a disaster, reputable info is unbearably rare. I discovered this the difficult way through the October wildfires in Northern California that threatened my hometown. As authorities, regional news, and persons on social media raced to share updates about where the fires had been spreading, the inconsistent info they provided reflected the chaos and uncertainty on the floor. My family members and close friends wanted a clearer picture.
🚨 There is an additional fire raging in Los Angeles appropriate now — if DigitalGlobe and Planet launch their information, you can use this guide to make your very own map. 🚨
I noticed an chance to support when DigitalGlobe and Planet begun releasing superior-resolution satellite imagery of the location through their open information plans. The imagery lined most of the affected region and was reasonably distinct, but it was locked inside of a several hundred gigabytes of GeoTIFF files — not considerably use if you escaped the fires with just your clothing and a smartphone. I set out to build something that much more persons could use.
You can test out the reside map here.
I selected to build with Mapbox since I wanted a system impressive more than enough to approach and host the imagery documents, but also nimble more than enough to make a map that would load quickly and be intuitive to use. In this write-up I’ll explain the very simple, manual workflow that I employed to publish an original map of the fires as swiftly as possible. I’ll then dive into optimizations I added to update the map and enhance the person practical experience.
The easy way
The 1st time all over, I downloaded a several illustrations or photos, stitched them together with gdal_merge.py, and uploaded the mosaic to Mapbox Studio. Once tiling was completed, I added my new tileset to a new design and style dependent on Mapbox’s Satellite Streets template and clicked publish. This bought me a quickly, cell-welcoming map in fewer than an hour, from locating the imagery to texting the URL to a pal. When my pal questioned why every thing seemed purple, I overloaded the title to incorporate a pretty transient explanation of phony coloration imagery. No coding, no styling — sensible defaults for the win!
But this approach would not scale. As much more and much more imagery grew to become readily available and targeted traffic to the map begun creeping up, I understood I wanted a better way to do updates.
The better way
In the conclude, I employed largely the similar equipment for preprocessing: GDAL and Mapbox Studio. But I also employed the Mapbox Uploads API and Amazon World wide web Services’ EC2 and S3 products and services. My code is readily available on Github here.
Functioning in Amazon’s cloud setting manufactured a several items considerably better. First, DigitalGlobe and Mapbox operate on AWS by now, and Planet is on Google Cloud — which has a quickly pipe to AWS. Functioning within just AWS would conserve me a lot of time on information transfer. Next, on my Ubuntu occasion I employed monitor to retain processing likely even if my notebook was off. I did not want to have to depend on a continuous net connection for a approach that could most likely get an overall day to full. You can test out this script to see how I set up my dependencies and prepped extra EBS storage.
The other major enhancement was the economical use of GDAL. This was important as information volumes exploded when I expanded the region lined by the map and new imagery grew to become readily available. Naively mosaicking and projecting hundreds of superior-resolution, 1-gigabyte illustrations or photos together working with gdal_merge.py was having much too prolonged. Also, Mapbox has a 10GB restrict so I couldn’t add the giant GeoTIFFs if I’d tried.
In the conclude, I relied on digital rasters (VRTs) and TIFF compression. VRTs are terrific since they are textual content documents describing processing actions they really do not essentially use the functions until finally you convert the VRT to a “proper” graphic file, like a GeoTiff. This “lazy” processing approach simplified my workflow so that I did not have to deal with a number of prolonged-jogging procedures or hundreds of files.
Step by step
My 1st procedure was to make a single mosaic of all the documents for a single day — which may well be 50 documents. Include the filenames to a textual content file, and you’re great to go!
ls *.tif > documents.txt
gdalbuildvrt -enter_file_checklist documents.txt mosaic.vrt
“Mosaicking” completed in < 1 second! This isn’t a “real” mosaic yet, but GDAL treats it as one thanks to the VRT format.
Then, I wanted to save Mapbox some work and project the mosaic from its initial WGS84 CRS to Web Mercator — the standard map tile projection — per the docs. Otherwise, this would have to be done on Mapbox’s end while it created the tileset, and I was already seeing some timeout errors.
gdalwarp -s_srs EPSG:4326 -t_srs EPSG:3857 -of vrt -overwrite mosaic.vrt mosaic_webmerc.vrt
“Reprojecting” a massive “mosaic” in < 1 second? Excellent!
Finally, I had to create a real GeoTIFF I could upload to Mapbox. Only at this point do the mosaicking and reprojection actually occur. I again followed Mapbox suggestions for optimizing these images and threw in a few compression flags I learned about on StackOverflow.
The most important optimization for the purposes of processing time and file size was using internal JPEG compression. JPEG compression is lossy, in that it throws away information in the image (mostly stuff that our eyes are less likely to notice). Normally for remote sensing that’s not ideal, but given the disaster, a bit of fuzziness at zoom level 22 for faster processing seemed like a reasonable tradeoff.
I also instructed GDAL to use a YCbCr photometric interpretation. I don’t really understand it, but apparently, it makes JPEG compression more efficient. A 50 cm resolution mosaic covering about 800 square miles came to about 3 gigs — very manageable.
gdal_translate -co tiled=yes -co bigtiff=yes -co compress=jpeg -co photometric=ycbcr -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 -of Gtiff mosaic_webmerc.vrt mosaic_final.tif
This step took 15 minutes or so, instead of many hours.
Now I needed to get the GeoTiff over to Mapbox for final processing. I wrote a simple uploader script using the Mapbox Python SDK that would get the tiling process started. The SDK handles authentication and request formatting for you. You just provide the data and the tileset name, upload the file, and start the tiling. Here’s the bare minimum:
from mapbox import Uploader
u = Uploader()
tileset = 'username.tileset_name' # yes, you must include your username
url = u.stage(open('mosaic_final.tif')) # upload happens here
u.create(url, tileset, name=fname) # starts the tiling job
A better map experience
The standard “shareable” URL for a Mapbox map style is super convenient. But I learned three ways to optimize it and my map.
First, the default URL bypasses caching. This makes sense given that you typically use this to preview a map you’re working on. But it’s really bad if your map goes viral. If you’re satisfied using the default map, do yourself a favor and tweak the URL you share: change one of the URL parameters from “fresh=true” to “fresh=false”. Even with caching turned on, the map will update within a few minutes if you change something, and you won’t overload the servers.
Second, the default view that appears when you share a map on social media is the extent of all the data. The Mapbox Satellite Streets layer is global the default view covers Null Island and Africa.
You can control this in Studio using the Map Position tool by clicking “unlock”, then browse to the default view you like. Lock it again and re-publish.
Third, the default map for a style is great, but there’s no address search or much else. In the case of the fire map, the content was what really mattered, and this default map got the majority of views. But I wanted a better user experience, so I needed to embed the map in my own website.
I used Github Pages, which can handle a lot of traffic to a static website. Embedding the map in a website gives you more freedom to add layers to the map and other map controls. With the help of the Mapbox team and a few tutorials, I added address search and a comparison slider so that people could compare the fresh imagery to the standard Mapbox Satellite layer, which is older. It’s shocking to see what the devastated neighborhoods looked like before and after the fire.
body[data-twttr-rendered=”true”] background-color: transparent.twitter-tweet margin: auto !important
Satellite imagery shows destruction of Santa Rosa fire: https://t.co/FdGJCyAFek By @robinkraft🛰️ open data @DigitalGlobe #santarosafire 🙏 https://t.co/JUmBoMF3N5
function notifyResize(height) height = height ? height : document.documentElement.offsetHeight var resized = false if (window.donkey && donkey.resize) donkey.resize(height) resized = trueif (parent && parent._resizeIframe) var obj = iframe: window.frameElement, height: height parent._resizeIframe(obj) resized = trueif (window.location && window.location.hash === “#amp=1” && window.parent && window.parent.postMessage) window.parent.postMessage(sentinel: “amp”, type: “embed-size”, height: height, “*”)if (window.webkit && window.webkit.messageHandlers && window.webkit.messageHandlers.resize) window.webkit.messageHandlers.resize.postMessage(height) resized = truereturn resizedtwttr.events.bind(‘rendered’, function (event) notifyResize()) twttr.events.bind(‘resize’, function (event) notifyResize())if (parent && parent._resizeIframe) var maxWidth = parseInt(window.frameElement.getAttribute(“width”)) if ( 500 < maxWidth) window.frameElement.setAttribute("width", "500")
Check out the source code for the embedded map page on Github. It also shows how to add Open Graph tags to control what appears on social media sites.
I’ve been working with satellite data and maps for a long time because I believe in the power of maps and geospatial data. But this disaster made me understand viscerally how powerful they can be. Using this imagery, I was fortunate to see my family home and wedding site intact.
Many others were not so fortunate. I was surprised that so much of the community found solace in this map, particularly since it put an intense loss on display for the world to see. But it turns out that it helped a lot of people move on. As one person wrote to me, “There are many homeowners in evacuated zones that have no idea whether their homes are still standing. This puts the power and knowledge back into their hands. It’s just awful waiting in suspense.”
My family, friends, and community are all grateful to Mapbox, DigitalGlobe, and Planet for their support during this disaster. Things like geocoded address searches, mobile-friendly maps, and rapid imagery updates are standard fare for those of us working in the geospatial world. But they make a huge difference for people in the midst of disaster.
There is now a fire raging in Los Angeles. Hopefully, DigitalGlobe and Planet will release their data ASAP, and then you can make the next viral fire map.
Read more about other disaster response organizations using our tools and explore our map design tutorials to inspire your next project.
Robin Kraft (@robinkraft) | Twitter
Santa Rosa fire map: How I built it was originally published in Points of interest on Medium, where people are continuing the conversation by highlighting and responding to this story.