# items

guid: tag:blogger.com,1999:blog-2082422783692631078.post-5872632875751805027
url: http://sk53-osm.blogspot.com/2018/07/coda-on-shop-completion-rates-on-osm.html
title: Coda on shop completion rates on OSM
updated: 2018-07-25 14:21:33.017000
published: 2018-07-25 14:21:00
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.603168
updated_at: 2019-10-26 08:30:24.875501
content:
  Thanks to John Baker (<a href="https://www.openstreetmap.org/user/Rovastar">Rovastar</a>) for a few suggestions discussing my <a href="http://sk53-osm.blogspot.com/2018/07/can-we-identify-completeness-of.html">recent blog</a> post in the <a href="https://wiki.openstreetmap.org/wiki/Nottingham/Pub_Meetup">pub</a> last night:<br /><br /><ul><li>What do the graphs of numbers of unique shop tags look like with heavier filtering of relatively poorly used tags.</li><li>E-cigarette shops are a recent phenomenon, and should represent a genuinely novel tag rather than the mix of typos, synonyms etc which characterise much of the long tail of shop tags. </li></ul><br />These were easy to follow up, so I present the graphs here:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://4.bp.blogspot.com/-J5iLbBcpDug/W1iC91-GsbI/AAAAAAAABJU/BXirGIeb_LIh8K5TetBUaKNhP79-M1X3ACLcBGAs/s1600/shop_tags_dist.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://4.bp.blogspot.com/-J5iLbBcpDug/W1iC91-GsbI/AAAAAAAABJU/BXirGIeb_LIh8K5TetBUaKNhP79-M1X3ACLcBGAs/s320/shop_tags_dist.png" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Unique shop tags over time on OpenStreetMap for Great Britain, <br />filtered to remove tags with a restricted number of uses as at June 2017.</td></tr></tbody></table>For virtually any level of filtering the curves level out around 2010-2011. Thus the core set of shop tags looks to be very stable. A good place to judge the extent of likely synonymy for shops in Britain is the LUA script used by <a href="https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/style.lua#L3095">SomeoneElse</a> for his "Useful Maps".<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-fzIRwj91ujs/W1iEaYMjAbI/AAAAAAAABJg/nlZn-vjmsTkPbg_1zxu9vNMY0jE2IS7WwCLcBGAs/s1600/shop_ecig.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="640" src="https://2.bp.blogspot.com/-fzIRwj91ujs/W1iEaYMjAbI/AAAAAAAABJg/nlZn-vjmsTkPbg_1zxu9vNMY0jE2IS7WwCLcBGAs/s320/shop_ecig.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Growth of mapped e-cigarette shops in GB on OSM</td></tr></tbody></table>As expected e-cigarette shops first appeared rather late, at the end of 2013, and there are a decent number mapped (over 200 by mid 2017). I haven't checked, but I suspect the sharp increase in 2017 was caused by some tagging rationalisation. It's not unusal for new things to acquire a range of synonyms before tagging stabilises and one value becomes favoured. (It's equally true that in some cases this does not happen).<br /><br />I've had a couple of other requests which it will take rather longer to look at, but if you have ideas relating to shops in Great Britain I can look at the data right now.<br /><br />


guid: tag:blogger.com,1999:blog-2082422783692631078.post-7824444007916416702
url: http://sk53-osm.blogspot.com/2018/07/can-we-identify-completeness-of.html
title: Can we identify 'completeness' of OpenStreetMap features from the data?
updated: 2018-07-24 15:35:24.510000
published: 2018-07-24 15:01:00.003000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.605850
updated_at: 2019-10-26 08:30:24.878100
content:
  At the Milan SotM conference <a href="https://twitter.com/sfkeller">Stefan Keller</a> from the <a href="https://www.ifs.hsr.ch/index.php?id=12520">Geometalab</a> at HSR (Rapperswil) will <a href="https://2018.stateofthemap.org/2018/A30-Areas-of-Interest_for_OpenStreetMap_with_Big_Spatial_Data_Analytics_/">talk</a> about recent work of his group on identifying "Areas of Interest" (<b>AoI</b>) from OpenStreetMap data. Stefan has been kind enough to involve me in some discussions about this work as it has progressed, but in this post I am solely concerned with a separate issue arising from the use of points of interest in this work.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-ADmygsg9NkU/W1YVDm_VNFI/AAAAAAAABIE/sLTBI2UsiQ4Ffc4fytakW5bkLCfuv5ehgCLcBGAs/s1600/shop_time_by_la3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://2.bp.blogspot.com/-ADmygsg9NkU/W1YVDm_VNFI/AAAAAAAABIE/sLTBI2UsiQ4Ffc4fytakW5bkLCfuv5ehgCLcBGAs/s640/shop_time_by_la3.png" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Growth of shops mapped on OSM for selected Local Authorities<br />(See Analysis section below for commentary)</td></tr></tbody></table><br /><br />Areas of Interest were <a href="https://blog.google/products/maps/discover-action-around-you-with-updated/">introduced</a> on Google Maps back in 2016. Loosely they correspond to shopping, entertainment and cultural areas with large clusters of relevant points of interest. No doubt Google not only used map features, but also other sources of data such as location of Android phones to calculate the footprints for Areas of Interest (shown in a pale orange or salmon colour on Google Maps).<br /><br />There are issues with the Google implementation, some discussed in this <a href="https://www.citylab.com/design/2016/08/google-maps-areas-of-interest/493670/">CityLab article</a> from 2016. My own examination of Google Maps confirms that shopping areas which are otherwise equivalent in range and type of shops are chosen as AoI in wealthy areas, but not in poorer areas dominated by social housing. I also found some places, notably the UBS IT centre in Altstetten, Zurich, which have erroneously been identified as AoI by Google. The work of Geometalab is therefore interesting not just in terms of whether OSM data can be used to calculate similar areas, but also to provide suitable data where biases based on socioeconomic status can, at least, be identified and corrected because data and code are open.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-OyL_umwZ_5g/W1cUYXa3iPI/AAAAAAAABI8/4VVwrWE8XO8RUh1kPj06pB9SsXBW7iLfQCLcBGAs/s1600/Screenshot%2B%25282%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="1138" height="252" src="https://1.bp.blogspot.com/-OyL_umwZ_5g/W1cUYXa3iPI/AAAAAAAABI8/4VVwrWE8XO8RUh1kPj06pB9SsXBW7iLfQCLcBGAs/s400/Screenshot%2B%25282%2529.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Zurich, centre and Aussersihl districts, showing Areas of Interest.<br />Work of Geometalab, derived from OpenStreetMap data.</td></tr></tbody></table>The starting point for this type of work relies on areas where POI mapping density is high and reasonably complete (for instance, the areas of Switzerland which Stefan's group have looked at, and areas  of the English <a href="http://sk53-osm.blogspot.com/2013/06/the-shopping-news-mapping-retail.html">East Midlands</a> and Germany which I have looked at  both recently, and <a href="http://sk53-osm.blogspot.com/2013/04/segmentation-of-retail-landuse-why-do.html">in the past</a>). Given that it is possible to calculate reasonable AoIs from OSM data where PoI density is high, the question arises "Can we identify which areas are 'reasonably' complete?". Normally, this type of work has involved comparing OSM data to some external reference data which are assumed for the purposes of comparison to be complete (for instance Peter Reed's <a href="http://tlatet.blogspot.co.uk/2015/08/osm-retail-survey-conclusions-1.html">work</a> on UK retail). However, in many parts of the world, and for many topic domains there is no readily usable data for this purpose. So the ancillary clause for the question is ", and we do this with OSM data alone?"<br /><br />This post is a first look at the problem for one class of POIs:&nbsp; shops.<br /><br /><br /><a name='more'></a><br /><br /><h4>Species Accumulation Curves</h4>My starting point comes from familiarity with something called a <a href="https://en.wikipedia.org/wiki/Species_discovery_curve">Species Accumulation Curve</a>. I believe that there are strong points of commonality between how OSM data is accrued and these curves.<br /><br />For many groups of plants, animals, and other biota, it is nigh on impossible to find, in a single survey, all the different species which grow or live in a particular area. Numerous factors influence this:<br /><ul><li><b>Surveyors' skills.</b> Not every surveyor has the same skill set, training, or even just visual acuity. One of the best naturalists I know is a care worker, who can trump national and international scientific authorities by finding more species than they can in the field.</li><li><b>Seasonality</b>. Plants flower at different times, birds migrate, some insects are on the wing for a short time.</li><li><b>Weather</b>. The hot dry weather in Britain has greatly reduced the number of flowers I have seen in the past few weeks, and consequently their insect visitors. On Sunday I was heartened to lead a field meeting where we found 44 species in our target group; but 10 years ago in the same location &amp; at the same time of year we found nigh on 30 more.</li><li><b>Predator Prey</b> relationships. Many species numbers go in cycles (for instance Lemming years), but at least for some insects population density has been estimated to be an order of 10^12 between the troughs and the peaks. Ideally one surveys through 2-3 full cycles: problematic if they are 17-year cicadas, or bamboos which flower and die on a 70-year cycle.</li><li><b>Increasing knowledge</b>. Sharing of techniques for searching or recognising different plants and animals can have an amazing influence on total numbers of species found. This is true even in Britain for as well studied a group as the higher plants. The BSBI's Atlas 2020 project which will be completed in 3 years time, will not only show changes in plant distribution brought about by agricultural intensification, increased urbanisation and climate change, but also changes from looking more closely for a wider range of plants (notably urban weeds and garden escapes).</li><li><b>Sheer cussedness.</b> Fungi are particularly awkward customers. Most spend their time invisibly underground, only showing fleetingly as fruiting bodies (mushrooms) when they feel the time and weather is right. Even with the most capable surveyors in the world the full extent of species complexity can only be appreciated by continual regular surveying of the same place. There are two locations in England which demonstrate this point. Esher Commons have been regularly surveyed for fungi by scientists, including global authorities on some fungal groups,&nbsp; from Kew Gardens for many years. No other place <a href="http://www.brc.ac.uk/psl/content/esher-commons"><b>in the world</b></a> is know to have as many fungi, and around 20% of all fungi known in the British Isles have been found at Esher. <a href="https://www.slnnr.org.uk/wildlife.aspx">Slapton Ley</a> in Devon has also received decades of regular surveying effort for fungi, and has over 2000 known fungi species. It may be come second to Esher for known fungal diversity!</li><li><b>Recorder effects</b>. Even for professional scientists it is often difficult to maintain a constant recording effort. Most biological data is gathered by citizen scientists who can only devote what leisure time they can spare for the activity. Recorders tend to be located in larger cities, rather than in potentially highly species diverse remote areas. For many species groups only a few people are seriously dedicated (as in my own interest in Plant Galls).</li></ul>The <b>key advantage</b> of species accumulation curves is that, whilst not impervious to these effects, they are a relatively robust measure. For my fungi-loving friends they are a useful tool to work out when to move on from one area to another. At the scientific level the curves are well studied and there is a good framework of statistical techniques for analysing them.<br /><br /><h3>Differences between OpenStreetMap &amp; Biological Record Data</h3><br />Data collection for biological recording differs from that for OSM in one particularly important aspect. For biological records all observations in each survey count. In OSM every repeat observation of the same POI (pub, shop, restaurant) is never collected. This means we can make no use of properties of each individual survey activity which contributed POIs. Also there is absolutely no equivalent of an OSM import for biological records. I've focused on Great Britain, so the latter has little impact on the results I present below.<br /><br />We can still look at two types of accumulation: <br /><ul><li>of individual shops;</li><li>of shop tags.</li></ul>Note that empty shops are meaningful data in the first grouping.<br /><br /><h3>Null Hypothesis</h3><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-EvvROuRMqS4/W1c1snUBhCI/AAAAAAAABJI/b85PhEIQ4ksW_-qReCZXkMrbg0WLmudcgCLcBGAs/s1600/IMG_8359a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="390" src="https://3.bp.blogspot.com/-EvvROuRMqS4/W1c1snUBhCI/AAAAAAAABJI/b85PhEIQ4ksW_-qReCZXkMrbg0WLmudcgCLcBGAs/s320/IMG_8359a.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Typical small shops which get mapped piecemeal on OSM: <br />near Conde de Casal Metro station<br />Av. Mediterraneo, Madrid, Nov. 2016</td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><br />My null hypothesis is that over time we should see the number of shops on OSM for a given area tailing off towards an asymptote once the area is well-mapped. I know that surveys I did in the Spring of 2013 changed the percentage of shops mapped in Nottingham from around 40% towards 90%. Even earlier <a href="https://twitter.com/JLZIMMERMANN">Jean-Louis Zimmerman</a> and <a href="https://twitter.com/tonyemery">Tony Emery</a> had mapped <a href="https://wiki.openstreetmap.org/wiki/Orange_(Vaucluse)#Plans_de_Ville">Orange</a> in great detail and a map of the town was published. I therefore took these two towns and a few others to see if this was plausible. Data were gathered point by point using Overpass-turbo, and plotted in LibreOffice.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-xfzBNucaK9g/W04JSM_hEuI/AAAAAAAABHY/0iF1j7mrh_MLrPn7oPvLf8cTAiBOfz3PgCLcBGAs/s1600/shop_graph2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="287" data-original-width="688" height="166" src="https://1.bp.blogspot.com/-xfzBNucaK9g/W04JSM_hEuI/AAAAAAAABHY/0iF1j7mrh_MLrPn7oPvLf8cTAiBOfz3PgCLcBGAs/s400/shop_graph2.png" width="400" /></a></div><br />The two towns (Nottingham &amp; Orange) where I had good reason to believe that shops were reasonably complete some years ago formed a baseline and did appear to show curves with asymptotic properties.<br /><br />The remaining places I chose on the basis that I knew that they are well-mapped, but without knowing if retail properties had been mapped to completion. I thought that it was plausible that this would prove to be true for Zurich &amp; Karlsruhe, certainly not for Madrid, and likely not for Dakar. San Francisco was chosen as a well-mapped location in North America. In practice none of the graphs for number of shops over times suggests that effort to map shops has reached an inflexion. Even for somewhere like Karlsruhe which has had active mappers for as long as anywhere the graph suggests that there is still scope for mapping shops.<br /><br />Gathering data point-by-point is fine for a quick test, but far too tedious (&amp; expensive in use of free resources), so my next step was to wrestle with OSM History files.<br /><br /><h3>Extracting OSM History data</h3>I already had an OSM History file for Great Britain for June 2017 downloaded from Geofabrik. Unfortunately, these files do not appear to have been updated since Geofabrik changed the user metadata available on their public servers. Also because history files contain user information protected by GDPR, these files are now only available through using an OSM sign-on now.<br /><br />Manipulating history files effectively means either using the command line <a href="https://osmcode.org/osmium-tool/">osmium tool</a> or writing programs using osmium library. This in turns means installing osmium. I therefore did this under Ubuntu 16.04. There is a packaged version of osmium for Ubuntu, but it is ancient, so it is necessary to compile and install the current version 1.8.<br /><br />Osmium is very much designed for heavy duty sophisticated processing of OSM data. It's not really a toolset for quick-and-dirty <i>ad hoc</i> investigations of the kind I do. I was apprehensive about getting tied up in knots getting the Osmium tool installed, particularly when I read the list of <a href="https://github.com/osmcode/osmium-tool/blob/master/README.md">dependencies</a>.<br /><br />In practice the only problem I had initially was due to not cloning a couple of packages into the right location in my osmium build directories. As I've never used Cmake in my life I was certainly intimidated by the simple statement "Please read the CMake documentation and get familiar with the <code>cmake</code> and <code>ccmake</code> tools which have many more options.". However, reassured by Richard Fairhurst &amp; Andy Townsend that it wasn't too difficult to install I preserved and soon had it installed. One thing I would have found helpful would have been an outline of the directory tree for a build.<br /><br />I also had a 5 minute attempt to compile Peter Mazdermind's <a href="https://github.com/MaZderMind/osm-history-renderer">OSM History</a> tool, but this has not been maintained and uses very ancient versions of osmium, so I did not preserver.<br /><br />The key reason for using the 1.8 version is that it has better support for extracting dependent data. Thus in a two step process it is possible to filter a history file for all elements tagged with shop and then find all their dependent elements. This is well covered in the osmium manual.<br /><br />For pragmatic reasons I chose the One Per Line (OPL) format, as I could very quickly load this data "as is" into a Postgres database.<br /><h3>Wrangling the Data</h3>As I have done for years I loaded the data exactly as it was stored in the source file, so that I could start from the raw data at any time all within Postgres. In practice I loaded nodes, ways &amp; relations with&nbsp; distinct COPY TO statements.<br /><br />I then processed each element type into base tables: transforming the main columns from strings to the proper datatype. Tags involves converting the string into an array separated by commas in the form key, value, key1, value1 .... This in turn converts simply to hstore.<br /><br />For each element the next thing was to calculate the end date for each version and add this to the . This can be done with a window function, or by joining the base table to itself (a left join). (See my very <a href="http://sk53-osm.blogspot.com/2011/05/on-histories-of-openstreetmap-data.html">old post</a> for one way to do this).<br /><br />The major disadvantage of my pragmatic approach is that one has to reassemble geometries, but before one can do that it is necessary to determine the potential number of distinct geometries for each version of a way or relation element. As others have done before me I ignored relations (very few shops are mapped as relations), and just worked with ways. To do this I first found all distinct start dates for all the nodes versions which contributed to any given way element, which can then be treated as minor versions of each way version. I actually prefer the term geometry version. You can see something very similar if you look at the history of a way in Potlatch2.<br /><br />Once I had the start and end dates for each geometry version the linestrings for the way can be assembled by joining the way_node_history, way_geom_history and node_history tables. All ways should produce valid linestrings. By storing the linestrings it is possible to preform multiple checks so that the code only attempts to assemble valid polygons (st_npoints(geom) &gt; 3 and st_isclosed(geom)) worked for me.<br /><br />The shop data is relatively small, under 70k ways, totalling around 125k versions, which expands to 180k geometry versions. For nodes of course versions is the total : 100k elements, 200k versions. Given the data goes back to 2007, the increase in data volume to handle history is very modest.<br /><br />The last thing I did was calculate a centroid for all the data (this can include ways which do not form polygons). All analysis used the shop centroids.<br /><br />It's worth noting that another paper in the SotM academic track by Alexander Zipf's group at Heidelberg may presage much easier analysis by anyone of OSM historical data without the need for this kind of data manipulation.<br /><br /><h3>Analysis</h3><h4>Shops by Local Authority </h4>My starting point was to look at how many shops have been mapped within each local authority across Great Britain. This enables looking at a much more representative sample than the few cities I looked at earlier. There is a disadvantage in that local authorities do not correspond to cities and therefore may not make natural mapping units.<br /><br />A first couple of quick plots show that there is a huge diversity in numbers of shops mapped, when they are mapped, intensity of mapping activity, and so on. When all are plotted together it's difficult to pick out any other trends:<br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-cQHmequL_xA/W04bFMdlmkI/AAAAAAAABHo/RVH9xvbbCyQD1AQUuY_QTctb-nY6qrUTgCLcBGAs/s1600/shop_hist_1stdraft.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="660" data-original-width="1366" height="231" src="https://1.bp.blogspot.com/-cQHmequL_xA/W04bFMdlmkI/AAAAAAAABHo/RVH9xvbbCyQD1AQUuY_QTctb-nY6qrUTgCLcBGAs/s640/shop_hist_1stdraft.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Progress in shop mapping for all Local Authorities in Britain<br />(the top lines with over 2000 shops are : Birmingham, Bristol, Edinburgh, Leeds, Nottingham and London Borough of Westminster)</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-WGiIw1CBPkI/W04bFxVRwCI/AAAAAAAABHw/QrFqKcvhKFcSMkJKZerYb-9TnbOgaXTeQCLcBGAs/s1600/shop_hist_2stdraft.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="660" data-original-width="1366" height="231" src="https://1.bp.blogspot.com/-WGiIw1CBPkI/W04bFxVRwCI/AAAAAAAABHw/QrFqKcvhKFcSMkJKZerYb-9TnbOgaXTeQCLcBGAs/s640/shop_hist_2stdraft.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Percentage of shops mapped at June 2017 in prior months, <br />all Local Authorities Great Britain.</td></tr></tbody></table>If we just look at the raw number of shops mapped, the accretion curve is more or less flat, with no sign of tapering off:<br /><br /><div class="separator" style="clear: both; text-align: center;"></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-zT_UEUD9pKg/W1YpG46ffSI/AAAAAAAABIc/NAZh0pI2Lm44pAv1oP_uMlC1HXogaWU0wCLcBGAs/s1600/shop_time_by_all.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://2.bp.blogspot.com/-zT_UEUD9pKg/W1YpG46ffSI/AAAAAAAABIc/NAZh0pI2Lm44pAv1oP_uMlC1HXogaWU0wCLcBGAs/s400/shop_time_by_all.png" width="520" /></a></div><br />Just looking at the top local authorities can highlight a few other features:<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://4.bp.blogspot.com/-TtKOfoBzTLg/W1Yo7zI0yiI/AAAAAAAABIg/E5clwiwLnUQp8uBXBDaMembCYRL3C-m6wCPcBGAYYCw/s1600/shop_s2000.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://4.bp.blogspot.com/-TtKOfoBzTLg/W1Yo7zI0yiI/AAAAAAAABIg/E5clwiwLnUQp8uBXBDaMembCYRL3C-m6wCPcBGAYYCw/s320/shop_s2000.png" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Local Authorities with more than 2000 shops mapped mid-2017</td></tr></tbody></table><br />Most of these places have seen a fairly steady increase in total shops mapped, but there are a few step changes:<br /><ul><li>Birmingham, 2015: Mattijs Melissen (<a href="https://www.openstreetmap.org/user/Math1985">Math1985</a>)<span class="d-block Box"></span> was a very active shop mapper at this time. The tailing off subsequently may merely because he returned to The Netherlands on completing his post doc.</li><li>Edinburgh, 2014. The <a href="https://www.ed.ac.uk/history-classics-archaeology/research/research-projects/mapping-edinburgh-s-social-history">Edinburgh MESH</a> (social history) project were actively mapping the inner city during this period.</li><li>Nottingham, 2013. My own deliberate attempt to map most shops from around March to June 2013.</li></ul>Clearly spurts of activity such as these are fairly typical of many areas. The extreme is Darlington where virtually all shops on OSM were mapped over at most a couple of months. Pulses of activity may therefore result in curves which are apparently asymptotic, but these only reflect individual mapper activity. I have not included counts of mappers touching elements tagged with shop, but this suggests some such metric may need to be used to avoid false positives from dedicated mapping of shops by single individuals.<br /><br />A much easier way to look at the data is by looking at graphs side by side. I selected all LAs with over 800 shops mapped, which gives a convenient set of 45 different ones. The graphs are at the head of the blog. Out of these 45, only 2 suggest there might be an asymptotic relationship in the data: Nottingham and Tendring. There are plenty more examples of many shops mapped over short periods (e.g., Gateshead, Sefton).<br /><br /><h4>Shops by Tag</h4>We can also look at whether there is any indication that we have mapped a given category of shop to exhaustion. Here are accumulation curves for the top 35 tag values (those with over 1000 elements mapped:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-K0Lbo-kicp4/W1YudO6jKkI/AAAAAAAABIw/Az6vQnc20mMXFJKaeGvAgHB7W9ii_j7xACLcBGAs/s1600/shop_tags_facet_s1000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://4.bp.blogspot.com/-K0Lbo-kicp4/W1YudO6jKkI/AAAAAAAABIw/Az6vQnc20mMXFJKaeGvAgHB7W9ii_j7xACLcBGAs/s400/shop_tags_facet_s1000.png" width="520" /></a></div>Only shop=supermarket and shop=doityourself have any appearance of slowing down, and it would be a long stretch to say they were trending towards a given number.<br /><br />In pretty much all cases the accumulation curves are linear. Thus individual shop tags are much less vulnerable to individual mapper activity. The one step function is shop=bookmaker where Math1985 initiated an effort to reduce the number of synonyms. Slightly worrying is the steady increase in shop=yes.<br /><br />Lastly we can look at the total number of shop tags:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-o0krwnzy8PU/W1YpDpz18nI/AAAAAAAABIk/UjMNlLI9lzkF96jqtBn9dmTnqO3z8gL_gCEwYBhgL/s1600/shop_tags.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://1.bp.blogspot.com/-o0krwnzy8PU/W1YpDpz18nI/AAAAAAAABIk/UjMNlLI9lzkF96jqtBn9dmTnqO3z8gL_gCEwYBhgL/s400/shop_tags.png" width="400" /></a></div>This looks more like the kind of graph I had been hoping to see. The drop around early 2015 was again, no doubt, due to Matt85's rationalisation efforts. It perhaps suggests no more than 2000 tags are needed to map shops in the UK.<br /><br />In <a href="http://sk53-osm.blogspot.com/2015/05/retail-outlets-on-openstreetmap.html">previous work</a> I showed that there is a long tail of very low usage shop tags, and that the presence of these tags is usually in the noise (I was able to assign 98-99% of all shop tags to specific general categories). It struck me that removing this noise may provide a more informative graph. I therefore excluded any shop tag which had been used 5 times or less in June 2017:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-Zk5kwETmxw4/W1Yo_6sTM2I/AAAAAAAABIk/-fS3DToNNDAKKgXq70iF7lHweWpE4TLsgCEwYBhgL/s1600/shop_tags_gt5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://4.bp.blogspot.com/-Zk5kwETmxw4/W1Yo_6sTM2I/AAAAAAAABIk/-fS3DToNNDAKKgXq70iF7lHweWpE4TLsgCEwYBhgL/s320/shop_tags_gt5.png" width="520" /></a></div>Finally, I have the kind of graph I predicted. It actually looks as though 500 shop tags pretty much meets all our tagging needs.<br /><br />Even without trying to fit curves to the rest of the data, it is clear that even in well-mapped cities, the data from 2017 suggest that there are plenty of shops to map. It may be with more tightly constrained boundaries we may see more curves suggestive of saturation. I'll look at this in the next post.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-2739591709161533064
url: http://sk53-osm.blogspot.com/2018/04/linear-or-1d-maps-from-openstreetmap.html
title: Linear or 1D maps from OpenStreetMap
updated: 2018-04-25 20:19:35.128000
published: 2018-04-25 20:18:00.002000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.608674
updated_at: 2019-10-26 08:30:24.881106
content:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/26829630157/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="1-D map of Clumber Street"><img alt="1-D map of Clumber Street" height="177" src="https://farm1.staticflickr.com/970/26829630157_f7647c0451.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Clumber Street, a pedestrian shopping street in Nottingham</td></tr></tbody></table><br />We are all familiar with 1D, or linear, maps.<br /><a name='more'></a><br />We see them in graphics at bus stops; use them to work out when it's our stop on the metro; there's even a <a href="https://www.theguardian.com/news/datablog/gallery/2013/jan/09/london-underground-alternative-tube-maps">whole genre</a> of using the style of the London Underground map for other purposes. Here are one or two examples: <br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:KK_A_New_Service_Brochure_Map.gif" style="margin-left: auto; margin-right: auto;" title="By New York City Transit Authority (The JoeKorNer) [Public domain], via Wikimedia Commons"><img alt="KK A New Service Brochure Map" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/45/KK_A_New_Service_Brochure_Map.gif/512px-KK_A_New_Service_Brochure_Map.gif" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">New York Subway line map from 1960s</td></tr></tbody></table><br /><br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:Extrait_Tablede_Peutinger_Secteur_Rh%C3%B4ne_et_Alpes.jpg" style="margin-left: auto; margin-right: auto;" title="By Niconaike [CC0], via Wikimedia Commons"><img alt="Extrait Tablede Peutinger Secteur Rhône et Alpes" src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d2/Extrait_Tablede_Peutinger_Secteur_Rh%C3%B4ne_et_Alpes.jpg/512px-Extrait_Tablede_Peutinger_Secteur_Rh%C3%B4ne_et_Alpes.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">An excerpt from the Tabula Peutingeriana </td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:R%C3%A9seau_Grand_R,_poteau_d%27arr%C3%AAt.jpg" style="margin-left: auto; margin-right: auto;" title="By P.poschadel [CC BY-SA 2.0 fr (https://creativecommons.org/licenses/by-sa/2.0/fr/deed.en)], from Wikimedia Commons"><img alt="Réseau Grand R, poteau d'arrêt" src="https://upload.wikimedia.org/wikipedia/commons/thumb/0/02/R%C3%A9seau_Grand_R%2C_poteau_d%27arr%C3%AAt.jpg/256px-R%C3%A9seau_Grand_R%2C_poteau_d%27arr%C3%AAt.jpg" width="256" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Bus stop showing typical public transport use of 1D maps</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:Jefferys%27s_itinerary;_or_travellers_companion_Fleuron_T221156-3.png" style="margin-left: auto; margin-right: auto;" title="By Jefferys, Thomas [Public domain], via Wikimedia Commons"><img alt="Jefferys's itinerary; or travellers companion Fleuron T221156-3" src="https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Jefferys%27s_itinerary%3B_or_travellers_companion_Fleuron_T221156-3.png/256px-Jefferys%27s_itinerary%3B_or_travellers_companion_Fleuron_T221156-3.png" width="256" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A 18th century road itinerary for Northumberland<br />Jeffrey's <i>Fleuron </i>(perhaps also a strip map)</td></tr></tbody></table><br />If you want to know more, <a href="https://twitter.com/Lorp">Laurence Penney</a>, from Bristol, gave a highly condensed version of <a href="http://www.youshouldliketypetoo.com/blog/misc/one-dimensional-maps/">his talk</a> on these maps at the international State-of-the-Map conference at  Birmingham back in 2013. He has a fantastic, and ever growing,  collection of examples going back at least as far as the <a href="https://en.wikipedia.org/wiki/Tabula_Peutingeriana">Tabula Peutingeriana</a> a  Roman scroll of road itineraries (as a 19th century facsimile.&nbsp; <br />&nbsp; <br />Like <a href="http://sk53-osm.blogspot.co.uk/2013/06/cartograms-and-openstreet-map-data.html">cartograms</a>, 1D maps were always a map style which I have wanted to create using OpenStreetMap data. Over the last few years I have made sporadic efforts to see how I might create them, and for quite some while I have had a reasonable notion for a workable approach. However, doing so required a fairly concerted effort to break through the little annoyances which occur at every step. In the past couple of days I have made just that effort, and seem to have something which is reasonable, and certainly meets my initial goals.<br /><br /><h4>Why Now? </h4>The reason why I knuckled down to the task was pragmatic.<br /><br />For several years we have spent an hour before our summer pub meetings in the East Midlands (Nottingham and Derby) mapping areas close to our meeting place. These partly help to improve OSM, but were also conceived as a way to show interested parties what was involved in mapping. This year I felt that it would be worthwhile to focus our energies in this hour on checking the centres of Derby &amp; Nottingham for change. <br /><br />As OpenStreetMap data becomes richer the issue of maintaining existing data becomes more pressing. Amenities, shops and other points of interest in town and city centres are particularly prone to change and are often changes are often difficult to spot in any systematic way. It seems axiomatic that we should strive for a means which makes checking existing data somewhat less arduous than the effort it took to create in the first place, but often that is not true. Some editors, do help a bit: Vespucci, for instance, highlights objects which have not been touched by an edit within the last year.<br /><br />Clearly something which reduces the quantity of information to validate ought to help: hence 1D maps.<br /><br /><h4>Outline of the technique.</h4>As I stated above, I've had a reasonable idea of how to approach the problem for a while. These are the key steps I envisaged :<br /><ul><li>Merge all ways belonging to a particular road (identified by a common name and/or ref).</li><li>Create a single centre line for the road (to eliminate spurs, dual carriageways, roundabouts, service roads etc).</li><li>Identify all POIs belonging to a given street: this might be because they have the appropriate street name or are within a given distance of the street centre line (corner buildings, for instance may belong to another street, but be equally prominent on the intersecting one).&nbsp;</li><li>Locate the POI along the centre line, and, importantly, which side of the line it is located.</li><li>Calculate distances along the centre line.</li></ul>If this all goes according to plan one ends up with a tabulation like this:<br /><div class="de1"><br /><table border="1" style="border: 1px black; width: 90%;"><tbody><tr><th>Name </th><th>POI Type </th><th>Distance Along (m) </th><th>Side of Road </th><th>Distance Offset (m) </th></tr><tr><td>Curry 77</td><td>restaurant</td><td>173</td><td>R</td><td>21</td></tr><tr><td>The Salutation</td><td>pub</td><td>192</td><td>R</td><td>24</td></tr><tr></tr><tr><td>Cumin</td><td>restaurant</td><td>218</td><td>L</td><td>18</td></tr><tr><td>Nottingham Credit Union</td><td>bank</td><td>227</td><td>R</td><td>29</td></tr></tbody></table></div><br />This format gives enough data to work on a suitable rendering.<br /><br />In practice the second point is what took most time, so in the rest of the post I'll describe the steps in detail. I make no claims for elegance, and some of the steps I certainly didn't use the perhaps the best tools. Be warned there is <b>a lot</b> of nitty-gritty stuff below!<br /><h3>Step by Step guide</h3><h4>The Road Network &amp; POIs</h4>I used two Overpass-Turbo queries to download all highway ways and a discrete set of POIs (all amenities, shops and offices) from the same bounding box. Data was downloaded as Geojson and immediately uploaded into PostGIS (using the QGIS DBManager). I converted everything to the British Grid on upload which simplifies distance calculations, but also gives some 'safe' areas to locate transformed data at the end.<br /><br />Next I grouped all highways with the same name into multilinestrings:<br /><blockquote class="tr_bq"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">SELECT nextval('street_seq') street_id<br />&nbsp;&nbsp;&nbsp;&nbsp; , name<br />&nbsp;&nbsp;&nbsp;&nbsp; , st_multi(st_union(geom))&nbsp; geom<br />&nbsp; FROM hwy_upload<br />&nbsp;GROUP BY name</span></span></blockquote>The additional <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">st_multi</span></span> is needed because most roads will still be a single linestring when merged with the union function. Operations also need to work with more complex roads, so they are all treated in the same way.<br /><h4>Generating a Centre Line 1 : creating an approximate medial axis </h4>The first operation in generating a central line is to buffer the entire multilinestring by some arbitrary value (20 metres worked for me), and then use <a href="http://postgis.net/docs/ST_ApproximateMedialAxis.html">st_approximatemedialaxi</a>s (discussed extensively in an <a href="http://sk53-osm.blogspot.co.uk/2016/10/skeletons-in-water.html">earlier post</a>) to create a first cut medial axis.<br /><br />There are several problems with the medial axis generated by the PostGIS functions:<br /><ul><li>In many cases it cant be resolved to a line.&nbsp; </li><li>It's a multilinestring (I think even if the linear segments can be merged into a linestring).</li><li>It appears that it is constructed of many 2-point lines.</li><li>Some of the piecewise linear segments are minute (nanometres in length IIRC)</li><li>There can be many short or very short stubs.</li></ul>These problems were what stymied me when I first looked into using the medial axis.<br /><br />Some can be reduced by using <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><a href="https://postgis.net/docs/ST_SnapToGrid.html">st_snaptogrid</a>.</span></span> Using a 1 m snapping grid is absolutely fine for my purposes and eliminates some of the more absurdly short segments. After considering various ways of pruning the medial axis graph I instead decided that routing through it was easier to try.<br /><h4>Generating a Centre Line 2: using pgrouting&nbsp;</h4>As I had the data in PostGIS <a href="http://pgrouting.org/">pgrouting</a> was a fairly obvious candidate to generate routes. It is a relatively lightweight set of tools for generating routes and trivial to install<br /><br />To prepare the data for pgrouting I decomposed the medial axis multtistrings using st_dump and stored them in a new table.<br /><br />I also extracted all the points (node) from the original road linestrings and for each street found the pair of points which were most distant from each other. These two points would be used later as the source and target nodes for routing:<br /><blockquote class="tr_bq"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">WITH st_pt_distance AS (<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SELECT street_id, <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; a.pt_id source_pt_id, b.pt_id target_pt_id, <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; a.pt_geom source_pt_geom, b.geom target_pt_geom,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; st_distance(a.pt_geom, b.pt_geom) distance<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FROM hwy_pts a, hwy_pts b<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; WHERE a.street_id&nbsp; = b.street_id<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; AND a.pt_id &lt;b.pt_id<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; )<br />SELECT street_id, <br />FROM st_pt_distance <br />WHERE distance = (SELECT street_id, max(distance)<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FROM st_pt_distance<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GROUP BY street_id)</span></span></blockquote>As I only want routing along an individual street it was not obvious if I could safely use a single table for pgrouting. I therefore decided to use a temporary table for each street, and iterate over each street using a Postgres stored procedure. Streets with inconvenient geometries (such as those having two exterior rings when buffered) are eliminated at this step.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-2pyA3vgmtZs/WuDciCc5YkI/AAAAAAAABGQ/gBSdh1_QBNgsbQzcCaTKTMyIKdVxJ7NSQCLcBGAs/s1600/ng_roads1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="541" data-original-width="897" height="288" src="https://1.bp.blogspot.com/-2pyA3vgmtZs/WuDciCc5YkI/AAAAAAAABGQ/gBSdh1_QBNgsbQzcCaTKTMyIKdVxJ7NSQCLcBGAs/s320/ng_roads1.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Examples of buffered roads with rejected geometries (mostly in 2 or more parts).<br />Some are correct others needed fixing</td></tr></tbody></table>The basic means of generating a route was as follows:<br /><ul><li>Create the temporary routing table: <span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><br />CREATE temp_table AS<br /> SELECT a.street_id, (st_dump(aml_geom)).path street_seg_id,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (st_dump(aml_geom)).geom as geom,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; st_length(st_dump(aml_geom)).geom) as cost,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">st_length(st_dump(aml_geom)).geom) as reverse_cost<br />&nbsp; FROM hwy_upload<br />&nbsp;WHERE a.street_id = &lt;value&gt;</span></span></span></span></li><li><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-size: small;">Prepare the table for pgrouting:</span></span><br />SELECT pgr_createtopology('temp_table', 0.0000001,'geom'); <br /><br /><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;">This routine creates another table <span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">temp_table_vertices</span></span> which contains the nodes (vertices) of the routing network). Useful to remember when cleaning up the temporary tables. (I'm not sure why I used such a small value for the second parameter. I think I was worried about the very short segments which I later eliminated).</span></span></span></span></li><li><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;">Create a route. I used the standard <a href="http://docs.pgrouting.org/latest/en/pgr_dijkstra.html#pgr-dijkstra"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">pgr_dijkstra</span></span></a> call which returns a list of all vertices and edges making up the route:<br /><br /><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">SELECT * <br />FROM pgr_dijkstra('select street_seg_id,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; source, <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; target, <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cost, <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; reverse_cost from temp_table',<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (SELECT id FROM temp_table_vertices_pgr<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; WHERE geom = (SELECT start_pt from hwy_upload <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; WHERE street_id =&lt;value&gt;)),<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (SELECT id FROM t</span></span></span></span></span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">emp_table_vertices_pgr</span></span></span></span></span></span><br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; WHERE geom = (SELECT end_pt FROM </span></span></span></span></span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">hwy_upload <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; WHERE </span></span></span></span></span></span></span></span></span></span></span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">street_id =&lt;value&gt;</span></span></span></span></span></span>))</span></span></span></span> </span></span></li><li>In principle I should have been able to just use <a href="https://postgis.net/docs/ST_LineMerge.html"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">st_linemerge</span></span></a> to assemble the parts of the route, but I found some small gaps which prevented the function from working. Instead I used the ordered set of nodes and made the line from them with <a href="https://postgis.net/docs/ST_MakeLine.html">st_makeline</a>. This approach rather relies on the medial axis segments only having 2 points. (This may have been related to the short segments, but I haven't tried the original technique since then):<br /><br /><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">SELECT st_makeline(pts) geom from (<br />&nbsp;&nbsp;&nbsp; SELECT c.*<br />&nbsp;&nbsp;&nbsp; FROM <br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; ((SELECT b.street_seg_id,seq,node,edge,agg_cost, geom,<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; st_pointn(b.geom,1) pts&nbsp; <br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; FROM pgr_dijkstra(&lt;see above&gt;) a <br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; JOIN hwy_aml_dump b on edge=</span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">street_seg_</span></span>id)<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ...</span></span></li></ul>And <b>VOILA</b>! we have a centre line, almost home and dry.<br /><h4>Adding the POIs</h4>It is now trivial to calculate position along the centre line for each of the selected POIs using <a href="https://postgis.net/docs/ST_LineLocatePoint.html"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">st_linelocatepoint</span></span></a>:<br /><br /><blockquote class="tr_bq"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;">SELECT street_id, poi_id, <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; st_linelocatepoint(<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; st_pointn(<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; st_shortestline(poi.geom,street.centreline_geom),<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2),<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; centreline_geom)<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; * st_length(centreline_geom) distance_along_street<br />&nbsp; FROM streets street, pois poi<br />&nbsp;WHERE street.street_id = poi.street_id</span></span></blockquote><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-size: small;">The offset from the centre line is given by st_shortestline, but we need one other piece of information: where the POI is to the left or right of the centre line. Unfortunately it is not possible to rely on the shortest line dropped to the street centre line. It needs to be extended slightly to ensure it crosses the street centre line. Doing this is fairly painful in PostGIS (best managed with a simple function, see <a href="https://gis.stackexchange.com/questions/104439/how-to-extend-a-straight-line-in-postgis">GIS Stack Exchange</a> for examples):</span></span><br /><blockquote class="tr_bq"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">SELECT st_makeline( /* line from POI to 5 m across centre line) */<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; geom,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; /* POI location */<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; st_translate(geom, /* xsect_geom is intersection point */<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sin(st_azimuth(geom, xsect_geom ) * <br />&nbsp; &nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (st_length(st_shortestline(geom,</span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">xsect_geom </span></span>))+5),<br /><br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cos(st_azimuth(geom, </span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">xsect_geom </span></span> ) *&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; (st_length(st_shortestline(geom,</span></span><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-size: x-small;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">xsect_geom </span></span>))+5))<br />&nbsp; FROM pois</span></span></blockquote>Now the <a href="https://postgis.net/docs/ST_LineCrossingDirection.html">st_linecrossingdirection</a> function works every time, and because these are simple straight lines we can assign POIs to left or right.<br /><h4>Pulling it together</h4><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-3LYAtDAUK8Y/WuDhV8e3kzI/AAAAAAAABGc/Rw0Hbm8godYCCzbqBP0Z6kOqiqVAZT1QQCLcBGAs/s1600/ng_roads2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="541" data-original-width="897" height="288" src="https://3.bp.blogspot.com/-3LYAtDAUK8Y/WuDhV8e3kzI/AAAAAAAABGc/Rw0Hbm8godYCCzbqBP0Z6kOqiqVAZT1QQCLcBGAs/s320/ng_roads2.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;">Maid Marian Way in <a href="https://www.openstreetmap.org/way/102189757#map=16/52.9516/-1.1518">central Nottingham</a> <br />showing the calculated central line (orange) <br />and short lines (red) from POIs.<br />This is a topologically complex road.</span></span></td></tr></tbody></table>Now all the components exist and it's possible to generate the tabulation of POIs shown above. This is in itself quite useful, we used my <a href="https://www.dropbox.com/s/soc802n8ynn53s7/nottingham_pois_by_road.csv?dl=0">initial tabulation</a> a little last night until the rain made paper-based mapping infeasible. However, the original goal is some kind of more usable visualisation such as that shown at the head of the post. I'll discuss how I used QGIS to achieve this in the next post.<br /><br />I will add queries &amp; other code to github in the near future, probably as GISTs for now. Much of this was driven by wanting something to hand for our first mapping evening, and in many cases I went with what I could get to work quickly rather than investigating why a given result was unexpected.<br /><br />One other unexpected benefit is that I discovered a few bugs in OSM data. As is often the case when one looks at a larger set of data inconsistencies and minor errors stand out rather quickly.<br /><br /><br /><br /><br /><br /><br /><br />


guid: tag:blogger.com,1999:blog-2082422783692631078.post-4854408616499962486
url: http://sk53-osm.blogspot.com/2017/08/mapping-specific-building-form.html
title: Mapping a specific building form
updated: 2017-08-30 21:30:27.571000
published: 2017-08-30 21:25:00.001000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.611609
updated_at: 2019-10-26 08:30:24.883997
content:
  <br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-Wi-4STylCBs/WXjlb-zwZrI/AAAAAAAABCA/5PWwn0BAqpA36805d33ILa7DIeLL0ZsOACLcBGAs/s1600/IMG_1158a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="390" src="https://1.bp.blogspot.com/-Wi-4STylCBs/WXjlb-zwZrI/AAAAAAAABCA/5PWwn0BAqpA36805d33ILa7DIeLL0ZsOACLcBGAs/s320/IMG_1158a.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Arts-and-Craft style semi-detacheed houses, Edwards Lane Estate, Nottingham</td></tr></tbody></table><span id="goog_1201215879"></span><span id="goog_1201215880"></span><br />My interest in many aspects of urban environments has increased greatly since I started contributing to OpenStreetMap.<br /><br />I suppose this was always there but largely latent. Wandering around familiar places to capture details to add to OSM often forces me to ask questions about the area. Why is it there? Why is it laid out in that way? Who designed the buildings? When was it built? Why are there gaps in house numbering? What was planned for the little stub street? What used to be on the land with newer houses?<br /><br /><a name='more'></a>A particular class of urban areas which I find interesting are social housing estates. I've written a little about these before: <a href="http://sk53-osm.blogspot.co.uk/2011/01/twitchell_13.html">here</a> and <a href="http://sk53-osm.blogspot.co.uk/2011/04/radburn-exemplar-or-nightmare.html">here</a>. Fortunately I am not alone in this interest. Planners, Architects, Social Historians, and others share it. As a result there is more documentation readily available to satisfy casual queries. In the lead is an inspirational blog called <a href="https://municipaldreams.wordpress.com/">Municipal Dreams</a>. Although this has, an understandable, London bias, <a href="https://twitter.com/MunicipalDreams">John Boughton</a> has managed to cover plenty of other cities across the country. Other blogs may cover individual estates in detail, for instance <a href="https://twitter.com/iwaites60">Ian Waites</a>' <a href="http://instancesofachangedsociety.blogspot.co.uk/">blog</a> on the Middlefield Estate in Gainsborough, . For Nottingham, I'm lucky that <a href="https://twitter.com/chrismatthew82">Chris Matthews</a> has written (an all too short) history of council housing in the city.<br /><br />This particular blog post originated when Municipal Dreams tweeted a picture of a characteristic Arts-and-Craft style house in Osmaston, Derby:<br /><br /><blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Another Addison Road, this one in Derby. Built, I would guess, under his great 1919 Housing Act. <a href="https://t.co/CwAaroDUcF">pic.twitter.com/CwAaroDUcF</a></p>&mdash; Municipal Dreams (@MunicipalDreams) <a href="https://twitter.com/MunicipalDreams/status/888819536807514112">July 22, 2017</a></blockquote><script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script><br />This house looked very familiar: there are hundreds of similar houses scattered across Nottingham on estates built by the city council between the 1920s and the 1940s. It prompted me to systematically add all such houses to OSM. So far, I may have missed a few, I have found over 900 across the city as shown in the map below.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><br /><a href="https://1.bp.blogspot.com/-PqjxbNhFIdg/WXjlX8ltVfI/AAAAAAAABB8/WJYKz0dY-M8m1OVzLYMpt-w-NknENL-TQCLcBGAs/s1600/corner_semi_locs.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1132" height="640" src="https://1.bp.blogspot.com/-PqjxbNhFIdg/WXjlX8ltVfI/AAAAAAAABB8/WJYKz0dY-M8m1OVzLYMpt-w-NknENL-TQCLcBGAs/s320/corner_semi_locs.jpg" width="452" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Distribution of this particular building style across the city of Nottingham.</td></tr></tbody></table><br />Just adding the footprints of the houses is not enough. Additional tags are needed to be able to identify this particular design. At present I am using personal tags as an interim measure. I hope that ultimately it will be possible to accurately identify the design reference used by the Nottingham City Housing department (see examples in <a href="https://twitter.com/Alexander_Ball">Alex Ball</a>'s post on their early work). I am documenting my approach on the wiki.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ASherwood_Perry_Road_2235.jpg" style="margin-left: auto; margin-right: auto;" title="By ClemRutter (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons"><img alt="Sherwood Perry Road 2235" src="https://upload.wikimedia.org/wikipedia/commons/thumb/a/ac/Sherwood_Perry_Road_2235.jpg/512px-Sherwood_Perry_Road_2235.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">One of the earliest examples of the design. Perry Road, Sherwood Estate.<br />Later examples (see below and above) either have hanging tiles covering&nbsp; first floor at the front, or are not rendered at all.</td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-hq8JShUIHug/Wacl_yZOktI/AAAAAAAABDY/cDRhLr11BZ8fEoq3FgEyx5SCmcGl2EisQCLcBGAs/s1600/P1020760-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://3.bp.blogspot.com/-hq8JShUIHug/Wacl_yZOktI/AAAAAAAABDY/cDRhLr11BZ8fEoq3FgEyx5SCmcGl2EisQCLcBGAs/s320/P1020760-1.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Houses on Charnock Avenue, Wollaton Park Estate<br />Only 4 pairs of houses of this type were built on this estate.</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://4.bp.blogspot.com/-rb87drYWgGA/WacoOVwJDRI/AAAAAAAABDs/4EYp_ml4AoARTuba4nB4HA6dbdSt3Q2xgCLcBGAs/s1600/P1020758-1-2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://4.bp.blogspot.com/-rb87drYWgGA/WacoOVwJDRI/AAAAAAAABDs/4EYp_ml4AoARTuba4nB4HA6dbdSt3Q2xgCLcBGAs/s320/P1020758-1-2.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Side view of Charnock Avenue Houses.</td></tr></tbody></table>Using Environment Agency Lidar open data and <a href="https://wiki.openstreetmap.org/wiki/Simple_3D_buildings">Simple 3D Building</a> tags I've had a go at capturing the form of a couple of these houses too: the image below shows how they are rendered on F4 Map.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-_nveNt8Z1sg/WacSmBSGNLI/AAAAAAAABDI/L85W_8T2f9wB4flNxJRnS9yYyszmlIWyACLcBGAs/s1600/Screenshot%2B%252869%2529-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="934" data-original-width="1024" height="291" src="https://1.bp.blogspot.com/-_nveNt8Z1sg/WacSmBSGNLI/AAAAAAAABDI/L85W_8T2f9wB4flNxJRnS9yYyszmlIWyACLcBGAs/s320/Screenshot%2B%252869%2529-1.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Screengrab from <a href="http://demo.f4map.com/#lat=52.9482375&amp;lon=-1.1864589&amp;zoom=21&amp;camera.theta=65.054&amp;camera.phi=58.671">F4 map</a>.<br />The S3DB rules for the two houses are slightly different.</td></tr></tbody></table><br /><br />Given that the houses were built over a period of about 20 years it can be awkward to date them. In addition to old maps and various information in the Matthews book and on-line, there are archival aerial photographs of the estates largely taken shortly after they were built.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-4cNhBkbMDCg/WaaxhmXhGJI/AAAAAAAABCs/QxHKrOUclQgj-u9birrBFsjSO0-FEcN1wCLcBGAs/s1600/Bulwell_Hall_Estate_EPW034135.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="491" data-original-width="580" height="405" src="https://1.bp.blogspot.com/-4cNhBkbMDCg/WaaxhmXhGJI/AAAAAAAABCs/QxHKrOUclQgj-u9birrBFsjSO0-FEcN1wCLcBGAs/s320/Bulwell_Hall_Estate_EPW034135.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/way/324227126#map=16/53.0150/-1.1913">Bulwell Hall Estate</a> photographed from the South-west 1930. <br />The semi-detached houses discussed in this post can be seen at the far end of the road in the foreground.<br />Source: <a href="https://britainfromabove.org.uk/en/image/EPW034135">Britain from Above</a>.</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-430w6zXy1y0/WaayWdnZktI/AAAAAAAABC0/zrJoOUxPwLsZ5APIj92zglYJxfHDrq_SACLcBGAs/s1600/Aspley_Estate_EPW036010.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="488" data-original-width="580" height="405" src="https://2.bp.blogspot.com/-430w6zXy1y0/WaayWdnZktI/AAAAAAAABC0/zrJoOUxPwLsZ5APIj92zglYJxfHDrq_SACLcBGAs/s320/Aspley_Estate_EPW036010.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/node/469813521#map=16/52.9759/-1.2000">Aspley Estat</a>e from the S in 1931.<br />Again the houses are most prominent in the corners of street intersections.<br />Source: <a href="https://britainfromabove.org.uk/en/image/EPW036010">Britain from Above</a>. </td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-rNUhK6doitw/Waayo-v6SuI/AAAAAAAABC4/BinQtumtGV0WHgJReexsCYvWoLOYQE-aQCLcBGAs/s1600/Sherwood_EPW021828.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="481" data-original-width="580" height="397" src="https://1.bp.blogspot.com/-rNUhK6doitw/Waayo-v6SuI/AAAAAAAABC4/BinQtumtGV0WHgJReexsCYvWoLOYQE-aQCLcBGAs/s320/Sherwood_EPW021828.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/#map=17/52.98554/-1.15113">Sherwood Estates</a> from the SW in 1928.<br />The rendered houses shown above are in front of the school centre left.<br />Source: <a href="https://britainfromabove.org.uk/en/image/EPW021828">Britain from Above</a>.</td></tr></tbody></table>These pictures also give a good impression of the sheer scale of Nottingham's social housing programme between the wars. These three estates were more or less complete by 1930 and several others followed before the start of WWII. For instance the Broxtowe Estate consists of <a href="http://overpass-turbo.eu/s/rmQ">around 1700 houses</a>. It's difficult to determine the exact year of construction, but surprisingly easy to identify the decade, so for some I have tagged them with decade_built=*.<br /><br />Mapping this level of detail: a single design for a single type of house is feasible within a city and when they are all built by a single organisation. It may also be suitable for many of the private housing developments built post-war by national firms (Wimpey, Taylor Woodrow, etc) as they often used identical designs. It becomes more problematic when very similar, but not identical houses turn up across the country. Being alerted to this by Municipal Dreams, I noted a pair at <a href="http://www.openstreetmap.org/#map=18/52.96562/-1.30327">the junction</a> of Park Drive and Nottingham Road in Ilkeston, and then I came across these houses in Station Road Awsworth:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-TG_yRllMZ64/WacntoGmIzI/AAAAAAAABDo/UpGD1nHQpiQ6hBtr-hxOxUMY8qDZK59ygCEwYBhgL/s1600/P1030997-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://1.bp.blogspot.com/-TG_yRllMZ64/WacntoGmIzI/AAAAAAAABDo/UpGD1nHQpiQ6hBtr-hxOxUMY8qDZK59ygCEwYBhgL/s320/P1030997-1.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/#map=18/52.99328/-1.28342">Station Road, Awsworth</a>. Note minor differences (absence of bay window, single window at side etc)</td></tr></tbody></table>Although we can identify these houses with a range of tags:<br /><ul><li>building=semi-detached,</li><li>building:architecture=arts_and_crafts,</li><li>decade_built=1920,</li><li>architect=* (if known),</li><li>developer=* (or building:developer);</li></ul>it is still very difficult to envisage a single typology which is likely to satisfy the demands of mappers, historians of social housing, architecture and planning. However, I hope I have shown here that much is possible with OSM data at a local level.<br /><br />


guid: tag:blogger.com,1999:blog-2082422783692631078.post-6020618178428525537
url: http://sk53-osm.blogspot.com/2016/10/using-open-data-for-statistical-purposes.html
title: Using Open Data for Statistical Purposes
updated: 2016-10-24 11:40:53.746000
published: 2016-10-24 11:40:00.004000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.614483
updated_at: 2019-10-26 08:30:24.886834
content:
  <div class="separator" style="clear: both; text-align: center;"></div>A <a href="https://twitter.com/owenboswarva/status/789541162583490561">tweet</a> by <a href="https://twitter.com/owenboswarva">Owen Boswarva</a> drew my attention to a <a href="https://publichealthmatters.blog.gov.uk/2016/10/21/obesity-and-the-environment-the-impact-of-fast-food/">recent report</a> by <a href="https://twitter.com/PHE_uk">Public Health England</a> (PHE) on the correlation of density of fast food outlets and deprivation.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-70Lx_CMFMSA/WA3oiBb0cjI/AAAAAAAABAY/Qy0I4VvGI549T3KFlYSUS-Fit2tjhK7JgCLcB/s1600/ff_outlet_density_fhrs.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://2.bp.blogspot.com/-70Lx_CMFMSA/WA3oiBb0cjI/AAAAAAAABAY/Qy0I4VvGI549T3KFlYSUS-Fit2tjhK7JgCLcB/s320/ff_outlet_density_fhrs.jpg" width="450" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Number of Fast Food outlets normalised to 100,000 population for Local Authorities in England<br />Source: Food Hygiene Rating Scheme (Takeaway class)</td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"></div>Specifically my interest was directed at the source of fast food outlet counts. PHE used data from <a href="http://www.pointx.co.uk/about.htm">PointX</a>, a joint venture of Landmark Information and the Ordnance Survey. I instantlly wondered if one could do the same thing with <a href="http://ratings.food.gov.uk/">Food Hygiene Ratings</a> (FHRS) open data. This is a quick report on doing exactly that.<br /><br />I already had a complete set of FHRS data for September 2016. I needed to download various administrative and census geographies, population figures for <a href="https://en.wikipedia.org/wiki/ONS_coding_system">Lower Layer Super Output Areas</a> (LSOAs), <a href="https://en.wikipedia.org/wiki/Indices_of_deprivation_2007">Index of Multiple Deprivation</a> (IMD) Scores for LSOAs and various files showing the linkages between the geographies.<br /><br />A certain amount of data wrangling was needed to merge this data (for instance linkages, population and IMD) all came in spreadsheets with awkward column names, multiple sheets and other minor inconveniences. Once these were sorted out I had a table with base figures at LSOA level which could be readily aggregated to Middle Layer Super Output Areas (MSOAs) and local authorities. The IMD score is rebased by summing LSOA scores multiplied by population and then dividing by total population.<br /><br />Using R I constructed simple scatter plots with a regression line and 95% confidence limits for both MSOA and Local authorities.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-FyfWqPhqpzw/WA3mvyYBWLI/AAAAAAAABAE/yJXLFW3a4SA3SLaLVwh0BLS4MqnknGqXgCLcB/s1600/imd_msoa_plot-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://2.bp.blogspot.com/-FyfWqPhqpzw/WA3mvyYBWLI/AAAAAAAABAE/yJXLFW3a4SA3SLaLVwh0BLS4MqnknGqXgCLcB/s320/imd_msoa_plot-1.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Number of Fast Food outlets (normalised) vs calculated <br />Index of Multiple Deprivation for Middle Super Output Areas</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-ZIXDpopVLJ8/WA3mxSnpx5I/AAAAAAAABAI/csKj2Y4K8oU7VqyeiUdcacWY7gbxI2ILACLcB/s1600/imd_plot2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://1.bp.blogspot.com/-ZIXDpopVLJ8/WA3mxSnpx5I/AAAAAAAABAI/csKj2Y4K8oU7VqyeiUdcacWY7gbxI2ILACLcB/s320/imd_plot2.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Number of Fast Food outlets (normalised) vs calculated <br />Index of Multiple Deprivation for Local Authorities<br />(outlier of City of London excluded)</td></tr></tbody></table><br />For comparison the relevant plot from the PHE report is shown below:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://publichealthmatters.blog.gov.uk/wp-content/uploads/sites/33/2016/10/fast-food-graph.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="405" src="https://publichealthmatters.blog.gov.uk/wp-content/uploads/sites/33/2016/10/fast-food-graph.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Scatter plot from PHE report for Local Authorities</td></tr></tbody></table><br />The final comparison I made was perhaps one I should have done at the outset. Comparing raw counts of fast food outlets from the Open Data source (FHRS) and the PointX data. PHE provided a table of counts at ward level. It took me a while to find a shape file and codes which fitted (the codes change year-on-year), but then it was easy to do a Point-in-Polygon count of the FHRS data for a direct comparison. The correlation of values was plotted in R again.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://4.bp.blogspot.com/-YQlj1o7PlA8/WA3oGPgr4QI/AAAAAAAABAQ/Eu75Fe5CgXkEh5_buQNDjl92SVuowcnhQCLcB/s1600/ff_plot2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://4.bp.blogspot.com/-YQlj1o7PlA8/WA3oGPgr4QI/AAAAAAAABAQ/Eu75Fe5CgXkEh5_buQNDjl92SVuowcnhQCLcB/s320/ff_plot2.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of number of Fast Food outlets by 2015 ward boundaries<br />derived from Food Hygiene Data or from Landmark/Ordnance Survey</td></tr></tbody></table><br />Doing this took longer than I hoped: but almost entirely because I don't know my way around the various formats of boundary data related to the census and more changeable boundaries such as the wards.<br /><br />I haven't done a formal comparison of the outputs, but the visuals presented above strongly suggest that FHRS data is just as useful as the PointX data for this purpose. The main explanation for the lower count coming from FHRS is that the PointX data includes outlets which do food delivery which may include places classified as Restaurants in FHRS.<br /><br />I had expected more issues with FHRS because there is clearly an under-reporting issue in inner city areas due to rapid turnover of management of takeaways (see the <a href="https://www.theguardian.com/world/2016/sep/25/day-in-life-food-safety-inspector-newham-east-london-hygiene">recent Guardian article</a> for an in-depth appreciation of this issue). The other week at the <a href="http://wiki.openstreetmap.org/wiki/London#Past_events">London OpenStreetMap</a> pub meeting in Islington I insisted that we should check the 'scores-on-the-doors' before choosing where to eat our Burritos (a habit I've learnt from <a href="https://twitter.com/drsiant">Dr Sian Thomas</a>). The three fast food outlets next to the pub didn't feature at all on the FHRS data.<br /><br />In conclusion: now that FHRS data covers nearly every major authority in the country (Rutland were the last still hold out) it is entirely suitable for a range of statistical purposes.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-600813469677196125
url: http://sk53-osm.blogspot.com/2016/10/skeletons-in-water.html
title: Skeletons in the Water
updated: 2016-10-07 10:48:23.706000
published: 2016-10-07 10:48:00.001000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.617296
updated_at: 2019-10-26 08:30:24.889532
content:
  For a number of years now I have, from time-to-time, made the odd stab at trying to find the flowline of a river from the mapped surface area of the watercourse using OpenStreetMap data.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AWindermere_Lake_District_from_hill.JPG" style="margin-left: auto; margin-right: auto;" title="By Abbasi1111 (Own work) [CC BY 3.0 (http://creativecommons.org/licenses/by/3.0)], via Wikimedia Commons"><img alt="Windermere Lake District from hill" src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/20/Windermere_Lake_District_from_hill.JPG/512px-Windermere_Lake_District_from_hill.JPG" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Windermere in the English Lake District, one of my test cases.</td></tr></tbody></table>I not infrequently find, being neither trained as a geospatial specialist nor a mathematician, that, although I have a fairly clear idea of what I want to do with some particular manipulation of geodata, I am stymied. More often than not this is simply because I don't know the most widely used term for a particular technique. It was therefore really useful to learn from <a href="http://www.imagico.de/">imagico </a>that the generic term for what I was trying to do is <a href="https://en.wikipedia.org/wiki/Topological_skeleton"><b>skeletonisation</b></a>. (I do hope my relative ignorance is not on <a href="https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/">this </a>scale.)<br /><br />Armed with this simple additional piece of knowledge immediately opened out the scope of resources available to me from wikipedia articles, blog posts, to software implementions. Unfortunately when I first tried to get the relevant extensions (<a href="http://www.sfcgal.org/">SFCGAL</a>) installed in PostGIS I was not able to get them to work, so I shelved looking at the problem for a while.<br /><br />Very recently I re-installed Postgres and Postgis from scratch with the latest versions and the <a href="http://postgis.net/docs/reference.html#reference_sfcgal">SFCGAL extensions</a> installed fine. So it was time to re-start my experiments.<br /><br />Once I was aware of skeletonisation as a generic technique I also recognised that it may be applicable to a number of outstanding issues relating to post-processing OpenStreetMap data. Off the top of my head &amp; in no particular order these include:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/4647841258" style="margin-left: auto; margin-right: auto;" title="Wiggly River Trent"><img alt="Wiggly River Trent" height="438" src="https://c3.staticflickr.com/5/4005/4647841258_9d9f480ef7.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">My earliest experiment using Ordnance Survey Open Data for the River Trent<br />Voronoi triangles based on modes of polygon, clipped back to polygon</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> <br /><ul><li><b>Waterway flowlines</b>. Replacing rivers mapped as areas by the central flowline where such a flowline has not already been mapped. Such data can then be used for navigation on river systems or for determining river basins (and ultimately watersheds/hydrographic basins). (It is this data which much of the rest of the post is concerned with).</li><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-Nl1isbZn3TI/V7RqIm2HfDI/AAAAAAAAA-I/FLb81Q3dIlsp5aGxOGHPlhqZ1L_g8RDdQCLcB/s1600/annapurna_himal_eg.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="275" src="https://2.bp.blogspot.com/-Nl1isbZn3TI/V7RqIm2HfDI/AAAAAAAAA-I/FLb81Q3dIlsp5aGxOGHPlhqZ1L_g8RDdQCLcB/s320/annapurna_himal_eg.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Earlier experiments with OpenStreetMap glacier data for the <a href="http://www.openstreetmap.org/#map=13/28.5394/83.8310">Annapurna region</a><br />Height (contours) &amp; slope(shading) data via Viewfinderpanorams.com<br />Voronoi triangulation clipped to glacier used to try &amp; find flowlines for the main Annapurna Glacier.<br />Some ideas originated from conversations with Gravitystorm.<br />Map data (c) OpenStreetMap contributors 2014.</td></tr></tbody></table><li><b>Glaciers</b>. Similarly for rivers although height also needs to be factored in. The idea is not just to identify flows on a glacier, but also simulate likely regions of higher speed flow with a view to creating an apparently more realistic cartographic depiction of the glacier. (Only apparent because in reality one needs lots of good aerial photography to correctly map ice-falls, major bergschrunds, crevasses, crevasse fields etc.).<br /></li><li><b>Creating Address Interpolation lines</b>.&nbsp; A small subset of residential highways have quite complex structures and therefore it is non-trivial to add parallel lines for address interpolation. Buffering the multilinestring of the highway centre lines &amp; then resolving that to a single line would help. (More on this soon).<br /></li><li><b>Dual Carriageways</b>. Pretty much the same issue as above except there is the additional problem of pairing up the two carriageways. Resolving them to a single way would make high-level routing and small scale cartography better (i.e., it's a cartographic generalisation technique). </li><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/--92Fcx_Ri8s/V7R0nX1qCrI/AAAAAAAAA-Y/PExYlGV-d8oGOqK9CDqALhiVCcrrL9DVwCLcB/s1600/old_market_square_ss.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="275" src="https://2.bp.blogspot.com/--92Fcx_Ri8s/V7R0nX1qCrI/AAAAAAAAA-Y/PExYlGV-d8oGOqK9CDqALhiVCcrrL9DVwCLcB/s320/old_market_square_ss.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The straight skeleton of Old Market Square Nottingham which allows routing across and close to most of the square<br />The skeleton does not take account of some barriers on the square, <br />but the hole at the left (a fountain shows the principle).<br />Data source: (c) OpenStreetMap contributors 2015.</td></tr></tbody></table><br /><li><b>Routing across areas for pedestrians</b>. Pedestrian squares, parks car parks etc. Skeletonisation of such areas may offer a quick &amp; dirty approach to this problem.</li></ul>What follows are some experiments I've done with water areas in Great Britain. I have mainly used the <a href="http://postgis.net/docs/manual-dev/ST_StraightSkeleton.html">ST_StraightSkeleton</a> function, with rather more limited time spent looking at <a href="http://postgis.net/docs/manual-2.2/ST_ApproximateMedialAxis.html">ST_ApproximateMedialAxis</a>. The two images below show my initial attempt to find hydrographic basins: this works merely by chaining together continuous waterway linestrings. These results are not bad, but several major rivers are divided into multiple watersheds. The map of Ireland shows the problem better because the Shannon system appears as a number of discrete watersheds, largely because the Shannon flows through a number of sizeable lakes. Other major rivers illustrating the issue in the UK are the Dee, Trent and Thames.<br /><br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22944754896/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="River Systems of Great Britain (derived from OSM)"><img alt="River Systems of Great Britain (derived from OSM)" height="667" src="https://c1.staticflickr.com/6/5784/22944754896_68b1af138f.jpg" width="472" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Identification of watersheds in Great Britain by contiguous sections of waterway in OpenStreetMap</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> <br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23379593606/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Irish Watersheds from OpenStreetMap"><img alt="Irish Watersheds from OpenStreetMap" height="640" src="https://c7.staticflickr.com/6/5802/23379593606_28cce6a05d_z.jpg" width="453" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Watersheds in Ireland derived from linear watercourses on OpenStreetMap.<br />Waterways are generally less well-mapped in Ireland, but also several major waterways pass through large lakes (e.g., the Bann (Lough Neagh), Shannon (Lough Ree, Lough Derg), and the <a href="https://en.wikipedia.org/wiki/River_Erne">Erne </a>(Upper &amp; Lower Lough Erne)) and no centre line is available. </td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> So the naive approach raised two problems:<br /><ul><li>Lakes, rivers mapped as areas etc also needed to be included in creating the elements of the watershed</li><li>Actual watersheds can be created by creating Concave shells around their constituent line geometries. <strike>Unfortunately I get a PostGIS non-noded intersection error when trying this, so wont discuss it further (although if someone can walk me through how to avoid such problems I'm all ears)</strike>. As later versions of PostGIS seem more robust I return to this later.</li></ul>Of course the simple way to address the first one is just to include areas of water as additional objects in the chain of connected objects. However I would also like to replace rivers as areas, and smaller lakes with linestrings as this type of generalisation can greatly assist cartography at smaller scales. The lack of a source of generalised objects derived from OSM has been a criticism of its utility for broader cartographic use, so this is another aspect of this investigation.<br /><br />So now with skeletonisation routines working in PostGIS time to look at some of the basics.<br /><br />I've taken Windermere, the largest lake in England, as an example to work through some of the issues. Windermere is a long thin lake which should have a fairly obvious median line. However, it does have some islands which complicate the matter.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-JemJGLoVP0k/V8RpeVNrBnI/AAAAAAAAA-s/E6-XpDLRg_kUEAacdB6Xwc_KrdPXp2SjwCLcB/s1600/windermere5.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="285" src="https://2.bp.blogspot.com/-JemJGLoVP0k/V8RpeVNrBnI/AAAAAAAAA-s/E6-XpDLRg_kUEAacdB6Xwc_KrdPXp2SjwCLcB/s400/windermere5.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Six versions of Windermere showing area, media axis (red), straight skeleton (thinner lines)<br />for different degrees of simplification (parameters of 0,5,25,125..). <br />Original shape is shown as a blue outline.`<br />All created as a single query using st_translate.</td></tr></tbody></table>Both the straight skeleton &amp; the medial axis are complicated multi-linestrings if I use raw OSM data for Windermere. Progressive simplification of the shape reduces this complexity with reasonable desirable medial axis appearing when simplified with the parameter of around 100 (assumed to be meters in Pseudo-Mercator). Unfortunately there are two problems: the derived axis passes through large islands; and inflow streams are not connected.<br /><br />I therefore took a different approach. I disassembled Windermere using ST_Dump and cut the line forming the outer ring at each point a stream or river way touched the lake. I then simplified each individual bit of shoreline between two streams &amp; then re-assembled the lake.<br /><br />When this is done all inflows &amp; outflows are connected to the straight skeleton of the simplified lake area. This can be input directly into my routines for collecting all ways making up a watershed.<br /><br />Additionally the straight skeleton can be pruned. The simplest one is to just remove all individual linestrings which dangle (i.e., are not connected to a waterway). Presumably one can iterate this until one has the minimum set necessary to a connected set of flows, but I haven't tried this.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-0P24EArhvR4/V8RqgN4BAnI/AAAAAAAAA-0/Mnerqlm8MMEBjsvb4p8L10eazU8Y8lREgCLcB/s1600/windermere_skelton_detail.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="281" src="https://1.bp.blogspot.com/-0P24EArhvR4/V8RqgN4BAnI/AAAAAAAAA-0/Mnerqlm8MMEBjsvb4p8L10eazU8Y8lREgCLcB/s400/windermere_skelton_detail.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Straight Skeletons for Windermere calculated for different simplification parameters.<br />The grey lines represent a parameter where details of islands are kept but the number of edges in the skeleton is greatly reduced.</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-o6rAXtd9hH8/V8RqjTCCGGI/AAAAAAAAA-4/9Rp1jBUc9j49UKP_bqHUVJm3Ig-V4TNrgCLcB/s1600/windermere_skelton_joined.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="270" src="https://2.bp.blogspot.com/-o6rAXtd9hH8/V8RqjTCCGGI/AAAAAAAAA-4/9Rp1jBUc9j49UKP_bqHUVJm3Ig-V4TNrgCLcB/s320/windermere_skelton_joined.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Windermere showing inflow &amp; outflow waterways</td><td class="tr-caption" style="text-align: center;"></td><td class="tr-caption" style="text-align: center;"></td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-PReM361HPhY/V8Rqt-h6PxI/AAAAAAAAA-8/alH1jZByAccrgi6ASXJIpX8wRa6v7-SDgCLcB/s1600/windermere_skelton_joined_pruned.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="270" src="https://2.bp.blogspot.com/-PReM361HPhY/V8Rqt-h6PxI/AAAAAAAAA-8/alH1jZByAccrgi6ASXJIpX8wRa6v7-SDgCLcB/s320/windermere_skelton_joined_pruned.jpg" width="280" /></a></div><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-sLmWqgCIcxU/V8RqxqWxwwI/AAAAAAAAA_A/qaRI6LwlQwQS0fX1P5PyPHvtqHxPQtP2gCLcB/s1600/windermere_skelton_joined_pruned_detail.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="270" src="https://3.bp.blogspot.com/-sLmWqgCIcxU/V8RqxqWxwwI/AAAAAAAAA_A/qaRI6LwlQwQS0fX1P5PyPHvtqHxPQtP2gCLcB/s320/windermere_skelton_joined_pruned_detail.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Detail of the centre of Windermere showing a reduced straight skeleton linked to inflowing streams (blue). The equivalent without reassembly and preserving stream topology is in red</td></tr></tbody></table>For a single lake it is possible to determine the appropriate degree of simplification to apply, but the complete set of lakes &amp; ponds in Great Britain is a completely different matter.<br /><br />Over simplification will result in too big a discrepancy between the original shape and adjacent geometries. Even for Windermere trying to include islands in a reassembly fails with too great a degree of simplification because geometries now cross each other.<br /><br />My approach has been to simplify geometries with parameters from 50 to 250 metres in ST_Simplify. I then compare a number of factors with the original:<br /><ul><li>Do I get a valid geometry</li><li>Number of interior rings </li><li>A measure of surface area</li></ul>With these I then choose one of the simplified geometries for further processing. In general large lakes and riverbank polygons will tolerate more simplification. The overall result is less complicated straight skeletons for further processing. (As an aside I think <a href="http://www.cs.nuim.ie/~pmooney/">Peter Mooney</a> of Maynooth did some work on comparing lake geometries using OSM data around 2010 or 2011).<br /><br />For my immediate practical purposes of finding watersheds I did not perform further pruning of skeletons, but such a process is needed for other applications such as cartographic generalisation.<br /><br />Even with my first approach which I thought was fairly robust I'm losing a fair number of&nbsp; waterways with simplification. I haven't looked into this further because it will delay finishing this particular post: and it's been on the stocks long enough.<br /><br />For further posts on the problems of skeletonisation read <a href="https://twitter.com/smathermather">Stephen Mathers</a> <a href="https://smathermather.wordpress.com/">blog </a>which I found very useful. <a href="https://github.com/StyXman">StyXman </a>is <a href="https://github.com/StyXman/osm-centerlines/tree/master/src/org/openstreetmap/josm/plugins/centerlines">developing a JOSM plugin</a> which uses some of these techniques to create centrelines too. A big thank you to him, and, of course, to Christoph Hormann (<a href="http://www.imagico.de/">imagico</a>).


guid: tag:blogger.com,1999:blog-2082422783692631078.post-3440320304825080633
url: http://sk53-osm.blogspot.com/2016/07/how-far-are-hedgehogs-from-road.html
title: How far are Hedgehogs from a road?
updated: 2016-07-01 10:58:12.422000
published: 2016-07-01 10:58:00
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.620015
updated_at: 2019-10-26 08:30:24.892225
content:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/27777325701/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="My last hedgehog siting (2010)2887a"><img alt="My last hedgehog siting (2010)2887a" height="375" src="https://c6.staticflickr.com/8/7444/27777325701_3f62054431.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">My last <a href="https://en.wikipedia.org/wiki/European_hedgehog">hedgehog </a>sighting in Britain: <a href="https://en.wikipedia.org/wiki/Elston">Elston</a>, Nottinghamshire 2010.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script><br /><br />One of my great joys with OpenStreetMap (and other (mainly) geographical Open Data) is that it provides a way into answering intriguing analytical questions.<br /><br />A few weeks ago the query was from a <a href="https://twitter.com/hedgehoghugh">Hedgehog ecologist</a>: naturally I learnt of the query through OSM (via IRC to be precise).<br /><br />The question was very simple: <i>&nbsp;</i><br /><br /><blockquote class="tr_bq"><i>What proportion of Britain's land area is more than 100 m from a road? </i>&nbsp;</blockquote><br />The reason it is germane for hedgehogs is that historically they have had a very high mortality from crossing roads. These days they are so rare, that spotting a squashed hedgehog is itself a rarity. Certainly <a href="https://www.cartoonstock.com/cartoonview.asp?catref=emc0071">this cartoon</a> would not have the same resonance it did when it first appeared in the 1970s.<br /><br />To answer the query is fairly straightforward: providing one has either a GIS tool or database to hand <b>AND</b> a full data set of British roads. QGIS and PostGIS were available &amp; I also have a full set of OSM data for May 2015 in the latter.<br /><br /><br /><a name='more'></a>The steps involve in answering the query were simple ones as follows:<br /><ul><li>Select a relevant subclass of highway elements from OSM (in this case deliberately selecting the main types of motorable roads)</li><li>Re-project into <a href="https://en.wikipedia.org/wiki/British_Grid">British Grid</a> (which enables more precise answers to this type of query)</li><li>Buffer all the roads by 100m</li><li>Eliminate overlaps</li><li>Sum the area covered&nbsp;</li><li>Subtract that from the known land area of Great Britain.</li></ul>In practice for this type of work I nearly always grid the data so that costly queries relying on the GIST indexes on geometries have to do less work. So I clip roads using a 2 or 5 km grid. All roads in a given grid square can either be merged into a multiline structure &amp; then buffered, or buffered and then merged into a single multipolygon. One way is probably more efficient than the other. Once buffered the buffer area<br /><br />In my initial analysis which I carried out very quickly I made three errors:<br /><ul><li>Failing to include trunk roads in the initial road selection through missing quotes in the SQL predicate</li></ul><ul><li>Using <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">st_within</span> &amp; not <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">st_intersects</span> to find roads crossing a grid square.</li></ul><ul><li>Clipping back to grid square boundaries in all cases. A road ending on the edge of one grid square still creates a 100 m semicircle in the adjacent square</li></ul>As usual it was visualisation of the data which hinted at the things I needed to fix. Certain main roads were missing, there were islands of roads in central Wales, and the grid boundaries were too obvious when visualised.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-2-CR4XBNCiI/V3JBWPeRW-I/AAAAAAAAA8I/NnxETZmOoZU9DoAaRwe1XKNHM9v9k3EhQCLcB/s1600/100m_roads.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="318" src="https://2.bp.blogspot.com/-2-CR4XBNCiI/V3JBWPeRW-I/AAAAAAAAA8I/NnxETZmOoZU9DoAaRwe1XKNHM9v9k3EhQCLcB/s320/100m_roads.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Initial visualisation. Note big gaps in Central Wales and absence of continuous lines on motorways.</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-JpkS8O4d_jM/V2rPpOwZkPI/AAAAAAAAA7k/ofkWNhHCuWINc5Fi6seJ98WH0-iw3WEfACLcB/s1600/road100b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="275" src="https://3.bp.blogspot.com/-JpkS8O4d_jM/V2rPpOwZkPI/AAAAAAAAA7k/ofkWNhHCuWINc5Fi6seJ98WH0-iw3WEfACLcB/s320/road100b.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Visualisation with fixed data. Compare Central Wales and motorways.</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-53BeGY4PZIs/V2rQYIl7VKI/AAAAAAAAA70/iKIlqYoK0swtZtL6Pg6aXsPkzOXSVGShwCLcB/s1600/road100a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://2.bp.blogspot.com/-53BeGY4PZIs/V2rQYIl7VKI/AAAAAAAAA70/iKIlqYoK0swtZtL6Pg6aXsPkzOXSVGShwCLcB/s320/road100a.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Area W of Birmingham enlarged. The main population centres are <a href="https://en.wikipedia.org/wiki/Shrewsbury">Shrewsbury </a>(top centre) and <a href="https://en.wikipedia.org/wiki/Telford">Telford </a>(right of top centre) with the <a href="https://en.wikipedia.org/wiki/West_Midlands_conurbation">West Midlands conurbation</a> on right edge. Welsh hill country is top loeft.</td></tr></tbody></table><br />The first I suspected anyway.&nbsp; I'd written the selection rather hurriedly &amp; there are 11 types of highway I include normally, so it's rather easy to mis-spell one, or as in this case not spot missing quotes. Easy to fix.<br /><br />The second was less immediately obvious, but it's one which has caught me out in the recent past too. This is writing <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">st_within</span> in a query where <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">st_intersects</span> is required. I suppose I do point-in-polygon type queries more frequently so instinctively type <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">st_within</span>. Also <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">st_within<span style="font-family: &quot;georgia&quot; , &quot;times new roman&quot; , serif;"> <span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;">is specifically a function which works on polygons whereas </span></span></span><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;"><span style="font-family: &quot;georgia&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">st_intersects</span> has broader applicability.</span></span></span><br /><br />Either way these were easy errors to fix. To do the calculation I used an Ordnance Survey Open Data file to hand (one of local authority boundaries) which results in a slight overstatement of land area as some LAs boundaries include marine areas (notably in <a href="https://en.wikipedia.org/wiki/The_Wash">The Wash</a> &amp; in the <a href="https://en.wikipedia.org/wiki/Bristol_Channel">Bristol Channel</a>) together with the OSM outline for the <a href="https://en.wikipedia.org/wiki/Bristol_Channel">Isle of Man</a> (as this is not available in OSGB Open Data). The area for Great Britain and Isle of Man which I used was 238,905 square km, but this can be corrected to any other widely available figure, such as that given on <a href="https://en.wikipedia.org/wiki/Countries_of_the_United_Kingdom">Wikipedia</a> of 229,519 square km. Equally the area of road buffers will include some sea for coastal roads. Ideally I need my grid cells to be land area only, something I'm thinking of doing for other purposes too.<br /><br />The sum of the area of the road buffers was for 100 m, for 200 m, for 500 m and 1000 m. I was also asked for a 5km buffer and I calculated that separately using a larger grid size. Note that gridding is purely something done to speed up the process. A recent talk highlights (sorry, I can't find the link for this slidedeck but it was from a recent Geo-conference) that many standard geometry routines have poor <a href="https://en.wikipedia.org/wiki/Computational_complexity_theory">computational complexity</a> properties so one often has to resort to divide-and-conquer approaches such as gridding the data.<br /><br />So here are the approximate figures, given the caveats above:<br /><ul><li>Within <b>100 m</b> of a road: 53,964 km2 or about 23.5% of land area of Great Britain </li><li>Within <b>200 m</b> of a road: 87,971 km2 or about 38.3% of land area</li><li>Within <b>500 m</b> of a road: 150,147 km2 or about 65.4% of land area</li><li>Within <b>1000 m</b> of a road: 189,562 km2 or about 82.6% of land area</li><li>Within <b>5000 m</b> of a road: 260,419 km2 or rather more than the land area concerned. </li></ul>The last figure made it abundantly clear that I needed to address the sources of approximation. Coastlines &amp; other borders introduce edge effects which can seriously <a href="http://sk53-osm.blogspot.co.uk/2015/05/retail-outlets-on-openstreetmap.html">complicate this type of analysis</a> and often require ensuring that one has appropriate data to handle it. So I did the following:<br /><ul><li>Created a shape file of the land area for Great Britain &amp; Isle of Man</li><li>Clipped each buffered result set by that shape file</li><li>Used each of these inputs for the final calculation</li></ul>In order to make the shape file&nbsp; needed something based on the coastline. Fortunately I remembered Geolytix had used OSGB Open Data of coastlines to produce refined postal area polygons. This was immediately available to me and I therefore used this as the starting point. I created a separate Isle of Man shapefile from OSM data and merged the two together. So here are the revised figures:<br /><ul><li>Within <b>100 m</b> of a road: 53,760 km2 or about 23.3% of land area of Great Britain </li><li>Within <b>200 m</b> of a road: 87,353 km2 or about 37.9% of land area</li><li>Within <b>500 m</b> of a road: 147,740 km2 or about 64.2% of land area</li><li>Within <b>1000 m</b> of a road: 186,533 km2 or about 79.7% of land area</li><li>Within <b>5000 m</b> of a road: 226,418 km2 or 98.3% of the land area.&nbsp;</li></ul>And just to enable checking here are some visualisations:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-6MCCrvhvR7Q/V3QqwphtWaI/AAAAAAAAA8Y/qpkfoKEoxS4DxGCReezLje2DW_Zixr6FACLcB/s1600/hedgehog_roads_e%2526W.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://1.bp.blogspot.com/-6MCCrvhvR7Q/V3QqwphtWaI/AAAAAAAAA8Y/qpkfoKEoxS4DxGCReezLje2DW_Zixr6FACLcB/s320/hedgehog_roads_e%2526W.jpg" width="450" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Distances for England &amp; Wales</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-37JQKK7bHwI/V3UsZ2I6YnI/AAAAAAAAA8o/IxsW7zqPLCgY56M5Z4lYg_zV2vVmXp37gCLcB/s1600/hedgehog_roads_scotland.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="https://3.bp.blogspot.com/-37JQKK7bHwI/V3UsZ2I6YnI/AAAAAAAAA8o/IxsW7zqPLCgY56M5Z4lYg_zV2vVmXp37gCLcB/s320/hedgehog_roads_scotland.jpg" width="452" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Mainland Scotland</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-Ax_Y2L-59EA/V3U7Pqpl_KI/AAAAAAAAA84/ZtaW4Ets--k61aMIio58cFpNEXAMbZaOgCLcB/s1600/hedgehog_roads_dark_peak.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://2.bp.blogspot.com/-Ax_Y2L-59EA/V3U7Pqpl_KI/AAAAAAAAA84/ZtaW4Ets--k61aMIio58cFpNEXAMbZaOgCLcB/s320/hedgehog_roads_dark_peak.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">English <a href="https://en.wikipedia.org/wiki/Peak_District">Peak District</a> (<a href="https://en.wikipedia.org/wiki/Dark_Peak">Dark Peak</a>)<br /><a href="https://en.wikipedia.org/wiki/Greater_Manchester">Manchester </a>is on W, <a href="https://en.wikipedia.org/wiki/Sheffield">Sheffield </a>on R</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-ufYicSnAG6s/V3U7WWPaNHI/AAAAAAAAA9A/WV5eJ21aVhUFAPeqoXblINOqACcYMethACLcB/s1600/hedgehog_roads_north_wales.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://3.bp.blogspot.com/-ufYicSnAG6s/V3U7WWPaNHI/AAAAAAAAA9A/WV5eJ21aVhUFAPeqoXblINOqACcYMethACLcB/s320/hedgehog_roads_north_wales.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/North_Wales">North Wales</a> &amp; <a href="https://en.wikipedia.org/wiki/Merseyside">Merseyside</a></td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-LLZGqX4U5tM/V3U7bUp328I/AAAAAAAAA9I/jIhPItWgGIEmd5BIDS_2mx9b9VV9H914wCLcB/s1600/hedgehog_roads_south_coast.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://2.bp.blogspot.com/-LLZGqX4U5tM/V3U7bUp328I/AAAAAAAAA9I/jIhPItWgGIEmd5BIDS_2mx9b9VV9H914wCLcB/s320/hedgehog_roads_south_coast.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Isle_of_Wight">Isle of Wight</a> and South Coast of England</td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"><a href="https://4.bp.blogspot.com/-J94_2Ltr62U/V3U7nXhOxaI/AAAAAAAAA9Q/R8I0htxm0Qwe23-fV_idqw0KXnSr8be6gCLcB/s1600/hedgehog_roads_south_ebudes.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://4.bp.blogspot.com/--mEUcX75RKk/V3U7ub-uAfI/AAAAAAAAA9Y/4sij8fuvtG4W8Lpgmtq-jqac3sAjoXjKACLcB/s1600/hedgehog_roads_south_west.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://4.bp.blogspot.com/--mEUcX75RKk/V3U7ub-uAfI/AAAAAAAAA9Y/4sij8fuvtG4W8Lpgmtq-jqac3sAjoXjKACLcB/s320/hedgehog_roads_south_west.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/South_West_England">South West England</a></td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-J94_2Ltr62U/V3U7nXhOxaI/AAAAAAAAA9Q/m_U97yWD7-c27BoN0Ykh1ifk-7jJJQGBQCKgB/s1600/hedgehog_roads_south_ebudes.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://2.bp.blogspot.com/-J94_2Ltr62U/V3U7nXhOxaI/AAAAAAAAA9Q/m_U97yWD7-c27BoN0Ykh1ifk-7jJJQGBQCKgB/s320/hedgehog_roads_south_ebudes.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Central_Belt">Central Belt</a> &amp; Southern <a href="https://en.wikipedia.org/wiki/Inner_Hebrides">Inner Hebrides</a>, Scotland<br /><a href="https://en.wikipedia.org/wiki/Kintyre">Kintyre </a>(bottom left of centre) shows forest tracks erroneously mapped as roads</td></tr></tbody></table><br />Even now there are one or two problems: specifically forestry tracks on Kintyre mapped as unclassified roads (I think I caught most of these and corrected them sometime ago). Some roads on Scottish Islands have a status which is unclear too. However the visualisations of the data do help to highlight such cases.<br /><br />The other thing is that for much of the country the patches of land which are further than 1 km from any road are often easily recognisable to the extent that they can often be readily names. The two big patches in the Peak District are <a href="https://en.wikipedia.org/wiki/Bleaklow">Bleaklow</a> and <a href="https://en.wikipedia.org/wiki/Kinder_Scout">Kinder Scout</a>. So the visualisation also shows other things which may be of interest for other reasons<br /><br /><br />The sad part of this post is that it wouldn't be necessary if hedgehogs were not in a steep decline across Britain.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-8680874859688519691
url: http://sk53-osm.blogspot.com/2016/05/bristol-new-brighton-buildings-from.html
title: Bristol (& New Brighton) Buildings from Lidar
updated: 2016-05-20 16:42:36.469000
published: 2016-05-20 16:42:00
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.622865
updated_at: 2019-10-26 08:30:24.895000
content:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><img alt="West front of Bristol Cathedral" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/44/West_front_of_Bristol_Cathedral.jpg/512px-West_front_of_Bristol_Cathedral.jpg" width="512" /></td></tr><tr><td class="tr-caption" style="text-align: center;">West Front, Bristol Cathedral<br />One of the buildings where we could use Lidar data to enhance its representation in OSM</td></tr></tbody></table>At <a href="http://odcamp.org.uk/">OpenData Camp 3</a> over the weekend I asked <a href="https://twitter.com/murraydata">John Murray</a> if he could give me a set of polylines extracted from the <a href="http://www.geostore.com/environment-agency/survey.html#/survey?grid=SJ39">Environment Agency Lidar Open Data</a>. Do read John's <a href="http://odcamp.org.uk/turning-lidar-data-into-actionable-insight/">amazing post</a> about the tools he has built for doing nifty things with Lidar data: <i>Turning Lidar Data into Actionable Insight</i>.<br /><br />I thought if might be a bit of fun to actually show directly how opendata produced by one of the ODCamp sponsors might end up in OpenStreetMap.<br /><br />In practice John got keen over the lunch break and wrote a bit of code to turn his polylines into polygons. So just before the last session of the day kicked off I had been sent a shape file for the 1km Ordnance Survey gird square where the meeting was taking place.<br /><br />We had one initial teething problem that the data was out by 1 km, but notwithstanding that it was very simple to perform some simple manipulations in the off-line OpenStreetMap editor JOSM.<br /><br />JOSM can read shapefles (and some other geo-formats such as geojson too) and automatically transforms these into OSM elements projected in WGS84. Therefore the additional data manipulation steps were pretty trivial:<br /><ul><li>Select all way elements (using a type:way search)</li><li>Add a source=EA Lidar Open Data tag</li><li>Select all type:relation elements to find multipolygons</li><li>Add building=yes tag</li><li>Select all way elements which were not part of multipolgons (type:way and not child type:relation)</li><li>Add building=yes tag</li><li>Select all way elements again and simplify them.</li></ul>&nbsp;The image below shows how this looks in the editor<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-6Hd-CPPuaME/VzoILMLY4ZI/AAAAAAAAA5s/9VcqaKm1KoQbTFaYtqdS6RweeOd31siAgCLcB/s1600/watershed_in_josm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="289" src="https://2.bp.blogspot.com/-6Hd-CPPuaME/VzoILMLY4ZI/AAAAAAAAA5s/9VcqaKm1KoQbTFaYtqdS6RweeOd31siAgCLcB/s400/watershed_in_josm.png" width="520" /></a></div><br />Now this is all pretty amazing, and if you read John's blog post there's lot more info which can be gleaned. However a close inspection of the data still shows a sizeable number of artefacts which would need cleaning up. Some John has dealt with in the intervening few days, but turning any automatically extracted feature into something of the sort of quality which can be one in OSM is another matter: and is not too dissimilar to the points I made some time ago about <a href="http://sk53-osm.blogspot.co.uk/2016/01/uk-open-data-and-buildings-in.html">OpenMap Local</a>.<br /><br />For me the real advantage is that it's a major step in making if more feasible to use Lidar data to enrich OSM data. For instance data on roof orientations could be combined with the algorithms &amp; crowd-sourced validation methods from OpenSolarMap. I hadn't realised until listening to John's talk just how valuable gable or eaves heights are in building datasets. It certainly persuaded me that they belong in OSM.<br /><br />Another downside is that it takes a whiz like John to create this software and it makes use of a powerful machine, powerful algorithms, optimised hardware &amp; proprietary storage. I have therefore spent a little time this week looking again at what is available in QGIS to do similar (but much less powerful) manipulation of the Lidar data.<br /><br />Basic transformations of Lidar have been described elsewhere (for instance see Chris Hill's <a href="http://chris-osm.blogspot.co.uk/2015/09/extracting-building-heights-from-lidar.html">posts</a>) so I won't dwell on them here. Suffice it to say I presume that the following have been created for a given area:<br /><ul><li>Combined Digital Surface Model (DSM). I usually do this as a virtual time set (can be done directly in QGIS)</li><li>Combined Digital Terrain Model (DTM). As above.</li><li>Delta of the two. DSM-DTM. This gives things (buildings, cars, trees etc) which are elevated above ground level.</li></ul>To get somewhere near what John's approach involves ideally requires:<br /><ul><li>Filtering out shorter objects (mainly cars, garages &amp; some street furniture)</li><li>Filtering out smaller objects (mainly trees)</li><li>Edge detection</li><li>Polygonisation</li></ul>In practice I found it relatively easy to do the first &amp; last and did not find a simple way of doing the other 2 in QGIS (although in part that might be because I'm short of disk right now).<br /><br />The other two can be achieved easily:<br /><ul><li>Filtering by Height: this is merely another raster calculation using the QGIS Raster Calculator. In my <a href="http://www.openstreetmap.org/#map=16/53.4361/-3.0497">test area</a> (New Brighton on the Wirral, OS grid ref SJ3093) most houses are Edwardian and much higher than 3 metres, whereas garages are usually a touch over 2 metres. I therefore used 3 metres as a cut-off.</li><li>Polygonisation. I used the height filtered data directly with the Raster...Conversion...Polygonize option in QGIS. This is a much cruder and more naive method than I was hoping to use, but there it is.</li></ul>I show the results of these steps below (in separate images to allow easier inspection &amp; then combined).<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-kTqdHhbKZRc/Vz8Ll-25OEI/AAAAAAAAA6c/tbZJRu3y-hwSMyW0Z4YLOGYay0hKB8e6wCLcB/s1600/newbrighton_lidar_heights.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://1.bp.blogspot.com/-kTqdHhbKZRc/Vz8Ll-25OEI/AAAAAAAAA6c/tbZJRu3y-hwSMyW0Z4YLOGYay0hKB8e6wCLcB/s320/newbrighton_lidar_heights.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Lidar Height data (DSM-DTM) filtered for &gt;3m</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://4.bp.blogspot.com/-ZV5nJAJhi9E/Vz8LiMuAtRI/AAAAAAAAA6Y/tyJOGpYFe0wibZOmN8HIGj8_Ziau8QNIQCLcB/s1600/newbrighton_building_polygons_only.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://4.bp.blogspot.com/-ZV5nJAJhi9E/Vz8LiMuAtRI/AAAAAAAAA6Y/tyJOGpYFe0wibZOmN8HIGj8_Ziau8QNIQCLcB/s320/newbrighton_building_polygons_only.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Extracted &amp; OSM Building polygons compared<br />(garages are deliberately excluded from OSM data)</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-kbG0TLOaOwY/Vz8LfP4uiBI/AAAAAAAAA6U/uT9KmBd04Ug7Vf5s58JG_22umtChZXUzACLcB/s1600/newbrighton_building_polygons.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://2.bp.blogspot.com/-kbG0TLOaOwY/Vz8LfP4uiBI/AAAAAAAAA6U/uT9KmBd04Ug7Vf5s58JG_22umtChZXUzACLcB/s320/newbrighton_building_polygons.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Height data combined with Polygons</td></tr></tbody></table><br /><br />Firstly it's worth noticing a few features from the raw height data:<br /><ul><li>Most buildings are tall, usually in excess of 8 metres (and probably at least that height at the gables).</li><li>There are a limited number of lower height buildings. The most obvious ones are near the top of the image and include two small factory premises N of the railway &amp; the platform canopy of the railway station. S of these the road bridge over the railway is obvious; and immediately to the SE there are apparently two largish buildings of low height, albeit quite a bit of noise in the height profile. (These are, in fact, Victoria View a development of flats which halted for several years). Further S still there are a small number of bungalows.</li><li>Terraces with a lower rear service area are obvious. </li><li>There are a significant number of linear features above 3 m in height. Most look to be walls, and indeed garden walls in the area tend to be high as most gardens are small &amp; given building heights would tend to be overlooked.</li><li>Isolated trees are obvious in one or two back gardens</li><li>Larger groups of trees are equally obvious along the railway line (&amp; elsewhere)</li><li>Swirly patterns in the 3-4 metre range occur in a number of places. These are mainly scrub (mainly gorse) or shrubberies.</li><li>There are still parked vehicles giving returns in the 3-4 metre height range.&nbsp;</li></ul><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-q-URt4Ly4KQ/Vz8nHuaID4I/AAAAAAAAA6w/Ps8tVg724pgnpzrKIXGN8dPfoJfc95opQCLcB/s1600/IMG_6362-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://1.bp.blogspot.com/-q-URt4Ly4KQ/Vz8nHuaID4I/AAAAAAAAA6w/Ps8tVg724pgnpzrKIXGN8dPfoJfc95opQCLcB/s320/IMG_6362-1.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Edwardian Streets <a href="http://www.openstreetmap.org/#map=18/53.43388/-3.05226">S of Mount Road</a> New Brighton (Dovedale &amp; Langdale Rds)</td></tr></tbody></table><br />I include a couple of photos of streets in the area to help with context. I would recommend strongly <a href="https://www.flickr.com/photos/liverpoolsuburbia/541180697/in/album-72157600356944866/">Russ Oakes' work</a> documenting suburban streets all over Merseyside for a much broader perspective. <br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-d4o5P_KiaYc/Vz8nKimnwhI/AAAAAAAAA60/7H_Mgql7q2gU6eghtTV2W-8LejfWx8qEACLcB/s1600/IMG_6347-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="252" src="https://3.bp.blogspot.com/-d4o5P_KiaYc/Vz8nKimnwhI/AAAAAAAAA60/7H_Mgql7q2gU6eghtTV2W-8LejfWx8qEACLcB/s320/IMG_6347-1.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Junction of <a href="http://www.openstreetmap.org/#map=17/53.43528/-3.05107">Dudley &amp; Hamilton Road</a>s, New Brighton</td></tr></tbody></table><br />Comparing the extracted polygons with OSM (and ignoring some OSM data which is missing) shows:<br /><ul><li>There is a fairly constant offset of OSM data (presumably inherited from the Bing imagery).</li><li>Building footprints are broadly comparable</li><li>Small gaps in terraces are resolve much better by tracing.</li><li>Some detail has not been added to some of the terraces in OSM which are still drawn as plain rectangles.</li><li>It's certainly possible to spot missing features &amp; use Lidar data as an aid to add them in (notably Victoria View, but the W part of the development was started after the Lidar data.</li></ul>Now as for deriving data to enhance OSM there's a fair more bit of processing needed.<br /><br />Absolute building height is relatively easy, one just needs to find the maximum height within a (location corrected) OSM polygon. Generating the other more useful <a href="http://wiki.openstreetmap.org/wiki/Simple_3D_buildings">Simple 3D building</a> (S3DB) tags is rather more involved, and certainly I have the impression that QGIS would be a fairly clunky way to do things. I really hope that some more technically-minded OSM folk can take inspiration from John's ideas and start thinking about tools to mainpulate Lidar data specifically for OSM.<br /><br />There is no doubt that the Environment Agency Lidar data was one of the most significant open data releases last year. Furthermore it is likely that other agencies &amp; local government bodies will make Lidar available more widely in the near future. For instance I believe much data from Kanton Zurich is open, including Lidar. This <a href="http://maps.zh.ch/?topic=LidarZH&amp;scale=6280&amp;x=670560.53&amp;y=250675.15&amp;offlayers=dom2014hillshade%2Cblatteinteilung-lidar%2Cblatteinteilung-dtm%2Cblatteinteilung-labels_lidar%2Cblatteinteilung-labels_dtm&amp;over=HoehenlOverlayZH%2CUpBackgroundZH">example </a>shows the extensive slumping caused by peri- and post-glacial phenomena in the woods near Bergietikon: so this is a reminder that it's not just buildings which are of interest.<br /><br />One last thing to note is that there's lots one can do with this data immediately (the subject of John's original article). Working how to add this data to OSM begins to look not dissimilar to creating authoritative datasets. It's of course worth spending time working out how to do this because once in OSM the data is potentially available for a multitude of purposes.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-3692626232245402959
url: http://sk53-osm.blogspot.com/2016/05/where-have-all-woods-gone-from-google.html
title: Where have all the woods gone from Google Maps?
updated: 2016-05-04 11:45:45.395000
published: 2016-05-04 11:45:00.001000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.625945
updated_at: 2019-10-26 08:30:24.897573
content:
  Very recently there was a nice post by <a href="https://twitter.com/justinobeirne">Justin O'Beirne</a> about the <a href="http://www.justinobeirne.com/essay/what-happened-to-google-maps">cumulative effect of change</a>s to the cartography of Google Maps.&nbsp; <a href="https://twitter.com/richardf">Richard Fairhust</a> summarised his views on twitter:<br /><br /><blockquote class="twitter-tweet" data-lang="en"><div dir="ltr" lang="en">On that <a href="https://twitter.com/justinobeirne">@justinobeirne</a> article: 1. Google Maps becoming less “map”, more “backdrop for search results”. Similar: road colours white/grey.</div>— Richard Fairhurst (@richardf) <a href="https://twitter.com/richardf/status/727190431260377088">May 2, 2016</a></blockquote><script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script> <br /><blockquote class="twitter-tweet" data-lang="en"><div dir="ltr" lang="en">On that <a href="https://twitter.com/justinobeirne">@justinobeirne</a> article: 1. Google Maps becoming less “map”, more “backdrop for search results”. Similar: road colours white/grey.</div>— Richard Fairhurst (@richardf) <a href="https://twitter.com/richardf/status/727190431260377088">May 2, 2016</a></blockquote><script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script> <br /><blockquote class="twitter-tweet" data-lang="en"><div dir="ltr" lang="en">On that <a href="https://twitter.com/justinobeirne">@justinobeirne</a> article: 1. Google Maps becoming less “map”, more “backdrop for search results”. Similar: road colours white/grey.</div>— Richard Fairhurst (@richardf) <a href="https://twitter.com/richardf/status/727190431260377088">May 2, 2016</a></blockquote><script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script> This is just my (very) minor contribution to the discussion.<br /><br />The <a href="http://bsbi.org/">Botanical Society of Britain &amp; Ireland</a> (BSBI) uses Google Maps as the background to their <a href="http://bsbi.org/maps/?taxonid=2cd4p9h.cba">maps of plant distributions</a>. Over the past couple of weeks I've been using it a lot because I've been interested in two things:<br /><ul><li>Where I might fund particular plants relatively close to where I live;</li><li>Which plants I see might be of interest to the county recorders.</li></ul>As at this time of year many of the botanical highlights are to be found in ancient woodlands it's damn useful to see where the woods are when assessing the BSBI records. That's why I noticed woods disappearing from the Google cartography as one zooms in.<br /><br />This screenshots shows successive zooms of an area in central Nottinghamshire which includes Clumber Park an two old woods, Gamston &amp; Eaton Woods. The latter two are centre right above the village of Askham.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-2Jsl6YnmQi8/Vynd61-HKrI/AAAAAAAAA5U/A8-E7LRgE5Mva1S2lursoFUJThpWlfK5QCLcB/s1600/google_woods1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="https://2.bp.blogspot.com/-2Jsl6YnmQi8/Vynd61-HKrI/AAAAAAAAA5U/A8-E7LRgE5Mva1S2lursoFUJThpWlfK5QCLcB/s320/google_woods1.png" width="520" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://3.bp.blogspot.com/-RpL6fVnm588/Vynd_Hsr7iI/AAAAAAAAA5Y/1IeO4pGzExY4aZOQInSxQUKHmvZNrBztwCLcB/s1600/google_woods2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="https://3.bp.blogspot.com/-RpL6fVnm588/Vynd_Hsr7iI/AAAAAAAAA5Y/1IeO4pGzExY4aZOQInSxQUKHmvZNrBztwCLcB/s320/google_woods2.png" width="520" /></a></div><br />All woodland just disappears between these two zoom levels.<br /><br /><div style="text-align: left;">Here's the active map so one can play with zooming in &amp; out.</div><br /><div style="text-align: center;"><br /><iframe allowfullscreen="" frameborder="0" height="450" src="https://www.google.com/maps/embed?pb=!1m14!1m12!1m3!1d76338.19460595223!2d-0.9269806048847737!3d53.27934919391607!2m3!1f0!2f0!3f0!3m2!1i1024!2i768!4f13.1!5e0!3m2!1sen!2suk!4v1462360791321" style="border: 0;" width="600"></iframe></div><br />Losing woods at high zoom levels is another example of loss of functionality. In practice it makes the maps layer useless for interpreting botanical data: I have to resort to using the satellite layer. Even that is not always easy because sometimes fields also appear dark green.<br /><br />Google does use a couple of other green shades for things like parks, golf courses, and possibly nature reserves (see Sherwood Forest NNR near Edwinstowe). I don't know if these come on and off in a similar arbitrary pattern.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-5972310337082154396
url: http://sk53-osm.blogspot.com/2016/02/distribution-of-contributions-in.html
title: Distribution of Contributions in Volunteer-generated Datasets : Gall or Fruit Fly Records
updated: 2016-02-14 14:43:16.935000
published: 2016-02-14 14:43:00.001000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.628664
updated_at: 2019-10-26 08:30:24.900224
content:
  I remarked in my <a href="http://sk53-osm.blogspot.co.uk/2015/04/interviewed-by-opencagedata.html">OpenCageData interview</a> that I see many similarities between biological recording and OpenStreetMap contributions. Indeed, I've had some interesting discussions about this with <a href="https://povesham.wordpress.com/">Prof. Muki Hakaly</a> at UCL. Muki's group now do extensive research across the gamut of activities which fall under the rubric of "citizen science", so I'm hopeful that they will elucidate which features are common across this spectrum.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/5817646991/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="Chaetorellia jaceae f : 5532b"><img alt="Chaetorellia jaceae f : 5532b" height="375" src="https://farm6.staticflickr.com/5240/5817646991_49307c1d98.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A female <i><a href="https://en.wikipedia.org/wiki/Chaetorellia">Chaetoraellia </a>jaceae</i>, a tephritid fly whose larvae feed on Knapweed.<br />Photo: (c) mausboam, Flickr.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> Basically, we know that there is a very long tail of smaller contributions to OpenStreetMap. Both <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_The_Long_Tail_of_OpenStreetMap">Harry Wood</a> and <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_More_than_Just_Data">Frederick Ramm</a> gave presentations on aspects of this at SotM-14 in Buenos Aires, and Richard Fairhurst also touched on this at SotM-US in 2013. Very recently <a href="http://www.openstreetmap.org/user/marczoutendijk/diary/37902">Marc Zoutendijk</a> has used data collected by Pascal Neis to examine a cohort of new Dutch OpenStreetMap contributors from 2014 and 2015.<br /><br />The usual hope expressed by people doing this type of analysis with OSM data is that by better understanding of these contributions we can improve the number of people who continue to contribute after the initial sign-up and first edit.<br /><br />My perspective is slightly different, because it is coloured by knowledge of the much longer history of biological recording.<br /><br /><br /><a name='more'></a>In Britain this can roughly be dated to the 17th century, and <a href="https://en.wikipedia.org/wiki/John_Ray">John Ray</a>'s publication of a flora of the <a href="https://books.google.co.uk/books?id=hz0-AAAAcAAJ&amp;dq=plantarum+circa+Cantabrigiam&amp;source=gbs_navlinks_s">Cambridge area</a>. In the early days compilation of records relied on exchanges of letters, but by the 18th century the collation of data had grown to the extent that for many counties it was possible to produce a flora for the local area. For instance in Nottinghamshire, the first flora was produced by <a href="http://www.biodiversitylibrary.org/item/188261#page/245/mode/1up">Deering </a>in 1738, followed by a second by Ordanyo in 1807. (In fact there was another written in the 1830s, followed by a gap of over&nbsp; hundred years until the last county flora was written by the Howitts in 1963).&nbsp; The sheer scope of this literature can be seen in Tim Rich's digitised copy of Simpson's 900 odd page <i><span style="font-size: 12pt;">A Bibliographical Index of the British Flora</span></i><span style="font-size: 12pt;"> (see the <a href="http://archive.bsbi.org.uk/index.html">BSBI website</a>). Here's a small extract of works which mention aspects of the Nottinghamshire Flora from around 1750-1825</span><br /><span style="font-size: 12pt;"><br /></span><br /><blockquote class="tr_bq"><span style="font-size: 12pt;"><span style="font-size: x-small;">(Nottinghamshire) A catalogue of plants ... about Loughborough; R. Pulteney 1747; Manuscript Leicester Museum 1749, and library Linnean Society</span></span><br /><span style="font-size: 12pt;"><span style="font-size: x-small;">Nottingham; Historical account of the town of, C. Deering 1751, 90.<br />(Nottinghamshire) An account of some of the more rare English plants observed in Leicestershire; R. Pulteney Philosophical Transactions XLIX 2 (1757) 803.<br />(Nottingham) A catalogue of some of the more rare plants found in the neighbourhood of Leicester, Loughborough and in Charley Forest; R. Pulteney Philosophical Transactions XLIX (1757) 803, 866; and in ‘The history ... of Leicester', I. Nichols I (1795) clxxvii.<br />Nottinghamshire. Plantae Cantabrigienses; T. Martyn 1763, 83, [from Deering's Catalogue].<br />Nottinghamshire The present state of all nations; T. Smollett II (1768) 408.<br />Nottingham. Description of England and Wales; [Society of Gentlemen] VII (1769) 135.<br />Nottingham. The complete English traveller; N. Spencer 1771, 495; 1773, 495.<br />(Nottinghamshire) Manuscript notes in Ray's Synopsis iii; J. Lightfoot [died 1788] &amp; J. Hill [died 1775] library Botany Department Oxford.<br />Nottingham; Topographical and statistical description of the county of, G. A. Cooke [c. 1802-10] 121.<br />Nottinghamshire. Botanist's guide; D. Turner &amp; L. W. Dillwyn II (1805) 482.<br />(Nottinghamshire) Midland flora; T. Purton 1817 2 volumes; appendix, parts 1, 2 &amp; 3 (in 2 parts) 1821.<br />Nottinghamshire. The scientific tourist in England ...; T. Walford II (1818).<br />Nottinghamshire. The new British traveller; J. Dugdale IV (1819) 6.<br />Nottinghamshire; Botanical calendar for, T. Jowett 1826. By "Il Rosajo" in local paper*.</span></span>&nbsp;  </blockquote>Perhaps my favourite quote from this period is about the Maiden Pink <i>Dianthus deltoides</i>, for instance in an <a href="https://ebooks.adelaide.edu.au/c/camden/william/britannia-gibson-1722/p14010002.html">english language</a> edition of <a href="https://en.wikipedia.org/wiki/William_Camden#Britannia">Camden's Britannia</a> around 1722 :<br /><blockquote>John Ray says, " I find this to be the same pink  which groweth so plentifully by the road side on the sandy hill you ascend going from  Lenton to Nottingham." <i>Catalogus Plantarum</i> 2 ed., 1677. p. 57.  </blockquote><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-7S7rC4EQcGI/VsCLbOtus1I/AAAAAAAAA30/ZDwtW3kqk50/s1600/IMG_5414-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://2.bp.blogspot.com/-7S7rC4EQcGI/VsCLbOtus1I/AAAAAAAAA30/ZDwtW3kqk50/s320/IMG_5414-1.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Dianthus_deltoides">Maiden Pink</a> in Nottingham,<br />sadly not native but an escape from a green roof.<br />Photo: copyright the author</td></tr></tbody></table><br /><br />Ray's correspondence has more (gruesome detail):<br /><br /><div style="text-align: center;"><br /><iframe frameborder="0" height="430px" src="https://archive.org/stream/correspondenceof48rayj?ui=embed#page/48/mode/1up" width="480px"></iframe></div><br /><div style="text-align: justify;">These details just emphasise how much has changed: the sandy hill between Lenton &amp; Nottingham is a four-lane road, Derby Road. This <a href="http://www.mapillary.com/map/im/6Got-g2pXQmqT0a8ngdnKw/photo">Mapillary </a>sequence is roughly in the relevant location.</div><br /><iframe frameborder="0" height="580px" src="http://www.mapillary.com/jsapi/?image=6Got-g2pXQmqT0a8ngdnKw&amp;showMap=false" width="480px"></iframe> Indeed the area was long known as Lenton Sands, although this name is, I think, falling into disuse.<br /><br />I wasn't not sure where the gallows were located, but a quick web search shows that <a href="http://www.ournottinghamshire.org.uk/page_id__943.aspx">they</a> were close to what is now the junction of <a href="http://www.openstreetmap.org/#map=16/52.9642/-1.1533">Forest and Mansfield Roads</a>.<br /><br />Another evocative historic plant location is <a href="https://en.wikipedia.org/wiki/Nottingham_Castle">Nottingham Castle Rock</a>. This is the type locality for the <a href="https://en.wikipedia.org/wiki/Silene_nutans">Nottingham Catchfly, <i>Silene nutans</i></a>, but it is long since extinct in the area. However, at the foot of the rock is another plant location known since John Ray's time: here Alexanders, <a href="https://en.wikipedia.org/wiki/Smyrnium_olusatrum">Smyrnium olusatrum</a>, still grows in a small patch along Peveril Drive.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-bPTk-eG70wk/VrcIVjDsKmI/AAAAAAAAA2o/I9eAnoswWMU/s1600/IMG_0148a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://2.bp.blogspot.com/-bPTk-eG70wk/VrcIVjDsKmI/AAAAAAAAA2o/I9eAnoswWMU/s320/IMG_0148a.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Foot of Castle Rock, Nottingham.<br />The gates are the old entrance gates to The Park Estate.<br />The green plants behind the left-hand gate are Alexanders.</td></tr></tbody></table>These little examples are just some of the one's most familiar to me. Throughout the British Isles there are thousands of such places where rare plants grow which have been known for hundreds of years. Occasionally the places get lost, and then by chance, or as the result of diligent research and field work, get relocated (see <a href="http://www.museumwales.ac.uk/articles/2009-04-01/The-rarest-plants-in-Wales/">here</a> for examples from Wales).<br /><br />It's not just plants either, a friend, David Brown, runs a regular field course in Scotland called <a href="http://staging.field-studies-council.org/individuals-and-families/courses/2015/kd/special-spring-moths-52071.aspx"><i>Special Spring Moths</i></a>. He takes the participants to locations which have been known for at least 170 years to see such exotica as the <a href="http://www.ukmoths.org.uk/species/brachionycha-nubeculosa">Rannoch Sprawler</a> and the <a href="http://www.ukmoths.org.uk/species/lycia-lapponaria/female/">Rannoch Brindled Beauty</a>.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ABrachionycha_nubeculosa2.JPG" style="margin-left: auto; margin-right: auto;" title="By Adam Furlepa (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons"><img alt="Brachionycha nubeculosa2" src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Brachionycha_nubeculosa2.JPG/512px-Brachionycha_nubeculosa2.JPG" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A Rannoch Sprawler in Poland, a rather better photo than <a href="https://flic.kr/p/4Dkhkn">mine </a>of a Scottish moth.<br />By Adam Furlepa&nbsp; <a href="http://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>, <a href="https://commons.wikimedia.org/wiki/File%3ABrachionycha_nubeculosa2.JPG">via Wikimedia Commons</a></td></tr></tbody></table>It should be obvious by now, as to why naturalists are often keenly interested in topography. Rarer species highlight why accurate records and a means of sharing them have been so important to British naturalists for such a long time.<br /><br />Until the mid-20th century the principle means of record keeping were personal card indexes (David Brown still does things this way). Collation of records over a particular area would be entrusted to a particularly knowledgeable and enthusiastic individual. The ideal was that these records would be periodically consolidated and published, either as a journal article, or, for larger groups, as a book.<br /><br />In 1964 this changed when the <a href="https://en.wikipedia.org/wiki/Biological_Records_Centre">Biological Record Centre</a> was set-up and records started to be computerised. Slightly earlier, in 1962, the first <a href="https://en.wikipedia.org/wiki/Atlas_of_the_British_Flora">Plant Atlas</a> of Britain &amp; Ireland was published. I think this was the first major publication to organise records on the basis of Ordnance Survey grid squares, which had only appeared relatively recently on consumer, rather than military, map products (the <a href="http://wiki.openstreetmap.org/wiki/New_Popular_Edition">New Popular Edition</a>). Since then major atlases have appeared for a number of groups, with birds and plants each having several editions.<br /><br />For these better known groups, atlas data pertains to surveys carried out in a defined period. For instance the <a href="http://www.bto.org/volunteer-surveys/birdatlas">last Bird Atlas</a> surveys ran from 2007-2011, and the next plant atlas recording period runs until <a href="http://www.bsbi.org.uk/atlas_2020.html">2020</a>. For most insects (the exceptions being <a href="http://butterfly-conservation.org/111/butterflies-for-the-new-millennium.html">Butterflies </a>and <a href="http://www.british-dragonflies.org.uk/content/national-dragonfly-atlas">Dragonflies</a>) there are just not enough people interested, or with the specialist knowledge, for such an effort. For these groups, all known records, however old, need to be used. There are currently around 100 beetles known from Nottinghamshire, where the last known sighting was pre-1916. But they can be re-found, as was the nationally rare Hazel Pot Beetle, <a href="https://en.wikipedia.org/wiki/Cryptocephalus">Cryptocephalus </a>coryli, by <a href="http://www.eakringbirds.com/eakringbirds6/insectinfocuscryptocephaluscoryli.htm">Trevor &amp; Dilys Pendleton</a> in 2008. So these older records can still be totally relevant today.<br /><br />Now to return to the long tail phenomenon.<br /><br />The point about the various BRC Atlas initiatives is that, in addition to all the information about plants and animals, they also form a range of large and valuable datasets for understanding aspects of how people contribute data in this type of undertaking. (Of course they don't answer the <b>WHY?</b>, but the National Biodiversity Network has recently <a href="http://nbn.org.uk/News/Latest-news/NBN-Recorder-Motivation-Research-summary-findings.aspx?utm_source=eshot&amp;utm_medium=email&amp;utm_campaign=NBNeNewsFebruary2016_1">surveyed contributor</a>s, and answers there may be of interest to people concerned with similar issues with OpenStreetMap).<br /><br />I, of course, don't have access to any of these large datasets at a granularity at which one can ask questions about relative frequency of contributions. However, I have contributed to a small niche dataset, that for <a href="https://en.wikipedia.org/wiki/Tephritidae">Tephritid </a>flies (often called Picture-winged Flies, or Gall Flies, but more widely called Fruit Flies; unfortunately in Britain fruit fly usually means <i>Drosphila melanogaster</i> which belongs to a <a href="https://en.wikipedia.org/wiki/Drosophilidae">different family)</a>. These are small, but distinctive flies most of which spend their larval phase feeding on fruits of various plants, some galling their hosts. They are common, but not well recorded.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/6016864881/in/photolist-9S5XLt-bzr1nw-a7icYc-a7icpT-a7m4fs-a7m47h-aaG1iF-aaJQdE-aaG1oB-9S5XHK-9S8SNj-9S8SCU-5PTALX-5f2qr6-56qHaA-51L6dc-51Qj21-4W3aT2-4W7pfd-LVj3Z" style="margin-left: auto; margin-right: auto;" title="Tephritis bardanae (m)"><img alt="Tephritis bardanae (m)" height="640" src="https://farm7.staticflickr.com/6005/6016864881_181814a7a0.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Tephritis_bardanae"><i>Tephritis bardanae</i></a> on <a href="https://en.wikipedia.org/wiki/Arctium_tomentosum"><i>Arctium tomentosum</i></a>, <a href="https://en.wikipedia.org/wiki/Puchberg_am_Schneeberg">Puchberg-am-Schneeberg</a>, Austria, 2011<br />Taken the day before SotM-EU 2011.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> Laurence Clemons, the co-ordinator of the scheme, is a near contemporary from university. He has run it for over 30 years, combining it with his day job of teaching. From time to time he circulates updated maps for each tephritid species. Fortunately for me he also includes a list of all recorders with the number of records and species submitted. The records go back to the 19th century, so although the data set is relatively small, 3 572 observations,&nbsp; it has both broad temporal and geographical extent. I have used this list from around 2008-9 to look at recorder contribution distributions.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/5979022584/in/photolist-9S5XLt-bzr1nw-a7icYc-a7icpT-a7m4fs-a7m47h-aaG1iF-aaJQdE-aaG1oB-9S5XHK-9S8SNj-9S8SCU-5PTALX-5f2qr6-56qHaA-51L6dc-51Qj21-4W3aT2-4W7pfd-LVj3Z" style="margin-left: auto; margin-right: auto;" title="Terellia tussilaginis (m) : 9037a"><img alt="Terellia tussilaginis (m) : 9037a" height="375" src="https://farm7.staticflickr.com/6142/5979022584_c6db6cbf41.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A male of <i><a href="https://en.wikipedia.org/wiki/Terellia">Terellia </a>tussilaginis</i> on <a href="https://en.wikipedia.org/wiki/Arctium_minus"><i>Arctium minus</i></a>, Nottingham.<br />Despite the name, these insects only feed on Burdocks.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> A quick scan of the people contributing records shows many entomologists who are specialists in other fields, for instance Raymond Uffen, a lepidopterist with 1 record, Brian Spooner, former head of Mycology at Kew Gardens, with 2 records. The people with lots of records are well known dipterists: <a href="http://www.omnesamici.co.uk/CGSthenandnow/WhiteleyDerek.html">Derek Whiteley</a> (371 records) from the Sheffield area, <a href="http://www.stevenfalk.co.uk/">Steven Falk</a> (865) from Warwickshire, John Coldwell (170) from Barnsley. Quite a few are museum professionals: Bill Ely (161), Jerry Bowdrey (184). Many of the recorders with a small number of records will be general naturalists who happen to come across galls, flies or leaf mines in the course of other surveying work.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-x4rRxoo6e8c/Vrcckr9AzcI/AAAAAAAAA3Q/t80tIzmNl8s/s1600/carr_I_p468_1L.tif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="https://1.bp.blogspot.com/-x4rRxoo6e8c/Vrcckr9AzcI/AAAAAAAAA3Q/t80tIzmNl8s/s320/carr_I_p468_1L.tif" width="422" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A page from Carr (1916) show early records of Tephritidae in Nottinghamshire<br />Not all of these records have found there way to the national scheme.</td></tr></tbody></table>I don't recognise many names of historical, rather than active recorders, but two stand out: J.L. Carr (1 record), and J.W. Saunt (209), see short biographies at <a href="http://www.coleopterist.org.uk/biogdict/c.htm">The Coleopterist</a>. Carr was the author of the formidable Invertebrate Fauna of Nottinghamshire (1916), keeper of the Nottingham Natural History Museum and Professor of Zoology. I know less of Saunt, other than he contributed many records to Carr's book, but he was apparently a car-body builder in Coventry.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://4.bp.blogspot.com/-1B_3ExGQjlA/VrcbPDhr0nI/AAAAAAAAA3E/PiTxmuu4u9k/s1600/tr_graph3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="342" src="https://4.bp.blogspot.com/-1B_3ExGQjlA/VrcbPDhr0nI/AAAAAAAAA3E/PiTxmuu4u9k/s320/tr_graph3.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Records / recorder GB &amp; Ireland Tephritid Recording Scheme 2008</td></tr></tbody></table><div class="separator" style="clear: both; text-align: justify;"><br /></div>I took the list of records from Laurence's PDF and lightly munged the data into a CSV file which I then pulled into R to plot some histograms. (As usual I find I always have to consult a few webpages to even get started again with R after a break of a few weeks).<br /><div class="separator" style="clear: both; text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;">Of course, and entirely as expected, the histogram shows an exponential decay of numbers of recorders against records. This is exactly like the graph which Marc created which set me of looking at these numbers. The extreme outlier with nearly 2000 records, is, again, as might be expected, Laurence Clemons. No-one devotes their leisure activity to running a recording scheme unless it is a passion. Also note that 2000 records implies considerably less than 100 a year: each record probably represents significant effort.</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;">If I change the size of bins and exclude the more extreme outlier values, the graph looks remarkably the same! Below I show the graph with 50 bins for the visible range with an upper limit of 200 and then 50.</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-JKCY2sghwJc/VrcfbQDTReI/AAAAAAAAA3c/CeBVfgwBSew/s1600/tr_graph4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="342" src="https://3.bp.blogspot.com/-JKCY2sghwJc/VrcfbQDTReI/AAAAAAAAA3c/CeBVfgwBSew/s320/tr_graph4.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The same graph as above, but for recorders with under 200 records</td></tr></tbody></table><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-qwB1SwpXc0w/VrcfdARtAzI/AAAAAAAAA3g/e3d--id8TS4/s1600/tr_graph5.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="342" src="https://3.bp.blogspot.com/-qwB1SwpXc0w/VrcfdARtAzI/AAAAAAAAA3g/e3d--id8TS4/s320/tr_graph5.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The same graph as above, but for recorders with under 200 records<br />I'm in the 16 records bin.</td></tr></tbody></table><div class="separator" style="clear: both; text-align: justify;">This is only one dataset. I'm pretty confident that the same patterns will be seen in other atlas datasets, and, as I stated at the outset, in other citizen science datasets. It's pretty much the same pattern we see when we look at OSM contributions.</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;">For me the main point about this is to emphasise that there is much to learn from the long experience of Biological Recording. I think the idea that somehow we can do magical things to change the shape of the contribution curve is belied by these other data collection experiences. Indeed, the co-ordinator of the fantastically successful <a href="http://www.earthwormsoc.org.uk/earthworm-identification/recording-earthworms">National Earthworm Recording</a> scheme <a href="https://www.timeshighereducation.com/comment/6510#comment-6510">wrote recently</a> that "I speak from experience when I say that teaching someone to identify a  group does not make them record (and build up the necessary experience  to become an expert)". Therefore we should accept that it is most likely a reality which we need to recognise.</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;">This does not of course mean that we should cease in efforts to make OpenStreetMap look: </div><ul><li>&nbsp;<b>important </b>(for instance as Missing Maps &amp; HOT have done most successfully);</li><li>&nbsp;<b>useful </b>(Richard Fairhurst's 'driven by cyclist' <a href="https://lists.openstreetmap.org/pipermail/talk-us/2015-June/014991.html">idea</a>);</li><li>&nbsp;<b>fun </b>(it is, really!);</li><li>&nbsp;<b>interesting </b>(that too); </li><li>&nbsp;or just providing <b>an excuse to get outside</b> (.</li></ul><div class="separator" style="clear: both; text-align: justify;">OSM is all of these things; it's also <b>educative</b>, a way to <b>meet like-minded folk</b>; a way of learning <b>new skills</b> (mainly in informatics, but also observational). Nor is this pattern a reason not to strive for a more diverse community of contributors. In fact, the very likelihood that mappers "<a href="https://www.cl.cam.ac.uk/~dh475/papers/icwsm13.pdf">are made not created</a>" tells us that new contributors are as important as ever.</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;">OSM is different in one other important way from biological records: there is a founder effect.&nbsp;</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;">If I see a bird in one place today and the same species tomorrow both are useful records. In OSM if something is already mapped one can't contribute it again (although one can alter what already exists). Over time the really easy things get mapped leaving those that are harder, more tedious or just less useful. This probably doesn't matter too much for the already engaged, but it does possibly put a limit on what a newcomer might feel able to do. However, to date, there is no sign of this happening: see Simon Poole's recent <a href="http://www.openstreetmap.org/user/SimonPoole/diary/36408">diary post</a> on the subject.</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;">On a more general level, as I said at the outset of this piece, I hope to see some much more detailed and meticulous academic research in the near future. There are now hundreds of different citizen science datasets which can be examined across a range of different domains and types of data acquisition.</div><div class="separator" style="clear: both; text-align: justify;"><br /></div><br />


guid: tag:blogger.com,1999:blog-2082422783692631078.post-2744274659797337751
url: http://sk53-osm.blogspot.com/2016/01/uk-open-data-and-buildings-in.html
title: UK Open Data and Buildings in OpenStreetMap
updated: 2016-01-18 12:38:54.485000
published: 2016-01-18 12:38:00.002000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.631617
updated_at: 2019-10-26 08:30:24.903100
content:
  I've finally (after 8 months) got around to looking  at the OpenMap Local buildings. This new dataset was launched at the first OpenDataCamp, and I've had the <a href="http://www.cucaera.co.uk/grp/?refs=SU&amp;map=road&amp;vcs=true&amp;graticule=true&amp;nsgrids=false&amp;zoom=9&amp;lat=51.31580815769595&amp;lon=-1.6900517278851757">SU</a> 100 kilometre square data on the PC since then (it's contains Southampton, where Ordnance Survey are based). I use Meridian 2 OS Open Data regularly and extensively, but these days don't make much use of the larger scale vector data.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22949197830/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Nottingham City Centre: OSM/OSGB Building Comparison"><img alt="Nottingham City Centre: OSM/OSGB Building Comparison" height="354" src="https://farm6.staticflickr.com/5624/22949197830_6c43f42980.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of Building polygons for Central Nottingham<br />OSM has more detail and does not merge discrete buildings.<br /><span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database right 2015. Image CC-BY-SA, the author.</span></td></tr></tbody></table>I needed them for something else which  caused me to download the <a href="http://www.cucaera.co.uk/grp/?refs=SK&amp;map=road&amp;vcs=true&amp;graticule=true&amp;nsgrids=false&amp;zoom=8&amp;lat=53.17694393740821&amp;lon=-2.071888715440977">SK</a> data. Co-incidentally Christian Ledermann had asked on <a href="https://lists.openstreetmap.org/pipermail/talk-gb/2015-November/018049.html">talk-gb</a> about using this data to add buildings to OpenStreetMap for <a href="https://en.wikipedia.org/wiki/Newark-on-Trent">Newark-on-Trent</a>. A little earlier the Environment Agency had released Lidar data for England, and this is also useful as input for mapping buildings.<br /><br /><h3>OpenMap Local </h3>Apart from the area I originally  needed which were in <a href="http://www.cucaera.co.uk/grp/?refs=SK41&amp;map=road&amp;vcs=true&amp;graticule=true&amp;nsgrids=false&amp;zoom=12&amp;lat=52.7390913199615&amp;lon=-1.3864769678061384">SK41</a> (no buildings in OSM), I've also looked at  areas which I know much better &amp; compared some selected areas where  we have good building coverage around Nottingham. The comparisons I made are shown visually, with my main observations summarised at the end. Note that comparisons have not been made on any systematic basis.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22648540073/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="uon_univertiy_park"><img alt="uon_univertiy_park" height="256" src="https://farm6.staticflickr.com/5642/22648540073_cc0e781001.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">University Park, University of Nottingham<br />an area of predominantly large academic buildings.<br />OpenStreetMap and OS OpenMap are largely in agreement: the minor differences applying to newer buildings which post-date the Bing imagery.<br /><span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database  right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data  from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database  right 2015. Aerial Imagery via Bing, (c) as in image. Image CC-BY-SA, the author.</span></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> <br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22648540053/in/photostream/" style="margin-left: auto; margin-right: auto;" title="uon_science_city_buildings"><img alt="uon_science_city_buildings" height="256" src="https://farm1.staticflickr.com/639/22648540053_2c1591a02a.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The Science City part of the University Park campus.<br />A new large lecture theatre block is not present in OpenMap data, and the outline of the building top centre (Tower Building) is over-simplified.<br /><span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database  right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data  from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database  right 2015. Image CC-BY-SA, the author.</span></td><td class="tr-caption" style="text-align: center;"></td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> <br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22880884667/in/photostream/" style="margin-left: auto; margin-right: auto;" title="newark_buildings2"><img alt="newark_buildings2" height="256" src="https://farm6.staticflickr.com/5734/22880884667_1b6ac26674.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Central Newark. OpenMap vs Lidar.<br />Many instances of building merging &amp; over-simplification are apparent here, notably with the outline of the parish church.<br /><span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database  right 2015, Lidar data  from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database  right 2015. Image CC-BY-SA, the author.</span></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> <br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22880884627/in/photostream/" style="margin-left: auto; margin-right: auto;" title="newark_buildings1"><img alt="newark_buildings1" height="256" src="https://farm6.staticflickr.com/5638/22880884627_4ac089031d.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Newark-on-Trent, residential areas, Showing inconistency in size for similar houses, and merging of terraced housing.<br /><span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database  right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data  from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database  right 2015. Image CC-BY-SA, the author.</span></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> I have not made  systematic comparisons, but these are my main observations (in brackets the 1km grid square where I've noted any particular issue): <br /><ul><li><b>Best for larger buildings</b>.  The data seem much more reliable (actually matching building footprints  fairly well) for larger buildings. Even for large detached houses I  would regard the data as unreliable: on our road of 40 detached houses,  at least 16 are represented as terraces (SK5439). Similar artefacts occur in  other areas with detached houses: apparently caused when a garage is  close to both houses. Smaller houses are inherently simplified: no  better than drawing one in JOSM and then copying the outline in fact.</li><li><b>Building fusion</b>. This is particularly clearly seen in the city centre image, where a whole block of buildings has been simplified to a single building (centre of image), but also occurs in suburban housing (see above).</li><li><b>Inconsistency in geometry simplification</b>. This is most noticeable in the city centre. (SK5739). For instance compare the OSM and the OpenMap Local outlines for St Peter's Church (bottom right in map above). In OpenMap Local the church is just shown as a rectangle, whereas in practice it is more complex. Modern buildings on the Jubilee Campus of Nottingham University are generally shown with more detail.</li><li><b>Inconsistency in building size</b>. In SK5439 there are a very large number of houses which were identical when built. However, in the OpenMap Local they are often of different sizes. (This is also probably true of OSM, if buildings have not been created by duplication). </li><li><b>Voids</b>. Gaps between closely packed buildings in the city centre appear slightly arbitrary in both placing and whether such a void exists or not.</li><li><b>Some selection inconsistency</b> with small size buildings. Only 2 garages are shown in an area of around 500 houses. With OSM the figure is nearer 200+.   (SK5439)</li><li><b>Demolished buildings</b>.  Whilst I would not expect the data to show the building demolished in  the past month, I would expect it to not show one demolished 2 years  ago, and I would certainly expect it not to show one demolished in 1970  (although MasterMap shows <a href="http://sk53-osm.blogspot.co.uk/2011/01/how-good-is-mastermap.html">this too</a>).   (SK5439)</li><li><b>Better locational accuracy</b>.  If using the full transform it may be useful to take advantage of the  better locational accuracy of this data. In the main OSM buildings are  rarely more than 3 m displaced from the OS OpenMap Local.  (SK5439) In general the more recently mapped buildings in Nottingham  city centre have better locational accuracy than this (SK5739).</li></ul>Taken together, my use of this directly within OSM would be along the following lines :<br /><ul><li><b>Selective  transfer of larger buildings</b> (schools, offices, public buildings,  factories, warehouses, larger shops) on a case-by-case basis from a  shapefile to a JOSM editing layer, or to Potlatch 2. Some minor refinement will probably  be needed (for instance a university building here has long narrow  courtyards which act as light wells which are not shown in OpenMap  Local.</li><li><b>Only use</b> it for <b>houses </b>and similar when shapes are <b>very  simple</b> and everything has been double checked, at the very least,  against aerial imagery. For simple shapes it's as quick to draw &amp;  copy in JOSM anyway. A similar principle holds for more complex building shapes on  modern estates, where one building can be cloned.</li><li><b>Watch out </b>for demolished buildings. This requires not just checking against Bing/MapBox imagery, but some local knowledge for sense checking.</li></ul><h3>Environment Agency Lidar Data</h3>Another source of building data is the recently released Environment Agency Lidar data. This does not cover the whole country, and in many places may only be at 1 or 2 m resolution. It may also be quite old. However, because it does not suffer from parallax artefacts it can be used in conjunction with both aerial imagery (whether from Bing, MapBox or more local sources) and OS OpenData. I have provided examples from Nottingham, Newark, and Melton Mowbray of this data, combined with one or more of OSM buildings data, OS OpenMap or Bing aerial imagery.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://wiki.openstreetmap.org/w/images/a/a2/Melton_dsm.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://wiki.openstreetmap.org/w/images/a/a2/Melton_dsm.png" height="285" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Melton_Mowbray">Melton Mowbray</a>. EA Lidar DSM (1m) overlaid on OSM. <br />The Lidar data was used to refine the OSM building outlines<br />which originally were traced from OS StreetView as block-sized polygons.<br />(see commentary)</td></tr></tbody></table>Melton Mowbray illustrates many of the benefits of Lidar data. It is a fairly typical country town, with many of the buildings in the town centre ranging in age from 10 to 500 years old. Many extend back from the street in a series of outbuildings (e.g., stables) which have eventually been incorporated into the main building, but this process leaves lots of small courtyards, service yards, etc which are more or less impossible to discern on aerial imagery.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/24431405886/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Butter Cross on Market Place, Melton Mowbray"><img alt="Butter Cross on Market Place, Melton Mowbray" height="375" src="https://farm2.staticflickr.com/1640/24431405886_024bc6c5c8.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Butter Cross in Market Place, Melton Mowbray<br />Despite the different styles &amp; ages of the buildings, several have long ranges at the rear.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> By doing a street-level ground survey one can identify which buildings are distinct on the street front. Lidar than helps to construct a building outline which is consistent with this. I surveyed the cetre of Melton in September, and this was the first place where I used Lidar data to aid in the interpretation of aerial imagery. In this case I find it essential to have adequate street level pictures to be able to relate to the aerial imagery: most useful are the presence and distribution of chimneys: because they throw shadows they are often visible even on poor quality imagery.<br /><br />The Lidar data also allows one to do some other things: notably find building heights. I've done this for a 1980's estate on the edge of Maidenhead: particularly easy as the residential buildings fall into a small number of categories: bungalows, two-storey-houses &amp; maisonettes (purpose built flats in a house-like structure.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://wiki.openstreetmap.org/w/images/3/3a/House_Heightw_from_Lidar.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://wiki.openstreetmap.org/w/images/3/3a/House_Heightw_from_Lidar.png" height="308" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A 1980s housing estate with building heights mapped from <a href="http://wiki.openstreetmap.org/wiki/English_Environment_Agency_LIDAR" title="English Environment Agency LIDAR">English Environment Agency LIDAR</a> Open Data. Buildings fall into 3 height categories: bungalows (green:  approx 4m high), 2-storey houses of various kinds (blue: approx 6 m  high), and maisonettes (condominiums) which are about 7 m high (red).  Heights were calculated in m, so the values represent minimum heights of  the highest part of the building, which is nearly always the gable line.<br />Outpur via Overpass Turbo, styled with MapCSS.</td></tr></tbody></table>There are many other useful blog posts about using this Lidar data, both specifically for OSM, but also generally. See posts by Chris Hill (<a href="http://chris-osm.blogspot.co.uk/2015/09/more-lidar-goodness.html">"More Lidar Goodness"</a> and <a href="http://chris-osm.blogspot.co.uk/2015/09/extracting-building-heights-from-lidar.html">"Building Heights</a>") and <a href="http://wiki.openstreetmap.org/wiki/User:EdLoach/Tendring_LIDAR">Ed Loach</a> for some of the specifics, and the write-up on the <a href="http://wiki.openstreetmap.org/wiki/English_Environment_Agency_LIDAR">wiki</a>. A <a href="http://www.emu-analytics.com/blog/post.php?s=2016-01-11-2-for-the-price-of-1">nice post </a>and <a href="http://www.emu-analytics.com/buildingheights">map</a> (v. slow in my browser) showing building heights in London on OpenMap Local may also be of interest. HousePrices has processed all the Lidar data from EA and Natural Resources Wales&nbsp; as a <a href="https://houseprices.io/lab/lidar/map">hillshaded slippy map</a> which is useful to look at what is available. Slightly unfortunately the map is in OSGB projection (ESPG:27700) and is not shown with other slippy maps which would make it a bit easier to locate oneself.<br /><h3>What kind of building data should be added to OSM?</h3>From  past experience single building outlines traced from OS StreetView, turn out to represent tens of buildings on the ground. Such simplified outlines just makes the work of  splitting the buildings properly quite a lot harder. This can be  particularly bad in town/city centres.<br /><br />Usually if adding detail of POIs and addresses it is important to have individual buildings mapped: this makes it much easier to correlate photos to roofline features such as chimneys, gables etc. A single very simple outline may be OK, because for more detailed mapping it should just be a question of deleting the original outline. However, the question must be asked, as to what purpose such an outline fulfils on OSM, when the source data can be readily combined with OSM data for downstream consumption.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AGranby_Street%2C_Leicester_(geograph_2296099).jpg" style="margin-left: auto; margin-right: auto;" title="Malc McDonald [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Granby Street, Leicester (geograph 2296099)" src="https://upload.wikimedia.org/wikipedia/commons/4/4c/Granby_Street%2C_Leicester_%28geograph_2296099%29.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Granby Street, Leicester.<br />The multiple buildings shown here are represented in OSM as single buildings for <a href="http://osm.org/go/eu5bTUIyE--">each block</a><br /> (imported from OS StreetView Open Data).<br /><a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a> &nbsp; © Copyright <a href="http://www.geograph.org.uk/profile/44954" title="View profile">Malc McDonald</a> and   licensed for reuse under this <a href="http://creativecommons.org/licenses/by-sa/2.0/" rel="license">Creative Commons Licence</a>.</td></tr></tbody></table>I think the fundamental question about straight imports into OpenStreetMap should be "Will it make life easier or harder for subsequent mappers?".<br /><br />If the work involved refining a building outline takes longer than re-drawing the building then I doubt if its worth importing the building at all. This is particularly true if the outline is actually of multiple buildings. This is why large building outlines are most valuable: they are generally pretty good compared with what an initial hand-traced outline might look like, and they lend themselves better to stepwise refinement. One group of buildings I find particularly tedious to do well are schools which tend to be a sprawling mass of interconnected buildings. Starting with a decent polygon with orthogonalised angles make adding such detail much easier. The current quarterly project for UK-based mappers might be the time to test this.<br /><br />Of course it may be that adding buildings assists in some other mapping goal. I've already mentioned that details of buildings are very useful for addresses. However OpenMap Local lacks the detail in precisely the areas where it would be most useful (city &amp; town centres). For suburban or inner-city housing similar polygons can be created as quickly in OSM editors (notably in JOSM, by duplicating existing buildings or using the Terracer (or even <a href="https://github.com/derickr/josm-plugin-uberterrace">UberTerracer</a>) plugins.<br /><br />The other thing which many people want is rendered maps largely derived from OSM, but showing more buildings. In practice, because many mappers do not have the know-how, wherewithal or time to create such a rendered view, they tend to want to import buildings. Historically, OSM tools for importing data are often much easier to use than ways to incorporate the same data <b>and </b>OSM data to&nbsp; render maps and make them accessible on the web. Perhaps we need to do more to help people in the latter task: which is now getting more complicated again with the move to vector tiles (at least outwith use of MapBox Studio), and TileMill's effective status of being a legacy application.<br /><h3>Summary&nbsp;</h3>Sadly, although the new building outlines are better than what preceded them, in most cases they don't offer a decent route for iterative refinement with OpenStreetMap.<br /><br />This absence of a simple way to improve building outlines means that ideally people wishing to use this data would merge it with OSM data outside of OSM. I do recognise this is often too much work, or too big a learning curve for many, and consequently there will always be a desire to add buildings to OSM because many people are much more comfortable with consuming only OSM data for their purposes. <br /><br />Existing tools for drawing buildings in OSM are pretty powerful &amp; getting more powerful all the time. Many of us, and I include myself in this group, are unaware of the full extent of these utilities. See bdiscoe's <a href="http://www.openstreetmap.org/user/bdiscoe/diary/37576">diary post</a> about mass adjustment of circular buildings (huts) for some insights.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-7254994088506831149
url: http://sk53-osm.blogspot.com/2015/12/how-accurately-have-townlands-in_2.html
title: How accurately have Townlands in Northern Ireland been mapped?
updated: 2015-12-02 12:09:42.706000
published: 2015-12-02 12:01:00.002000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.634615
updated_at: 2019-10-26 08:30:24.905816
content:
  From time-to-time newly released Open Data provides a nice opportunity to check OpenStreetMap for its accuracy in all its forms (see <a href="http://www.ucl.ac.uk/~ucfamha/OSM%20data%20analysis%20070808_web.pdf">Hakaly </a>(2008) for a breakdown of what this can mean).<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540162/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Coastal Townlands, Cos. Derry &amp; Antrim"><img alt="Coastal Townlands, Cos. Derry &amp; Antrim" height="287" src="https://farm6.staticflickr.com/5789/23357540162_3898949e2a.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Coastal Townlands, Counties Derry/Londonderry and Antrim.<br /><span style="font-size: xx-small;">Boundary lines see below. The deeper the colour of the area, the greater discrepancy in the area of the OSM polygon and the OSNI one. The pale base colour represents a divergence of under 2%. Townlands on the coast and on the UK/Ireland border seem to&nbsp; be most likely to diverge in size. The small cluster centre right is caused by different ways of handling townlands which cross a Civil Parish boundary (OSM &amp; the original source GSGS 3906 split these, the OSNI data does not).</span></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> We have known for a while that both the Ordnance Survey of Northern Ireland and the Ordnance Survey of Ireland were planning OpenData releases. When they came it was all in a rush. For now the hard work starts of checking license conditions for suitability for use in OSM and other places, as well as then working out what is really useful. However, because the townland boundaries of Northern Ireland are complete, it was an ideal opportunity to look at accuracy.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-t_IpfTgHMZM/Vl7cuFAeIeI/AAAAAAAAA2U/n-nqzNm1F_s/s1600/IMG_9905a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="http://2.bp.blogspot.com/-t_IpfTgHMZM/Vl7cuFAeIeI/AAAAAAAAA2U/n-nqzNm1F_s/s320/IMG_9905a.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">View along N side of MacGilligan Peninsula towards Inishown from Umbra</td></tr></tbody></table><br />My reasons for doing this are not just pure interest. The usefulness of the <a href="http://sk53-osm.blogspot.co.uk/2015/10/irish-vice-counties-creation-of.html">Irish Vice County</a> boundaries depends of their positional accuracy. Earlier my prediction was that such boundaries ought to be within 10 metres of their true location on the ground where they were based on townland boundaries, but this was largely based on experience with other OSM data rather than an objective statement. Thus investigating the accuracy using an independent data set provides an excellent way of testing this statement. The tests need to be done now, because (as we shall see) the nature of OSM is to fix issues spotted very quickly, and thus datasets become loosely coupled.<br /><br />I adopted two approaches:<br /><ol><li>A straight comparison of areas (or their ratios).</li><li>Using a series of buffered boundaries from one source (OSNI) and seeing what proportion of the other source (OSM) was included in each buffer.</li></ol>To choose which townlands to compare I followed a suggestion of Rory McCann and for each OSM townland selected the one which shared the most area in common from the OSNI data set. (I have also done it on matching names for a smaller set of data &amp; get similar results). Note that I am comparing townland with townland, not boundary segment with boundary segment. This means that each boundary segment (other than coastal, lacustrine or riverine ones) will be included twice.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23403267576/in/photostream/" style="margin-left: auto; margin-right: auto;" title="umbra_townland_cf"><img alt="umbra_townland_cf" height="354" src="https://farm1.staticflickr.com/646/23403267576_6dc9d77ef9.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Buffering approach to investigating boundary accuracy.<br />Demonstrated with Umbra townland in County Derry/Londonderry.<br />This is predominantly coastal sand dunes, with a small river running along its S boundary.</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540122/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Northern Ireland Townlands OSNI comparison"><img alt="Northern Ireland Townlands OSNI comparison" height="354" src="https://farm1.staticflickr.com/652/23357540122_e166446070.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Northern Ireland using the same colouring.<br />At this scale very few boundary mismatches are apparent.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> The buffering approach is based on that described by Hakaly (2008). I used buffers of 5, 10, 15 and 20 m, and then clipped the initial OSM way be each in turn.&nbsp; On the scale of the whole country it is clear that most boundaries match closely. This is confirmed by checking what proportion of the boundaries fall into each buffer class: over 80% are within 5m, over 90% within 10m and nearly 95% within 20m.<br /><br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-tWK-UiB2_co/Vl7RlbaAa5I/AAAAAAAAA2E/VnqLhNH5bhc/s1600/townland_graph.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="277" src="http://4.bp.blogspot.com/-tWK-UiB2_co/Vl7RlbaAa5I/AAAAAAAAA2E/VnqLhNH5bhc/s400/townland_graph.png" width="500" /></a></div><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> Closer inspection (as with the <a href="http://www.openstreetmap.org/relation/4784274">Umbra</a>) shows much of the discrepancy to be present along the coast. This is not surprising, coastlines on OSM were originally derived automatically, and even when refined by hand are unlikely to accord with Mean High Water (MHW). Certainly, for my purposes, it is merely important that the OSM coastlines do not stray above MHW.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540202/in/photostream/" style="margin-left: auto; margin-right: auto;" title="NI Townlands, all boundaries within 5 m of OSNI"><img alt="NI Townlands, all boundaries within 5 m of OSNI" height="354" src="https://farm1.staticflickr.com/679/23357540202_01c4be999f.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">OSM townland Boundaries within 5m of OSNI data</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> The analysis described so far focusses on positional accuracy. Looking at areas highlights a range of other accuracy issues.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22840038813/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="townlands_ni_cf9"><img alt="townlands_ni_cf9" height="354" src="https://farm6.staticflickr.com/5658/22840038813_8993ddb144.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Area comparison. Townlands are coloured according to absolute variance of ratio of areas from 1. <br />The redder they are the further the ratio is from 1.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> Area discrepancies of over, say 5%, may be the result of any of the following:<br /><ul><li>Boundary discrepancy (such as coastlines). Mainly caused by coastlines, or difficulty of delineating some boundary feature, such as the course of the Umbra river above)&nbsp;</li><li>Erroneous interpretation of the boundary on old maps causing selection of the wrong feature. This transfers land from one townland to another, therefore these should cluster.&nbsp;</li><li>Missing townlands. When a single townland has been created without noticing one or more others inside it (<a href="http://www.openstreetmap.org/#map=14/55.0704/-6.5098&amp;layers=N">Town Parks townland</a> at Ballymoney is an example).&nbsp;</li><li>Different treatment of townlands bisected by a Civil Parish. See caption of first image above. Incorrect tagging.&nbsp;</li><li>Higher level administrative units having tags appropriate to a townland. I've noted two cases of this one of which was <a href="http://www.openstreetmap.org/relation/5136591">Ballyphilip </a>CP on the Ards peninsula in County Down.&nbsp;</li><li>Islands. Some offshore islands appear to be missing from the OSNI data (see The Skerries N of <a href="http://www.openstreetmap.org/#map=14/55.2215/-6.6145">Portrush</a>) </li></ul>We've already caught a few examples in each of these classes through this analysis, and no doubt will find a few more. I have not yet investigated the very apparent discrepancy along the borders.<br /><br />To conclude, townland boundaries show exactly the kind of positional accuracy we expected (or perhaps hoped). Perhaps 1% of the total data (90-100 townlands from about 9000) may need some form of correction. I'm biased, but this seems pretty good, for a project principally relying on rectified photo-reduced maps from 1939! It's also worth remembering, that unlike road comparisons, there is no widely available sensor data (ie GPS tracks/point) to help boundary alignments.<br /><br />When time permits I'll extend this to include OSI Open Data too. A big thanks to both organisations for releasing their Open Data. OSNI staff have been contributors to OSM for a while: they host <a href="https://twitter.com/osni_maps/status/631832070952779778">Missing Maps lunchtime sessions</a> in their offices.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-2239225893201820462
url: http://sk53-osm.blogspot.com/2015/11/urban-areas-4-derivation-from.html
title: Urban Areas 4 : Derivation from OpenStreetMap using road density
updated: 2015-11-15 15:16:56.105000
published: 2015-11-15 15:16:00.004000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.640389
updated_at: 2019-10-26 08:30:24.910916
content:
  Another variation on the theme from <a href="http://sk53-osm.blogspot.com/2015/11/urban-areas-3-derivation-from-osm-using.html">the last post</a>: this time looking for some measure of road density.<br /><br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22413481824/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="Butler Co, PA: derived Urban Areas"><img alt="Butler Co, PA: derived Urban Areas" height="354" src="https://farm6.staticflickr.com/5705/22413481824_f379de3860.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of Urban Areas derived using "block method" and gridded road density.<br />Only grid squares with over 500 m of road included.<br />The area shown <a href="http://www.openstreetmap.org/#map=13/40.8569/-79.9092">is around</a> Butler, Butler Co, PA</td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> The easiest way is to sum road lengths in individual cells. The cells have to be quite small (say 250 metre square) to achieve the resolution desired. I've excluded link roads and motorway &amp; trunk highway classed in this calculation.<br /><br /><br /><a name='more'></a>Again I've used Butler County data from OpenSteetMap. I created a 250 m square grid in the (Google) Spherical Mercator projection, and simply summed up the road lengths in each grid square (roads were coerced to a geography type in PostGIS). Visual inspection of binned data suggested a value of between 400 and 500 metres/square gave reasonably discrete areas, and reduced artefacts along major roads. I show the result obtained above compared with the "block method" described in the previous post.<br /><br />An alternative way is to apply a finer grid of lines not polygons and split these grid lines each time they are intersected by a road. I've also done this for Butler County, although PostGIS doesn't provide any simple way to split a single line into multiple parts based on intersections.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22413481794/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="butler_co_urban_density2"><img alt="butler_co_urban_density2" height="640" src="https://farm1.staticflickr.com/633/22413481794_bb0595bd0c_z.jpg" width="453" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Urban Area derivation, with results of grid intersection methods superimposed<br />on gridded road density and block method results<br />Area around Butler, Butler Co, PA</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> I used lines 100 m apart and in this case I filtered out any segments longer than 200 m. A second step is then to further restrict the selections to lines which intersect each other: this has the primary effect of removing dual carriageway artefacts. The former are shown as green lines, and the latter as red lines on the map above. I did not exclude motorway &amp; trunk in this analysis, resulting in some artefacts around junctions.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22413481724/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="butler_co_urban_density1"><img alt="butler_co_urban_density1" height="500" src="https://farm1.staticflickr.com/737/22413481724_d19b0e4134.jpg" width="354" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">As above, Butler County as a whole.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> In general this produces similar results to other methods: but can miss areas where the roads are precisely aligned with the grid (a not uncommon phenomenon in the USA), suggesting that in practice grids should be rotated slightly from the cardinal directions.<br /><br />The two road density methods described here produce somewhat smaller polygons than the block method, but seem to be better than the latter for picking up smaller residential areas which tend to get eliminated in the latter method. Both methods may suffer for larger areas as the size of such closely spaced grids is likely to have adverse effects on performance.<br /><br />I think this concludes discussion of urban area derivation techniques applicable for complete road networks with unclear or inaccurate attribution, such as US Tiger data.<br /><h4>Appendix: Cutting a line at each intersection in PostGIS</h4>One is so used to the power and flexibility of modern FOSS GIS tools, notably those from OSGeo, that it comes as a shock when what seems like a common operation is not supported directly. The OSM desktop editor JOSM, for instance provides a tool to split a way into multiple fragments.<br /><br />I'd hoped to split each x &amp; y grid line at each point when it was crossed by a road. Filtering for only small segments would identify when roads were more frequent. The problems associated with doing this in PostGIS are reasonably well described here on <a href="http://gis.stackexchange.com/questions/115973/split-lines-at-intersection-points">GIS StackOverflow</a>. The second answer looks interesting but provides no guidance, and <span style="font-family: &quot;Courier New&quot;,Courier,monospace;"><a href="http://postgis.net/docs/ST_Node.html">ST_NODE</a></span> must remain a mystery as there is no clear explanation of the magic it works, other than it replaces another bit of magic using <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">ST_UNION</span>. <br /><br />So I did something quick &amp; dirty. I found all the intersection points between the grid lines &amp; the road nework, taking care to retain the grid line identifiers. I then constructed lines from any intersection point to points either further east or north (using &lt;&lt; and &lt;&lt;| operators) located <b>on the same line</b> &amp; selected the shortest one. I then filtered to get those shorter than 200 m. In practice if I had known I was going to use 200 m as my filter I could have thrown out all the longer lines at that stage.<br /><br /><br />


guid: tag:blogger.com,1999:blog-2082422783692631078.post-4795471978162340110
url: http://sk53-osm.blogspot.com/2015/11/urban-areas-3-derivation-from-osm-using.html
title: Urban Areas 3 : derivation from OSM using residential blocks
updated: 2015-11-14 15:17:41.521000
published: 2015-11-14 15:17:00.001000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.643125
updated_at: 2019-10-26 08:30:24.913550
content:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AFromCoL_8234828991.jpg" style="margin-left: auto; margin-right: auto;" title="By Zack Weinberg (Flickr  Uploaded by crazypaco) [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="FromCoL 8234828991" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/FromCoL_8234828991.jpg/512px-FromCoL_8234828991.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">View S from the <a href="https://en.wikipedia.org/wiki/Cathedral_of_Learning">Cathedral of Learning</a> in <a href="https://en.wikipedia.org/wiki/Oakland_%28Pittsburgh%29">Oakland, Pittsburgh</a>, <br />showing some urban areas used as tests in this post.<br /><div style="text-align: justify;"><span style="font-size: xx-small;">The incised valley of the  Monogahela in the background contained railways and steel works. The plateau beyond has residential suburbs of Pittsburgh. To the left foreground are the woods and ravines of <a href="https://en.wikipedia.org/wiki/Schenley_Park">Schenley Park</a>, with a residential area beyond. Source: Zack Weinberg via Wikimedia Commons CC-BY-SA</span></div></td></tr></tbody></table><br />One of the obvious features of the highway network for the USA on OpenStreetMap is that road density is much higher in built-up areas. I started looking at how to measure this, when I recalled a method for <a href="https://github.com/lucasmation/osm_cnefe_import">identifying city blocks</a> introduced to me by a Brazilian user of OpenStreetMap data.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22574566097/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="butler_co_urban_blocks"><img alt="butler_co_urban_blocks" height="640" src="https://farm6.staticflickr.com/5742/22574566097_c13845bc0f_z.jpg" width="453" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Residential Areas for Butler Co, Pennsylvania, identified with the block method<br />from OpenStreetMap data. Orange line outlines Butler County.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> My idea was simple, a greater road density implies smaller areas for the polygons enclosed by a set of roads. By choosing some maximum polygon size, one should be able to pick out urban areas. <br /><br />The method itself is also really quite simple:<br /><ul><li>Take the main road network for some area and make a union of it (which will be a MULTILINESTRING).</li><li>Polygonize this data, and decompose to individual polygons.</li></ul>In Lucas' implementation the first step is done by municipal areas. I wanted to try the approach for a whole state without using administrative area data. I therefore once again turned to my trusty standby of using a gridded method.<br /><br /><a name='more'></a>First off I did this for Butler County, to test the process and iron out teething problems with the SQL (see above). The main changes I made are:<br /><ul><li>&nbsp;to perform the initial creation of blocks using a 3 minute grid. In order to avoid losing blocks at the edges of each grid square, the grid square as a linestring needs to be included in the block generation. I achieved this by:</li></ul><blockquote class="tr_bq"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">SELECT grid_guid, geom <br />&nbsp; FROM pa_roads_gridded<br />&nbsp; UNION ALL <br />SELECT grid_guid, st_exteriorring(geom) <br />&nbsp; FROM pa_3minute_grid</span></blockquote><ul><li>In the decomposition phase the area of each polygon is calculated, and only polygons with an area of under 100000&nbsp; included (units in Spherical Mercator, actual areas are about half the size in square metres).</li><li>All blocks within a grid square are merged if they touch to form a number of larger polygons. (This uses the same method as before for identifying each distinct graph).</li><li>Finally the larger polygons which touch are merged across grid squares.</li></ul>I then applied it to the whole of Pennsylvania. Initially I was too enthusiastic and tried to do the polygon merging in a single step, but soon returned to the gridded approach when I saw how slow this was in iterating to completion.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23017578711/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Pittsburgh Urban Areas :derived from OSM using the block method"><img alt="Pittsburgh Urban Areas :derived from OSM using the block method" height="354" src="https://farm1.staticflickr.com/565/23017578711_36e981289e.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Urban Areas for the Pittsburgh region, derived using the 'block' method described here.<br />Each polygon is coloured randomly to help show the boundaries.<br />Interstates &amp; other dual-carriageways create significant artefacts.</td></tr></tbody></table><br />Looking at Pennsylvania as whole, and the Pittsburgh conurbation in particular, reveals quite reasonable identification of urban areas with <b>one big</b> problem. The problem is freeways, specifically interstates, but in practice it applies to any dual carriageway. These not only produce long linear extensions of the main urban areas, but actually cause areas which are not contiguous to fuse.<br /><br />There are a number of approaches to correcting these artefacts:<br /><ul><li>Simply exclude <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">highway=motorway</span> in the initial step. This would probably work OK for Pennsylvania, but would not, for instance, work in the UK where many <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">highway=trunk</span> roads are dual carriageways.</li><li>Introduce some compactness measure for each initial polygon (e.g., ratio of perimeter to area, or ratio of area to area of an enclosing circle) , and exclude polygons over a certain threshold. This is probably the easiest to do in the short term.</li><li>Generalise dual carriageways to a single line feature before creating blocks. This is complex, the only solution I know of specifically for OSM data is the <a href="http://gis.stackexchange.com/questions/104359/how-to-get-a-low-level-road-network-from-openstreetmap-data?lq=1">VB6 one written</a> for <a href="https://vimeo.com/73118945">CityGuide</a>. However, this is a general problem (<a href="https://en.wikipedia.org/wiki/Topological_skeleton">skeletonization</a>) whose solution would help in a whole range of areas: for instance, better cartography for small scales; 1-dimension maps; automated address interpolation.&nbsp;</li></ul><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22995995762/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Pittsburgh Urban Areas: block method"><img alt="Pittsburgh Urban Areas: block method" height="354" src="https://farm1.staticflickr.com/757/22995995762_9c619d5ae9.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Urban Areas for the Pittsburgh region, derived using the modified 'block' method described above.<br />Each polygon is coloured randomly to help show the boundaries.<br />Compare with image above.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> The second was easy to test: I excluded polygons with an <a href="http://mathworld.wolfram.com/IsoperimetricQuotient.html">isoperimetic ratio</a> of less than 0.3 and this seems to have eliminated the artefacts due to dual carriageways. As with some of my other grid-based techniques there are still some artefacts based on using a grid.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22995995842/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Pennsyvlania Urban Areas : block method"><img alt="Pennsyvlania Urban Areas : block method" height="354" src="https://farm1.staticflickr.com/590/22995995842_9029e9496f.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Pennsylvania Urban Areas using modified block method.<br />Grid artefacts are very clear at this scale.<br />The vignetting of OSM tiles is not deliberate, but due to vagaries of the OpenLayers plugin for QGIS</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> As with all the methods described in this series of posts, this technique could do with refinement if used in anger. However, it does look to be generating relatively usable urban area polygons, albeit with some false positives, for US road data from OpenStreetMap.<br /><br />Some fairly simple improvements to the polygons — for instance clipping to ensure they don't overlap existing landuse type polygons — could make them suitable for reincorporation within OSM as residential areas. Two utilities may be available, the <a href="http://wiki.openstreetmap.org/wiki/Snapshot_Server">Snapshot Server</a> and <a href="http://wiki.openstreetmap.org/wiki/OSMLY">OSMLY</a>, which would allow such data to be used on a case-by-case basis when suitable by individual mappers.<br /><br />Another application would be to more easily reduce the number of highway=residential roads which are candidates for retagging.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-1779132849671688715
url: http://sk53-osm.blogspot.com/2015/11/urban-areas-2-derivation-from.html
title: Urban Areas 2 : Derivation from OpenStreetMap using Residential Roads
updated: 2015-11-13 10:37:14.794000
published: 2015-11-12 13:58:00.001000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.645949
updated_at: 2019-10-26 08:30:24.916473
content:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-8gDJHa1Z9M0/VkIxz0EAV4I/AAAAAAAAA0Q/xYomYCVXA4I/s1600/IMG_4143-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="http://4.bp.blogspot.com/-8gDJHa1Z9M0/VkIxz0EAV4I/AAAAAAAAA0Q/xYomYCVXA4I/s320/IMG_4143-1.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Street corner, Retiro, Buenos Aires<br />(<a href="http://www.openstreetmap.org/#map=19/-34.59284/-58.38377">Libertad/ Juncal</a>)<br />CC-BY-SA, the author</td></tr></tbody></table>Following on from my last post I have now been looking in more detail at how one might start using OpenStreetMap (OSM) to create a global dataset of Urban Areas. As OSM does not have any widely used notation for urban areas I have been looking at several ways in which other OSM data can be used to identify such areas prospectively. In this post I look at the use of residential roads (and I'm not the first <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_El_uso_de_OSM_para_la_identificaci%C3%B3n_de_Nucleos_Poblados">to do so</a>). Later posts will look at other techniques.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22438725688/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="ar_ba_urban2"><img alt="ar_ba_urban2" height="253" src="https://farm6.staticflickr.com/5712/22438725688_b97f59b27d.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Buenos Aires and hinterland, showing comparison between urban polygons<br />derived from OSM (green) and the Natural Earth data (light brown).</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> <br />I have chosen the following places as suitable test areas for these investigations:<br /><ul><li><a href="https://en.wikipedia.org/wiki/East_Midlands"><b>East Midlands</b></a> of England. Not only my home turf, but also a well-mapped area with extensive use of landuse tags, and in excess of 99% of all residential roads. In addition Ordnance Survey Meridian 2 Open Data contains a layer corresponding to urban areas which provides an excellent control for checking results from this area.</li><li><a href="https://en.wikipedia.org/wiki/Pakistan"><b>Pakistan</b></a>. Not only one of the most populous countries in the world, but one of the least well mapped in OpenStreetMap. Pakistan is a likely candidate for cities which are barely mapped. I would also expect other very populous Asian countries (notably China, India and Bangladesh) which are poorly mapped to be similar to Pakistan.</li><li><a href="https://en.wikipedia.org/wiki/Nigeria"><b>Nigeria</b></a>. Similar criteria to Pakistan: the most populous country in Africa. The .pbf file for Nigeria is approximately 50% larger than that for Pakistan, but both are smaller than that for <a href="https://en.wikipedia.org/wiki/Lesotho">Lesotho </a>with a population of 2 million compared to 180 million (Nigeria) and 200 million (Pakistan).</li><li><a href="https://en.wikipedia.org/wiki/Ivory_Coast"><b>Côte d'Ivoire</b></a>. Close to Nigeria, but a place which I know has an active OSM community. Quite a number of mapping activities. (Note to Geofabrik, it's not called the Ivory Coast any more).</li><li><a href="https://en.wikipedia.org/wiki/Argentina"><b>Argentina</b></a>. Latin American cities are often laid out in a grid, nowhere more so than in Argentina. The prevalence of the grid system, and my believe that the urban road system is largely complete were reasons for choosing this as a Latin American example. My own experience of travelling in Argentina after SotM-14 suggests that, for the most part, urban road systems are mapped. One known gap, the newer western suburbs of Ushuaia has recently been rectified by the kind provision of aerial imagery from the Argentine National mapping agency.</li><li><a href="https://en.wikipedia.org/wiki/Pennsylvania"><b>Pennsylvania</b></a>. It was essential to include some US data&nbsp; because of the <a href="http://wiki.openstreetmap.org/wiki/TIGER">TIGER </a>import problem: all rural roads being tagged residential. Since I spent part of my childhood in Pennsylvania it is also a place I know and which I have edited (sporadically) to improve the rural road network.</li></ul>Briefly I expected the following: good urban areas for the East Midlands and Argentina (i.e., better than <a href="https://en.wikipedia.org/wiki/Natural_Earth">Natural Earth</a> (NE)); middling to poor for the three developing nations (gaps relative to NE, but in some cases better precision); hopeless for Pennsylvania.<br /><br /><a name='more'></a>The basic process is very simple:<br /><ol><li>Extract residential roads</li><li>Merge all roads which link together into a single multiline segment</li><li>Buffer each group of merged roads (I use 100 metres)</li><li>Buffer again by a larger amount &amp; then again by the same amount but negated: this smooths the outline and fills any residual holes.</li></ol>In practice this needs to be slightly more complex to improve performance. To this end I add the following techniques:<br /><ul><li>&nbsp;Roads are clipped into grid squares (typically 10 km, 3 minutes or 7.5 minutes).</li><li>Merging is first performed within the grid</li><li>Some tidying up of this data is then done, notably buffering and re-clipping to the grid</li><li>A second round of merging is performed ignoring the grid</li></ul>I'm using the postgres routine I described a couple of years ago to identify the independent graphs of roads in each grid, and in the second step. <strike>It still lacks all the performance optimisation stuff I meant to do back then!</strike> (After much messing about it turned out that it only needed indexes on the work table).<br /><br /><h3>East Midlands </h3><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22479309648/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Comparison of 2 Urban Area sourced for East Midlands"><img alt="Comparison of 2 Urban Area sourced for East Midlands" height="640" src="https://farm6.staticflickr.com/5753/22479309648_a8634eae10_z.jpg" width="567" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of Urban Areas for English East Midlands</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> As expected, there is a high degree of concordance between areas identified as urban from the OSM residential road data, and the available Open Data from the Ordnance Survey. Areas not identified as urban from the OSM data, but so identified as such in the Meridian data are shown in cyan in the map above. For the most part the differences fall into two classes:<br /><ol><li>Commercial, retail &amp; industrial areas in towns and cities which are not identified through merely looking at residential roads in OSM. This initial aspect of the data is of course wholly predictable.</li><li>Small towns and villages in the countryside. Firstly, the goal is not to find these in these experiments. Various refinements (such as adding isolated island residential roads back at various steps in the process) have not been made (see further below). Secondly many villages will be aligned along road classes other than residential and would not be found anyway. Most of these areas not identified are under 60 ha in size.</li></ol>More surprising is the very small number of false positives in the data derived from OSM. These are all erroneous classifications of rural roads as residential ones (sometimes because the same road starts in a town and runs into the countryside). For Britain, at least, this offers a useful visual QA tool for OSM data.<br /><h3></h3><h3>Argentina</h3>I had also chosen Argentina as an area to validate the concept.&nbsp; But this was also influenced by the particular problems with Natural Earth data in this region. The huge <a href="https://en.wikipedia.org/wiki/Megalopolis_%28city_type%29">megalopolis </a>of <a href="https://en.wikipedia.org/wiki/Buenos_Aires">Buenos Aires </a>was also useful for making sure the chosen algorithms of this technique will work with the largest cities in the world.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-Umn9UbXgSYI/VkC8BH3g_CI/AAAAAAAAAzw/0s9rFmnQkbw/s1600/ar_chubut_urban.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://4.bp.blogspot.com/-Umn9UbXgSYI/VkC8BH3g_CI/AAAAAAAAAzw/0s9rFmnQkbw/s320/ar_chubut_urban.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Northern <a href="https://en.wikipedia.org/wiki/Santa_Cruz_Province,_Argentina">Santa Cruz</a> &amp; S. <a href="https://en.wikipedia.org/wiki/Chubut_Province">Chubut </a>provinces, <a href="https://en.wikipedia.org/wiki/Patagonia">Patagonia</a><br />comparison of NE &amp; derived OSM data. </td></tr></tbody></table>Once again there is a highly satisfactory match with Natural Earth data around Buenos Aires. In <a href="https://en.wikipedia.org/wiki/Patagonia">Patagonia</a>, not only are many cities which were missed by Natural Earth identified, but there do not seem to be any obvious false positives. Indeed with some light tidying-up (clipping to coastlines, filling small holes, some smoothing) this could make a viable replacement for the NE data here.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-15W-RHJwl5w/VkC9SWoZ-pI/AAAAAAAAA0A/0U0_jHp3BZs/s1600/ar_nequen_urban.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://3.bp.blogspot.com/-15W-RHJwl5w/VkC9SWoZ-pI/AAAAAAAAA0A/0U0_jHp3BZs/s320/ar_nequen_urban.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Neuqu%C3%A9n">Neuquén </a>city (bottom centre) and province: More false positive Urban Areas from Natural Earth compared to OSM<br />Sources: as before</td></tr></tbody></table><h4>Addendum</h4>It turns out that I'm not the first to use this approach in Argentina. After I'd published this post <br /><ahref https:="" nicolas09f9="" twitter.com="">Nicolás Álvarez</ahref> drew my attention to <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_El_uso_de_OSM_para_la_identificaci%C3%B3n_de_Nucleos_Poblados">a talk</a>, which I had missed, at State of the Map 2014 in Buenos Aires. The authors are :<a href="https://twitter.com/vlasvlasvlas">Vladimiro Bellini</a>, Fernando Pino, Martin Moroni of the Ministry of Energy. Another great example of how people are doing interesting things with OSM data all over the world.<br /><br />I haven't looked at the details yet, but from the abstract this looks very similar to my approach with an important refinement, restricting the length of roads included to those under 2 km. Slides of this talk are available on <a href="http://es.slideshare.net/VladimirBellini/sotm-2014-session-el-uso-de-osm-para-la-identificacin-de-nucleos-poblados">Slideshare</a>. <br /><h3>Pakistan</h3><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-PAjsqp88zfk/VkNrP8zZnUI/AAAAAAAAA1s/A574Qd36nqU/s1600/pk_urban3.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="339" src="http://3.bp.blogspot.com/-PAjsqp88zfk/VkNrP8zZnUI/AAAAAAAAA1s/A574Qd36nqU/s320/pk_urban3.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Punjab,_Pakistan">Punjab </a>area of Pakistan: comparison with Natural Earth.<br /><a href="https://en.wikipedia.org/wiki/Lahore">Lahore </a>is bottom right &amp; <a href="https://en.wikipedia.org/wiki/Rawalpindi">Rawalpindi </a>top left.</td></tr></tbody></table><br />A first cursory look at the results for Pakistan compared with Natural Earth data suggests that most major urban areas are being detected. Looking at a larger scale reveals many mid-size cities being ignored. At even larger scales the inaccuracy of the Natural Earth data itself becomes more apparent.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-ntusEFOjO_o/VkNokjy_UUI/AAAAAAAAA1c/Xn34YQDX2ZI/s1600/pk_urban2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="246" src="http://3.bp.blogspot.com/-ntusEFOjO_o/VkNokjy_UUI/AAAAAAAAA1c/Xn34YQDX2ZI/s320/pk_urban2.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Slightly more detail with Lahore (bottom centre) &amp; Sialkot (centre).<br /><span style="font-size: xx-small;">Many rural as well as urban roads have the residential tag in the latter area,</span><br /><span style="font-size: xx-small;">Note the reticulated appearance of the polygon (see below, Pennsylvania)</span></td></tr></tbody></table>It seems that a significant problem exists with my algorithm for Pakistan. Even large cities may have only a handful of roads mapped as highway=residential. These either form isolated islands in the first phase or are merged into a single island after the first phase. At present I make no attempt to add back these polygons with no adjacent ones. The sensible place to do it is after the second phase of merging. To do it earlier would be to introduce too many artefacts from inappropriately tagged roads or isolated residential districts in rural areas. I have not done this for this post, but show below the difference between 1st phase (dark blue) &amp; 2nd phase (light blue) urban polygons above &amp; below.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-gu0fRCFcf0Y/VkNokcqkjyI/AAAAAAAAA1Y/hGlzF6N6wt4/s1600/pk_sialkot2_urban.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="246" src="http://1.bp.blogspot.com/-gu0fRCFcf0Y/VkNokcqkjyI/AAAAAAAAA1Y/hGlzF6N6wt4/s320/pk_sialkot2_urban.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Higher zoom of area West of&nbsp; Sialkot (right).<br /><span style="font-size: xx-small;">Many urban areas have either a tiny number of residential roads or none at all.<br />Source: Natural Earth &amp; derived OSM polygons<br />Background Imagery: Landsat via Bing Maps</span></td></tr></tbody></table>Lahore is is a megalopolis, but it has remarkably few residential streets mapped. A major factor is that in the older parts of the city, most thoroughfares in residential areas are narrow, and lined by multi-storey buildings. This makes them more or less impossible to pick out on available aerial imagery, and perhaps on any imagery likely to be available to OSM in the near future. However, my choice of buffer size to build residential roads works reasonably well even when the grid is partially mapped, as in this case.<br /><br />Notwithstanding this there are still many wider roads which can be mapped. Once again I have spent a little time whilst writing this post adding a few in Lahore.<br /><br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AA_Busy_Street_in_Sialkot.jpg" style="margin-left: auto; margin-right: auto;" title="By Yaminjanjua at en.wikipedia (Transferred from en.wikipedia) [Public domain], via Wikimedia Commons"><img alt="A Busy Street in Sialkot" src="https://upload.wikimedia.org/wikipedia/commons/2/26/A_Busy_Street_in_Sialkot.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A busy street, Sialkot 2008<br />CC-BY-SA via Wikimedia</td></tr></tbody></table><br />Elsewhere, a local mapper in <a href="https://en.wikipedia.org/wiki/Sialkot">Sialkot </a>has been <a href="http://www.openstreetmap.org/#map=17/32.49392/74.53604">adding this type of detail,</a> but using the tag <a href="http://wiki.openstreetmap.org/wiki/Tag:service%3Dalley"><span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">highway=service, service=alley</span></a>. This seems an entirely reasonable choice of tags: if it is a local convention then taking account of this by adding such ways to the choice of residential roads is easy to do. (I would have to look at the use of this tag combination globally to see if there were any problems in doing this by default).<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-iYVCnrNRsNo/VkNsVPL3NEI/AAAAAAAAA10/PnkVBcm8Lmo/s1600/pk_res_ua.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="246" src="http://3.bp.blogspot.com/-iYVCnrNRsNo/VkNsVPL3NEI/AAAAAAAAA10/PnkVBcm8Lmo/s320/pk_res_ua.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Pakistan showing both Urban &amp; OSM Residential areas.<br />Note the density in <a href="https://en.wikipedia.org/wiki/Sindh">Sindh</a>, the result of a <a href="http://wiki.openstreetmap.org/wiki/Humanitarian_OSM_Team/Pakistan_Tharparkar_Drought">HOT activation</a></td></tr></tbody></table>It was no surprise that OSM data for Pakistan are incomplete in many areas. I have drawn most of my examples from a particularly densely populated parts of the Punjab. In the process of examining the area it is clear that much can be achieved in a relatively short time by adding more residential areas and prominent residential roads. We should bear in mind that the <a href="http://wiki.openstreetmap.org/wiki/2010_07_Pakistan_Floods">dreadful flooding events of 2010</a> are unlikely to be isolated events. At the time map data available to relief organisations was poor, and the sheer extent of the area covered was far too large for a <a href="http://www.mapaction.org/disasterresponse/depldetail/196.html">responsive mapping campaign</a>.<br /><br />OSM data for Pakistan is still poor over 5 years on, we never managed to achieve the same leverage as Google Map Maker with the diaspora of people with Pakistani-heritage. But we can try again.<br /><h3>Nigeria</h3><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ARound_about_in_Ibadan-1%2C_by_Adebisi_Adewoyin.jpg" style="margin-left: auto; margin-right: auto;" title="By Adebisi Adewoyin (World66) [CC BY-SA 1.0 (http://creativecommons.org/licenses/by-sa/1.0)], via Wikimedia Commons"><img alt="Round about in Ibadan-1, by Adebisi Adewoyin" src="https://upload.wikimedia.org/wikipedia/commons/b/bf/Round_about_in_Ibadan-1%2C_by_Adebisi_Adewoyin.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Roundabout <a href="https://en.wikipedia.org/wiki/Ibadan">Ibadan</a><br />Whilst looking for CC-BY-SA images for urban areas of Nigeria I noticed a few shots of roundabouts with distinctive sculptures in the middle. These seem <a href="http://thenationonlineng.net/imos-magnificent-roundabouts-city-gates/">quite common</a> across the country.<br />Source: Adebisi Adewoyin via Wikimedia Commons CC-BY-SA</td></tr></tbody></table>In many ways the situation in Nigeria is similar to Pakistan: at an overall level the results look reasonable, a closer look reveals that large cities are getting missed. This is most noticeable in the Southern part of the country.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22323292523/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Urban Area comparison, SW Nigera"><img alt="Urban Area comparison, SW Nigera" height="354" src="https://farm1.staticflickr.com/728/22323292523_c953269028.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of Natural Earth &amp; derived OSM Urban Areas for SW Nigeria</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> For the area shown above at least one factor is the absence of high resolution imagery. However, some significant cities were missing in any other form than an imported GNIS node ( e.g., <a href="https://en.wikipedia.org/wiki/Owo,_Nigeria">Owu</a>). Looking at the country as a whole, it appears the North is much better mapped, with a more extensive road system and lots of minor settlements mapped as residential areas.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22323292563/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Nigeria: urban &amp; residential areas"><img alt="Nigeria: urban &amp; residential areas" height="256" src="https://farm6.staticflickr.com/5817/22323292563_3c605cb2a9.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Urban (NE &amp; derived OSM) and residential areas in Nigeria</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> I have no idea why this should be, but clearly one would expect as much or more detail in other parts of the country based on available demographic data (see below). I am aware that <a href="https://en.wikipedia.org/wiki/Kano">Kano </a>seems to have the most active mappers, that there have been <a href="http://wiki.openstreetmap.org/wiki/WikiProject_Nigeria#North_Nigeria_crisis_-_HOT_Monitoring_-_mapping_goals">HOT initiatives</a> in NE Nigeria, and that eHealth Africa have been doing some mapping of residential areas (see this recent <a href="http://www.openstreetmap.org/user/olub/diary/36265">blog post</a>): it is probably the latter activity which has had the biggest effect. I dont know quite how long this has been running, but it does demonstrate how much data can be added.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-zIY3Cy4trZI/VkNaLmKN1oI/AAAAAAAAA1I/7wm4xGTc1Fo/s1600/ngadens.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="533" src="http://2.bp.blogspot.com/-zIY3Cy4trZI/VkNaLmKN1oI/AAAAAAAAA1I/7wm4xGTc1Fo/s400/ngadens.jpg" width="412" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Nigeria Population Density, 2000<br />Compare with density of residential mapping on OSM<br />Source: see map CC-BY-SA</td></tr></tbody></table><h3>Côte d'Ivoire</h3><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AAbidjanpyramid.JPG" title="By Zenman (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons"><img alt="Abidjanpyramid" src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/9b/Abidjanpyramid.JPG/512px-Abidjanpyramid.JPG" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Central Abidjan, quartier Plateau (on <a href="http://www.openstreetmap.org/#map=17/5.31941/-4.01760">OSM</a>)<br />Source: Zenman via Wikimedia Commons</td></tr></tbody></table>I wanted to look at an area with some similarities to Nigeria, but with a better established mapping community to see if this makes for any significant difference. The obvious choice was Côte d'Ivoire. There always seems to be something interesting going on there: mapping parties, software training events, conferences. And it's all nicely documented on the <a href="http://www.openstreetmap.ci/">community website</a>. One of the things I particularly like is a <a href="http://umap.openstreetmap.fr/fr/map/pharma-carte_3999#12/5.3563/-4.0134">map of pharmacies in Abidjan</a> on umap: a reminder that maps help everywhere for little daily tasks or personal emergencies. Looking at Pascal Neis' maps Abidjan looks to be the place with the <a href="http://resultmaps.neis-one.org/oooc?zoom=8&amp;lat=6.63751&amp;lon=-4.65558&amp;layers=B00TFFFFFT">most dedicated mappers</a>.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-pRynF9sHssU/VkNCoKxJNeI/AAAAAAAAA0g/iPWiiPTGGUo/s1600/ci_south_ua.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://2.bp.blogspot.com/-pRynF9sHssU/VkNCoKxJNeI/AAAAAAAAA0g/iPWiiPTGGUo/s320/ci_south_ua.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of NE &amp; derived OSM Urban Areas, Southern Côte d'Ivoire<br />Legend: road network (OSM), NE urban (cyan), OSM derived urban (magenta)<br />Imagery: Landsat via Bing Maps</td></tr></tbody></table>Again on the small scale we have a pattern of significant settlements being captured from OSM and some smaller places being missed in a fairly haphazard way. We also get the impression that NE urban areas are frequently too large.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-IRhTJsD4Tlk/VkNCpC7r37I/AAAAAAAAA0o/vfkPUc9g7BI/s1600/ci_central_ua.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://3.bp.blogspot.com/-IRhTJsD4Tlk/VkNCpC7r37I/AAAAAAAAA0o/vfkPUc9g7BI/s320/ci_central_ua.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of NE &amp; derived OSM Urban Areas, Southern Côte d'Ivoire<br />Legend: road network (OSM), NE urban (cyan), OSM derived urban (magenta), OSM residential (blue)<br />Imagery: Landsat via Bing Maps</td></tr></tbody></table>This is confirmed by zooming in.<br /><br />Adding an extra layer from OSM, mapped residential landuse shows what is most different between Nigeria &amp; the Côte d'Ivoire. Individual settlements of different sizes have been mapped in the latter as <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">landuse=residential</span>, even if no-one has had time to add the individual roads. In Côte  d'Ivoire it is detail of settlements that is missing not settlements themselves.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-OnRN617f6FE/VkNCpOn2jjI/AAAAAAAAA0s/zpz1h8V4H8s/s1600/ci_adidjan_ua.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="http://4.bp.blogspot.com/-OnRN617f6FE/VkNCpOn2jjI/AAAAAAAAA0s/zpz1h8V4H8s/s320/ci_adidjan_ua.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison of NE &amp; derived OSM Urban Areas, Abidjan<br />Legend: road network (OSM), NE urban (cyan), OSM derived urban (magenta)<br />Imagery: Standard OSM layer</td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;">Looking at <a href="https://en.wikipedia.org/wiki/Abidjan">Abidjan </a>itself, we can see how well mapped it is, the precise correspondence of derived urban areas with landuse mapping, and, once again, that the NE polygon is too large. I do note that the derived urban area includes some large industrial landuse polygons. This suggests that <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">highway=residential</span> has been used rather than <span style="font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;">highway=unclassified</span> in these areas.</div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Unlike Nigeria &amp; Pakistan (and the US, see below), I have not felt the need edit OSM for Côte  d'Ivoire, either to add really obviously missing data, or to modify tagging. </div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Looking at these three countries together, the key point is that, unless the local OSM community is large, and mature, residential roads will not be adequate on their own to identify urban areas. In Britain, it was not until after we had official open data (Ordnance Survey) that many residential roads were added, and that was in 2010, 5 years into mapping the country. Without that external source we would probably still have significant towns &amp; cities only partially mapped. Much of how the map&nbsp; Côte  d'Ivoire looks now is very reminiscent of Great Britain before we had open data: great detail in places, but fairly scanty away from where most mappers lived. </div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;">Adding landuse is an excellent way to build up a picture of what needs to be mapped: in places like Côte d'Ivoire the combination of using both landuse &amp; derived urban areas looks promising (more later).</div><h3>Pennsylvania (and elsewhere in US)</h3><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22832538622/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Pittsburgh area: Residential Roads on OSM"><img alt="Pittsburgh area: Residential Roads on OSM" height="354" src="https://farm6.staticflickr.com/5805/22832538622_34cb180708.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Pittsburgh area : Residential Highways &amp; Landuse mapped on OpenStreetMap<br />Source: (c) OpenStreetMap contributors</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> A quick look at road data for Pennsylvania shows that the network of residential roads reaches into the furthermost reaches of the state. The human eye can quickly discern the much greater density of the road network which represents truly urban locations. Finding a suitable automatic substitute will be the topic of a later post in this series.<br /><br />However, the reason for looking at the US was to look at the problem where residential roads have not been distinguished from other minor roads (i.e., the confusingly named "unclassified" highway tag in OSM). The first thing is just to take a look at the output from the algorithm described above.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22438725678/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="pittsburgh_ua2"><img alt="pittsburgh_ua2" height="354" src="https://farm1.staticflickr.com/616/22438725678_f628c65875.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The same area as above, with each contiguous group of residential roads given a random colour.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>  Actually it's not quite as bad as I feared: rivers, railways and major roads all serve to break up the continuous network of residential roads. This effect is more pronounced in cities and large towns. The network of rural roads end up as very large continuous 'swiss cheese' polygons. In many cases these can be broken up by re-tagging only a small subset of roads from residential to unclassified (or service/track if applicable).<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22832538602/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="Kansas Residential Roads on OSM"><img alt="Kansas Residential Roads on OSM" height="256" src="https://farm6.staticflickr.com/5730/22832538602_b78d21dec6.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">All residential roads in Kansas from OSM<br />The background is the 3 minute interval grid used in building a graph of the roads.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> I also had a quick look at Kansas (a typical mid-Western state with a gridded road system determined by township boundaries), and Oregon (mountains, a very sparsely populated interior, but lots of active mappers in the Portland &amp; <a href="https://en.wikipedia.org/wiki/Willamette_Valley">Willamette Valley</a> areas).<br /><br />I'd half expected <a href="http://www.openstreetmap.org/user/ToeBee">ToeBee </a>to have done some tidying up of highway=residential in Kansas. He's certainly tidied up lots of other things, and he knows the rural roads well as a regular participant in the annual <a href="https://en.wikipedia.org/wiki/Biking_Across_Kansas">Biking Across Kansas</a> ride (many of his Mapillary pictures stem from these rides). The effect of a regular grid of roads, and no work to reclassify them from the original TIGER import is really obvious.<br /><br /><br />Oregon is substantially better, The local mapping community has clearly worked to improve the Tiger data along the Willamette Valley extending south from Portland. In these cases the built-up areas of local towns and cities stand out clearly. Like Pittsburgh, Portland itself is represented by several polygons divided by rivers, railways etc. Away from the centres of population we return to the 'swiss cheese; polygons. At least in some parts of the state, many of these residential roads are nothing more than forest tracks, or old farm tracks (as in the part of the John Day Fossil Beds where the family of my great-grandfather's brother ranched until around 1975). Some are, at best, vestigial.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22869041515/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="Urban areas derived from residential roads, Willamette Valley, OR"><img alt="Urban areas derived from residential roads, Willamette Valley, OR" height="640" src="https://farm1.staticflickr.com/681/22869041515_7fe8bf9375_z.jpg" width="453" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Willamette Valley, Oregon, showing urban areas derived from residential roads.<br />Outlying areas can be seen to be uncorrected Tiger data, resulting in 'swiss cheese' polygons.</td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> What this area of Oregon shows is that concerted local efforts to correct Tiger data can achieve decent results.<br /><br />Remember that just the process of reviewing Tiger data can lead to substantial improvements in alignment, road detail and other things. The real problem with Tiger data is its sheer abundance, which ends up being so off-putting (and boring) that rarely do mappers stick with it. Typical rural counties in Pennsylvania have 5000 or so ways tagged highway=residential, which is a hell of a lot to review, realign, check the surface type, correct other errors, etc. The sheer amount of data involved in imports often just overwhelms the capacity of local mappers to check and enhance the imported data. This factor should always be allowed for when planning imports: if the amount of data is beyond the capacity of the community to assimilate it then it often remains untouched. Even when the data is of good quality it will get dated quite quickly.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-RKcppWoq58g/Vj9SsoQ9Y0I/AAAAAAAAAzQ/G5dWYMWkL7c/s1600/imgus65_215c.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="http://3.bp.blogspot.com/-RKcppWoq58g/Vj9SsoQ9Y0I/AAAAAAAAAzQ/G5dWYMWkL7c/s320/imgus65_215c.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A barn near Valencia, Butler Co, 16th July 1966</td></tr></tbody></table>Whilst writing up this post, I've experimented with revising the classification of residential roads in Butler County, Pennsylvania. Butler is to the north of Pittsburgh, and even in the 1960s we had friends and neighbours who moved there, whilst continuing to work in the city. So although the county is still predominantly rural, there is a lot of dispersed residential settlement: either as small spur residential roads off main highways, or as non-contiguous lots along these roads. This type of settlement pattern definitely makes deciding whether residential or unclassified is the appropriate tag. One mapper has made a determined effort to map this type of landuse in a similar area just north of the Mason-Dixon Line.<br /><br /><div style="text-align: center;"><iframe frameborder="0" height="350" marginheight="0" marginwidth="0" scrolling="no" src="http://www.openstreetmap.org/export/embed.html?bbox=-77.86208152770996%2C39.7523372243077%2C-77.7662944793701%2C39.79092937807007&amp;layer=mapnik" style="border: 1px solid black;" width="425"></iframe><br /></div><div style="text-align: center;"><small><a href="http://www.openstreetmap.org/#map=14/39.7716/-77.8142">View Larger Map</a></small></div><br />I focussed solely on changing tags, with a small amount of addition of a surface tag when this was really obvious. Equally, you can see, that I mainly looked at longer roads. Not only are they much easier to assess at lower zoom levels, but changing the tags of a few longer roads has a disproportionately useful effect on things like routing or my goal. I also steered clear of the larger towns which would have required more detailed examination of the aerial imagery.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-sD3MATqfWZ8/Vj-E6feGWII/AAAAAAAAAzg/Z20Pl6ZvJ84/s1600/export%252818%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="374" src="http://4.bp.blogspot.com/-sD3MATqfWZ8/Vj-E6feGWII/AAAAAAAAAzg/Z20Pl6ZvJ84/s320/export%252818%2529.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Butler Co, PA: roads reassigned to unclassified.<br />My edits whilst writing this blog.</td></tr></tbody></table>This approach worked fine for my goal, and should also improve things like cycle routing. Many roads are still poorly aligned, but my overall feeling is that it is more productive to concentrate on one type of change.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22476112828/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Butler County, Pennsylvania : Urban Areas derived from OSM"><img alt="Butler County, Pennsylvania : Urban Areas derived from OSM" height="640" src="https://farm6.staticflickr.com/5692/22476112828_ecd745ba66_z.jpg" width="453" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Derived polygons for Butler Co, PA. <br />Pastel shades original polygons, grey edged with red, <br />derived after reclassification of rural roads (as seen aove)</td></tr></tbody></table><br /><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> The results of these changes compared to the PA dataset I started with are shown above.<br /><br />Obviously for the past several years most people making use of OSM data have not been impacted by the profusion of residential roads in rural areas (and indeed in BLM lands and National Forests). Is this because most applications are agnostic to such data in the US, or is OSM data just not used in such places? The only consumer I know which does place importance on such data is Richard Fairhurst's <a href="http://cycle.travel/">cycle.travel</a>, and he already makes use of landuse data to improve the quality of selected routes. It may be that highway=residential is so pervasive in the US that all data consumers have to work around this tagging, with the effect that there will be little incentive for regular mappers to change the tags.<br /><h3>Conclusions</h3>This post has taken a rather meandering route.<br /><ul><li><b>Firstly</b>, the general question: "Can useful results be obtained using a very naive approach to identify urban areas?" has been answered. It works well in areas with good mapping coverage, is a decent starting points for poorly mapped places, and only falls down when tagging practices are away from the norm.</li></ul><ul><li><b>Secondly</b>, the data produced can provide useful visualisations which can rapidly demonstrate areas where OSM lacks data, or existing data might be better tagged.<br />&nbsp;</li><li><b>Thirdly</b>. As for so many things with OSM, often the simplest way to work towards the data set one wants is to add more data to OSM or improve data which already exists.</li><li><b>Fourthly. </b>This approach crucially depends on availability of high quality aerial or satellite imagery for at least basic urban road networks to be mapped. Areas where only landsat data are available can never be identified with this technique.</li></ul>This is not the only approach I have considered. Next up will be using OSM landuse polygons directly.<br /><br />As a final note I'd like to express my particular appreciation of the work of local mappers in Pakistan, Nigeria &amp; Cote d'Ivoire. Much of the detailed analysis and issues discussed above was based on that work.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-6327863938317651791
url: http://sk53-osm.blogspot.com/2015/10/urban-areas-meditation-on-why-simple.html
title: Urban Areas: a meditation on why simple global geographical datasets are so poor
updated: 2015-10-25 15:38:33.011000
published: 2015-10-25 15:38:00.002000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.648925
updated_at: 2019-10-26 08:30:24.919500
content:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3APuerto-Vallarta.jpg" style="margin-left: auto; margin-right: auto;" title="By No machine-readable author provided. Intersofia assumed (based on copyright claims). [GFDL (http://www.gnu.org/copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/) or CC BY 2.5 (http://creativecommons.org/licenses/by/2.5)], via Wikimedia Commons"><img alt="Puerto-Vallarta" src="https://upload.wikimedia.org/wikipedia/commons/thumb/0/01/Puerto-Vallarta.jpg/512px-Puerto-Vallarta.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Puerto Vallarta, an aerial view of an urban area missing many roads on OpenStreetMap.<br />The area in the middle distance away from the sea was particularly lacking.<br />Fortunately the centre of the hurricane didn't pass over this area.<br />Source: Wikimedia Commons, (c) CC-BY-SA</td><td class="tr-caption" style="text-align: center;"></td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><br />The other night, as <a href="https://en.wikipedia.org/wiki/Hurricane_Patricia">Hurricane 'Patricia' </a>bore down on the Pacific coast of Mexico, I had a twitter conversation with Bill Morris and others regarding how well mapped <a href="https://en.wikipedia.org/wiki/Puerto_Vallarta">Puerto Vallarta</a> was on OpenStreetMap. (BTW: I'm sure it's much better mapped <a href="http://overpass-turbo.eu/s/cfs">now</a>).<br /><blockquote class="twitter-tweet" lang="en"><div dir="ltr" lang="en">Latin America is a weak spot for <a href="https://twitter.com/openstreetmap">@openstreetmap</a>. A town in the <a href="https://twitter.com/hashtag/Patricia?src=hash">#Patricia</a> track: <a href="https://t.co/pk4UYHHan3">https://t.co/pk4UYHHan3</a> <a href="https://t.co/3t7ZQkWkWz">pic.twitter.com/3t7ZQkWkWz</a></div>— Bill Morris (@vtcraghead) <a href="https://twitter.com/vtcraghead/status/657567974442835968">October 23, 2015</a></blockquote><script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script> Of course, OSM is about fixing things, so I carried out the conversation in between adding around a hundred streets to the city. However the really interesting question was this one:<br /><blockquote class="twitter-tweet" data-conversation="none" lang="en"><div dir="ltr" lang="en"><a href="https://twitter.com/vtcraghead">@vtcraghead</a> <a href="https://twitter.com/hotosm">@hotosm</a> <a href="https://twitter.com/openstreetmap">@openstreetmap</a> Would there be some sort of algorithm that can be run to detect built up areas from imagery not mapped?</div>— Antonio Locandro (@antoniolocandro) <a href="https://twitter.com/antoniolocandro/status/657664713224929281">October 23, 2015</a></blockquote><script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script> <br />Whilst at breakfast I thought a little more about this. I decided it ought to be possible to do something fairly simple with data which already exists.<br /><br /><a name='more'></a>My basic idea was as follows: create a set of urban area polygons (from the <a href="http://www.naturalearthdata.com/downloads/10m-cultural-vectors/10m-urban-area/">Natural Earth data</a>) for cities over a certain size (from <a href="http://download.geonames.org/export/dump/">Geonames/GNS data</a>), and use these polygons to count total road length and other parameters of OSM data. I would then plot these values against population for various areas (continents, countries), choose well-mapped places and build a regression model to predict poorly mapped places.<br /><br />Of course I though the long-winded bit would be extracting OSM data for several thousand cities and then processing to get the various metrics.&nbsp; Unfortunately I haven't got that far!<br /><br />As with any sensible analytical approach it's always worthwhile visualising the data: mainly because this is the easiest way to do some basic sense checks. Sadly, both my datasets failed this initial step.<br /><br /><h3>Natural Earth Urban Areas</h3>Even before I loaded the city data I was suspicious of the utility of the Natural Earth data. I looked in two sparsely populated areas which are familiar to me: <a href="https://en.wikipedia.org/wiki/Patagonia">Patagonia </a>&amp; <a href="https://en.wikipedia.org/wiki/Extremadura">Extremadura</a>. In both cases there were urban areas which looked spurious at first glance: NW of Rio Gallegos, and somewhere close to the <a href="https://en.wikipedia.org/wiki/Sierra_de_Gata">Sierra de Gata</a>.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-arf_UTiDY2I/ViteAuJcXUI/AAAAAAAAAwI/3ETv5eFmcDQ/s1600/urban_patagonia.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://3.bp.blogspot.com/-arf_UTiDY2I/ViteAuJcXUI/AAAAAAAAAwI/3ETv5eFmcDQ/s320/urban_patagonia.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Spurious Urban Areas (outlined in orange) in Patagonia from Natural Earth<br />Imagery via Bing (most likely Landsat).</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-FBlbp9JLlhg/Vitf-NeUKbI/AAAAAAAAAwk/jPTIhKPdkFQ/s1600/rio_gallego_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://2.bp.blogspot.com/-FBlbp9JLlhg/Vitf-NeUKbI/AAAAAAAAAwk/jPTIhKPdkFQ/s320/rio_gallego_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/R%C3%ADo_Gallegos,_Santa_Cruz">Rio Gallegos</a> (population 100,000) on the same scale as above.<br />The urban area is clearly visible, but not in Natural Earth</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-oNDNMXmsoCw/Vitf8diymzI/AAAAAAAAAwg/ak66RVvL85Y/s1600/ushuaia_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://2.bp.blogspot.com/-oNDNMXmsoCw/Vitf8diymzI/AAAAAAAAAwg/ak66RVvL85Y/s320/ushuaia_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Ushuaia">Ushuaia </a>(population 50,000) on the same scale as above.<br />The urban area is clearly visible.</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-lcs0SGY21IY/VitgAMfzfHI/AAAAAAAAAww/0F8BmdiPP3I/s1600/punta_arenas_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://4.bp.blogspot.com/-lcs0SGY21IY/VitgAMfzfHI/AAAAAAAAAww/0F8BmdiPP3I/s320/punta_arenas_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Punta_Arenas">Punta Arenas </a>(population 127,000) on the same scale as above.<br />The urban area is clearly visible.</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-p_Rpcgsodhk/Vitf3fx_JLI/AAAAAAAAAwY/HYdnRVqKUNg/s1600/el_calafate_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://4.bp.blogspot.com/-p_Rpcgsodhk/Vitf3fx_JLI/AAAAAAAAAwY/HYdnRVqKUNg/s320/el_calafate_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/El_Calafate">El Calafate</a> (population 22,000) at a much larger scale.<br />For once some correspondence of urban areas.</td></tr></tbody></table>Furthermore many of the larger cities in these areas had no urban areas in the Natural Earth data: in Patagonia these include Rio Gallegos, Ushuaia and Punta Arenas; in Extremadura, Caceres is missing.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-N_pjvgAa-I4/Viti2VIRyyI/AAAAAAAAAw4/3hcLJ2_Ej8E/s1600/caceres_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://4.bp.blogspot.com/-N_pjvgAa-I4/Viti2VIRyyI/AAAAAAAAAw4/3hcLJ2_Ej8E/s320/caceres_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/C%C3%A1ceres,_Spain">Cáceres</a> (population 90,000), Extremadura.<br />The urban area is clearly visible.</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-IlQPtKwXLTU/Viti4S6-P-I/AAAAAAAAAxA/l3nfXJnXBaw/s1600/coria_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://3.bp.blogspot.com/-IlQPtKwXLTU/Viti4S6-P-I/AAAAAAAAAxA/l3nfXJnXBaw/s320/coria_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Coria,_C%C3%A1ceres">Coria </a>(pop 12,000) a small town in Extremadura <br />which has an urban polygon in Natural Earth.</td></tr></tbody></table>This all without looking at the data in more populated parts of the world. So I had a look at Western Europe. Firstly, I looked at my main mapping area Nottinghamshire. Two problems were immediately apparent: many urban areas are shown as one contiguous strip; minor towns were appearing with substantial urban areas.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-7yLEQwOzFYo/VitmDLrLKYI/AAAAAAAAAxM/_NkrWadyQkE/s1600/northern_powerhouse_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://2.bp.blogspot.com/-7yLEQwOzFYo/VitmDLrLKYI/AAAAAAAAAxM/_NkrWadyQkE/s320/northern_powerhouse_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Natural Earth Urban Areas: showing the '<a href="https://en.wikipedia.org/wiki/Northern_Powerhouse">Northern Powerhouse</a>' area.<br />Urban areas for 7 major cities are run together in the centre.</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-y7nEzOQJ4q4/VitmgBolSqI/AAAAAAAAAxU/ynU3ITFfXkU/s1600/gainsborough_ne.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://4.bp.blogspot.com/-y7nEzOQJ4q4/VitmgBolSqI/AAAAAAAAAxU/ynU3ITFfXkU/s320/gainsborough_ne.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Urban area polygon for <a href="https://en.wikipedia.org/wiki/Gainsborough,_Lincolnshire">Gainsborough </a>(pop: 20,000).<br />This is very rural agricultural land as can be seen by my <a href="http://www.mapillary.com/map/im/CXBj9akiFy17dmB2rciELA/photo">Mapillary </a>traces!</td></tr></tbody></table>In practice false positives don't matter too much as they can be filtered out by only selecting urban areas for larger cities. Contiguous areas can be separated using <a href="https://en.wikipedia.org/wiki/Voronoi_diagram">Voronoi </a>triangulation of cities to split the polygons (as suggested on the Natural Earth site, where it is called Thiessen polygons), but more on this below.<br /><br />The key point is that I want a polygon set which is reliable for places I don't know in Central America, Brazil, Indonesia, Africa, India and China.&nbsp; A cursory inspection shows that it is unreliable in places I do know.<br /><br />Straightforward conclusions about this dataset are:<br /><ul><li>It is not usable for analytical applications</li><li>It has major errors for cartographical applications</li><li>Relying on Remote Sensing alone doesn't always give good data</li><li>No-one sense checked the data.</li><li>Not all Natural Earth data has been curated and checked for fitness for purpose.</li></ul>I still need a decent set of urban area polygons! <br /><h3>Geonames (15k+ populated places)</h3>The other dataset I needed was a geolocated list of cities, of say over 100,000.<br /><br />Wikipedia has such a list, but the idea was to do something quick, not learn how to scrape Wikipedia.<br /><br />There seemed to be two readily available datasets: from the UN Statistical Division and from Geonames. The UN data lacks lat/lon so that was out. I've used the Geonames data of cities with populations over 15,000 before: it certainly has the types of data I wanted.<br /><br />Once again I did a quick visualisation, and just as before problems were immediately apparent. Take a look at the image of Northern England. Firstly, note the node labelled High Peak: this is not a city/town it's an administrative district. Secondly, look for <a href="https://en.wikipedia.org/wiki/Derby">Derby </a>(pop: 250,000) &amp; <a href="https://en.wikipedia.org/wiki/Grimsby">Grimsby </a>(pop: 90,000).: they're not there, are they. Scotland is affected too. No <a href="https://en.wikipedia.org/wiki/Perth,_Scotland">Perth</a>, <a href="https://en.wikipedia.org/wiki/Dundee">Dundee </a>or <a href="https://en.wikipedia.org/wiki/Inverness">Inverness</a>. I didn't bother looking further.<br /><br />For my purpose it's important that the population data are also reasonably accurate.<br /><br />My first check, Nottingham, is shown as having a population of 246,000. The last time the city had such a low population was around 1900 (see the Demographics section <a href="https://en.wikipedia.org/wiki/Nottingham#History">here</a>): current estimated population is 314,000 for the city and 730,000 for the urban area. A 33% error is probably too much for the purpose.<br /><br />So I need a better list of cities, with more useful population figures too!<br /><h4>Combining the two</h4>On a worldwide basis around 4000 places with populations over 100,000 are not covered by the Natural Earth urban polygons. <a href="https://en.wikipedia.org/wiki/Mindanao">Mindanao </a>in the Philippines seems particularly badly affected:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-QPrRRFLsOOo/Vitx0W7Bv5I/AAAAAAAAAxk/gTZ87AtcHpY/s1600/mindanao_gns.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="303" src="http://2.bp.blogspot.com/-QPrRRFLsOOo/Vitx0W7Bv5I/AAAAAAAAAxk/gTZ87AtcHpY/s320/mindanao_gns.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Places over 100,000 from GNS data without corresponding urban area from Natural Earth</td></tr></tbody></table>Again not good.<br /><br />Some of the missing places are actually a reflection of the generalisation of the NE data: this mainly shows up along coastlines. In these cases the centroid of the city is not just outside the urban area of NE data, but in the sea!<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-79ShWtQcZ7o/Vit4VZBH1-I/AAAAAAAAAx0/aTreMKf15Cg/s1600/missing_cities_ne_gns.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="293" src="http://1.bp.blogspot.com/-79ShWtQcZ7o/Vit4VZBH1-I/AAAAAAAAAx0/aTreMKf15Cg/s320/missing_cities_ne_gns.jpg" width="480" /></a></div><br /><br />And of course losing places like Derby makes the idea of cutting the urban areas up using Voronoi triangulation also suspect:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-JVN-Bwt9wtQ/ViuEBnOJ5FI/AAAAAAAAAyA/Jov1Tm40HNA/s1600/city_voronoi.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="276" src="http://1.bp.blogspot.com/-JVN-Bwt9wtQ/ViuEBnOJ5FI/AAAAAAAAAyA/Jov1Tm40HNA/s320/city_voronoi.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Urban Areas for GNS populated places over 50000<br />cut using Voronoi polygons&gt;</td></tr></tbody></table>Obviously Nottingham now includes most of Derby and has a population value which is too small. This is a heavily mapped area so I'd predict this as a significant outlier. Doncaster also seems to be missing, and Rotherham is mopping up many smaller places.<br /><br />At least this is an area I can look at statistically. I just did this for a single metric (non-motorway road length in metres) using OSM data from May 2015. I counted road length within each polygon and tweaked the population figure by adding in the population of all city nodes within the polygon. I plotted the results using R.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-xMu81_1g-L0/Vium-808ubI/AAAAAAAAAyQ/L9imuoMMcF0/s1600/gb_roads_vs_pop.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="231" src="http://2.bp.blogspot.com/-xMu81_1g-L0/Vium-808ubI/AAAAAAAAAyQ/L9imuoMMcF0/s320/gb_roads_vs_pop.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Plot of road length (in m) on Y axis (V3) against Population<br />for most urban areas in Great Britain</td></tr></tbody></table>This uncovered another problem with the data: there were nodes for both London and City of London each with assigned the population of the Greater London area. As the data also included most significant London suburbs these two records caused lots of problems. Fortunately their Voronoi Polygons were small and just removing them allowed a more sensible plot (see above), and even a decent enough regression line.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-EqL67N0HaDI/ViuowTbSnuI/AAAAAAAAAyc/O0xLzWfoKH0/s1600/cities_regression.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="231" src="http://3.bp.blogspot.com/-EqL67N0HaDI/ViuowTbSnuI/AAAAAAAAAyc/O0xLzWfoKH0/s320/cities_regression.png" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Plot as above Road Length on OSM (V3) vs Population (V4)<br />with added regression line.</td></tr></tbody></table>&nbsp;I spent too long fiddling with the data to have time to spend tarting up my R plots. Apologies.<br /><br /><h3>Urban Areas in OpenStreetMap</h3>So we could do with better Urban Area data: why not OSM? <br /><br />Analytical uses did not figure over much at the outset of OSM. I suspect I was relatively unusual in seeing OSM as a potential way to capture data for analytical uses. As I have tried to show for other data there are no good technical reasons why OSM data can't be used for analytics: it is absence, patchiness and inconsistency of data which are much more problematic.<br /><br />Whilst not of any consolation for people who want the data now, it's not something I worry about over much. A few years ago OSM wasn't much use for car routing: more mappers and a greater variety of source data fixed it.<br /><br />I think it's a fairly natural progression for mappers to move on to capturing other features once the road network is in place: in the main these are the things of most use for analytical applications. Furthermore in places where mappers are focused on maintaining data consistency becomes more desirable <b>for the mapper</b>: it really helps spot change. Mappers are also often motivated by seeing the impact of their activities: thematic maps can nudge contributors to add things outside their own direct interests: <a href="https://twitter.com/rjw62">Robert Whittaker</a>'s <a href="http://robert.mathmos.net/osm/postboxes/">Post Hoc</a> site is a good example.<br /><br />There is another problem, which I find more intriguing. Urban areas exemplify this perfectly. We tend to map things at a fine granular level; and the grains get smaller as detail gets added. A city mapped in a lot of detail will have well over ten different types of landuse/landcover: residential, commercial, parks, recreation grounds, pitches, hospitals, education, railways, brownfield, construction, retail, allotments, industrial, cemeteries &amp; so on. There is no single tag to identify a built-up urban area.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-8txlTqWesjM/VizpkJ_NlkI/AAAAAAAAAys/1BHMi3nHcAA/s1600/osm_builtup2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="261" src="http://4.bp.blogspot.com/-8txlTqWesjM/VizpkJ_NlkI/AAAAAAAAAys/1BHMi3nHcAA/s320/osm_builtup2.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Candidate Urban Areas from OpenStreetMap cf. NE polygons<br />OSM data includes a wide range of polygon types.<br />Note industrial area in North Sea, and Chatsworth Park in Peak District: good examples of potential spurious data<br />Source: (c) OpenStreetMap contributors &amp; Natural Earth</td></tr></tbody></table>It also turns out that we can't rely on some concatenation of all the different landuses: there are numerous exceptions for many of the categories which are not urban. Examples include industrial areas in the countryside (nuclear power stations, water works), formal parks associated with country estates, sports clubs and so on. However, at least for Great Britain, it does provide a reasonably convincing candidate dataset.<br /><br />We can try a couple of fudges: first merge polygons matching those criteria which touch or overlap. Merged areas can be tidied up by enlarging them with a positive buffer value and then shrinking them again. This will probably fill any holes left my unmapped landuse, or categories not selected initially. I had a quick go at this for Derby. The results compared with the NE data, my source data from OSM and urban areas from the Ordnance Survey Meridian 2 Open Data are shown below.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-vo1QS_565XM/VizwSBdhc5I/AAAAAAAAAy8/MtcQbHZwh5U/s1600/derby_ua.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="http://2.bp.blogspot.com/-vo1QS_565XM/VizwSBdhc5I/AAAAAAAAAy8/MtcQbHZwh5U/s320/derby_ua.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Comparison between non-OSM sources for Urban Area around Derby<br />and candidate polygons from OSM (orange: original; blue stipple: merged with some gaps filled using buffering).<br />Sources: OpenStreetMap; Ordnance Suryey Meridian 2 (Open Data); Natural Earth. </td></tr></tbody></table><h3>Conclusions &amp; Next Steps</h3>Doing simple geo-experiments with existing worldwide datasets is not really possible. Most available data has not really been created for any kind of analytical purpose. Furthermore the datasets often require substantial cleaning up to remove obvious artefacts (erroneous data, duplicates etc). Many really require additional sources of data, and in many cases OpenStreetMap is a really convenient way to do so (more on this at some later time).<br /><br />I'm actually going to have a go at trying to improve the datasets referenced here, and shortly will create projects on GitHub to do so. It turns out that decent Urban Area data would be useful for all sorts of things.<br /><br />However, I'll first return to the issue of deriving built-up areas from OSM: it's something I've thought about from time-to-time in the past, and the Derby experiments look better than I expected. I'll also have to give more thought as to how we can identify less well mapped cities given that my first idea won't work.


guid: tag:blogger.com,1999:blog-2082422783692631078.post-6591896709243389584
url: http://sk53-osm.blogspot.com/2015/10/irish-vice-counties-creation-of.html
title: Irish Vice Counties : the creation of a specific dataset on OpenStreetMap
updated: 2015-10-16 14:52:21.709000
published: 2015-10-16 13:52:00.002000
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.651915
updated_at: 2019-10-26 08:30:24.923344
content:
  I've been meaning to write about <a href="http://www.openstreetmap.ie/" target="_blank">OpenStreetMap Ireland</a>'s <a href="https://en.wikipedia.org/wiki/Townland" target="_blank">townland</a> <a href="http://wiki.openstreetmap.org/wiki/Ireland/Mapping_Townlands" target="_blank">mapping project </a>for some time.<br /><br />It's a wonderful example of how historical maps are of significant value in creating really useful data on OpenStreetMap which is just as relevant as it today as was in the past.<br /><br />The immediate reason for writing about them is that I have been creating vice county boundaries for Ireland. In doing so I have not just been using the data, but the fantastic range of resources made available through the activities of the <a href="http://www.townlands.ie/" target="_blank">townlands project</a>.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AIrish_vice_counties_ex_osm_multicolour.svg" style="margin-left: auto; margin-right: auto;" title="By Semudobia (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons"><img alt="Irish vice counties ex osm multicolour" src="https://upload.wikimedia.org/wikipedia/commons/thumb/f/f4/Irish_vice_counties_ex_osm_multicolour.svg/512px-Irish_vice_counties_ex_osm_multicolour.svg.png" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The Vice Counties of Ireland<br />My first complete draft of the boundaries on OpenStreetMap</td></tr></tbody></table><br /><a name='more'></a>I've written a little about <a href="https://en.wikipedia.org/wiki/Vice-county" target="_blank">vice counties</a> (VCs) before (<a href="http://sk53-osm.blogspot.co.uk/2013/02/mundane-cartography-mapping-vice.html" target="_blank">here </a>and <a href="http://sk53-osm.blogspot.co.uk/2013/01/ashes-to-ashes-mapping-mappers.html" target="_blank">here</a>). I was in Northern Ireland to attend the Summer Meeting of the BSBI (see <a href="https://twitter.com/oshduffy" target="_blank">Oisin Duff</a>y's great write-ups and pictures on <a href="https://oisinduffy.wordpress.com/tag/bsbi/" target="_blank">his blog</a>). As the meeting was attended by botanical recorders for 5 out of the 40 vice counties in Ireland and I have long known that decent data on boundaries is not widely available, I thought I would try &amp; repay a little of their kindness by using OSM to create a set of boundaries suitable for use in GIS systems (and possibly publications: many recent <a href="http://www.bsbi.org.uk/rare_plants.html" target="_blank">Rare Plant Registers</a> make use of reproductions of the original maps). (And just to be clear: VCs  really exist, are not a funny kind of barony, don't have anything to do  with Viscounts, and are used in real life.)<br /><h2>A Bit of Background </h2>&nbsp;Both townlands and vice counties are slightly specialised geographic units. <br /><h4>The Townlands Project </h4>Ireland has over 60,000 <a href="https://en.wikipedia.org/wiki/Townland">townlands</a>. They are not only the smallest administrative unit, used to construct larger units, but also have a long history. Some townlands reflect land tenure before the Normans arrived in Ireland; others have been created for later country estates. Even today townland boundaries experience minor changes: often to reflect natural changes in part of the boundary.<br /><br />Until very, very recently, townlands were also often an important part of the address, particularly in rural areas. (The very recently is because the Republic of Ireland has finally introduced a postcode, <a href="https://en.wikipedia.org/wiki/Postal_addresses_in_the_Republic_of_Ireland#Eircode">EirCode</a>; Northern Ireland has had postcodes since the 1970s). As both administrative units &amp; address components, townlands have been very important for people tracing their ancestry in Ireland.<br /><br />Townland names are still used in day-to-day life throughout the island. On the BSBI trip, two of the locations we visited were described using the townland name (<a href="http://www.openstreetmap.org/relation/4784274">Umbra</a>, <a href="http://www.openstreetmap.org/relation/4873450">Moys</a>). Some counties in Northern Ireland incorporate the townland name on street signs.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AModern_Streetname_sign_with_Townland_Name_(geograph_4700760).jpg" style="margin-left: auto; margin-right: auto;" title="SK53 [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Modern Streetname sign with Townland Name (geograph 4700760)" src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/24/Modern_Streetname_sign_with_Townland_Name_%28geograph_4700760%29.jpg/512px-Modern_Streetname_sign_with_Townland_Name_%28geograph_4700760%29.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A road sign with the townland name (Ballyholly) in green under the road name.<br />Photo: the author via Geograph/Wikimedia Commons CC-BY-SA</td></tr></tbody></table>The antiquity of many townland boundaries (and, perhaps, also because Ireland has not suffered agricultural intensification to the same extent as other parts of Western Europe), means that lots of these are readily visible in aerial photographs: walls, hedgerows, streams, rivers, landuse. With old maps showing the boundaries it is possible to match these to features still extant in the landscape.<br /><br />Thus <a href="http://wiki.openstreetmap.org/wiki/Ireland/Mapping_Townlands">townlands are highly useful things</a> to add to OpenStreetMap. They help delineate larger administrative units, provide useful information for addressing, for genealogists, local historians and naturalists. They are also an aspect of cultural identity which is precious to many (hence their relatively recent appearance on road signs).<br /><br />The first person to map a townland on OSM was larryone who added <a href="http://www.openstreetmap.org/relation/1016461/history">Donard Mountain </a>back in <a href="http://www.openstreetmap.org/changeset/5091148">June 2010</a>. I certainly remember being shown a solitary townland a few years ago, probably <a href="http://www.openstreetmap.org/relation/1143928">Caulstown</a>, mapped by Dermot McNally in August 2010. Later on most of the townlands of County Carlow were added by IrlJdel and brianh and Rory McCann created a simple website to show what had been done.<br /><br />At this point things started happening rather more rapidly: <a href="http://www.openstreetmap.ie/resources/maps/">GSGS 3906</a> (UK military maps of Ireland) became available from the <a href="https://www.tcd.ie/Library/map-library/">Glucksman Map Library, Trinity College Dublin</a>; Rory revamped his webpages to create the current <a href="http://townlands.ie/">townlands.ie</a> site; more areas started to look complete; Dave Corley compiled an extremely thorough set of <a href="https://www.youtube.com/playlist?list=PLwUy6ph8y4kFO8mmUgv87f16e4dxRh8NV">"How To.." videos</a>; and overlays were made available on the OSM Ireland website.<br /><br />Even a couple of years ago this looked like, and was described as, a multi-year project. However, the combination of more people, greater experience, better feedback, shared knowledge, and, perhaps, a little bit of competitive feeling, has dramatically shortened this timescale. At the time of writing, the <a href="http://www.townlands.ie/progress/rate/">predicted completion date</a> is now around June next year.<br /><h4>Vice Counties in Ireland: a bit of background </h4><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/16271433@N02/18366619054/" style="margin-left: auto; margin-right: auto;" title="Binevenagh Mountain"><img alt="Binevenagh Mountain" height="265" src="https://farm1.staticflickr.com/430/18366619054_9be01a27f9.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">VC40 Londonderry &amp; VC35 East Donegal: view from Binevenagh<br />The Foyle river &amp; estuary divide these two VCs.<br />The flat farmland on the right bank of the estuary was the site of the baseline of the primary triangulation of Ireland.<br />Photo<a href="https://www.flickr.com/photos/16271433@N02/"> John Purvis</a> by on Flickr <a href="https://creativecommons.org/licenses/by/2.0/">cc-by2.0</a></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> The key purpose of Vice Counties is to have <b>permanent unalterable</b> units for recording wildlife. This in turn allows sensible comparison of data across time periods. Attempts to follow administrative boundaries just falter because they change too frequently. They were first introduced for Great Britain and surrounding Islands by <a href="https://en.wikipedia.org/wiki/Hewett_Watson" target="_blank">Watson</a>; whilst the Irish equivalent was created by <a href="https://en.wikipedia.org/wiki/Robert_Lloyd_Praeger" target="_blank">Robert Lloyd Praeger</a>, a towering figure in <a href="http://www.botanicgardens.ie/herb/census/webbheyday.htm" target="_blank">Irish Natural History</a> during the late 19th and early 20th centuries.<br /><br />Vice Counties in Ireland are rather simpler to map than their equivalents in Great Britain. Firstly the traditional Irish counties have fewer size outliers than their British counterparts (no <a href="https://en.wikipedia.org/wiki/Rutland">Rutlands</a>, or <a href="https://en.wikipedia.org/wiki/Clackmannanshire">Clackmannans</a>). Thus many of the vice counties are directly based on the traditional county boundaries. The main exceptions being where towns on border rivers had grown sufficiently large, that even by the end of the 19th century, the urban areas had been transferred from one county to another. Even the number of these cases is quite small: for example. <a href="https://en.wikipedia.org/wiki/New_Ross" target="_blank">New Ross</a>, <a href="https://en.wikipedia.org/wiki/Waterford" target="_blank">Waterford</a>, <a href="https://en.wikipedia.org/wiki/Carrick-on-Suir" target="_blank">Carrick-on-Suir</a>, <a href="https://en.wikipedia.org/wiki/Bray" target="_blank">Bray</a>, and <a href="https://en.wikipedia.org/wiki/Limerick" target="_blank">Limerick</a>. They are also very easy to deal with because the VC boundary is the river. One additional case, which is more intriguing, is the City of <a href="https://en.wikipedia.org/wiki/Derry">Derry</a> or <a href="https://en.wikipedia.org/wiki/Derry/Londonderry_name_dispute">Londonderry </a>which lies on the left bank of the <a href="https://en.wikipedia.org/wiki/River_Foyle">Foyle</a>. The vice county boundary follows the Foyle, placing the city in East Donegal. Thus the vice county not only does not follow internal administrative boundaries but includes the international boundary. Of course this is sensible: wildlife don't have passports.<br /><br />The larger counties along the western seaboard and in <a href="https://en.wikipedia.org/wiki/Munster" target="_blank">Munster </a>were too big to be single units, and were split. Four (<a href="https://en.wikipedia.org/wiki/County_Kerry">Kerry</a>, <a href="https://en.wikipedia.org/wiki/County_Tipperary">Tipperary</a>, <a href="https://en.wikipedia.org/wiki/County_Mayo">Mayo </a>&amp; <a href="https://en.wikipedia.org/wiki/County_Donegal">Donegal</a>) into 2 vice counties each, and two (<a href="https://en.wikipedia.org/wiki/County_Cork">Cork </a>and <a href="https://en.wikipedia.org/wiki/County_Galway">Galway</a>) into three. The means chosen for choosing an internal boundary were rather diverse:<br /><ul><li>Historical <a href="https://en.wikipedia.org/wiki/Barony_%28Ireland%29" target="_blank">Barony </a>boundaries (Kerry &amp; Donegal).</li><li>Railway Lines (Tipperary, Mid/East Cork, North-East/South-East Galway boundaries)</li><li>Lines of Crow's Flight (Mid/West Cork)</li><li>Vague descriptions (more later)</li></ul>Overall one has the impression that Praeger took a bit of care over most, but was rushed when delineating the Cork divisions (perhaps he just needed to finish his book, <i>Irish Topographical Botany</i>)<br /><br />In addition to these internal divisions there were a number of areas which moved between administrative counties at the end of the 19th century. Praeger was obviously aware of some and, ignored, or was ignorant of, others. In most cases he used the older boundaries, but in a couple of places he used the new ones.<br /><br />The most useful reference describing all these aspects, that is, apart from Praeger's original work is a paper in by <a href="https://en.wikipedia.org/wiki/D._A._Webb">Webb </a>(conveniently made available <a href="http://www.botanicgardens.ie/herb/census/webbvcs.htm">online</a> by the <a href="https://en.wikipedia.org/wiki/County_Tipperary">National Botanical Garden</a> at <a href="https://en.wikipedia.org/wiki/Glasnevin">Glasnevin</a>). Also of interest are Praeger's itineraries whilst preparing the book, now made <a href="http://www.botanicgardens.ie/herb/books/itbitineraries.htm#segalway">available online</a> by the NBG.<br /><br /><h3>Mapping the VC Boundaries</h3><h4>First Steps </h4>The obvious thing was to map the simple ones first: those where the VC boundaries directly corresponded with the historical county boundary. These are copies of the existing boundary for the historical/traditional county.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AGeograph-2425274-by-Oliver-Dixon.jpg" style="margin-left: auto; margin-right: auto;" title="By Oliver Dixon (From geograph.org.uk) [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Geograph-2425274-by-Oliver-Dixon" src="https://upload.wikimedia.org/wikipedia/commons/4/44/Geograph-2425274-by-Oliver-Dixon.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">VC H16 West Galway: Foot of Lough Nafooey<br />The Mayo/Galway county boundary runs along the river draining the lough, but all of the area is in VC16.<br />Mainly unimproved and rough pasture in foreground. Moorland &amp; blanket bog on hills.</td></tr></tbody></table>Next I attacked those where the differences between the VC and the traditional county were minor (mainly river boundaries). Again a simple copy of the existing boundary, addition of the river and deletion of the overlap was all that was needed).<br /><br />Finally I looked at the split counties (and major changes which actually mainly involved bits of County Mayo). For Cork, Tipperary and Galway where the railway line is used to split the county I could simply use the existing ways from OSM. The Mid/West Cork boundary is even simpler as it mainly involves two straight lines. The complicated ones were Kerry, Donegal and Mayo: where the boundaries were more involved. For the first two the split line follows barony boundaries: I was able to use the Memorial Atlas (courtesy of Library, NUI Galway) which shows barony and civil parish boundaries. In the first case I very crudely followed the line. For Mayo I made a very crude approximation of the split line from the top of Lough Mask to Ballina.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3APanorama_Inis_Me%C3%A1in.jpg" style="margin-left: auto; margin-right: auto;" title="By Paucabot (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons"><img alt="Panorama Inis Meáin" src="https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Panorama_Inis_Me%C3%A1in.jpg/512px-Panorama_Inis_Me%C3%A1in.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">VC9 Clare: Inis Meáin <br />The Aran Isles are place in VC9, but administratively are part of County Galway</td></tr></tbody></table>At this stage I had 40 VCs, and had repaired any inadvertent damage to other boundaries (see below).<br /><br /><h4>Fixing the complicated ones </h4>The next step was to manage the larger transfers between counties: around Joyce Country in SW Mayo, around Ballina on Sligo/Mayo border and between Roscommon and Mayo. In order to do this effectively I had to map the townlands in the affected area, and civil parish boundaries too. It would appear that, in the main, the transfers between administrative counties in the late 1800s were of civil parishes: at least by using civil parishes I appear to get results consistent with Webb's paper.<br /><br /><h4>Tweaks &amp; Refinement </h4>Finally, the boundaries need to be made more accurate.<br /><br />For many counties this has already been done in the process of mapping townlands. However, for Munster in particular, the county boundaries on OSM are only very approximate (e.g., based on a line enclosing places in one particular county). I use an iterative approach:<br /><ul><li>Refine the boundary using the <a href="http://maps.openstreetmap.ie/oocmaps.html?zoom=8&amp;lat=53.55822&amp;lon=-7.85303&amp;layers=0000BFFFFF">Memorial Atlas</a> or the <a href="http://maps.openstreetmap.ie/oocmaps.html?zoom=8&amp;lat=53.55822&amp;lon=-7.85303&amp;layers=00B00FFFFF">Bartholomew </a>quarter inch map.</li><li>Refine it again using the <a href="http://maps.openstreetmap.ie/oocmaps.html?zoom=8&amp;lat=53.55822&amp;lon=-7.85303&amp;layers=0B000FFFFF">GSGS 4136</a> 1 inch map (this has not been rectified with a high degree of accuracy and it helps to use image offsets in JOSM, or shift the image in other OSM editors). At this stage the boundary should be no more than around 100m from its actual location.</li><li>Final refinement requires mapping the townlands along one side of the border. This is more laborious, and so far I have only done this completely for the Cork/Limerick border. On the other hand this is a direct contribution to the townlands project <b>and </b>results in significantly better boundaries for actual administrative units. The processes involved are fully described in Dave Corley's <a href="http://wiki.openstreetmap.org/wiki/Ireland/Mapping_Townlands#Mapping_townlands_video_series">videos</a>, but use both the GSGS 3906 map (photo-reduced from older 6 inch maps), Bing imagery, and existing OSM data. Ideally this process should produce data of similar accuracy to&nbsp; hand-held GPS. In practice fairly featureless upland terrain and wooded incised river valleys cant be resolved with such confidence.</li></ul>As more data are added to OSM, and in particular data collected in the field, the fine-scale mapping can be further refined: so this is always a process of continual refinement &amp; iteration. <br /><br /><h4>Technical Aspects (OSM)</h4>It's worth making a few comments about the technical aspects, and specifically large scale manipulation of boundary relations.<br /><br />Creating a whole set of boundaries encompassing one country and part of another is not a light undertaking on OSM. It is fiddly work, and involves manipulating objects with many dependencies. In practice I find it somewhat reminiscent of software migration projects: mainly mundane but you need to keep your wits about you.<br /><br />Contrary to what some believe, none of the OSM editors can prevent damage to other objects in this process. Mapping boundaries on this scale inevitably involves relations, and relations are not semantically clean objects at the level of the OSM data model. It is obvious that some inter-relationship between geographical elements is needed in the data model, so relations make sense. However, they are also in many ways a 'get-out-of-jail-card', enabling one to say "of course OSM can do this, just use relations", In other words a kludge. I am thinking of only widely used relations in this context (multipolygon, route, boundary and enforcement): many others are either semantically obscure or too little used. In many ways they are the equivalent in SQL UPDATE statements which delete rows, or a failure to distinguish between a string with no characters and NULL (and Oracle, I'm thinking of you!).<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AEarth_wall_towards_Knockmealdown_-_geograph.org.uk_-_1154339.jpg" style="margin-left: auto; margin-right: auto;" title="Bryan Conlon [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Earth wall towards Knockmealdown - geograph.org.uk - 1154339" src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d8/Earth_wall_towards_Knockmealdown_-_geograph.org.uk_-_1154339.jpg/256px-Earth_wall_towards_Knockmealdown_-_geograph.org.uk_-_1154339.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Wall in Knockmealdown Mountains, probably a boundary wall<br />Source: Bryan Conlon via Wikimedia Commons CC-BY-SA</td></tr></tbody></table>In addition with OSM, one is always working on a live database (even when it's a cached off-line copy, as with JOSM), so it is not possible to manipulate the data in ways which would facilitate testing changes (locking data, rollbacks etc).<br /><br />The objective of the person making the edits should therefore to ensure that at the end of the process, what existed before is in the state it was before. It is important to accept &amp; understand that during the process of making edits on this scale some things will break. This can be embarrassing: I may well be responsible for flooding Ireland in the summer. But it is best to accept that it is probably inevitable, and ensure one uses as many of the available tools to check (in this case OSM Inspector, Boundary reports from OSM-IE Nominatim, JOSM validator etc).<br /><br />To start with I grabbed a given administrative county, selected all the members of the boundary relation and then created a new relation added those members to it, added some tags and then checked that it was closed. (all in JOSM relation editor). I usually did this by downloading the relation and its members directly.<br /><br />This is actually a matter of selecting the elements of the boundary  and adding them to the new VC boundary. I used JOSM and did the  following steps:<br /><ol><li>Select the historical county relation (usually tagged with <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">admin_level=6</span>).</li><li>Select all the members of the relation (ensure that they are all downloaded)</li><li>Create a new relation, tagged <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">type=boundary</span>, <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">boundary=vice_county</span>.</li><li>Open the relation editor and add all the previously selected members</li><li>Sort them, &amp; check that the boundary is closed.</li><li>Save.</li></ol>In  fact I did exactly the same for the more complex cases too: it's easier  to remove ways than try &amp; select hundreds of them accurately. This is particularly important for coastal counties: there are lots and lots of islands which it would otherwise be all too easy to miss.<br /><br />For  the boundaries which followed the river rather than the historical  administrative boundary, revision was fairly simple. Firstly I ensured that  existing boundary ways joined the relevant river and were cut at that  point; then I added the river ways to the relation; and finally removed  the ways not wanted. Once again checking that the boundary was closed.<br /><br />For  the split counties (see above), the process was similar, but first I  had to create a way (or ways defining how the county was split).  Initially I used very simple geometries which I could refine later. Then  I added this way to one of the already created VCs for the county  (e.g., East Donegal for Donegal) and removed the two ways immediately  adjoining this belonging to the other vice county. Ideally this would  leave a closed boundary for the VC and a long chain of ways which wasnt  needed. These could then be selected and removed from the relation. For  the other VCs I could then use JOSM's select option to find all ways of  the county and then using remove from selection take away those already  in the other VC.<br /><br />Thus a reasonable approximation of  the VCs could be produced fairly quickly, thereafter it was all about  refining the boundaries.<br /><br /><h3>Making the Boundaries Available</h3>Obviously the point of mapping Vice County boundaries is so that people can use them. The typical uses which I can foresee are:<br /><ul><li><b>Presentation </b>&amp; informational material: slides etc. Largely to explain the Vice County system, or to illustrate the geography of an individual vice county (e.g., the maps included in many <a href="http://www.bsbi.org.uk/rare_plants.html">Rare Plant Registers</a>).</li><li>Illustrating <b>Wildlife Distributions</b>. Distributions of plants, animals or other wildlife features using the vice county boundaries to assist in interpreting the information. This is a common method of providing distribution information in books and websites: for instance Paul Whelans' Lichens of Ireland both as <a href="http://www.lichens.ie/lichens-of-ireland-book/">a book</a> and <a href="http://www.lichens.ie/">a website</a> uses this idea.</li><li><b>Use in GIS</b>. This may be informational (showing the VC boundary along with other information) or analytical (selecting data based on inclusion within the VC boundary).</li><li><b>In the field</b> with a GPS. To provide a tool to keep an eye on which VC particular records belong to when recording along boundaries.&nbsp;</li></ul>I've listed these in rough order of how demanding the applications are for accurate delineation of VC boundaries. Maps of the island of Ireland will necessarily not require the same degree of accuracy as ensuring records of riparian vegetation are accurately reported in the correct VC.<br /><br />The important point is that, as they stand, the boundaries may need more work for the latter two types of applications, but are perfectly suitable for the first two. My first use of the data in the informational sense was to add the map as both <a href="https://en.wikipedia.org/wiki/Vice-county#/media/File:Irish_vice_counties_ex_osm_multicolour.svg">a raster file and SVG to Wikipedia</a>, so that now the article on Vice Counties has maps for both Great Britain and Ireland.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/19669737993/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="H40 County Londonderry (test)"><img alt="H40 County Londonderry (test)" height="500" src="https://farm1.staticflickr.com/405/19669737993_28876cdccf.jpg" width="500" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A simple raster map of H40 Londonderry suitable for displaying and assisting the entry of biological records.<br />See previous blog article for principles behind this.<br /><span style="font-size: xx-small;">Contains OpenStreetMap Data (c) OpenStreetMap contributors 2015. Image CC-BY-SA the author.</span></td><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script> I've also produced <a href="https://flic.kr/p/vY9xPF">a simple raster map</a> of H40 Londonderry suitable for use as a background layer in MapMate, a widely used biological recording package. I plan to produce these for all counties eventually. Certain features, notably woodland, are currently not well mapped on OSM. I contemplate using <a href="http://wiki.openstreetmap.org/wiki/Republic_of_Ireland_Corine_2006_Import">Corine</a> data to improve coverage. Irish Vice Counties may also benefit from <a href="https://en.wikipedia.org/wiki/Hypsometric_tints">hypsometric tinting</a> as upland areas are often quite discrete and distinct.<br /><br /><h4>Github </h4>What is problematical is that extracting OpenStreetMap data requires a lot of knowledge and a degree of technical skill.<br /><br />Most users of this data will be ecologists, albeit probably familiar with GIS tools, and amateur naturalists, mostly unfamiliar with GIS.<br /><br />A second consideration is that as townland mapping advances the boundaries will improve. Taken together this suggested that I needed to publish the data in more easily consumable forms (e.g., Geojson, KML and Shapefiles), and furthermore that these need to be identified with some release mechanism.<br /><br />Github seems perfect for this job. I can refresh the extracted data locally and sync it once I've checked that its OK. I can eventually make releases with release notes so that people can work with data suitable for their needs and also decide if they want a more up-to-date version. Github's geodiff facility means that changes between versions can be visualised too: very useful for the user wanting to see if newer data addresses particular issues. I can also use it to release additional formats (e.g., Garmin IMG files) in the future. Lastly of course it provides a simple link I can point people to.<br /><br /><b>Here it is: <a href="https://github.com/SK53/Irish-Vice-Counties">https://github.com/SK53/Irish-Vice-Counties</a></b><br /><br />I've still got work to do in organising the folder structure, and a lot more work to do getting the basic update automated. Ultimately there should be a folder for each Vice County containing Shapefiles, KML, IMG, and Geojson.<br /><h4>OpenSteetMap Ireland</h4>Thanks to <a href="https://twitter.com/IrlJidel">Donal Diamond</a>, the Vice County boundaries have been made available on the main <a href="http://maps.openstreetmap.ie/?zoom=7&amp;lat=53.5&amp;lon=-6&amp;layers=B00FFFFFTFFFF">openstreetmap.ie</a> site for several months. They can be viewed together with other mapped boundaries, or against a background of historical mapping. This is the best place to start if you want to check whether the current mapping is suitable for your needs.<br /><br /><h4>Accuracy &amp; Quality</h4>I had planned to discuss how accurate borders are across the country and what factors militate against accuracy in various places, but this is quite a complex subject. For now I'll merely state the main points:<br /><ul><li>Boundaries which share that of a <b>townland </b>are likely to be accurate to around 10 m.</li><li><b>Topological accuracy</b> in such cases will generally be good (i.e., if a boundary follows a wall, stream, road etc., this will be shown correctly, even if the position of the relevant feature might not be precisely located geographically).</li><li>Ways tagged with <b>source=GSGS 3906</b> which are not yet incorporated with townland boundaries, will be slightly less accurate. This is because the boundaries shown on the old maps will not have been fully reconciled to features visible in aerial photos.</li><li>Ways tagged with <b>source=GSGS 4136</b> will be much less accurate. Perhaps up to 100 m divergence from their true position.</li><li>Ways tagged with <b>source=<a href="http://www.libraryireland.com/Atlas/Contents.php" target="_blank">http://www.libraryireland.com/Atlas/Contents.php</a></b>, or will be even less accurate. Most, if not all, of these have been eliminated as part of creating Vice County boundaries. These were drawn between localities known to belong to different counties, and in less densely populated areas were diverged as much as 1-2 km from their true position.</li></ul>I plan to produce a map summarising the likely accuracy of boundaries, but this will need to wait for another post.<br /><br />Certain features are either harder to map accurately, or represent particular problems:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ACladaghgarriff_-_geograph.org.uk_-_286148.jpg" style="margin-left: auto; margin-right: auto;" title="Richard Webb [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Cladaghgarriff - geograph.org.uk - 286148" src="https://upload.wikimedia.org/wikipedia/commons/b/bc/Cladaghgarriff_-_geograph.org.uk_-_286148.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Upland area on the H1 South Kerry/H3 Cork boundary: <br />Cladaghgarriff Ridge along to Knockowen from the Healy Pass.             <br />Photo: Richard Webb via Wikimedia Commons CC-BY-SA</td></tr></tbody></table><ul><li><b>Coastline </b>is probably reasonably accurate, but the nature of OSM is that the coastline does not follow high or low water. Places with big cliffs may not be accurately shown because of parallax on aerial imagery.</li><li><b>Wooded watercourses</b>. In many places in Ireland, streams and rivers cut deep narrow glens which are often heavily wooded. This makes accurate alignment of old maps a lot harder, and accurate tracing of watercourses impossible. Fortunately in nearly all cases the boundaries follow the watercourse. This problem was most noticeable along the Cork/Waterford boundary, and for mapping townland boundaries in County Waterford.</li><li><b>Moorland</b>. Boundaries which follow watersheds on hills, mainly moorland, are difficult to align precisely. We would really need some of the cairns shown on old maps to be accurately located in order to improve map rectification in these areas.</li><li><b>Changes in Watercourses</b>. In many places streams will have been straightened, or altered course naturally since 1900 when the base survey for GSGS 3906 maps was carried out. Furthermore where a watercourse has changed significantly this could affect administrative boundaries. In the ,main using 1900 base data is ideal for Vice County boundaries as this is close to the time they were defined. However, as boundaries become refined on OSM divergences between former and current administrative borders may need to be checked.</li><li><b>Alignment along Roads</b>. It is apparent that in many cases a townland (and therefore a county) boundary which follows a road, does not follow the mid-line, but rather one of the verges. As in most cases the difference is less than other sources of inaccuracy this has been ignored in many cases. In the future this type of detail may want refining.</li><li><b>Railways</b>. I have conventionally followed the southernmost track when a VC boundary follows a railway line. Like roads this is unlikely to create any real problems, until, that is, a rare alien plant is found growing on the trackbed.</li><li><b>Straight-lines</b>. Choosing the points where these begin and end is a little arbitrary, not particularly helped by lack of precision in Praeger's original description</li><li><b>Cork City</b>. The location of the boundary between H4 &amp; H5 in Cork City is particularly vexed. I have not yet updated this to follow the suggestions of&nbsp; Mary Scannell (1992): <i>A Re-Definition of the H4/H5 Vice-County Boundary in Urban Cork</i>&nbsp;&nbsp; <b>The Irish Naturalists' Journal</b>, 24(1), 11-13.</li></ul>In all these latter cases additional information fed back to OpenStreetMap should gradually resolve these issues.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ACharleville_station_(geograph_4166551).jpg" style="margin-left: auto; margin-right: auto;" title="Robert Ashby [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Charleville station (geograph 4166551)" src="https://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/Charleville_station_%28geograph_4166551%29.jpg/512px-Charleville_station_%28geograph_4166551%29.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Charleville Station, near the tri-point of H4/H5/H10<br />I'm not exactly sure which of the lines here should be used for the H4/H5 boundary.<br />Please do not find unusual plants/animals on the platforms.<br /><span style="font-size: xx-small;">Source: Robert Ashby via Geograph/Wikimedia Commons CC-BY-SA</span></td></tr></tbody></table><h3>Conclusions</h3>This has been a very interesting exercise. It is in effect, an attempt to create a single coherent dataset within OpenStreetMap, making use of OSM data, tools, knowledge, resources etc., for primary use by a different community of users. This has led me to consider issues of data versioning, how to make data available more easily, and how to add more extensive metadata regarding quality etc (see the above section &amp; to be covered in a later post). It is by no means a completed work: some boundaries still need work; the github repository, particularly automated updating, is incomplete; raster &amp; Garmin files are not yet done for all VCs.<br /><br />Lastly, as with all OSM-based data, this project is fundamentally always unfinished. We work by <b>incremental</b>, <b>iterative </b>refinement, and the biggest driver for that is <b>interaction </b>with users who can drive <b>improvement</b>. These <b>4 "I"s </b>distinguish OSM data from many typical sources of environmental data which are created once, and tend to decay once the funding has disappeared.<br /><h3>Building on the Shoulders of Giants (Acknowledgements)</h3>Digitising the Vice County boundaries for Ireland is notionally a big task (it would be interesting to know what resources were used by Landmark for Great Britain, and NPWS for Ireland). I completed the basics comfortably in under a week (although much work remains in tidying the data). What this shows is how the collective efforts of the OpenStreetMap community (both worldwide and specifically in Ireland), massively reduce much of that complexity through the availability of data, skilled individuals, historic maps, a plethora of software tools, and sharing of knowledge. Of course OSM in turn has benefited from broader OpenGeo community in the same way (think PostGIS, gdal, ogr, QGIS etc).<br /><br />As I try to improve the Vice County boundaries, I am not just doing work for myself; I am improving the boundaries of existing administrative units, and adding many townlands myself. Similarly anyone mapping townlands on which include a county border improve Vice Counties. We expect all townlands to be mapped sometime next Summer (July 2016).<br /><br />Many thanks to <a href="https://twitter.com/murlong77" target="_blank">Maria Long</a>, Irish Officer, BSBI for providing background, useful references and encouragement. <a href="https://twitter.com/lalonde" target="_blank">Rory McCann</a> for the townlands project. <a href="https://twitter.com/DaCor_ie" target="_blank">Dave Corley</a>, KDDA and IrlJdel for resources and help with historical maps and boundary issues. <a href="https://twitter.com/tim_waters" target="_blank">Tim Waters</a> of <a href="http://topomancy.com/" target="_blank">Topomancy LLP</a> and the <a href="http://digitalcollections.nypl.org/divisions/lionel-pincus-and-princess-firyal-map-division" target="_blank">New York Public Library</a> for <a href="http://mapwarper.net/" target="_blank">MapWarper</a> without which none of this would have been possible. Lastly, I should mention <a href="https://twitter.com/skimacker" target="_blank">Dermot McNally</a>, who not only did much of the original mapping of the Irish coastline, but <strike>I believe </strike>was also the <strike>first </strike>second person to map <a href="http://www.openstreetmap.org/relation/1143928">a townland.</a><br /><br /><span style="font-size: xx-small;"><i>Revisions: 2015-10-16: a) updated info about mapping of Co. Carlow; b) corrected info. on first townland mapped; c) add info on Dermot McNally's mapping of Caulstown. </i></span><br /><br /><br /><br /><br /><br />


guid: tag:blogger.com,1999:blog-2082422783692631078.post-6599960315666249481
url: http://sk53-osm.blogspot.com/2014/10/plantations-woods-forests-or-something.html
title: Plantations : woods, forests or something else?
updated: 2015-09-16 14:12:53.469000
published: 2015-09-16 14:12:00
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.655144
updated_at: 2019-10-26 08:30:24.927402
content:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AStand_of_trees_at_New_Fen_-_geograph.org.uk_-_636879.jpg" style="margin-left: auto; margin-right: auto;" title="Alison Rawson [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Stand of trees at New Fen - geograph.org.uk - 636879" src="https://upload.wikimedia.org/wikipedia/commons/2/2b/Stand_of_trees_at_New_Fen_-_geograph.org.uk_-_636879.jpg" width="480" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Poplars at Lakenheath<br /><span style="font-size: xx-small;"><a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA 2.0</a> &nbsp; © Copyright <a href="http://www.geograph.org.uk/profile/12764" title="View profile">Alison Rawson</a> and   licensed for reuse under this <a href="http://creativecommons.org/licenses/by-sa/2.0/" rel="license">Creative Commons Licence</a>.</span></td></tr></tbody></table>One type of woodland area I have alluded to a couple of times in the past are <a href="http://en.wikipedia.org/wiki/Plantation">plantations </a>(see <a href="http://sk53-osm.blogspot.co.uk/2014/09/woodland-cartography.html">here</a> and <a href="http://sk53-osm.blogspot.co.uk/2014/07/weingartnermoor-woodland-walk-sotm-eu.html">here</a>). I've always been frustrated at not having found good illustrations, but in the past couple of weeks I've noted a few which either already have good open images available or I've been able to snap a picture myself.<br /><br />Plantations run the gamut from small areas to fully-fledged forests. In general what connects them is that the trees are planted in orderly rows, and the plantation has an expected lifetime, after which the trees will be harvested or replanted. Photographs enable some of the variety to be shown. In turn this should highlight the sorts of information we might want to capture by OpenStreetMap tags. <br /><br /><a name='more'></a><h3>Three Plantations</h3><h4>Ratby Burroughs</h4><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-UniLaIebfgM/VC61fk8-dlI/AAAAAAAAApg/ZFnZS4Qy6FA/s1600/geograph-4188081-by-SK53.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="http://4.bp.blogspot.com/-UniLaIebfgM/VC61fk8-dlI/AAAAAAAAApg/ZFnZS4Qy6FA/s1600/geograph-4188081-by-SK53.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Planted trees at Ratby Burroughs<br /><a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA 2.0</a> &nbsp; © Copyright <a href="http://www.geograph.org.uk/profile/102131" title="View profile">SK53</a> and   licensed for reuse under this <a href="http://creativecommons.org/licenses/by-sa/2.0/" rel="license">Creative Commons Licence</a>.</td></tr></tbody></table>The first plantation I came across was a relatively newly planted extension to Burroughs Wood which is a <a href="https://en.wikipedia.org/wiki/Woodland_Trust">Woodland Trust </a>reserve.and part of the <a href="https://en.wikipedia.org/wiki/National_Forest_%28England%29">National Forest</a>. These plantings are designed to add additional woodland cover quickly, and are certainly fairly common on Woodland Trust sites. There's always a bit of controversy about <a href="http://europeantrees.wordpress.com/2014/07/02/its-less-about-planting-trees-more-about-saving-trees/">how necessary</a> such plantings might be. Apparently deer populations in Leicestershire are sufficiently high as to cause real concern about natural regeneration of woodlands. However, a deer-proof fence on land adjacent to an existing wood might allow the wood to extend itself naturally.<br /><br />As can be seen the trees are perhaps 10-15 years old, and are reasonably well established. I think it is unfortunate that they were planted in straight rows. Many of the oaks had dead branches suggesting that the trees need <a href="https://en.wikipedia.org/wiki/Thinning">thinning</a>, or perhaps some will die (self-thinning) and then the obvious grid will be lost. Perhaps because the trees were somewhat stressed they were absolutely wonderful for finding plant galls.<br /><br />This type of planting is common in other situations: in country parks; on post-industrial sites as part of remediation schemes; as screens for factories, business parks, busy roads (or even on nature reserves); and on newer parkland golf courses. Newer plantings can often to spotted on aerial photos before the canopy closes over because they show up as closely packed green (or, in Autumn, yellow-orange) dots.<br /><h4>Hope Valley, Derbyshire</h4><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-nD4nx9eG5d8/VCws7Yb2m1I/AAAAAAAAApQ/xmlTNSL2ZLQ/s1600/geograph-4166983-by-SK53.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="http://1.bp.blogspot.com/-nD4nx9eG5d8/VCws7Yb2m1I/AAAAAAAAApQ/xmlTNSL2ZLQ/s1600/geograph-4166983-by-SK53.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Small plantation of Poplars at moribund waterworks.<br /><a href="http://en.wikipedia.org/wiki/Hope_Valley,_Derbyshire">Hope Valle</a>y, The Peak District<br /><a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA 2.0</a> &nbsp; © Copyright <a href="http://www.geograph.org.uk/profile/102131" title="View profile">SK53</a> and   licensed for reuse under this <a href="http://creativecommons.org/licenses/by-sa/2.0/" rel="license">Creative Commons Licence</a>.</td></tr></tbody></table><br /><br />The second I came across was rather anomalous in several ways.<br /><br />It was next to a stream, <a href="http://en.wikipedia.org/wiki/Peakshole_Water">Peakshole Water,</a> in the Peak District National Park. Although the actual altitude was quite low, this is an upland area, with high rainfall. Woodlands tend to be dominated by Ash or Sycamore in the immediate vicinity. <br /><br /><a href="http://en.wikipedia.org/wiki/Populus_nigra">Black Poplars</a>, <i>Populus nigra</i> (or their hybrids, Populus x canadensis) are certainly not a tree species one expects <a href="https://www.derby.gov.uk/apps/flora/?SpeciesID=1348">to encounter</a>: although I did see a few <a href="http://en.wikipedia.org/wiki/Populus_tremula">Aspens</a> (<i>Populus tremula</i>) which looked <a href="https://www.derby.gov.uk/apps/flora/?SpeciesID=1352">far more in place</a> a little further along towards <a href="http://en.wikipedia.org/wiki/Hope,_Derbyshire">Hope</a>. Poplars tend to quite short-lived, and these trees have the look of senescence to them. Given that the trees look very similar I would imagine they are all clones of a single stock.<br /><br />It is now possible to use&nbsp; the National Library of Scotland's <a href="http://maps.nls.uk/geo/find/#zoom=15&amp;lat=53.34261&amp;lon=-1.76554&amp;layers=00000B0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFTFFFFFFFFT">collection of out-of-copyright 6 inch maps</a> for England and Wales to work out when these trees were planted. They seem to be planted entirely within an area which is marked as a waterworks (filter bed) on the older maps, but they do <a href="http://maps.nls.uk/view/101600130#zoom=6&amp;lat=1994&amp;lon=2562&amp;layers=BT">not appear </a>on the map surveyed in 1938. Presumably they were planted post-WWII. It is difficult to surmise for what purpose. The original plantation would have consisted of less than 50 trees, and it's not a particularly congenial location. Perhaps this was an early <a href="http://en.wikipedia.org/wiki/Environmental_remediation">environmental remediation</a> strategy.<br /><br /><h4>Lakenheath Fen </h4><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-tuifbhfDVuA/VCwqCdwMHBI/AAAAAAAAApE/kclFFDCl-6U/s1600/geograph-3751580-by-Hugh-Venables.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="http://3.bp.blogspot.com/-tuifbhfDVuA/VCwqCdwMHBI/AAAAAAAAApE/kclFFDCl-6U/s1600/geograph-3751580-by-Hugh-Venables.jpg" width="520" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Poplar plantation at Lakenheath RSPB Reserve, Suffolk<br /><span style="font-size: xx-small;"><a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA 2.0</a> &nbsp; © Copyright <a href="http://www.geograph.org.uk/profile/3176" title="View profile">Hugh Venables</a> and   licensed for reuse under this <a href="http://creativecommons.org/licenses/by-sa/2.0/" rel="license">Creative Commons Licence</a>.</span></td></tr></tbody></table><br />The last group of plantations were ones I saw from the train from Ely to Thetford. These are rather well-known. Originally planted to provide wood for making match-sticks they are now part of a major nature reserve run by the RSPB at <a href="http://en.wikipedia.org/wiki/Lakenheath_Fen_RSPB_reserve">Lakenheath Fen</a>. I had no idea that the train ran so far from the village of <a href="http://en.wikipedia.org/wiki/Lakenheath">Lakenheath</a>, or that the <a href="http://en.wikipedia.org/wiki/Lakenheath_railway_station">station </a>was on the doorstep of the RSBP reserve (so scope for a later visit next time I'm in Cambridge). Once the plantations came into view they were very obviously different from other woodlands seen from the train.<br /><br />The plantations are of a hybrid Black Poplar cultivar. Poplars, like other trees in the <a href="http://en.wikipedia.org/wiki/Salicaceae">Salicaceae</a>, the Willow family, do not grow heartwood. This means that they can grow very rapidly, but equally are short-lived and less robust than trees with the additional structural rigidity gained from heartwood. In this case they were planted by the firm <a href="http://en.wikipedia.org/wiki/Bryant_and_May">Bryant &amp; May</a> with a view to making <a href="http://en.wikipedia.org/wiki/Match">matchsticks </a>(someone noted on the walk at the <a href="http://sk53-osm.blogspot.com/2014/07/weingartnermoor-woodland-walk-sotm-eu.html">Weingartnermoor </a>that I seemed obsessed with uses of particular trees)<br /><br />Typical Poplar plantations are planted with the <a href="http://en.wikipedia.org/wiki/Cultivar">cultivar </a>favoured at the time of planting; a cultivar which may no longer be available by the time the trees mature. There is widespread research aimed at producing new varieties, at least in part supported by the <a href="http://en.wikipedia.org/wiki/Food_and_Agriculture_Organization">FAO's </a><a href="http://www.fao.org/forestry/ipc/en/">International Poplar Commission</a>. Such plantations can be found all over Europe, in the northern parts of the Indian sub-continent, and no doubt elsewhere. In many parts of Europe they can be easily recognised in winter, not just because the trees are obviously in rows, but usually because one or more are hosts to <a href="http://mistletoe.org.uk/homewp/">Mistletoe </a>(<a href="http://en.wikipedia.org/wiki/Viscum_album"><i>Viscum album</i></a>).<br /><br />At Lakenheath, they are known for supporting the only regular breeding site of the <a href="http://en.wikipedia.org/wiki/Eurasian_golden_oriole">Golden Oriole </a>(<i>Oriolus oriolus</i>) in Britain. Anyone wanting to know more about the reserve and it's history should consult a recent article in <a href="http://www.britishwildlife.com/viewbackissue.asp?issueid=77"><i>British Wildlife </i>magazine</a> (although this has little to say about the poplars).<br /><h3>Elsewhere</h3><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AThe_Slate._-_geograph.org.uk_-_452589.jpg" style="margin-left: auto; margin-right: auto;" title="Steve Partridge [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="The Slate. - geograph.org.uk - 452589" src="//upload.wikimedia.org/wikipedia/commons/5/56/The_Slate._-_geograph.org.uk_-_452589.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Forestry Plantations around The Slate and Killypole, Kintyre, Argyll<br /><span style="font-size: xx-small;">Steve Partridge <a href="http://creativecommons.org/licenses/by-sa/2.0">CC-BY-SA-2.0</a>, via Wikimedia Commons</span></td></tr></tbody></table>In Britain by far and away the largest area covered by plantations are forests of conifers. These have been planted all over the country since the creation of the <a href="https://en.wikipedia.org/wiki/Forestry_Commission">Forestry Commission</a>, but are commonest on former moorlands and rough grazing in the uplands. As I've said before these form the archetypal images of forests in the UK.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AEucalipto_Galicia.JPG" style="margin-left: auto; margin-right: auto;" title="By Martin253 (Taken by Martin253) [Public domain], via Wikimedia Commons"><img alt="Eucalipto Galicia" src="//upload.wikimedia.org/wikipedia/commons/thumb/3/32/Eucalipto_Galicia.JPG/512px-Eucalipto_Galicia.JPG" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Eucalyptus plantation near <a href="http://www.openstreetmap.org/relation/341077">Viveiro</a>, Galicia<br />Source: Martin253 via Wikimedia Commons, Public Domain</td></tr></tbody></table>Forestry plantations are no doubt common elsewhere in Europe, although in many places they will be part of an integrated approach to forest management which is rather different from that seen in Britain and Ireland. However, in Spain during the Franco era there were considerable efforts to plant fast growing trees as plantations: notably of <a href="https://en.wikipedia.org/wiki/Eucalyptus#Europe">various <i>Eucalyptus </i>species</a>, with perhaps the commonest species being <a href="https://en.wikipedia.org/wiki/Eucalyptus_globulus"><i>E. globulus</i>.</a><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AOilpalm_malaysia.jpg" style="margin-left: auto; margin-right: auto;" title="By Craig (Own work) [Public domain], via Wikimedia Commons"><img alt="Oilpalm malaysia" src="//upload.wikimedia.org/wikipedia/commons/thumb/8/8d/Oilpalm_malaysia.jpg/512px-Oilpalm_malaysia.jpg" width="512" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Malaysian Palm Oil Plantation<br />Source: Wikimedia</td></tr></tbody></table>Several types of trees are notable for being planted extensively in the tropics: for example, <a href="https://en.wikipedia.org/wiki/Natural_rubber">Rubber </a>for its sap, <a href="https://en.wikipedia.org/wiki/Teak">Teak </a>for timber, and <a href="https://en.wikipedia.org/wiki/Palm_oil">Coconut Palms</a> for oil.<br /><br />The current tagging for orchards <a href="http://wiki.openstreetmap.org/wiki/Tag:landuse%3Dorchard">does allow</a> for Palm plantations to be tagged as orchards, although they don't look very like typical orchards. At least around <a href="https://en.wikipedia.org/wiki/Lumut,_Malaysia">Lumut </a>in Malaysia these appear quite characteristic in <a href="http://binged.it/1BJHBrj">aerial photos</a>. (As an aside, using <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">landuse=orchard</span> for <a href="https://en.wikipedia.org/wiki/Dehesa_%28pastoral_management%29">Dehesa </a>wood pasture seems going too far).<br /><br /><h3>Conclusions</h3>Plantations exist throughout the globe.and are a significant part of the world's woodlands. Many plantations are of economic importance, and most probably have an adverse effect on both local and global biodiversity. Furthermore they often differ in many ways from conventional ideas of woodland: forest plantations in Britain are often impenetrable for walkers, the ground layer is often absent.<br /><br />I therefore think there are a wide range of reasons why we might want to be able to distinguish them from other woodlands on OpenStreetMap. To date I have marked a very small number using <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">plantation=yes</span>, but this is very much a placeholder tag and I'm open to other suggestions. At one end of the spectrum orchards and olive groves can be regarded as  particular types of plantations. To an extent there is a continuum between traditional orchards and many large scale woodland plantation. In addition there are issues as  to appropriate tagging when trees are planted for crop products other than  timber and wood.<br /><br />I also believe that we should keep a distinction between a plantation and newly (or obviously) planted woodland (just as we will retain a distinction between an orchard and a plantation). I think the key criteria for identifying a plantation are:<br /><ul><li><b>Monoculture </b>(or nearly so). Trees are all of a single species. In some cases a mix of species may be planted around the edge of the plantation (for instance in modern forestry practice in Britain)</li><li><b>Single Age Class</b>. Trees are obviously of a similar size (and therefore age). Evidence of sapling growth will be minimal.</li><li><b>Planted in Rows</b>. As most plantations either have a commercial objective, or are planted in such a way to minimise costs, they tend to be planted in long rows. This not only assists the process of planting, but also harvesting.</li><li><b>Short to Medium term lifecycle</b>. The trees of the plantation are expected to be harvested within 20-50 years (less if for Christmas Trees). If the plantation is producing a crop (as in Oil Palm plantations), the trees will be replaced in the same time-frame. </li></ul>This definition therefore tends to exclude areas of woodland planted to enhance amenity or for environmental remediation. In these cases I have from time to time used <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">planted=yes</span> to identify such woodlands. In practice it would be nice to have a <span style="font-family: &quot;Courier New&quot;,Courier,monospace;">planted_date</span> key as well.<br /><br />If you are interested in developing on these ideas I suggest using the OSM <a href="https://lists.openstreetmap.org/listinfo/tagging">tagging</a> mailing list for further discussion. <br /><br /><br />


guid: tag:blogger.com,1999:blog-2082422783692631078.post-181866406377556306
url: http://sk53-osm.blogspot.com/2015/09/why-does-guardian-think-that.html
title: Why does The Guardian think that OpenStreetMap is owned by Google
updated: 2015-09-10 11:57:09.728000
published: 2015-09-10 11:57:00
fetched: 2019-10-26 08:30:23.899059
created_at: 2019-06-09 18:12:13.658631
updated_at: 2019-10-26 08:30:24.930626
content:
  My attention was drawn by a tweet to <a href="http://www.theguardian.com/travel/gallery/2015/sep/10/cool-cartography-the-art-of-mapmaking">an article</a> in The Guardian online showing some nice examples of cartography:<br /><br /><blockquote class="twitter-tweet" lang="en"><div dir="ltr" lang="en">Some great <a href="https://twitter.com/hashtag/map?src=hash">#map</a> and <a href="https://twitter.com/hashtag/cartography?src=hash">#cartography</a> art here <a href="http://t.co/QesV8DLeGO">http://t.co/QesV8DLeGO</a></div>— Ordnance Survey (@OrdnanceSurvey) <a href="https://twitter.com/OrdnanceSurvey/status/641915350062342144">September 10, 2015</a></blockquote><script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>To my amazement it contained this:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-fhr-ODAzK0Q/VfFmOvIhs_I/AAAAAAAAAv4/KhbHAn5kn_4/s1600/guardian_error.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="388" src="http://4.bp.blogspot.com/-fhr-ODAzK0Q/VfFmOvIhs_I/AAAAAAAAAv4/KhbHAn5kn_4/s320/guardian_error.jpg" width="520" /></a></div><br />To quote "Google's OpenStreetMap"!<br /><br />Now, The Guardian has generally covered OpenStreetMap well over the years, and has made use of appropriately credited OSM data from time-to-time over the years. Last year it <a href="http://www.theguardian.com/technology/2014/jan/14/why-the-world-needs-openstreetmap">re-published </a>Serge Wroclawski's influential blog post <a href="http://blog.emacsen.net/blog/2014/01/04/why-the-world-needs-openstreetmap/">"Why the world needs OpenStreetMap"</a>. It's former Technology editor, <a href="https://twitter.com/charlesarthur?">Charles Arthur</a>, was very familiar with OpenStreetMap, and in turn played a big part in the campaign to get open data from the Ordnance Survey.<br /><br />This error is symptomatic of two things:<br /><ul><li>A widespread assumption that anything to do with on-line maps must come from Google (most often seen in the belief that the images are taken with Google's own statellites).</li><li>An absence of care in fact-checking when taking information from other webpages.&nbsp;</li></ul>There is not a great deal that the OpenStreetMap community can do about the former. After all commercial players such as DigitalGlobe have to put up with the same sort of thing. However, we can do something about the latter: but in general probably don't put enough effort into such things. (Perhaps unsurprising, its more fun to survey &amp; add data than try and get people to get their facts right).<br /><br />A quick google search reveals more or less the identical string in numerous webpages referring to <a href="https://www.behance.net/gallery/24276859/City-Layouts">Luis Dilger's 3-D city</a> visualisations:<br /><br /><ul><li><a href="http://fullscream.com/incredible-3d-city-layouts-luis-dilger-and-remodelling-google-maps/">Full Scream</a></li><li><a href="http://designcollector.net/city-layouts-by-luis-dilger/">Design Collector</a> credited to Arseny Verishin</li><li><a href="https://trouge.com/extraordinary-city-layout-visualisation/">Trogue</a> not credited</li><li>&nbsp;<a href="http://www.fubiz.net/en/tag/luis-dilger/">Fubiz </a>not credited</li><li>and many more</li></ul>Clearly most of these are just straight copying from a single source. However it is not the original statement by Dilger (see link above). So it looks like The Guardian really has not checked it's facts and has the text is probably unoriginal too.<br /><br />Thankfully I got a speedy response from them (thanks to twitter):<br /><br /><br /><blockquote class="twitter-tweet" lang="en"><div dir="ltr" lang="en"><a href="https://twitter.com/SK53onOSM">@SK53onOSM</a> thanks for pointing that out - the caption's been amended now</div>— Gdn Readers' editor (@GdnReadersEd) <a href="https://twitter.com/GdnReadersEd/status/641938082967777280">September 10, 2015</a></blockquote><script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script> All-in-all the episode highlights that as a group we in OpenStreetMap have a long way to go in communicating who we are and what we do to mainstream media types.


guid: https://derickrethans.nl/buildings.html
url: https://derickrethans.nl/buildings.html
title: Terracing Buildings
published: 2015-08-05 08:25:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.854106
updated_at: 2019-07-20 20:30:22.473776
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="terracing_buildings"/>Terracing Buildings</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Wednesday, August 5th 2015, 09:25 BST</div>
        </div>
        <p>Following up from the <a href="https://derickrethans.nl/flats.html">previous</a> article on addressing flats, in this article we deal with drawing buildings. The new area has lots of similarly looking buildings, and they're not exactly square. They do however all (broadly) share the same shape:</p>
        <img src="derickrethans.nl/rst/images/buildings.png"/>
        <p><a href="http://josm.openstreetmap.de">JOSM</a> has a <a href="http://wiki.openstreetmap.org/wiki/JOSM/Plugins/Terracer">terracer</a> plugin, which allows you to draw a rectangle, and then quickly convert it to a row of terraced houses and at the same time add house numbers to the generated houses.</p>
        <p>You do the following steps with the plugin:</p>
        <table>
          <thead>
            <tr>
              <td>
                <p>Image                            </p>
              </td>
              <td>
                <p>Action</p>
              </td>
            </tr>
          </thead>
          <tbody>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace01.png"/>
              </td>
              <td>
                <p>Find a spot that needs buildings added</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace02.png"/>
              </td>
              <td>
                <p>Draw the outline</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace03.png"/>
              </td>
              <td>
                <p>Add the <code>building=residential</code> tag</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace04.png"/>
              </td>
              <td>
                <p>Select road and outline, and press <code>Shift-T</code> to run the terracer plugin.</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace05.png"/>
              </td>
              <td>
                <p>Select every other middle node at the back</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace06.png"/>
              </td>
              <td>
                <p>Use JOSM's Way Accuracy Tool <code>w</code> to add in the other nodes for the back</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace07.png"/>
              </td>
              <td>
                <p>Repeat for the other 5 buildings</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace08.png"/>
              </td>
              <td>
                <p>Select all buildings and use the Orthogonalise Shape tool <code>Q</code> to "square" the buildings.</p>
              </td>
            </tr>
          </tbody>
        </table>
        <p>Having to do step 6 and 7 for every building, especially when there are a lot of them, is a very tedious, and inaccurate process. It is sometimes even faster to draw all the building outlines by hand, select them all, and orthogonalise them.</p>
        <p>But the terracer does already save a lot of work, so what if I could make it better and make it support the type of buildings that are common in my new neighbourhood?</p>
        <p>I had a look into the plugin, and it seems that I have to use Java to update it. I don't really use Java, and hardly ever written any code in it. It also sadly lives on some SVN repository, so pull requests against it are difficult too. Instead, I pushed my version of the repository to <a href="https://github.com/derickr/josm-plugin-uberterrace">https://github.com/derickr/josm-plugin-uberterrace</a> — and called my version Überterrace. Oh, and I learned some Java.</p>
        <p>After a while of figuring out some ... syntax issues, and figuring out how to build this stuff, I managed to improve the plugin with another option. The new option is "fancy outline" and automatically draws in the "extensions at the back". The new work flow is now:</p>
        <table>
          <thead>
            <tr>
              <td>
                <p>Image                            </p>
              </td>
              <td>
                <p>Action</p>
              </td>
            </tr>
          </thead>
          <tbody>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace01.png"/>
              </td>
              <td>
                <p>Find a spot that needs buildings added</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace10.png"/>
              </td>
              <td>
                <p>Draw the outline, including the backends</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace11.png"/>
              </td>
              <td>
                <p>Add the <code>building=residential</code> tag</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace12.png"/>
              </td>
              <td>
                <p>Select road and outline, and press <code>Shift-T</code> to run the terracer plugin with the "fancy outline" checkbox set.</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace13.png"/>
              </td>
              <td>
                <p>Drag the selected middle nodes to the right position.</p>
              </td>
            </tr>
            <tr>
              <td>
                <img src="derickrethans.nl/rst/images/terrace14.png"/>
              </td>
              <td>
                <p>Press <code>Shift-E</code> to select all buildings, and press <code>q</code> to orthogonalise them.</p>
              </td>
            </tr>
          </tbody>
        </table>
        <p>As you can see, this makes it much faster to draw such building outlines. When you look at it in 3D however, you can see there is still a problem though:</p>
        <img src="derickrethans.nl/rst/images/terrace-3d-now.png"/>
        <p>The 3D image that is being drawn does not take into account that the "extension" at the back, isn't part of the gabled roof nor that it is only two instead of three levels. There are various possibilities of building layouts:</p>
        <img src="derickrethans.nl/rst/images/terrace-3d-alternatives.png"/>
        <p>So the next versions of Überterrace are going to support (some of) these as well!</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/flats.html
url: https://derickrethans.nl/flats.html
title: Adressing Flats
published: 2015-07-21 08:11:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.856921
updated_at: 2019-07-20 20:30:22.476889
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="adressing_flats"/>Adressing Flats</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Tuesday, July 21st 2015, 09:11 BST</div>
        </div>
        <p>I recently moved into a new area in London, and that means, a new area to map. That means drawing in all the buildings, but also adding shops and addresses. Unlike my old area, the new neighbourhood has lots of buildings with flats. In <a href="http://openstreetmap.org">OpenStreetMap</a>, you can tag addresses on flats with the <a href="http://wiki.openstreetmap.org/wiki/Key:addr:flats">addr:flats</a> tag. In most cases, the addresses of the flats are formed by a flat number, building name (<a href="http://wiki.openstreetmap.org/wiki/Key:addr:housename">addr:housename</a>) and street name (<a href="http://wiki.openstreetmap.org/wiki/Key:addr:street">addr:street</a>). Only occasionally the address consists of flat number, house number, and street name.</p>
        <p>When I started tagging a nearby collection of flat blocks, and the flat number/house name/street name on their entrance nodes, I noticed that it rendered as:</p>
        <img src="derickrethans.nl/rst/images/flatnr-before.png"/>
        <p>As you can see, it ignored the <a href="http://wiki.openstreetmap.org/wiki/Key:addr:flats">addr:flats</a> tag, and instead just rendered the <a href="http://wiki.openstreetmap.org/wiki/Key:addr:housename">addr:housename</a> tag over and over again. Not particularly useful.</p>
        <p>So I thought, why not "fix" the rendering. In the past, I already added a basic rendering for <a href="http://wiki.openstreetmap.org/wiki/Tag:natural%3Dtree_row">natural=tree_row</a> to the "default" rendering style. So I had another look on how to make <a href="http://wiki.openstreetmap.org/wiki/Key:addr:flats">addr:flats</a> render and show up instead of the house name 20 times.</p>
        <p>OpenStreetMap's style sheet is developed through the Git repository at <code>git@github.com:gravitystorm/openstreetmap-carto.git</code>. I cloned the repository and searched after the places where I needed to make modifications to add the flat numbers. The style sheets first require a "style" file to tell which data to import from the main database (or <code>.xml</code> file). Only fields that are copied into the rendering database are available for use in the style sheets. OpenStreetMap Carto stores these definitions in the <code>project.mml</code> file. But that's a nasty JSON file, so instead, you add the definition into the <code>project.yaml</code> file. There is a script to convert the YAML file into the <code>project.mml</code> file which the <code>carto</code> tool can use to generate a <a href="http://mapnik.org/">Mapnik</a> stylesheet. Mapnik is then responsible for rendering the map.</p>
        <p>In <code>project.yaml</code>, just above the <code>- id: "housenumbers"</code> line, I added the following section:</p>
        <pre>- id: "flatnumbers"
    name: "flatnumbers"
    class: ""
    geometry: "point"
    <<: *extents
    Datasource:
      <<: *osm2pgsql
      table: |-
        (SELECT
            way,
            "addr:flats",
            way_area/NULLIF(!pixel_width!::real*!pixel_height!::real,0) AS way_pixels
          FROM planet_osm_polygon
          WHERE "addr:flats" IS NOT NULL
            AND building IS NOT NULL
        UNION ALL
        SELECT
            way,
            "addr:flats",
            NULL AS way_pixels
          FROM planet_osm_point
          WHERE "addr:flats" IS NOT NULL
          ORDER BY way_pixels DESC NULLS LAST
        ) AS flatnumbers
    properties:
      minzoom: 17
    advanced: {}

  </pre>
        <p>It basically instructs the stylesheet to run an extra query to obtain information on <code>addr:flats</code> and make that information available to the actual styles.</p>
        <p>It requires an extra column in the database (<code>addr:flats</code>), the import "style" needs to include this data while importing from the data source into the rendering database. The file containing the import definitions is <code>openstreetmap-carto.style</code>. I added the following line in between the definitions for <code>access</code> and <code>addr:housenumber</code>:</p>
        <pre>node,way   addr:flats   text         linear

  </pre>
        <p>Adding an extra column into the "style" means that it is necessary to reimport the data from the main database (or XML file) into the rendering database. My original source is the <code>openstreetmap.osm</code> file, which I re-imported with:</p>
        <pre>/home/derick/install/osm2pgsql/osm2pgsql \
          -S openstreetmap-carto.style --slim -d gis -C 2400 openstreetmap.osm

  </pre>
        <p>To render the flat numbers, and attach a style to them, I had to add a section to the <code>addressing.mss</code> file which is a sub-stylesheet that deals with address information. This stylesheet is linked from the project file. I added the following lines in between the <code>#interpolation</code> and <code>#housenumbers</code> sections:</p>
        <pre>#flatnumbers {
    [zoom >= 17] {
          text-name: "[addr:flats]";
          text-placement: interior;
          text-min-distance: 1;
          text-wrap-width: 0;
          text-face-name: @book-fonts;
          text-fill: #666;
          text-size: 8;
    }
  }

  </pre>
        <p>I picked the same style as for standard house numbers, but one size smaller.</p>
        <p>With all the sections added to the right files I had to regenerate the <code>project.mml</code> file from <code>project.yaml</code> with:</p>
        <pre>./scripts/yaml2mml.py < project.yaml > project.mml

  </pre>
        <p>And generate a <a href="http://mapnik.org/">Mapnik</a> stylesheet from the <code>project.mml</code> file with:</p>
        <pre>carto project.mml > mapnik.osm

  </pre>
        <p>With all that in place, I re-rendered the area. The flat numbers are now visible on the map:</p>
        <img src="derickrethans.nl/rst/images/flatnr-after.png"/>
        <p>I have packaged up all the changes, and prepared a <a href="http://github.com/gravitystorm/openstreetmap-carto/pull/1665">pull request</a> for <code>openstreetmap-carto</code>. With some luck, the changes will make it to the OpenStreetMap website at some point in the future.</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/mapping-lubumbashi.html
url: https://derickrethans.nl/mapping-lubumbashi.html
title: Mapping Lubumbashi
published: 2014-05-06 18:26:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.859266
updated_at: 2019-07-20 20:30:22.479735
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="mapping_lubumbashi"/>Mapping Lubumbashi</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Tuesday, May 6th 2014, 19:26 BST</div>
        </div>
        <img class="right" align="right" src="derickrethans.nl/images/content/hot-logo.png"/>
        <p>I spent last Saturday helping out HOT, the <a href="http://wiki.openstreetmap.org/wiki/Humanitarian_OpenStreetMap_Team">Humanitarian OpenStreetMap Team</a>, to improve the map of Lubumbashi, Congo. This session focussed on digitizing the data that was acquired on the ground by volunteers (with pen and paper).</p>
        <p>But this was not where the effort started. It started a few weeks ago when <a href="https://twitter.com/Zjwarie">Jorieke</a> set off to Lubumbashi to organise the mapping party. You can read more about this on the <a href="http://hot.openstreetmap.org/updates/2014-04-01_a_week_in_lubumbashi_drc">HOT blog</a>.</p>
        <p>In cooperation with <a href="http://www.msf.org.uk/">Doctors without Borders</a>/Médecins Sans Frontières UK (MSF UK), HOT organised this mapping effort, with the help of students from the University of Lubumbashi. The goal was to map the whole or at least as much as possible of the city of Lubumbashi.</p>
        <p>At the start of the <a href="https://wiki.openstreetmap.org/wiki/2014_London_HOT_Congo_Mapathon">Mapathon</a>, <a href="https://twitter.com/ivangayton">Harry Wood</a> went through his usual spiel of introducing OpenStreetMap to the crowd. I dropped in the moment he was done, just in time to hear <a href="https://twitter.com/ivangayton">Ivan Gayton</a>'s talk on the work of MSF and their Congo mapping work. Mapping in Congo is not nearly as simple as it is here in London, where streets have street names that are neatly printed on street signs. This is not the case in Congo. There no street signs and often, people use names that are either not official, or have no official name anyway. The only way to find out how streets are called, is by asking the people living there!</p>
        <p>MSF is interested in street name information, or more precisely, the neighbourhood names so that they have an identifier for statistical analysis for epidemiology. The accurate location names allows them to pinpoint for example cholera outbreaks, and hopefully in the future they can take preventative action against disease outbreaks.</p>
        <p>The first step was creating a basic road layout, which was handled by a <a href="http://wiki.openstreetmap.org/wiki/MSF_Congo_Mapping_Party,_Berlin">Mapathon in Berlin</a>. Coordination of such large mapping efforts happen through the <a href="http://tasks.hotosm.org/">OSM Tasking Manager</a>. You can see the job for Lubumbashi at <a href="http://tasks.hotosm.org/job/456">http://tasks.hotosm.org/job/456</a> They did not stop at Lubumbashi though, and traced a whole of of other towns as well. And the mapping was not only remotely done from Berlin, but OpenStreetMap communities from <a href="https://twitter.com/OSM_HU/statuses/447367461310971905">Hungary</a>, <a href="https://twitter.com/ndongamadou/statuses/447435101035847680">Togo</a>, <a href="https://twitter.com/FoFB7/statuses/447794067540176896">Burkina</a> helped as well—a true global community.</p>
        <img class="left" align="left" src="derickrethans.nl/images/content/hot-lubumbashi.jpg"/>
        <p>Besides this initial road tracing job, there are a few other things still to trace too. If you have time, give them a hand for <a href="http://tasks.hotosm.org/job/459">http://tasks.hotosm.org/job/459</a></p>
        <p>With a basic road layout traced from areal images you have a rough idea on how a city looks like, but nothing beats a survey on the ground to see how things actually are. It is not always easy to spot things on areal imagery.</p>
        <p>While remote mapping parties were nearing their completion of the road network, a group of volunteers from Department of Geography of the University of Lubumbashi went out into the neighbourhoods with <a href="http://fieldpapers.org/">FieldPapers</a> to do the survey. Through hints from Ivan I understood that it took MSF some time to convince their professor to help out for free. He could hardly believe that some organisation was collecting all this data and then making it available for free. Not just for "no money", but also under an open license. Perhaps he was worried about what Google <a href="http://wiki.openstreetmap.org/wiki/Google_Map_Maker">MapMaker</a> is trying to do in underdeveloped countries—getting volunteers to collect data for them and then locking it up in GoogleMaps. Having open access to the data is of course much more valuable.</p>
        <p>FieldPapers allows you to create <a href="http://fieldpapers.org/atlas.php?id=vsm3w2nd">atlases</a> of areas. It divides this up into bite size chunks so that you can print them out and mark your survey information on the paper. For the Lubumbashi mapthon, we had an immense stack of those FieldPapers with notes on them. And our task was to go through this stack and digitize the notes on the map through an OpenStreetMap editor, such as JOSM. I think I managed to go through about 100 to 150 pages of them during the mapathon. One of the <a href="http://www.openstreetmap.org/changeset/22108730">areas</a> that I did was about 30 sheets of paper!</p>
        <p>In the end, we made it through all of the FieldPapers. Task completed! But that does not mean we're finished. This effort was only a small section of Lubumbashi and the local volunteers are going to continue their effort. But they are also learning how to edit the data themselves. And that is the only sustainable way, but I will try to help out whenever I can!</p>
        <div class="credits">
          <a href="https://www.flickr.com/photos/harrywood/13909444627/">
            <p> Derick going through FieldPapers for Lubumbashi — CC by-sa Harry Wood</p>
          </a>
        </div>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/postbox-hunting.html
url: https://derickrethans.nl/postbox-hunting.html
title: Hunting for Postboxes (part 1)
published: 2014-01-28 09:11:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.861567
updated_at: 2019-07-20 20:30:22.482490
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="hunting_for_postboxes_part_1"/>Hunting for Postboxes (part 1)</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Tuesday, January 28th 2014, 09:11 GMT</div>
        </div>
        <img class="left" align="left" src="derickrethans.nl/images/content/postbox-pebble.png"/>
        <p>The new year brings new hobby projects! In this year, I am going to try to photograph as many UK postboxes as I can. For this I am going to build some apps and overview maps, but I will write more about those later. A teaser for a future post is here to the left ;-).</p>
        <p>The first thing that I want my application(s) to do is to give a description of where a postbox is located and this post is the tale of how I went about just doing that.</p>
        <p>As my data source, I am using an <a href="http://openstreetmap.org">OpenStreetMap</a> extract for the <a href="http://metro.teczno.com/#london">London region</a>. I import that through my "Import OSM data into MongoDB" script that I wrote about in an <a href="https://derickrethans.nl/importing-osm-into-mongodb.html">earlier article</a>. The script is available on GitHub as part of the 3angle repository. The latest version is at <a href="https://raw.github.com/derickr/3angle/master/import-data.php">https://raw.github.com/derickr/3angle/master/import-data.php</a> and it also requires <a href="https://raw.github.com/derickr/3angle/master/classes.php">https://raw.github.com/derickr/3angle/master/classes.php</a> for some GeoJSON helper classes and <a href="https://raw.github.com/derickr/3angle/master/config.php">https://raw.github.com/derickr/3angle/master/config.php</a> where you can set the database name and collection name (in my case, <code>demo</code> and <code>poiConcat</code>).</p>
        <p>My goal is to get from a longitude/latitude pair to the closest postbox's reference number (<code>EC2 201</code>), a description like (<code>On Paul Street, on the
  corner with Epworth Street</code>), and a direction and distance (<code>East, 86m</code>).</p>
        <p>The first thing I do is to look-up the closest postbox. I can very easily do that with the following aggrgation query in MongoDB:</p>
        <pre><?php
  $m = new MongoClient( 'mongodb://localhost' );
  $d = $m->selectDb( 'demo' );
  $c = $d->selectCollection( 'poiConcat' );

  $center = new GeoJSONPoint( (float) $_GET['lon'], (float) $_GET['lat'] );

  $res = $c->aggregate( array(
          '$geoNear' => array(
                  'near' => $center->getGeoJson(),
                  'distanceField' => 'distance',
                  'distanceMultiplier' => 1,
                  'maxDistance' => 5000,
                  'spherical' => true,
                  'query' => array( TAGS => 'amenity=post_box' ),
                  'limit' => 1,
          )
  ) );

  </pre>
        <p>I am using the aggregation framework instead of a normal query because the aggregation framework also can return the distance to the found object. The above returns <code>$res</code>, and our first result is located in <code>$res['result'][0]</code>, which looks like:</p>
        <pre>array(
          '_id' => 'n905143645',
          'l' => array(
                  'type' => 'Point',
                  'coordinates' => array( -0.1960, 51.5376 ),
          ),
          'm' => array(
                  'v' => 5,
                  'cs' => 14576087,
                  'uid' => 37137,
                  'ts' => 1357659884,
          ),
          'ts' => array(
                  0 => 'amenity=post_box',
                  1 => 'operator=Royal Mail',
                  2 => 'ref=NW6 14',
          ),
          'ty' => 1,
          'distance' => 162.15002059040299,
  ),

  </pre>
        <p>So postbox <code>NW6 14</code> is <code>162.15</code> meters away from <code>-0.1979, 51.5385</code> at <code>-0.1960, 51.5376</code>, as you can see in this image:</p>
        <img src="derickrethans.nl/images/content/postbox1.png"/>
        <p>To find a description of where the postbox is, we first find the closest road:</p>
        <pre>$r = $res['result'][0];
  $query = [
          LOC => [
                  '$near' => $r['l']
          ],
          TAGS => new     MongoRegex(
                  '/^highway=(trunk|pedestrian|service|primary|secondary|tertiary|residential|unclassified)/'
          )
  ];
  $road = $c->findOne( $query );

  </pre>
        <p><code>highway</code> is an <a href="http://openstreetmap.org">OpenStreetMap</a> tag that is also used for footpaths, service roads and alleys. These generally don't have names, so with the regular expression we restrict the query to only return "normal" roads. After executing this query, <code>$road</code> now contains:</p>
        <pre>array (
          '_id' => 'w4442243',
          'ty' => 2,
          'l' => array (
                  'type' => 'LineString',
                  'coordinates' => array (
                          array (
                                  -0.2046823,
                                  51.5346008,
                          ),
                          …
                          array (
                                  -0.1940129,
                                  51.5384693,
                          ),
                  ),
          ),
          'ts' => array (
                  'hgv=destination',
                  'highway=secondary',
                  'lit=yes',
                  'name=Brondesbury Road',
                  'note=Signed as maxweight 7.5T for goods vehicles except for access, so have tagged as hgv=destination',
                  'ref=B451',
                  'sidewalk=both',
                  'source:ref=OS OpenData StreetView',
          ),
          'm' => array (
                  'v' => 15,
                  'cs' => 18802367,
                  'uid' => 37137,
                  'ts' => 1384017096,
          ),
  )

  </pre>
        <p>As an image this looks like:</p>
        <img src="derickrethans.nl/images/content/postbox2.png"/>
        <p>We are interested only in the name (<code>name=Brondesbury Road</code>) and the geometry (<code>l</code>). Right now, we can already assemble the description <code>NW6
  14, on Brondesbury Road</code>. But we also want to know the closest cross road, which we can find by finding all roads that intersect with our geometry (in <code>l</code>) by running the following query:</p>
        <pre>$q = $c->find( [
          'l' => [
                  '$geoIntersects' => [ '$geometry' => $road['l'] ]
          ],
          'ts' => new MongoRegex(
                  '/^highway=(trunk|pedestrian|service|primary|secondary|tertiary|residential|unclassified)/'
          ),
          '_id' => [ '$ne' => $road['_id'] ],
  ] );

  </pre>
        <p>This returns nineteen roads. An extract looks like:</p>
        <pre>array(19) {
          'w4211713' => array(5) {
                  '_id' => string(8) "w4211713"
                  'ty' => int(2)
                  'l' => array(2) {
                          'type' => string(10) "LineString"
                          'coordinates' => array(22) { ... }
                  }
                  'ts' =>
                  array(4) {
                          [0] => string(19) "highway=residential"
                          [1] => string(15) "maxspeed=20 mph"
                          [2] => string(23) "name=Brondesbury Villas"
                          [3] => string(13) "sidewalk=both"
                  }
                  …
          }
          'w245650577' =>
          array(5) {
                  '_id' => string(10) "w245650577"
                  'ty' => int(2)
                  'l' => array(2) {
                          'type' => string(10) "LineString"
                          'coordinates' => array(5) { ... }
                  }
                  'ts' =>
                  array(6) {
                          [0] => string(15) "highway=primary"
                          [1] => string(7) "lit=yes"
                          [2] => string(15) "maxspeed=30 mph"
                          [3] => string(22) "name=Kilburn High Road"
                          [4] => string(6) "ref=A5"
                          [5] => string(13) "sidewalk=both"
                  }
                  …
          }
  }

  </pre>
        <p>As an image this looks like:</p>
        <img src="derickrethans.nl/images/content/postbox3.png"/>
        <p>We are only interested in the roads that have a name and have a <strong>different</strong> name than the road we have run the intersection query for. In some cases, <a href="http://openstreetmap.org">OpenStreetMap</a> splits up a road in more than one segment carrying the same name. We discard both those in a loop and are then left with an array of intersecting road IDs in the <code>$intersectingWays</code> variable:</p>
        <pre>$intersectingWays = array();
  foreach ( $q as $crossRoad )
  {
          $crossTags = Functions::split_tags( $crossRoad[TAGS] );
          if ( !in_array( "name={$roadName}", $crossRoad ) && array_key_exists( 'name', $crossTags ) )
          {
                  $intersectingWays[] = $crossRoad['_id'];
          }
  }

  </pre>
        <p>With these IDs, we then search for the closest road(s) to the initially found postbox location:</p>
        <pre>$res = $c->aggregate( array(
          '$geoNear' => array(
                  'near' => $r['l'],
                  'distanceField' => 'distance',
                  'distanceMultiplier' => 1,
                  'maxDistance' => 5000,
                  'spherical' => true,
                  'query' => [
                          '_id' => [ '$in' => $intersectingWays ],
                          'ts' => [ '$ne' => "name={$roadName}" ]
                  ],
                  'limit' => 1,
          )
  ) );

  </pre>
        <p>Again, the result in <code>$res</code> is in a similar format as before, so I won't repeat that. We use the aggrgation framework again so that we also get the distance of this intersecting road to the originally found postbox location. Depending on the distance to the intersecting road, we either use <code>on the
  corner of <roadname></code> (less thatn 25m) or <code>near <roadname></code> if it's further away than 25m. For our example postbox, that makes <code>NW6 14, on Brondesbury
  Road, near Algernon Road</code> which is illustrated by this image:</p>
        <img src="derickrethans.nl/images/content/postbox4.png"/>
        <p>The full code for this example can be found at <a href="https://github.com/derickr/3angle/tree/master/maps-postbox">https://github.com/derickr/3angle/tree/master/maps-postbox</a> and you see it in action (for London) at: <a href="http://maps.derickrethans.nl/?l=postbox,lat=51.5&lon=-0.128&zoom=17">http://maps.derickrethans.nl/?l=postbox,lat=51.5&lon=-0.128&zoom=17</a></p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/splines.html
url: https://derickrethans.nl/splines.html
title: Smoothing lines with splines
published: 2014-01-21 08:44:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.863924
updated_at: 2019-07-20 20:30:22.485155
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="smoothing_lines_with_splines"/>Smoothing lines with splines</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> La Clusaz, France</div>
          <div class="date">Tuesday, January 21st 2014, 09:44 CET</div>
        </div>
        <p>For my <a href="https://derickrethans.nl/year-of-edits-2013.html">OpenStreetMap year of edits</a> videos I use <a href="http://povray.org">PovRay</a> to animate edits of <a href="http://openstreetmap.org">OpenStreetMap</a>. Over the course of a year I show all the edits that have been made. In previous years, I used a simple sine function to rotate the Earth alone it's North/South and East/West axises. This year I wanted to highlight specific events so I needed a different method to rotate the Earth.</p>
        <p>I started out with finding events and created two paths for them:</p>
        <img src="derickrethans.nl/images/content/path2.png"/>
        <p>This one starts in Greenland, moves towards Brazil and then heads over Russia, India, the Philippines and ends in Indonesia.</p>
        <img src="derickrethans.nl/images/content/path1.png"/>
        <p>The second one starts in Korea, moves through Africa, heads towards Antarctica and then heads back to France over Brazil.</p>
        <p>Each of the two paths have points highlighted which are (about) 15 days apart. The paths I converted into data files, which look like</p>
        <pre>0000  -24   70   8500
  0180  -64   58   9500
  0240  -81   29  10000
  0480  -43  -16  15000
  0720   -8   36  20000
  0840   27   44  15000
  1020   76   20  15000
  1140  115   25  15000
  1320  127   -2   9000
  1460  157  -24  10000

  </pre>
        <p>The first column is the frame number (four frames per day), then the longitude and the latitude and finally the height of the camera over the point. As you can see, I have not included a line for each of the points in the path images.</p>
        <p>From the points in the data files, I then needed to generate all the intermediate points for each of the three columns. And my first (naive) attempt was to simply do a linear interpolation. Point <em>1</em>'s longitude can be calculated by: <code>-24 + (1-0) * ((-64 - -24) / (180 - 0))</code> and point <em>856</em>'s longitude with: <code>27 + (855 - 840) * ((76 - 115) / (1020 - 840))</code>. Doing that for all of the three axis (for the first 300 frames) results in the following video:</p>
        <div class="video">
          <div class="videoWrapper">
            <iframe src="https://player.vimeo.com/video/83049723?title=0&byline=0&portrait=0" frameborder="0"/>
          </div>
          <p><a href="https://vimeo.com/83049723"/>Linear interpolation of a line between a set of points</p>
        </div>
        <p>As you can see, the changes in direction (and height) are really abrupt, and doesn't make for a nice smooth rotating body. So I had to come up with something else: a smoothed line.</p>
        <p>There are various different ways of smoothing line, but one of the easier ones that I found was the <a href="http://en.wikipedia.org/wiki/Spline_%28mathematics%29">Spline</a>. A spline is a smoothing function that connects all the data points with sufficiently smooth curves. I used the implementation at <a href="http://www.script-tutorials.com/smooth-curve-graphs-with-php-and-gd/">http://www.script-tutorials.com/smooth-curve-graphs-with-php-and-gd/</a> with the other script available through <a href="https://github.com/derickr/osm-year-in-edits/blob/master/changes/create-camera-from-file.php">https://github.com/derickr/osm-year-in-edits/blob/master/changes/create-camera-from-file.php</a></p>
        <p>If we now look at the same section of the video, we see that the turns and zoom level are much smoother:</p>
        <div class="video">
          <div class="videoWrapper">
            <iframe src="https://player.vimeo.com/video/83049724?title=0&byline=0&portrait=0" frameborder="0"/>
          </div>
          <p><a href="https://vimeo.com/83049724"/>Interpolation between a set of points with a spline</p>
        </div>
        <p>Just focussing on the zoom aspect, I am also reproducing a line graph of the interpolated points here:</p>
        <img src="derickrethans.nl/images/content/spline.png"/>
        <p>As you can see, between 1172 and 1328 it makes a very big dip—and that wrecked with my video as it zoomed in too much. I've fixed that manually for the video, but I would like to find an algorithm that did produce smooth lines without such a big "dip". Any ideas are most welcome.</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


