# items

guid: https://derickrethans.nl/year-of-edits-2013.html
url: https://derickrethans.nl/year-of-edits-2013.html
title: OpenStreetMap: A Year of Edits 2013
published: 2014-01-01 14:02:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.866340
updated_at: 2019-07-20 20:30:22.487659
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="openstreetmap_a_year_of_edits_2013"/>OpenStreetMap: A Year of Edits 2013</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Wednesday, January 1st 2014, 14:02 GMT</div>
        </div>
        <p>Just like the <a href="https://derickrethans.nl/year-of-edits-2013.html">last few years</a> I've analysed all the edits to OpenStreetMap and produced a visualisation. This year I wanted to highlight different projects that OpenStreetMap contributors have done and so instead of having an uniform rotational spinning globe it moved between interesting events.</p>
        <p>The result is: <strong>OpenStreetMap: A Year of Edits (2013)</strong>:</p>
        <div class="video">
          <div class="videoWrapper">
            <iframe src="https://player.vimeo.com/video/83164362?title=0&byline=0&portrait=0" frameborder="0"/>
          </div>
          <p><a href="https://vimeo.com/83164362"/>OpenStreetMap: A Year of Edits (2013)</p>
        </div>
        <p>This animation shows all additions and modifications of nodes (flashes) and cumulative edits (fading to purple) in 2013.</p>
        <p>Some of the highlighted events in no particular order are:</p>
        <ul>
          <li>
            <p>
              <a href="http://wiki.openstreetmap.org/wiki/Antarctica/Import_2013">Antarctica coastline imports</a>
            </p>
          </li>
          <li>
            <p><a href="http://www.openstreetmap.fr/36680-communes">Completion of the boundary data in France</a> (In French)</p>
          </li>
          <li>
            <p>
              <a href="http://hot.openstreetmap.org/updates/2013-11-10_remote_hot_activation_in_the_philippines_for_typhoon_yolandahaiyan">Relief mapping in the Philippines</a>
            </p>
          </li>
        </ul>
        <p>You can see the video in HD on <a href="http://vimeo.com/derickr/osm-2013">Vimeo</a>.</p>
        <p>I have started to make the code for this available through GitHub, but the <a href="https://github.com/derickr/osm-year-in-edits">code</a> is very much not modularized and it is likely it will only work on my set-up. Perhaps I manage to fix this better during the upcoming year!</p>
        <p>
          <strong>Mappy New Year!</strong>
        </p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent24.html
url: https://derickrethans.nl/advent24.html
title: Contributing Advent 24: Wrapping up!
published: 2013-12-24 09:33:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.868798
updated_at: 2019-07-20 20:30:22.490117
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_24_wrapping_up"/>Contributing Advent 24: Wrapping up!</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> Dieren, Netherlands</div>
          <div class="date">Tuesday, December 24th 2013, 10:33 CET</div>
        </div>
        <p>Yay, the last one! Writing 24 blog posts in 24 days is a lot more work than actually contributing 24 times. Let's go over the different things that I managed to accomplish, and what you can do to continue some of that work yourself.</p>
        <p>
          <strong>PHP Date/Time</strong>
        </p>
        <p>A few of my contributions are related to PHP's DateTime support. On the <a href="https://derickrethans.nl/advent06.html">6th</a> and the <a href="https://derickrethans.nl/advent09.html">9th</a> I wrote about a bug fixes in "Cookie formats" and "Day of the week" calculations. The bug reports (<a href="https://bugs.php.net/bug.php?id=61599">#61599</a>, <a href="https://bugs.php.net/bug.php?id=63391">#63391</a>, <a href="https://bugs.php.net/bug.php?id=63776">#63776</a> and <a href="https://bugs.php.net/bug.php?id=64294">#64294</a>) mentioned issues all came with a good reproducible script. Please continue filing bug reports, and make sure you follow the guidelines (if possible/necessary) from the <a href="https://derickrethans.nl/advent23.html">23rd</a>.</p>
        <p>The other DateTime issue that I touched on the <a href="https://derickrethans.nl/advent21.html">21st</a> is probably something that you can't really contribute too—unless you want to start petitions and lobby to get rid of Daylight Saving Time altogether…</p>
        <p>
          <strong>Xdebug</strong>
        </p>
        <p>I have been working on and off on <a href="http://xdebug.org">Xdebug</a> for about 13 years now, and it is still a project I am very interested in. So it could not have come as a surprise that there were four posts about it. Some of them, such as on the <a href="https://derickrethans.nl/advent01.html">1st</a> ("Xdebug and hidden properties)" were helped by a good <a href="http://bugs.xdebug.org/view.php?id=987">bug report</a>. But there are plenty of more <a href="http://bugs.xdebug.org/view_all_bug_page.php">bugs</a> in Xdebug. Some of them I can easily reproduce, others only with difficulty and some bugs are just weird. Going over the bug list of your favourite project and trying to reproduce and clarify them is of great help to project maintainers that do not always have a lot of time themselves. In some cases, they might not even have access to the same platforms (ugh, Windows) as what the bug report was filed for.</p>
        <p>Other ways to help an Open Source project is by answering questions on the project's mailing list, or on IRC as I did in the post on the <a href="https://derickrethans.nl/advent08.html">8th</a>. Project maintainers are not awake 24 hours, and especially if you are in a different timezone, your knowledge can help other people. I often lurk in channels on IRC to help out on Xdebug, PHP, OpenStreetMap and MongoDB.</p>
        <p>And of course you can contribute to Open Source projects by just submitting pull requests like I did for the <a href="https://derickrethans.nl/advent15.html">15th</a> and the <a href="https://derickrethans.nl/advent20.html">20th</a>. Most projects will have guidelines on how to contribute. For example Xdebug has a <a href="http://xdebug.org/contributing.php">contributing</a> page, and the MongoDB driver for PHP has a <a href="https://github.com/mongodb/mongo-php-driver/blob/master/CONTRIBUTING.md">page</a> on GitHub.</p>
        <p>
          <strong>OpenStreetMap</strong>
        </p>
        <p>Another large group of contributions (7!) are related to <a href="http://openstreetmap.org">OpenStreetMap</a>. There are many things you can do yourself to help fighting commercial monopolies on geographic data. Two of the posts on OpenStreetMap (on the <a href="https://derickrethans.nl/advent02.html">2nd</a> and <a href="https://derickrethans.nl/advent10.html">10th</a>) dealt with ITO World's <a href="http://www.itoworld.com/static/openstreetmap_tools/osm_analysis.html">OSM Analysis</a>. This tool highlights differences between Ordnance Survey's OS Locator product and OpenStreetMap. If you are in the UK, why not have a look if there are any errors near you? For example, in <a href="http://www.itoworld.com/product/data/osm_analysis/area?name=Manchester">Manchester</a> there are 189 discrepancies right now. If you don't know how to get the right data into OpenStreetMap, take a (geolocated) photo of the street sign and add a link to it in an <em>OpenStreetMap note</em>. Please do not just believe OS Locator's variant of the name, or copy any data of any other map. The only thing that will suffice is proof (a photo) that the name is different. In some cases, you might of course need more photos as I showed in the post for the <a href="https://derickrethans.nl/advent16.html">16th</a>.</p>
        <p>I posted an article on OpenStreetMap notes on the <a href="https://derickrethans.nl/advent05.html">5th</a>. Notes <strong>can</strong> be posted anonymously but then we see the text "This note includes comments from anonymous users which should be independently verified." You can <a href="https://www.openstreetmap.org/user/new">sign up</a> as an OpenStreetMap user, where it will be explained that you <strong>can never copy data from other maps</strong>, but a nice photo of a street sign is of course good enough as well. You can also add comments to notes as a user, and if you sign up as a user, then you can also add and modify the rest of the map like I demonstrated on the <a href="https://derickrethans.nl/advent19.html">19th</a>.</p>
        <p>
          <strong>Hacking time!</strong>
        </p>
        <p>The last group of contributions that I want to talk about are for your <em>own</em> project. I do lots of little things here and there, for example with the <a href="https://github.com/php-geospatial/geospatial">geospatial</a> extension that I wrote about on the <a href="https://derickrethans.nl/advent12.html">12th</a>. In my opinion there is nothing more fun that to hack on your pet project. However, sometimes that results in a project that you work on for 12 years (Xdebug!), so be warned!</p>
        <p>Merry Christmas, and I hope this series of posts inspired you to make your own contributions. Perhaps make it a new year's resolution!</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent19.html
url: https://derickrethans.nl/advent19.html
title: Contributing Advent 19: Lunch time mapping
published: 2013-12-19 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.871170
updated_at: 2019-07-20 20:30:22.492680
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_19_lunch_time_mapping"/>Contributing Advent 19: Lunch time mapping</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Thursday, December 19th 2013, 09:10 GMT</div>
        </div>
        <p>I like going out for walks at lunch time. At least during winter times, it's the only amount of fresh air during daylight I am actually getting. During my walks, I also like to do something useful. Then, my laptop's background image shows me an <a href="http://www.openstreetmap.org/about">OpenStreetMap</a> map of the area the laptop is currently at. (This now works a lot better again after Mozilla started doing their <a href="https://derickrethans.nl/advent11.html">location project</a>). When I am at home, this background however has this (empty) spot on it:</p>
        <img src="derickrethans.nl/images/content/advent2013-19a.png"/>
        <p>And empty spots are annoying! So, during a lunch break in the last few days, I went out and walked over <a href="http://www.openstreetmap.org/way/4072795">The Avenue</a> to do some mapping. I find it way easier to do this with pen and paper, as compared to an application like <a href="http://wiki.openstreetmap.org/wiki/Vespucci">Vespucci</a>. Pen and paper is both faster, and I don't have to fiddle with a tiny phone keyboard. I do however use Vespucci for things like changing a shop's name or type.</p>
        <p>Notes on pen and paper I make plenty, and they look like this:</p>
        <img src="derickrethans.nl/images/content/advent2013-19b.png"/>
        <p>Over the past few years I have developed a short hand for all kinds of things. You can see house numbers (a), how many floors, the colour of the building, the shape and colour of the roof (b), crossings (c) and fences (d) on the extract above.</p>
        <p>With the notes on paper, I then use <a href="http://josm.openstreetmap.de">JOSM</a> to add the information to the map. JOSM's interface looks like:</p>
        <img src="derickrethans.nl/images/content/advent2013-19c.png"/>
        <p>JOSM is a little bit like the <a href="http://vim.org">VIM</a>, but then for editting OpenStreetMap. It is not the easiest editor, but so much more powerfull than the online editor <a href="http://wiki.openstreetmap.org/wiki/ID">iD</a>. With some practice, and learning key combinations, it allows you to pretty quickly add the information from your notes to the map. The following video (at 8 times speed) shows how I added the information from the notes from above to the map:</p>
        <div class="video">
          <div class="videoWrapper">
            <iframe src="https://player.vimeo.com/video/81856483?title=0&byline=0&portrait=0" frameborder="0"/>
          </div>
          <p><a href="https://vimeo.com/81856483"/>Contributing Advent 19: Lunch time mapping</p>
        </div>
        <p>In the end, when everything has been added and uploaded, and the <a href="http://www.openstreetmap.org/about">OpenStreetMap</a> website rendered my updates, the map now looks like:</p>
        <img src="derickrethans.nl/images/content/advent2013-19d.png"/>
        <p>Now it is just a matter of adding the information from the other eight pages of notes I made on this trip!</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent18.html
url: https://derickrethans.nl/advent18.html
title: Contributing Advent 18: Mapping without access
published: 2013-12-18 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.873622
updated_at: 2019-07-20 20:30:22.495029
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_18_mapping_without_access"/>Contributing Advent 18: Mapping without access</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Wednesday, December 18th 2013, 09:10 GMT</div>
        </div>
        <p>So, this is another one about chasing down issues on ITO World's analysis, and probably not the last one. However, in this case it needed a little bit more than photographing street name signs. This is the tale of <a href="http://www.openstreetmap.org/way/32954706#map=18/51.49746/-0.19013">St. Mary's Place</a>.</p>
        <p>OpenStreetMap had this street as St. Mary's Gate, a road in the same <a href="http://www.openstreetmap.org/#map=18/51.49741/-0.19055">area</a>. ITO World's analysis had flagged this up as being incorrect, so before the last <a href="http://bit.ly/londonosm10">OpenStreetMap winter meet-up</a> I went to have a look. After checking out an apparently <a href="http://www.openstreetmap.org/way/26741199">missing street</a>, and adding it to OpenStreetMap I walked towards the supposed location of St. Mary's Place. However, I couldn't really get near as there was a <a href="http://www.openstreetmap.org/node/376497546">gate</a> blocking my way.</p>
        <p>Not really a problem I thought, so walked to the other side of the estate and encountered <a href="http://www.openstreetmap.org/node/371156450">another set of gates</a>. Now I got annoyed as I had walked quite a bit, and I still had to check out <a href="http://www.openstreetmap.org/way/157471066">Palace Green</a> before the meet-up as well. In any case, I was looking around to see whether there was a public entrance after all, when a pedestrian left through a gate. Instead of trying to sneak in, I just asked him about the name of the street… and that was it. He could confirm that indeed St. Mary's Gate and St. Mary's Place split in the middle.</p>
        <p>Getting to know information about Palace Green is tricky too, and that conclusion will have to wait until I've spoken to <a href="http://www.openstreetmap.org/user/Paul%20The%20Archivist">Paul</a> who mapped it in the first place.</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent16.html
url: https://derickrethans.nl/advent16.html
title: Contributing Advent 16: St. Johns vs. St John's
published: 2013-12-16 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.875960
updated_at: 2019-07-20 20:30:22.497413
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_16_st_johns_vs_st_john_s"/>Contributing Advent 16: St. Johns vs. St John's</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Monday, December 16th 2013, 09:10 GMT</div>
        </div>
        <p>I already wrote about comparing <a href="http://data.gov.uk/dataset/os-locator1">OS Locator</a> data with OpenStreetMap for <a href="https://derickrethans.nl/advent02.html">day 2</a> and <a href="https://derickrethans.nl/advent08.html">day 8</a>, but there are plenty of other highlights in London that I can easily have a look at. So a certain lunch time walk brought me to the borough of <a href="http://en.wikipedia.org/wiki/Kensington_and_Chelsea_London_Borough_Council">Kensington and Chelsea</a>. I first looked at <a href="http://www.openstreetmap.org/way/34150447">Camborne Mews</a> (which OS Locator had as Cambourne Mews) and <a href="http://www.openstreetmap.org/way/34227497">Cowling Close</a> (which OpenStreetMap had as Cowling Cross).</p>
        <p>But the more interesting thing to look at was the last one of the three. Apostrophes are always a tricky part, both for mappers as well as councils. I had already run into <a href="http://www.openstreetmap.org/way/4981900/history">St. Marks Road</a> <a href="http://www.openstreetmap.org/changeset/5449937">back in 2010</a> but ITO World now also flagged up St. Johns Gardens. Both segments <em>were</em> mapped as St. John's Gardens in OpenStreetMap. OS Locator has the street as two <strong>different</strong> names. For the  <a href="http://ris.dev.openstreetmap.org/oslmusicalchairs/map?osl_id=695310">western segment</a> it has <em>St John's Gardens</em>:</p>
        <img src="derickrethans.nl/images/content/advent2013-16a.png"/>
        <p>For the <a href="http://ris.dev.openstreetmap.org/oslmusicalchairs/map?osl_id=695945">eastern segment</a> it has <em>St Johns Gardens</em>:</p>
        <img src="derickrethans.nl/images/content/advent2013-16b.png"/>
        <p>These images are extracts from another comparison tool between OS Locator and OpenStreetMap, called <a href="http://ris.dev.openstreetmap.org/oslmusicalchairs/map?osl_id=695310&zoom=18&lat=51.51014&lon=-0.20641&layers=B0TT&view_mode=pseudorandom">OS Locator musical chairs</a>, made by <a href="http://wiki.openstreetmap.org/wiki/User:Robert">Robert Scott</a>.</p>
        <p>In any case, after visiting Cowling Close slightly to the west, I came through Clarendon Road and there on the corner with St John's Gardens, I spotted the following sign:</p>
        <img src="derickrethans.nl/images/content/advent2013-16c.png"/>
        <p>Ah! So I concluded, the name of the street is St John's Gardens, with apostrophe.</p>
        <p>I continued along the road towards St Johns Church, and on the corner with Lansdowne Road I found the following two signs:</p>
        <img src="derickrethans.nl/images/content/advent2013-16d.png"/>
        <img src="derickrethans.nl/images/content/advent2013-16e.png"/>
        <p>Argh, both without apostrophe! On the north end around the church, on the corner with Lansdowne Crescent, there two more signs without apostrophe (sorry, one was too blurry to add here):</p>
        <img src="derickrethans.nl/images/content/advent2013-16f.png"/>
        <p>I took another photo on the corner with Ladbroke Grove but didn't pay much attention to it and assumed it was without apostrophe as well. So with five signs saying <em>St Johns Gardens</em> (without apostrophe) and sign with <em>St John's Gardens</em> (with apostrophe), the street is surely just called <em>St Johns Gardens</em>. Back home, I then <a href="http://www.openstreetmap.org/changeset/19256893">edited</a> OpenStreetMap to say that the name of the streets are <em>St Johns Gardens</em> and <strong>not</strong> <em>St John's Gardens</em>.</p>
        <p>Just now, while writing the article I had a better look at the sign on the corner with Ladbroke Grove:</p>
        <img src="derickrethans.nl/images/content/advent2013-16g.png"/>
        <p>And yes, there is indeed an apostrophe… What surprises me most however is that it's really up to the council to decide the spelling of street names, and convey that information to Ordnance Survey. Apparently, they can't make their minds up on what to print on the signs, and what to tell Ordnance Survey (considering they had two parts of the same street with two different names). In any case, OpenStreetMap <em>maps what is on the ground</em>, and right now, with three against two, <em>St Johns Gardens</em> (without apostrophe) wins.</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent14.html
url: https://derickrethans.nl/advent14.html
title: Contributing Advent 14: Brown Hart Gardens
published: 2013-12-14 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.878323
updated_at: 2019-07-20 20:30:22.499803
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_14_brown_hart_gardens"/>Contributing Advent 14: Brown Hart Gardens</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Saturday, December 14th 2013, 09:10 GMT</div>
        </div>
        <p>Besides fixing bugs from <a href="https://derickrethans.nl/advent02.html">ITO World</a> or <a href="https://derickrethans.nl/advent05.html">OpenStreetMap notes</a> I also tend to do some general mapping when I am out and about. Coinciding with my "buying the whisky of the month" event (if you want to contribute to this, see the <a href="http://www.amazon.co.uk/registry/wishlist/SLCB276UZU8B">wishlist</a> ;-) ), we ended up walking past a rather curious building with a park on top in London: <a href="http://en.wikipedia.org/wiki/Brown_Hart_Gardens">Brown Hart Gardens</a>.</p>
        <p>The gardens, new café, and the "Duke Street Electricity Substation" that lays underneath were not mapped at all, and of course I could not let that be. From photos and memory, I re-created the <a href="http://www.openstreetmap.org/relation/3354838#map=19/51.51315/-0.15145">gardens</a> in <a href="http://josm.openstreetmap.de/">JOSM</a>, my preferred OpenStreetMap editor and now it is available on the <a href="http://www.openstreetmap.org/#map=19/51.51310/-0.15151">map</a> — including the <a href="http://www.openstreetmap.org/way/249455143">steps</a> and the <a href="http://www.openstreetmap.org/node/2560520283">elevator</a>:</p>
        <img src="derickrethans.nl/images/content/advent2013-14a.png"/>
        <p>There is also a 3D rendering <a href="http://map.f4-group.com/#lat=51.5129156&lon=-0.1514470&zoom=19&ui.discoveryOpen=false&camera.theta=71.087&camera.phi=141.853">available</a>:</p>
        <img src="derickrethans.nl/images/content/advent2013-14b.png"/>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent13.html
url: https://derickrethans.nl/advent13.html
title: Contributing Advent 13: Haiyan Progress Video
published: 2013-12-13 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.880651
updated_at: 2019-07-20 20:30:22.502195
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_13_haiyan_progress_video"/>Contributing Advent 13: Haiyan Progress Video</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Friday, December 13th 2013, 09:10 GMT</div>
        </div>
        <p>Once in a while, the <a href="http://wiki.openstreetmap.org/wiki/London/Winter_2013-2014_events">London OpenStreetMap group</a> organises a <a href="http://wiki.openstreetmap.org/wiki/London/London_Hack_Weekend_Nov_2013">hack weekend</a> where a bunch of us hack on various different projects, such as the <a href="https://www.mapbox.com/blog/redesigning-openstreetmap-org/">new design</a>, <a href="http://project-osrm.org">project OSRM</a> or <a href="http://sk53-osm.blogspot.co.uk/2013/12/harry-we-missed-you-london-osm-hack.html">using Food Hygiene Open Data</a> to improve OpenStreetMap.</p>
        <p>The last few that I have attended were always close to the end of the year, so I tend to use them to make my "year of edits" videos:</p>
        <ul>
          <li>
            <p>2010: <a href="https://vimeo.com/34406096">https://vimeo.com/34406096</a></p>
          </li>
          <li>
            <p>2011: <a href="https://vimeo.com/34404102">https://vimeo.com/34404102</a></p>
          </li>
          <li>
            <p>2012: <a href="https://vimeo.com/56374742">https://vimeo.com/56374742</a></p>
          </li>
        </ul>
        <p>This year instead I focussed on creating a mapping party progress video for the great work that the Humanitarian OpenStreetMap Team (HOT) has been doing for organising and mapping the destruction of Typhoon <a href="http://en.wikipedia.org/wiki/Typhoon_Haiyan">Haiyan</a> in the Philippines. There have been lots of <a href="http://wiki.osm.org/wiki/Typhoon_Haiyan#Media_coverage">media coverage</a> but I wanted to show how much they have added in the two weeks since the thyphoon hit—and what better to do this then with a progress video.</p>
        <p>I have made many videos before, mostly for mapping parties in London, but also for other areas, such as <a href="https://vimeo.com/channels/osm/54113868">Burgos</a> in Spain, and for the <a href="https://vimeo.com/channels/osm/62772466">Carse of Gowrie</a>. At first I wanted to make them for the large area around Tacloban but in the end it made more sense to just do a slightly smaller area:</p>
        <div class="video">
          <div class="videoWrapper">
            <iframe src="https://player.vimeo.com/video/80922315?title=0&byline=0&portrait=0" frameborder="0"/>
          </div>
          <p><a href="https://vimeo.com/80922315"/>HOT: Tacloban (Typhoon Haiyan mapping)</p>
        </div>
        <p>I have used a slightly tweaked variant of my OSM progress software that you can find on GitHub: <a href="https://github.com/derickr/osm-progress">https://github.com/derickr/osm-progress</a> — I'll soon clean up my changes and push them to the repository as well.</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent12.html
url: https://derickrethans.nl/advent12.html
title: Contributing Advent 12: Geospatial algorithms
published: 2013-12-12 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.882970
updated_at: 2019-07-20 20:30:22.504496
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_12_geospatial_algorithms"/>Contributing Advent 12: Geospatial algorithms</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Thursday, December 12th 2013, 09:10 GMT</div>
        </div>
        <p>This is technically not a new contribution, but one I've been working on and off at for the past months. The <a href="https://github.com/php-geospatial/geospatial">geospatial</a> PHP extension contains a whole bunch of algorithms that deal with geospatial problems. The <a href="http://en.wikipedia.org/wiki/Haversine_formula">Haversine</a> and <a href="http://en.wikipedia.org/wiki/Vincenty%27s_formulae">Vincenty</a> functions calculate distances on a globe, an implementation of the <a href="http://en.wikipedia.org/wiki/Helmert_transformation">Helmert transformation</a> algorithm converts between different <a href="http://en.wikipedia.org/wiki/Datum_%28geodesy%29">datums</a>, and some general functions to convert from f.e. 16°45'12" N to 16.7533° N.</p>
        <img class="left" align="left" src="derickrethans.nl/images/content/advent2013-12a.png"/>
        <p>These are all useful functions, but I'd like to talk about an algorithm that I've added some time ago now. It is an implementation of <a href="http://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm">Ramer-Douglas-Peucker</a> that can be used to simplify a polyline. I have used that for the display of the timezone boundaries that I show in one of my maps examples: <a href="http://maps.derickrethans.nl/?l=timezone&lat=51.5&lon=0&zoom=8">http://maps.derickrethans.nl/?l=timezone&lat=51.5&lon=0&zoom=8</a>. The boundary data that I have for the UK for example, is so complex that it takes quite a bit of time to render them on the map. On higher zoom levels where that amount of detail simply does not show up, we can of course do with a simpler line, and that is what RDP helps us to achieve.</p>
        <p>The algorithm has as input an array of coordinate pairs:</p>
        <pre>$points = [
          [  0,  10 ],
          [  1,   6 ],
          [  3,   0 ],
          [  5,   6 ],
          [  7,   8 ],
          [  8,   7 ],
          [  9,   4 ],
          [  13, 10 ],
  ];

  </pre>
        <p>As well as a parameter (called ε) that determines the maximum distance from a line that a point is allowed to be so that it is included in the returned array. The algorithm is recursive and will for each segment—starting with the whole array:</p>
        <ul>
          <li>
            <p>Find the further point from the line as marked by the first and last point</p>
          </li>
          <li>
            <p>If the point is closer than ε to the line segment then any points not currently marked to keep can be discarded without the simplified curve being worse than ε.</p>
          </li>
          <li>
            <p>If the point furthest from the line segment is greater than ε from the approximation then that point must be kept. The algorithm recursively calls itself with the first point and the worst point and then with the worst point and the last point (which includes marking the worst point being marked as kept).</p>
          </li>
        </ul>
        <p>With the above <code>$points</code> array, we can run the function like:</p>
        <pre>$points = rdp_simplify( $points, 2.75 );

  </pre>
        <p>We are left with:</p>
        <pre>$points = [
          [  0,  10 ],
          [  3,   0 ],
          [  5,   6 ],
          [  9,   4 ],
          [  13, 10 ],
  ];

  </pre>
        <p>The image below shows the effect of <code>ε = 2.75</code> and other values:</p>
        <img src="derickrethans.nl/images/content/advent2013-12b.png"/>
        <p>In case you do not want to install a PHP extension to get that functionality, I also have an example implementation in PHP at <a href="https://github.com/derickr/3angle/blob/master/rdp.php">https://github.com/derickr/3angle/blob/master/rdp.php</a>, however, the time it takes to run the algorithm in PHP is about the same as the browser needs to draw the additional points of the lines.</p>
        <p>Expect to see an updated version in the <a href="https://github.com/php-geospatial/geospatial">geospatial</a> extension that also handles multi-polygons in the future. This should speed up running this over the large amount of UK polygons, as there are plenty of islands that also need to be done.</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent11.html
url: https://derickrethans.nl/advent11.html
title: Contributing Advent 11: Mozilla's geolocation services
published: 2013-12-11 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.885331
updated_at: 2019-07-20 20:30:22.506915
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_11_mozilla_s_geolocation_services"/>Contributing Advent 11: Mozilla's geolocation services</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Wednesday, December 11th 2013, 09:10 GMT</div>
        </div>
        <p>About a months and a half ago Mozilla <a href="https://blog.mozilla.org/services/2013/10/28/introducing-the-mozilla-location-service/">announced</a> its <a href="https://wiki.mozilla.org/CloudServices/Location">Location Services</a>. It is "an experimental pilot project to provide geolocation lookups based on publicly observable cell tower and WiFi access point information". In other words, it is going to provide a similar service as Google's location service that Firefox currently uses for its <a href="http://www.mozilla.org/en-GB/firefox/geolocation/">location aware browsing</a>.</p>
        <img class="left" align="left" src="derickrethans.nl/images/content/advent2013-11a.png"/>
        <p>In order to provide this service, they need geolocated data so that they know where WiFi access points and cell towers are. They have released a tool, <a href="https://github.com/dougt/MozStumbler/releases">MozStumbler</a>, that allows everybody to collect this data and provide it to Mozilla. I believe that there should be open alternatives for important services like this, and I think that Mozilla seems like a good caretaker of this data, respecting privacy. So I have installed <a href="https://github.com/dougt/MozStumbler/releases">MozStumbler</a> and turn it on whenever I go on a new route through town.</p>
        <img class="right" align="right" src="derickrethans.nl/images/content/advent2013-11b.png"/>
        <p>The application isn't the prettiest as you can see, but it does work well. There is also a "Test Mozilla Location Services" which uses their API to lookup the location—without using the GPS. <a href="https://mozilla-ichnaea.readthedocs.org/en/latest/">Ichnaea</a> is the public API that you can use to do lookups yourself. The <a href="https://mozilla-ichnaea.readthedocs.org/en/latest/api/geolocate.html">Geolocate</a> API implements the same interface as the <a href="https://developers.google.com/maps/documentation/business/geolocation/">Google Maps Geolocation API endpoint</a>. But there is a more extensive <a href="https://mozilla-ichnaea.readthedocs.org/en/latest/api/search.html">Search</a> API as well. I will soon be upgrading <a href="http://derickrethans.nl/projects.html#haunt">Haunt</a> to use this new API as the one it currently uses is discontinued.</p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: https://derickrethans.nl/advent10.html
url: https://derickrethans.nl/advent10.html
title: Contributing Advent 10: Dragons in Dollis Hill
published: 2013-12-10 09:10:00
fetched: 2019-07-20 20:30:22.386321
created_at: 2019-06-09 18:12:13.887602
updated_at: 2019-07-20 20:30:22.509234
summary:
  <div class="article">
    <div class="body">
      <div class="articleListItem">
        <h1><a name="contributing_advent_10_dragons_in_dollis_hill"/>Contributing Advent 10: Dragons in Dollis Hill</h1>
        <dl class="head"/>
        <div class="articleMetaData">
          <div class="location"> London, UK</div>
          <div class="date">Tuesday, December 10th 2013, 09:10 GMT</div>
        </div>
        <p>I had previous managed to clear out all the discrepancies on ITO World's Analysis for <a href="http://www.itoworld.com/product/data/osm_analysis/area?name=Camden">Camden</a>, <a href="http://www.itoworld.com/product/data/osm_analysis/area?name=Kensington+And+Chelsea">Kensington and Chelsea</a> and the <a href="http://www.itoworld.com/product/data/osm_analysis/area?name=City+Of+Westminster">City of Westminster</a>. But I have so far not yet managed to address all of the issues in my home borough <a href="http://www.itoworld.com/product/data/osm_analysis/area?name=Brent">Brent</a>. I guess this is mostly because the "issues" are not really in the nicest parts of the borough—perhaps there are even dragons there. But I am trying to make a point to get to them anyway, so I set out by bus to Dollis Hill to tackle <a href="http://www.openstreetmap.org/browse/way/248940442">Grunwick Close</a>, <a href="http://www.openstreetmap.org/browse/way/67752051">Flowers Close</a> and <a href="http://www.openstreetmap.org/browse/way/4081584">St Andrews Close</a>.</p>
        <p>The first one, Grunwick Close was a new road and I couldn't find any street sign at all. I had to ask a local resident which indeed confirmed that I had the correct street name. The latter two actually had a "FIXME" tag on them, and no name tag. The first one, Flowers Close, was <a href="http://www.openstreetmap.org/browse/way/67752051/history">previously</a> incorrectly mapped as Flower Road (and OS Locator had it right), but the second one was wrong on OS Locator as it had thought there was an apostrophe in "Andrews"—the street sign disagreed.</p>
        <img class="right" align="right" src="derickrethans.nl/images/content/advent2013-10a.png"/>
        <p>The <a href="http://www.openstreetmap.org/browse/way/67254155/history">platform at Dollis Hill</a> had this annoyingly nagging "FIXME" tag on it and I couldn't let that go unfixed of course. (Did I mention OpenStreetMap is addictive and that I have some form of OCD?) So I spend some time mapping all the aspects of the tube station: entrances, tunnels, stairs, platforms, tracks and stop positions—all following the "new" <a href="http://wiki.openstreetmap.org/wiki/Proposed_features/Public_Transport">public transport scheme</a>. The image here shows how the new station looks like on the map.</p>
        <p>You can see it on the website at <a href="http://www.openstreetmap.org/browse/relation/3350099">http://www.openstreetmap.org/browse/relation/3350099</a></p>
        <div class="patreon">
          <a href="https://www.patreon.com/bePatron?u=7864328">
            <img src="https://derickrethans.nl/images/become_a_patron_button.png" alt="Become a Patron!"/>
          </a>
        </div>
        
        
      </div>
    </div>
  </div>


guid: 1488
url: http://shtosm.ru/all/na-si-bystree/
title: Быстрее загружай
published: 2019-06-07 17:23:36
fetched: 2019-09-19 22:00:21.994743
created_at: 2019-06-09 18:12:14.066845
updated_at: 2019-09-19 22:00:22.123290
summary:
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/uploading_changes.jpg" width="851" height="243" alt="" />
  </div>
  <p>Возможно, вы заметили, что правки передаются на сервер быстрее. Или не заметили, потому что это не особенно тормозило и раньше. На прошлой неделе код сайта osm.org, Rails Port, <a href="https://git.openstreetmap.org/chef.git/commitdiff/4869af95cca2e03c7d84de5daae9e70cba72be46">отстранили</a> от ещё одной части API: загрузки пакетов правок. Ваши пакеты теперь обрабатывает Cgimap.</p>
  <p>Cgimap — это программа на C++, которая обрабатывает часть запросов к OSM API. Сайт написан на Ruby on Rails, поэтому каждый запрос к нему проходит через длинную цепочку обработчиков, не всегда оптимальную. Кроме того, передача данных от базы к клиенту в Rails требует много памяти и задействует сборщик мусора, который напрягает систему. Поэтому запрос /<i>map</i> на скачивание данных был переписан на C++ ещё в 2009 году.</p>
  <p>Предполагается, что если появится API 0.7, то он целиком будет сделан внутри Cgimap. Мы давно хотим отвязать API от кода сайта, чтобы дневнички и интерактивная карта остались в Rails Port, а запросы к базе были отдельно. Но работа не кипит, скрипт обрабатывает лишь малую часть запросов. Сложным барьером была поддержка авторизации OAuth, но в 2016 году Мэтт Эймос, главный разработчик Cgimap, её сделал. В мае 2018 Mmd приступил и написал обработчик второго по количеству перекачиваемых данных запроса: /<i>upload</i> для отправки пакетов правок на сервер. Только в начале этого года у разработчиков дошли руки подчистить концы, отрефакторить и приступить к тестированию.</p>
  <p>Второго марта бета-версию Cgimap 0.7.0 (это версия программы, не API) подключили к dev.osm.org, 28 марта тестирование закончилось и версию отправили в репозитории. Но у Тома, как водится, руки дошли подключить только после <a href="https://github.com/openstreetmap/operations/issues/293">третьего напоминания</a>. Пара дней ушла на настройку доступа к базе данных, и с 28 мая пакеты правок обрабатываются сервером быстрее, особенно большие. Поскольку это первый запрос, на котором Cgimap требует аутентификации, в ней нашлись несколько проблем, которые поправили в трёх версиях Cgimap за три дня.</p>


guid: 1487
url: http://shtosm.ru/all/chto-za-maxar/
title: Что за Maxar?
published: 2019-05-30 20:54:58
fetched: 2019-09-09 12:00:31.639783
created_at: 2019-06-09 18:12:14.070356
updated_at: 2019-09-09 12:00:31.726186
summary:
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/kaliningrad-dg-maxar.jpg" width="1047" height="448" alt="" />
  </div>
  <p>Три недели назад вы могли заметить новый слой спутниковых снимков в вашем редакторе: DigitalGlobe Vivid. А неделю назад — что его заменили два других, тоже на весь мир: Maxar Standard и Maxar Premium. Откуда они и зачем?</p>
  <p>Быстрое гугление показывает, что Maxar Technologies получилась в октябре 2017 года слиянием MDA Holdings и DigitalGlobe. Объединённая компания занимается всем, что связано с космосом и спутниками. Теперь ещё и продажей космоснимков. Все сотрудники перешли в Maxar, и они не собираются отзывать разрешение на обрисовку снимков у OpenStreetMap. Никто не делал заявлений на этот счёт, но <a href="https://github.com/osmlab/editor-layer-index/pull/655">пул-реквесты</a> в каталог подложек показывают, что в Maxar помнят про сотрудничество и обновляют ссылки.</p>
  <p>Новые слои отличаются от старых. Технически — тем, что они не проходят через серверы Mapbox, а отдаются напрямую с DG. Содержанием тоже. В описании Maxar Premium упоминается, что это то же, что DG +Vivid, который нам два года назад <a href="http://shtosm.ru/all/digitalglobe-nash/">выдали</a> в слое DG Premium. <a href="http://osmz.ru/imagery/">Сравнение снимков</a> в десяти городах показало, что чаще всего оба новых слоя совпадают, а если нет, то свежее может быть как Standard (в Новосибирске), так и Premium (в Калининграде). Но в паре мест — в районах Москвы, Петербурга и Саранска — самым свежим оказался DG Premium.</p>
  <p>Это печально, потому что старые слои DigitalGlobe <a href="https://github.com/osmlab/editor-layer-index/pull/668">отключат</a> не позднее 30 июня. Мы потеряем как самые свежие снимки, по которым можно нарисовать стадион в Саранске, так и самые старые, которые напомнят, как выглядели города до строительного бума. Представители Maxar пока не отвечают на вопросы, но на всякий случай, не откладывайте рисование новых жилых комплексов и инфраструктуры. Конечно, через год-два они появятся на всех слоях, но у нас же самая актуальная и точная карта сегодня, а не через год.</p>


guid: 1485
url: http://shtosm.ru/all/vision-of-a-future/
title: Vision of a Future
published: 2019-05-03 08:57:46
fetched: 2019-09-08 13:00:19.713229
created_at: 2019-06-09 18:12:14.073415
updated_at: 2019-09-08 13:00:19.808256
summary:
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/mapbox-vision-leftlane.jpg" width="1400" height="600" alt="" />
  </div>
  <p>Вы наверняка знаете про <a href="https://vision.mapbox.com/index.html">Mapbox Vision SDK</a>. Удивительная библиотека, которая прямо на устройстве прогоняет видео с телефонной камеры через нейронную сеть. Ищет в кадре машины, пешеходов и знаки, размечает кадр попиксельно, рисует на экране маршрут и напоминает об ограничениях. Продвинутый видеорегистратор. <a href="https://www.youtube.com/watch?v=9FFMJ1Xa7TQ">Видеоролик</a> лучше любого описания: да, техника дошла, будущее уже в телефоне и почти не тормозит.</p>
  <p>Его проблема в том, что он не нужен водителям. Несомненно, эта разработка на порядок интереснее самодвижущихся машин, которыми прожужжали все новости: такие машины купят несколько тысяч человек, а телефоны есть у миллионов водителей. Но авторы, как это часто бывает с технарями, для гениального ядра придумали яркий, но совершенно бесполезный обвес. Забыли одно из главных правил вождения: не отвлекаться. Все их линии и напоминания на практике не нужны.</p>
  <p>Когда я за рулём, экран моего телефона выключен. Там работает навигатор и в правильные моменты он подсказывает: «поверните налево», «через триста метров направо». Никогда не удавалось оценить, сколько это — триста метров; порой приходилось резко тормозить и сдавать назад. Если прикрепить телефон к приборной панели, я начинаю его палить: контролировать скорость и время по GPS, смотреть на окрестности на карте. Читать уведомления из мессенджеров.</p>
  <p>Поэтому AR, дополненная реальность, не нужна. Остальные части презентации тоже по-своему бесполезны. Предупреждения, например, либо избыточны и их игнорируешь — вспомните «динь» о превышении скорости в яндекс-картах, — либо редки и слишком неожиданны, чтобы вовремя среагировать. Картинки на сайте Vision SDK предлагают и третий вид уведомлений: слишком ярких и перегруженных, чтобы читать за рулём. Последний слайд в «Use Cases» особенно зловещ: внедрение системы приведёт к штрафам за то, что водители не роботы.</p>
  <p>Когда-то я целый год смотрел ролики ру-чп и теперь при езде я вижу сотни собственных ошибок, которые могли бы привести к аварии. Отвлёкся на медиафасад. На узкой полосе мечусь между зеркалами и машиной впереди. Автобус справа закрыл сигнал светофора и я его предположил. Мчу по трассе и ожидаю, что пешеходы не самоубийцы. Перестраиваюсь влево и каждый раз жду, что влечу в машину либо в слепой зоне, либо впереди, когда проверяю эту зону.</p>
  <p>Vision SDK захватывает воображение как обещание безопасного будущего. Водителю нужна помощь — но не в навигации: с ней справляются и обычные навигаторы. Этот технологический костыль приблизит то, о чём безосновательно грезят апологеты автономного транспорта: более безопасные и эффективные дороги. Для этого не нужно покупать новую машину и бояться, что встретишь непредсказуемый для искусственного интеллекта таз. Достаточно поставить две камеры сзади и спереди, запустить приложение и построить маршрут.</p>
  <p>Главное в разработке — заткнуть внутреннего технаря и решать проблемы, а не предоставлять возможности. Например, не показывать, а говорить. Не «триста метров», а «перед забором». «Две минуты до съезда, перестройся вправо». Или «десять секунд», если машин вокруг мало. Это всё базовые улучшения, для которых не нужно камер. Vision SDK позволяет оценивать окружающую обстановку так, как не сможет никакой водитель, особенно усталый или в сумерках. «Впереди пешеход, притормози».</p>
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/empty-screen-dash.jpg" width="1400" height="600" alt="" />
  </div>
  <p>Обучение привычкам водителя, плюс данные с камер и автомобильных датчиков, плюс картография решат все затруднения, знакомые каждому водителю. Стоит ли разгоняться, или всё равно на светофоре стоять? Безопасно ли обгонять таз с прицепом, когда встречка далеко, но слепит фарами? Прервать обгон и встроиться между фурами? Быстрее ли соседняя полоса и точно ли там сейчас никого нет в слепой зоне? Сколько на этой дороге полос, не стоит ли принять чуть вправо? Тормозить или резко свернуть перед препятствием? Не влетит ли в меня обгоняющий, если я поверну налево?</p>
  <p>Нужно не вываливать всё, что вычислили, а ненавязчиво подсказывать и, возможно, светить крупными, понятными индикаторами. Знаю, что превышаю скорость, но пора бы понять, что я люблю ездить плюс пять к ограничению, за которые не штрафуют. Если я в правой полосе, то зачем предлагать держаться правее? Линия маршрута нафиг не нужна, когда о повороте можно сказать голосом, а вот индикатор «можно обгонять» очень бы пригодился. Как и индикаторы безопасности перестроения на левом и правом боковых зеркалах.</p>
  <p>Будут ли аварии из-за водителей, излишне положившихся на такого помощника? Конечно. Это предсказуемая угроза безопасности, с которой приложение, единственная задача которого — минимизация угроз, — должно бороться. Производители автономных машин сейчас думают, как водитель, следящий за правильностью решений, должен сигнализировать о своём внимании. Наша гипотетическая система тоже должна постоянно давать сигналы, что она следит за дорогой. При этом не раздражая водителя. Пониженная уверенность в данных — тоже красный флаг, который должен привлекать внимание. Эффективные сигнальные системы потребуют работы когнитивистов, которых не каждая компания догадается нанять. Это не лампочки на приборной панели, а интерфейс более высокого, интуитивного уровня.</p>
  <p>Будущее вождения может быть построено на Vision SDK, если не произойдёт неминуемого. Я жду традиционного: Mapbox выпустит SDK и со временем переключит разработчиков на другие задачи. Несколько стартапов вырастут вокруг этой библиотеки и каждый запилит свою часть головоломки. Будут несколько приложений с платными подписками (потому что использование SDK стоит по доллару за человека), которые почти, но не совсем, помогают водителю, каждое по-своему. Пазл из них не сложится в идеального помощника. И в финале, как недавно нейронные сети для определения и маскировки лиц, технология обработки автомобильного видео станет ещё одним не взлетевшим технологическим прорывом.</p>
  <p>Но я надеюсь, что когда-нибудь перестану волноваться об обгонах, перестроениях и пропущенных поворотах. Технологии возрождаются в лучшем виде после того, как полежат несколько лет в болоте разочарования. Вождение будет безопасным.</p>


guid: 1484
url: http://shtosm.ru/all/osrm-vsyo/
title: Ничей OSRM
published: 2019-04-30 17:41:21
fetched: 2019-09-07 19:30:23.949693
created_at: 2019-06-09 18:12:14.076699
updated_at: 2019-09-07 19:30:24.029321
summary:
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/osrm-commit-chart.png" width="729" height="177" alt="" />
  </div>
  <p>История самого популярного движка для прокладки маршрутов, похоже, приближается к финалу. Или, если повезёт, к новой странице. Проект уже несколько месяцев сидит без программистов: можно сказать, что разработка мертва.</p>
  <p>В начале 2012 года OSRM <a href="http://shtosm.ru/all/priyatno-posmotret-na-bystry-routing-i-otzyvchivy-interfeys/">всех удивил</a> необычно быстрой прокладкой маршрутов: линия перемещалась быстрее, чем загружались тайлы подложки. Этот движок хитрым образом сворачивает дорожную сеть в предобработке и затем держит все данные в памяти — поэтому вычисления выполняются за миллисекунды, а не секунды, как в то время у альтернативных проектов.</p>
  <p>Не удивительно, что спустя год его автора, Денниса Люксена, <a href="https://blog.mapbox.com/dennis-luxen-joins-mapbox-8d81db32aa68">завербовали в Mapbox</a> — сразу после защиты диплома в университете Карлсруэ. В то время это было позитивной новостью: OSRM не откладывался на полку отработанных дипломных проектов, как это случилось, например, с MapSurfer, а получал поддержку деньгами и коллегами от крупной компании. С тех пор OSRM становился всё умнее, получая полезные на практике функции, вроде улучшенных пошаговых инструкций, поддержки полос и <i>restriction:conditional</i>.</p>
  <p>Всё поменялось в январе 2018 года. Именно тогда Mapzen объявил о закрытии всех своих сервисов, включая движок роутинга Valhalla. Его особенностью была разбивка данных по тайлам, что предположительно помогало обновлять данные и загружать их в память фрагментами. В то время (да и в это) векторные тайлы захватили воображение руководства Mapbox, поэтому не очень удивляет, что всего через три дня все пять разработчиков <a href="https://blog.mapbox.com/mapbox-%EF%B8%8F-valhalla-1fdda95ac4f">присоединились</a> к компании. Спустя месяц Valhalla уже строила маршруты по пробкам на сотнях серверов Mapbox.</p>
  <p>А что OSRM? В апреле 2018 активность главного репозитория проекта внезапно снизилась на два порядка: с двадцати коммитов в день до пары коммитов в неделю. Деннис ушёл ещё в начале 2015. Часть разработчиков перевели на другие проекты, но большинство, судя по слухам и активности на GitHub, просто уволили. Из верхних десяти участников проекта (одиннадцатый — Лев из Juno) активность после того апреля проявлял только сотрудник Mapbox Дэниел Паттерсон. Да и то — он лишь проверял и принимал пул-реквесты от пользователей библиотеки, не тратя много времени на разработку. По косвенным признакам ощущалось, что изменения даже не тестировались как следует.</p>
  <p>«Линия жизни» репозитория, <a href="https://github.com/Project-OSRM/osrm-backend/graphs/commit-activity">график коммитов</a>, дважды за последний год прерывалась на месяц: в августе и в декабре. Когда я залил очередной несложный пул-реквест, я ожидал, что нынешний перерыв тоже скоро закончится. Но с последней правки кода прошло уже больше двух месяцев. Дэниела, судя по строчкам вида «42 contributions in private repositories», загрузили работой в Mapbox, и OSRM остался без мейнтейнера. Некому даже нажимать на кнопку принятия пул-реквестов. Другими словами, проект мёртв.</p>
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/ors-route-example-nq8.png" width="1202" height="515" alt="" />
  </div>
  <p>Почему так получилось? У всех всё работает. OSRM как-то страхует Valhalla на серверах Mapbox, но в нём ничего не изменится, потому что вся работа теперь в векторных тайлах. Maps.Me, который когда-то удивил работой движка на смартфонах, пару лет как выкорчевал OSRM из кода, заменив своей, более гибкой навигацией. Для публичных серверов, типа недавно добавленного на глагне роутинга от немецкого FOSSGIS, достаточно старых версий. Вероятно, OSRM просто исчерпал возможности для развития, а отсутствие даже теоретической возможности поддержать различные профили, пробки и прочие динамические ограничения толкают корпоративных пользователей к разработке альтернативных решений. Если энтузиастов не найдётся и среди сообщества OpenStreetMap, проекту крышка.</p>
  <p>Не сказать, что это плохо: <a href="https://www.graphhopper.com/">GraphHopper</a> ещё жив. Разработка движка <a href="https://github.com/valhalla/valhalla">Valhalla</a>, разумеется, очень активна, но как и другие проекты Mapbox, поднять его на своём сервере будет непросто. А в недрах Гейдельбергского университета Амандус Бутцер и Тим Макколи растят достойную смену: <a href="https://maps.openrouteservice.org/">OpenRouteService</a>.</p>


guid: 1483
url: http://shtosm.ru/all/desyat-let-api/
title: Вечно ноль шестой
published: 2019-04-22 07:09:35
fetched: 2019-09-06 17:00:24.918946
created_at: 2019-06-09 18:12:14.079723
updated_at: 2019-09-06 17:00:25.001234
summary:
  <p>Ровно десять лет назад проект перешёл на API версии 0.6. API, или протокол, — это то, к чему обращаются приложения-редакторы карты для скачивания и загрузки данных на сервер. Как видно, этот протокол менялся пять раз, причём каждый раз серьёзно: то отношения появятся, то сегменты (отрезки между двумя точками) пропадут. Шестая версия API — последняя. Вчера у неё был юбилей.</p>
  <h2>Зачем мне об этом знать?</h2>
  <p>API — это не столько протокол обмена информацией, сколько описание модели данных. Именно он определяет, что карта состоит из точек, линий, отношений и тегов, что у объектов есть версии, что в линии не может быть более двух тысяч точек и так далее. Когда менялся протокол, менялась модель данных. Как в реальном мире полезно знать основы физики, в OpenStreetMap полезно знать API.</p>
  <h2>Насколько всё странно было раньше?</h2>
  <p>Изменения в четвёртой, пятой и шестой версиях <a href="https://wiki.openstreetmap.org/wiki/API#Summaries_of_editing_API_changes">перечислены в вики</a>. Вторая версия API стала работать через ссылки (REST), а не через сложные запросы XML-RPC, и в ней появились теги. В третьей версии теги вынесли из строкового атрибута xml в отдельные элементы &lt;tag>, а сегменты объединили в линии (way).</p>
  <h2>Что изменилось в версии 0.6?</h2>
  <p>Появились пакеты правок и номера версий. Раньше объекты загружали по одному, а история адресовалась только по меткам времени. Линии ограничили 2000 точками, пакеты правок — 50 тысячами объектов (позже снизили до десяти тысяч), а члены отношений стали упорядочены (раньше они возвращались в случайном порядке). Кроме того, базу данных перевели с MySQL на PostgreSQL.</p>
  <h2>Зачем понадобилось менять протокол?</h2>
  <p>В феврале 2008 года, через четыре с половиной месяца после включения API 0.5, Фредерик Рамм написал пропозал «<a href="https://wiki.openstreetmap.org/w/index.php?title=Original_Changesets_and_Reverts_Proposal_2008&oldid=89886">Пакеты правок и откаты</a>», который поднимает важную проблему проекта: отмену ошибочных правок — и выводит из неё всё, что со временем вошло в API 0.6.</p>
  <p>3-4 мая участники из Великобритании, Германии, Австрии и Нидерландов собрались в Лондоне на «Monitoring and Rollback Hack-a-thon», где обсудили новый API, составили его черновик в вики и наметали новые функции в коде.</p>
  <h2>Кто и как писал новый API?</h2>
  <p>Короткий ответ: Cloudmade. С 2007 по примерно 2010 годы эта компания была крупнейшей, связанной с OpenStreetMap. Венчурное финансирование помогло фокусироваться не на зарабатывании денег, а на развитии OSM. Среди её сотрудников были известные разработчики Стив Кост, Энди Аллан, Мэтт Эймос, Шон Макдональд и Гарри Вуд. В октябре 2008 они плотно взялись за код Rails Port, оценили фронт работ и разметили задачи. Примерно за две недели большая часть работы была сделана — но было непонятно, где её конец.</p>
  <p>Финишный рывок разработчики сделали на неделе 3-9 ноября в лондонском офисе Cloudmade. Присутствовали те же сотрудники плюс Фредерик, Ричард и Грант. Сначала на встрече «APIzza 0.6» участники договорились об окончательном виде API. Затем на хакатоне на выходных они запрограммировали тесты и оставшиеся методы. Немного допилили напильником — и в середине ноября код был готов, а Бретт выпустил Osmosis с полной поддержкой нового API. Напомним, что Osmosis до сих пор является центральным инструментом в OSM: на нём работает репликация, а в то время только эта программа умела вырезать куски из карты и фильтровать по тегам.</p>
  <h2>Как переход проходил для пользователей?</h2>
  <p>11 декабря Шон Макдональд объявил о <a href="http://web.archive.org/web/20090409124456/https://wiki.openstreetmap.org/wiki/OSM_Protocol_Version_0.6/Crowd_sourced_Testing">публичном тестировании</a> нового API. На временный сервер загрузили карту Лондона и предлагали пользоваться Potlatch, JOSM, Merkaartor и Osmosis, чтобы отловить все возможные ошибки перед окончательной миграцией.</p>
  <p>В конце января Стив Кост объявил о переходе на новый протокол через два месяца. Новая база данных должна была встать на новый сервер, но его не привезли вовремя: переход отложили на 17-20 апреля 2009 года. За день до назначенного времени Энди Робинсон предоставил официальную <a href="http://web.archive.org/web/20090425114841/http://wiki.openstreetmap.org/wiki/OSM_Protocol_Version_0.6/Information">информацию</a> о переходе. 21 апреля в 9:43 UTC Ричард Фэйрхёрст объявил: «Мы вернулись — с API 0.6, постгресом и новым сервером».</p>
  <h2>Так что, отменять правки теперь легко?</h2>
  <p>Впервые этот вопрос задали через три часа после включения API 0.6. Тогда был ответ «нет», и сейчас он тоже «нет». Но, по крайней мере, нынче есть <a href="https://forum.openstreetmap.org/viewtopic.php?id=6129">выбор инструментов</a> разной степени сложности. До кнопки отмены в интерфейсе а-ля википедия нам ещё очень далеко.</p>
  <h2>Отлично, а когда перейдём на API 0.7?</h2>
  <p>Теперь уже очевидно, что никогда.</p>
  <h2>Почему?</h2>
  <p>Протокол и модель данных достаточно хороши.  Участники проекта накидывали идеи для следующей версии <a href="https://wiki.openstreetmap.org/wiki/API_v0.7">в эту вики-страницу</a>; в её разделе «See Also» вы найдёте ещё несколько списков. Но не встретилось ничего настолько важного, что могло бы сподвигнуть сообщество взяться за написание API 0.7. Ни один человек самостоятельно не сможет улучшить протокол, потому что в OpenStreetMap слишком много несущих систем и участников, с которыми нужно договариваться.</p>
  <h2>Нельзя ли как-нибудь изменить по мелочам?</h2>
  <p>Современный API 0.6 заметно отличается от того, что мы получили десять лет назад. Примерно с 2012 года к нему начали прикручивать дополнительные функции, не затрагивающие модель данных. В мае 2012 приложения смогли узнать, какие у них права. В августе для перелицензирования в API добавили скрытие версий объектов. В апреле 2013 <a href="http://shtosm.ru/all/bez-oshibok/">появились заметки</a>. В ноябре 2014 пакетам правок <a href="http://shtosm.ru/all/plody-programmirovaniya/">добавили комментарии</a>. Последний раз протокол улучшали полгода назад, когда улучшили поиск заметок через /notes/search.</p>
  <h2>Перелицензирование?</h2>
  <p>Забавный факт: 27 февраля 2009 года, в разгар подготовки к замене сервера и протокола, рабочая группа по юридическим вопросам предоставила <a href="https://wiki.openstreetmap.org/w/index.php?title=Open_Database_License/Implementation_Plan&oldid=237497">план перехода на ODbL</a>. По нему, весь процесс должен был занять всего три-четыре месяца — именно во столько поначалу Фредерик оценивал время на переход на API 0.6.</p>


guid: 1482
url: http://shtosm.ru/all/chto-proizoydyot-zavtra-s-gps/
title: Что произойдёт завтра с GPS?
published: 2019-04-05 08:35:51
fetched: 2019-09-05 13:30:22.895034
created_at: 2019-06-09 18:12:14.083922
updated_at: 2019-09-05 13:30:22.971833
summary:
  <p>Короткий ответ — ничего.</p>
  <p>Вы уже слышали эту историю: приёмники хранят номер недели в десятибитном поле, и раз в двадцать лет оно переполняется. Завтра те устройства, которые давно не обновлялись, могут начать party like it’s 1999. Ошибка не повлияет на позиционирование, потому что внутренние вычисления всегда консистентны. Секунды и минуты тоже будут правильны. Сломаться могут только метки дня, месяца и года.</p>
  <p>GPS на сегодня даёт самое точное время, с ошибкой в пределах трёх наносекунд. Для сравнения, сетевой протокол NTP, который обновляет время в вашем компьютере, имеет точность около миллисекунды. Точность важна в GPS, потому что время используется для определения координат. Десять наносекунд — это примерно три метра ошибки. Этой точностью пользуются сейсмологи, энергетики и другие учёные.</p>
  <p>Пользовательские навигаторы, которые могут быть обновлены, уже обновлены. Обычное решение — запрограммировать дату отсечки: например, если год внезапно получается меньше 2015, то прибавить двадцать лет. Многие так и латают дыру: очевидно, этот способ только отсрочивает ошибку, что и сломало даты в некоторых навигаторах в 2016-2017 годах.</p>
  <p>Реальные проблемы это доставляет только исследователям, которые используют метки времени GPS в научных устройствах, разбросанных по Арктике и горам. Устройствах, замеряющих движение земной коры или синхронизирующих детекторы частиц. В этом году они мотаются по командировкам и срочно обновляют приёмники. Конечно, у многих есть опыт двадцатилетней давности: тогдашние проблемы кто-то решил заменой дорогих штучных устройств на более дешёвые и простые.</p>
  <p>Трёх других спутниковых систем — ГЛОНАСС, BeiDou и Galileo — проблема не касается. Всю информацию я вытащил из <a href="https://www.gps.gov/governance/advisory/meetings/2017-11/powers.pdf">этой презентации</a> 2017 года для американского совета по позиционированию и замерам времени, и из <a href="https://www.nature.com/articles/d41586-019-01048-2">позавчерашней статьи в Nature</a>.</p>


guid: 1481
url: http://shtosm.ru/all/konferenciy-psto/
title: Конференций псто
published: 2019-04-03 08:00:13
fetched: 2019-09-05 13:00:29.845247
created_at: 2019-06-09 18:12:14.086910
updated_at: 2019-09-05 13:00:29.928380
summary:
  <p>В этом году нас ждут много конференций про OpenStreetMap и картографии в целом. Готовиться к ним нужно заранее. Эти конференции сейчас ждут от вас заявок на доклады, и негоже их разочаровывать. Пройдите по списку и порадуйте организаторов каждой:</p>
  <ul>
  <li>State of the Map 2019 в Гейдельберге 21-23 сентября: главная конференция нашего проекта снова рядом, в Западной Европе — грех не съездить. <a href="https://2019.stateofthemap.org/calls/general/">Присылайте доклады</a> до 25 апреля. Лучше раньше.</li>
  <li>HOT Summit 2019 там же 19-20 сентября: главная конференция Гуманитарной команды выросла до двух дней, прикрепилась к SotM как три года назад в Брюсселе, и тоже <a href="https://summit2019.hotosm.org/call-for-proposals/">ждёт ваших историй</a>, включая «5-minute failures». Срок — до 30 апреля.</li>
  <li>FOSS4G 2019 в Бухаресте 26-30 августа: самая главная ГИС-конференция с недешёвым входным билетом (330 евро без мастер-классов) и количеством треков как на Московском вокзале. <a href="https://2019.foss4g.org/call-for-papers/general-sessions/">Присылайте заявки</a> до 15 апреля и потом помогайте выбрать триста достойнейших.</li>
  <li>State of the Map US 2019 в Миннеаполисе 5-8 сентября: крупнейшая региональная конференция про OSM уютненько втиснулась между международными. Если можете позволить себе билет в город на полпути из Чикаго в Фарго, <a href="https://sessionize.com/sotmus2019">заявляйте доклад</a> до 1 июня и ждите тёплого приёма.</li>
  <li>State of the Map France 2019 в Монпелье 14-16 июня: прибрежный французский город недалеко от Марселя в середине лета — сам по себе хороший повод для поездки, а если вы знаете язык, то встретиться с местными осмерами сам Кост велел. <a href="http://sotm2019.openstreetmap.fr/contribution.html">Вписывайтесь в докладчики</a> до 15 апреля и готовьте купальник.</li>
  <li>State of the Map Africa 2019 в Гран-Басаме, Кот-д’Ивуар 22-24 ноября: доклады они пока не собирают, но язык ожидается французский, океан тоже рядом, так что это неплохое продолжение французской осмерской конференции. Фоточку и новостные каналы <a href="https://wiki.openstreetmap.org/wiki/State_of_the_Map_Africa_2019">смотрите в вики</a>.</li>
  <li>OSCAL 2019 в Тиране, Албания 18-19 мая: ежегодная конференция касается всего открытого, и докладам про OSM <a href="https://oscal.openlabs.cc/speakers/">они тоже будут рады</a>. Тема этого года — этика в технологиях.</li>
  <li>«Цифровые геотехнологии» 20 апреля (суббота) в Петербурге: очередная регулярная встреча любителей картографии, каждая из которых имела свою направленность. В этом месяце обсудят открытые ДЗЗ: если у вас есть мысль доклада, <a href="https://spbgeotex.ru/">зарегистрируйте её на сайте</a>.</li>
  </ul>
  <p>Кроме того, прямо завтра в Москве, в четверг в 19:00 сотрудники «Урбики» бесплатно расскажут про дизайн картографических сервисов — скорее <a href="https://rambler-internet-holding.timepad.ru/event/931842/">записывайтесь</a>, таких знаний вы больше нигде не найдёте. И 16 апреля команда NextGIS покажет, что они нового придумали за полтора года с прошлого их Demo Day: <a href="http://nextgis.ru/d-day/">регистрируйтесь</a> и готовьтесь удивляться.</p>


guid: 1480
url: http://shtosm.ru/all/podkasty/
title: Картопокалипсис уже здесь
published: 2019-04-01 13:23:54
fetched: 2019-08-20 10:30:23.328416
created_at: 2019-06-09 18:12:14.089801
updated_at: 2019-08-20 10:30:23.402766
summary:
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/mapocalypse_700x300.jpg" width="821" height="352" alt="" />
  </div>
  <p>15 марта мы с Максимом Дубининым, руководителем компании <a href="http://nextgis.ru/">NextGIS</a>, и с Сергеем Голубевым, редактором великолепного блога <a href="http://городшахты.рф/">городшахты.рф</a>, запустили русскоязычный подкаст про картографию: «<a href="https://mapokalypse.podster.fm/">Картопокалипсис</a>». В каждом выпуске мы примерно час обсуждаем одну тему: уже поговорили про конференции и про открытые ГИС в России. У каждого из нас свой опыт работы и свои подходы к картографии, мы «варимся» в ней очень много лет и забыли больше, чем многие из вас когда-либо знали. Отсюда, наверное, слегка мрачноватые шутки на фоне неоправданного и тщательно скрываемого оптимизма.</p>
  <p>Это не первый русскоязычный картографический подкаст. Два года назад прервались еженедельные выпуски передачи про OpenStreetMap, «<a href="http://shtosm.ru/all/radio-osm-na-russkom/">Радио OSM</a>». Когда её запускали, типичным подкастом было «Радио-Т»: несколько друзей два часа обсуждают новости. Довольно унылый формат, замена блогам для ленивых авторов. За два с половиной года, как я записывал этот подкаст вместе с десятками соведущих из OSM, я ни разу не послушал ни выпуска других подкастов: было неинтересно. Как и со сбором данных для карты, интереснее было записывать: из дома, с балкона пляжного домика в Греции, из гостиницы перед конференцией или из латвийского леса с комарами.</p>
  <p>Но формат не стоял на месте эти годы. Когда в январе перед FOSDEM 2019 я послушал в дороге «Heavyweight», я немного офигел: качественное оформление, захватывающая история, эмоциональная подача. Ощущение было как от просмотра хорошего фильма. Но в отличие от других форматов, особенно от книг, подкасты очень удобны в дороге: не портят глаза, позволяют есть или смотреть в окно, имеют предсказуемую длительность. Вместе с восхождением сериалов, на новый уровень качества вышли подкасты, и теперь не слушать их довольно странно — так же, как не читать книг.</p>
  <p>Прослушав десяток передач на разные темы, мы поняли, что среди тем не хватает картографии. То есть, на английском языке уже выходят два подкаста: <a href="https://www.themappyisthour.com/podcast">The Mappyist Hour</a>, в котором ровно год двое ведущих болтают на отвлечённые темы и звонят гостям (но их плохо слышно). И нестареющий <a href="http://veryspatial.com/">VerySpatial</a>, которому примерно столько же лет, сколько OpenStreetMap. Последний хорош, но русскому человеку может быть непросто слушать английскую речь: Google Translate на звуковой файл не натравишь, да и темы в подкасте нередко слишком коммерческие.</p>
  <p>Теперь у нас есть приличный подкаст и на русском языке. «Картопокалипсис» выходит раз в две недели, следующий выпуск ждём 12-13 апреля. Подписывайтесь на него в <a href="https://itunes.apple.com/ru/podcast/%D0%BA%D0%B0%D1%80%D1%82%D0%BE%D0%BF%D0%BE%D0%BA%D0%B0%D0%BB%D0%B8%D0%BF%D1%81%D0%B8%D1%81/id1456377757?mt=2">iTunes</a>, <a href="https://castbox.fm/channel/%D0%9A%D0%B0%D1%80%D1%82%D0%BE%D0%BF%D0%BE%D0%BA%D0%B0%D0%BB%D0%B8%D0%BF%D1%81%D0%B8%D1%81-id2056524">Castbox</a> (лучшее приложение для Android), <a href="https://overcast.fm/itunes1456377757">Overcast</a> (приличное для iOS), либо скормите <a href="https://podster.fm/rss.xml?pid=66387">ссылку на RSS</a> вашему плееру. У нас есть план на ближайшие пару месяцев, но мы будем рады предложениям тем и отзывам на опубликованные выпуски. Пишите их в iTunes, Castbox или в телеграм-группу <a href="https://t.me/ruosm">@ruosm</a>.</p>


guid: 1479
url: http://shtosm.ru/all/pod-gnyotom-meritokratii/
title: Под гнётом меритократии
published: 2019-03-28 07:40:44
fetched: 2019-07-18 07:00:22.207321
created_at: 2019-06-09 18:12:14.092735
updated_at: 2019-07-18 07:00:22.285631
summary:
  <p>Долгое время мы считали, что в OpenStreetMap меритократия. Сегодня уже понятно, что властью в проекте обладают скорее те, кто в нём дольше: жест Стива Коста (ушёл в 2012 году) был беспрецедентным. И что меритократия — такой же миф, как «здравый смысл» или предрасположенность мужчин к математике. <a href="https://aeon.co/ideas/a-belief-in-meritocracy-is-not-only-false-its-bad-for-you">Восьмимартовская статья в Aeon</a> объясняет это коротко и доходчиво.</p>
  <p>Начинают они с того, что поверье (у 70% населения, по опросам), будто усилия приводят к процветанию, — полное заблуждение и не подтверждается практикой. При этом, заблуждаются обе стороны: бедные считают, что если бы они чуть меньше ленились и взяли, например, побольше задач, то сразу же разбогатели бы. А богатые — что они всего добились собственными умом и упорством. Разумеется, это не так, иначе бы петербуржские академики не продавали свои книжки в переходах, а миллионы на благотворительность жертвовал бы не Билл Гейтс, а программисты поспособнее.</p>
  <p>К богатству ведёт не труд, а сочетание начальных условий (родиться в богатой семье лучше, чем в бедной; в Америке — лучше, чем в Польше), предрасположенности к предпринимательству, воспитания и, конечно, удачи. Сто программистов могут придумать алгоритм, но только один продаст его условному фейсбуку — и не потому, что он самый умный. Множество талантливых и упорных людей собирают вам айфоны на китайских фабриках.</p>
  <p>Хуже того, продолжает Клифтон Марк, научные исследования показывают, что вера в меритократию откровенно вредит. Люди, убеждённые в ней, ведут себя более эгоистично, менее самокритично и допускают дискриминацию по любым признакам. Например, есть известная игра «ультиматум», когда есть условные 100$ и одному человеку нужно предложить второму такую долю, чтобы он или она согласились. Обычно первый предлагает 40-50$. Но если перед этим с ним или с ней сыграть в другую игру, убеждающую в уме или усидчивости, то вне зависимости от результата, в «ультиматуме» этот игрок предлагает другому меньшую сумму. Второй игрок, если не проходил аналогичную «подготовку», был готов на неё согласиться.</p>
  <p>Многие успешные люди не считают свои деньги и статус следствием удачи. Интересно, что если про неё напомнить, у них повышается вероятность пожертвований и в целом щедрых жестов, хотя обычно они таким людям не свойственны. Когда в компаниях провозглашают следование принципам меритократии, возрастает зарплатное неравенство между людьми разных полов или рас. Игнорируют меритократию — разница исчезает. То есть, принцип, который как будто должен устранить неравенство и воздать каждому по заслугам, приводит ровно к противоположному результату.</p>
  <p>Сообщество OpenStreetMap взращивалось на принципах меритократии, <a href="https://wiki.openstreetmap.org/w/index.php?title=Foundation/Core_Values&oldid=732254">do-ocracy</a>: они работали, когда этим сообществом была группа друзей (технари мужского пола из Западной Европы). Когда мы все присоединялись лет десять назад, мы верили: трудись — и сможешь влиять на проект. Теперь ежу понятно, что проектом рулят только те, кто успел войти в него до 2008 года и разделить ценности создателей. Только решив эту проблему, сменив политический строй, OpenStreetMap сможет повернуться лицом к пользователям и перейти на следующий уровень качества.</p>


guid: 1478
url: http://shtosm.ru/all/ot-upload-filters-k-sanity-filter/
title: От Upload Filters к Sanity Filter
updated: 2019-03-21 14:00:34
fetched: 2019-06-09 18:12:14.035276
created_at: 2019-06-09 18:12:14.095592
updated_at: 2019-06-09 18:12:14.095592
summary:
  <div class="e2-text-picture">
  <img src="http://shtosm.ru/pictures/broken_de_tiles-nq8.png" width="1056" height="453" alt="" />
  </div>
  <p>Какое главное требование к любому сервису для коммерческого использования? Не скорость, не дешевизна, не качество результата. <i>Надёжность</i>. Если сервис надёжен, то есть, гарантирует, что запрос не обвалится и не выдаст неожиданный результат когда-нибудь в будущем или один раз из тысячи, то компании готовы платить за него миллионы долларов.</p>
  <p>Пока американские картографы в Slack обсуждают, почему для Waze карту обновляют так много добровольцев (как и в России для Яндекса), немецкий картографический интернет тупо отключается, сервис за сервисом. На openstreetmap.de карта зияет чёрными тайлами. Overpass API <a href="https://lists.openstreetmap.org/pipermail/talk/2019-March/082312.html">отключен</a> до 23:00 по Москве. Выгрузки Geofabrik тоже отключены. Вместо всей немецкой википедии — <a href="https://de.wikipedia.org/wiki/Wikipedia:Hauptseite">чёрная страница</a> с горой длинных слов. Если вы использовали хоть один из этих сервисов, вы теперь знаете, что такое надёжность открытого проекта.</p>
  <p>Да, тринадцатая статья, за которую во вторник будет голосовать Европарламент, <a href="https://blog.openstreetmap.org/2019/03/21/the-eu-copyright-directive-threatens-openstreetmap/">сломает нам весь интернет</a>. Контент-проектам типа OSM и Википедии придётся вводить сложные системы цензурирования контента, чтобы не платить штрафы за нарушение авторского права. Потому что отвечать за это будут не пользователи их данных, как сейчас, а сами проекты. Этот законопроект подобен таким же идиотским проектам русского правительства. Если бы я жил в Европе, я бы вышел 23 марта на улицу вместе со всеми. Но сейчас ни я, ни люди из США, Африки, Азии, России не могут пользоваться сервисами, и почему меня должны касаться проблемы белых людей?</p>
  <p>Реакция на 13 статью подобна русским контрсанкциям: назло бабушке отморожу сервер. Парламентарии не заметят, а тысячи пользователей со всего мира поймут, что этим немцам (которые до сих пор рулят во всех направляющих дискуссиях в OpenStreetMap) доверять нельзя, какие-то они нервные и не думают о других. Лучше пойти на сервисы, которые обеспечивают надёжность, пусть и не такие дешёвые.</p>
  <p>Проблема только в том, что адекватных платных сервисов на базе OpenStreetMap нет. Mapbox приближается к этой планке, но от вида их картостиля хочется биться головой о карту на столе, а качество геокодера заставляет лить слёзы. И мы не говорим о свежести данных ещё: известно же, что слепок OSM — это не OSM. И это всё: остальные не тянут объёма запросов и кастомизацию. Хороши только пользовательские продукты, отвязанные от корневых серверов: тот же Maps.Me или <a href="https://cycle.travel/">Cycle.Travel</a>.</p>
  <p>Вот и получается: почему обычные пользователи правят не OSM, а проприетарные карты? Потому что они с ними работают. Почему они работают не с открытой альтернативой? Потому что если бы компании, производящие продукт, её выбрали, то потом бы ежедневно обливались слезами. Открытым данным доверять нельзя — и не потому, что они открытые (это, как раз, хорошо), а потому что вокруг них вырастают ненадёжные, часто откровенно вредящие сервисы. Проще пожертвовать источником, чем пытаться их исправить.</p>


guid: http://scanningpages.wordpress.com/?p=622
url: https://scanningpages.wordpress.com/2013/06/13/state-of-the-map-us-2013/
title: State Of The Map US 2013
published: 2013-06-13 03:19:05
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.532386
updated_at: 2019-10-26 23:00:23.604064
summary:
  I am sitting on a plane flying eastbound at 37,000 feet over middle of America. A few hours ago we left San Francisco the host city for State Of The Map US 2013. State of the Map US is the American regional conference for OpenStreetMap participants and enthusiasts. A common theme for the weekends events [&#8230;]

content:
  <div data-shortcode="caption" id="attachment_628" style="width: 310px" class="wp-caption aligncenter"><a href="https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg"><img aria-describedby="caption-attachment-628" data-attachment-id="628" data-permalink="https://scanningpages.wordpress.com/2013/06/13/state-of-the-map-us-2013/code_sprint/" data-orig-file="https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg" data-orig-size="1500,1004" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;9&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;NIKON D3000&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;1370886418&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;18&quot;,&quot;iso&quot;:&quot;400&quot;,&quot;shutter_speed&quot;:&quot;0.05&quot;,&quot;title&quot;:&quot;&quot;}" data-image-title="code_sprint" data-image-description="&lt;p&gt;Developers working hard at the SOTMUS 2013 code sprint&lt;/p&gt;
  " data-medium-file="https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg?w=300" data-large-file="https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg?w=614" src="https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg?w=300&#038;h=200" alt="Code Sprint Photo" width="300" height="200" class="size-medium wp-image-628" srcset="https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg?w=300&amp;h=200 300w, https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg?w=598&amp;h=400 598w, https://scanningpages.files.wordpress.com/2013/06/code_sprint1.jpg?w=150&amp;h=100 150w" sizes="(max-width: 300px) 100vw, 300px" /></a><p id="caption-attachment-628" class="wp-caption-text">OSM hard at work</p></div>
  <p>I am sitting on a plane flying eastbound at 37,000 feet over middle of America.  A few hours ago we left San Francisco the host city for <a href="http://www.stateofthemap.us" target="_blank">State Of The Map US 2013</a>.  State of the Map US is the American regional conference for OpenStreetMap participants and enthusiasts.  A common theme for the weekends events was building community.</p>
  <p><span id="more-622"></span></p>
  <p>I go to open-source conferences so that I can spend time informally talking with other members of my open source community.  In San Francisco this week I was able to meet OpenStreetMap celebrities such as Richard Fairhurst (author of Potlach) and Frederick Ramm (former JOSM maintainer, Geofabrick founder and OSMF board member).  I also spent time informally chatting with dozens of people who attended the conference to learn more about OpenStreetMap.   Many people I spoke with told me they were interested in OpenStreetMap and were looking into using the data but had only done a small amount of mapping.</p>
  <p>Conferences typically have three sets of activities the most popular of these are the sessions.  My favorite sessions of this conference was the session Ryan Closner gave on on using Voxel.js to render OpenStreetMap data and a set of sessions on who is doing mapping.  Voxel.js is a open-source clone of Minecraft written in Javascript.  The presenter has built software that allows your character to &#8216;walk through&#8217; a world consisting of real streets and building locations through OpenStreetMap data.  My other favorite session was the track on community analysis.  Richard Fairhurst, Alan McConchie, and Martjin van Exel presented a set of very related talks on who is doing the mapping. I hope to use some of their techniques and code to perform an analysis of activities in the Canadian community.</p>
  <p>The second conference track is the hall track where everyone talks informally between sessions in the hall or conference while waiting for a session to start.  I find that I get the most out of a conference by making a point of saying hi to the person sitting next to me while waiting for a session to start.  I might ask them where they are from, why they are at the conference and what types of things they have mapped.  I learned that there are many people new to the OpenStreetMap community.  I have been adding to data to OpenStreetMap for almost 5 years and have spoken on OpenStreetMap at at least 3 conferences and numerous mapping parties and hack days.  This was the first OpenStreetMap conference for many of the people I spoke with. Many of them are coming either from a traditional geography background or web background where they are starting to use OpenStreetMap for stuff.  As a community we,  need to keep these people engaged as regular casual mappers to make sure that they help us keep the map strong. I spoke with many people that we were able to convince too spend their weekend at a OpenStreetMap conference but we haven&#8217;t yet been able to convince them to spend their weekend adding to the map. How do we convince these people to occasionally take half a day and spend it on improving the map?</p>
  <p>We also need to make sure that they can easily use our data. We need to reach out and build bridges with communities that can use our map. A group of us met in a BOF to discuss improving the diversity of the people involved in OpenStreetMap. We agreed to create a <a href="http://lists.openstreetmap.org/listinfo/diversity-talk" target="_blank">diversity-talk</a> mailing list to discuss ways we can expand the OpenStreetMap community.   I hope to spend some time thinking of ways I can grow our community by making inroads into other established communities.</p>
  <p>The third conference track I attended was the party track.  I attended three parties in San Francisco. There was a party Saturday night at The Raven a popular bar on Mission in San Francisco.  On Sunday, Stamen design welcomed the OpenStreetMap world into their office for a party. The Stamen office is filled with cool maps. Monday night Mapbox also hosted a small party at a pub following the code-sprint.  I missed the Code for America party on Friday but I don&#8217;t doubt that was fun.  Eating and drinking with other mappers allows you to get to know them in a more relaxed atmosphere.  I find that it is much easier to ask someone for help or their honest opinion when you&#8217;ve spent time socializing with them.</p>
  <p>I want to thank OpenStreetMap.US and the sponsors for organizing this conference and all of the folks from Europe and Asia (I met people who flew in from Japan and New Zealand). </p>


guid: http://scanningpages.wordpress.com/?p=589
url: https://scanningpages.wordpress.com/2013/03/13/maproulette-hack-weekend/
title: Maproulette Hack Weekend
published: 2013-03-13 02:11:29
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.535596
updated_at: 2019-10-26 23:00:23.606905
summary:
  This weekend we had the second annual Toronto OpenStreetMap developer weekend. The nice folks at the Ryerson Department of Geography hosted us. My focus this weekend was to work the Serge and Martijn on maproulette Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark [&#8230;]

content:
  <p>This weekend we had the second annual <a href="http://wiki.openstreetmap.org/wiki/Toronto_Hack_Weekend_March_2013">Toronto OpenStreetMap developer weekend</a>.   The nice folks at the <a href="http://www.ryerson.ca/geography">Ryerson Department of Geography</a> hosted us.  My focus this weekend was to work the <a href="http://wiki.openstreetmap.org/wiki/User:Emacsen">Serge</a> and <a href="http://oegeo.wordpress.com/martijnvanexel/">Martijn</a> on <a href="http://www.maproulette.org">maproulette</a></p>
  <p>Maproulette is software that presents an easy to do mapping task to users which they can complete and then mark the task as completed.  Examples of past maproulette mapping challenges include fixing connectivity errors or fixing objects touched by the license change.<br />
  <span id="more-589"></span><br />
  Serge and Martijn wanted to redesign parts of maproulette to make it easier develop more challenge modules.  I am going to discuss some of the design decisions we debated during the weekend.</p>
  <p><strong>Challenges</strong><br />
  A Challenge is a type of mapping activity that map roulette can assign to users.   Examples of challenges might be &#8216;Add bridges to roads with where we suspect a bridge exists&#8217;  or &#8216;Add the street address tags to restaurants that don&#8217;t have addresses&#8217;. </p>
  <p><strong>Tasks</strong><br />
  A task is a specific work item within a challenge.  In the &#8216;Add bridges to roads&#8217; challenge a task might be to add a bridge to a specific intersection (or determine that bridge should be there)</p>
  <p>Maproulette needs to provide a web based interface that displays challenge tasks to users(task assignments).  This requires us to somehow store information about the tasks that make up a challenge.  Challenge specific modules will compute tasks (ie find missing bridges).  Figuring out how the web interface and the programs that find challenge tasks should be connected was one of the questions we discussed this weekend.</p>
  <p><strong>Proposal 1: Discrete Modules</strong></p>
  <p>Each challenge has a challenge module that supports operations such as </p>
  <ul>
  <li> Give me some available tasks near this lat/lon</li>
  <li> Mark this task as assigned to a user </li>
  <li>Mark this task as complete/skipped</li>
  </ul>
  <p>Each challenge module would be responsible for storing task data in it&#8217;s own<br />
  format (or using database tables specific to the module).  The user&#8217;s web-browser would talk to a common front-end component that would invoke the individual challenge modules.  The challenge modules would return HTML/Javascript/JSON suitable that the browser would use in displaying information about the task to the user. </p>
  <p><strong>Proposal 2: Shared Task Representation</strong></p>
  <p>Challenge modules compute a list of tasks by examining a mixture of OSM and external data sources.  The challenge modules then convert these tasks into a common representation made up of a task and associated sets of task_objects.  A task to find add a missing bridge might be made up of three objects. An object for the place we think the bridge belongs, an object for the road the bridge is part of and a third object for the road/stream that the bridge crosses over.  These tasks and associated objects get stored in a database.  Tasks and associated objects for different challenges get stored in the same pair of database tables.  The web application that users interact with can then display and manipulate information about tasks from all challenges.<br />
  <strong><br />
  Contrasting the options<br />
  </strong><br />
  When designing software systems you often have two or more different approaches to a given problem.  Deciding which option to proceed with is the challenge software designers have to face.  Some of the questions I asked myself this weekend included</p>
  <ul>
  <li><em>How can code between different challenges be shared or re-used in each of the options? </em>.  The second approach sounds like it has more places where code can be re-used because all challenges use the same data model for assigning tasks</li>
  <li><em>Is one approach more flexible than the other? Are there things different challenges might deal with differently that will be hard to accommodate with one of the designs? </em> Our concern with the second proposal was that we were generalizing all challenges to object + geometry and not much else.  What if some challenges needed to display extra things such as textual hints?</li>
  <li><em>How do the proposals make it easier or harder for someone to come along and develop a new challenge?</em>   In the first approach a challenge developer can has free reign to implement things in any fashion they like as long as they implement the task assignment API I discuss above.  In the second proposal challenge authors can still pick their programming language and methods for detecting challenges but they must use the database schema we provide for storing information about tasks.  With the second approach challenge developers only need to worry about writing programs to find challenge tasks and store them in the database. Task assignment/management is handled by the common code</li>
  </ul>
  <p><strong><br />
  A third proposal<br />
  </strong><br />
  We were still debating the merits of these different approaches on  late-morning on Sunday when we decided to take a break and go for lunch.  One would think that a group of mappers would know lots of places nearby for lunch in downtown Toronto but we seemed to be wondering aimlessly back and forth down the street.   While we were doing this we came up with the idea of a shared database schema approach like proposal 2 but letting each challenge store a blob of  JSON with the task.  Each challenge would also implement challenge specific rendering/display could that would know how to take the JSON describing the task and display information about the task to the user in a meaning full way.</p>
  <p>I want to thank <a href="http://weait.com">Richard Weait</a> for organizing this weekend and Serge and Martijn for letting me participate in the design process.  Next we need to finish implementing what we discussed.  This should help us figure out if our ideas were good ideas.</p>


guid: http://scanningpages.wordpress.com/?p=576
url: https://scanningpages.wordpress.com/2012/12/29/python-urllib2-cookies-and-openstreetmap/
title: Python urllib2, cookies and OpenStreetMap
published: 2012-12-29 06:00:36
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.538245
updated_at: 2019-10-26 23:00:23.609455
summary:
  I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly. I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website. The first step is to setup a urllib2 opener instance that is configured [&#8230;]

content:
  <p>I found the documentation on urllib2 a bit unclear about how to get cookie handling working properly.</p>
  <p>I was working on a python script that needed to contact the OpenStreetMap web server, login with my OSM credentials and interact with the website.</p>
  <p>The first step is to setup a urllib2 opener instance that is configured to store cookies.</p>
  <p><code></p>
  <pre>
  import cookielib,Cookie,urllib2,urllib
  import xml.etree.cElementTree as ElementTree

  cookies = cookielib.CookieJar()
  opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies))
  </pre>
  <p></code></p>
  <p>This will create an opener that can be used to retrieve URL&#8217;s.  Any cookies set in the HTTP response will be stored in the cookie&#8217;s cookie jar.  If I needed to add additional openers (ie for special redirect handling) I would just add them as additional parameters to the build_opener call.   ie urllib2.build_opener(handler1, handler2,handler3)&#8230;</p>
  <p>Next we need to contact OpenStreetMap to get a blank login form.  The blank login screen has a hidden variable &#8216;authenticity_token&#8217; that needs to be passed back as part of the POST with my login credentials<br />
   <code></p>
  <pre>
  inptag = '{http://www.w3.org/1999/xhtml}input'
  formtag = '{http://www.w3.org/1999/xhtml}form'
  # fetch the blank login form
  response_tokenfetch = opener.open(request)
  html = response_tokenfetch.read()
  htmlfile=StringIO.StringIO(html)
  # parse the HTML elements in the form
  # extract any input fields for later resubmission
  # this will pick up the authenticity_token and anything else
  xml_tree = ElementTree.parse(htmlfile)
  for form in xml_tree.getiterator(formtag):                
      for field in form.getiterator(inptag):
          if 'name' in field.attrib and 'value' in field.attrib:
                  login_payload[field.attrib['name']] = field.attrib['value']
  login_payload['username'] = username
  login_payload['password'] = password
  login_payload['remember_me'] = 'yes'
  login_payload['cookie_test'] = 'true'
  </pre>
  <p></code></p>
  <p>Next we submit the LOGIN request as a POST.  Any session cookies returned as part of the blank form will be added to the second request.<br />
  <code></p>
  <pre>
  cookies.add_cookie_header(request)
  response = opener.open(request,urllib.urlencode(login_payload))
  </pre>
  <p></code></p>
  <p>If our login was successful then cookies contains an _osm_session and _osm_username that will be used in subsequent API calls.</p>
  <p><code></p>
  <pre>
  request2=urllib2.Request('http://api06.dev.openstreetmap.org/user/stevens/inbox)
  cookies.add_cookie_header(request2)
  response2 = opener.open(request2)
  html=response2.read()
  </pre>
  <p></code></p>
  <p>You could then parse the HTML to extract a list of messages.<br />
  If your using the formal OpenStreetMap API (ie calls under /api/0.6/&#8230;) then you should instead use oauth for authentication instead of logging in through the website.  Some OSM features such as messaging can only be accessed by pretending to be a web session and parsing/faking HTML.</p>


guid: http://scanningpages.wordpress.com/?p=549
url: https://scanningpages.wordpress.com/2012/08/16/learning-from-mistakes/
title: Learning from mistakes
published: 2012-08-16 23:38:02
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.540589
updated_at: 2019-10-26 23:00:23.611744
summary:
  Last year I attended and wrote about FOSS4G 2011 in Denver. The 2012 FOSS4G event was supposed to take place in Beijing this September. Last month OSGEO announced that the conference was cancelled. Cameron Shorter has posted a detailed analysis on how the conference fell apart. His post does a great job of looking at [&#8230;]

content:
  <p>Last year I attended and wrote about <a href="https://scanningpages.wordpress.com/2011/09/17/big-tent-conferences/">FOSS4G 2011</a> in Denver.  The 2012 FOSS4G event was supposed to take place in Beijing this September.  Last month OSGEO announced that the conference was cancelled.<br />
  <a href="http://cameronshorter.blogspot.ca/">Cameron Shorter</a> has posted a <a href="http://lists.osgeo.org/pipermail/conference_dev/2012-August/001886.html">detailed analysis</a> on how the conference fell apart.  His post does a great job of looking at the sequence of events that lead up to the current situation.</p>
  <p><span id="more-549"></span></p>
  <p>In the corporate world when an event, company or product falls on its face the detailed reasons are hushed up and rarely get a public airing.  When the government screws up sometimes we get an expensive public inquiry where lawyers argue about how someone else did or didn&#8217;t screw up.  OSGEO has a tradition of building consensus and discussing non-technical issues on their mailing lists.  Good open-source projects not only be give us amazing software but they allow us to learn valuable lessons on public relations, finances and management by letting us watch their inner-workings.</p>
  <p>When I was learning too fly airplanes I made a point of regularly reading the investigation reports of plane crashes. A common theme was that a single mistake didn&#8217;t cause the accident but a chain of mistakes, malfunctions and bad luck all added up to the crash. Common thinking in the aviation community is that if you learn about the mistakes others have made then you will be more likely to avoid making the same mistakes.  The story Cameron tells is about limited options, ignoring red flags, communication break downs and bad luck.  These are all issues we see everyday across all aspects of life.</p>
  <p>The mailing list discussion about the lessons that can be learned for future FOSS4G events is still on-going but as outsiders we can follow their discussion and try to learn from their mistakes.  When our events, initiatives or products don&#8217;t succeed we should also conduct a carefully look at how we got to that state and what things, with 20/20 hindsight, we should have paid more attention too.</p>


guid: http://scanningpages.wordpress.com/?p=515
url: https://scanningpages.wordpress.com/2012/05/22/pgcon2012-making-maps/
title: PGCon2012 – Making Maps
published: 2012-05-22 01:16:34
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.543528
updated_at: 2019-10-26 23:00:23.614534
summary:
  Slides from my 2012 PGCon presentation Making your Own Maps are now available The presentation covered Common reasons people render their own maps Where to get OpenStreetMap data and how to load it into your PostGIS database How to use Tilemill to design your own map style How to render map tiles, both statically and [&#8230;]

content:
  <p>Slides from my 2012 PGCon presentation <a href="http://www.pgcon.org/2012/schedule/events/432.en.html" target="_blank"><em>Making your Own Maps</em></a> are now  <a href="https://scanningpages.files.wordpress.com/2012/05/pgcon_makingmaps2012.pdf">available</a></p>
  <p><a href="https://scanningpages.files.wordpress.com/2012/05/pgcon_makingmaps2012.pdf"><img data-attachment-id="516" data-permalink="https://scanningpages.wordpress.com/2012/05/22/pgcon2012-making-maps/pgcon2012/" data-orig-file="https://scanningpages.files.wordpress.com/2012/05/pgcon2012.png" data-orig-size="1058,794" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;}" data-image-title="pgcon2012" data-image-description="" data-medium-file="https://scanningpages.files.wordpress.com/2012/05/pgcon2012.png?w=300" data-large-file="https://scanningpages.files.wordpress.com/2012/05/pgcon2012.png?w=614" src="https://scanningpages.files.wordpress.com/2012/05/pgcon2012.png?w=300&#038;h=225" alt="" title="pgcon2012" width="300" height="225" class="aligncenter size-medium wp-image-516" srcset="https://scanningpages.files.wordpress.com/2012/05/pgcon2012.png?w=300&amp;h=225 300w, https://scanningpages.files.wordpress.com/2012/05/pgcon2012.png?w=600&amp;h=450 600w, https://scanningpages.files.wordpress.com/2012/05/pgcon2012.png?w=150&amp;h=113 150w" sizes="(max-width: 300px) 100vw, 300px" /></a></p>
  <p>The presentation covered</p>
  <ul>
  <li>Common reasons people render their own maps</li>
  <li>Where to get OpenStreetMap data and how to load it into your PostGIS database</li>
  <li>How to use Tilemill to design your own map style</li>
  <li>How to render map tiles, both statically and dynamically </li>
  <li>How to use OpenLayers to display your map</li>
  </ul>
  <p>The presentation was recorded.  I will update this post when the recording comes online.</p>
  <p>I&#8217;ve been a regular attendee of PGCON since the first year it was held in Ottawa.  Like past years I enjoyed the conference and I would like to thank Dan Langille for putting together another first-rate conference.  My favourite part of PostgreSQL conferences is meeting and reconnecting with users people in the community.   </p>
  <p>Updated: The video is available <a href="http://www.youtube.com/watch?v=yctOoHoiQJ4">here</a></p>


guid: http://scanningpages.wordpress.com/?p=501
url: https://scanningpages.wordpress.com/2012/05/06/penlake-map-may-2012/
title: Penlake Map May 2012
published: 2012-05-06 18:53:50
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.545953
updated_at: 2019-10-26 23:00:23.616841
summary:
  I have published a new version of the Peninsula Lake map. This edition includes new cartography and map updates based on my survey work last summer. I had written the previous map style in Cascadnik, a CSS like language that translates to mapnik XML. Cascadnik has since been replaced by Carto so I have decided [&#8230;]

content:
  <div data-shortcode="caption" id="attachment_508" style="width: 310px" class="wp-caption aligncenter"><a href="https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05.png"><img aria-describedby="caption-attachment-508" data-attachment-id="508" data-permalink="https://scanningpages.wordpress.com/2012/05/06/penlake-map-may-2012/penlake-2012-05-small/" data-orig-file="https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05-small1.png" data-orig-size="640,354" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;}" data-image-title="penlake.2012.05.small" data-image-description="&lt;p&gt;The May 2012 edition of the Peninsula lake map.&lt;/p&gt;
  " data-medium-file="https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05-small1.png?w=300" data-large-file="https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05-small1.png?w=614" src="https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05-small1.png?w=300&#038;h=165" alt="Peninsula Lake Map" title="penlake.2012.05.small" width="300" height="165" class="size-medium wp-image-508" srcset="https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05-small1.png?w=298&amp;h=165 298w, https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05-small1.png?w=597&amp;h=330 597w, https://scanningpages.files.wordpress.com/2012/05/penlake-2012-05-small1.png?w=150&amp;h=83 150w" sizes="(max-width: 300px) 100vw, 300px" /></a><p id="caption-attachment-508" class="wp-caption-text">Peninsula Lake, Huntsville Ontario</p></div>
  <p>I have published a new version of the Peninsula Lake map.  This edition includes new cartography and map updates based on my survey work last summer. </p>
  <p>I had written the previous map style in Cascadnik, a CSS like language that translates to mapnik XML.   Cascadnik has since been replaced by Carto so I have decided to rewrite the map style using <a href="http://mapbox.com/tilemill" target="_blank">TileMill</a> (Map design software that uses Carto and Mapnik). </p>
  <p>One of my goals while redesigning the map was to put to use what I have learned about cartography over the past year.  The original PenLake map style was based on a sample Cascadnik style adapted to my colour scheme, nautical features and a print scale.   The new style was designed by me from scratch to focus on</p>
  <ul>
  <li>The lake and nearby rivers, streams and marshes</li>
  <li>Recreational attractions around the lake including hotels and golf courses</li>
  <li>Roads near the lake</li>
  <li>The water depth of the lake including suspected rocks</li>
  </ul>
  <p>This style is much smaller than the previous one ( the style is less than 300 lines of CSS) and it behaves well at different zoom levels between 12 and 18.   I suspect the number of layers can still be further reduced for faster rendering times.</p>
  <p>I have made a web version of the map available on <a href="http://www.mapexplorer.org/penlakemap" title="Peninsula Lake Map" target="_blank">Peninsula Lake Map</a> my map <a href="http://www.mapexplorer.org">hosting site</a>.  The map tiles are currently being served by <a href="http://www.mapbox.com" title="MapBox" target="_blank">MapBox.</a></p>
  <p>Some of the water depth data in this version of the map is based on observations I manually made while kayaking on the lake in the summer of 2011.   I have tried to correct this data to match the sounding datum used by the CHS in their observations.  I don&#8217;t expect the observations taken by me with a kayak, rope and a tape measure to match the accuracy of the CHS with their fancy sonar but we have to work with the equipment that we have available to us.  The depths on the map should be based on a datum of 913 feet.</p>
  <p>The Source code is still available from <a href="http://github.com/ssinger/penlakemap" target="_blank">github</a></p>


guid: http://scanningpages.wordpress.com/?p=485
url: https://scanningpages.wordpress.com/2012/03/18/osm-hack-weekend-toronto-2012/
title: OSM Hack Weekend (Toronto 2012)
published: 2012-03-18 15:34:02
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.548271
updated_at: 2019-10-26 23:00:23.619171
summary:
  The Toronto OSM hack weekend (2012) started on friday with an introduction to OpenStreetMap presentation at the Ryerson Geography department. Experienced OpenStreetMap presenter, Richard Weait gave a talk to a room of Geography students, staff and faculty. After Richard explained the basics of OpenStreetMap A.J. and Tristen gave a short presentation on some of the [&#8230;]

content:
  <p>The <a href="http://wiki.openstreetmap.org/wiki/Toronto_Hack_Weekend_March_2012">Toronto OSM hack weekend (2012)</a> started on friday with an introduction to OpenStreetMap presentation at the <a href="http://www.ryerson.ca/geography/">Ryerson Geography</a> department. Experienced OpenStreetMap presenter, <a href="http://wiki.openstreetmap.org/wiki/User:Rw">Richard Weait</a> gave a talk to a room of Geography students, staff and faculty.  After Richard explained the basics of OpenStreetMap A.J. and Tristen gave a short presentation on some of the work that they have been doing for <a href="http://www.mapbox.com">MapBox</a>.</p>
  <p>After the presentations we did a QA session with the audience.  The OSM &#8216;<em>experts</em>&#8216; in the room tried to answer the questions from the audience.  Most people in the audience were geographers and GIS people who had already heard of OSM.  A number of the attendees already had worked with various components in the OSM stack. We had questions on projections, layers, TileMill, Mapnick, tagging and an assortment of other topics.</p>
  <p>After the Q/A  I went with A.J and Tristen to Chiplote for some Burritos. At Chipotle we learned three things. 1) Chipotle in Washington does more of the meat preperation in the back room than they do in Toronto; Toronto should take a hint.  2) Keeping the Burrito in the tinfoil wrap it in keeps it from falling apart as you eat it.  3) Next time I need to remember to bring Buritos back to my fellow hackers. Friday night we had a social at the hack weekend social HQ, there was plenty of beer and the room was packed.</p>
  <p>Saturday morning I took the train back downtown for the first day of hacking. We had local OSM developers and out of town hackers from three different countries participate along with a group of Ryerson Geography students.  I spent some time Saturday morning helping someone get the <a href="http://wiki.openstreetmap.org/wiki/Rails_port">rails port</a> working on their OSX laptop.   The part they seemed to have the most difficulty with was getting PostgresSQL running.  This tells me two things. First that the work people in the OSM community have done to make the rails port easier to install has paid off and some PostgreSQL binary distributions could make their user experience a bit better.</p>
  <p>I had suggested the EDB one-click installer for OSX but they opt&#8217;d for the <a href="http://www.kyngchaos.com/">KyngChao</a> binaries.  They were okay installing the package but they weren&#8217;t clear how to startup PostgreSQL and connect to it.  I feel that the EDB installer makes it easier for an inexperienced PostgreSQL user to get up and running.</p>
  <p>In addition to helping people with installing PostgreSQL I also helped some people with SQL queries and helped with changes to the website look cleaner for users logged in but haven&#8217;t uploaded any GPX traces. </p>
  <p>Saturday night we had another party at social HQ. Party highlights included beer with ingredients from seven continents, comparision of political systems, talk of election rigging. Thai food, wine and beer were also spotted. Most of the geography students from the hack sessions were able to make it out and saw how fun OSM parties can be.  I hope to see them at the regular Toronto OSM pub nights.  </p>
  <p>The <a href="http://www.gotransit.ca">GO train</a> back to Oakville was pretty full. This was St. Patricks day and the streets of Toronto were filled with drunk people in green.  The 11:43pm train to the suburbs is a bit too early for serious leprecons.   Many of the riders had been at a Van Halen concert, it didn&#8217;t seem like many of them were interested in making maps.</p>
  <p>There is a saying in the OpenStreetMap world that if you want to build an local OpenStreetMap community you should import a bunch of Germans.  If you can&#8217;t import some Germans the next best thing is to import Richard Weait.   He almost single-handedly organized the Toronto hack weekend, secured a venue and provided food, drinks and social entertainment.  I also want to thank the Ryerson Geography department and <a href="http://www.ryerson.ca/~crinner/">Claus Rinner </a>for letting us use their facilities and providing eager GIS students.</p>


guid: http://scanningpages.wordpress.com/?p=421
url: https://scanningpages.wordpress.com/2011/11/17/virgin-roads/
title: Virgin Roads
published: 2011-11-17 03:43:10
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.550730
updated_at: 2019-10-26 23:00:23.621623
summary:
  The neighborhood I live in is between 20 and 30 years old, other parts of Oakville are under constant growth. When a new sub-division is built the developer will put in sewers, roads and start building houses. Sometimes the street signs will look home-made other times proper street signs will already be up. When I [&#8230;]

content:
  <p><a href="https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg"><img data-attachment-id="423" data-permalink="https://scanningpages.wordpress.com/2011/11/17/virgin-roads/dsc_0400-1/" data-orig-file="https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg" data-orig-size="640,429" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;}" data-image-title="DSC_0400.1" data-image-description="&lt;p&gt;Some houses under construction in Oakville&lt;/p&gt;
  " data-medium-file="https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg?w=300" data-large-file="https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg?w=614" src="https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg?w=300&#038;h=201" alt="" title="DSC_0400.1" width="300" height="201" class="aligncenter size-medium wp-image-423" srcset="https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg?w=300&amp;h=201 300w, https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg?w=600&amp;h=402 600w, https://scanningpages.files.wordpress.com/2011/11/dsc_0400-1.jpg?w=150&amp;h=101 150w" sizes="(max-width: 300px) 100vw, 300px" /></a></p>
  <p>The neighborhood I live in is between 20 and 30 years old, other parts of Oakville are under constant growth. When a new sub-division is built the developer will put in sewers, roads and start building houses. Sometimes the street signs will look home-made other times proper street signs will already be up.</p>
  <p>When I see a new subdivision going up near my I try drive (or walk) the roads with my GPS  capturing the data for OpenStreetMap. Often my GPS traces in OpenStreetMap will be the first time these roads show up on a map outside of the towns GIS. It might take years before some of the commercial maps send trucks to map these roads.</p>
  <p>On Sunday I was driving past a sub-division that was being built and decided to <a title="map" href="http://www.openstreetmap.org/?lat=43.46939&amp;lon=-79.74569&amp;zoom=16&amp;layers=M" target="_blank">map</a> some of the roads. I got about GPS traces and names for about half of the roads. I need to make a point of going back and finishing the soon. Sub-division construction sites usually have construction workers and heavy equipment moving about.  It is important to pay close attention to your surroundings and the condition of the road your mapping often it is a work in progress. </p>
  <p>I doubt anyone is going to be moving into these houses until the spring. In the spring when people move into their new houses they will be able to use OpenStreetMap to invite there friends to the house-warming party.</p>


guid: http://scanningpages.wordpress.com/?p=413
url: https://scanningpages.wordpress.com/2011/09/18/postgis-replication-foss4g/
title: PostGIS Replication @ FOSS4G
published: 2011-09-18 21:42:23
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.553064
updated_at: 2019-10-26 23:00:23.623834
summary:
  My talk on PostGIS replication at FOSS4G 2011 went well. It looked like there were about 150 people in the room. Most of them had not yet deployed a PostGIS replication solution. My talk covered Slony and streaming replication. It gave an overview of different replication patterns that can crop on in the GIS space. [&#8230;]

content:
  <p>My talk on PostGIS replication at FOSS4G 2011 went well.  It looked like there were about 150 people in the room.  Most of them had not yet deployed a PostGIS replication solution.</p>
  <p>My talk covered Slony and streaming replication.  It gave an overview of different replication patterns that can crop on in the GIS space.  I then gave an overview of the key features and limitation of Slony and streaming replication.</p>
  <p>A video of the talk is available at <a href="http://www.fosslc.org/drupal/content/postgresql-replication-0">FOSSLC</a></p>
  <p>My slides are available <a href='https://scanningpages.files.wordpress.com/2011/09/postgis_replication_foss4g_2011.pdf'>here</a></p>


guid: http://scanningpages.wordpress.com/?p=401
url: https://scanningpages.wordpress.com/2011/09/17/big-tent-conferences/
title: Big Tent Conferences
published: 2011-09-17 22:49:50
fetched: 2019-10-26 23:00:23.545561
created_at: 2019-06-09 18:12:14.555397
updated_at: 2019-10-26 23:00:23.626104
summary:
  This past week I attended FOSS4G in Denver, a conference run by the open-source geospatial foundation (OSGEO) that also happened to be the largest PostgreSQL conference in North America. FOSS4G is a big tent conference that attracted about 900 attendees from all over the world with over 400 of them from the United States, over [&#8230;]

content:
  <p>This past week I attended <a href="http://2011.foss4g.org" target="_blank">FOSS4G</a> in Denver, a conference run by the open-source geospatial foundation (<a href="http://www.osgeo.org" target="_blank">OSGEO</a>) that also happened to be the largest PostgreSQL conference in North America.</p>
  <p>FOSS4G is a big tent conference that attracted about 900 attendees from all over the world with over 400 of them from the United States, over 50 from Canada.<br />
  A big tent conference is a conference is similar to a big-tent political party.  The idea is appeal to a broad base of people by catering to as many groups as possible while keeping true to some key common themes.  Out of the 900 people at the conference I&#8217;d say 75% raised their hands when asked if it was their first FOSS4G.   Last years FOSS4G was held in Europe and also had over 800 attendees.<br />
  <span id="more-401"></span><br />
  At FOSS4G the common theme is open source geo-spatial software.  This includes open-source desktop GIS systems like QGIS and GRASS.  It includes open source map rendering platforms like Mapserver and Mapnik.  It also includes open source spatial databases such as PostGIS and MongoDB.   The tent is also big enough to include commercial vendors such as ESRI, Safe(authors of FME) who produce commercial products that interface with some open-source geospartial software.   Many of the software packages (open-source and commercial) represented at the conference compete with each other but they are able to share the same tent.   At the opening reception one attendee (who I won&#8217;t name) said <em>&#8220;I&#8217;m need to find Chris Schmidt and tell him why OpenLayers sucks</em>&#8221;  conferences that get everyone under one roof for a week allow us to learn from each other so we can improve our software, our documentation and our communities.  It is much harder to have a flame-war on a mailing list with someone you have sat down and had a beer with than it is to have a flame-war with someone you have never met.</p>
  <p>At FOSS4G almost everyone I spoke to used PostGIS in some fashion, some of them used couchDB or MongoDB for tasks, others had legacy Oracle Spatial or ESRI systems but everyone was aware of PostGIS and most had used it in some fashion.   If you are running PostGIS then you are running PostgreSQL. FOSS4G 2011 was the largest collection of PostgreSQL users under one roof in North America to date. As a member of the PostgreSQL community I feel we need to do a better job at making sure that geospatial types feel welcome in our community.   Excluding the core PostGIS developers I recognized less than 5 people from the PostgreSQL community at the conference and most of us were there because of our connections to other communities.  Having said that, most people I spoke to said that they loved PostGIS and PostgreSQL.</p>
  <p>The flight home from FOSS4G is also a good time to reflect on what it means to be a community conference.</p>
  <ul>
  <li> Each year OSGEO asks for submissions/proposals from the community on groups that want to host the next FOSS4G.  An OSGEO committee examines these submissions and selects a host city for the next conference.</li>
  <li> OSGEO moves the conference around the world each year.  They won&#8217;t hold the conference in the same continent two years in a row.  This makes it easier for people in different parts of the world too attend.  It also means that when the conference is in your backyard you make a special effort to attend because it probably isn&#8217;t coming back for at least three years.</li>
  <li> FOSS4G has a call for papers, the papers are then voted on by the wider community as an input into the paper selection processes.  Anyone in the community can have input into which talks are selected.</li>
  <li> After FOSS4G a financial report on the conference along with many other documents used in planning the conference are published in a subversion repository.  This acts as a resource for future groups considering hosting FOSS4G.  It also means that the revenues and expenses are public so people can can see where the money goes.</li>
  <li> FOSS4G does not just appeal to developers.  In addition to developers and sysadmins there where many cartographers and geographers at the conference</li>
  </ul>
  <p>Next year FOSS4G will be held in Bejing.  Various political, cultural, logistical and financial reasons will make it hard for many of the American attendees to go to FOSS4G.  Some people in Denver were talking about how there should be a North American FOSS4G next year as well. My thoughts on this are</p>
  <ul>
  <li>It is important to not dilute the FOSS4G brand, FOSS4G is the big geo-spatial conference that travels around the world.  If you start splitting the FOSS4G brand you will soon end up with  a FOSS4G-EU, FOSS4G-US , FOSS4G-Asia instead of a FOSS4G.   Other communities do this (*BSD as an example).   Figure out the relationship between FOSS4G, OSGEO and regional conferences before someone announces FOSS4G-US, maybe these regional conferences should be called something other than FOSS4G</li>
  <li> If you are thinking about creating another North American geo-spatial conference think about what other communities your conference can reach out to  (OSM, PostgreSQL, Python, etc&#8230;).  State of the Map was fun for a lot of the same reasons PGCon is fun.  You get to spend a few days meeting and hanging out with people that you regularly chat with online and everyone shares a deep passion of a very specific project.  A lot of the fun I had with FOSS4G was from the energy of being at such a large conference.</li>
  <li> If you do hold some sort of regional conference, for god sakes don&#8217;t hold the conference two weeks before the Bejing conference.  Space the conferences out so that it is easier for speakers, attendees and sponsors to attend both.  I think the next year will turn out to be a great year for the struggling OpenStreetMap community in the US because of the interest and enthusiasm generated at  SOTM and FOSS4G.  I hope FOSS4G in Bejing brings a lot more great Asian contributors into the open source geospatial tent next year.</li>
  </ul>
  <p>It feels like the plane is descending.  I will try to send this off once I&#8217;m home.</p>


guid: https://osmand.net/blog/osmand-ios-2-8-released
url: https://osmand.net/blog/osmand-ios-2-8-released
title: OsmAnd 2.80 (iOS)
published: 2019-06-09 14:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.781776
updated_at: 2019-10-20 15:30:23.197474
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd 2.80 (iOS)</h2><div class="date">June 09, 2019</div><p><a href="https://itunes.apple.com/us/app/osmand-maps-travel-navigate/id934850257">OsmAnd 2.80 is now available!</a></p><img src="https://osmand.net/images/blog/ios-2-72/01.png" alt="(Image not available offline)"/><h4>Mapillary</h4><p>We continue to follow our <a href="https://osmand.net/blog/2019-ny-resolutions">New Year’s resolutions</a>, where we have promised to catch the iOS version of OsmAnd up with the one on Android, and we are getting there.</p><p>We're proud to have collaborated with Mapillary for the new release. Now you can view the street-level imagery in your location or anywhere in the world using the Mapillary in OsmAnd for iOS.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-72/1.jpg"/><img src="https://osmand.net/images/blog/ios-2-72/2.jpg"/><img src="https://osmand.net/images/blog/ios-2-72/3.jpg"/><img src="https://osmand.net/images/blog/ios-2-72/4.jpg"/><div style="clear:both;"></div></div><p>Using the widget on the map, you'll be able to view images or videos and even add your own. To view the images and videos, you'll have to go to 'Configure screen' menu and enable the 'Mapillary' layer.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-72/7.jpg"/><img src="https://osmand.net/images/blog/ios-2-72/8.jpg"/><img src="https://osmand.net/images/blog/ios-2-72/9.jpg"/><img src="https://osmand.net/images/blog/ios-2-72/10.jpg"/><div style="clear:both;"></div></div> <p>Now you have access to an extensive database created by volunteers. To view media attached to a certain point on the map, just make a long tap there and open the context menu. You'll be able to scroll through images of the location easily.</p><p>Zoom in closer to view the images even more conveniently in a Mapillary window over your map. You can also press the white arrows to 'move' along the streets. To contribute to Mapillary, just press on the widget and take a picture.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-72/5.jpg"/><img src="https://osmand.net/images/blog/ios-2-72/6.jpg"/><div style="clear:both;"></div></div><h3>What's more in the latest version of 2.80?</h3><p>• Fixed city and postal code search.</p><p>• Improved performance of Hillshade and Contour Lines map rendering.</p><p>• Added Turkish TTS.</p><p>And remember that only together we can achieve the best results!</p><p>New features are coming SOON!</p><p><a href="https://itunes.apple.com/us/app/osmand-maps-travel-navigate/id934850257"><img src="http://osmand.net/images/app-store-badge.png" alt="Get it on App Store" height="60"/></a><p>OsmAnd at <a href="https://www.facebook.com/osmandapp/">Facebook</a>, <a href="https://www.twitter.com/osmandapp/">Twitter</a>, and <a href="https://www.reddit.com/r/OsmAnd/">Reddit</a>! Join us at our <a href="https://t.me/OsmAndMaps">group of Telegram</a>.</p>


guid: https://osmand.net/blog/osmand-tracker-0-5-released
url: https://osmand.net/blog/osmand-tracker-0-5-released
title: OsmAnd Online GPS Tracker 0.5
published: 2019-06-03 18:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.784930
updated_at: 2019-10-20 15:30:23.199637
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd Online GPS Tracker 0.5</h2><div class="date">June 03, 2019</div><p>Hello everybody!</p><p>We prepared a new version of our OsmAnd Tracker - 0.5.</p><p>What the new functions are?</p><img src="https://osmand.net/images/blog/tracker-0-5/01.png" alt="(Image not available offline)"/><h3>Added contact search</h3><p>Yes, it was a problem to find the contact or your group from a telegram list. Now it's easy!</p><div class="blogimageblock">    <img src="https://osmand.net/images/blog/tracker-0-5/1.jpg"/>    <img src="https://osmand.net/images/blog/tracker-0-5/2.jpg"/>        <div style="clear:both;"></div></div><h3>Added proxy settings</h3><p>Many of our users asked us to add proxy. We did it. Use proxy inside of OsmAnd Tracker. Go to Settings and write your proxy data: <i>Settings-> Privacy-> Proxy Settings</i></p><div class="blogimageblock">    <img src="https://osmand.net/images/blog/tracker-0-5/3.jpg"/>    <img src="https://osmand.net/images/blog/tracker-0-5/4.jpg"/>        <div style="clear:both;"></div></div><h3>GPS settings: minimum speed, accuracy, and distance between points</h3><p>Now, you can control your points. You can set settings of points. If you stay at one place some time without moving, for example, your points don't send and your GPX track will be nicer. Go to <i>Settings-> GPX settings</i></p><div class="blogimageblock">    <img src="https://osmand.net/images/blog/tracker-0-5/5.jpg"/>        <div style="clear:both;"></div></div><h3>Active marker on the OsmAnd map: distance, speed, and angle of movement of your contacts</h3><p>It's very useful to see distance, speed, and angle movement of your contacts at  OsmAnd map and text in Telegram. Now you can understand at once where and how your contact move.</p><div class="blogimageblock">    <img src="https://osmand.net/images/blog/tracker-0-5/6.jpg"/>    <img src="https://osmand.net/images/blog/tracker-0-5/7.jpg"/>        <div style="clear:both;"></div></div><p>Further instructions and helpful advice can be found <a href="https://osmand.net/features/tracker">here</a>.</p><p>Get OsmAnd Tracker today from <a href="https://play.google.com/store/apps/details?id=net.osmand.telegram&hl">Google play</a>.</p>.</p><p><a href="https://play.google.com/store/apps/details?id=net.osmand.telegram" target="_blank"><img src="https://raw.githubusercontent.com/osmandapp/osmandapp.github.io/9fc59e5136b6a07045eb96d052b3ce6ddde805c3/website//images/help/badge_store_google_play.png" alt="Get it on Google Play" height="60"/></a>


guid: https://osmand.net/blog/osmand-ios-2-7-released
url: https://osmand.net/blog/osmand-ios-2-7-released
title: OsmAnd 2.70 (iOS)
published: 2019-05-10 18:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.787440
updated_at: 2019-10-20 15:30:23.201852
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd 2.70 (iOS)</h2><div class="date">May 10, 2019</div><p><a href="https://itunes.apple.com/us/app/osmand-maps-travel-navigate/id934850257">OsmAnd 2.70 is now available!</a></p><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7.png" alt="(Image not available offline)"/><h3>OpenStreetMap (OSM) Editing</h3><p>We continue to follow our <a href="https://osmand.net/blog/2019-ny-resolutions">New Year’s resolutions</a>, where we have promised to catch the iOS version of OsmAnd up with the one on Android, and we are getting there. OSM Editing allows you to make contributions to <a href="https://www.openstreetmap.org">OpenStreetMap</a>, a global community aimed at creating a comprehensive map of the world and providing up-to-date open-source data to every user.</p><p>The feature lets you create new objects, so-called <a href="https://wiki.openstreetmap.org/wiki/Points_of_interest">points of interest or POI</a> on the map and <a href="https://wiki.openstreetmap.org/wiki/Notes">OSM notes</a>.</p><p>So how to do it? Tap on the map where the new POI has to be placed -> 'Actions' and press 'Create POI' or 'Open OSM Note' add its name and other details like working hours, website, etc. You'll also have to register at <a href="https://www.openstreetmap.org">OpenStreetMap</a> and then provide your OSM credentials to introduce changes.</p></ul><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-1.PNG"/><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-2.PNG"/><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-3.PNG"/><div style="clear:both;"></div></div><p>There are all your "Edits" and "Notes" at general menu -> 'My Places' -> 'OSM Edits'. From this menu, you can delete or upload all your edits and notes.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-4.PNG"/><div style="clear:both;"></div></div><h3>My places menu</h3><p>Combined "Favorites", "Trips" and "OSM Edits" screens into a new 'My Places' screen.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-5.PNG"/><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-6.PNG"/><div style="clear:both;"></div></div><h3>What's new in the latest version of 2.70?</h3><p>• Added options to control the size of map elements: use “Text Size” option to control the size of the text and icons on the map and “Map Magnifier” to control the overall size of map elements.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-8.PNG"/><img src="https://osmand.net/images/blog/ios-2-7/ios-2-7-9.PNG"/><div style="clear:both;"></div></div><p>• Opening a search result on the map will now zoom in on the selected point.</p><p>• Fixed icon sizes on map.</p><p>• Optimized multiple menus for iPhone X screen.</p><p>• Added Hungarian and Hungarian (formal) TTS, added alphabetical sorting to the TTS language list.</p>	<p>And remember that only together we can achieve the best results!</p><p>New features are coming SOON!</p><p><a href="https://itunes.apple.com/us/app/osmand-maps-travel-navigate/id934850257"><img src="http://osmand.net/images/app-store-badge.png" alt="Get it on App Store" height="60"/></a>


guid: https://osmand.net/blog/guideline-pt
url: https://osmand.net/blog/guideline-pt
title: Guideline for public transport
published: 2019-03-16 17:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.789930
updated_at: 2019-10-20 15:30:23.204217
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>Guideline for public transport</h2><div class="date">March 16, 2019</div><ul class="list"><li><a href="#Guidelines">Guidelines on how to build or correct public transport routes</a></li><li><a href="#mistakes">The most common mistakes</a></li><li><a href="#Validation_1">"OSM Inspector" final check (Data validator)</a></li><li><a href="#Validation_2">"JOSM validator" final check (Data validator)</a></li><p>Hi!</p><p>As you already know we have <a href="https://osmand.net/blog/osmand-3-3-released">recently released</a> an update of our app on Android!</p><p>And guess what? It allows our users to work out their routes using public transport, which will certainly help one to move through the concrete jungle faster. The OSM database, however, has some unchecked and dubious info, so we decided to give you a hand and show how to use the newest update to your advantage. Our instructions will be detailed, but, overall, not too complex, because we want you to engage in the process of forming OSM. Try not to break it, ok? Just kidding! If you need a bit more support use <a href="https://learnosm.org/en/josm/start-josm/">LearnOSM</a>.</p> <div class="subtitle" id="Guidelines"><h3>Guidelines on how to build or correct public transport routes</h3></div><p>So, let us go through the process of mapping out a route together using <a href="https://wiki.openstreetmap.org/wiki/JOSM">JOSM</a>. We have chosen to create a bus route. For starters, let’s mark our stops. There are two options in the settings, so one can either mark it as a regular bus stop (<a href="https://wiki.openstreetmap.org/w/index.php?title=Proposed_features/Public_Transport&oldid=625726#Stop_Position">public_transport=stop_position</a>), or as a place where the passengers will be waiting for the arrival of their bus (<a href="https://wiki.openstreetmap.org/w/index.php?title=Proposed_features/Public_Transport&oldid=625726#Platform">public_transport=platform</a>). These stops should be included directly into your route. Then, you should attach some tags, such as <i>public_transport=stop_position</i>, <i>name</i>, etc. You also can use presets: find them in Presets - Transport - Public transport.</p><img src="https://osmand.net/images/blog/guideline-pt/image_1.png"/><p>As a general rule, the platform for waiting passengers is located in front of a bus stop. It can be a node, a line or even a polygon. We will set two tags, one is <i>public_transport=platform</i>, and the other - <i>highway=bus_stop</i> (to meet the requirements of PT v1). We also encourage you to give it a name.</p><img src="https://osmand.net/images/blog/guideline-pt/image_2.png"/><p>You need to match the first stop with the starting point of your route. Logically, the last stop has to be near the finish. If, however, you have to put it in the middle, you will need to split the segment there.</p><p>After checking whether we have tagged all the stops correctly, we will go on mapping out the route. We will need to search a panel on the right side of the screen for “Relations”. If you cannot find it, go directly to Windows -> “Relations”. Found it? Click on “Create a new relation”.</p> <img src="https://osmand.net/images/blog/guideline-pt/image_3.png"/><p>A window will pop up. Here, we will create the relations. One can find the description of any tag they may need <a href="https://wiki.openstreetmap.org/w/index.php?title=Proposed_features/Public_Transport&oldid=625726#Route">here</a>.</p><p>Let’s go ahead and add some tags:</p><p><i>type=route</i></p><p><i>route=bus</i></p><p><i>from=Bus stop 1</i></p><p><i>to=Bus stop 3</i></p><p><i>name=Bus 5: Bus stop 1 => Bus stop 3</i></p><p><i>ref=5</i></p><p>Our next task is to add other elements of our route. Go to the main window, do not close the window where you have been creating the relation, though. Successively select all the segments of the route, using the CTRL button for multiple marking. These segments will appear in the tab Selection in the window where we have been creating the relation.</p><p>Using the buttons circled in red, we will add them to the “Members” tab.</p><img src="https://osmand.net/images/blog/guideline-pt/image_4.png"/><img src="https://osmand.net/images/blog/guideline-pt/image_5.png"/><p>Our next move is to add nodes <i>public_transport=stop_position</i> and <i>public_transport=platform</i>. Select them on the map and, then, add to the relation (higher than ways). Enumerate them to avoid further difficulties. For example, <i>stop_position 1, platform 1, stop position 2, platform 2</i> and etc. As a result, our relation editing window looks like this:</p><img src="https://osmand.net/images/blog/guideline-pt/image_6.png"/><p>The roles for stops and platforms, however, will be automatically filled out. For instance, in the column called “Role” stop_position has an assigned “stop” role, while platform has “platform” role. We will change it just a bit here. In particular, we will change the first and the last platform and stop_position roles. Add “_entry_only” to the objects related to the first stop (<i>stop_entry_only, platform_entry_only</i>), it means that this stop has just one purpose - entrance. The last stop, accordingly, should be marked “_exit_only” (<i>stop_exit_only, platform_exit_only</i>). Sometimes the roles are not assigned automatically, then you will need to fill them in manually.</p><img src="https://osmand.net/images/blog/guideline-pt/image_7.png"/><p>We have marked the buttons for moving and sorting the relation members as 1 and 2 (in red).</p><p><b>If the order of stops is incorrect, OsmAnd will NOT be able to plan a proper route for you!</b></p><p>To check whether the route has any gaps, use small arrows located on the right side of the way. They also will help you to determine the direction of the route.</p><p>Now, as you can see, we have a complete route, which starts at <i>Bus stop 1</i> (on Fedinskaya street) and ends at <i>Bus stop 3</i> (A-108).</p><p>As a rule, buses go both ways. So, why not to create a backtrack? Just start from copying the relations by pressing “Relations - Create copy of this relation”. You can also access the button in the redactor. In order not to get confused, we can delete all <i>stop_position</i> and <i>platform</i> from the relation copy and leave only the ways. By clicking on a button called “Reverse the order of the relation members” in the redactor window, we will reverse the direction of the route and, furthermore, will be able to check whether they are correct. Now, let us add stops and platforms, adding tags in the process.</p><p>All the directions, variants of route, and details attached to the route should be united relation type "route_master" which is required by the scheme PTv2. Our task now is to create a new relation and fill in the tags: </p><p><i>type=route_master</i></p><p><i>route_master=bus</i></p><p><i>ref=5</i></p><p>The tag “ref” is reserved for the number of the route.</p><p>Not closing the redactor, let’s find the previously created route and choose “Select relation” from the context menu. If you did alright, which we are sure of, you will see your route change color. It will become purple, and the tags will appear on the panel “Tags / Membership”.</p><img src="https://osmand.net/images/blog/guideline-pt/image_8.png"/><p>We will add it to “Members” using the corresponding buttons and save the relation.</p><img src="https://osmand.net/images/blog/guideline-pt/image_9.png"/><div class="subtitle" id="mistakes"><h3>The most common mistakes</h3></div><div class="subtitle" id="Validation_1"><h3>"OSM Inspector" final check (Data validator)</h3></div><p><a href="http://tools.geofabrik.de/osmi/?view=pubtrans_routes&lon=13.39599&lat=52.49897&zoom=12&overlays=ptv2_routes_invalid,ptv2_error_,ptv2_error_ways,ptv2_error_nodes">OSM Inspector</a> can help you to check whether routes are correct. <a href="https://blog.geofabrik.de/?p=436">Use it</a>, it does a pretty nice work!</p><p>In case there is a mistake, click on it in the tab “Selection” and detailed info will appear. A vertical scroll bar will show up automatically if there are multiple mistakes on different levels. Each mistake has a letter J next to it. Click on it, and the mechanism of <a href="https://josm.openstreetmap.de/wiki/Help/Preferences/RemoteControl">the remote control</a> will open the corresponding area in JOSM.</p><img src="https://osmand.net/images/blog/guideline-pt/image_10.png"/><p>Should the message “The OSM server ‘api.openstreetmap.org’ reported a bad request. The area you tried is too big or your request was too large. Either request a smaller area or use an export file provided by the OSM community” appear on the screen, the easiest way to resolve it would be to download this area through JOSM -> File -> Download data -> Download from Overpass API. Using <a href="https://wiki.openstreetmap.org/wiki/Overpass_API">Overpass API</a> allows you to get more data from the server in one request as well as to download only the required type of data.</p><p><i>Gap / Gap or unordered before this way</i></p><p>Any unintentional split of your track will result in this mistake. Search for the relation with this mistake in the editing panel.</p><img src="https://osmand.net/images/blog/guideline-pt/image_11.png"/><p>On this screenshot, we can see that the line which shows our route indeed has some gaps. To get to the root of the problem, click on the route segment near the split in the Members list. This way will be marked with red coloring in the main window of the program.</p><img src="https://osmand.net/images/blog/guideline-pt/image_12.png"/><p>To move across the segments of a relation you can also use the keyboard. By going through the track manually, it is easier to spot the segment that was placed incorrectly. And when we already know what has caused the mistake, we can just drag it to its right place. You can also use buttons on your left, if you wish.</p><p>The neighboring segments may have an inverted order, which is a fairly common mistake. To correct it, we can select the disorganized tracks in the list of Members and click the button  on the left, which is “Reverse the order of relation members”.</p><p>The image of a bus has been placed on the button in the latest versions of JOSM to automatically correct simple mistakes. And, finally, when we understand how to make corrections manually, we can run the program, fully controlling the process!</p><img src="https://osmand.net/images/blog/guideline-pt/image_13.png"/><p><i>Empty role for non-way object</i></p><p>In most cases, it means that we have left out one stop_position or platform role. We will need to fill in the empty role.</p> <p><i>Road vehicle route over non-road</i></p><p>This message will appear when the road is closed or does not allow a certain type of transport to pass through it. Just check the track and place some corrections.</p>  <p><i>Stop without proper tags</i></p><p>At the place of stop, there are no tags indicating the type of transport (bus=yes, train=yes and etc.).</p><p><i>Route has only stops/platforms</i></p><p>There are no ways in the route. Just add them.</p> <p><i>Trolley bus without trolley wire</i></p> <p>Tag trolley ways with trolley_wire=yes to mark contact electrical network.</p><p><i>Unknown route type</i></p> <p>This type of the route is unknown. Add some details.</p> <p><i>Unknown role</i></p> <p>The type of the role should be specified.</p><p><i>Stop is not node</i></p> <p>The place of a stop (stop_position) is not a node.</p>  <p><i>Stop not on way</i></p> <p>A stop is not implemented in your route, it should be put directly in it.</p><div class="subtitle" id="Validation_2"><h3>"JOSM validator" final check (Data validator)</h3></div><p>JOSM has an inbuilt validator which checks data. It will open automatically should you try to send the data to the server, it can also be handled manually. In order to do this, search for the tab “Validation Results” on the right of the main window.   Still cannot see it? Let’s turn it on through Windows -> Validation results. Click “Check” button and the whole area (if none selected) validation should start. Should you try sending data to the server, this check will work out only recently changed info.</p>  <img src="https://osmand.net/images/blog/guideline-pt/image_14.png"/><p>Let us go through the results related to public transport:</p> <p>- <i>Route relation contains a ‘forward/backward/alternate’ role</i></p><p>The PTv1 scheme implies additional role assignment to a route, but it goes against PTv2 schema. Delete all the roles and, if necessary, create your route again from scratch.</p>  <p>- <i>Route relation contains a gap</i></p><p>The same as <i>Gap / Gap or unordered before this way</i> in OSM Inspector.</p><p>- <i>Route scheme is unspecified. Add public_transport:version</i></p><p>If a route matches with the PTv2 scheme (as mentioned above), add the tag <i>public_transport:version=2</i> to the relation.</p> <p>- <i>Stop position is not part of the route</i></p><p>Look up <i>Stop not on way</i> in OSM Inspector, because they are identical.</p><p>____________</p> <p>As you already know, a public transport function has been added in <a href="https://osmand.net/blog/osmand-3-3-released">the newest release</a> of our app.  We have received some valuable feedback from our users who had already tried this function out. Unfortunately, not all the data in OSM is reliable, which causes a few problems. We sincerely hope that, with the help of these guidelines, it will be much easier for you to create and check the public transport routes.</p><p>P.S. We will do our best to upgrade these and other functional aspects of the app in the future. And remember that each user makes all the difference. Let’s create OSM together!</p>


guid: https://osmand.net/blog/osmand-3-3-released
url: https://osmand.net/blog/osmand-3-3-released
title: OsmAnd 3.3
published: 2019-03-07 20:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.792315
updated_at: 2019-10-20 15:30:23.206776
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd 3.3</h2><div class="date">March 7, 2019</div><p>We will be releasing the new version patch by patch during the next two weeks, and the info about app updates will be published in this article. Here is what has been done so far.</p><p>We follow our <a href="https://osmand.net/help-online/privacy-policy">Privacy Policy</a> and now we have completely <b>removed Facebook and Firebase analytics from the free version (OsmAnd+ didn't include it)</b>.</p><h3>Navigation on public transport: Metro, Buses, Tram and so on.</h3><p>First, we would like to introduce Navigation for public transport which is currently in the beta phase. <b>Each of you can try to navigate in your city by public transport now with OsmAnd</b>. We want to receive your feedback for customization of this feature. Most importantly, we use the latest New Public Transport Schema that is also called Public Transport Version 2 (PTv2) for our Public Transport navigation algorithm. You can check your public transport <a href="http://tools.geofabrik.de/osmi/">here</a>. <a href="https://osmand.net/blog/guideline-pt">Guidelines on how to build or correct public transport routes </a>is provided in our blog.</p></ul> <div class="centeredimageblog"><img src="https://osmand.net/images/blog/3.3/pt_1_blog.png"/><img src="https://osmand.net/images/blog/3.3/pt_2_blog.png"/><img src="https://osmand.net/images/blog/3.3/pt_3_blog.png"/><img src="https://osmand.net/images/blog/3.3/pt_4_blog.png"/><div style="clear:both;"></div></div><h3>Redesigned Directions menu</h3><p>We have made a more detailed Directions menu to help you navigate to your Home and Work and see the History of your destinations. When you click “Options”, you can see Settings for Navigation like the list of active GPX and others.</p></ul><div class="centeredimageblog"><img src="https://osmand.net/images/blog/3.3/pt_5_blog.png"/><img src="https://osmand.net/images/blog/3.3/pt_6_blog.png"/><div style="clear:both;"></div></div><h3>Improve Quick action</h3><p>As requested by many users, we have <b>added Show/Hide tracks, Day/Night mode switches in Quick action</b>. Quick action is a flexible menu of shortcuts providing direct access to whatever features you need to access most frequently: just select and add them to this 'toolbar'. In this way, the next time you need to add a new favorite, add a navigation destination, enable or disable voice guidance, etc, you can do it in one click using Quick action. The Quick action button can be added via the Configure screen menu. To add actions to the menu, please press 'Add action' right on its menu bar. To remove an action, go to Configure screen and tap the 'Quick action' menu, then remove any shortcuts not needed.</p></ul><div class="centeredimageblog"><img src="https://osmand.net/images/blog/3.3/pt_7_blog.png"/><div style="clear:both;"></div></div><h3>Additional info in the Route details.</h3><p>While building a route,  you can view information about the road itself like <b>road types, surface, steepness and smoothness</b> which will help you in understanding the road conditions of your route.</p></ul><div class="centeredimageblog"><img src="https://osmand.net/images/blog/3.3/pt_8_blog.png"/><img src="https://osmand.net/images/blog/3.3/pt_9_blog.png"/><div style="clear:both;"></div></div><p>Besides these changes, we've also made a range of fixes of some critical bugs, including flooded territories and crashes at some public transport stops.</p><p>Please update OsmAnd and share your opinion on our <a href="https://www.facebook.com/osmandapp/">Facebook</a>, <a href="https://www.twitter.com/osmandapp/">Twitter</a>, and <a href="https://www.reddit.com/r/OsmAnd/">Reddit</a>!</p><p><a href="https://play.google.com/store/apps/details?id=net.osmand" target="_blank"><img src="https://raw.githubusercontent.com/osmandapp/osmandapp.github.io/9fc59e5136b6a07045eb96d052b3ce6ddde805c3/website/images/help/badge_store_google_play.png" alt="Get it on Google Play" height="60"/></a><a href="https://www.amazon.com/OsmAnd-Maps-Navigation/dp/B00D0SA8I8" target="_blank"><img src="https://raw.githubusercontent.com/osmandapp/osmandapp.github.io/9fc59e5136b6a07045eb96d052b3ce6ddde805c3/website/images/help/badge_store_amazon.png" alt="Get it on Amazon" height="60"/></a></p>


guid: https://osmand.net/blog/osmand-tracker-0-4-released
url: https://osmand.net/blog/osmand-tracker-0-4-released
title: OsmAnd Online GPS Tracker 0.4
published: 2019-02-19 20:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.794725
updated_at: 2019-10-20 15:30:23.209324
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd Online GPS Tracker 0.4</h2><div class="date">Feb 19, 2019</div><p>Hi, Everybody!</p><p>Guess what - we have got something new for you! OsmAnd Tracker version 0.4 is fresh from the oven and ready to be downloaded. It does not matter whether you are an amateur or a professional sportsperson, or a traveler  - the app will pleasantly surprise you anyway.</p><p>In <a href="https://osmand.net/blog/osmand-telegram-released">our blog</a> we have already announced that we have chosen Telegram to access all the necessary data. It is an open social platform that perfectly satisfies our requirements. Not only does it provide the list of contacts, but it also allows sharing one’s location. A Telegram account will be used during the process of registration in OsmAnd Tracker. The cherry on top is that you do not have to download the Telegram app to use your account in OsmAnd Tracker, and vice versa - one does not need OsmAnd app to share their location with another user via Telegram. You do not have to worry about the safety of personal information and chats, though. Select the accessible data and do not worry. By the way, your messages will stay intact and secure.</p><p>The Tracker has an amazing new function that is Timeline. We used to have two tabs “Live now” and “My location”. Now our users can organize their tracks the way they like, sending their location with a timer. The transmission will stop the moment the set time runs out. You have time options up to 24 hours to choose from. The users you have sent the GEO position to will see either a text message with written coordinates or a dot on an actual map. And what if you want to share a road to work that you made a few days ago? Well, now - with OsmAnd - you can! Here comes “Timeline”, where you can find the exact time span and send it. That is it! As easy as it possibly could be. In this tab, you can also view the history of tracks that were either sent by you or to you on a chosen day. One also can have multiple chats with different settings and levels of integration. The application also allows you to access saved info chronologically to avoid any confusion and difficulties in finding a particular moment in your movement history, including the locations that have been shared with you by other users. Any user <a href="https://osmand.net/features/trip-recording-plugin">can master their tracks and work on the details directly in OsmAnd app</a>, which is fantastic!</p><div class="blogimageblock">    <img src="https://osmand.net/images/features/4-1-Timeline.png"/>    <img src="https://osmand.net/images/features/Timeline_2.png"/>        <div style="clear:both;"></div></div><p>Further instructions and helpful advice can be found <a href="https://osmand.net/features/tracker">here</a>.</p><p>Get OsmAnd Tracker today from <a href="https://play.google.com/store/apps/details?id=net.osmand.telegram&hl">Google play</a>.</p>.</p><p><a href="https://play.google.com/store/apps/details?id=net.osmand.telegram" target="_blank"><img src="https://raw.githubusercontent.com/osmandapp/osmandapp.github.io/9fc59e5136b6a07045eb96d052b3ce6ddde805c3/website//images/help/badge_store_google_play.png" alt="Get it on Google Play" height="60"/></a>


guid: https://osmand.net/blog/osmand-ios-2-5-released
url: https://osmand.net/blog/osmand-ios-2-5-released
title: OsmAnd 2.5 (iOS)
published: 2019-02-12 21:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.797009
updated_at: 2019-10-20 15:30:23.211526
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd 2.5 (iOS)</h2><div class="date">February 12, 2019</div><p><a href="https://itunes.apple.com/us/app/osmand-maps-travel-navigate/id934850257">OsmAnd 2.5 is now available!</a></p><h3>OsmAnd Live</h3><p>In a nutshell - you can access OsmAnd Live from an IPhone now. Do you remember our <a href="https://osmand.net/blog/2019-ny-resolutions">New Year’s resolutions</a>? We have promised to catch the iOS version of OsmAnd up with the one on Android, and we are getting there. This update is just the beginning, so we will continue gradually adjusting the features of the app. OsmAnd Live gives you a unique chance to collaborate with other active app users to create a verified detailed map. Once you have inserted a correction in OpenStreetMap, it takes an hour for the online map to adjust. And, finally, there are no limits on downloading maps and contour lines.</p><h3>Subscription options</h3><p>Even though, the subscription for OsmAnd Live is inexpensive, it gives you numerous benefits, such as unlimited downloads, the access to contour lines, and hourly updates. In fact, by subscribing to the app you help to support our developers and <a href="https://osmand.net/osm_live">the contributors that edit the info on OpenStreetMap</a>. Just choose your subscription plan and you no longer need to update the map data on the device, it will be done automatically every hour or so. There are three options to choose from: a monthly, trimester, and annual. The subscribed users have all the downloads and plugins available. You can also manage the subscription and payment details using App Store or IPhone Settings.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/ios-2-5/ios-2-5.png"/><div style="clear:both;"></div></div><p>And remember that only together we can achieve the best results!</p><p>New features are coming SOON!</p><p><a href="https://itunes.apple.com/us/app/osmand-maps-travel-navigate/id934850257"><img src="http://osmand.net/images/app-store-badge.png" alt="Get it on App Store" height="60"/></a>


guid: https://osmand.net/blog/2019-ny-resolutions
url: https://osmand.net/blog/2019-ny-resolutions
title: 2019 New Year resolutions
published: 2018-12-31 16:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.799354
updated_at: 2019-10-20 15:30:23.213817
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>2019 New Year resolutions</h2><div class="date">Dec 31, 2018</div><p>Happy New 2019 Year!</p><p>Following the nice tradition of creating the list of New Year Resolutions, on behalf of our team, I will create and share it here and hope it would be an appropriate post for the community.</p><h3>2018 achievements</h3><img src="https://osmand.net/images/blog/2018/2018-NY-1.png"/><ul class="textlist">	<li><b>Measure Distance.</b><br> This is a long-expected UI simplification and enhancement to create GPX tracks as lines and as route points using a mobile phone. It allows:		<ul>			<li>To draw lines and measure straight distances</li>			<li>To draw a route between selected points as car, pedestrian, bike even as boat on channels route. Basically it calculates a route between all intermediate points and so far we've tested up to 50-100 points.</li>			<li>To save all edits as GPX track.</li>			<li>Edit an existing GPX track.</li>			<li>Easily accessible via Drawer menu.</li>		</ul>	</li><br>	<li><b>OsmAnd Live maps.</b><br> In 2018 we've moved our implementation to Overpass and managed to achieve average delay 20 minutes after the improvements (90th percentile delay is less than 40 minutes). If we exclude all medium outages for 2-3 hours, which happens once a month, then average time between your edits and appearance on OsmAnd would be 10-15 minutes. So it gives a great tool to quickly verify changes on a spot. I've made a presentation at SOTM 2018 about it - https://youtu.be/ZHY0KE4IzWI</li><br>	<li><b>Travel - Wikivoyage.</b><br> We've launched a new travel category based on open data https://www.wikivoyage.org/ project. There were already some improvements how to open GPX files in OsmAnd and select specific categories i.e. Eat / Accommodation / See. We went further and made all that data available offline with cached images. By downloading 300 MB, you can have all travel articles around the World and enjoy your trips. Of course, Wikivoyage data is linked to Wikipedia articles about specific places, so for a complete picture, you will need to download Wikipedia as well.</li><br>	<li><b>iOS Navigation, Ruler and other.</b><br> We've finally made huge progress in our iOS version and managed to organize a small &amp; stable development group. One of the primary important features was Navigation which is a full copy of Android version including navigation on GPX and all parameters and vehicle support. This also included a variety of widgets added to iOS with an advanced version of Ruler which works in 3D as well.</li><br>	<li><b>OsmAnd Telegram.</b><br> It was a hard decision for us but we decided to go into a completely new area and make a product for active travelers OsmAnd Telegram. This is an online GPS tracker which allows you to share your location within a group of people and also see everybody on the map. Sharing your location is always linked with chat-messaging, so we decided to build on the most open social technology Telegram. We've put the OsmAnd API on a completely different level to support a complex application such as OsmAnd Telegram and we also plan to enhance that API, so more and more side-apps could enrich OsmAnd and make it easier to use for specific cases.</li><br></ul><img src="https://osmand.net/images/blog/2018/2018-NY-2.png"/><h3>2019 resolutions</h3><p>We knew our list up front and it was based on your emails and issues. We selected 5 most important directions for our small team of 9 people to make it happen.</p><ul class="textlist">	<li><b>Public Transport.</b><br> This feature is almost ready and we planned it for 2018 release though we will need to polish it and do a proper beta test. OsmAnd already has all transport lines clickable which dramatically improved in 2018, so you can view and validate the public transport lines.</li><br>	<li><b>Place Reviews!</b><br> This is probably the most important goal for 2019. We fully understand that we need to do something to fight for Open Reviews Data against Google, TripAdvisor who takes the data bound to its own products, rules and etc. It is time to make something like OpenStreetMap for Reviews and we've already found connections with other OSM app developers to build a unified platform. Obviously the data will be open for everybody and that is the only way to get enough reviews to rate the places.</li><br>	<li><b>Travel + Wikivoyage.</b><br> We've got positive feedback for integrating with Wikivoyage data but it is very obvious we need to do more. We plan to let people contribute back to Wikivoyage more actively, so the open data will be more up to date. We also need to split it by regions to make the maps smaller and generate them every month. So, there are lots of small and very important features to make it useful :)</li><br>		<li><b>iOS improvements: OsmAnd Live, Quick Action, OSM Editing.</b><br> 2018 was decisive for iOS version and we've started to work hard on features to reduce the gap between Android / iPhone version. In 2019, we are planning to release the most requested features, and the good sign is that iOS audience is growing very fast, reacting to the changes we do. OsmAnd Live, OSM Editing and Quick Action are even a shorter list what we plan to achieve in 2019.</li><br>	<li><b>OpenGL-rendering (speed up rendering).</b><br> This is a long-awaited feature and it is ready for iOS users but unfortunately, we didn't publish it for Android due to a big variety of devices on Android. It is available for public test as https://download.osmand.net/latest-night-build/OsmAnd-qt-arm-nightly.apk. Obviously, there are pros and cons: it is faster but more power consuming; it has 2.5D but not all layers are currently supported; it requires more resources for the same operations.</li><br></ul><p><b>Happy New 2019 Year!</b></p>


guid: https://osmand.net/blog/giveaway-start
url: https://osmand.net/blog/giveaway-start
title: Giveaway
published: 2018-12-29 18:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.801838
updated_at: 2019-10-20 15:30:23.216062
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>Giveaway</h2><div class="date">Dec 29, 2018</div><p>We know that not everyone is able to purchase OsmAnd on Google Play / Amazon store or to purchase maps for OsmAnd on iTunes store due to various reasons, but many of you would still like to support the application by using it and spreading it around the world.</p><p>OsmAnd team decided to do a small present for our subscribers during this Christmas. And we started our True Giveaway based on Bitcoin. A little bit technical information: the number of promocodes is limited and each round 1 participant is selected for 1 promocode based on random number. The source of random is the latest available Bitcoin  Block hash. More information you can read and find out <a href="https://osmand.net/giveaway">on our website</a>. </p><h3>Christmas - our first giveaway</h3><p>During this Christmas weekend we started our first giveaway and we sent a special letter for our subscribers. At the end of the letter there was a special button to uniquely identify user with participant id.</p><img src="https://osmand.net/images/blog/giveaway-start/giveaway_sub_1.png"/><p>At first time we gave away <b>75</b> promocodes for OsmAnd+. At the last round there were <b>322</b> participants, so <b>25%</b> were winners.</p><img src="https://osmand.net/images/blog/giveaway-start/giveaway_win_1.png"/><p>Each hour (round) the winner was selected by latest available Bitcoin Block, so our giveaway is totally transparent and also respects privacy. </p><p>We were very surprised with your activity and engagement, so we decided to do on a regular basis. Also we plan to give away OsmAnd+, OsmAnd iOS and other paid plugins. Right now you can only subsribe for a giveaway at the Download screen in OsmAnd Android but we plan to enable subscription soon on the <a href="https://osmand.net">main page</a></p><p>Subscribe and Good luck!</p><p>We love you, our users!</p>


guid: https://osmand.net/blog/osmand-live-support-changes-count
url: https://osmand.net/blog/osmand-live-support-changes-count
title: Detailed OSM changes stats
published: 2018-09-30 11:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.804083
updated_at: 2019-10-20 15:30:23.218270
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>Detailed OSM changes stats</h2><div class="date">Sep 30, 2018</div><p>This is a tremendous example of how a small change in the OpenStreetMap API could make an impact on other applications. As part of our OsmAnd Live initiative, we've collected detailed statistics about OSM changes over the last 2 years. As of today, we've gathered information about all changesets, all users and all editors. What is unique about our stats is that we are also collecting them by multiple world regions. </p><p>But we've always been asked to focus on how many objects were changed rather than how many changesets are submitted. This month that change was finally merged into the platform <a href="https://github.com/openstreetmap/openstreetmap-website/issues/1947">https://github.com/openstreetmap/openstreetmap-website/issues/1947</a>. So the rest of the implementation was pretty clear for us and it is finally in place as well.</p><h3>URLs and APIs</h3><p>You can view all stats at <a href="https://osmand.net/osm_live">https://osmand.net/osm_live</a> and browse by specific month and regions. It is also available via JSON API i.e. <a href="https://osmand.net/reports/query_report?report=all_contributions_requests&month=2018-09&region=">Worldwide</a> and <a href="https://osmand.net/reports/query_report?report=all_contributions_requests&month=2018-09&region=belarus_europe">Belarus</a> for example. </p><img src="https://osmand.net/images/blog/changes-count/stats.png"/><h3>Stats for 29 September 2018</h3><table border="1">	<tr>		<th>Changesets</th>		<th>Unique users</th>		<th>Object edits</th>		<th>Editor</th>	</tr>	<tr><td>14223</td><td>1991</td><td>810839</td><td>iD 2.1</td></tr>	<tr><td>7334</td><td>1171</td><td>1591351</td><td>JOSM</td></tr>	<tr><td>1169</td><td>247</td><td>7382</td><td>StreetComplete</td></tr>	<tr><td>818</td><td>476</td><td>1324</td><td>MAPS.ME</td></tr>	<tr><td>623</td><td>142</td><td>55098</td><td>Potlatch</td></tr>	<tr><td>305</td><td>115</td><td>3666</td><td>Vespucci</td></tr>	<tr><td>305</td><td>149</td><td>537</td><td>OsmAnd</td></tr>	<tr><td>304</td><td>99</td><td>8582</td><td>Go Map</td></tr>	<tr><td>124</td><td>10</td><td>8716</td><td>Merkaator</td></tr></table><p>Don't forget statistics lie.</p>


guid: https://osmand.net/blog/osmand-telegram-released
url: https://osmand.net/blog/osmand-telegram-released
title: OsmAnd Telegram — Location Sharing
published: 2018-09-14 10:00:00
fetched: 2019-10-20 15:30:23.069255
created_at: 2019-06-09 18:12:14.807048
updated_at: 2019-10-20 15:30:23.220995
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd Telegram — Location Sharing</h2><div class="date">Sep 14, 2018</div><p>We are pleased to announce that we have a new application in OsmAnd family. This application helps to see your contacts on the map in OsmAnd. It has a social aspect; you need to have contacts, that's why it was challenging for us to choose the platform to build it on.</p><p>We have chosen Telegram as the most open social platform since it has an Open API, Open SDK and, moreover, it will eventually have an Open Server implementation (Blockchain).</p><div class="blogimageblock">        <img src="https://osmand.net/images/blog/telegram/1-My_Location.jpg"/>        <img src="https://osmand.net/images/blog/telegram/4-Map.jpg"/>        <div style="clear:both;"></div></div><p>It is important that we have a great level of interoperability between the applications. You can easily exchange your location with your contacts without having 'Telegram mobile app' (OsmAnd Telegram is an independent telegram client) and, the other way around, your contacts should not necessarily have OsmAnd Telegram in order to see your location. We designed OsmAnd Telegram so you could still use your Telegram app and OsmAnd app naturally without flaws.</p><p>The application sends the Live Location messages to the selected chats for the time you specified, displays a list of your contacts and groups, and checks the chats for the presence of the messages with a location, which are then displayed on the map in OsmAnd.The application does not send or view your text messages.</p><span>Log in using your phone number registered in Telegram to enable the following:<ul>    <li>Manage the list of contacts and groups that send you their location</li>    <li>View the location of contacts and groups in real time on a map in OsmAnd</li>    <li>Set the time for sharing the location separately for each chat</li>    <li>Set the location update frequency</li></ul></span><div class="blogimageblock"><img src="https://osmand.net/images/blog/telegram/2-My-location.jpg"/><img src="https://osmand.net/images/blog/telegram/3-Live.jpg"/><div style="clear:both;"></div></div><h3>Settings:</h3><p>There are several settings for managing the frequency of sharing locations and controlling the display of the locations sent to you.</p><p><b>Send my location</b> — allows you to set the interval for sending your position from once per second (for increasing accuracy) to once in 5 minutes (for reducing power consumption).</p><p><b>Stale location</b> — allows you to set the time (from 1 minute to 1 hour) after which the location of your contact will be considered outdated. If the location becomes outdated, the position becomes gray.</p><p><b>Location history</b> — allows setting the time (from 5 minutes to 24 hours) after which the contact with the deleted location will be hidden from the list and the OsmAnd map.</p><p>The <b>OsmAnd Connect</b> setting allows you to select the version of OsmAnd in which your contacts with the position will be displayed on the map.</p><div class="blogimageblock">    <img src="https://osmand.net/images/blog/telegram/Settings-1.jpg"/>    <img src="https://osmand.net/images/blog/telegram/Settings-2.jpg"/><div style="clear:both;"></div></div><h3>OsmAnd Assistant:</h3><p>We have also prepared a Telegram bot called 'OsmAnd Assistant,' so you can integrate OsmAnd Telegram with 3rd party trackers or APIs. Hence, you can not only share your location but also the locations of your bicycle or a group of individuals taking part in racing which is available via API, and OsmAnd Telegram will make sure you have a nice presentation of locations on the map in OsmAnd itself.</p><p>OsmAnd Telegram represents the idea that we are not trying to lock-in our users into a single app, and we are in line with our mission to bring the value to end-users by focusing on the strong side of our app, meanwhile integrating with others. Recently we have also expanded a lot of our <a href="http://osmand.net/build_it">OsmAnd APIs</a>, so you can fine-tune or write another app that will seamlessly integrate with OsmAnd. Please share with us what you think about this new app on <a href="https://twitter.com/osmandapp">Twitter</a> and other social channels.</p><p>To view the GPS positions of contacts on the map, you need the latest version of <a href="https://play.google.com/store/apps/details?id=net.osmand">OsmAnd</a> or <a href="https://play.google.com/store/apps/details?id=net.osmand.plus">OsmAnd+</a>. The minimum supported version of OsmAnd or OsmAnd + is 3.0.4.</p><p>Get OsmAnd Telegram - Location Sharing <a href="https://play.google.com/store/apps/details?id=net.osmand.telegram">Google Play</a>.</p>


guid: https://osmand.net/blog/osmand-3-1-released
url: https://osmand.net/blog/osmand-3-1-released
title: OsmAnd 3.1
published: 2018-08-02 10:00:00
fetched: 2019-09-03 06:30:25.527721
created_at: 2019-06-09 18:12:14.809341
updated_at: 2019-09-03 06:30:25.686115
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd 3.1</h2><div class="date">Aug 2, 2018</div><p>OsmAnd 3.1 is here! Here are some of the most important updates.</p><h3>Wikipedia and Wikivoyage</h3><p>We have significantly improved the Parsing algorithms of Wikipedia articles, fixed the problems with  French and Italian languages in which there were practically no articles.    The mistakes in the articles content processing in Wikivoyage have also been eliminated.     The updated Wikipedia and Wikivoyage files are already available.</p><div class="blogimageblock"><img src="https://osmand.net/images/blog/3.1/img-1-1.jpg"/><img src="https://osmand.net/images/blog/3.1/img-1-2.jpg"/><div style="clear:both;"></div></div><h3>OpenStreetMap editing</h3><p>Some great news for  <a href="https://www.openstreetmap.org">OpenStreetMap.org</a> contributors who use OsmAnd.We have added synchronization of OSM Edit to Google App Backup, so now in the case of OsmAnd reinstallation, all your OSM edits that have not been uploaded yet will not be lost.<p>We have corrected the error in the map notes. If the author of the note was anonymous, OsmAnd did not display the names of all the commentators.</p><p>The most important innovation for contributors: we have added the ability to edit tags for non-point objects and ways.</p></p><div class="blogimageblock"><img src="https://osmand.net/images/blog/3.1/img-2-1.jpg"/><img src="https://osmand.net/images/blog/3.1/img-2-2.jpg"/><div style="clear:both;"></div></div><h3>Map Makers</h3><p>As requested by many users, we have added the possibility to hide the layer with markers from the map. To do this, open OsmAnd > Menu > Configure Map > Markers > Turn off. The layer will turn on as soon as you add a new marker. Also, the problems with turning off the groups of markers have been fixed.</p><div class="blogimageblock">        <img src="https://osmand.net/images/blog/3.1/img-3-1.jpg"/>        <div style="clear:both;"></div></div><h3>Context menu</h3><p>The tables with the numbers of public transport routes in the night mode are now fixed, the numbers became readable regardless of the background color. Furthermore, the problems with the size of Additional options on screens with a resolution of less than 360dp are eliminated.</p><div class="blogimageblock"><img src="https://osmand.net/images/blog/3.1/img-4-1.jpg"/><img src="https://osmand.net/images/blog/3.1/img-4-2.jpg"/><div style="clear:both;"></div></div><h3>Boat navigation</h3><p>We have included support for waterway fairway as well as added navigation by polygons in the test mode.    Boat mode for navigation can be enabled together with the <a href="http://osmand.net/features?id=nautical-charts"> Nautical plugin.</a></p><div class="blogimageblock"><img src="https://osmand.net/images/blog/3.1/img-5-1.jpg"/><img src="https://osmand.net/images/blog/3.1/img-5-2.jpg"/><div style="clear:both;"></div></div>


guid: https://osmand.net/blog/mission
url: https://osmand.net/blog/mission
title: OsmAnd Mission
published: 2018-06-28 10:00:00
fetched: 2019-07-11 14:30:29.214022
created_at: 2019-06-09 18:12:14.811656
updated_at: 2019-07-11 14:30:29.365227
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>OsmAnd Mission</h2><div class="date">June 28, 2018</div><p>The OsmAnd team has been working on open-source maps mobile applications and receiving support from our community for many years now.    The time has come to create a manifest describing our place in the world of IT by explaining why we prefer some solutions/products/technologies over others,    clarifying our values, and strengthening our long-term relationship with the community.</p><span class="quote">Our defined goal is to create universally helpful open-source offline travel tools and to encourage crowdsourcing initiatives worldwide.</span><h3>"Open-Source" for "Crowd-Source"</h3><p>We strongly believe that open-source applications make the world a better place by not competing for revenue, but competing for    making the best product. The same applies to crowd-sourced projects where the data is open to everyone. Open-source    and crowd-source are not the same: Some truly open-source projects feature non-accessible    data, and some crowd-sourced projects feature valuable data but no accessible product.</p><p>When OsmAnd was first launched, the app was based solely on OpenStreetMap. Later, we moved on adding Wikipedia support, Mapillary    imagery, and Wikivoyage. In this fashion, everybody can contribute and see results in the app within a month, or even within the hour (via    an OsmAnd Live subscription).</p><p>Along these lines, our most important plan for OsmAnd is to continue to be a long-sustained project    open for any kind of contribution, and at the same time our code can be used by anyone for non-commercial purposes.</p><h3>Respect for Privacy</h3><p>In OsmAnd, we strongly respect the privacy of every individual seeing it as an integral part of human dignity. Our principles    of data protection include:</p><ul style="list-style: none;">    <li>        <b>Geolocation data</b>: We recognize geolocation data as critical and sensitive personal information which we will never store, nor share		   with any third party.</li>    <li>        <b>Digital footprint</b>: We rate IT footptint-capable data like your screen resolution or IP address as sensitive.        Global services aggregating such information can use it to assemble a digital footprints, track users,        deliver personalized ads, etc., hence we entirely avoid using such services and do not transfer such data to third parties.</li>    <li>        <b>Simple Analytics</b>: We aim at making the best application possible and want to ensure it runs smoothly under all        common conditions. For this reason we may collect anonymous and geo-imprecise information, used strictly for internal purposes. Of course, we        always provide a way to opt out.</li>    <li>        <b>Ads</b>: Again, in the spirit of protecting everyone's privacy, we intend to keep OsmAnd entirely ad-free. We strongly rely        on the 'pay for the app' model as a sustainable and reliable solution.</li>    <h3>Community Support</h3>    <p>Communities have great power and potential to evolve and make a difference for themselves and their goals. Our goal is to create tools        that will help people share and update information about their surroundings and environment. In OsmAnd, you can almost instantly add new data to a map,        and make it available to others. In this fashion, maps quickly become extremely detailed, which in turn improves OsmAnd's use and value		for activities like driving, hiking, cycling, sailing, traveling abroad, and in many other ways.</p>    <p>OsmAnd also supports mappers through the 'OsmAnd Live' initiative, where users purchase a subscription to frequent        map updates and other tools, while those funds become rewards for the contributors.</p>    <p>We are strong supporters of a <i>'do it yourself'</i> spirit. When you support the project, you do it for yourself as well as for        the global community. When you translate, update, code, test, or leave feedback – you fine-tune the very instrument        people all over the world use every day. And they are doing the same in return.</p>    <h3>Reliable Maps, Available to Anyone</h3>    <p>We are convinced that OsmAnd has to be accessible to anyone. That is why the free version is globally available, featuring a        complete functionality. Everyone deserves to have access to reliable maps working without the internet and we intend        to develop new possibilities in this area.</p>    <p>P.S. Please support us by funding, purchasing, contributing to OsmAnd or to related crowd-source projects. We will keep        supporting you by creating open-source apps, maintaining them, funding mappers, and spreading openness in the world.</p>    <p>Let's sustain a cycle of support in an Open World!</p>    <div align="right">        <p>            <i>OsmAnd CEO – Victor Shcherb</i>        </p>    </div>


guid: https://osmand.net/blog/custom-travel
url: https://osmand.net/blog/custom-travel
title: Custom Travel - creating your one-of-a-kind travel guide
published: 2018-06-26 10:00:00
fetched: 2019-07-11 14:30:29.214022
created_at: 2019-06-09 18:12:14.813968
updated_at: 2019-07-11 14:30:29.367485
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2>Custom Travel - creating your one-of-a-kind travel guide</h2><div class="date">June 26, 2018</div><p>Custom Travel tool helps you create your own travel guide and use it via OsmAnd Travel feature. Basically, you can use the text from any source (from Wikipedia to your friend's blog) and pair it with a GPX track.</p><div class="centeredimageblog">    <img src="https://osmand.net/images/blog/custom travel/customtravel_1.png"/>    <img src="https://osmand.net/images/blog/custom travel/customtravel_2.png"/>    <div style="clear:both;"></div></div><h3>Prepare the files</h3><p>To start, please prepare your files. Save your track with '.GPX' extension and the text file as '.html'. In order to create your custom database successfully, the names of the GPX file and the html have to be identical. For example, 'Milan.gpx' and 'Milan.html'.</p><p>The GPX tracks can be downloaded from the internet or you can create one (using OsmAnd or any online GPX creation tool).</p><p>For example, you can use <!-- <a href="https://ridewithgps.com">RideWithGps</a> service,  --><a href="https://www.gpsies.com/createTrack.do">GPSies</a>, or a simialr one. </p><p>Add the points you'd like to visit into your route and then save that route as a GPX track. <!-- After that, upload it to your phone and open using OsmAnd. You can also view other people's routes or check  <a href="https://www.openstreetmap.org/traces">OpenStreetMap GPX tracks</a>-->.</p><img src="https://osmand.net/images/blog/custom travel/gpsies_track_create.png"/><p>Preparing your GPX track in OsmAnd is also possible, you can read about it here: <a href="https://osmand.net/features/measure-distance">Measure distance</a></p><p>Choosing the source for your guide's text is completely up to you. It can be  <a href="http://www.touropia.com/tourist-attractions-in-milan/">an article</a> in a travel blog, a <a href="https://en.wikipedia.org/wiki/Milan">Wikipedia page</a>, etc. Simply copy the text, then paste it into a text editor and save it with an 'HTML' extension. Make sure to make the name of the GPX track and the HTML file identical.</p><h3>Launch the tool</h3><h3><a href="http://download.osmand.net/latest-night-build/OsmAndMapCreator-main.zip">Download OsmAnd MapCreator</a></h3><p><b>Linux</b>.<ol><li>Download MapCreator and extract the files from the archive.</li><li><p>Start your console and open the MapCreator folder.</p><p>Example: cd /home/user/OsmAndMapCreator-main/</p></li><li>Run <b>./utilites.sh travel-guide-creator</b></li><li><p>When asked to provide the path, please paste the path to the folder where you have your files prepared.</p></li></ol><p>Example: cd /home/user/MyCustomGuides/</p><p>We do not recommend locating your files in the 'TravelGuideCreator' folder itself. Kindly see the 'Read me' file for step-by-step commands.</p><p><b>Windows</b>. Open <b>utilities.bat</b> file.</p><h3>Import files</h3><p>Your file should be created in seconds. By default, it will be named 'travel_guide.sqlite', but you can rename it as you want making sure to save it with 'sqlite' file extension. For example, 'Milan_weekend_tour.sqlite'. After that, please transfer it to your device. Specifically, to the folder where OsmAnd travel guides are stored. You can check the path to that folder in 'OsmAnd > Settings > General > Data storage folder > Manually specified. Please copy the file there using any mobile file manager or by connecting your phone to computer. Restart the app.</p><h3>Use your guide</h3><p>After you start the app, please go to 'Travel' menu. Press Options > 'Travel book' to select your database as a source. After that, you'll be able to use search normally in order to view the articles. You'll find all the points from your GPX track in 'Points' menu. Please use 'Show on map' button to view your points on the map. You can bookmark any article to have a quicker access to it.</p><div class="centeredimageblog">    <img src="https://osmand.net/images/blog/custom travel/customtravel_3.png"/>    <img src="https://osmand.net/images/blog/custom travel/customtravel_4.png"/>    <div style="clear:both;"></div></div>


guid: https://osmand.net/blog/travel
url: https://osmand.net/blog/travel
title: Travel guides in OsmAnd
published: 2018-06-08 10:00:00
fetched: 2019-07-06 15:30:24.205117
created_at: 2019-06-09 18:12:14.816264
updated_at: 2019-07-06 15:30:24.352767
summary:
  <head><style type="text/css">
  .blogimageblock img {
      max-width: 300px;
      padding: 10px;
      margin-top: 20px;
      margin-right: 10px;
      margin-bottom: 20px;
  }
  .blogimageblock {
      margin: 30px auto 0;
      text-align: center;
      border-radius: 6px;
      background-color: #fafafa;
      border: 1px solid #e6e6e6;
  }

  </style></head>
  <h2 id="Travel_guides_in_OsmAnd">Travel guides in OsmAnd</h2><div class="toc"><ul class="list"><li><a href="#How_to_use">How to use the guides</a></li><li><a href="#Options">Options</a></li><li><a href="#Points">Points</a></li><li><a href="#Other_guides">Other guides</a></li></ul></div><p>We always strived to make OsmAnd not just your navigator, but your all-around travel guide. That's why, in version 3.0, the 'Travel guides' option has been added.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/travel/travel_1.png" alt="(Image not available offline)"/><img src="https://osmand.net/images/blog/travel/travel_3.png" alt="(Image not available offline)"/><div style="clear:both;"></div></div><p>Enjoy offline guides for basically any destination. The guides are based on Wikivoyage, a community project similar to Wikipedia, where volunteer writers create articles with real and up-to-date information. Such information includes the main facts about a place, transport, landmarks, shopping spots, etc. Going to <a href="https://en.wikivoyage.org/wiki/London">London</a>, <a href="https://en.wikivoyage.org/wiki/Shanghai">Shanghai</a> or planning a visit to <a href="https://en.wikivoyage.org/wiki/Grand_Canyon">the Grand Canyon</a>? Your guide is there to help you.</p><div class="subtitle" id="How_to_use"><h2>How to use the guides</h2></div><p>To start using the feature, please go to OsmAnd > Travel guides. Make sure to download the travel guide file to be able to use the feature. This file is also available in Download maps > 'Travel guides' menu. Now all your guides are saved on device and you can look up any information, even if you're abroad or out of a mobile network area.</p><p>As soon as the file is downloaded, you can start using the guides. Just go to OsmAnd > 'Travel guides' and enter the name of the city into the search field. As soon as you open the article, you'll see the information conveniently arranged by topic. The information often includes topics like 'Understand' with the details about local culture and habits, 'Talk', 'Get in', 'Get Around', 'See' that covers main places of interest, 'Buy' with the recommended shopping spots, 'Sleep', 'Drink', and 'Go next' suggesting the locations you may be interested in after visiting the current one. You can also press 'Bookmark' to have a quick access to this article later. All such articles will be saved in your 'Bookmarked articles' list.</p><p>The articles are sharable. Just press Share button in the upper-right corner and send it to your friend. That person will be able to open your link in OsmAnd directly if they have the app on their device.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/travel/travel_2.png" alt="(Image not available offline)"/><img src="https://osmand.net/images/blog/travel/travel_5.png" alt="(Image not available offline)"/><div style="clear:both;"></div></div><div class="subtitle" id="Options"><h2>Options</h2></div><p>You can also choose if you'd like to download the images or not, clear image cache or delete the search history. To do that, please press the 'Options' button in the upper-right corner of the screen.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/travel/travel_4.png" alt="(Image not available offline)"/><img src="https://osmand.net/images/blog/travel/travel_6.png" alt="(Image not available offline)"/><div style="clear:both;"></div></div><p>The articles are available in different languages. By default, the language you're using in OsmAnd will be also picked for Travel guides. But if you'd like, you can view your article in a different one. To do that, press the translation button next to the name of the article.</p><p>For example, the article about London is available in Chinese, Dutch, English, French, German and many other languages.</p><div class="subtitle" id="Points"><h2>Points</h2></div><p>Articles often contain points grouped by topic. They are available in the 'Points' menu at the bottom of the article. You'll see food locations,  airports, railway stations, places to shop, main historical attractions, accommodation options, etc. To view them all on the map, please enable 'Show on map' option. The track with the points will be also available in My places > My tracks > Travel. To remove the points from the map, please go to 'Configure map' menu and disable the 'GPX track' option.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/travel/travel_7.png" alt="(Image not available offline)"/><img src="https://osmand.net/images/blog/travel/travel_8.png" alt="(Image not available offline)"/><div style="clear:both;"></div></div><div class="subtitle" id="Other_guides"><h2>Other guides</h2></div><p>Besides the city guides, you can read about regions or countries, and even scroll through special guides like the list of UNESCO Global Geoparks Network or even a phrasebook of a country you're visiting. To see more exciting articles, please see the 'Explore' tab of the travel guides menu.</p><div class="centeredimageblog"><img src="https://osmand.net/images/blog/travel/travel_9.png" alt="(Image not available offline)"/><img src="https://osmand.net/images/blog/travel/travel_10.png" alt="(Image not available offline)"/><div style="clear:both;"></div></div><p>What do you think of this feature? Please join us on <a href="https://www.facebook.com/osmandapp/">Facebook</a> and <a href="https://www.twitter.com/osmandapp/">Twitter</a> to share your thoughts.</p>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-8346970140380752493
url: http://ksmapper.blogspot.com/2015/02/a-note-about-josm.html
title: A Note About JOSM
published: 2015-02-04 00:43:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.799506
updated_at: 2019-10-25 14:30:27.155457
summary:
  Over the past couple of months I have been slowly working to add support for OSM notes into JOSM. Back in July 2013, Ian Dees and I initially did a rewrite of the OpenStreetBugs plugin to support notes. We actually both started out in a friendly competition to see who could get a working version out first as we traded words on IRC. But soon Ian gave me access to his GitHub <a href="https://github.com/iandees/josm-notes" target="_blank">repository</a> where we continued to work on the plugin for a while.<br /><br />This plugin basically worked but I decided that since notes are now a core feature of OSM, they should also be more integrated into JOSM. In particular I wanted the workflow of notes to follow the standard JOSM download-work-upload sequence. The plugin was designed to be always online and issued a new query against the notes API every time you panned or zoomed the map. It had an "offline" mode which would stop this behavior and allow you to work without constant network traffic. However in this mode, changes to notes and newly created notes were not shown on the map which was a bit confusing. And unfortunately it was not a simple fix because the plugin just wasn't designed that way. In offline mode, it held actions in queue which were then executed when you went back online. So there was no note object in queue to draw on the map until after the upload happened and another download was initiated from the API. Also, the most useful way of using it was to zoom out, let it download notes and then go into offline mode to work. This was not at all intuitive.<br /><br />After receiving a favorable response from the <a href="https://lists.openstreetmap.org/pipermail/josm-dev/2014-August/007033.html" target="_blank">josm-dev mailing list</a>, I began submitting patches in September 2014. Throughout the rest of 2014 I slowly built up behind-the-scenes functionality to download, parse and display notes in JOSM. As of the December release, the notes functionality is actually usable. It is not turned on by default yet because there is still some polishing that needs to be done, especially in the user interface department. But I would love to get some more people testing it. The secret handshake to enable notes in JOSM is as follows:<br /><ol start="0"><li>Download some data. Anything, anywhere. This just initializes a setting that you will change</li><li>Open JOSM preferences</li><li>Go to the advanced preferences</li><li>Search for "notes"</li><li>Change the preference <i>osm.notes.enableDownload</i> to "true"</li><li>Restart JOSM</li></ol><div>In graphical format:&nbsp;</div><div><div class="separator" style="clear: both; text-align: left;"><a href="http://4.bp.blogspot.com/-MkutEB88e4I/VM6zw1NSXcI/AAAAAAAAJAE/lgDUEiMcHzY/s1600/josm_note_screenshot.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-MkutEB88e4I/VM6zw1NSXcI/AAAAAAAAJAE/lgDUEiMcHzY/s1600/josm_note_screenshot.png" height="440" width="625" /></a></div><br /><br />When you open the download dialog after restarting JOSM, you will notice a new "Notes" checkbox alongside the "OpenStreetMap Data" and "Raw GPS data":<br /><div class="separator" style="clear: both; text-align: center;"></div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-hDWQ2l5wVGU/VM7AOul50XI/AAAAAAAAJAg/fl2uHvvUWuc/s1600/josm_note_download_screenshot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-hDWQ2l5wVGU/VM7AOul50XI/AAAAAAAAJAg/fl2uHvvUWuc/s1600/josm_note_download_screenshot.png" height="251" width="400" /></a></div><br />After your first download of notes, you will also need to open the note panel or you won't be able to manipulate notes. Click the note icon in the left hand panel. If you still have the notes plugin installed, there will be two identical buttons. Either figure out which one is which or uninstall the plugin. Here is a view of things:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-ZUl2WVluywU/VM7P7nRUcvI/AAAAAAAAJAw/kujSZ99gTzc/s1600/josm_note_view.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-ZUl2WVluywU/VM7P7nRUcvI/AAAAAAAAJAw/kujSZ99gTzc/s1600/josm_note_view.png" height="490" width="625" /></a></div>The area download limit for the notes API is far greater than the limit on downloading OSM data. You can download 1 square degree at a time if you are only downloading notes. The limit for OSM data is 0.25 square degrees (although realistically you usually run into the 50,000 object limit before you run into the square degree limit). The above screenshot was downloaded in one request. It is about half the state of Kansas.<br /><br />Protip: By default, JOSM will only download 1,000 notes per download request. The OSM API has a hard limit of 10,000 per request. What JOSM asks for is controlled by the&nbsp;<i>osm.notes.downloadLimit</i> setting&nbsp;in the advanced preferences. The other setting&nbsp;worth mentioning is <i>osm.notes.daysClosed</i>. This determines whether or not closed notes are downloaded. If set to -1, it will download all notes ever created within the requested bounding box. If it is set to 0 it will only download currently open notes. If it is set to a positive number, it will download notes which have been closed for that many days. By default, osm.org shows notes that have been closed within the last week. To copy this behavior, this setting should be set to 7. I believe the default right now is 1 day.<br /><br />There is some discussion around these options as well as some of the other polishing that is still needed in the&nbsp;<a href="https://josm.openstreetmap.de/query?status=!closed&amp;component=Core+notes" target="_blank">JOSM bug tracker</a>. And of course that is also where any new issues can be reported.<br /><br />In particular, one of the open tickets which I have already submitted a patch for, but hasn't been accepted yet is the ability to download notes via the note search API. I use this periodically to download all the notes created by Craigslist users (they all start with the word "bounds" so that's the query I send to the note search API) and close all the useless ones.<br /><br />One last screenshot. One of the features of the notes code is that it can read note dump files. It is actually a byproduct of being able to save and load the note layer. Anyway, here is a screenshot of the latest note dump from&nbsp;<a href="http://planet.osm.org/notes/">http://planet.osm.org/notes/</a>&nbsp;loaded into JOSM. All 309,000 of them.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-LgIndMEM91U/VM9DRt0W3GI/AAAAAAAAJBA/bDFIQRxahSM/s1600/Screenshot%2B2015-02-02%2B03.21.28.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-LgIndMEM91U/VM9DRt0W3GI/AAAAAAAAJBA/bDFIQRxahSM/s1600/Screenshot%2B2015-02-02%2B03.21.28.png" height="360" width="625" /></a></div><br /><br />Happy noting!<br /><br />P.S. Dear iD: <a href="https://github.com/openstreetmap/iD/pull/1895" target="_blank">Your move</a>!


guid: tag:blogger.com,1999:blog-7044416767097884399.post-4415861582832030737
url: http://ksmapper.blogspot.com/2014/02/workflow-for-fixing-county-borders.html
title: Workflow For Fixing County Borders
published: 2014-02-05 18:32:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.803462
updated_at: 2019-10-25 14:30:27.158959
summary:
  Well, <a href="http://ksmapper.blogspot.com/2014/02/county-borders-in-openstreetmap.html" target="_blank">my last post</a> got some requests for workflow and I got another snow day off from work today so why not! Follow along as I fix Wyoming's county borders.<br /><div><br /></div><div>Two things you will need to have open: JOSM with the mirrored_download plugin installed and the Wikipedia page for the list of counties in the state. Do a search on wikipedia for "List of counties in &lt;state&gt;" to find it. Here is the one for <a href="http://en.wikipedia.org/wiki/List_of_counties_in_Wyoming" target="_blank">Wyoming</a>.</div><div><br /></div><div>Next you need to download all the county relations and their members. I do this using the <a href="http://wiki.openstreetmap.org/wiki/Overpass_API" target="_blank">Overpass API</a>. JOSM has a "Download from Overpass API" option in the File menu (Update: turns out this is a feature of the mirrored_download plugin, not in JOSM core) but we aren't going to use that one just yet because it always adds a bbox parameter to the query which we don't want in this one. Instead, use the "Open Location" option and paste in the following URL after changing the 56 to the appropriate FIPS code for your state which should be listed on the Wikipedia page.</div><pre class="source-code">http://overpass-api.de/api/interpreter?data=relation["nist:state_fips"="56"];&gt;&gt;;out meta;<br /></pre><div><br />This tells the Overpass API to return all relations with a nist:state_fips=56 tag plus all of the relation members (&gt;&gt;) and finally to add OSM metadata that is needed to edit and upload (out meta). This includes things like version number, last edit timestamp and last user to touch the object. By default it excludes this metadata to reduce download size.<br /><br />After that finishes you should have a nice outline of the state and all its counties. Make sure the number of downloaded relations matches the number of counties in the state. If not, it probably means the nist:state_fips tag is missing from some counties. If you can mange to find the relation through another means, download it by ID using the "Download object" option in JOSM. Or... I don't know... Ping me on <a href="http://irc.openstreetmap.org/" target="_blank">IRC</a> I guess :)<br /><br />Next, let's load up all the place nodes in the state. For this I use the "Download from Overpass API" option in JOSM. Start off by drawing a box around the desired state on the map. So far I have only done mostly square states. If your state is an odd shape then you can download the bounding box and then select things outside of your state and use the JOSM "Purge" option in the edit menu to remove them from your dataset. Or just ignore them. Then paste this in this text into the query box:</div><pre class="source-code">[timeout:300];node["place"];out meta;<br /></pre><div>Now the screen will look a little busy so I set up a filter to only show me place=county nodes until I need the other stuff. To do this, create a new filter with the filter string</div><pre class="source-code">place=county or type:way or type:relation<br /></pre><div>This will initially hide the county borders and nodes that you want to see so after the filter is created, make sure all three checkboxes in the filter panel are checked. The far right one will invert the filter so instead of hiding whatever matches the filter string it will hide everything <b>except</b> the things that match the filter string. The second checkbox on the left makes them completely disappear instead of just making them inactive but still visible. The first checkbox can be used to quickly disable the filter when you need to see everything.<br /><br />At this point my JOSM window looks like this:<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-ctczbpgXw5E/UvJnp6LxJoI/AAAAAAAAFQw/3C4C-qnFlEw/s1600/WY_counties.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-ctczbpgXw5E/UvJnp6LxJoI/AAAAAAAAFQw/3C4C-qnFlEw/s1600/WY_counties.png" height="384" width="632" /></a></div><br />Note the "Relations: 23" on the relation toolbox matches the expected number of counties in Wyoming.<br /><br />At this point I usually remove the is_in tag from all the place=county nodes just to get that out of the way. Do a search (CTRL-F) for "type:node place=county" and it should select all of the county nodes. If you have some for a neighboring state loaded that you don't want to touch you can deselect them by holding CTRL while clicking. Then just delete the is_in tag.<br /><br />Now for the real meat. I just go down the list of counties in Wikipedia. Bring up the first county's Wikipedia page and copy the page title for use in the wikipedia=* tag and note the name of the county seat.<br /><br />Now in JOSM hit CTRL-F to bring up the search dialog and search for the county seat. If you are lucky it will find exactly one match. Note that because of the filter, you won't actually see the selected object but it is still selected! This could be considered a bug but in this case it works out. If the search found multiple things then you will have to disable the filter and find the right one.<br /><br />Now:<br /><ul><li>Hold down shift and select the county node in addition to the city node that is already selected</li><li>Select the county relation in the relation list and click the "Edit" button</li><li>Add the two selected nodes to the bottom of the relation member list using the 4th button down on the lower right panel of the window</li><li>Add the role "admin_centre" to the city node</li><li>Add the role "label" to the county node</li><li>Add the alt_name tag to the relation in the top panel</li><li>Add the wikipedia tag. Remember how you copied the page title from Wikipedia? Just type "en:" and then paste the title.</li></ul>Here is a shot of Albany county (Wyoming, not New York!) when I'm done with it:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-mDtYRRPR9rg/UvJzaIDPNbI/AAAAAAAAFRA/joJTFCWHazQ/s1600/Relation_editor.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-mDtYRRPR9rg/UvJzaIDPNbI/AAAAAAAAFRA/joJTFCWHazQ/s1600/Relation_editor.png" height="615" width="632" /></a></div><br /><br />And that's pretty much it. Now just repeat the process through the list of counties.<br /><br />After I'm done I go back and look for any place=hamlet or place=village county seats and bump them up to place=town. I do this by doing CTRL-A in the relation list, then right clicking and doing a "Select members" which selects all the relation members. Then I use the find dialog to find all place=village and make sure to select the "find in selection" radio button so you are only finding things that are already selected. Otherwise you will promote ALL villages in the whole state to towns! As a sanity check, use the selection toolbox to make sure the number of objects selected matches your expectation. It should be fewer than the number of counties in the state.</div><br /><div>I will leave you with a quick video of me editing the next county in Wyoming.</div><iframe allowfullscreen="" frameborder="0" height="375" src="//www.youtube.com/embed/8CAt9Uec5R4" width="500"></iframe>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-8114342771946681699
url: http://ksmapper.blogspot.com/2014/02/county-borders-in-openstreetmap.html
title: County Borders in OpenStreetMap
published: 2014-02-05 07:21:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.807499
updated_at: 2019-10-25 14:30:27.161882
summary:
  I have had a fairly steady relationship with county borders in OSM. They were originally imported before I joined the project in 2010. However they were imported as overlapping closed ways which is not ideal. In some places people had started working on de-duplicating the ways and turning them into relations. However this is tedious work and some of the people who attempted it didn't quite understand how relations worked so it was pretty much a big mess.<br /><br />Throughout 2011 and 2012 I periodically came back and worked on county borders and eventually got to a place where there was a relation for every county in the nation. There are still some minor differences in tagging but I wasn't as concerned with that and tried not to change the work of other mappers as long as the relation existed and had valid geometry. At the same time I also reduced complexity at state borders. There used to be a mess of admin boundary ways. One for the state, one for each of the counties on either side of the state border and possibly also some city boundaries. They usually didn't match up very well either so it looked terrible. I didn't mess with cities but at least I made the state and county borders share a single way to reduce clutter.<br /><br />At some point I also noticed (thanks to the MapQuest Open rendering which renders county names) that the place=county nodes were all placed at the extreme eastern border of all the counties. This caused the county names to be rendered in unexpected locations. So I ended up fixing those as well. Here is a screenshot of the last changeset where I moved over 1,100 nodes to the middle(ish) of their respective counties:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-cSoG666Bk9s/UvE9ZqZ6wOI/AAAAAAAAFOI/ts6ch7_tfEU/s1600/county_nodes_move.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-cSoG666Bk9s/UvE9ZqZ6wOI/AAAAAAAAFOI/ts6ch7_tfEU/s1600/county_nodes_move.png" height="425" width="632" /></a></div><br /><br /><h4>The Effects</h4>So that was nice. But there was still some weirdness with counties, particularly when geocoding in Nominatim. Especially if there was a city with the same name as a county. For example, I searched for "Kearney, NE" and the first result was the place=county node for Kearney County in Nebraska. What I was looking for and expected to find first was the city of Kearney which is in Buffalo county. Then there is the issue of Nominatim finding both the county node and the boundary relation when you search for a county. The same thing often happens with cities. Really, they are the same thing and should show up as one result. And sometimes it might. Nominatim tries to be smart about this and does remarkably well but sometimes it needs help.<br /><br />You might say we should just delete the node since we already have the boundary information. But very few renderers use boundary relations to render place names from. They all rely on these nodes. The same goes for cities, states and even countries I believe. There is also the oddness of what to use for county name. The relations all have a "name=Kearney County" tag while the nodes just have "name=Kearney".<br /><br />After poking around a little and bugging&nbsp;<a href="http://wiki.openstreetmap.org/wiki/User:Lonvia">lonvia</a>&nbsp;(maintainer of nominatim.osm.org) about it a couple of times, I learned a few things about Nominatim and was able to make some fixes. It was my intent to come back to this some day and figure it all out and fix up county relations once and for all. I still hadn't gotten around to that but just in the last few days, new user <a href="http://www.openstreetmap.org/user/revent">revent</a>&nbsp;was getting annoyed at the geocoding problems in Texas and really dove into Nominatim to figure out the best practices. He wrote up a <a href="http://www.openstreetmap.org/user/revent/diary/20934" target="_blank">diary entry</a> on the subject and we have since then talked together some more on IRC and since I had work cancelled today because of snow I thought I would write up this post about my current understanding of things:<br /><ul><li>Nominatim is able to "link" multiple elements into a single feature. It tries to do this automatically but doesn't always get it right.</li><li>Nominatim uses Wikipedia to help determine the importance of a place.</li><li>Nominatim makes use of the alt_name tag.</li><li>After editing the map, Nominatim does not always fully reindex an area. It might depend on current workload of the server so things can seem inconsistent. But a manual reindex can be forced by an admin after a large area has undergone a lot of edits.</li></ul><br />The linking is important because this allows the county border and the node to be linked into a single feature so that only one result is returned when you search for a particular county. It also improves addressing since Nominatim no longer has to choose between two different things to associate addresses with. This linking can be accomplished by adding the node to the county border relation with a role of "label".<br /><br />The use of Wikipedia is interesting and (I think) the source of my problems with Kearney, NE that I mentioned above. Nominatim was linking the county node to the <a href="http://en.wikipedia.org/wiki/Kearney,_Nebraska">Kearney, Nebraska</a>&nbsp;Wikipedia page based on name tag. This helped bump its importance up to where it was returned before the city. Now that I have added the Kearney county node to the Kearney county relation and specified a "wikipedia=en:Kearney County, Nebraska" tag, it links things to the correct Wikipedia page and the search results return the city first, as expected.<br /><h4><br /></h4><h4>The Fix</h4>So to summarize, I believe revent and I have settled on the ideal way to map a county. Using Kearney county as an example:<br />One relation with the following tags:<br /><pre class="source-code">type=boundary<br />boundary=administrative<br />admin_level=6<br />name=Kearney County<br />alt_name=Kearney<br />wikipedia=en:Kearney County, Nebraska<br /></pre><br />In addition, the following tags are in widespread use:<br /><pre class="source-code">border_type=county<br />nist:state_fips=31<br />nist:fips_code=31099<br /></pre><br />I'm not sure the border_type tag is actually used for much but it has a fair amount of existing use and seems to further differentiate borders that are the same admin_level based on local conventions - county vs borough and such. The two nist: tags were part of the import and I have actually used the state_fips as an easy way to download all county relations in a given state to work on. These FIPS codes have been withdrawn by NIST recently but are still commonly used. We may need to look at replacing or augmenting these tags with an ISO standard at some point but for now they work.<br /><br />In addition to the relation tags the following relation memberships are recommended:<br /><ul><li>Obviously the ways that make up the border with a role of "outer" in the correct order so that they form a closed ring</li><li>The node with the following two tags: place=county and name=Kearney with a role of "label"</li><li>The place=city/town node of the county seat with a role of "admin_centre"</li></ul><br />The county seat city node as an admin centre (yes, use the British spelling) is kind of a new thing to me. I'm not 100% sold on it but it does seem like useful information to have.<br /><br />If you have looked at very many county nodes you might notice that the "is_in" tag is conspicuously absent. It turns out, it causes more problems than it solves and should probably be left off or removed. I have always found it a bit silly anyway. OSM is a geodatabase - there is no need to indicate location in a tag.<br /><br />I have also been promoting any county seats that were place=village to place=town. I know I saw this as a thing on the OSM wiki at one point although I can't find it right now. While they may not meet the population guidelines of a "town" according to the wiki, I think county seats carry extra importance due to their governmental functions. It also means that at least some city names are rendered on maps at low zoom levels in low population areas. Without this promotion you could pan around in the Kansas/Nebraska/Dakotas area for several screens at zoom level 9 or 10 and not see a single city name on the map on osm.org. In some ways this is a rendering issue but I don't feel like this is an egregious case of tagging for the renderer.<br /><br /><h4>Your Turn!</h4>So if you are looking for something to do, jump in and make sure your county is correctly linked and tagged. I have already done Kansas, Nebraska and South Dakota but I probably won't have time to do everything in the near future and I will concentrate on lower population areas first so please feel free to jump in! If there is interest, I may do another post about my workflow to do a state-wide edit. Otherwise you can always find me on the #osm or #osm-us <a href="http://irc.openstreetmap.org/" target="_blank">IRC channels</a> if you have questions.<br /><br /><b>UPDATE!</b>&nbsp;I have since written another post detailing my workflow: <a href="http://ksmapper.blogspot.com/2014/02/workflow-for-fixing-county-borders.html" target="_blank">http://ksmapper.blogspot.com/2014/02/workflow-for-fixing-county-borders.html</a><br /><br />Incidentally, a lot of the same concepts apply to cities as well. If you do a Nominatim search for "<a href="http://nominatim.openstreetmap.org/search.php?q=Los+Angeles%2C+CA" target="_blank">Los Angeles, CA</a>" right now it will return two results. One for the node and one for the boundary relation. Making the node a member of the relation should fix that.<br /><span style="font-size: x-small;">(sigh. In doing that search I just saw that the relation for LA is completely broken anyway. Admin boundaries suck.)</span><br /><span style="font-size: x-small;"><br /></span> In closing, here is a screen shot of the completed county relations in South Dakota:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-2ILEFfM1e_8/UvHkmYVvEJI/AAAAAAAAFO4/yrarFtR5VJo/s1600/SD_counties.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-2ILEFfM1e_8/UvHkmYVvEJI/AAAAAAAAFO4/yrarFtR5VJo/s1600/SD_counties.png" height="425" width="632" /></a></div>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-6006119363182568753
url: http://ksmapper.blogspot.com/2013/01/too-much-of-good-thing.html
title: Too Much of a Good Thing
published: 2013-01-23 07:13:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.811290
updated_at: 2019-10-25 14:30:27.166615
summary:
  What makes OpenStreetMap great? Well there may be multiple answers from different people. But one thing is certain: Our geodata is awesome. I mean it is kind of the reason that we exist. So more is better, right? We are, after all, trying to map the entire planet. <br /><br />But it turns out, there can actually be too much of a good thing. One of my first encounters with such a situation was while doing some fixing up of imported TIGER data somewhere out west. I came across this:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-8wUDDLz28FM/UO5PohggKAI/AAAAAAAABkA/FccY7q8W7SI/s1600/TIGER_exponential_nodes.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="448" src="http://1.bp.blogspot.com/-8wUDDLz28FM/UO5PohggKAI/AAAAAAAABkA/FccY7q8W7SI/s640/TIGER_exponential_nodes.png" width="635" /></a></div><br />Notice the scale. We are looking at a little over 100 meters of country highway and there are over 300 nodes in this image. This is obviously beyond ridiculous and I can assure you that this particular instance was cleaned up long ago and those nodes met their appropriate end in a special level of /dev/null.<br /><br />But this is not an isolated incident. I keep coming across such over-noding. So I decided to use my local planet database (which I <a href="http://ksmapper.blogspot.com/2011/04/parse-planet-into-pgsnapshot-database.html" target="_blank">previously blogged</a> about setting up) to try and find out how much this really happens and where such things come from.<br /><a name='more'></a><br />Well, the "ways" table in the database is 85 GB in size and has over 165 million entries in it. Doing anything with the whole data set tends to be slow. So I decided to filter out some things. While discussing it on IRC, Andrew Buck suggested starting by ignoring ways with fewer than 10 nodes in them. After some consideration this actually seemed reasonable. It is hard to over-node something with 10 nodes. So I made a new table with only these ways in it and with only a subset of columns to make it smaller and more manageable. This actually ended up eliminating a lot more ways than I anticipated. It turns out, 75% of the ways in the OSM database have fewer than 10 nodes in them. This left me with about 40.5 million ways to investigate.<br /><br />The next question is what does it mean to be "over-noded" anyway? Since this is a geospatial database, I can use functions to determine the length of a given way. And I also know how many nodes each way has in it. So a logical unit is nodes per meter. <br /><br />Next up is to determine what is "normal" for typical mapping. Taking the 40 million ways with more than 10 nodes, I ran some basic statistical functions and found the following numbers:<br />Minimum nodes/meter: 0.00000282<br />Median nodes/meter: 0.043<br />Average nodes/meter: 0.088<br />Maximum nodes/meter: 65.23<br /><br />Well that is quite a range. Let's see what is going on here. The way with the fewest nodes/meter turns out to be <a href="http://www.openstreetmap.org/browse/way/105818922" target="_blank">way 105818922</a> which is part of the border of Indonesia. It has 14 nodes and stretches almost 5,000 kilometers, averaging one node every 350 kilometers. It could maybe use a little more refinement but it's a national border in the middle of the ocean... it is probably not too bad.<br /><br />On the other end of the spectrum is <a href="http://www.openstreetmap.org/browse/way/44342937" target="_blank">way 44342937</a>. It supposedly represents a round "building" with a diameter of about 1.5 meters. It has 339 nodes. What the... <b>WHAT?!</b> What is this, the world's most accurately mapped dog house? Since it is likely that this way is going to be deleted in the very near future, I include here a screen shot of it loaded up in JOSM:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-NtISA2J1tcE/UO5o7s4CLpI/AAAAAAAABkQ/h7HTXMOBfw8/s1600/most_overnoded.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="327" src="http://1.bp.blogspot.com/-NtISA2J1tcE/UO5o7s4CLpI/AAAAAAAABkQ/h7HTXMOBfw8/s400/most_overnoded.png" width="400" /></a></div><br />Yes, that is a tiny circle in the middle of someone's back yard. All that red around it is JOSM trying to draw its normal way direction arrows on such a ridiculous object.<br /><br />Now that we have examined both ends of the spectrum, what's in the middle? Well, there is <a href="http://www.openstreetmap.org/browse/way/101521179" target="_blank">way 101521179</a>. 174 meters long with 14 nodes which yields a nodes/meter reading of 0.08000019. That is... well... entirely reasonable. The fact that the median is half of the average would seem to indicate that the numbers are skewed towards the high end by a comparatively few number of ways with a very high number of nodes - like that one with 65 per meter.<br /><br />So if 65 nodes/meter is the extreme upper limit and most people map at less than 0.1 then what exactly is "too much"? Well I don't really know but I'm just going to throw out a number for further analysis. Let's make it... 1 node per meter. This limits the number of ways down to about 105 thousand.<br /><br />Since my database has geographic knowledge about these ways and I have the number down to a reasonable level, let's hook up&nbsp;<a href="http://www.qgis.org/" target="_blank">QGIS</a>&nbsp;to it and make a map of the ways with more than 1 node per meter. Let's start with the US:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-fJlA8bkjm2w/UPrnIj5SzkI/AAAAAAAABpI/ygwGpcA-JLE/s1600/over_noded_US.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="448" src="http://2.bp.blogspot.com/-fJlA8bkjm2w/UPrnIj5SzkI/AAAAAAAABpI/ygwGpcA-JLE/s640/over_noded_US.png" width="635" /></a></div><br />Well that's actually not as dense as I was expecting. It looks like some of the TIGER over-noding might actually come in at under 1 node/meter. Which kind of makes sense if you think about it. A lot of TIGER ways are really long and the over-noding is sometimes clumped into one section so on average the nodes/meter ratio will be less than 1. But for fun let's look at what other interesting things might be in this data set. How about Europe?<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-Ei22FD6v07I/UProXdkS_pI/AAAAAAAABpY/AaI4ONjTwbE/s1600/over_noded_EU.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="449" src="http://3.bp.blogspot.com/-Ei22FD6v07I/UProXdkS_pI/AAAAAAAABpY/AaI4ONjTwbE/s640/over_noded_EU.png" width="635" /></a></div><br /><br />Well now that's different. What's with the outbreak in France?! I have my suspicions. Let's see how this works out...<br /><br />There is a "source" tag that is often used in OSM data to indicate where a certain map object came from. It is especially often used during imports. Technically it usually makes more sense to put the source tag on the changeset instead of the map object itself, especially for large imports but a lot of people still put it on the map objects anyway. So let's see what these over-noded ways reveal in their source tags. Here are the results of a query that groups things by source tag and counts how many ways with a node/meter ratio of over 1 have that source tag. I truncated the source information at 70 characters for display purposes.<br /><br /><pre class="source-code">&nbsp;WAYS  | SOURCE<br />-------|-----------------------------------------------------------------------<br /> 62995 | cadastre-dgi-fr source : Direction Générale des Impôts - Cadastre. Mis<br /> 22713 | <br />  3535 | extraction vectorielle v1 cadastre-dgi-fr source : Direction Générale <br />  1313 | Bing<br />  1045 | 3dShapes<br />   744 | bing<br />   687 | Kolding Kommune<br />   628 | NHD<br />   608 | dcgis<br />   602 | WakeGIS<br />   585 | WroclawGIS<br />   526 | Planimetria de Vitoria<br />   482 | MassGIS Buildings (http://www.mass.gov/mgis/lidarbuildingfp2d.htm)<br />   454 | http://www.bakersfieldcity.us/gis/downloads/gis_spatial_data.htm<br />   393 | NextView<br />   390 | Regione Emilia Romagna<br />   369 | cadastre-dgi-fr source : Direction Générale des Impôts - Cadastre ; mi<br />   349 | kapor2<br />   336 | Bing Sat<br />   324 | SO!GIS Import<br />   260 | geoimage.at<br />   244 | Regione_del_Veneto_LR28_16.7.1976_Formazione_CTR_auth_39164-5700-1100_<br />   230 | http://www.sogis.ch SO!GIS Import<br />   225 | MGC<br />   214 | Kreis_Viersen_Katasteramt_2012_06<br />   214 | lukr<br />   206 | Ajuntament de Girona<br />   184 | CCH<br />   183 | NRCan-CanVec-10.0<br />   175 | Orthophotos 2011 du SITG (Système d'Information du Territoire Genevois<br />   133 | DEP Wetlands (1:12,000) - April 2007 (http://www.mass.gov/mgis/wetdep.<br />   128 | vuv:dibavod:a05<br />   125 | http://www.roseville.ca.us/services/maps_n_data/data_clearinghouse.asp<br />   115 | City of Kamloops<br />   112 | cadastre-dgi-fr source : PaysDeBrest - 20100331<br />   105 | OS_OpenData_VectorMapDistrict<br /></pre><br />Well then. I recognize that top source tag as the ongoing import of french building outlines and some other features from some cadastre data they got their hands on over there. A spot check shows that some of this isn't actually because of excessive nodes being used but rather the odd way in which the import is creating building geometries. For example, here is&nbsp;<a href="http://www.openstreetmap.org/browse/way/67157454" target="_blank">way&nbsp;67157454</a>.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-otQ5Y7wm0IA/UPx2hj3kfRI/AAAAAAAABuc/1hmKTcG0Qcs/s1600/overnoded_FR.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="429" src="http://3.bp.blogspot.com/-otQ5Y7wm0IA/UPx2hj3kfRI/AAAAAAAABuc/1hmKTcG0Qcs/s640/overnoded_FR.png" width="635" /></a></div><br /><br />Note the wall=no tag. I believe this implies that it is some kind of porch or veranda. It has a rounded front which is where all the nodes are that make it have a nodes/meter ratio of just over 1.0. If a human mapper had mapped this building it would have likely been a single way that included the porch as well as the rest of the house instead of the 8 individual areas that the import created. This would have resulted in a single 70 meter long way with 47 nodes which comes out to 0.67 nodes/meter.<br /><br />Checking a few more of these ways in France, it does look like some care was taken to prevent over-noding. Most of them are just barely over 1.0 nodes/meter. And at this point it should also be noted that closed ways are actually having their first/last node counted twice because of the way the database stores the node membership information. Since I was originally looking for over-noded highways I didn't take this into account. So technically a lot of these French buildings might be just under 1 node/meter. But the fact that there are so many of them right on this (arbitrary) limit is still interesting.<br /><br />There are a few other source tags I recognize as well. CanVec is imported from Canadian government data. NHD is the National Hydrography Dataset here in the US from which some people have imported rivers and lakes. I know several of these water feature imports have had problems with ridiculous over-noding. Some of it has been fixed but a lot remains. MassGIS is the&nbsp;Massachusetts&nbsp;GIS office which has been used for some local imports. In particular, the most over-noded way that I showcased above is from this MassGIS building import.<br /><br />Of course the second most popular source tag is blank which doesn't tell us much. A tiny random sample shows there are a couple of imports that didn't use any source tag and some are just very detailed manual mapping.<br /><br />I think finding what I was originally looking for (over-noded TIGER ways) is going to take some more digging. This post kind of got hijacked by the big red blob in France but I think I'll call it quits for now and do some more poking to find the TIGERs I'm after. If there is a lesson so far, I think it can be summed up as:<br /><br /><ol><li><b>Sanity check your imports!</b> Tiny objects with a large number of nodes are an obvious sign that there is something weird going on in your conversion to .osm format. It is possible that the object was represented as a parametric curve in the source and the conversion to OSM format tried to recreate that as closely as possible within the constraints of our x/y coordinate system.</li><li><b>Imports are not human data.</b> Even in France where this building import is generally viewed as a good thing and a lot of checking and verifying of the data has taken place, the data is still very distinctly different from most of the OSM database that has been created by hand. This may not be a bad thing in every case but it is definitely something to think about when proposing and executing an import.</li></ol>Lastly, I leave you with one more way. As I was randomly browsing the 105 thousand ways, I happened to click on way number&nbsp;<a href="http://www.openstreetmap.org/browse/way/129485933" target="_blank">129485933</a>&nbsp;which has 1.00526 nodes per meter. (again, with the first/ last node double counted) If you look back at version 1 and 2 you will see a certain someone's user name. Turns out I am the culprit! This way was originally created on my phone during the <a href="http://2011.stateofthemap.org/friday-night-fun-rockies-game/" target="_blank">baseball game</a> we attended during SOTM 2011 in Denver. How random is that?


guid: tag:blogger.com,1999:blog-7044416767097884399.post-4177457460665795371
url: http://ksmapper.blogspot.com/2012/11/preparing-for-operation-cowboy.html
title: Preparing for Operation Cowboy
published: 2012-11-21 17:10:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.814688
updated_at: 2019-10-25 14:30:27.169640
summary:
  If you haven't heard, there's a major mapping event about to happen. Dubbed <a href="http://wiki.openstreetmap.org/wiki/Operation_Cowboy" target="_blank">Operation Cowboy</a>, this will be the second event of its kind, the first being <a href="http://wiki.openstreetmap.org/wiki/Night_of_the_living_maps" target="_blank">Night of the Living Maps</a> which happened back in February. As described on the wiki, these events are kind of like a global LAN party for maps. For one weekend people who choose to participate will focus their mapping efforts on a common goal. This time the organizers decided to focus on the USA. <br /><br />We of course have plenty of mapping to do around here. Cleaning up and updating the TIGER road import is kind of a never-ending task. In some of the sparsely populated areas of the country there aren't really any local mappers to speak of so there is ample work for "armchair mappers" to come in and do some map maintenance.<br /><br />Considering the audience and the event, I decided to try and get a few things tweaked before Operation Cowboy goes down.<br /><br /><h3>TIGER 2012 road name tiles</h3><br />First up was improving the <a href="http://wiki.openstreetmap.org/wiki/TIGER_2012" target="_blank">TIGER 2012 road name tiles</a> being served up by the OSM-US tile server. They were originally set up by Ian Dees and are a great resource for tracing roads that were built since the original TIGER data was imported. The only issue I had with them is that they show the street name in the Census Bureau's abbreviated format. In OSM we prefer to use unabbreviated names because it reduces ambiguity. Does "St" mean "Street" or "Saint"? While that one might not be too hard to figure out and expanding "Ave" to "Avenue" might be trivial, there are a lot of abbreviations in the data and some of them are not easy to guess if you aren't familiar with the area.<br /><br />Any guesses as to what the following mean? Brg. Cll. Natl Lkshr. Ofc. Pso. RM. BIA Rte? <span style="font-size: xx-small;">(see answer key at end of post)</span> Most people in the US can probably guess some of them but there are definitely some regional ones in there that many people may not be familiar with. Then add in abbreviated directional prefixes. And now suppose you are a European trying to figure all of this out during Operation Cowboy. It's going to look like a secret code.<br /><br />I won't go into the technical details in this post (might save that for another one after I clean up the process doumentation) but it did involve 27 GB of data in 3,200 .dbf files, about 10 different command line utilties and several large SQL queries. The end result was a mapping of 10.8 million line IDs (linearid field in TIGER speak) to fully expanded road names which I applied to the rendering database. The result can be seen on this <a href="http://tile.openstreetmap.us/tiger2012_roads_expanded/preview.html#16/37.8112/-122.2747" target="_blank">tile preview page</a>.<br /><br /><h3>Imagery preset updates</h3><br />Related to the road name tiles, I noticed that Potlatch2 had a TIGER road name tiles preset in the "Background" menu but it was still pointing at tiles with 2011 data in them. Thanks to the magic of github, I was able to <a href="https://github.com/systemed/potlatch2/pull/90" target="_blank">get that fixed</a> easily.<br /><br />Then I noticed that the <a href="http://josm.openstreetmap.de/wiki/Maps" target="_blank">JOSM imagery sources</a> page didn't have this tile set listed at all. That one was a little trickier to fix. It involved coming up with a simplified polygon of the US border to define the available extent of the tile layer as well as putting a base64 encoded image into an XML tag on the JOSM wiki. The image+base64+XML+wiki thing seems a little nasty but the JOSM functionality that this enables is actually pretty neat. If you download OSM data inside of the US, it will pop up a notice that informs you that that there is additional background imagery available and lets you add it to the "Imagery" menu with a single click. The base64 encoded image is displayed next to the menu option. In this case it is the TIGER logo:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-Qx5r646Pjec/UKxy_pMqK6I/AAAAAAAABas/NpOb_1FOc7s/s1600/smalltiger.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-Qx5r646Pjec/UKxy_pMqK6I/AAAAAAAABas/NpOb_1FOc7s/s1600/smalltiger.gif" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">RAWR!</td></tr></tbody></table><span id="goog_1398736279"></span><br /><h3>TIGER tag removal</h3><br />The TIGER road import put some tags on imported objects that weren't really necessary. As people edit roads, some of the tags become even less useful as the information from the tiger tags diverges from the "normal" OSM tags. Some of the tags were useful at the time for the upload process itself but are now just annoying bloat that confuses new users.<br /><br />Back in July I started a <a href="http://lists.openstreetmap.org/pipermail/talk-us/2012-July/008830.html" target="_blank">mailing list thread</a> about having editors automatically drop some of these tags like they already drop the created_by tag. Out of that discussion came a JOSM <a href="https://josm.openstreetmap.de/ticket/7915" target="_blank">trac ticket and patch</a> that does exactly this. Right before uploading, JOSM checks to see if any of the modified objects it is about to upload have "discardable" TIGER tags on them and removes them.<br /><br />In September, I opened <a href="https://trac.openstreetmap.org/ticket/4552" target="_blank">a ticket</a> against Potlatch2 with the same feature request. Since nothing had happened with that ticket and I already had the Potlatch2 repository cloned from the tile URL change pull request, I decided to take matters into my own hands. I had never so much as glanced at ActionScript but fortunately P2 was a breeze to get running in a development environment. The <a href="http://wiki.openstreetmap.org/wiki/Potlatch_2/Developer_Documentation" target="_blank">developer documentation page</a>&nbsp;on the wiki got me up and running in a matter of minutes. Although I did have to give ant more PermGen space to get through compilation. I should probably update the wiki with that tidbit...<br /><br />After a little over an hour I had <a href="https://github.com/systemed/potlatch2/pull/91" target="_blank">another pull request</a> fashioned which Richard was kind enough to merge the next day. I believe this went live on osm.org within the last few hours. So now every way that is touched during Operation Cowboy will help to clean this clutter out of the database without the user having to do anything.<br /><br /><h3>YEEEEHAW!</h3><div><br /></div><div>So that's how I prepared for Operation Cowboy. Unfortunately the actual event will take place over the Thanksgiving weekend which means I will be in Nebraska with very limited internet access so I won't be able to participate much with actual mapping. I will probably manage to hang out in the IRC channel a bit to try and field some questions if they come up. Please join me! (#opc2012 on irc.oftc.net)</div><div><br /></div><div><br /></div><div><span style="font-size: x-small;">Answer key for TIGER abbreviations:</span></div><div><div><span style="font-size: x-small;">Brg = Bridge</span></div><div><span style="font-size: x-small;">Cll = Calle (Spanish)</span></div><div><span style="font-size: x-small;">Natl Lkshr = National Lakeshore</span></div><div><span style="font-size: x-small;">Ofc = Office</span></div><div><span style="font-size: x-small;">Pso = Paseo (Spanish)</span></div><div><span style="font-size: x-small;">RM = Ranch to Market Road</span></div><div><span style="font-size: x-small;">BIA Rte = Bureau of Indian Affairs Route</span></div></div>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-6660965256320040292
url: http://ksmapper.blogspot.com/2012/10/licensed-to-map-what-happened-to-la.html.html
title: Licensed to Map (What happened to Los Angeles?!)
published: 2012-10-24 03:53:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.818491
updated_at: 2019-10-25 14:30:27.172623
summary:
  I recently got back from the <a href="http://stateofthemap.us/" target="_blank">State of the Map - USA</a>&nbsp;conference. It was great and you should have been there! But for those who weren't... I presented one session on Saturday afternoon with the same title as this post. Instead of just putting my slides out there I thought I would write a blog post to tell more of the story. This is a fairly long post but don't worry, it has a lot of pictures! (click to see full resolution)<br /><br />So, the license change. It happened. We lost some data. But what happened before that to try and save as much as we could? And what exactly did we lose?<br /><br />First, a brief timeline:<br /><ul><li>Many moons ago, the OSM Foundation voted to change our license from Creative Commons to the Open Database License. This actually happened before I knew that OSM existed.</li><li>In order to make the change, permission had to be secured from everyone who had contributed map data to the database.</li><li>Contributions from anyone who did not agree to the new terms had to be removed from the database.</li><li>At the end of 2011 the board set April 1st, 2012 as a target date for the license switch. This turned out to be wildly optimistic but at least it was a concrete goal to shoot for instead of just an ongoing process with no end in sight.</li><li>The process of removing non-relicensable data was done by a bot that went through the entire database in the 2nd half of June</li><li>After a little more cleanup, the first ODbL planet file was finally delivered in September.</li></ul>Even though the April 1st target date was not really reasonable to hit, (especially in hindsight of course) it still gave the community something to work towards. And work we did.<br /><br /><h2><span style="font-size: large;">Contacting inactive mappers</span></h2>The first priority was to contact undecided users to make them aware of the change and that they needed to log in and indicate a decision. There were several rounds of emails sent out from the foundation to undecided users, trying to get them to respond. In addition to the emails from the foundation, users actively contacted other people in their area to try and make them aware of the decision. Lastly, the foundation supplied the account email address of some of the accounts with the most map data to a small group of volunteers for more targeted contact.<br /><br />The result of this contact effort can be seen in this graph that I have been presenting since the beginning of the process:<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-TnzlkN1nvXQ/UH4qdtYkgOI/AAAAAAAABUo/5KjReAixVZs/s1600/osm_license_acceptance.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="217" src="http://4.bp.blogspot.com/-TnzlkN1nvXQ/UH4qdtYkgOI/AAAAAAAABUo/5KjReAixVZs/s640/osm_license_acceptance.png" width="640" /></a></div><br />The green line&nbsp;corresponds&nbsp;to the scale on the left while the red line follows the scale on the right. Of those who responded, it was a 99.5% landslide in favor of&nbsp;re-licensing their data. You can see that the people who were opposed to the change were very quick to log in and enter their decision. The big bumps in the green line clearly show when the mass emails were being sent out by the foundation.<br /><a name='more'></a><br />I was the volunteer who did targeted contact here in the U.S. It was an interesting project. <a href="http://wiki.openstreetmap.org/wiki/User:SimonPoole" target="_blank">Simon Poole</a> came up with a prioritized list of users to contact per country/region based on his site&nbsp;which&nbsp;shows undecided users ranked by how much map data they have in the database. (<a href="http://odbl.poole.ch/" target="_blank">http://odbl.poole.ch/</a>)<br /><br />I ended up contacting people through a variety of websites including obvious social networking sites like Facebook, Google+, Twitter, Flickr and LinkedIn. I also sent a few though more obscure sites like tripadvisor and soundcloud. Beyond that, I attempted to determine their real name and any contact info I could get my hands on. A simple web search with their OSM user name plus the location of their first edit was remarkably effective. Often it would yield a twitter account which would then give me a real name and lead to accounts on other sites. Here is a screenshot of the spreadsheet I used to track my contact with users:<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-4qTQLT3RbwQ/UH4xE1HaTWI/AAAAAAAABU4/wLhS0odL_CU/s1600/user_contact_list.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://1.bp.blogspot.com/-4qTQLT3RbwQ/UH4xE1HaTWI/AAAAAAAABU4/wLhS0odL_CU/s400/user_contact_list.png" width="342" /></a></div><br />I kept track of dates I sent messages, when I heard back and any accounts around the internet that were possibly related to the user in question. All in all, I attempted to contact a total of 168 users. 71% of them accepted the new&nbsp;contributor terms. Only one or two of those who responded did not accept the new terms. The rest I was unable to contact.<br /><br />As part of my contact efforts I ended up calling 7 people on the telephone. This led to a few awkward conversations. One person said "I don't remember giving OSM my phone number" to which I replied "Well... you didn't. I stalked you across the internet." Another call landed me in a conversation with a high school student's mother. When I asked to speak to him she asked "who is this?" to which I responded that her son did not know me but that I am from a website that her son had used. I am happy to report that both of these users agreed to the new terms within 5 minutes of talking to me! However a few times I felt kind of like this:<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-R4q8iTfeMEQ/UH416ryUwAI/AAAAAAAABVI/4iJh9V_UGy0/s1600/fromtheinternet.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="http://1.bp.blogspot.com/-R4q8iTfeMEQ/UH416ryUwAI/AAAAAAAABVI/4iJh9V_UGy0/s320/fromtheinternet.png" width="308" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Found on&nbsp;http://www.freewebs.com/jhnbytwoo/</td></tr></tbody></table><br /><h2><span style="font-size: large;">Remapping</span></h2>Besides contacting undecided users, there was also an effort to remove and replace the contributions of users who would not agree to the new terms. There were several tools to assist us in this effort. Unfortunately I didn't get good screenshots of most of them. At the time I wasn't planning on doing a presentation about this and now that the license change is over, they are either shut down or not displaying useful information.<br /><br />But they basically all had a different way of highlighting map objects that were likely going to be either deleted or modified by the removal of license tainted data. Both JOSM and Potlatch 2 had a license change plugin/mode that would draw red halos around objects that were in danger of deletion and yellow ones around things that would be modified. Next was the OSM Inspector license change view. It also displayed objects that would be touched by the bot and let you click on them to get details as well as links to the object history and edit links to go fix it. These were based on the "<a href="http://wiki.openstreetmap.org/wiki/WTFE" target="_blank">Quick History Service</a>" which was a pretty good approximation of the algorithm the bot would use to remove data. There was also&nbsp;<a href="http://wiki.openstreetmap.org/wiki/CLEANMAP" target="_blank">cleanmap/badmap</a> which was two different map renderings. Cleanmap showed a rendering of what the map would look like after the license change and badmap showed only things that needed to be remapped.<br /><br /><h2><span style="font-size: large;">The license bot</span></h2>In the second half of July, the "license bot" was run over the planet to remove all license tainted data from the database. After the full bot run there was some additional cleanup to do. For example there were some obvious cases of copy/paste remapping where people "remapped" by straight up copying the data of users who had not agreed to the new terms. This is obviously copyright infringement and needed to be taken care of. In the bot removal process, data which was originally created by decliners or non-responders was completely removed from the database. Objects which had been modified by decliners/non-responders were reverted to an older state to remove the data that couldn't be relicensed.<br /><br />The official number I heard after the process was done is that OpenStreetMap lost 1.2% of its data. Of course removing 1% of the Interstate system makes it completely useless for routing across the country so the true impact of the bot was much higher but I'm not sure it is possible to put a specific number on it. The effects of the bot were also highly variable by location. Australia and Poland got hit disproportionately hard as did some cities here in the U.S.<br /><br />The bot had a slight blind spot having to do with splitting ways. When a way is split in OSM, a brand new way is created and attributed entirely to the user who split the way. No link is kept to the user who originally created the long way. This issue was known about ahead of time but the effort required to try and guess on way splits was deemed too high in comparison to the difference it would have made on the data.<br /><br />So, what does 1.2% of OSM data look like? Here are two graphs from&nbsp;<a href="http://osmstats.alltogetherlost.com/">http://osmstats.alltogetherlost.com/</a>&nbsp;which show the total number of nodes and ways in the OSM database over time:<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-DGQxalunzKc/UH5F83GI-LI/AAAAAAAABVY/efqr95m2H3g/s1600/bot_nodes_graph_clipped.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="151" src="http://1.bp.blogspot.com/-DGQxalunzKc/UH5F83GI-LI/AAAAAAAABVY/efqr95m2H3g/s320/bot_nodes_graph_clipped.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-3VNAxQCiDPc/UH5F9Zx8PGI/AAAAAAAABVg/NHd_J_6eM-I/s1600/bot_ways_graph_clipped.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="154" src="http://1.bp.blogspot.com/-3VNAxQCiDPc/UH5F9Zx8PGI/AAAAAAAABVg/NHd_J_6eM-I/s320/bot_ways_graph_clipped.png" width="320" /></a></div><br />You can clearly see a small bump in July. But really, it's little more than a blip. The growth of data in OSM appears to be kind of unstoppable.<br /><br /><br /><h2><span style="font-size: large;">The bot in America</span></h2><div>Since I presented this information at SOTM-US, I focused on some U.S. specific issues. We have some unique considerations, mostly having to do with the fact that a lot of our road network is imported from the TIGER/Line data set which is public domain. This means that the vast majority of our road network is not impacted by license considerations. However some of the TIGER fixup work that has happened since the import was definitely impacted. For example, there was one armchair mapper from Europe who did a lot of work on the interstate system in the eastern 1/3 of the country which was lost in the license change. As I mentioned above, way splitting was not taken into account by the license bot. Since splitting ways is a relatively common thing to do when fixing up TIGER data, this did lead to some data being removed that was technically from a public domain source. Luckily it tends to be relatively easy to restore this data, especially on interstates where a lot of the splitting happened to map bridges which are clearly visible in aerial imagery.</div><div><br /></div><div>To visualize the bot's activities, I made a map by analyzing the daily diffs that were published during the time the license bot was running in America. It only shows node edits. Red means the bot deleted a node, blue means it modified a node, likely reverting it to an earlier version.</div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-m7Kx9BYnJdw/UIcsP3RuPpI/AAAAAAAABV4/axn6as7w2To/s1600/bot_affected_nodes_USA.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="395" src="http://4.bp.blogspot.com/-m7Kx9BYnJdw/UIcsP3RuPpI/AAAAAAAABV4/axn6as7w2To/s640/bot_affected_nodes_USA.png" width="640" /></a></div><div><br /></div><div>Well that looks kind of bad. But it is exaggerated a bit because of the rendering style and low zoom. Let's look at some individual locations. First up: Kansas.&nbsp;</div><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-iJYg63hstwI/UIcszeIm7oI/AAAAAAAABWI/MlJmCrcvJQY/s1600/bot_affected_nodes_Kansas.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="378" src="http://2.bp.blogspot.com/-iJYg63hstwI/UIcszeIm7oI/AAAAAAAABWI/MlJmCrcvJQY/s640/bot_affected_nodes_Kansas.png" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"></div><div><br /></div><div>Well that's pretty good. I got most of the license problems here in Kansas taken care of either by contacting users and getting them to agree to the change or by remapping things.</div><div><br /></div><div>Next up is Dallas. On the national map it looks like a fairly decent sized blob.</div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-4RngWLQJnuo/UIcsy_agKvI/AAAAAAAABWA/vFzj7Zf209g/s1600/bot_affected_nodes_Dallas.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="380" src="http://1.bp.blogspot.com/-4RngWLQJnuo/UIcsy_agKvI/AAAAAAAABWA/vFzj7Zf209g/s640/bot_affected_nodes_Dallas.png" width="640" /></a></div><div>Upon closer inspection, it isn't really that bad. There are a few neighborhoods that were removed from the map and some damage to that highway heading east. But the overall structure of the road network isn't too bad.</div><div><br /></div><div>Next, South Carolina:</div><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-8LE19I9KC7Y/UIcs1f4JYRI/AAAAAAAABWY/2i2-hoKlm9I/s1600/bot_affected_nodes_SouthCarolina.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="378" src="http://2.bp.blogspot.com/-8LE19I9KC7Y/UIcs1f4JYRI/AAAAAAAABWY/2i2-hoKlm9I/s640/bot_affected_nodes_SouthCarolina.png" width="640" /></a></div><div>That's a bit more serious. Obviously there was a lot of damage to the interstate system here.</div><div><br /></div><div>And finally, Los Angeles:</div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-igx_Vp15xPM/UIcs0cO9LRI/AAAAAAAABWQ/B6f7rJzpsC8/s1600/bot_affected_nodes_LA.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="382" src="http://1.bp.blogspot.com/-igx_Vp15xPM/UIcs0cO9LRI/AAAAAAAABWQ/B6f7rJzpsC8/s640/bot_affected_nodes_LA.png" width="640" /></a></div><div>OK, now that's pretty heavy. Most of it is due to a single, very prolific, mapper who refused to agree to the new terms.</div><div><br /></div><div>So we know where the license bot was most active. But what exactly did it do to our data? Turns out, many different things. Here is an example of some pretty obvious damage:</div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-6hu8wgRcs7Q/UIcy0busdaI/AAAAAAAABWs/2XCtPRfMpzk/s1600/oj_co_bot_damage.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="412" src="http://3.bp.blogspot.com/-6hu8wgRcs7Q/UIcy0busdaI/AAAAAAAABWs/2XCtPRfMpzk/s640/oj_co_bot_damage.jpg" width="640" /></a></div><div>The original TIGER import was pretty terrible in this area. A declining user corrected a bunch of the road geometry. Then an accepting user came along and added some details like turning circles and gates. When the bot ran, it did not touch the new additions made by the accepting user but moved all the nodes touched by the declining user back to their original positions, shearing the map rather severely.</div><div><br /></div><div>Here is a somewhat amusing example:</div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-apHSW0kuiN0/UIczscIpvZI/AAAAAAAABW0/KR3gUr_wZEQ/s1600/redaction_weird_offramp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="302" src="http://3.bp.blogspot.com/-apHSW0kuiN0/UIczscIpvZI/AAAAAAAABW0/KR3gUr_wZEQ/s320/redaction_weird_offramp.png" width="320" /></a></div><div>In this case, the original TIGER had this onramp going the wrong direction. A declining user reversed the direction of the way. Then an accepting user came along and refined the shape of the way by adding more nodes to it. The bot reversed the order of the original nodes but not of the new nodes added by the accepting user, leading to the odd "zig" in the middle.</div><div><br /></div><div>Now for a little more subtle damage:</div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-T8vID_DMNg0/UIc0tJ1tIQI/AAAAAAAABW8/Y44iXMe5faI/s1600/license_bot_LA_slippy.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="452" src="http://3.bp.blogspot.com/-T8vID_DMNg0/UIc0tJ1tIQI/AAAAAAAABW8/Y44iXMe5faI/s640/license_bot_LA_slippy.png" width="640" /></a></div>You can see some obviously weird zigs and zags in the roads but nothing seems to be completely out of whack here. However, when viewed in JOSM, the damage becomes clearer.<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-CspfEJs_0oE/UIc0vpushiI/AAAAAAAABXE/avERPOaIWj4/s1600/license_bot_LA_JOSM_annotated.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="448" src="http://3.bp.blogspot.com/-CspfEJs_0oE/UIc0vpushiI/AAAAAAAABXE/avERPOaIWj4/s640/license_bot_LA_JOSM_annotated.png" width="640" /></a></div><div>In the upper left, there are nodes that used to be part of a way. Both the nodes themselves and the way made it through the license bot. But the fact that the nodes were a part of the way was lost leaving empty nodes and a misshapen way. In the lower part of the image you can see that there is a way which does not actually connect to any of the ways it crosses. So much for routability.</div><div><br /></div><div><h2><span style="font-size: large;">Fixing the damage</span></h2></div><div>After the license bot ran, the community immediately started focusing on how to fix the damage. Several tools were developed or adapted to help guide and focus this effort. At the end of the day, we have risen to the challenge and I have actually been surprised at how quickly most of the damage has been fixed.&nbsp;</div><div><br /></div><div>I made a rough graph of remapping activity based on changeset comments. I used the weekly changeset dump along with my&nbsp;my&nbsp;<a href="https://github.com/ToeBee/ChangesetMD" target="_blank">ChangesetMD</a>&nbsp;tool&nbsp;to create a database of changeset metadata which I then queried for these keywords in the changeset comments:&nbsp;odbl, license, remapping, redaction.&nbsp;</div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/--bOCbXxoMW0/UIc21_wVDfI/AAAAAAAABXM/GGoWdefxQy4/s1600/remapping_activity_graph.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="426" src="http://3.bp.blogspot.com/--bOCbXxoMW0/UIc21_wVDfI/AAAAAAAABXM/GGoWdefxQy4/s640/remapping_activity_graph.png" width="640" /></a></div><div>It starts back in December 2011 when remapping started in earnest. The orange line follows the axis on the left and indicates the number of objects touched in these remapping changesets. This includes deletes, modifications and additions. The blue line follows the axis on the right and is just the number of remapping changesets created per day.&nbsp;</div><div><br /></div><div>I believe the huge spike in the orange line at the end of January was me and a few other people really putting our backs into getting the coastlines cleaned up. <a href="http://www.paulnorman.ca/blog/" target="_blank">Paul Norman</a> came up with a good way to highlight shoreline ways that were in danger of being damaged by the bot and it turns out there were a lot of them. I nuked most of the shoreline along the west coast and replaced it with NHD data. Some others did the same or manually remapped the shoreline around the Great Lakes and the east coast.</div><div><br /></div><div>In May and June, remapping activity seems to have slowed to a crawl. I think people were just getting tired of it. I certainly was. The big blue spike on the right happened a couple of days after the license bot started running. It was actually still in progress at the time. But since Europe was hit first, it seems that the Europeans made quick work of cleaning up after it.</div><div><br /></div><div>My own contribution to the remapping effort was a map that highlighted ways that seemed to be missing their highway=* tag. I noticed that a fair number of interstates and on/off ramps had a lot of their tags stripped but one of the following tags remained: oneway, lanes, bridge, tunnel. Any of these tags without a highway tag is rather suspicious so I put them on a map to highlight and guide people to them for remapping. Once in the area of such a way I often found other problems in addition to the missing highway tag. Here is an animation of how my map evolved between August 8th and September 30th:</div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-pEgarfZJWyY/UIc8dPZ9Q6I/AAAAAAAABXg/QXHymdZCp8I/s1600/oneway_map.gif" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="384" src="http://4.bp.blogspot.com/-pEgarfZJWyY/UIc8dPZ9Q6I/AAAAAAAABXg/QXHymdZCp8I/s640/oneway_map.gif" width="640" /></a></div><div><br /></div><div>It started out only showing ways with a oneway tag. You can see some new red appear on the map when I added the other tags to the list. By August 28th, the U.S. was completely clear on this map. The Europeans still have quite a mess to clean up. You can find the map still online here:&nbsp;<a href="http://ni.kwsn.net/~toby/OSM/maps/redaction.html" target="_blank">http://ni.kwsn.net/~toby/OSM/maps/redaction.html</a></div><div><br /></div><div>Another effort to detect problems on major roads was the routing grid that Kai Krueger set up. It queried <a href="http://project-osrm.org/" target="_blank">OSRM</a> periodically for a route between 40 cities and compared the distance to a reference distance. If it was off by a lot, it indicated a likely problem along an interstate or US highway between the two cities. The difference was color coded and put into a display matrix. Here is another animation showing how the routing grid evolved between July 21st and September 30th:</div><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-F6p29qzO_6w/UIc-a6xIbII/AAAAAAAABXo/7NvGTXYKY_I/s1600/routing_grid_3.gif" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="574" src="http://4.bp.blogspot.com/-F6p29qzO_6w/UIc-a6xIbII/AAAAAAAABXo/7NvGTXYKY_I/s640/routing_grid_3.gif" width="640" /></a></div><div>Again, you can see that things got fixed up pretty quickly. Also, I'm not sure why everything went orange again right at the very end.</div><div><br /></div><div>One other strategy I used to fill in hard hit areas was to re-import the area from new TIGER 2011 and even some 2012 data when it became available. To do this, I downloaded both the shapefile and the feature names relationship file from the <a href="http://www.census.gov/geo/www/tiger/shp.html" target="_blank">Census Bureau</a>. The feature names file allowed me to do a better job of expanding street names from their abbreviated forms. I joined the two files in QGIS and then ran them through ogr2osm using my own <a href="https://github.com/ToeBee/ogr2osm-translations" target="_blank">translation file</a> which extracted only a few of the shapefile attributes into OSM tags and expanded abbreviations. This gave me a .osm file which I was able to load into JOSM and then selectively copy roads from there into OSM.</div><div><br /></div><div>Here is a screen shot of JOSM while I had some TIGER data loaded in the background with my "inactive" color set to a bright green to contrast the red of selected ways in the OSM layer on top of it. This highlighted areas that were missing in OSM. My apologies to any red-green colorblind readers.</div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-5jWKsj_KfeU/UIdA2tmCmcI/AAAAAAAABXw/CeEpdInMGfw/s1600/riverside_JOSM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="458" src="http://3.bp.blogspot.com/-5jWKsj_KfeU/UIdA2tmCmcI/AAAAAAAABXw/CeEpdInMGfw/s640/riverside_JOSM.png" width="640" /></a></div><div>After zooming in on the desired area I used the search string "type:way allinview" to automatically select all the ways in the TIGER layer, copied them and then pasted them into the OSM layer. Then of course I had to do some quality assurance to connect the new ways to the existing data and remove some common TIGER problems such as over-noded ways and duplicate ways that were created when TIGER had more than one name for a road. Typically this was just things like "Drive" vs "Avenue" or some other minor spelling difference.<br /><br />One excellent tool that&nbsp;<a href="https://oegeo.wordpress.com/" target="_blank">Martijn van Exel</a> came up with is called "Remap-a-tron." I did not talk about it at the conference at all because&nbsp;Martijn did a presentation about it right before my presentation. But you can read about it in his own <a href="http://oegeo.wordpress.com/2012/08/29/remapathon/" target="_blank">blog post</a>. Basically, he did some digging to find ways that had been deleted by the license bot. Then when you hit the web page, it would display one deleted way at a time and give you the opportunity to go remap it in your favorite editor using aerial imagery, surrounding data that wasn't removed or whatever other source of license acceptable data you could find. After remapping the way you would come back and indicate that you had remapped it. It would record the remapping and immediately show you the next way that needed attention. This tool has now been rebranded as "Maproulette" and is available at&nbsp;<a href="http://maproulette.org/" target="_blank">http://maproulette.org</a>&nbsp;where it has been retasked to find other common map errors that are easy to fix.<br /><br />Finally, there is also a "<a href="http://tools.geofabrik.de/osmi/debug.html?view=redactionbot" target="_blank">Redaction Bot</a>" view in the Geofabrik OSM Inspector tool. It shows all objects that were touched by the bot. You can click on them to get details about what the bot did to a particular object. I don't think it tries to match things that were deleted by the bot to new mapping done to replace them so over time the signal to noise ratio of this tool will decrease. Here is a screenshot of it with one way selected.<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-6eUopxuY498/UIdgoCZOjsI/AAAAAAAABYQ/cXzesoT31HI/s1600/geofabrik_OSMI.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="394" src="http://4.bp.blogspot.com/-6eUopxuY498/UIdgoCZOjsI/AAAAAAAABYQ/cXzesoT31HI/s640/geofabrik_OSMI.png" width="640" /></a></div><br /></div><div><br /></div><div><h2><span style="font-size: large;">The End</span></h2></div><div>In conclusion, the license change definitely had a major impact on OpenStreetMap. Both our map data and the community were impacted, sometimes to great detriment. However we are getting past it remarkably quickly thanks to the hard work of mappers like you. Some damage will remain in the map for a long time to come. Deleted points of interest aren't nearly as obvious as broken interstates and will take much longer to fill back in. But at the end of the day, we will survive and continue to thrive. In some ways, 2012 really was the year of OpenStreetMap, despite the license change. Let's make 2013 even better!<br /><br /></div><div><h2><span style="font-size: large;">Map the planet!</span></h2></div><div><br /></div>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-2574779849454093341
url: http://ksmapper.blogspot.com/2012/06/making-location-badge-using-osmand-and.html
title: Making a location badge using OsmAnd and the MapQuest Open static map API
published: 2012-06-08 02:50:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.821819
updated_at: 2019-10-25 14:30:27.175409
summary:
  I'm about to embark on my second <a href="http://www.bak.org/" target="_blank">Biking Across Kansas</a> tour. When I did it last year, I used Google Latitude to create a little "My Location" badge which I put here on my blog to let people track me across the state. The Google one looks like this:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-f3H4To1O__c/T88CsQsnd0I/AAAAAAAAAvk/SlOCc_mg3tU/s1600/google_location_badge.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="http://4.bp.blogspot.com/-f3H4To1O__c/T88CsQsnd0I/AAAAAAAAAvk/SlOCc_mg3tU/s200/google_location_badge.png" width="141" /></a></div><br /><br />However at some point during the last year I noticed that <a href="http://www.osmand.net/" target="_blank">OsmAnd</a> added a feature that allows it to periodically ping a web service of your choosing with the current location of the phone. This allows me to build my own location publishing service. <br /><br />I did a little searching for how other people have used this feature in OsmAnd and found a few people talking about feeding the data into a Google map display. The example in the OsmAnd wiki posts the location to a Google doc which then somehow magically updates a Google map display. No offense to Google but I'd rather use OSM based products. Fortunately the good people over at <a href="http://open.mapquest.com/" target="_blank">MapQuest Open</a> provide a service that suits this situation: The <a href="http://open.mapquestapi.com/staticmap/" target="_blank">static map API</a>.<br /><a name='more'></a><br />Behold my creation (and location):<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-jZTj5KLUAiA/UFz5DUwyDtI/AAAAAAAABG4/kF4JgODSwus/s1600/tracker.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="http://2.bp.blogspot.com/-jZTj5KLUAiA/UFz5DUwyDtI/AAAAAAAABG4/kF4JgODSwus/s640/tracker.png" width="622" /></a></div><br />Note that this is no longer an active link since BAK is over. This is just a screenshot of what it looked like.<br /><br />Now for the details. The system has two components. A web service to collect the location information from the phone and a display page that creates a pretty map to look at.<br /><br />I hacked together some PHP in about an hour. Not the biggest fan of PHP and I'm pretty sure I'm doing some things wrong but I got something working and don't have time to make any improvements before I leave for BAK in under 12 hours so there you have it. <a href="https://github.com/ToeBee/OsmAnd-Tracker" target="_blank">The code is on github</a>.<br /><br />The location web service takes the GET request from the phone and writes the information contained in it to a file. The display page reads the file, pulls out the coordinates and uses them to construct a URL that retrieves a static map image from the MapQuest Open static map API. It isn't an interactive map, just a quick "I am here" image. However I have wrapped the image in a link that takes youto an interactive map on openstreetmap.org. Throw that into an iframe tag and you're good to go!<br /><br />Possible future improvements might include the following<br /><ul><li><div>Perform a nominatim query to get a nearby city name and display that</div></li><li>Better controls over the accuracy of the position displayed. The google badge lets you only reveal which city you are in if you wish. Currently I only allow rounding coordinates to a given number of decimals (default is two)</li><li>Dynamic sizing of the map based on parameters passed in to the iframe from the parent page. This example is rather huge and not really a "badge" but that's what I want in this situation.</li><li>Track more than just the last location ping and display distance traveled between updates or something else interesting. The static map API is pretty flexible with displaying multiple POIs or even lines or shapes.</li><li>...</li><li>PROFIT!</li></ul>I am packing my bags as I finish this post so I don't really have time to brainstorm anything else. I'm sure others will have ideas :)<br /><div><br /><div>See you in 500 miles!</div></div>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-5540953464009813785
url: http://ksmapper.blogspot.com/2012/03/remapping-using-tiger-2011.html
title: Remapping using TIGER 2011
published: 2012-03-11 22:36:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.824549
updated_at: 2019-10-25 14:30:27.178235
summary:
  The license change mapping process I talked about in my <a href="http://ksmapper.blogspot.com/2012/01/license-change-mapping.html">previous post</a> works well where there is still some clean data to go off of and you're just having to redo some work that a decliner did in splitting ways or adding some nodes to refine the geometry. But what about when an entire area is on the chopping block? I present the <a href="http://tools.geofabrik.de/osmi/?view=wtfe">OSM Inspector</a> view in Glendale, CA:<br /><span id="goog_639475318"><br /></span><span id="goog_639475319"></span><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-d8qLLPU5sJQ/T10KNM0Z2wI/AAAAAAAAAnE/MFOar-B8CYM/s1600/glendale_dirty.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="440" src="http://1.bp.blogspot.com/-d8qLLPU5sJQ/T10KNM0Z2wI/AAAAAAAAAnE/MFOar-B8CYM/s640/glendale_dirty.png" width="630" /></a></div><br />The area I worked on was just south of here and was actually even worse although I didn't get a "before" screenshot. Less yellow, more red. Here is what it looked like on Simon Poole's "<a href="http://cleanmap.poole.ch/?layers=00B0">badmap</a>"&nbsp;rendering. Pretty much the entire road network is going to get nuked:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-Ef7upQxWi6Q/T10MjBS7ftI/AAAAAAAAAnM/uhKgHH2C954/s1600/remapping_before.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="352" src="http://2.bp.blogspot.com/-Ef7upQxWi6Q/T10MjBS7ftI/AAAAAAAAAnM/uhKgHH2C954/s640/remapping_before.png" width="630" /></a></div><br />Most of this data is from a single, very prolific contributor who is not likely to agree to the new terms. It is also highly unlikely that anyone in the area is going to be able to redo all of his work before April 1st. So I reached for a bigger paintbrush: <a href="http://www.census.gov/geo/www/tiger/tgrshp2011/tgrshp2011.html">TIGER 2011</a>.<br /><a name='more'></a><br />To use TIGER data in OSM is no simple matter. For one thing, the Census Bureau uses abbreviations rather heavily and OSM is strongly anti-abbreviation, and for good reason. Unfortunately it is not a simple matter of throwing a regular expression at it and expanding all "Ave" to "Avenue" and "N" to "North" - what if there is a street named "N Street" which runs parallel to "O Street"? But then again, there may be a legitimate "North Street" somewhere. The TIGER data does indicate what pre- and post-fixes are present on a given road so the full name can be reconstructed with some effort. But to use this information is a little cumbersome. And of course the accuracy is always an issue but that's kind of a given with this data.<br /><br />First I downloaded the shapefile for Los Angeles county. Then I had to get the "Feature Names Relationship File" and use QGIS to join the two based on the LINEARID field. I ended up with a shapefile with all the name information present. The complication here is that sometimes roads have multiple entries in the relationship file and I'm not sure what QGIS does in this case. Since I was only working on a small area, I reviewed the data and made sure it all looked sane.&nbsp;Since a whole county's worth of data can be hard to work with in JOSM, I clipped out just the area I wanted to work on and exported that as a new shapefile in QGIS.<br /><br />The next step was to convert this data to .osm format. I used a new version of the <a href="https://github.com/andrewguertin/ogr2osm">ogr2osm</a> python script written by Andrew Guertin to do this. To translate shapefile attributes to OSM tags requires a translation file. I cooked one up based on some previous traffic on the talk-us mailing list for mapping TIGER road types to OSM highway=* tags. For the name, I did some more looking at the data and came up with a bare minimum implementation for the few hundred highways I had to work with. I don't expect this to work well in other places without modification. But here is what I came up with. Patches welcome :)<br /><a href="https://github.com/ToeBee/ogr2osm-translations/blob/master/tiger2011.py">https://github.com/ToeBee/ogr2osm-translations/blob/master/tiger2011.py</a><br /><br />And that was the easy part.<br /><br />Now to integrate this data into OSM. First I downloaded the entire area in JOSM, selected everything and ran the <a href="http://wiki.openstreetmap.org/wiki/JOSM/Plugins/LicenseChange">LicenseChange</a> plugin against it. Of course the whole screen lit up in red. Then I opened the .osm file generated by ogr2osm. I also changed my JOSM settings to render the "inactive" layer in a bright color so I could easily see where things were missing.<br /><br />Then it was just a matter of spending a few hours deleting ways in the area, switching to the TIGER layer, copying the missing roads and then pasting them into the "live" layer of OSM data. Then of course I had to find all the places where the old and the new data intersected and recombine them to maintain correct topology. The JOSM validator can help with this although I'm sure some mistakes made it in. The larger an area you can do at once, the less fiddling you have to do with the topology since all the data you copy/paste from the TIGER data will (for the most part) have correct internal topology. But bigger areas are harder to keep track of as you go. I usually tried to find relatively self-contained areas with as few as possible connections to the rest of the world. This spot on the map is a little unique in that it seems to be in a hilly area and it isn't your average grid pattern of streets. So there were actually isolated networks that I could deal with in chunks.<br /><br />Of course I always had to keep an eye out for TIGER strangeness. Some fictional roads and that kind of thing. 99% of what I touched was just residential roads. Where I did touch tertiary/secondary ones I did some additional comparison since TIGER's road classifications don't always match up with reality.<br /><br />This whole process was kind of complicated when there was actually something I &nbsp;<b>didn't</b> have to replace. I always prefer local mapper contributions over TIGER data so I had to make sure I didn't steamroll over what little clean data I did manage to find.<br /><br />So at the end of the day (literally) the area now looks like this on badmap. Obviously I was only concentrating on the road network so the parks are still there.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-4iJTeiSmruE/T10a-AzF3SI/AAAAAAAAAnU/XXBPI7hYbyA/s1600/remapping_after.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="445" src="http://2.bp.blogspot.com/-4iJTeiSmruE/T10a-AzF3SI/AAAAAAAAAnU/XXBPI7hYbyA/s640/remapping_after.png" width="630" /></a></div><br />And even from a lower zoom, there is a noticeable hole in OSMI:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-FItX3Dgye70/T10ldsptkxI/AAAAAAAAAnc/vkXzNu09nZ8/s1600/osmi_after.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="530" src="http://3.bp.blogspot.com/-FItX3Dgye70/T10ldsptkxI/AAAAAAAAAnc/vkXzNu09nZ8/s640/osmi_after.png" width="630" /></a></div><br />Definitely an improvement but there is still much, MUCH more work to do :(<br /><br />The changeset I did this in can be seen here:<br /><a href="http://www.openstreetmap.org/browse/changeset/10937678">http://www.openstreetmap.org/browse/changeset/10937678</a>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-4911668036047260885
url: http://ksmapper.blogspot.com/2012/01/license-change-mapping.html
title: License Change Mapping
published: 2012-01-08 06:04:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.827273
updated_at: 2019-10-25 14:30:27.181358
summary:
  So far I have avoided saying anything about the OpenStreetMap&nbsp;<a href="http://www.osmfoundation.org/wiki/License/We_Are_Changing_The_License">license change</a> on this blog. It is a very sensitive subject for a lot of people for a million different reasons. This is <b>not</b> a post about how right/wrong the license change is. I personally have agreed to the new terms but then again I don't really care what license my data gets redistributed under. While I prefer (in most situations) a share-alike clause, if we were switching to public domain instead of ODbL, I would probably still agree. Whatever.<br /><br />But the license change is happening and we must prepare the map for it, ahead of the <a href="http://blog.osmfoundation.org/2011/11/16/odbl-progress/">April 1 cutover date</a> set by the license working group. Today I decided to take a look at what it would take to get I-70 in Kansas cleaned up, license-wise, and thought I'd share my findings. Note that this particular process may or may not apply to other areas. The interstate system is kind of special in that it was imported from a public domain source and has very long stretches that are identical except for tags that don't really matter. This means that there isn't really very much unique or unrecoverable information present. If there is a small section of road (say a bridge) that is dirty, it can just be removed and replaced from clean data that surrounds it plus imagery to verify that it is indeed a bridge. However whenever possible it is preferred to maintain the history of objects in OSM so I don't want to just nuke and replace large sections of I-70. Plus, deleting and recreating ways gets more complicated because of the route relations that they may be a part of.<br /><br />A majority of I-70 in Kansas is already clean but there is still a&nbsp;noticeable&nbsp;fraction of dirty objects. The goal is to end up with all objects being "clean" to re-license under ODbL. First task: find dirty objects. For this&nbsp;I use the "License Change" view in the OSM inspector tool created by Geofabrik:&nbsp;<a href="http://tools.geofabrik.de/osmi/?view=wtfe">http://tools.geofabrik.de/osmi/?view=wtfe</a><br /><a name='more'></a><br />Here is a section of I-70 I will be working on.<br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-zF8DpseBHMg/TwjMkIA5frI/AAAAAAAAAjI/R--1Nxm3uEc/s1600/1_inspector_dirty_view.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="400" src="http://3.bp.blogspot.com/-zF8DpseBHMg/TwjMkIA5frI/AAAAAAAAAjI/R--1Nxm3uEc/s640/1_inspector_dirty_view.png" width="635" /></a></div><br />You can mouse over the things on the left to find out exactly what the colors and symbols mean. Basically red and orange are bad. In this screen shot I have already clicked on the right-most red dot. This brought up the information in the upper right panel which indicates that there are two items of interest near where I clicked. Node 388499694 and way 33918862 were both created by declining user moonwashed.<br /><br />Now I click on the clock icon of the way. This brings up Ian Dees' "Deep Diff" service which displays a nice visual history of the way.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-ZrMcaKHQhXg/TwjNvJlfu0I/AAAAAAAAAjQ/iDkthBniUxY/s1600/2_deep_history_view.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="356" src="http://2.bp.blogspot.com/-ZrMcaKHQhXg/TwjNvJlfu0I/AAAAAAAAAjQ/iDkthBniUxY/s640/2_deep_history_view.png" width="635" /></a></div><br />From this I can see that the declining user created the way and made one edit to it. The subsequent 11 edits have all been by users who have accepted the new terms. The only original tags that have survived unchanged from the declining user are highway=motorway and oneway=yes. In this case, I consider these both trivial tags that should not prevent this way from being re-licensed. I have yet to encountered this along I-70 but there may be non-trivial tags that were added by a declining user. In this case, such tags would have to be removed to make the way "clean."<br /><br />Now, what about the nodes? I have the "remote control" feature enabled in JOSM which means that I can go back to the OSM inspector and click on the icon with the little "J" on it next to the way. This will cause JOSM to automatically download a bounding box around that way and select the way after it is downloaded. (<strike>This doesn't seem to work from Opera though</strike> see comment below) I also have the <a href="http://wiki.openstreetmap.org/wiki/JOSM/Plugins/LicenseChange">License Change Plugin</a>&nbsp;installed which I used to do a license check on the selected way. Here is the resulting JOSM window:<br /><br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-NNYIHQVQpgU/TwjSn7lCDQI/AAAAAAAAAjo/-j1nSBHmF9k/s1600/3_josm_loaded.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="396" src="http://1.bp.blogspot.com/-NNYIHQVQpgU/TwjSn7lCDQI/AAAAAAAAAjo/-j1nSBHmF9k/s640/3_josm_loaded.png" width="635" /></a></div><br />I have also selected all of the dirty nodes in the license problem list. Now if I click the "Select" button, it will select all the dirty nodes. Next I inspect them to make sure none of them are nodes that connect to other ways. This could cause problems in the next step. And in fact, the one at the left end of the way is a junction node with a bridge. So I will unselect it using a CTRL+click and deal with it later. Now that only dirty nodes are selected, I delete them all. This will leave me with a way that is of course horribly wrong in terms of geometry. But now I can turn on the Bing imagery and go back and re-improve the way from Bing using the <a href="http://ksmapper.blogspot.com/2011/12/most-interesting-mapper.html">ImproveWayAccuracy</a> plugin:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-UV8bY5TXcxQ/TwjVGf_szDI/AAAAAAAAAjw/dL3MIEeIW2U/s1600/4_reimproving_way.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="395" src="http://1.bp.blogspot.com/-UV8bY5TXcxQ/TwjVGf_szDI/AAAAAAAAAjw/dL3MIEeIW2U/s640/4_reimproving_way.png" width="635" /></a></div><br />And of course the screen shot software intentionally left out the mouse cursor... but it is where the green line makes a bend which is where I am about to add a node. Now I check along the rest of the way to make sure everything else is ok.<br /><br />Now back to that dirty node I left out. The difficulty in this case is that the node is part of a 2-node way (a short bridge) so if I delete the dirty node, the bridge will get deleted as well. Since I don't want that, I will create a new node in the middle of the bridge, then delete the dirty node. This will create a gap between the two ways which needs to be fixed by re-extending one of the ways to connect with the other. Then all that I need to do is fix the position of the junction node according to imagery. This leaves all the nodes clean.<br /><br />Note that for untagged nodes, simply moving might be enough to clean them, depending on what rules end up being applied during the license chage. But simply moving a node by 1 meter to make sure it is ODbL clean doesn't give me warm fuzzy feelings. I'd rather just scrap them and start from scratch to avoid uncertainty.<br /><br />But the way itself is still considered dirty. For the following 3 reasons, I believe that the way can still be&nbsp;re-licensed:<br /><ul><li>The way was originally created from a public domain import. All the declining user did was split it, probably to make the bridge.</li><li>All tags are either trivial (as mentioned above) or have been added/changed by agreeing users</li><li>The geometry is now completely clean as I removed all dirty nodes and recreated them as I would when mapping something from scratch: aerial imagery and GPS traces</li></ul>So to indicate this, I will add the following two tags to the way:<br /><ul><li>odbl=clean</li><li>odbl:note=decliner touched tags that have been overwritten or are trivial. All decliner nodes have been removed</li></ul><br />The first tag is documented on the <a href="http://wiki.openstreetmap.org/wiki/Remapping/License_Change_View_on_OSM_Inspector#Per-object_Overrides_with_odbl.3Dclean">Remapping/License Change View on OSM Inspector</a> wiki page. I'm not sure if there has been an official nod to obeying this tag in the final license change process but I'm not the only one using it so hopefully it will. And if not, it gives me a simple tag that I can query on to retrieve all ways that need further work. The second tag is just something I decided to use to briefly explain why I think this way is clean.<br /><br />And that's it. After the OSM inspector updates its data, this way should show up as a green line instead of a red one.<br /><br />This might seem like a lot of work but once I worked out a process I was able to move along pretty quickly. Plus, there were only a few ways where I had to deal with more than a few nodes. I got all of I-70 in eastern Kansas done in a little over an hour and I should be able to finish the rest within another hour or two. And along the way I of course get sucked into improving other things too so it isn't entirely busywork.<br /><br />If only we lived in a perfect world where everyone got along and no licenses were required. But then what would the lawyers do?


guid: tag:blogger.com,1999:blog-7044416767097884399.post-805300550809964529
url: http://ksmapper.blogspot.com/2011/12/most-interesting-mapper.html
title: The Most Interesting Mapper
published: 2011-12-18 06:09:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.830004
updated_at: 2019-10-25 14:30:27.184378
summary:
  <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-KYLioR5qEKI/TuU3nYQelTI/AAAAAAAAAeY/je7fOM1yqrk/s1600/most_interesting_mapper.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-KYLioR5qEKI/TuU3nYQelTI/AAAAAAAAAeY/je7fOM1yqrk/s1600/most_interesting_mapper.jpg" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: large;">The Most Interesting Mapper in the World</span></td></tr></tbody></table><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">In case you live under a rock, this particular meme has its origins in a set of beer commercials that I find kind of amusing. Here is <a href="http://www.youtube.com/watch?v=TzXRPphJouA">a decent compilation</a> of them.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">&lt;sorry to interrupt this completely awesome post but quick note: This plugin has been turned into a core JOSM feature. See "Update 2" at the bottom of this post&gt;</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">But seriously, the ImproveWayAccuracy plugin is a wonderful tool. Perhaps the prime use case to show it off is a smoothly curving road that was originally either mapped from low resolution imagery or (around here) imported from TIGER data that may have used far too few nodes to make a smooth curve. To fix this by hand is highly annoying. You have to drag nodes around and then grab the + between two existing nodes to create a new one and drag it to where you want it, repeating that as often as needed to achieve a smooth enough curve. Each node requires several precise mouse movements.&nbsp;</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">It is much easier to draw a way from scratch where you just move your mouse along the curve and click as needed to form a smooth looking curve. Well, the ImproveWayAccuracy plugin makes this possible to do with existing nodes/ways!</div><a name='more'></a><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">For basic usage you can look at the plugin's wiki page:&nbsp;<a href="http://wiki.openstreetmap.org/wiki/JOSM/Plugins/ImproveWayAccuracy">http://wiki.openstreetmap.org/wiki/JOSM/Plugins/ImproveWayAccuracy</a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">The first thing you will want to do after installing the plugin is to change JOSM's "highlight" color to something that has a high contrast to whatever the "selected" color is. By default they are both shades of red which makes the plugin harder to use. I changed mine to a bright green (#66FF33).&nbsp;</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Basically you activate the plugin by pressing k and selecting a way. Then if you move the mouse around you will see that the node that is closest to your mouse cursor is highlighted and there are temporary lines going from your cursor to the two nodes next to that one. If you click, the highlighted node will be moved to your cursor's current location. It is kind of hard to describe but pretty easy to get the hang of if you try it.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Holding down CTRL and clicking will add a new node to the way between the two nearest nodes. Holding down SHIFT will lock the node selection to the current node even if you move the mouse so that another node is now closer to the cursor. This can be helpful while trying to fix some weird geometries.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Here is a short clip of me aligning and smoothing out a curve using this plugin. You can tell when I am holding down CTRL because the cursor gets a small + symbol added to it:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/umUhtXsPgwU?feature=player_embedded' frameborder='0' /></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Now... I do have one beef with this plugin. The wiki page states that deleting nodes is not a feature and that you should re-use nodes whenever possible. While I generally agree that as much history as possible should be preserved while editing, it is not always practical. Especially with the TIGER imports that I am frequently editing. Sometimes imported ways have too many nodes along them. This makes editing them more of a pain and the correct way to handle it really is to delete nodes.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Of course the source code for this plugin is available in the OSM SVN repository (<a href="http://trac.openstreetmap.org/browser/applications/editors/josm/plugins/ImproveWayAccuracy">see here</a>) so I hacked this feature in myself. Just like the CTRL key adds nodes to the way, the ALT key removes them in my version. However it performs a sanity checks first. If you try to delete a node that is a member of two different ways (a junction node) then the plugin will throw up an error message. The ImproveWayAccuracy plugin is only intended to modify the geometry of a way. If you want to change the topology of the data, you should not be in the IWA mode.&nbsp;</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">One gotacha: Some linux window managers use ALT plus a mouse drag to move windows. So my window manager was swallowing my ALT+click event and it never got passed on to JOSM so the plugin just didn't do anything. But I don't use ALT+drag to move my windows around so I just disabled it. I looked at trying to use a different key but the way the existing code was written, it was easiest to just use another meta key and since CTRL and SHIFT were already taken... I guess I could have maybe used CTRL+SHIFT.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Update: Turns out window managers only listen for the left ALT key but JOSM will take the left or right one. So you can use the meta keys on the right side of the keyboard without problems. Oh and CTRL+SHIFT is already used by the plugin although I have never actually used this feature.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><strike>If you want to use my version of the plugin, the easiest way is to first install the "official" plugin through the JOSM plugin options. Then close JOSM and copy my ImproveWayAccuracy.jar file over the one in the JOSM plugin directory (~/.josm/plugins/ on linux or C:\Documents and Settings\&lt;username&gt;\Application Data\JOSM\plugins\ on windows) You can </strike><a href="http://ni.kwsn.net/~toby/OSM/ImproveWayAccuracy.jar"><strike>download the jar file here</strike></a><strike>. Or if you want to see the code, </strike><a href="http://ni.kwsn.net/~toby/OSM/IWAMode.java.patch"><strike>here is the svn diff</strike></a><strike> for the IWAMode.java file.&nbsp;(as of revision 27186) </strike></div><div class="separator" style="clear: both; text-align: left;"><strike><br /></strike></div><div class="separator" style="clear: both; text-align: left;"><strike>I submitted the patch to the original author of the plugin so at some point this might get rolled in to the real plugin. But he said he is very busy right now and probably won't get a chance to look at it any time soon. So I decided to just put it out there in this form for now. The only problem is that letting JOSM update plugins may overwrite the custom .jar file in the future.</strike></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Update: I <a href="http://josm.openstreetmap.de/ticket/7253">submitted a patch</a> to the JOSM trac system. There was some discussion about keys and other checks to do before deleting nodes. But at the end of the day my changes (plus a tweak or two) have been incorporated into the live code and a new version of the plugin has been released. So just update your plugins and you will be able to take advantage of this feature!</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Update 2: Turns out, this code is useful! The JOSM developers saw fit to pull this functionality into JOSM core. They even rearranged some keyboard shortcuts to make room for it on W instead of K. So no more plugin required. Just hit "W" and go! You can see the discussion in <a href="http://josm.openstreetmap.de/ticket/7184">ticket #7184</a> over on the JOSM trac.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Enjoy.</div>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-4560975720766338525
url: http://ksmapper.blogspot.com/2011/12/random-computer-upgrade-notes.html
title: Random computer upgrade notes
published: 2011-12-06 03:50:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.832800
updated_at: 2019-10-25 14:30:27.187161
summary:
  I recently did some reshuffling of the computers in my home. I built a brand new gaming system from parts I bought on newegg. Intel i5 Sandybridge, 16 GB of memory and an nvidia video card that not only has a 120mm fan, but has <b>two</b> of them. The insanity. Then I used the parts from the old gaming system (first generation i5, 12 GB, ATI graphics) to give my ancient linux box a major upgrade.<br /><br /><span style="font-size: large;">Of Linux and Distros</span><br />The Linux box was so old (AMD Athlon x2 4400+ with 3 GB) that it was becoming a pain to use for any kind of desktop applications. I ended up neglecting it and spending most of my time on my windows box (I know! The horror!) with multiple putty sessions open. Also, it was running Gentoo Linux. In my local Linux user group, we have determined that people who are over 30 don't use Gentoo. It is a worthy distro and has many strengths. I have been using it for over 10 years on some system or another and because of its minimal nature, I learned a lot about Linux that I would otherwise be oblivious to. But it can be a real pain, especially to maintain for desktop use. I was getting tired of every other <i>emerge --sync</i> generating package conflicts, lengthy config file merges and the occasional system toolchain breakage that you <b>had</b> to resolve before rebooting (or loosing power!)&nbsp; otherwise the system would completely eat itself. So, being a distinguished 31 years of age now, I decided it was time to drop Gentoo. So long and thanks for all the fish.<br /><br /><a name='more'></a>So what to replace it with? Well I've been an Ubuntu user for a while as well. But they really worked hard to tick me off with the whole Unity business. It's not that I am completely opposed to new things. But I think Ubuntu messed up pretty good in this transition. First, they rolled out Unity to the netbook edition a year ago, before it was ready. It crashed the first time I logged in. Then it continued to be buggy. Also, I really, really don't like the mac-style combined launch/task bar that is central to Unity. I want a taskbar that displays currently running programs. Not some auto-hiding, moving-with-your-mouse contraption that shows a whole host of programs, some of which may be running and some of which may not. The running ones have a dot on them or something? Whatver. Just give me a taskbar. And now with Ubuntu 11.10 the option to go back to regular old gnome 2 is mostly gone. It's either Unity or Gnome3 with gnome-shell. On my netbook I now use gnome-shell and it works well enough for such a limited device with a small screen. But I'm not really a fan of some of the unilateral decisions they made when it comes to running it on a dual monitor desktop.<br /><br />Enter <a href="http://blog.linuxmint.com/?p=1851">this blog post</a> from Linux Mint. The condensed version: "Gnome 2 is dead so we <b>have</b> to move to something newer. But we'll ease the transition by giving you similar functionality to what you've been used to for the last 15 years and let you choose how much new stuff you want." Perfect. So right now I'm typing this on a Linux Mint 12 release candidate install. So far I'm pretty happy. I even find myself using some of the new gnome-shell features from time to time. But I still have my taskbar, traditional alt-tab and always-visible notification area. I'm still trying to hammer out a few bugs I'm encountering. One thing that annoyed me greatly was the default workspace switching for dual monitors. Only my left monitor switched. The right one always displayed the same workspace. Turns out it is just a dumb default setting made by a gnome developer. Unfortunately they don't provide an easy checkbox to change this in the settings. Luckily I found <a href="http://gregcor.com/2011/05/07/fix-dual-monitors-in-gnome-3-aka-my-workspaces-are-broken/">this blog post</a> that fixed it for me in 30 seconds. <br /><br />Gnome-shell does seem to be a little bit crashy but I'm I'm going to blame that on the ATI video card I have in this thing. I bought this card several years ago right after ATI announced that they would open up their hardware specs to allow people to write decent open source drivers for ATI cards. It seems that hasn't happened as quickly as I had hoped. Notice I'm back to nvidia for my new gaming card. Luckily when gnome-shell crashes it doesn't take down the whole system and I can usually hit ATL+F2 to bring up a run dialog (even though it doesn't appear on the screen) and execute "r" which tells gnome-shell to restart itself. Otherwise, I can also switch to a virtual terminal run a <i>gnome-shell --replace </i>command to make everything happy again.<br /><br /><strong>Update</strong>: I think I have tracked down the gnome 3 crashiness. It is in the Mint media player plugin. <a href="https://bugs.launchpad.net/linuxmint/+bug/892316">This bug on launchpad</a> indicates that there is a fix which should be rolled out to the masses shortly. The workaround until then is to just disable the extension.<br /><br /><span style="font-size: large;">Of Zabbix and Migrating Old Data</span><br />I have been using <a href="http://www.zabbix.com/">Zabbix</a> to monitor and track system metrics for a while. Not that I have anything super critical running on my boxes here at home but sometimes it yields some interesting data and I'm kind of a sucker for graphs.<br /><br />I installed the latest zabbix on my new machine but I didn't really want to migrate my whole database from the old one because there was some cruft in it and several old hosts that haven't existed for a while. But there are a few monitored items that I wanted to migrate. In particular, the data that generates my <a href="http://ni.kwsn.net/~toby/OSM/license_count.html">license acceptance graphs</a>. Zabbix does have an export feature that will export a given item but it only exports the definition, not the data. Fortunately it keeps all its data in a pretty plain mysql database and I'm kind of handy with databases so I managed to transition the data as follows:<br /><ul><li><div>Create the item in the new Zabbix install. I did an export from the old system and imported it into the new one to create the item. In this case I also had to copy a shell script that zabbix calls to update the counts and then configure the new zabbix agent to poll it.</div></li><li>Export the data from the old mysql database. First you need the item ID. This can be found by looking at the item you are exporting in the zabbix web web UI and examining the URL. You are looking for an <i>itemid=????</i> parameter. Then plug that in to the following query which will give you a CSV file. Note that the mysql user you connect as must have the <i>file</i> permission - I just logged in as the mysql root user.<br /><pre class="source-code">select * <br />into outfile '/tmp/history_uint_agreed.csv'<br />fields terminated by ','<br />lines terminated by '\n'<br />from history_uint <br />where itemid = 19913;<br /></pre></li><li>Now, find the item ID in the target zabbix install. Then use sed or vim or something to do a string substitution. I used vim with the following command: <br /><pre class="source-code">:%s/19913/22304/</pre></li><li>Import the data into the new database:<br /><pre class="source-code">load data infile '/tmp/history_uint_agreed.csv'<br />into table history_uint<br />fields terminated by ','<br />lines terminated by '\n'</pre></li><li>Finally, activate the item in the new zabbix so that it starts updating with new data.</li></ul><br /><span style="font-size: large;">Of PostGIS, osmosis and the jxapi</span><br />Since I was using PostgreSQL 8.4 on my old system and 9.1 is out now, I ended up just dumping my existing OSM database that I <a href="http://ksmapper.blogspot.com/2011/04/parse-planet-into-pgsnapshot-database.html">blogged about in the past</a> and setting up a brand new one from scratch. I set up the database right before I left for Thanksgiving. I started the planet file torrent and then drove to Nebraska where my only access to the internet was tethering through my phone which only gets EDGE data service up there. Plus, I was of course being (a little) social with family so I didn't have as much time to tinker with it. For a few tweaks dealing with   PostgreSQL 9.1 vs 8.4 I followed <a href="http://www.paulnorman.ca/blog/2011/11/loading-a-pgsnapshot-schema-with-a-planet-take-2/">Paul Norman's post</a> on the subject. But I didn't want to do the dump files with osmosis for a couple of reasons so I went with the <i>--write-pgsql</i> task instead of the <i>--write-pgsql-dump</i> one so that the data would get read out of the planet file and put straight into the database. Well almost. It still creates some temporary files along the way but you don't have to mess with loading them into the database after it finishes. Unfortunately the planet file has grown too much since I last did this and the nodes no longer fit in my 12 GB of memory so I had to use the TempFile option for node location storage. It actually didn't impact the speed nearly as much as I thought it would. I'm guessing Linux's aggressive caching probably paid off and a good chunk of the temporary node file ended up in the 12 GB of memory the system has. My final osmosis command line looked like this:<br /><pre class="source-code">bzcat planet-111123.osm.bz2 | <br />osmosis --fast-read-xml file=- <br />--log-progress interval=30 <br />--write-pgsql nodeLocationStoreType=TempFile <br />              host=localhost database=xapi user=xapi password=xapi </pre><br />Osmosis ran for 53 hours but when it was done, the database was ready to go.<br /><br />For fun, I ran a few <a href="http://wiki.openstreetmap.org/wiki/User:Pnorman/jxapi">benchmark queries</a> that Paul came up with a while ago against my old   PostgreSQL 8.4 and then again against the 9.1 database on new hardware. Before running each query I restarted   PostgreSQL and flushed the disk cache using the command  <br /><pre class="source-code">sync; echo 3 &gt; /proc/sys/vm/drop_caches</pre>to make sure I was getting the raw hardware speed, not assisted by the various caches that the operating system and the database both have. Vital stats of the old and new systems:<br /><b>Old:</b> AMD Athlon x2 4400+ (2.2 GHz) with 3 GB of memory<br /><b>New:</b> Intel Core i5 (2.67 GHz) with 12 GB of memory<br />The I/O system remained pretty much the same: software RAID 5 of four 7,200 RPM 1TB drives with LVM on top of it.<br /><br /><table border="1"><tbody><tr><th>Query</th><th>Old system</th><th>New system</th><th>speedup</th></tr><tr><td>map?bbox=11.54,48.14,11.543,48.145</td>    <td>61 sec</td>    <td>46 sec</td>    <td>25%</td></tr><tr><td>*[bbox=11.54,48.14,11.543,48.145]</td>    <td>56 sec</td>    <td>42 sec</td>    <td>25%</td></tr><tr><td>*[amenity=library][bbox=-0.57,51.24,0.31,51.75]</td>    <td>172 sec</td>    <td>153 sec</td>    <td>11%</td></tr><tr><td>node[power=generator]</td>    <td>55 sec</td>    <td>85 sec</td>    <td>-55%</td></tr><tr><td>*[amenity=parking][capacity:disabled=*][bbox=-0.57,51.24,0.31,51.75]</td>    <td>192 sec</td>    <td>177 sec</td>    <td>8%</td></tr><tr><td>*[waterway=sluice_gate]</td>    <td>40 sec</td>    <td>9 sec</td>    <td>78%</td></tr><tr><td>*[bdad6d21a=1378b02bc]</td>    <td>6.3 sec</td>    <td>3.3 sec</td>    <td>47%</td></tr><tr><td>*[power=generator]</td>    <td>405 sec</td>    <td>367 sec</td>    <td>9%</td></tr><tr><td>map?bbox=11.54,48.14,11.543,48.145</td>    <td>45 sec</td>    <td>43 sec</td>    <td>4%</td></tr><tr><td>map?bbox=-124,49,-123,50</td>    <td>272 sec</td>    <td>227 sec</td>    <td>17%</td></tr></tbody></table><br />Several noticeably faster ones but nothing too earth shattering. Then there is that one that is 55% slower. No clue what's up with that. Of course this is all pretty anecdotal. So much changed between the two runs that it is hard to pinpoint whether the speedup came from the faster CPU, more memory, improvements in &nbsp; PostgreSQL or changes in the data. Probably a little from all of the above.<br /><br />I leave you with a picture of the ridiculous video card fans that I mentioned in the first paragraph. Notice at the upper right corner of the card that it also requires two power connections. I hope it will help me achieve some epic frags at the upcoming semiannual LAN party I have with my college friends. It has already propelled my <a href="http://boinc.berkeley.edu/">BOINC</a> score to new heights.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-aIDpanlgrlo/TtxRzGoB0aI/AAAAAAAAAeM/0KBMS_E6Apg/s1600/SAM_1247.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="470" src="http://1.bp.blogspot.com/-aIDpanlgrlo/TtxRzGoB0aI/AAAAAAAAAeM/0KBMS_E6Apg/s640/SAM_1247.JPG" width="625" /></a></div><br />


guid: tag:blogger.com,1999:blog-7044416767097884399.post-4551257701545339899
url: http://ksmapper.blogspot.com/2011/08/simple-openstreetmap-tile-rendering.html
title: Simple OpenStreetMap tile rendering with Maperitive
published: 2011-08-21 03:44:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.835425
updated_at: 2019-10-25 14:30:27.190072
summary:
  Lately there have been a lot of people popping into the OSM IRC channel (<a href="http://irc.openstreetmap.org/">#osm on oftc.net</a>) because they have been getting the "you've been bad" tile from the OSM tile server:<br />
   <br />
   <div class="separator" style="clear: both; text-align: center;">
   <a href="http://wiki.openstreetmap.org/w/images/d/df/Tile_Abuse.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://wiki.openstreetmap.org/w/images/d/df/Tile_Abuse.png" /></a></div>
   <br />
   The reason is that they are using tile scraping applications that violate the OSM <a href="http://wiki.openstreetmap.org/wiki/Tile_usage_policy">tile usage policy</a>. Typically people just want to download tiles for their city or a vacation destination or travel route in order to pre-load them on a mobile device for offline use.<br />
   <br />
   This is of course a perfect use case for OSM. The problem is that OSM is a project run by volunteers and with limited server capacity. Our primary product is geographic data, not map tiles. So when these tile scraping programs start hitting high zoom levels, it can cause serious problems for the tile server. The solution is to download OSM data instead of rendered tiles. Then you can render your own tiles or even use offline vector rendering.<br />
   <br />
   One tool to make your own tiles is Maperitive. It is pretty easy to use and by default generates tiles that are fairly similar in style to the default mapnik style you see on osm.org although it is very customizable. It is a .NET application although it claims to work in Linux/OSX with mono. Also, while it is free to download, it is not open source which is kind of rare for the OSM world. These two things may cause some people to have an allergic reaction (I'm looking at you, Serge) but if you're still reading, here is a quick lesson on how to use it to render your own tiles. If you keep reading, you should be able to start rendering your own tiles in about 10 minutes. There is a tutorial video and some detailed documentation on the Maperitive website but I don't see a basic text based tutorial that walks you through the common "render some tiles for this area" use case.<br />
   <br />
   <span class="Apple-style-span" style="font-size: large;"><b>Getting Started</b></span><br />
   Installation is pretty straightforward. Just head over to&nbsp;<a href="http://maperitive.net/">http://maperitive.net/</a> and find the "Download" link. Grab the latest .zip file (Maperitive-1228.zip right now) and download it. Once the download finishes, just unzip the Maperitive folder inside of the zip file somewhere and you're done. Open the Maperitive.exe file to fire it up.<br />
   <a name='more'></a><br />
   Here is a screen shot of my Maperitive with some OSM data already loaded in. It should all make sense by the end of this post.<br />
   <div class="separator" style="clear: both; text-align: center;">
   <a href="http://2.bp.blogspot.com/-Eh2NEY6j3Mw/TlAarp7qc0I/AAAAAAAAAaM/BlfupgaW-eU/s1600/Maperitive_screenshot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="http://2.bp.blogspot.com/-Eh2NEY6j3Mw/TlAarp7qc0I/AAAAAAAAAaM/BlfupgaW-eU/s640/Maperitive_screenshot.png" width="600" /></a></div>
   <br />
   The first time you fire up Maperitive, you will notice there is already a map on the screen! By default it just starts loading map tiles from osm.org so you can get your bearings and navigate to an area of interest. Next, look at the bottom of the window. On the left is a text area (labeled "Commander" in the tab above it) where status messages and command outputs are displayed. Below that is a command line. This is a graphical program but a lot of the advanced functionality is done via this command line. A note about typing commands: Maperitive automatically completes commands for you which is nice... but it does it in a rather annoying way. As soon as you have typed enough for it to identify a unique command, it immediately fills in the rest of the command. I am much more used to a user-driven tab completion style as seen in most linux shells so the automatic completion in the middle of typing a command means my fingers keep typing the command after Maperitive auto-completes it for me and I have to backspace to erase extra characters.<br />
   <br />
   On the right side is a "Sources" section that lists what map sources are currently being displayed in the main window. You can turn each one on and off by clicking the star by it. In the menus are various options. When you select one it will execute a command that you can see being executed in the Commander pane at the bottom. For example, find your home city and then go to the View menu and select "Set Home Location." You will see the text "<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">&gt; set-home</span>" appear in the Commander. You can do the same thing by typing "<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">set-home</span>" in the command prompt and hitting enter. Now you can use the "Go Home" option to snap back to your "home" location from anywhere.<br />
   <br />
   <b><span class="Apple-style-span" style="font-size: large;">Selecting an area</span></b><br />
   Next, you must select an area of the world to render. You can navigate with your mouse just like on osm.org. Holding down shift while dragging with the mouse will draw a box. After you release the mouse button the view will zoom to that box. I'm not really sure what the limits of Maperitive are as far as volume of data. Note that the way it downloads data from the XAPI puts a limit of 10 square degrees on the download area. But this translates to a box of over 100 kilometers on a side and I'm guessing Maperitive will have a hard time with that much data anyway. I tried my county which is about 30x60 kilometers (18x37 miles). It contains about 26 MB of data in .osm format. I was able to render tiles on a 2.8 GHz Core i5 system down to zoom level 18 in about an hour with a memory footprint of under 300 MB. This should be a big enough example to cover many use cases so I will be using this area in my examples.<br />
   <br />
   By default, Maperitive renders everything you see in the map on your screen. So if you are happy with that, you can go to the next step. To customize the area to something more specific than the shape of the window, read on.<br />
   <br />
   You can set a more specific bounding box for all your operations by using the command prompt. One easy way to find a bounding box is to use the <a href="http://openstreetmap.us/uixapi/xapi.html">XAPI frontend</a>. Browse the map and then hold shift while dragging the mouse to draw a box. After drawing the box, check the "Search by Area" checkbox above the map. Now look down at the bottom of the page and copy the latitude/longitude coordinates after "bbox=" - in my case:&nbsp;-96.964,39.025,-96.547,39.573 (do NOT copy the ] at the end!)<br />
   <br />
   Now go back to Maperitive and use the bounds-set command:<br />
   <pre class="source-code">bounds-set -96.964,39.025,-96.547,39.573</pre>
   Maperitive will reply with "Locking the bounds to ..." and will draw a thick dotted line on the map to show the bounding box you have selected. It is clearly visible in the screenshot up above.<br />
   <br />
   <b><span class="Apple-style-span" style="font-size: large;">Downloading data</span></b><br />
   Now that you have your area of operations defined, it is time to download data. This should be as simple as selecting the "Download OSM Data" option in the Map menu. This will hit the java XAPI service hosted by MapQuest Open and download the data within your specified bounds. One problem with Maperitive is that its network timeout is very short. So if you get an error that says "The operation has timed out" then the XAPI server just responded too slowly. You can try again but you may need to reduce the size of your bounding box or try one of the other XAPI servers listed on the <a href="http://wiki.openstreetmap.org/wiki/Xapi#Servers">XAPI wiki page</a>. To specify the server you add an "xapi-url=" argument to the command. I hit my own XAPI server that I have set up here at home. To do this I issued this command:<br />
   <pre class="source-code">download-osm xapi-url=http://localhost:8080/xapi/api/0.6/*</pre>
   <br />
   Alternatively&nbsp;you can download the data using another method. (in JOSM or the "Export" tab on osm.org to name two) &nbsp;Then you can go to the File menu and use the "Open Map Sources" option to load the file.<br />
   <br />
   Once the data is loaded, turn off the Web map display by clicking the star over in the Map Sources tab. This will remove the tiles downloaded from osm.org from the display and show only what Maperitive has rendered from the downloaded data. It will be fairly similar but not identical.<br />
   <br />
   <b><span class="Apple-style-span" style="font-size: large;">Generating the tiles</span></b><br />
   Almost done! Now all you have to do is generate tiles. First, make sure that the web map layer is disabled or deleted in the Map Sources tab. Otherwise Maperitive uses map tiles from osm.org as a background layer. This causes it to scrape tiles which is what we are trying to avoid in the first place! There is a command in the Tools menu that will generate tiles. By default it uses how far you are zoomed in on the map window and renders 5 zoom levels of tiles based on that. If you want to generate a wider range of zoom levels, go back to the command line. This will render your bounding area from zoom level 8 down to 18:<br />
   <pre class="source-code">generate-tiles minzoom=8 maxzoom=18</pre>
   <br />
   The tiles will be generated in a directory named "Tiles" inside of the Maperitive directory, wherever you unzipped it to. If you are going all the way to zoom level 17 and 18, it could take a while to render so go outside and collect some map data or something.<br />
   <br />
   When the process finishes, you will have a standard tile directory tree just like you would have had by scraping tiles from osm.org with Mobile Atlass Creator or whatever other scraping software people have come up with. Load it onto a phone for offline viewing or throw it up on a web server using OpenLayers... whatever your heart desires.<br />
   <br />
   <b><span class="Apple-style-span" style="font-size: large;">Easy updating</span></b><br />
   Maperitive supports making simple scripts. So if you intend on updating your tiles periodically you might want to make a script so you can just fire and forget. Scripts are run from the File menu. Here is one that performs all the steps listed above for my county but only renders tiles between level 8 and 15. It takes about 90 seconds to execute including download time from the XAPI which can vary depending on current server load and size of area. You can copy this script, change the numbers in the bounds-set command and the zoom levels you want to render and then run it to generate tiles for your desired area. Maperitive expects script files to have a .mscript extension.<br />
   <br />
   <pre class="source-code">clear-map
  <br />use-ruleset alias=default
  <br />bounds-set -96.964,39.025,-96.547,39.573
  <br />download-osm
  <br />generate-tiles minzoom=8 maxzoom=15
  <br /></pre>
   <br />
   Maperitive can do a lot of other things like adding hillshading or relief contours (elevation information) to the map.There are also several styles that come with Maperitive besides the mapnik look-alike one. You can poke around the menus to see some of this functionality or execute the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">help-commands</span> command on the command line to get a list of all available commands. More information about a specific command can be seen by using help-commands again and adding the specific command you want more information about. Example: <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">help-commands generate-tiles</span> will give you more details about the command that renders tiles.<br />
   <br />
   I think I am going to do another post about Maperitive that will go in to more details about customizing rendering rules for a specific purpose. Until then, happy rendering!


guid: tag:blogger.com,1999:blog-7044416767097884399.post-5515794768347354757
url: http://ksmapper.blogspot.com/2011/06/cleanup-on-map-aisle.html
title: Cleanup on the Map Aisle
published: 2011-06-25 22:04:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.838265
updated_at: 2019-10-25 14:30:27.192917
summary:
  Spills happen. A sticky honey jar in the grocery store. A glass of orange juice at breakfast. Oil spills in the Gulf of Mexico. Node spills in OpenStreetMap. Some can be cleaned up with a mop and some water. Others require a more technical solution. What is a "node spill" you ask? It's when nodes get uploaded to OSM that don't have any tags on them and aren't part of a way. These tagless, unconnected nodes add no useful information to the map and are just dead weight in the database. Where do they come from? There are at least a couple of common sources of node spills. One is <a href="http://trac.openstreetmap.org/ticket/2700">editor bugs</a> and simple user error. These are generally pretty small spills of 10s or maybe 100s of nodes. The bigger problem is imports.<br /><br />The topic of imports deserves its own post. For now I'll just say that badly performed imports and insufficient checking after imports can lead to 10s of thousands of empty nodes. The basic problem is that nodes get uploaded first and don't become part of a way until the way is uploaded later. So if something goes wrong with the way upload, the nodes end up in an orphaned state.<br /><br />I have come across a few of these nodes before but what really caught my attention was a failed NHD (National Hydrography Dataset) import in Oklahoma. It happened to poke up into Kansas a little bit where I noticed it next to a state highway I was editing. After a <a href="http://lists.openstreetmap.org/pipermail/talk-us/2011-May/thread.html#5782">lengthy thread</a> on the talk-us mailing list I eventually found a good way to detect these nodes. Since then I have made my way across the US and some of Canada deleting these useless nodes from the database. As detailed in <a href="http://lists.openstreetmap.org/pipermail/talk-us/2011-May/005824.html">this message</a>&nbsp;it goes something like this:<br /><br /><ul><li>Use the XAPI to perform a query of the form /node[not(way)][bbox=a,b,c,d]</li><li>Open the result of this query in JOSM and apply a filter to hide all nodes with tags</li><li>Perform some checks to make sure there really is no useful data</li><li>Delete nodes and upload</li></ul><br /><a name='more'></a><b>XAPI Query</b><br />The [not(way)] xapi predicate is documented on the OSM wiki <a href="http://wiki.openstreetmap.org/wiki/Xapi#Child_Element_Predicates">XAPI page</a>. In addition you need to add a bounding box to the query. An easy way to come up with a bounding box is to use the uixapi page found at &nbsp;<a href="http://openstreetmap.us/uixapi/xapi.html">http://openstreetmap.us/uixapi/xapi.html</a>. Hold down shift and draw a box around the area you wish to query. Then tick the "Search by Area" checkbox above the map. &nbsp;The URL at the bottom of the page will change to include a [bbox=...] portion. Note that the bbox won't update automatically if you draw another box. You need to uncheck and recheck the box for it to update.<br /><br />If you are using one of the public XAPI services then the size of the bounding box will be limited to 10 square degrees. Since I am running these queries against my own XAPI installation, I removed that limitation so I've just been doing entire states at a time. Depending on how much data is in a given area, the queries take very different times to return. All of Alaska took under 10 minutes while some of the small states on the east coast took about an hour.<br /><br />I started out downloading the data straight into JOSM using the "Open Location..." feature in the File menu but for particularly dense areas this caused some strange problems so I switched to using wget to download the data to a file and then opened the file in JOSM. I learned something about wget: it has a default timeout of 15 minutes. So if the query takes longer than 15 minutes, wget will give up and try again which will result in two identical queries running simultaneously and slowing things down even more. So remember to use the --timeout=0 option to disable the timeout in wget. Here is a sample wget command for Kansas:<br /><pre class="source-code">wget --timeout=0 -O KS.osm "http://localhost:8080/xapi/api/0.6/node[not(way)][bbox=-102.12,36.87,-94.46,40.11]"</pre><br /><b>Loading into JOSM</b><br />The previous step gives you a .osm file that contains all nodes that are not part of a way in the area you queried. Obviously this includes all POIs that contain useful data. The file can be rather large and may require that you give JOSM a couple of GB of memory to use. Here is a screen shot of Maine:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-ye1QAVc6Y8M/TgYzFw3Kn8I/AAAAAAAAAWE/FXvLTyyudYo/s1600/Maine_all_nodes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="490" src="http://4.bp.blogspot.com/-ye1QAVc6Y8M/TgYzFw3Kn8I/AAAAAAAAAWE/FXvLTyyudYo/s640/Maine_all_nodes.png" width="625" /></a></div><br />Now let's filter out all nodes with useful information. In this case you can see a filter in the bottom right of the screen of the form "-untagged" which hides all nodes with tags except for nodes that <b>only</b> have the following tags: attribution, created_by, source, fixme and note. (I won't guarantee that this is a complete list) While nodes with only these tags aren't really useful map data, you do need to be careful because nodes with a fixme or a note tag could be useful information to other mappers. The other stricter JOSM filter that I used sometimes was "tags:1-999" which filtered out all nodes with <b>any</b>&nbsp;tags on them. After filtering, things are a little more&nbsp;manageable:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-iLCeTe7BKHc/TgY1Dtzo1yI/AAAAAAAAAWI/ULWkEq53iKY/s1600/Maine_all_empty.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="490" src="http://2.bp.blogspot.com/-iLCeTe7BKHc/TgY1Dtzo1yI/AAAAAAAAAWI/ULWkEq53iKY/s640/Maine_all_empty.png" width="625" /></a></div><br />Now for a bit of data analysis. Over on the right side of the window you can see I have the "Authors" panel open which lists who the last person was to touch the currently selected objects. The most frequent user in the US was woodpeck_fixbot. After some digging I found out why. The Census Bureau TIGER data was originally imported with several tags such as source and an upload_uuid tag on every single node. After the import, it was determined that these tags on the 10s of millions of nodes in the TIGER data were&nbsp;unnecessary and bloated the database and the planet file so much that it was decided to remove them. This removal was done by woodpeck_fixbot. So all these nodes are originally from the TIGER import and I deleted these without a second thought since they have been sitting in the database for several years.<br /><br />However for other cases I did more checking. I investigated the changesets that left a lot of empty nodes to determine what was being done in these uploads and whether the user noticed the failure. Sometimes it was obvious that the failure was noticed and the data was re-uploaded so that the empty nodes were actually duplicates of data that was successfully uploaded later. In this case deletion was also obviously&nbsp;warranted. One way to check this is to just zoom in on some of the nodes with the OSM tiles in the background:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-tbn3wqVvH_I/TgY4WypegqI/AAAAAAAAAWM/ISrPPD_2tEs/s1600/Maine_NHD_Verify.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="490" src="http://4.bp.blogspot.com/-tbn3wqVvH_I/TgY4WypegqI/AAAAAAAAAWM/ISrPPD_2tEs/s640/Maine_NHD_Verify.png" width="625" /></a></div><br />Obviously these nodes were intended to be part of the border of this lake which exists. In this case it was doubly confirmed by looking at the changeset comment where it stated that this was an upload of NHD data.<br /><br />Unfortunately sometimes the uploader did not notice the error so no re-uploading took place and there were no features present. In some of these cases I contacted the user to make sure they knew about the error and see if they had a way of recovering from it. A surprising number of them knew there was a problem during the upload process and thought they had fixed it but they didn't get everything. Just about everyone I contacted said to go ahead and delete the nodes.<br /><br />The other thing you really have to watch out for is currently running imports. I hit one of those in St Louis. Someone was actively importing NHD data while I was cleaning up empty nodes in the area. If I hadn't noticed this and blindly deleted the nodes it would have caused their upload to fail horribly and led to all kinds of headaches including even more empty nodes. One good way to do this is to select all the nodes you are thinking about deleting and then using the JOSM search feature with the string "timestamp:2011" to find nodes last touched in 2011 within your current selection. These can then be more thoroughly investigated.<br /><br /><b>Uploading</b><br />After sufficient checking had been performed, I deleted the nodes and uploaded them with a helpful changeset comment.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-Sgb2moqKD-c/TgY-FObHmWI/AAAAAAAAAWQ/uFZ4lssrbno/s1600/Maine_upload.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="440" src="http://2.bp.blogspot.com/-Sgb2moqKD-c/TgY-FObHmWI/AAAAAAAAAWQ/uFZ4lssrbno/s400/Maine_upload.png" width="400" /></a></div><br />There is one final kink that popped up a couple of times during upload. While all these nodes are not used in a way, a few of them were members of a relation. Most of these cases were clear errors. For example some empty node in the middle of Nebraska was a member of an administrative boundary in France. No clue how that happened. I'm assuming some editor or import bug. There were several in California that were members of turn restriction relations with a "location_hint" role. Not sure what that's supposed to mean but of course I left them alone. In these cases the API will return an error saying that the node is still in use by a relation and you can't delete it. To resolve the conflict you can take the relation ID from that error message and download the relation using JOSM's "Download Object" feature. It will complain that there is a conflict which can then be resolved in the conflict editor. Then the upload can continue.<br /><br />Here is a shot of <a href="http://wiki.openstreetmap.org/wiki/LiveMapViewer">LiveMapViewer</a>&nbsp;&nbsp;after a day of work near the Great Lakes. Obviously this includes the work of other mappers as well but most of that red is the result of my deletions.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-1SAEJjh6P50/TgZPdQ2OIYI/AAAAAAAAAWU/d69v62hESHY/s1600/node_cleanup.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="http://1.bp.blogspot.com/-1SAEJjh6P50/TgZPdQ2OIYI/AAAAAAAAAWU/d69v62hESHY/s640/node_cleanup.png" width="625" /></a></div><br /><br /><b>Conclusion</b><br />As stated above, I have covered the entire US with this cleanup. In most places I was pretty conservative in what I deleted in that I only deleted those nodes that I positively identified as part of an import or otherwise determined what had happened and why. Other nodes that I wasn't sure about I left in place. I have yet to receive a single complaint about any of these edits so I think I did ok. Despite the conservative deletion I have nuked just over 411,000 nodes. This translates to a space savings of about 50 MB in the planet.osm file assuming each empty node takes about 125 bytes of space. Guess that's not even a drop in the bucket. But still a worth while cleanup effort. It even got me pretty high up on the monthly list of most active OSM users on the <a href="http://www.openstreetmap.org/stats/data_stats.html">data stats</a>&nbsp;page!<br /><br />Some of the more extreme cases of bad data I found were a failed upload of parcel data in Arkansas that was supposed to have been cleaned up by the original uploader but was not due to a bug in the editor. It took <a href="http://www.openstreetmap.org/browse/changeset/8186396">three</a>&nbsp;<a href="http://www.openstreetmap.org/browse/changeset/8186543">changesets</a>&nbsp;to <a href="http://www.openstreetmap.org/browse/changeset/8186642">clean</a> that up. Then there was the case of ridiculous over-noding. There were <a href="http://www.openstreetmap.org/browse/changeset/8501065">14,000 nodes</a> that were mostly contained in about a 4 block area near San Bernadino in California. That amount of bad data can actually make it difficult for casual mappers to do anything in the area since some editors may not handle that volume of data very well.<br /><br />Time to go get some fresh mop water and continue the cleanup.


guid: tag:blogger.com,1999:blog-7044416767097884399.post-650569282152703009
url: http://ksmapper.blogspot.com/2011/05/main-attraction.html
title: The Main Attraction
published: 2011-05-15 08:29:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.840964
updated_at: 2019-10-25 14:30:27.195665
summary:
  I have mentioned the TIGER import several times. The initial import tagged roads with abbreviations in their name. For example "Ave" instead of "Avenue" or "St" instead of "Street." However the general rule in OSM is to <b>not</b> abbreviate things. So a script was devised to automatically expand abbreviations. This script was run by the user <a href="http://www.openstreetmap.org/user/balrog-kun">balrog-kun</a>. Until tonight I assumed that it had been run over all of the TIGER data. But then I saw a few abbreviations while working on some <a href="http://wiki.openstreetmap.org/wiki/Kansas_state_highways">highways in Kansas</a> and decided to dig a little deeper.&nbsp;I did some XAPI queries (using the database I talked about in a <a href="http://ksmapper.blogspot.com/2011/04/parse-planet-into-pgsnapshot-database.html">previous post</a>!) &nbsp;to see if anything interesting stood out. I did one query for all streets named "Main St" and another for all roads named "Main Street" and here is what I got back.<br /><br />Green is "Main St" and the rest is "Main Street"<br /><br /><div class="separator" style="clear: both; text-align: left;"><a href="http://4.bp.blogspot.com/-7FlvxJcl_UA/Tc9-2zprC_I/AAAAAAAAAVk/wh5_sInE53E/s1600/MainSt.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="441" src="http://4.bp.blogspot.com/-7FlvxJcl_UA/Tc9-2zprC_I/AAAAAAAAAVk/wh5_sInE53E/s640/MainSt.png" width="625" /></a></div><div class="separator" style="clear: both; text-align: left;"></div><a name='more'></a><br /><div class="separator" style="clear: both; text-align: left;">First of all, I am amused at just how many "Main" streets there are in the US. But also it looks like the name expansion bot was only set loose west of the Mississippi. The bot was run long before I joined OSM so I was not around for any discussions about why this was done. I'm sure it is buried somewhere in the mailing list archives...</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">I haven't done any substantial editing east of the Mississippi which explains why I hadn't noticed abbreviated street names before. I'm still not sure why I found some in western Kansas. The most likely explanation is that a few of the bot uploads failed for some reason, leaving pockets of abbreviations around.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">I pinged balrog-kun about it on IRC so the mystery may be solved in a few hours but I thought the image was amusing enough to share anyway. And for a few more giggles, here is the whole world:</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><a href="http://3.bp.blogspot.com/-t1KPId7_Sik/Tc-FXSbFybI/AAAAAAAAAVs/V0NpWCpVCQA/s1600/MainSt_world.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="410" src="http://3.bp.blogspot.com/-t1KPId7_Sik/Tc-FXSbFybI/AAAAAAAAAVs/V0NpWCpVCQA/s640/MainSt_world.png" width="625" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">It seems the UK has a fair share of Main streets as well, plus a few scattered mostly in former British colonies. I'm sure they started it and then I guess we here in the US just <a href="http://www.youtube.com/watch?v=EbVKWCpNFhY">turned it up to 11</a>.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Update After talking to balrog-kun: Apparently he was moving west-to-east and at some point some people started complaining for some reason so he stopped. Fair enough I guess. He's not sure about the ones I found in western Kansas though.&nbsp;</div>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-2963103839147233697
url: http://ksmapper.blogspot.com/2011/04/one-mapper-one-year.html
title: One Mapper, One Year
published: 2011-04-10 20:36:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.843646
updated_at: 2019-10-25 14:30:27.198379
summary:
  Today is the one year anniversary of<a href="http://www.openstreetmap.org/browse/changeset/4388684"> my first OpenStreetMap edit</a>! I thought I would take the opportunity to reflect on how OSM and myself have had an effect on each other. My intention is not to brag about all the mapping I have done but rather to point out some different ways in which contributions can be made and some of the tools I use when mapping.<br /><br />My very first edit was to add a foot/bike path near my apartment so that it would show up on <a href="http://opencyclemap.org/">OpenCycleMap</a>. Unfortunately the imported TIGER roads were so far off that I had to realign them before I could add the path in the correct location. And of course correcting the ones right around the path led me to correct the ones connecting to those and the ones connecting to <b>those</b> and so on... At the time I didn't even know what TIGER was or what all those tiger:* tags were on all the roads.<br /><br />Some highlights since then:<br /><ul><li>GPS trackpoints <a href="http://www.openstreetmap.org/user/ToeBee/traces">uploaded</a>: 283,271 in 75 traces (most from my Garmin Edge 305)</li><li>410&nbsp;<a href="http://www.openstreetmap.org/user/ToeBee/edits">changesets</a>&nbsp;created&nbsp;using Potlatch, JOSM, OsmAnd and Vespucci</li><li>Pictures taken: over 2,000&nbsp;(4.5 GB saved)&nbsp; Most were geotagged to a GPS trace to aid in mapping features (uploaded some of them to&nbsp;<a href="http://openstreetview.org/?lat=39.941890036188&amp;lon=-100.7890033705&amp;zoom=5">openstreetview.org</a>)</li><li>Some of the locations I have edited:&nbsp;</li><ul><li>My <a href="http://www.openstreetmap.org/browse/changeset/4711369">university campus</a></li><li>My <a href="http://www.openstreetmap.org/browse/changeset/4972565">city</a></li><li>My <a href="http://www.openstreetmap.org/browse/changeset/5642488">state</a></li><li>My <a href="http://www.openstreetmap.org/browse/changeset/5214118">country</a></li><li>My <a href="http://www.openstreetmap.org/browse/changeset/4682033">high school</a></li><li>My <a href="http://www.openstreetmap.org/browse/changeset/7301352">childhood home</a></li><li>Some <a href="http://www.openstreetmap.org/browse/changeset/7662208">random place</a> where there was an earthquake</li></ul><li>Map objects affected (according to&nbsp;<a href="http://hdyc.neis-one.org/?ToeBee">hdyc.neis-one.org</a>&nbsp;- last updated in January)</li><ul><li>Nodes: about 165,000</li><li>Ways: about 24,500</li><li>Relations: about 500 (I created over 200 county relations in 3 states since the last update)</li></ul><li>Contacted multiple government agencies in Kansas to get explicit permission to use their data to assist in mapping. Documented these data sources on the <a href="http://wiki.openstreetmap.org/wiki/Kansas">Kansas </a>page in the wiki</li><li>Started a <a href="http://ksmapper.blogspot.com/">blog</a>&nbsp;(self referring link!)</li></ul><br />Now for a pretty picture. Here are all the GPS traces in my city. I think there may be one or two from other people but most of it is me.<br /><a name='more'></a><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-rtpNSOAPLeM/TaHL1wKKeVI/AAAAAAAAAVY/kVy8PBm5R2s/s1600/MHK_gps_traces.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="537" src="http://4.bp.blogspot.com/-rtpNSOAPLeM/TaHL1wKKeVI/AAAAAAAAAVY/kVy8PBm5R2s/s640/MHK_gps_traces.png" width="630" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">GPS traces in Manhattan, KS</td></tr></tbody></table><br />Most of them were collected on my bicycle although there are a few roads in town that I do not bike on because I do not have a death wish so those were done in the car. But I still have work to do! On Saturday I was talking to some people in the <a href="http://www.fhabclub.org/">Flint Hills Area Bike Club</a> and heard laments of new streets missing from maps. They were of course talking about google maps but if these new streets are added to OSM first, I may have some converts!<br /><br />And speaking of converts, I have also had the pleasure of working with some other locals to enhance and use OSM data in interesting applications. For example, <a href="http://ksuanth.weebly.com/wesch.html">Michael Wesch</a> is a professor at Kansas State University. He used OSM data to make a <a href="http://bikemanhattan.info/map.htm">bicycle safety map</a>&nbsp;for the city. In trying to get the map into a usable state, he added a lot of the bigger landuse areas in the city that you now see on the map. He also convinced the College of Architecture,&nbsp;Planning&nbsp;and Design to focus its "<a href="http://www.k-state.edu/media/newsreleases/aug10/bikedesign82010.html">Design Days</a>" project on bicycle friendliness and to use OSM as their map. Almost 200 students were sent out into the city with <a href="http://walking-papers.org/">walking papers</a> to find any bicycle oriented path or amenity that wasn't on the map and report back to people on campus who then entered the information into the OSM database. I had about 10 minutes to teach 10 people how to use JOSM. Oh the things I would now do differently.<br /><br />Several new paths were found that connect neighborhoods and open up the option of safely commuting on a bicycle or by foot to school and the university. Many of these paths will never make it to google/yahoo/bing/etc maps because they are not officially maintained or even known about by city officials. They are just paths that people have worn into the ground by repeated use. At the end, each group presented plans for improving one section of the city and most used OSM renderings in their presentations. I will admit that I felt a sense of satisfaction seeing mapnik renderings of the city printed out on three foot wide paper and being displayed in projected presentations.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-ihwSCvjgkBE/TaHVRTTPFrI/AAAAAAAAAVc/X17LC-ezZEI/s1600/2010-08-25+17.52.11.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="475" src="http://4.bp.blogspot.com/-ihwSCvjgkBE/TaHVRTTPFrI/AAAAAAAAAVc/X17LC-ezZEI/s640/2010-08-25+17.52.11.jpg" width="630" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">A giant printed map for people to mark on at the Design Days public presentation</td></tr></tbody></table><br />And speaking of the university, my involvement in OSM got me interested in geographic data in general and I decided to take advantage of the university employee tuition discount. I took a cartography course last fall and am now in the middle of the GIS I course. It is interesting to see differences in the way OSM and "professional" GIS applications (mostly ESRI ArcGIS for these courses) handle data and allow you to interact with it. Both have different strengths and weaknesses.<br /><br />After my dramatic and internationally anticipated* entrance into the blogosphere with my <a href="http://ksmapper.blogspot.com/2010/10/tiger-ways-in-kansas.html">TIGER ways in Kansas</a> post, I was made aware of and encouraged to attend <a href="http://www.wherecamp5280.org/">WhereCamp5280</a> in Denver. Here I met <a href="http://www.stevecoast.com/">Steve</a>&nbsp;and&nbsp;<a href="http://www.hurricanemcewen.com/">Hurricane</a>&nbsp;Coast, <a href="http://weait.com/">Richard Weait</a> and <a href="http://blog.samat.org/">Samat Jain</a> among others. I learned several things that day and had some fun while I was at it. Ski mapping in <a href="http://www.openstreetmap.org/?lat=39.87943&amp;lon=-105.77013&amp;zoom=15&amp;layers=M">Winter Park</a> the next day didn't hurt the "fun" aspect either!<br /><span class="Apple-style-span" style="font-size: xx-small;">*for some definition of "dramatic" and "internationally anticipated"</span><br /><br />A lot of my mapping has obviously been local to <a href="http://www.openstreetmap.org/?lat=39.1981&amp;lon=-96.5812&amp;zoom=13&amp;layers=M">Manhattan, KS</a>. (Note: I have heard rumors that some other "Manhattan" exists up in the northeast of the country. I have been unable to verify this rumor personally and assume it is false based on my own&nbsp;empirical&nbsp;evidence) This is OSM's strength. Get people who know their own neighborhood to add detail to the map and then update it as it changes. As evidenced by the GPS traces above, I have explored much of the city. Doing so on a bicycle has been really fun. I have discovered interesting neighborhoods and parts of the city that I had never seen before. It surprised me how much I did <b>not</b>&nbsp;know about this relatively small city even after having lived here for over 10 years.<br /><br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-JrqBEw1MfPM/TaIBnm05lFI/AAAAAAAAAVg/xlAlO8vz11Q/s1600/map.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="590" src="http://2.bp.blogspot.com/-JrqBEw1MfPM/TaIBnm05lFI/AAAAAAAAAVg/xlAlO8vz11Q/s640/map.png" width="630" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Current map of Manhattan, KS</td></tr></tbody></table><br />Other edits have touched places that I have visited on trips. After the trip to Denver, I edited all along I-70. Mostly just improving alignment plus adding the occasional gas station, hotel or other business along the interstate that I managed to snap a picture of as I went past. I also ended up fixing a major cloverleaf intersection in Denver that the local OSMers hadn't gotten to yet. Tsk. Tsk.<br /><br />Some of the more regional editing started with tracing the <a href="http://www.openstreetmap.org/browse/relation/1061544">Kansas River</a> from USGS imagery. Someone else happened to be doing the same thing, starting from the east. I started from the west and we met up in the middle. More recently I have started doing some edits that touch much larger areas. For example, while browsing the <a href="http://open.mapquest.com/">MapQuest Open</a> map, I noticed that county name labels were being placed towards the southeast of the county instead of the middle where you would expect them to be. After some investigation I realized that they are rendering the county name from the place=county nodes and that when these were originally imported, they were all off in the extreme southeast of their counties for some reason. So I used an XAPI query to get all the county nodes in Kansas and another one to get all the county borders. Then I moved all the nodes to be approximately in the center of their respective counties. Now the MapQuest site shows county names in the correct location.<br /><br />While local mappers are key to OSM's value, people with some data analysis skills and above average technical abilities are also needed to tend to some of the "bigger" data that local mappers may not notice or care about. In the US, a lot of this data that needs tending to comes from data imports. Administrative boundaries, TIGER data, etc. Imports is an entirely separate topic that I won't get in to here though.<br /><br />Another issue that requires some data analysis is tagging. Luckily we have some great tools such as <a href="http://tagstat.hypercube.telascience.org/">tagstat </a>and <a href="http://taginfo.openstreetmap.de/">taginfo </a>to help explore tag usage. One problem with tags is typos. Recently I got a message from a mapper who discovered through one of these tools that I had mistyped the word "petroleum" when mapping some of the oil wells south of the city. Thanks to JOSM's auto complete feature when entering tags, the typo had been applied to all of the wells in the area. Thanks to this other mapper discovering the outlying values in this tag, the typo was discovered and corrected.<br /><br />Another problem with tags is inconsistencies. Unfortunately, as with many open source projects, documentation seems to be a low priority. When I first tried to figure out how to tag a doctor's office, I found 3 different pages on the OSM wiki describing how to tag them. One of them was just a stub page. The other two were fairly well fleshed out. But by looking at the tagstat I found that the one that was being used the most by far was the tag&nbsp;corresponding&nbsp;to the stub page. It looks like this has been cleaned up somewhat now. There are still <a href="http://wiki.openstreetmap.org/wiki/Proposed_features/Doctor">two</a>&nbsp;different <a href="http://wiki.openstreetmap.org/wiki/Tag:amenity%3Ddoctors">pages</a>&nbsp;with a proposal to merge them.<br /><br />Initially I was really frustrated by the disorganized tagging rules. In my day job, inconsistent data is sometimes worse than missing data. I still haven't entirely gotten over it but at the end of the day, I consider the various sources and tag things however it makes the most sense to me. The bad thing about such an approach is that it makes the OSM data harder for our consumers to use because they have to consider all the tagging variations for a given type of object. The good news is that since I had these problems, a lot of commonly mapped objects have been added to presets that various editors have so most users don't need to know about the intricacies of tag values or the impressive debates that surround them on various <a href="http://lists.openstreetmap.org/listinfo">mailing lists</a>. Just use the preset and you're done - and the data is consistent.<br /><br />In summary: It has been a fun year learning how to map, meeting other mappers, getting involved in the mapping community and experimenting with map data. I plan to continue! In fact, I just recently decided to make the relationship official and became a card-carrying member of the&nbsp;<a href="http://www.openstreetmap.us/">OSM-US</a> foundation. Although I'm still waiting on the actual card to arrive. In the meantime...<br /><br /><span class="Apple-style-span" style="font-size: large;">Map the planet!</span>


guid: tag:blogger.com,1999:blog-7044416767097884399.post-4792440131163113296
url: http://ksmapper.blogspot.com/2011/04/keeping-database-up-to-date-with.html
title: Keeping a database up to date with osmosis
published: 2011-04-09 04:34:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.846376
updated_at: 2019-10-25 14:30:27.201340
summary:
  This is a follow-up to my <a href="http://ksmapper.blogspot.com/2011/04/parse-planet-into-pgsnapshot-database.html">previous post</a> about importing a planet file into a pgsnapshot database schema. However the same basic process applies to other schemas that osmosis can write to.<br />
   <br />
   To briefly set the stage: After downloading the weekly planet file and processing/importing it into a database, the data is now probably 3 or 4 days old. OpenStreetMap provides both minutely and hourly change files that can be applied to the database to bring it up to date with what is live right now in the master OSM database. If you think about it, that is actually a pretty sweet deal. Someone on the other side of the planet can spot a new restaurant while on their lunch break, use an application on their phone to upload it to OSM and you will get the change applied to your local database in less than &nbsp;120 seconds. Does this blow anyone else's mind just a little? Open data is awesome like that.<br />
   <br />
   <a name='more'></a><br />
   <span class="Apple-style-span" style="font-size: x-large;">Initializing the working directory</span><br />
   <br />
   So what does it take to pull of this feat of mad wizardry? The "<a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#Replication_Tasks">Replication Tasks</a>" section of the osmosis usage page is of interest. Again, not the easiest read in the world. Let's break it down.&nbsp;First, you need to set up a working directory for osmosis to keep some information about the current state of the data. To do this, create a directory. I chose to put this in a hidden directory in my home dir: ~/.osmosis-minutely/ Next, enter this directory and execute this command (again, I am assuming you have osmosis installed in your home directory):<br />
   <br />
   <pre class="source-code">~/osmosis/bin/osmosis --rrii
  <br /></pre>
   <br />
   This will create two files: download.lock and configuration.txt. You can ignore download.lock. It is used by osmosis to ensure that only one copy is running at a time. The interesting bits are in configuration.txt. First of all, this is where you have to decide if you want to do hourly or minutely updates. Minutely allows you to stay in sync with live changes to the map as people upload them as described above. However&nbsp;this means that your computer will always be spending anywhere from maybe 10 seconds to a full minute every minute importing OSM data. This won't make it unusable for other things but will certainly have an impact on the system. In theory using the hourly files might be slightly more efficient. I haven't played with this to determine if this is true or not but Paul Norman indicated that it didn't seem to be for him. Once you have decided which to use, put either&nbsp;<i>http://planet.openstreetmap.org/minute-replicate</i> or&nbsp;<i>http://planet.openstreetmap.org/hour-replicate</i> in the <i>baseUrl </i>line of the configuration.txt file.<br />
   <br />
   Now you need to determine the time stamp of your planet file. It is in the 2nd line of the planet file. You can get it like this:<br />
   <br />
   <pre class="source-code">bunzip2 -c planet-110309.osm.bz2 | head</pre>
   <br />
   In my case it reads <i>timestamp="2011-03-09T01:11:06Z"</i><br />
   <br />
   Now you can do it the easy way or the hard way. The easy way is to use the&nbsp;<a href="http://toolserver.org/~mazder/replicate-sequences/">replicate-sequences tool</a>. Note: this tool only supports minutely updates. As mentioned in the osmosis documentation, I would suggest taking the time stamp from above and subtracting an hour and using that time just to make sure you don't miss an update. After you enter the time and click the button, take the text it returns and copy/paste it into a file named state.txt in the working directory (~/.osmosis-minutely/)<br />
   <br />
   If you want to know what is going on behind the scenes, read on here. Otherwise feel free to skip to the next section. Every minute (and every hour) files are created on&nbsp;<a href="http://planet.openstreetmap.org/">http://planet.openstreetmap.org</a>&nbsp;that contain the changes that have been uploaded to the database since the last minute/hourly file was generated. These files contain only the new attributes of objects that have been changed. So applying a change twice is fine. It just updates the data to the same values it had before. This is why it is safe to go back an hour from your time stamp. However if you miss an update, all the objects affected by that update will be out of sync with the master database until (if ever) they are updated again.<br />
   <br />
   Along with the change file itself, there is also file containing metadata about the change. For example, <a href="http://planet.osm.org/minute-replicate/000/751/000.state.txt">this one</a>. It has a sequence number that increments for each change file plus a time stamp of when exactly the last change in the associated change file was applied. If you want to find the correct state file without using the tool listed above (or if you want to use hourly updates) you will need to browse through the directories to find an appropriate state file to start off from, again going back in time at least a few minutes or an hour. The time stamps listed next to the files are one hour off from the time stamp inside of the file (the time stamp in the file is in UTC and the server is one time zone away from UTC) so you can use it as a rough guide for locating the correct file but be sure to check the time stamp inside of the file before using it! As above, paste the contents of the appropriate state file into your ~/.osmosis-minutely/ directory as <i>state.txt</i>.<br />
   <br />
   <span class="Apple-style-span" style="font-size: x-large;">Catching up</span><br />
   <br />
   You should have everything in place now! Time to fire up osmosis! Paul reached this point long before I did (because I was playing around with things) so I swiped this command from him:<br />
   <br />
   <pre class="source-code">~/osmosis/bin/osmosis --rri workingDirectory=~/.osmosis-minutely/ 
  <br />                      --sc 
  <br />                      --wpc user="xapi" database="xapi" password="xapi"</pre>
   <br />
   Let's look at the command:<br />
   <ul>
   <li>--rri or "<a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.39#--read-replication-interval_.28--rri.29">read replication interval</a>:" Not much to add to the wiki description. This makes osmosis download a set of change files and tells it where to look for the state.txt file to know which files to download.</li>
   <li>--sc or "<a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.39#--sort-change_.28--sc.29">sort change</a>:" I'm not 100% sure about the function here. I'm assuming it puts the changes in chronological order so that you don't end up applying a newer change before an older change which would lead to an outdated object in your database.</li>
   <li>--wpc or "write pgsql change:" writes the change to the database using the given credentials. (modify to match your database/username/password) This task can be replaced by other "write change" tasks like <a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.39#--write-apidb-change_.28--wdc.29">--wdc</a> or <a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.39#--write-xml-change_.28--wxc.29">--wxc</a>&nbsp;to write changes to other destinations.</li>
   </ul>
   <br />
   By default osmosis will download one hours worth of changes (even if you are using minutely updates - it will just download 60 at a time) and then sit there applying them. This will take anywhere from a few minutes to a full hour. It just depends on how many changes are in the files it is processing and how complex the changes are. Applying changes to huge multipolygons is known to be slow and weekends are always a lot busier than week days.<br />
   <br />
   If you want quicker feedback to make sure it is working, open the ~/.osmosis-minutely/configuration.txt file and change the <i>maxInterval </i>value to something lower. This controls the maximum number of changes that osmosis will download in one go. As mentioned above, the default is one hour (3600 seconds) so you can set it to 60 seconds to just download one minute at a time which should only take a few seconds to apply. Once the process finishes you can check the state.txt file. The time stamp and sequence number should have changed to reflect the update that was applied.<br />
   <br />
   Due to my tinkering around and multiple runs I did with osmosis, I had to apply 488 hours worth of updates to my database. I did it by hand in 10 and 48 hour chunks by changing maxInterval to 36000 and 172800, respectively. After it was all said and done, it took about 163 hours to apply 488 hours of updates. That averages out to about 20 minutes of processing time per hour of updates. But I saw variation, especially in the 10 hour runs. Anywhere from 1.8 to 4.5 hours to apply 10 hours of updates. As before, disk I/O tends to be the bottleneck and I'm using a software RAID5 composed of three 1TB 7,200RPM drives so not exactly tuned for speed.<br />
   <br />
   Now all you need to do is put the osmosis command into a minutely or hourly cron job and you're done! And really you can do this before the database is all caught up. I just did the catching up by hand to monitor and record progress. As I mentioned above, it is safe to call osmosis every minute. Osmosis puts a lock on the download.lock file and if the previous run is still executing, the next one will exit immediately without doing anything. I chose to put the command in a script and then put a call to the script into cron. You might want to redirect stdout and stderr to either a log file or /dev/null to avoid cron sending out emails every time the script runs. My cron job looks like this, minus the newline put in for readability here:<br />
   <br />
   <pre class="source-code">* * * * * /home/toby/bin/osmosis-update.bash &gt;&gt; 
  <br />          /home/toby/.osmosis-minutely/osmosis.log 2&gt;&amp;1</pre>
   <br />
   And that's all, folks! You now have an automatically updating OSM database to query as you please. Have fun!


guid: tag:blogger.com,1999:blog-7044416767097884399.post-4708535036302350732
url: http://ksmapper.blogspot.com/2011/04/parse-planet-into-pgsnapshot-database.html
title: Parse the Planet! (into a pgsnapshot database)
published: 2011-04-03 22:35:00
fetched: 2019-10-25 14:30:26.938078
created_at: 2019-06-09 18:12:15.849091
updated_at: 2019-10-25 14:30:27.204121
summary:
  What makes OpenStreetMap... well... <b>Open</b>? In a link: <a href="http://planet.openstreetmap.org/">http://planet.openstreetmap.org/</a><br /><br />Yes, of course there are the legal issues of having an open license, the transparent community processes to govern the project and a whole host of open source software to create, manipulate, consume and display map data. But at the end of the day, none of that would matter if the raw data wasn't available.<br /><br />As is obvious from my previous posts, I have dabbled in manipulating this raw data in the past, with varying degrees of success. After learning from my previous attempts and hanging around on the mailing lists and IRC channel, picking up information here and there I decided to take another crack at processing a planet file. I ended up finding a partner in crime: <a href="http://www.paulnorman.ca/blog/">Paul Norman</a>. He was wanting to import a planet file for use in a private XAPI server using the new <a href="https://github.com/iandees/xapi-servlet">java based XAPI implementation</a> written by <a href="http://www.openstreetmap.org/user/iandees/diary">Ian Dees</a>. This sounded like a good idea so I decided to follow along. We bounced commands and results off of each other for a while. I ended up going a slightly different route in some of the processing because I had some different hardware available to me.<br /><br />The first thing to know about processing planet data is that there are several different database schemas that can be used to hold the data. This topic probably deserves its own post at some point. Here is a brief list of the ones I am at least minimally familiar with. I know there are some others (like the gazetteer database used by nominatim):<br /><ul><li>API database: Creates a duplicate of the master OSM database. Does not use any GIS features. Nodes are stored with lat/lon values as simple numbers. Contains tables for: ways, tags, users, changesets. Can track history of objects once you start applying updates.</li><li>"pgsnapshot" schema: creates a schema that utilizes the GIS features of postgis that allows you to run spatial queries against OSM data. Still maintains all tag information but they are in a single "hstore" column, not in their own relational tables. Changesets are not stored nor is any information about users except the ID and username. History is not maintained as updates are applied. There is also a related "pgsimple" schema that seems to be an older version of the pgsnapshot schema that may have some advantages. Differences are discussed in the "pgsnapshot_and_pgsimple.txt" file in the osmosis <i>script </i>directory.</li><li>osm2pgsql rendering schema: This schema is created by the osm2pgsql tool and used by mapnik to render map tiles. It uses GIS features as well but it is lossy and very de-normalized. Only objects and tags that you care about rendering are imported. This means it is also smaller and faster to get set up.</li></ul><br />The XAPI uses the pgsnapshot schema so this is what I set up. The instructions on how to get going are provided in the <a href="https://github.com/iandees/xapi-servlet/blob/master/readme.md">xapi readme file</a> but I will go ahead and copy it here with some additional notes.<br /><br /><a name='more'></a><br /><br /><span class="Apple-style-span" style="font-size: large;"><b>Setting up the database</b></span><br /><br />First of all, you will need a postgresql server with some spare drive space. If you are processing the entire planet file, you will end up creating a database that is about 430 GB in size. However during processing, you may end up needing anywhere between 200 and 400 GB of additional space for temporary files. Some of this depends on exactly how you execute the process. And some of the storage doesn't need to be fast. I used an external USB drive for some of the temporary files and it worked great.<br /><br />Next, you might want to take a quick look at the "Configure the PostGIS database" section in <a href="http://weait.com/content/build-your-own-openstreetmap-server">this post by Richard Weait</a>. I am pretty new to postgresql so I have no tuning recommendations of my own. I used the values he suggested and they seem to work.<br /><br />As the postgres user on your database system execute the following commands from a shell: <br /><pre class="source-code">createdb xapi<br />createlang plpgsql xapi<br />createuser xapi<br /></pre>Note: this may not be the best way to do things, but answer "yes" to the superuser question. Otherwise you have to modify postgres ownership which wasn't quite straightforward. Mostly because I'm new to postgres.<br /><br />Now to set up the GIS part of postgis and the hstore for tags. This step sets up the database to hold spatial data instead of just being a bland relational database. Note that your exact path may vary depending on what Linux distro you are using. You can always try a quick <i>locate</i> command. On Ubuntu you have to install the <i>postgresql-8.4-hstore-new</i> package first. On Gentoo I actually had to compile the hstore-new plugin myself.<br /><pre class="source-code">psql -d xapi -f /usr/share/postgresql/8.4/contrib/postgis.sql<br />psql -d xapi -f /usr/share/postgresql/8.4/contrib/spatial_ref_sys.sql<br />psql -d xapi -f /usr/share/postgresql/8.4/contrib/hstore-new.sql<br /></pre><br />Now we set up the actual OSM tables. You will need to download <a href="http://wiki.openstreetmap.org/wiki/Osmosis">osmosis</a>. The pgsnapshot schema was just recently introduced. I had to use a development snapshot but as of April 2, it is included in the release of osmosis 0.39. In this post I will assume that you have osmosis unpacked in your home directory.<br /><pre class="source-code">psql -d xapi -f ~/osmosis/script/pgsnapshot_schema_0.6.sql<br />psql -d xapi -f ~/osmosis/script/pgsnapshot_schema_0.6_linestring.sql<br /></pre><br /><br /><span class="Apple-style-span" style="font-size: large;"><b>Getting the data</b></span><br /><br />Now that the database is ready, let's get some data to work on. For the boring people, just go to <a href="http://planet.openstreetmap.org/">http://planet.openstreetmap.org/</a> and download the file <i>planet-latest.osm.bz2.</i> It will likely take several hours to get the 15 GB file. For the cool kids, check out <a href="http://osm-torrent.torres.voyager.hr/">http://osm-torrent.torres.voyager.hr/</a> where you can find a torrent file with the latest planet dump in it. Planet files are typically produced every Wednesday. The torrent is updated a day or two later. I downloaded several planet files via the torrent and each time I truly did get a <i>torrent </i>of data. During most of the transfer, my downstream bandwidth was maxed out at 2 megabytes/sec.<br /><br />If this is your first time working with OSM data I would strongly recommend testing the waters with a country or state extract. Extract providers are listed on the <a href="http://wiki.openstreetmap.org/wiki/Planet">Planet page</a> on the wiki. I would look at geofabrik and cloudmade. Geofabrik has the data available in a format called PBF which is more compact and quicker to work with than the bzip2 compressed XML file you get from planet.osm.<br /><br /><br /><span class="Apple-style-span" style="font-size: large;"><b>Processing with Osmosis</b></span><br /><br />You might want to go refill your drink. This is going to be long.<br /><br />Osmosis is an amazing tool. Also, amazingly confusing. Just take a look at the <a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage">"Detailed Usage"</a> page on the wiki. The page <b>summary</b> is two full screens tall in my browser. And once you start to actually read things, it just gets worse. It talks about tasks, pipes and streams as casually as most people would discuss the weather. These are all useful things if you're an osmosis expert wanting to look up how to do a specific task like splitting a planet file into 16 polygon defined extracts, all at the same time - while filtering out relations. But it is a little daunting to new users.<br /><br />One thing to keep in mind: argument order <b>matters</b>. For example, If you want to use the --log-progress option to output progress information, it must go between the read and write tasks - you can't simply add it to the end of the command line. There are also multiple ways to do some things. This can be a good thing as it lets you optimize the process based on what kind of hardware you have available to you.<br /><br />Osmosis is a java program. As such, you may need to specify certain options to the JVM to make it do what you want. These JVM level options are controlled by a file. On linux this file is named <i>.osmosis</i> and needs to be in your home directory. On windows 7 it needs to be at <i>C:\Users\&lt;user name&gt;\osmosis.bat</i>. The most common option you may need to use is to give java more memory using the -Xmx option. The other one I needed for one of my attempts was -Djava.io.tmpdir=/some/other/path/ to set the location of the temporary directory to something other than /tmp/<br /><br />So my ~/.osmosis file in linux looks like this to give it 2 GB of memory and redirect the temp directory<br /><pre class="source-code">export JAVACMD_OPTIONS="-Xmx2G -Djava.io.tmpdir=/mnt/database/tmp/"</pre><br />The format is identical in windows except you replace "export" with "set"<br /><br />One more important note: Osmosis claims to be able to read bzip2 compressed files natively and will happily set about doing this for a few minutes. But <b>do not attempt this!</b> It will end in tears and very odd "invalid XML" errors being thrown. The reason is that java's native bzip2 implementation can not read files with multiple streams in them. So it reads the first stream and thinks it is done, truncating any XML elements it might be in the middle of at the time. Instead, use an external utility to unzip the file to standard out and then tell osmosis to read the uncompressed data from standard in. There is a note about this at the bottom of the osmosis page on the wiki but it is easy to miss (I did at first) so it bears repeating.<br /><br />Now that we have some basics down (I told you this was going to be long) let's look at what it takes to actually process the data. The first thing we need to do is read the XML using the <a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--read-xml_.28--rx.29">--read-xml</a> task.<br /><br />Next, there is a decision to make. There are two basic ways to get the data into the database. One is to let osmosis do it all automatically using the <a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--write-pgsql_.28--wp.29">--write-pgsql</a> task. This will cause osmosis to read the XML into some temporary files in your tmp directory and then shove it into the database. The first step may take up to several days (depending on if you are using memory node storage or not - keep reading) and create close to 200 GB of temporary files before it even touches the database. If something fails during the database load, the temporary files are deleted and you start over from scratch. Why the temporary files? Osmosis has to do some processing of the data to correctly construct ways out of nodes and relations out of ways and nodes before it can put them in the database.<br /><br />The other option is to use the <a href="http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#--write-pgsql-dump_.28--wpd.29">--write-pgsql-dump</a> task to process the XML into a series of files that can be loaded into postgres using the \copy command which is much faster than issuing INSERT queries. It also gives you a fall back point if something goes wrong during the database import. The primary reason I chose to use it is that it lets you easily run this process on another computer and then take the resulting files and import them into postgres on a different computer. This is also the only reason I have been mentioning Windows. My file server that has the disk space and is running postgres is an older machine with only 3 GB of memory and a slower CPU. My windows desktop is a much newer machine since I use it for gaming. So it has a quad core CPU and 12 GB of memory. I found a good use for the memory.<br /><br />I will focus on the --write-pgsql-dump task that I used but some of the same things apply to the --write-pgsql task. I will start out with the full command line I used and then explain the parts. As I mentioned above, I ran this on my windows box because it has the memory. So please excuse the ugly file paths :)<br /><pre class="source-code">c:\bzcat.exe o:\planet-110309.osm.bz2 | bin\osmosis.bat <br />  --read-xml file=- --log-progress interval=30 <br />  --write-pgsql-dump directory=g:\osmosis <br />    enableLinestringBuilder=yes <br />    nodeLocationStoreType=InMemory<br /></pre>Now for the parts:<br /><ul><li><b>c:\bzcat.exe o:\planet-110309.osm.bz2 | bin\osmosis.bat</b>: As I mentioned, it is a bad idea to let osmosis unzip the file so we must do this using an external utility and pipe the result into osmosis</li><li><b>--read-xml file=-</b>: This tells osmosis to read an XML document from standard in. It is easy to miss, but don't forget the - at the end!</li><li><b>--log-progress interval=30</b>: This will cause osmosis to report its progress every 30 seconds. The output simply tells you which node/way/relation it is working on and how many per second it is processing so you can make sure it is actually doing something and possibly estimate completion time.</li><li><b>--write-pgsql-dump directory=g:\osmosis</b>: Tells osmosis to format its output as a series of postgres dump files in the given directory. The following three sub-options affect the behavior of this task.</li><li><b>enableLinestringBuilder=yes</b>: This builds a geometric representation of the way which is then stored in a "geometry" type column in postgres that can be used for spatial queries. Building the linestring can be done in the database but you may not live to see it finish if you are processing the entire planet. Seriously.&nbsp;</li><li><b>nodeLocationStoreType=InMemory</b>: This tells osmosis to hold geometric information about every node in memory. By default it writes this to a temporary file on disk. If you have enough memory you really, really, <b>really</b> want to do this as it cuts processing time from about a week down to, in my case, about 8 hours.</li></ul><br />I mentioned "enough" memory. I did this on a computer with 12 GB of memory. The first time I tried this command I gave java 12 GB (-Xmx12G) and java threw an out of memory exception even though my system monitoring (I have <a href="http://www.zabbix.com/">zabbix </a>set up) showed that there was still 1GB free. So the second time I gave java 16GB of memory. Yes, that is more memory than the system has and I would not recommend doing this on a regular basis... but it worked in this case. In fact, zabbix still showed that there was almost 1 GB free except for a brief dip to 200 MB. I'm guessing there is some interplay between the JVM and the operating system going on that complicated things a bit. Also, keep in mind that my 12 GB was able to process the planet file today with about 1 billion, 100 million nodes in it but the map is constantly growing (at about <a href="http://wiki.openstreetmap.org/wiki/Statistics#Database_Statistics_-_Graphical">a million nodes per day</a>!) so hardware requirements will go up over time. If you don't have enough memory, you can still use osmosis but you will need to use a different node location store type and this process will take several days - maybe up to a week - instead of 8 hours. Anyway, here is a picture of memory usage as this process ran:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-DYT4FLeWj5M/TZPeFNeb7CI/AAAAAAAAAVQ/6chrLh30qNQ/s1600/osmosis_total_memory.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="184" src="http://1.bp.blogspot.com/-DYT4FLeWj5M/TZPeFNeb7CI/AAAAAAAAAVQ/6chrLh30qNQ/s640/osmosis_total_memory.png" width="610" /></a></div><br />The initial downhill slide is where osmosis reads all the nodes into memory. After that, memory usage is relatively constant as it processes ways and relations using the nodes it has in memory. The first upwards bump in the graph is when I closed Starcraft 2 and went to bed :)<br /><br />One tip if you are doing this across the network like I was: have your output directory be a local disk or even an external USB drive. My windows system didn't have enough disk space so at first I tried putting the output on a network mounted drive that pointed to my file server which is running a samba share. Bandwidth was not a problem but network latency killed performance. Switching to an external USB drive almost tripled the speed at which osmosis processed objects. The planet file can reside on the network. But the output directory needs to be something local.<br /><br />Here are the resulting files after this 8 hour long memory sucking excursion:<br /><br /><pre class="source-code">nodes.txt            105 GB<br />relation_members.txt 231 MB<br />relations.txt        165 MB<br />users.txt              2 MB<br />way_nodes.txt         26 GB<br />ways.txt              79 GB<br />---------------------------<br />total:               210 GB<br /></pre><br /><br /><span class="Apple-style-span" style="font-size: large;"><b>Loading into postgres </b></span><br /><br />Now for the long part. These 6 files need to be loaded into a database. Luckily there is another script in the osmosis distribution that will assist you. However it will need some modifications. The file is ~/osmosis/<i>script/pgsnapshot_load_0.6.sql</i>. If you open this file you will see that it first drops all constraints and indexes to make the mass inserts go faster. Then it actually imports the data, rebuilds indexes and constraints and finishes up with some database maintenance. The first thing I did was enable query timing just so I could see how long things took. To do this, insert the following line. All of the index/constraint drops will be instantaneous so you can put it after those:<br /><pre class="source-code">\timing</pre><br />Next, comment out the next two lines that start with <i>SELECT DropGeometryColumn</i>. The bbox one isn't needed for the xapi and, assuming you followed my advice above about building the linestring in osmosis, that should already be there. You will also need to comment out the two <i>SELECT AddGeometryColumn</i> statements and the two bigger UPDATE queries below that. These sections are all preceded by comments indicating that they may need to be commented out. To comment them out, just add two dashes at the beginning of the line.<br /><br />I am also unconvinced as to the necessity of the CLUSTER queries. I let the one for the nodes table run and it took almost 3 days. Then I lost power during a spring thunderstorm while the one on the ways table was running. I didn't go back and re run it and my database still seems to be working fine. From what I learned in the <a href="http://www.postgresql.org/docs/8.4/interactive/sql-cluster.html">postgres documentation on the subject</a>, in theory it makes things faster by copying the entire table in an ordered fashion so that nodes that are geographically near each other are also close to each other in the database. This can increase the chances that a single disk read operation will load many of the nodes that you need into memory at once. In practice my use of the database via the XAPI tends to touch a wide area at once so I doubt I'm seeing much benefit from this. You will also see that the postgres documentation states that this needs to be run on a regular basis to maintain the theoretical benefit. As objects are added/deleted/changed, the clustering is slowly lost. And the CLUSTER query puts an exclusive lock on the table so your database will effectively be offline for at least a day any time you do this. So feel free to comment these out as well if you want to save some time. You can always run them by hand later if you determine that they are actually needed.<br /><br />I also did not run the VACUUM command. <a href="http://www.postgresql.org/docs/8.4/interactive/sql-vacuum.html">VACUUM </a>is a database maintenance task that compacts your tables to remove "holes" left by deleted and changed records. In my case, the CLUSTER query completely copied my nodes table, so I don't think that a VACUUM immediately following this would have done anything. Even without the CLUSTER, I'm not sure it would have done much to a brand new database. I may be wrong about that but I have yet to see any adverse effects from not running it. Again, you can always run it by hand later - and may need to do so periodically anyway.<br /><br />However it is very important to run the ANALYZE query! This will cause postgres to do a quick (10 minutes or so) analysis of all the tables in the database and will store some basic information that postgres can use later to optimize your joins. I initially didn't do this either and some simple queries I tried looked like they were going to take days to finish. After running the ANALYZE query, they finished in seconds. It is <b>THAT </b>important!<br /><br />But back to the import process itself. Now that you have tweaked the <i>pgsnapshot_load_0.6.sql</i> file, it is time to wield it. Change to the directory where the .txt files from osmosis are. Then execute:<br /><pre class="source-code">psql -U xapi -d xapi -f ~/osmosis/script/pgsnapshot_load_0.6.sql<br /></pre><br />At this point you should go to sleep or spend time with your family or something. Here are the timings from my load:<br /><pre class="source-code">users.txt:            20 seconds<br />nodes.txt:            13 hours<br />ways.txt:             20 hours<br />way_nodes.txt:         2 hours<br />relations.txt:        40 seconds<br />relation_members.txt: 80 seconds<br /><br />Creating indexes:<br />node primary key:              1.5 hours<br />way primary key:               30 minutes<br />way_nodes primary key:         1.5 hours<br />relations primary key:         40 seconds<br />relation_members primary key:  50 seconds<br />nodes geometry index:          25 hours<br />way_nodes node ID index:       2 hours<br />realtion_members member index: 45 seconds<br />ways bounding box index:       1.7 hours<br />way linestring index:          1.6 hours<br /></pre><br />This ads up to about 70 hours. And that's without the CLUSTER query. If something goes wrong, you can truncate the tables (only the 6 affected by the load... leave the other GIS tables alone!) and then just rerun the script. It will spit out errors when it tries to drop the indexes at the beginning because they don't exist but that is fine.<br /><br />You can monitor the progress of the \copy commands in linux by using  <br /><pre class="source-code">lsof -o -p &lt;pid&gt; </pre>where &lt;pid&gt; is the process ID of the pgsql process. It will give you a readout the position of the current read within the file in bytes - in hex. Convert to decimal and divide by 1073741824 to get it in gigabytes.<br /><br />If you are tight on space, it is safe to delete or compress the input files after pgsql has finished the \copy command on that file.<br /><br />Here is a pretty picture of my disk usage during the import process:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-fQ-ZbxcoLog/TZju3ENW5MI/AAAAAAAAAVU/7CsDA3JQscU/s1600/pgsnapshot_disk.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="216" src="http://4.bp.blogspot.com/-fQ-ZbxcoLog/TZju3ENW5MI/AAAAAAAAAVU/7CsDA3JQscU/s640/pgsnapshot_disk.png" width="610" /></a></div><br />As you can see, this covers a 6 day period. 3 days (from some time on the 16th to pretty much the end) is the CLUSTER query that I'm not sure is needed. The occasional drops in space usage towards the end is me deleting old files from previous osmosis test runs to make sure it didn't fill up the partition. The big drop at the end is where it finally finished the CLUSTER query and was able to drop the temporary nodes table. Total partition size is 670 GB.<br /><br /><br /><span class="Apple-style-span" style="font-size: large;"><b>Conclusion</b></span><br /><br />And that's it! Once that process finishes, you will have a complete pgsnapshot database containing the OpenStreetMap database! As I mentioned up towards the top, one use for this is to run your own XAPI server to do specific map queries however you could also use this database to do other things with the OSM data. Depending on what you are doing, you may need to set osmosis up to keep your database in sync with new edits. Obviously by the time this process finishes, the data will already be several days out of date. I will cover the syncing process in a separate post soon.<br /><br />The total processing time for me was about 80 hours. To make that at least somewhat meaningful I should list the specs of the computer doing all the importing. I already mentioned that the computer I ran osmosis on was a quad core with 12 GB of memory. The computer that is running the database and that actually ran the import process is a little older. It has an AMD 4400+ (first generation dual core) with 3 GB of memory. But the most important part for the database processes is disk. Mine aren't really spectacular: 3 1TB Samsung Spinpoint F1 drives (7,200 RPM) in a software RAID-5 (so total of 2TB usable storage). I am also running LVM on top of the RAID which splits the space into two partitions. Pretty much all of the database related processes are slowed down by disk access times. Faster disks or a better RAID setup (0 or 10) would drop these times. Or SSDs. Donations welcome :)<br /><br />As I said above, the total size of the database after it is all said and done is 430 GB. However if you execute the CLUSTER query on the nodes table, you will need an additional 180 GB or so because it makes a complete copy of the nodes table. Plus remember the temporary files created by osmosis need another 210 GB although they can be deleted/moved/compressed as the load script finishes using them.<br /><br /><strike>Check back soon (or subscribe!) to catch</strike> <a href="http://ksmapper.blogspot.com/2011/04/keeping-database-up-to-date-with.html">Check out my next post</a> about keeping the database up to date.


