# items

guid: 1099 at http://www.dankarran.com
url: http://www.dankarran.com/blog/2013/04/google-map-maker-openstreetmap-uk
title: Google Map Maker vs OpenStreetMap in the UK
published: 2013-04-14 16:20:00
fetched: 2019-07-02 14:30:07.801716
created_at: 2019-06-09 18:11:40.241054
updated_at: 2019-07-02 14:30:07.868563
summary:
  <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>Earlier this week Google <a href="http://google-latlong.blogspot.co.uk/2013/04/welcoming-united-kingdom-to-google-map.html">announced</a> that their <a href="http://www.google.com/mapmaker">Map Maker</a> product was being launching in the UK. Map Maker lets users make changes to Google's popular Maps service, helping Google improve their map data with the assistance of the crowd. The product has been around since 2008, originally available only in countries that didn't really have any map data, and later expanding into countries that already had good coverage but could be improved with additional information.</p>
  <p>Map Maker works in a similar way to the <a href="http://www.openstreetmap.org">OpenStreetMap project</a> (which started around 2006) in that they both let people edit the details on their respective map, but there are some noticeable differences between Map Maker and OpenStreetMap. </p>
  <p>(I've been using and contributing to OpenStreetMap since the early days of the project, so I'm going to be biased towards it, but I think both Map Maker and OSM have things going for them.)</p>
  <p><strong>User base</strong><br />
  As probably the most widely used map in the world, Google Maps has a large number of people looking at their maps for local information. Most of them have local knowledge about an area, and a very small percentage of them may want to help improve their local maps for their own benefit and for others. On the other hand, OpenStreetMap's user base is much smaller (though has been growing quickly for years), and for the most part consists only of those people who want to help build the maps. Both Google Maps and OpenStreetMap can be embedded into other websites, but the end users of those sites are unlikely to care about making updates to the map, it's only people who use Google Maps or OpenStreetMap directly who'd do that.</p>
  <p>Editing a place in Google Maps means that your information is currently going to be seen by many more casual users than if you were to edit OpenStreetMap, but with the inclusion of OpenStreetMap data in many other projects (like Apple Maps, MapQuest, MapBox) there are also massive advantages to making sure the information is up to date in there too. If you're promoting your business, you should definitely make sure to add it to OpenStreetMap as well.</p>
  <p><strong>Moderation</strong><br />
  With any service where you're giving lots of unknown people a chance to share information that is visible to the public, you're likely to get people vandalising that information, whether it be accidentally, for a bit of fun, or for any other reason. Google Map Maker has a moderation feature which stops all edits being added to the map automatically, and only lets them through after they've been vetted by trusted users. OpenStreetMap doesn't have up-front moderation, but has tools in place to let users keep an eye on areas they know. Both approaches have their merits and disadvantages.</p>
  <p><strong>Editing interface and user experience</strong><br /><img src="http://www.dankarran.com/sites/dankarran.com/files/google-mapmaker-startup.png" style="float: right; padding: 10px; width: 200px;" alt="Google Map Maker welcome screen" />The interface to Google Map Maker is designed to be easy to access when you're looking at a map (look out for the 'edit in Map Maker' link in the bottom right corner of the map) and makes it as simple as possible for people to add new places, edit existing ones (accurate business information is what Google is hoping for here), add roads, and review other peoples' edits. The interface is perhaps a bit more intuitive than OpenStreetMap might be for a new user at the moment, but some of that is because it has to cater for less technical users. OpenStreetMap is making efforts to improve this initial user experience too though, with improvements to the website and in particular the development of the new <a href="http://ideditor.com/">iD editor</a> which aims to make editing simpler.</p>
  <p><strong>Using the maps and map data</strong><br />
  Both Google Maps and OpenStreetMap can be embedded into your own website to show the location of your business, or to overlay whatever other information you'd like to display. With Google Maps you can change some of the styling, and choose to hide some of the information (with some technical know how), but you never have full control over what you're showing. If you want to show where your business is without also promoting your competitors, you're out of luck. With OpenStreetMap however, you can have complete control over what you show and what you don't, especially if you use a service like <a href="http://mapbox.com">MapBox</a> to customise your maps, or download any one of a number of opensource tools to build your maps exactly as you like.</p>
  <p>Any information that you add into Google is owned by Google, and shown on a service that could in theory be withdrawn at any time (see Google Reader). Any information you add into OpenStreetMap is licensed freely for people to reuse as they like, so if you're adding information about your business into OSM, it's got the potential to be promoted much more widely than if it's just in Google Maps. The information in OSM will always be available to use as it's not dependent on any one commercial entity.</p>
  <p><strong>So, which should I use?</strong><br />
  Personally, I'm happy to add some information into Google Maps, such as if I'm helping to promote a local business, but I'll also make sure that information is mirrored (and bettered) in OpenStreetMap. If I'm going to add in any sizable chunk of information, I'll just add that into OpenStreetMap as adding it into Google would be doing work for them for free without being able to reuse that information myself. </p>
  <p>As an example, the maps below show Foxdale in the Isle of Man, in both Google Maps and OpenStreetMap. I help out with the online bits for some <a href="http://www.iomfarmholidays.com">self-catering farm cottages</a> in the village, so I've added <a href="http://www.iomfarmholidays.com/location">their location</a> and a bit of detail around the farm into Google Maps, but if you look in OpenStreetMap you'll see a lot more detail for the whole village as well.</p>
  <div class="photo">
  <a href="https://maps.google.com/?ll=54.171064,-4.633892&amp;spn=0.008767,0.020878&amp;t=m&amp;z=16"><img src="http://www.dankarran.com/sites/dankarran.com/files/foxdale-google2.png" alt="Map of Foxdale, Isle of Man on Google Maps" /></a>
  <div class="caption"><a href="https://maps.google.com/?ll=54.171064,-4.633892&amp;spn=0.008767,0.020878&amp;t=m&amp;z=16">Map of Foxdale, Isle of Man on Google Maps</a> (© Google)</div>
  </div>
  <p>Foxdale looks a bit bare in Google Maps, but check it out in OpenStreetMap:</p>
  <div class="photo">
  <a href="http://www.openstreetmap.org/?lat=54.16841&amp;lon=-4.63855&amp;zoom=16&amp;layers=M"><img src="http://www.dankarran.com/sites/dankarran.com/files/foxdale-openstreetmap.png" alt="Map of Foxdale, Isle of Man on OpenStreetMap" /></a>
  <div class="caption"><a href="http://www.openstreetmap.org/?lat=54.16841&amp;lon=-4.63855&amp;zoom=16&amp;layers=M">Map of Foxdale, Isle of Man on OpenStreetMap</a> (© OpenStreetMap contributors, <a href="http://www.openstreetmap.org/copyright">CC-BY-SA</a>)</div>
  </div>
  <p>If this post has inspired you to add your business information into OpenStreetMap, check out this <a href="http://www.northrivergeographic.com/for-beginners-add-point-openstreetmap">quick guide to adding your first point</a>.</p>
  </div></div></div>


guid: 1096 at http://www.dankarran.com
url: http://www.dankarran.com/blog/2013/02/presenting-bomb-sight-geomob
title: Presenting Bomb Sight at geomob
published: 2013-02-12 23:30:00
fetched: 2019-07-02 14:30:07.801716
created_at: 2019-06-09 18:11:40.243766
updated_at: 2019-07-02 14:30:07.870990
summary:
  <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>Last week I presented a talk on the <a href="http://www.bombsight.org">Bomb Sight project</a> at the first <a href="http://geomobldn.org/first-geomob-of-2013-7-feb-at-the-bcs">geomob</a> meetup of the year. I posted the <a href="http://www.slideshare.net/GeobitsLtd/bomb-sight-presentation-for-geomob-london">slides on SlideShare</a> afterwards, but I'll add some more narrative here in case you missed the presentation and want to find out more about the project. </p>
  <iframe src="http://www.slideshare.net/slideshow/embed_code/16417151" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC;border-width:1px 1px 0;margin:5px;float:right;width:427px;height:356px;max-width:427px;" allowfullscreen="" webkitallowfullscreen="" mozallowfullscreen=""> </iframe><p>
  Bomb Sight was a year long academic project to map the London Blitz Bomb Census, funded by <a href="http://www.jisc.ac.uk">JISC</a> with the support of the <a href="http://www.nationalarchives.gov.uk">National Archives</a> and the <a href="http://www.port.ac.uk">University of Portsmouth</a>. The <a href="http://www.bombsight.org/about/team/">project team</a> was led by <a href="http://www.port.ac.uk/departments/academic/geography/staff/title,109932,en.html">Dr Catherine (Kate) Jones</a> of the University of Portsmouth, detailed knowledge of the archive information provided by <a href="https://twitter.com/cartivist">Andrew Janes</a> from the National Archives, the web app built by <a href="http://www.linkedin.com/in/pweber">Patrick Weber</a> of <a href="http://www.locationinsights.co.uk">Location Insight Ltd</a>, the mobile app by me (<a href="http://www.geobits.co.uk">Geobits Ltd</a>) and the design of both by <a href="http://jasia.net">Jasia Warren</a>.</p>
  <p><a href="http://en.wikipedia.org/wiki/The_Blitz">The Blitz</a> was a period of London's history that saw prolonged bombing with high explosive bombs during 1940 and 1941. A million houses were damaged or destroyed in London alone, and many other UK cities suffered a similar fate. More bombing continued later in World War II with the arrival of self-propelled V1 and V2 bombs.</p>
  <p>The Bomb Sight project used maps from the <a href="http://www.nationalarchives.gov.uk/records/research-guides/maps-bomb-census-survey.htm">Blitz Bomb Census Survey</a> which were compiled by the Ministry of Home Security to give an overall picture of where bombs fell across the UK. There are a lot of maps held by the National Archives that relate to WWII bombing of London and other cities, and because we only had a limited amount of time, the project is only covering the area around London (referred to then as Civil Defence Region 5) and for the period of 7 October 1940 - 6 June 1941. The Blitz started in London a month before this, on 7 September, but mapping of bomb locations by the Ministry of Home Security only began on 7 October. We have also included data from the <a href="http://www.guardian.co.uk/news/datablog/2010/sep/06/london-blitz-bomb-map-september-7-1940">first night of the Blitz</a>, which comes from London Fire Brigade records. The maps were previously only available to access in their original form in the Reading Room at the National Archives, but the Bomb Sight project has now made them available to citizen researchers, students and academics who want to explore the maps.</p>
  <p>The main features of the project have been creating digital maps of the Bomb Census, digitising bomb location data from these, analysing the information within various geographical boundaries, combining the information with geolocated stories and photos from the time, creating a web mapping application to explore the information, and creating a mobile application to help people explore bomb locations while they are out and about in London. </p>
  <p>The maps for the project came from two collections held at the National Archives. The first set of maps (<a href="http://discovery.nationalarchives.gov.uk/SearchUI/details?Uri=C4483204">HO193/13</a>) shows bombs that fell over night during the eight month period from 7 October 1940 - 6 June 1941 without any additional information to show when each bomb fell during that period. We used 35 of these map sheets to cover the London area. The second set of maps (<a href="http://discovery.nationalarchives.gov.uk/SearchUI/details?Uri=C4483211">HO193/01</a>) details the bombs that fell on a day by day basis, using colour to depict the day of the week, and different symbols to show the type of bomb that fell. Due to the large number of these map sheets (over 500) we just selected nine of them that show bombing for the week between 7 - 14 October 1940. Each of these maps was then photographed, loaded into a GIS package and georectified to match its geography up with a modern map.</p>
  <p>Manually digitising all of the bomb locations from the aggregate maps provided the landing sites of 31,373 bombs, 28,325 of which fell within the current day boundaries of Greater London. </p>
  <p>Visualising all of these points on a <a href="http://www.bombsight.org/#9/51.5369/0.0645">zoomed out map</a> shows a mass of red dots covering the London area. We investigated some alternative ways of visualising the data at this scale (such as a heat map), but we decided this method helped to convey the impact of The Blitz on London as a whole. The image was one that was widely used by the media when reporting the project.</p>
  <p>The website brings together the various datasets (first day of the Blitz, first week of mapping, 8 months of mapping) overlaid on modern OpenStreetMap data or on the original map images (see the layers icon in the top right of the map and click '1940s Bomb Maps' to enable them). You can explore the map by panning around, search for a location you're interested in, or browse information within various geographical boundaries using the '<a href="http://www.bombsight.org/explore/">Explore London</a>' menu. When you are <a href="http://www.bombsight.org/#16/51.5052/-0.0770">zoomed in to the map</a> you can click on any of the markers to view more information about the bomb as well as some contextual information.</p>
  <p>Each digitised bomb has its own information page (e.g. <a href="http://www.bombsight.org/bombs/29402/">this one near Bank Station</a>) showing the type of bomb, a map of its location and others around it, a current day address for the location, stories from locals mentioning nearby places, and a selection of nearby photos from the time. The photos came from the <a href="http://www.iwm.org.uk/collections/search?query=london%20blitz&amp;items_per_page=10&amp;filter[contentDate][0]=%22Second%20World%20War%22&amp;f[0]=mediaType%3Aimage">Imperial War Museum archives</a>, where they are provided in an easily searchable format and with a <a href="http://www.iwm.org.uk/corporate/privacy-copyright/licence">licence</a> to use them free for non-commercial purposes.</p>
  <p>The example bomb information page shown in the presentation was for a <a href="http://www.bombsight.org/bombs/29402/">bomb that landed near Bank station</a>. I think this bomb is probably the one that <em>The Register</em> claimed was missing from the map in their article <a href="http://www.theregister.co.uk/2012/12/12/blitz_map/">London Blitz bomb web map a hit-and-miss affair</a>, but it's difficult to be certain. </p>
  <p>There are a number of considerations around accuracy that should be taken into account when dealing with these maps. The original maps were an aggregation of data that was coming in from various sources, presumably of varying levels of accuracy and precision due to the sheer amount of damage that each attack would cause. The base maps would potentially have had accuracy issues as well, especially around the edges, where one sheet meets another, and over the years the paper may have warped slightly. There is then the digitisation process, which can introduce errors and inaccuracies during the georectification of the maps and also while digitisation of each point, which is converted from the original pen mark to a pair of coordinates near the centre of the mark.</p>
  <p>As part of the digitisation process, the location of each bomb was <a href="http://en.wikipedia.org/wiki/Reverse_geocoding">reverse-geocoded</a> to determine the nearest street address, to help give some context to the data, and to assist with searching. Looking up a historic location using a modern dataset can give some incorrect results at times, whether its because the bomb didn't actually fall where we think it did (see the note about accuracy above) or the road name has changed name over time, including as old buildings have been knocked down to make way for new developments (e.g. <a href="http://www.bombsight.org/bombs/36197/">bomb at More London Place</a>).</p>
  <p>The technology behind the website is all open source, with <a href="http://www.postgresql.org">PostgreSQL</a>/<a href="http://www.postgis.org">PostGIS</a> as the database, <a href="http://www.geoserver.org">GeoServer</a> to provide <a href="http://en.wikipedia.org/wiki/Web_Feature_Service" title="Web Feature Service">WFS</a> and tiled <a href="http://en.wikipedia.org/wiki/Web_Map_Service" title="Web Map Service">WMS</a> interfaces to the data, <a href="http://leafletjs.com">Leaflet</a> and <a href="http://openstreetmap.org">OpenStreetMap</a> for the mapping, <a href="http://www.djangoproject.com/">Django</a>/<a href="http://geodjango.org">GeoDjango</a> to build the dynamic website and <a href="http://twitter.github.com/bootstrap/">Bootstrap</a> to provide a responsive user interface framework suitable for desktop, tablet and mobile devices. The website sits behind the <a href="http://www.cloudflare.com">CloudFlare</a> content delivery network (CDN), which caches all of the pages and map data from the site and serves them from their own servers located closest to the end user, meaning that the load on our server is much reduced.</p>
  <p><a href="https://play.google.com/store/apps/details?id=org.bombsight.app"><img src="http://geobits.co.uk/bombsight/google_launch.png" style="float:right; margin:10px; width:200px; height:200px;" alt="Bomb Sight Android app" /></a>As well as the mobile-friendly website, we built a mobile application (currently <a href="https://play.google.com/store/apps/details?id=org.bombsight.app">available for Android</a> and hopefully for iOS in the near future) which replicated some of the mapping features and also added an <a href="http://en.wikipedia.org/wiki/Augmented_reality">Augmented Reality</a> interface to help explore the data in the context of your own surroundings. The application was built using the <a href="http://developer.android.com/sdk/index.html">Android SDK</a>, <a href="http://phonegap.com/">Phonegap</a> to provide a framework for cross-platform development, Leaflet and OpenStreetMap for mapping, and the <a href="http://www.wikitude.com/developer">WIkitude SDK</a> for Augmented Reality functionality that allows you to see bomb locations overlaid on what your camera is seeing in front of you. If you want to find out more, you may also be interested in my post on <a href="http://blitzbombcensusmaps.wordpress.com/2012/09/24/choosing-an-augmented-reality-library-for-android/">choosing an AR library for Android</a>, a <a href="http://blitzbombcensusmaps.wordpress.com/2012/10/05/technical-overview-of-the-android-mobile-application/">technical overview</a>, or a <a href="http://blitzbombcensusmaps.wordpress.com/2012/11/06/building-the-android-mobile-prototype-app-for-bomb-sight/">walkthrough of the prototype</a>. There are some <a href="http://blitzbombcensusmaps.wordpress.com/2012/12/20/announcing-the-release-of-the-android-app/">screenshots posted here</a> too.</p>
  <p>The outcomes of the Bomb Sight project were the georeferenced bomb maps, providing a digital record of an important event in the history of London and the country, the sharing of all of the data with the National Archives to help reduce use of the original paper maps and aid with their preservation, the opening up of maps that were previously only accessible if you were able to visit the National Archives, making the information available to a wider audience and providing a geographic framework for the study of the impact of bombing on the social and economic climate.</p>
  <p>The media response to the project was phenomenal, and the initial response completely overwhelming, as we struggled to keep the site up and running under the immense traffic (see my post on <a href="http://blitzbombcensusmaps.wordpress.com/2012/12/11/on-scaling-from-40-visitors-a-day-to-6-every-second/">scaling from 40 visitors a day to 6 every second</a>).</p>
  <p>The project is officially at an end, but there are a few things that are still to come as we find the time to work on them. In the next couple of months, we will be providing data downloads for non-commercial purposes, allowing researchers to investigate the data in more detail, and after that we will be adding some tutorials to the site to help people find their way around. I'm also planning to port the Android app to iPhone when I get a chance. </p>
  <p>All of us on the project team would quite like to work more on the project as and when possible, on these planned updates, and also on expanding the project further if we can... </p>
  <p>The data we are using is only a small subset of the data available in the National Archives. There is potential - if we can find both time and funding - to expand the temporal coverage of bomb data to cover the entire period of the Blitz and the rest of the war, including mapping the V1 and V2 bombs that were launched into London later in the war. We could add more detailed reports (known as <a href="http://www.nationalarchives.gov.uk/catalogue/displaycataloguedetails.asp?CATLN=3&amp;CATID=7753&amp;SearchInit=4&amp;SearchType=6&amp;CATREF=HO+198&amp;j=1">BC4 reports</a>) to the site, which were written reports that described many of the bomb sites. We could also include more contextual information around each bomb location, such as more photos, user-submitted stories and comments. And of course, London wasn't the only city that was bombed during the war - there are many other British cities that were bombed, and many European cities that were hit even worse by Allied bombing.</p>
  <p>If you would like to find out more about the <a href="http://www.bombsight.org">Bomb Sight project</a>, have a read of the <a href="http://blitzbombcensusmaps.wordpress.com">project blog</a>, or tweet us on the project at <a href="http://twitter.com/BombSightUK">@BombSightUK</a>.</p>
  </div></div></div>


guid: 1093 at http://www.dankarran.com
url: http://www.dankarran.com/blog/2012/11/bomb-sight-explore-london-during-the-blitz
title: Bomb Sight: explore London during the Blitz
published: 2012-11-29 15:30:03
fetched: 2019-07-02 14:30:07.801716
created_at: 2019-06-09 18:11:40.246785
updated_at: 2019-07-02 14:30:07.873486
summary:
  <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>For the past few months I've been part of a team working on a project that's helping make available information about where bombs fell in London during the period of World War II known as The Blitz.</p>
  <p><img src="http://www.dankarran.com/sites/dankarran.com/files/until-1972.jpg" alt="Secret until 1972" style="float: right; padding: 10px; width: 160px; height: auto;" />Records and maps from the <a href="http://www.nationalarchives.gov.uk/records/research-guides/maps-bomb-census-survey.htm">WWII Bomb Census survey 1940-1945</a> have been available at the <a href="http://www.nationalarchives.gov.uk">National Archives</a> in Kew since they were declassified in 1972. The public has been able to access the original records there if they knew what they were looking for, and can get to Kew to visit the archives (something that's well worth doing, if you get a chance) but the maps hadn't been available online... until today.</p>
  <p><a href="http://www.bombsight.org"><img src="http://www.dankarran.com/sites/dankarran.com/files/96ae7407e0c07c7da2e33cdc5170d23f.png" style="float: left; padding: 10px; width: 60px; height: 60px;" alt="Bomb Sight" /></a>The <a href="http://www.bombsight.org">Bomb Sight</a> website launched today to make this information more accessible to the public. Covering the period between 7th October 1940 to 6th June 1941, you can find out where bombs fell during the Blitz and also access photos and stories that relate to that area during the war.</p>
  <p>The project was funded as part of the <a href="http://www.jisc.ac.uk/whatwedo/programmes/digitisation/content2011_2013.aspx">JISC Content Programme</a> to digitise a subset of the maps and make them available online - in both raster and vector forms - and provide a mobile augmented reality interface to explore where bombs fell in the area around you.</p>
  <p>The main <a href="http://www.bombsight.org">bomb damage map</a> is the place to start exploring. Zoom out to get an idea of how badly hit each and every part of London was, or zoom in to an area (e.g. <a href="http://www.bombsight.org/#16/51.5059/-0.0801">bombs falling near Tower Bridge</a>) to see more detail.</p>
  <p><a href="http://www.bombsight.org"><img src="http://www.dankarran.com/sites/dankarran.com/files/bomb-sight-screenshot.png" style="float: left; padding: 10px;" /></a>You may want to use the layers menu to switch from the modern map to the original maps used to record bomb locations. </p>
  <p>You can click through to see a little more information about each bomb (e.g. these two <a href="http://www.bombsight.org/bombs/8411/">east</a> and <a href="http://www.bombsight.org/bombs/8193/">west</a> of Tower Bridge) and see related information about the area. </p>
  <p>You can also <a href="http://www.bombsight.org/explore/">explore statistics</a> for all areas of London down to a borough (e.g. <a href="http://www.bombsight.org/explore/greater-london/southwark/">Southwark</a>) and ward level (e.g. <a href="http://www.bombsight.org/explore/greater-london/southwark/riverside/">Riverside, Southwark</a>).</p>
  <p>Keep an eye out for the Android app I'm working on for the project, which will be coming soon. In the mean time, you can already explore an area using your phone, simply by visiting <a href="http://www.bombsight.org">bombsight.org</a> in your browser.</p>
  <p>If you're interested in the work we've been doing with the project, follow us on Twitter <a href="http://twitter.com/BombSightUK">@BombSightUK</a>, and please help spread the word to others who may be interested in exploring London's past. You can also read some more information about the work over the past year in the <a href="http://blitzbombcensusmaps.wordpress.com">project blog</a>.</p>
  </div></div></div>


guid: 1087 at http://www.dankarran.com
url: http://www.dankarran.com/blog/2011/06/qr-code-treasure-trails
title: QR code treasure trails
published: 2011-06-18 17:00:00
fetched: 2019-07-02 14:30:07.801716
created_at: 2019-06-09 18:11:40.249397
updated_at: 2019-07-02 14:30:07.875838
summary:
  <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><img src="http://www.dankarran.com/sites/dankarran.com/files/qr-code-trail-sticker.jpg" alt="QR code trail sticker in-situ" style="float: right; padding: 10px;" />I've been working with QR codes since the tail end of last year and have seen an <a href="http://www.google.com/trends?q=qr+code&amp;ctab=0&amp;geo=gb&amp;date=all&amp;sort=0">impressive growth in interest</a> from marketers and consumers, with QR codes appearing in all sorts of different places, from the Metro newspaper to estate agents' windows. Doing social media work with the <a href="http://tag.st">Tag Street</a> twitter account <a href="http://twitter.com/tag_st">@tag_st</a>, I get to keep an eye on how these codes are being adopted around the world.</p>
  <p>Whenever you're working for any length of time with a technology that's new to you, it's quite common to try and think up other uses for it, testing things out in new ways to see if they'll work.</p>
  <p>So, a month or so back, I <a href="http://twitter.com/#!/dankarran/status/70443317439102977">had the idea</a> of creating a QR code trail using little stickers that could lead people from one point to the next. I asked on Twitter to see if any of my followers would scan a random QR code they found on the street. Some <a href="https://twitter.com/#!/The_Maz/status/70448753340989440">seemed curious</a>, others <a href="https://twitter.com/#!/chillly/status/70474524084281344">unsure</a>, or even <a href="http://twitter.com/#!/almien/status/70466483624480768">concerned</a>, and others <a href="https://twitter.com/#!/T_RRed/status/70445458471927808">definitely wouldn't</a>. A good cross-section, I thought, and enough to convince me to get some stickers printed up (<a href="http://uk.moo.com/products/stickers.html">Moo stickers</a> were perfect for this) and give it a shot.</p>
  <p>On 1st June I <a href="http://twitter.com/#!/dankarran/status/75855091009204224">tweeted</a> the start point of the trail, on the northern part of Tower Bridge and it was soon picked up and <a href="http://twitter.com/#!/se1/status/75859136511881216">retweeted</a> by the <a href="http://www.london-se1.co.uk">London SE1</a> hyperlocal site.</p>
  <p>The same day, the first tag got scanned by a couple of people, but nobody made it to the second tag, or any further than that. Nobody has scanned any of the tags since, despite all (but one, the final tag) still being in place*.</p>
  <p><img src="http://www.dankarran.com/sites/dankarran.com/files/qr-code-destination.jpg" alt="Chairs in Potters Fields park" style="float: right; padding: 10px;" />So, what does this leave me thinking about QR codes? Actually, not much differently than I did already... For QR codes to work, and prove useful to to the people they're there to help, they must:</p>
  <ul><li>be clear about the content they'll deliver, and their value</li>
  <li>provide alternative ways to access their content - a short URL</li>
  <li>be easy to see, and invite people passing by who could gain value</li>
  </ul><p>These tags had their own unique website address printed on them (at <a href="http://qrtrail.co.uk">QRtrail.co.uk</a>/<em>something</em>) but that was about it. They didn't provide a clear way that a person could benefit from scanning the code. In fact, the last tag - <em>the treasure</em> if you will - was simply a nice relaxing place to sit, with a nice view of City Hall, Tower Bridge, and Potters Fields park.</p>
  <p>If I'm going to try this again, it'll need some rethinking, probably some better publicity, and a prize at the end to encourage people to try it out. Meanwhile, if you'd like to try out a sheet of 6 stickers wherever you are, let me know and I'll sort you out with a set of tags that you can edit at <em>qrtrail.co.uk</em>.</p>
  <p>If you'd like to try the SE1 trail, visit <a href="http://qrtrail.co.uk/trail/se1">QRtrail.co.uk/trail/se1</a> and let me know how you get on.</p>
  <p><em>* I'll take the stickers down before too long so they don't stay as litter in the city</em></p>
  </div></div></div>


guid: 1083 at http://www.dankarran.com
url: http://www.dankarran.com/blog/2011/03/review-openstreetmap-book
title: Book Review: OpenStreetMap: Using and Enhancing the Free Map of the World
published: 2011-03-21 20:07:14
fetched: 2019-07-02 14:30:07.801716
created_at: 2019-06-09 18:11:40.251842
updated_at: 2019-07-02 14:30:07.878179
summary:
  <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><em>A review of <a href="http://www.amazon.co.uk/OpenStreetMap-Using-Enhancing-Free-World/dp/1906860114/ref=sr_1_1?ie=UTF8&amp;qid=1300738070&amp;sr=8-1">OpenStreetMap: Using and Enhancing the Free Map of the World</a> by Ramm, Topf and Chilton</em></p>
  <p>This book is a great introduction to, and overview of, the OpenStreetMap project, with plenty of detail and pointers to further information, all of which is sure to get the reader hooked in the realm of open geodata.</p>
  <p>As an open source community-oriented project, there is a lot of information already available on the <a href="http://wiki.openstreetmap.org">OpenStreetMap wiki</a> website, but it can be disjointed at times, and difficult to get a good overview of the project. This book provides a good alternative, condensing all the information you need to start updating the map, using the map data on GPS devices, and also gives plenty of examples to developers keen on using the wealth of map data in their own projects.</p>
  <p>The chapters of the book are broken up into small enough chunks that readers can easily dip in and out, while also providing plenty of information to refer back to during mapping.</p>
  <p>The first half of the book is geared towards those readers who want to get started mapping their surroundings and contributing geographic information to the OpenStreetMap project. The second half goes into more detail about the technical aspects of OpenStreetMap, introducing the data API and other ways of accessing and modifying the data for use in third party projects.</p>
  <p>Throughout the book, readers are given links to the OpenStreetMap wiki for further information about topics, should they be interested in finding out more.</p>
  <p>This book would be of interest to a range of different audiences, from walkers and geocaching enthusiasts keen to explore and document their surroundings, to GIS users wanting to know more about emerging sources of geodata, and web developers who are looking for maps or a new source of information to use in their projects... or anyone else who has an interest in geography and how emerging technologies are helping develop the subject.</p>
  </div></div></div>


guid: 1080 at http://www.dankarran.com
url: http://www.dankarran.com/blog/2010/11/logos-and-locations-in-qr-codes
title: Logos and locations in QR codes
published: 2010-11-14 21:33:13
fetched: 2019-07-02 14:30:07.801716
created_at: 2019-06-09 18:11:40.254281
updated_at: 2019-07-02 14:30:07.880423
summary:
  <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><img src="http://www.dankarran.com/sites/dankarran.com/files/qr-code-douglas-info-themap.png" alt="Custom design QR code for themap.im" width="180" height="210" style="float:right;padding: 10px" />I have spent a bit of time this afternoon looking into <a href="http://en.wikipedia.org/wiki/QR_Code">QR codes</a>, and how they can be customised to incorporate logos or other information. I think the first I saw of this trend was a <a href="http://2d-code.co.uk/bbc-logo-in-qr-code/">BBC logo</a> embedded into a QR code, but numerous people have tried it out, such as <a href="http://d-qr.net/">these ones from Japan</a>, where the QR code has already been used much more widely than in the UK.</p>
  <p>If you haven't come across QR codes before, they are much like barcodes, but have the ability to store much more information in them. Perhaps the most common use is to store a website address in them. When scanned with a mobile app like <a href="http://itunes.apple.com/us/app/bakodo-barcode-scanner-qr/id371932548?mt=8">Bakodo</a> or the <a href="http://itunes.apple.com/us/app/google-mobile-app/id284815942?mt=8">Google app</a> for the iPhone, the handset can load up the website it points to, and instantly give the user more information about the tag they have scanned.</p>
  <p>These 2-dimensional barcodes have a tolerance for errors, meaning that bits of it can be missing or covered up, while still being able to be read and used. That's quite handy, if you want to make them look a little more interesting and include a logo or something to attract people's attention to them. Even better, it's as simple as generating a normal QR code from the <a href="http://code.google.com/apis/chart/docs/gallery/qr_codes.html">Google Charts API</a> (<a href="http://chart.apis.google.com/chart?cht=qr&amp;chs=500x500&amp;chld=H|0&amp;chl=http://www.dankarran.com">e.g. one for dankarran.com</a>), downloading it, opening it in your favourite graphics editor, and inserting your logo. Check to make sure your scanner can still read it, tweak it if necessary, and you're set. Set the error correction level to the highest value possible (chld=H in the chart URL) and keep your URL as short as possible to give you most flexibility around your logo and less chance of it breaking barcode scanning applications.</p>
  <h2>Using QR codes to improve location-based information</h2>
  <p><img src="http://www.dankarran.com/sites/dankarran.com/files/qr-code-douglas-osm.png" alt="Custom design QR code for OpenStreetMap" width="180" height="180" style="float:right;padding: 10px" />QR codes could be quite useful for tourist information signs, to give people quick access to more information about the local area, a map of local amenities, or directions to whatever they are looking for. Most phones that have the ability to recognise a QR code probably also have a GPS or some other form of positioning built in, which could help them find their location on the standard Google Maps, but doesn't necessarily help them get at other detailed information, perhaps provided by the likes of <a href="http://openstreetmap.org">OpenStreetMap</a> or provided by local information sites or the local government. Using these codes to point to targeted local information could be of great use to visitors (as long as there's a note to tell them how to make use of the QR code).</p>
  <p>A QR code like the one at the top of the post could be useful for someone standing at a tourist information point in the Sea Terminal in Douglas, Isle of Man, as it takes them to <a href="http://goo.gl/KppjI">a map centred on that point</a>. The site it points to doesn't do much more than providing <a href="http://themap.im">maps of the Isle of Man</a> at the moment, and doesn't work too well on a mobile yet, but could (and hopefully will, before too long) provide much more information that could be of use to visitors. Similarly, the one to the right points to a <a href="http://osm.org/go/evGiECzt">map of Douglas</a> on the OpenStreetMap site.</p>
  <p>I haven't seen many QR codes in use yet, but hopefully they could become much more widespread in coming years. Could you see yourself making use of codes you found out on the street?</p>
  </div></div></div>


guid: 1079 at http://www.dankarran.com
url: http://www.dankarran.com/blog/2010/11/new-maps-of-the-isle-of-man
title: New maps of the Isle of Man
published: 2010-11-02 13:42:43
fetched: 2019-07-02 14:30:07.801716
created_at: 2019-06-09 18:11:40.256686
updated_at: 2019-07-02 14:30:07.882803
summary:
  <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>Last week I launched a section on <a href="http://themap.im" title="the map of the Isle of Man">themap.im</a> to show a gallery of <a href="http://themap.im/places">maps of the Isle of Man</a>. The aim is to create localised maps for different towns and villages around the Island, so for example you could bookmark a <a href="http://themap.im/castletown">map of Castletown</a>, <a href="http://themap.im/colby">Colby</a> or <a href="http://themap.im/cregneash">Cregneash</a>, or any other place listed in the <a href="http://themap.im/places">gallery</a>.</p>
  <p><a href="http://themap.im/places"><img src="http://www.dankarran.com/sites/dankarran.com/files/isle-of-man-maps.png" alt="Maps of the Isle of Man" /></a></p>
  <p>The maps got some great coverage with a <a href="http://news.bbc.co.uk/local/isleofman/hi/people_and_places/newsid_9132000/9132086.stm">BBC Isle of Man article</a>, which has generated a good amount of interest in the site. The more people the map is available to, the sooner we can iron out any issues in the map data. There may be the odd typo, things accidentally added in the wrong place, or things that are missing completely, so all <a href="mailto:suggestions@themap.im">feedback is welcome</a>.</p>
  <p>For me, building this site is a great opportunity to use <a href="http://dankarran.com/web-development/drupal">Drupal</a> to help promote <a href="http://www.openstreetmap.org">OpenStreetMap</a> data, using the <a href="http://openlayers.org">OpenLayers</a> mapping library (and <a href="http://drupal.org/project/openlayers">Drupal module</a>) to display the maps on the site, and I'm looking forward to building the site out further.</p>
  </div></div></div>


guid: http://blog.gravitystorm.co.uk/?p=866
title: Moderation, Authorisation and Background Task Improvements for OpenStreetMap
published: 2018-11-09 16:33:58
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.702473
updated_at: 2019-10-23 07:00:09.067968
url:
  https://blog.gravitystorm.co.uk/2018/11/09/moderation-authorisation-and-background-task-improvements-for-openstreetmap/

summary:
  We&#8217;re making steady progress with improving the OpenStreetMap codebase. Although it&#8217;s been a busy year for me, I managed to wrap up the moderation pull request, and it was merged and deployed in mid June. Lots of different people have worked on this over the years, starting way back as a GSoC project in 2015! [&#8230;]

content:
  <p>We&#8217;re making steady progress with improving the OpenStreetMap codebase. Although it&#8217;s been a busy year for me, I managed to wrap up the <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1576">moderation pull request</a>, and it was merged and deployed in mid June. Lots of different people have worked on this over the years, starting way back as a GSoC project in 2015! It was great to finally get it integrated into the site.</p>
  <p>Last week <a href="https://www.openstreetmap.org/user/mavl">mavl</a>, one of our site moderators, posted that <a href="https://www.openstreetmap.org/user/mavl/diary/46806">nearly a thousand issues were logged by OSM volunteers in the first 3 months</a> &#8211; at an average rate of over 10 per day! That&#8217;s a lot more than I was expecting, so future development ideas could involve working with the OpenStreetMap moderators to support their workflows.</p>
  <p>Also in June, the <a href="https://rubyforgood.org/">Ruby for Good</a> team picked us as one of their projects for their annual development event. A bunch of good stuff came out of that, in particular a <a href="https://github.com/openstreetmap/quad_tile/">new &#8220;quad_tile&#8221; gem</a>, which builds one of our C extensions automatically during gem installation, and therefore saves new developers from having to deal with that.</p>
  <p>In addition, they also kick-started work on our new authorisation framework, based on CanCanCan, which I <a href="https://blog.gravitystorm.co.uk/2018/04/11/groundwork-for-new-features-in-osm/">outlined previously</a>. It&#8217;s taken longer than I would have liked to get that ready, but <a href="https://github.com/openstreetmap/openstreetmap-website/pull/2023">the code was merged and deployed</a> last week, and I&#8217;ve started refactoring the rest of our controllers to use it. This will allow us to remove a lot of home-grown authorisation code and use a standard approach instead, which means less custom code to maintain as well as being more familiar to new developers. I see this authorisation framework as a key enabler for future projects &#8211; we&#8217;ll be able to develop new features faster, while making sure that only the right people have access.</p>
  <p>Another recent upgrade has been to <a href="https://github.com/openstreetmap/openstreetmap-website/pull/2037">set up</a> and start using the <a href="https://edgeguides.rubyonrails.org/active_job_basics.html">Active Job</a> framework. This allows tasks to be run in the background, and is now a standard part of Ruby on Rails. We&#8217;re already using it for <a href="https://github.com/openstreetmap/openstreetmap-website/pull/2038">sending notification emails</a>, so now if you post a comment to a popular diary entry, you don&#8217;t need to wait while the system sends dozens of notification emails to all the subscribers &#8211; they are queued and dealt with separately. A small improvement, perhaps, but the job framework will really come alive when we start using it for processing GPX trace uploads. I hope to have more news on that soon.</p>
  <p>If you want to follow our progress more closely, or get involved in the development, head over to our <a href="https://github.com/openstreetmap/openstreetmap-website">GitHub repository</a> for full installation and development guides, along with all the issues and pull requests that we are dealing with!</p>


guid: http://blog.gravitystorm.co.uk/?p=861
url: https://blog.gravitystorm.co.uk/2018/04/11/groundwork-for-new-features-in-osm/
title: Groundwork for new features in OpenStreetMap
published: 2018-04-11 09:53:03
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.705160
updated_at: 2019-10-23 07:00:09.070685
summary:
  Last summer I finished a large refactoring, and thought it would be nice to change tack, and so I decided to try to push through a new feature as my next project. Refactoring is worthwhile, but it has a long-term pay-off. On the other hand, new features show progress to a wider audience, and so [&#8230;]

content:
  <p>Last summer I <a href="https://blog.gravitystorm.co.uk/2017/06/01/factory-refactoring-done/">finished a large refactoring</a>, and thought it would be nice to change tack, and so I decided to try to push through a new feature as my next project. Refactoring is worthwhile, but it has a long-term pay-off. On the other hand, new features show progress to a wider audience, and so new features are another avenue to getting people interested and involved in development.</p>
  <p>I picked an <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1576">old Google Summer of Code project</a> that hadn&#8217;t really been wrapped up, and immediately spotted a bunch of changes that would be needed to make it easier for others to help review it. Long story short, it needed a lot more work than anyone realised and it&#8217;s taken me a few months to get it ready. I&#8217;ve learned a few lessons about GSoC projects along the way, but that&#8217;s a story for another time.</p>
  <p>I want to keep going with the refactoring, since a better codebase leads to happier developers and eventually to better features. But it&#8217;s worthwhile having some sort of a goal, otherwise it&#8217;s hard to decide what&#8217;s important to refactor, and to avoid getting lost in the weeds. There have been discussions in the past about <a href="https://vimeo.com/68097488#t=1348s">adding some form of Groups</a> to OpenStreetMap, and it&#8217;s a topic that keeps on coming up. But I know that if anyone tried implementing Groups on top of our current codebase, it would be impossible to maintain, and it&#8217;s far too big a challenge for a self-contained project like GSoC.</p>
  <p>So what things do I think would make it easier to implement Groups? The most obvious piece of groundwork is <a href="https://github.com/openstreetmap/openstreetmap-website/issues/1626">a proper authorisation framework</a>. Without that, deciding who can view messages in or add members to each group would be gnarly. I also don&#8217;t want to add many more new features to the site with our &#8220;default allow&#8221; permissions &#8211; it&#8217;s too easy to get that wrong, particularly adding something substantial and complex like Groups.</p>
  <p>I had a stab at adding the authorisation framework a few weeks ago, but quickly realised some more groundwork would help. We can make life easier for defining the permissions if we use <a href="http://guides.rubyonrails.org/routing.html#resource-routing-the-rails-default">standard Rails resource routing</a> in more places. However, that involves refactoring controller methods and renaming various files. That refactoring becomes easier if we use <a href="http://guides.rubyonrails.org/routing.html#creating-paths-and-urls-from-objects">standard Rails link helpers</a>, and if we use <a href="http://guides.rubyonrails.org/i18n.html#lazy-lookup">shortened internationalization lookups</a>.</p>
  <p>So there&#8217;s some more groundwork to do before the groundwork before the groundwork&#8230;</p>


guid: http://blog.gravitystorm.co.uk/?p=856
url: https://blog.gravitystorm.co.uk/2017/06/01/factory-refactoring-done/
title: Factory Refactoring – Done!
published: 2017-06-01 18:21:52
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.708105
updated_at: 2019-10-23 07:00:09.073456
summary:
  After over 50 Pull Requests spread over the last 9 months, I&#8217;ve finally finished refactoring the openstreetmap-website test suite to use factories instead of fixtures. Time for a celebration! As I&#8217;ve discussed in previous posts, the openstreetmap-website codebase powers the main OpenStreetMap website and the map editing API. The test suite has traditionally only used [&#8230;]

content:
  <p>After over 50 Pull Requests spread over the last 9 months, I&#8217;ve <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1556">finally finished</a> refactoring the <a href="https://github.com/openstreetmap/openstreetmap-website">openstreetmap-website</a> test suite to use factories instead of fixtures. Time for a celebration!</p>
  <p>As I&#8217;ve discussed in <a href="https://blog.gravitystorm.co.uk/2016/09/15/refreshing-the-openstreetmap-codebase/">previous</a> <a href="https://blog.gravitystorm.co.uk/2017/02/21/steady-progress-osm-website/">posts</a>, the openstreetmap-website codebase powers the main OpenStreetMap website and the map editing API. The test suite has traditionally only used <a href="http://guides.rubyonrails.org/testing.html#the-low-down-on-fixtures">fixtures</a>, where all test data was preloaded into the database and the same data used for every test. One drawback of this approach is that any change to the fixtures can have knock-on effects on other tests. For example, adding another diary entry to the fixtures could break a different test which expects a particular number of diary entries to be found by a search query. There are also more subtle problems, including the lack of clear intent in the tests. When you read a test that asserts that a given Node or Way is found, it was often not clear which attributes of the fixture were important &#8211; perhaps that Node belonged to a particular Changeset, or had a particular timestamp, or was in a particular location, or a mixture of other attributes. Figuring out these hidden intents for each test was often a major source of the refactoring effort &#8211; and there&#8217;s 1080 tests in the suite, with more than 325,000 total assertions!</p>
  <p>Changing to <a href="https://github.com/thoughtbot/factory_girl/blob/master/GETTING_STARTED.md#defining-factories">factories</a> has made the tests independent of each other. Now every test starts with a blank database, and only the objects that are needed are created, using the factories. This means that tests can more easily create a particular database record for the test at hand, without interfering with other tests. The intent of the test is often clearer too, since creating objects from factories happens in the test itself and are therefore explicit about what attributes are the important ones.</p>
  <p>An example of the benefits of factories was when I <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1348/files">fixed a bug</a> around encoding of diary entry titles in our RSS feeds. I easily created a diary entry with a specific and unusual title, without interfering with any other tests or having to create yet another fixture.</p>
  <pre class="brush: ruby; title: ; notranslate">
  def test_rss_character_escaping
    create(:diary_entry, :title =&gt; &quot;&lt;script&gt;&quot;)
    get :rss, :format =&gt; :rss

    assert_match &quot;&lt;title&gt;&amp;lt;script&amp;gt;&lt;/title&gt;&quot;, response.body
  end
  </pre>
  <p>All in all, this took much, much longer than I was expecting! <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1279">Looking back</a>, I feel I might have <a href="https://github.com/gravitystorm/openstreetmap-website/issues">picked a different task</a>, but I&#8217;m glad that it&#8217;s all done now. I&#8217;m certainly glad that I won&#8217;t have to do it all again! Moving on, it&#8217;s now easier for me and the other developers to write robust tests, and this will help us implement new features more quickly.</p>
  <p>Big thanks go to Tom Hughes for reviewing all 51 individual pull requests! Thanks also to everyone who lent me their encouragement.</p>
  <p>So the big question is &#8211; what will I work on next?</p>


guid: http://blog.gravitystorm.co.uk/?p=852
url: https://blog.gravitystorm.co.uk/2017/02/21/steady-progress-osm-website/
title: Steady progress on the OpenStreetMap Website
published: 2017-02-21 16:06:00
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.710819
updated_at: 2019-10-23 07:00:09.075812
summary:
  Time for a short status update on my work on the openstreetmap-website codebase. It&#8217;s been a few months since I started refactoring the tests and the work rumbles on. A few of my recent coding opportunities have been taken up with other projects, including the blogs aggregator, the 2017 budget for the OSMF Operations Working [&#8230;]

content:
  <p>Time for a short status update on my work on the <a href="https://github.com/openstreetmap/openstreetmap-website">openstreetmap-website codebase</a>. It&#8217;s <a href="https://blog.gravitystorm.co.uk/2016/09/15/refreshing-the-openstreetmap-codebase/">been a few months</a> since I started refactoring the tests and the work rumbles on. A few of my recent coding opportunities have been taken up with other projects, including <a href="https://blog.gravitystorm.co.uk/2017/01/31/upgrading-the-osm-blogs-aggregator/">the blogs aggregator</a>, the <a href="https://github.com/openstreetmap/operations/issues/132">2017 budget</a> for the OSMF Operations Working Group (OWG), and the <a href="https://operations.osmfoundation.org/">new OWG website</a>.</p>
  <p>With the fixtures refactoring I&#8217;ve <a href="https://github.com/gravitystorm/openstreetmap-website/issues/4">already tackled</a> the low-hanging fruit. So now I&#8217;m forced to tackle the big one &#8211; converting the Users fixtures. The User model is unsurprisingly used in most tests for the website, so the conversion is quite time-consuming and I&#8217;ve had to break this down into multiple stages. However, when this bit of the work is complete most future Pull Requests on other topics can be submitted without having to use any fixtures at all. The nodes/ways/relations tests will then be the main thing remaining for conversion, but since the code that deals with those changes infrequently, it&#8217;s best to work on the User factories first.</p>
  <p>As I&#8217;ve been working on replacing the fixtures, I&#8217;ve come across <a href="https://github.com/gravitystorm/openstreetmap-website/issues">a bunch of other things I want to change</a>. But before tackling all that I&#8217;m going to mix it around a bit. My goal is to alternate between the work I think is the most important, and also helping other developers with their own work. We have around <a href="https://github.com/openstreetmap/openstreetmap-website/pulls">40 outstanding pull requests</a> and some need a hand to complete. There are plenty of straightforward coding fixes among the <a href="https://github.com/openstreetmap/openstreetmap-website/issues">250 open</a> issues that I can work on too. I hope that if more of the issues and particularly the pull requests are completed, this will motivate some more people to get involved in development.</p>
  <p>If you have any thoughts on what I should be prioritising &#8211; particularly if you&#8217;ve got an outstanding pull request of your own &#8211; then let me know in the comments!</p>


guid: http://blog.gravitystorm.co.uk/?p=843
url: https://blog.gravitystorm.co.uk/2017/01/31/upgrading-the-osm-blogs-aggregator/
title: Upgrading the OpenStreetMap Blogs Aggregator
published: 2017-01-31 15:14:08
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.713170
updated_at: 2019-10-23 07:00:09.078209
summary:
  One of my projects over the winter has been upgrading the blogs.openstreetmap.org feed aggregator. This site collects OpenStreetMap-themed posts from a large number of different blogs and shows them all in one place. The old version of the site was certainly showing its age. The software that powered it, called PlanetPlanet hasn&#8217;t been updated for [&#8230;]

content:
  <p>One of my projects over the winter has been upgrading the <a href="https://blogs.openstreetmap.org">blogs.openstreetmap.org</a> feed aggregator. This site collects OpenStreetMap-themed posts from a large number of different blogs and shows them all in one place. The old version of the site was certainly showing its age. The software that powered it, called <a href="http://www.planetplanet.org/">PlanetPlanet</a> hasn&#8217;t been updated for over 10 years, and can&#8217;t cope with feeds served using https, so an increasing number of blogs were disappearing from the site. Time for an upgrade.</p>
  <p>My larger goal was moving the administration of the site into the open, in order to get more people involved. <a href="http://shaunmcdonald.me.uk/">Shaun</a> has maintained the old system for many years and has done a great job. Unfortunately there were tasks that could only be done by him, such as adding new feeds, removing old feeds, and customising the site. To make any changes you had to know who to ask, and hope that Shaun had time to work on it for you. It was also unclear what criteria a blog feed had to meet to be added, and even more so, if and when a blog should be removed.</p>
  <p>The challenge was therefore to move the configuration to a public repository, move the deployment to OSMF hardware, create a clear policy for managing the feeds, and thereby reduce the barriers to getting involved all round.</p>
  <p>Thankfully, other people did almost all of the work! After I investigated the different feed aggregation software options &#8211; most of them are barely maintained nowadays &#8211; I reckoned that <a href="https://github.com/feedreader/pluto">Pluto</a> was the best choice. It turns out Shaun had previously come to the same conclusion, and had almost completed the migration himself. Then <a href="https://compton.nu/">Tom</a> put together the Chef <a href="https://github.com/openstreetmap/chef/tree/master/cookbooks/blogs">deployment configuration</a>, which was the second half of the work. So all I had to do was finish converting the list of feeds, make a few template changes, and everything was ready to go. After <a href="https://github.com/openstreetmap/chef/pull/102">a few weeks delay</a> while the sysadmins worked on other tasks, the new site <a href="https://git.openstreetmap.org/dns.git/commit/f172b80fc79f43e7c1645bcb841d284a58a06b9b">went live on Friday</a>.</p>
  <p>If you know of a blog that should be added to the site, please <a href="https://github.com/gravitystorm/blogs.osm.org/blob/master/FEEDS.md">check our new guidelines</a>. If you have any changes you want to make to the <a href="https://github.com/gravitystorm/blogs.osm.org">site</a>, the deployment, or the software that powers it, now is you chance to get involved!</p>


guid: http://blog.gravitystorm.co.uk/?p=836
url: https://blog.gravitystorm.co.uk/2016/09/15/refreshing-the-openstreetmap-codebase/
title: Refreshing the OpenStreetMap Codebase
published: 2016-09-15 14:48:38
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.715804
updated_at: 2019-10-23 07:00:09.080613
summary:
  The codebase that powers OpenStreetMap is older than any other Rails project that I work on. The first commit was in July 2006, and even then, that was just a port of an existing pre-rails system (hence why you might still see it referred to as &#8220;The Rails Port&#8221;) It&#8217;s a solid, well-tested and battle-hardened [&#8230;]

content:
  <p>The <a href="https://github.com/openstreetmap/openstreetmap-website">codebase that powers OpenStreetMap</a> is older than any other <a href="http://rubyonrails.org/">Rails</a> project that I work on. The <a href="https://github.com/openstreetmap/openstreetmap-website/commit/06f3473d1c64d2e0a1ecc6f1a9c4679a52bc4761">first commit</a> was in July 2006, and even then, that was just a port of an existing pre-rails system (hence why you might still see it referred to as &#8220;The Rails Port&#8221;)</p>
  <p>It&#8217;s a solid, well-tested and battle-hardened codebase. It&#8217;s frequently updated too, particularly its dependencies. But if you know where to look, you can see its age. We have <a href="https://github.com/openstreetmap/openstreetmap-website/graphs/contributors">very few enduring contributors</a>, with is surprising given its key position within the larger OpenStreetMap development community. So I&#8217;ve been taking a look to learn what I can do to help.</p>
  <p><a href="https://github.com/openstreetmap/openstreetmap-website/graphs/contributors"><img data-attachment-id="840" data-permalink="https://blog.gravitystorm.co.uk/2016/09/15/refreshing-the-openstreetmap-codebase/openstreetmap-website-contributions/" data-orig-file="https://gravitystorm.files.wordpress.com/2016/09/openstreetmap-website-contributions.png" data-orig-size="600,193" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="OpenStreetMap-website Contributions" data-image-description="" data-medium-file="https://gravitystorm.files.wordpress.com/2016/09/openstreetmap-website-contributions.png?w=300" data-large-file="https://gravitystorm.files.wordpress.com/2016/09/openstreetmap-website-contributions.png?w=600" src="https://gravitystorm.files.wordpress.com/2016/09/openstreetmap-website-contributions.png?w=625" alt="OpenStreetMap-website Contributions"   class="aligncenter size-full wp-image-840" srcset="https://gravitystorm.files.wordpress.com/2016/09/openstreetmap-website-contributions.png 600w, https://gravitystorm.files.wordpress.com/2016/09/openstreetmap-website-contributions.png?w=128 128w, https://gravitystorm.files.wordpress.com/2016/09/openstreetmap-website-contributions.png?w=300 300w" sizes="(max-width: 600px) 100vw, 600px" /></a></p>
  <p>For someone just getting started with Rails, they&#8217;ll find that many parts of our code don&#8217;t match what&#8217;s in any of the books they read, or any the guides online. More experienced developers will spot a lot of things that were written years ago, and would nowadays been done differently. And for some of our developers, particularly our Summer of Code students, they are learning what to do by reading our existing code, so our idiosyncrasies accumulate.</p>
  <p>I started trying to fix a minor bug with the diary entries and a bunch of things struck me as needing a thorough refresh. I started the process of <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1285">refactoring the tests</a> to use factories instead of fixtures &#8211; 2 down, 37 to go. I&#8217;ve started rewriting the controllers to use the standard rails CRUD method names. And I&#8217;ve <a href="https://github.com/gravitystorm/openstreetmap-website/issues/2">made a list</a> of plenty of other things that I&#8217;d like to tackle, all of which will help lower the barrier for new (and experienced) developers who want to get stuck into the openstreetmap-website code.</p>
  <p>I hope that progress will snowball &#8211; as it becomes easier to contribute, more people will join in, and in turn help make it even easier to contribute.</p>
  <p>But it&#8217;s time-consuming. I need help to share these projects around. If you&#8217;re interested, please <a href="https://github.com/openstreetmap/openstreetmap-website">get stuck in</a>!</p>


guid: http://blog.gravitystorm.co.uk/?p=818
url: https://blog.gravitystorm.co.uk/2016/07/28/getting-involved-in-owg/
title: Getting Involved in the Operations Working Group
published: 2016-07-28 18:31:28
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.718474
updated_at: 2019-10-23 07:00:09.083049
summary:
  For the last few years I&#8217;ve been trying to get more people involved in the Operations Working Group &#8211; the team within the OpenStreetMap Foundation that runs all of our services. Each time I think &#8220;why aren&#8217;t more people involved&#8221;, I try to figure out some plausible barriers to entry, and then work on fixing [&#8230;]

content:
  <p>For the last few years I&#8217;ve been trying to get more people involved in the <a href="http://wiki.osmfoundation.org/wiki/Operations_Working_Group">Operations Working Group</a> &#8211; the team within the <a href="http://www.osmfoundation.org">OpenStreetMap Foundation</a> that runs all of our services. Each time I think &#8220;why aren&#8217;t more people involved&#8221;, I try to figure out some plausible barriers to entry, and then work on fixing them.</p>
  <p>One of the reasons is that we&#8217;re very quiet about what we do &#8211; there&#8217;s a pride in keeping OpenStreetMap humming along and not causing too much of a fuss. But the lack of publicity hurts us when we&#8217;re trying to get more people involved. Hence this blog post, among other reasons.</p>
  <p>I&#8217;ve been working recently (as in, for the last few years) on making as much of our OWG activities public as possible, rather than hidden away on our mailing list or in our meetings. So we now have <a href="https://github.com/openstreetmap/operations/issues">a public issue tracker showing our tasks</a>, and we also <a href="https://gravitystorm.github.io/owg-log/">publish a monthly summary of our activities</a>.</p>
  <p>To make OpenStreetMap work, we run a surprisingly large number of servers. For many years we maintained a list of our hardware and what each server was being used for on the OpenStreetMap wiki, which helped new people find out how everything works. Maintaining this information was a lot of work and the wiki was often outdated. For my own projects I use <a href="https://jekyllrb.com">Jekyll</a> to create static websites (which are easier to find hosting for than websites that need databases), and information on Jekyll sites can be generated with the help of <a href="https://jekyllrb.com/docs/datafiles/">data files</a>. Since OWG uses <a href="https://www.chef.io/">Chef</a> to configure all of our servers, and Chef knows both the hardware configuration and also what the machines are used for, the idea came that we could automate these server information pages entirely. That website is now live on <a href="https://hardware.openstreetmap.org">hardware.openstreetmap.org</a> so we have a public, accurate and timely list of all of our hardware and the services running on it.</p>
  <p>Now my attention has moved to <a href="https://github.com/openstreetmap/chef">our Chef configuration</a>. Although the configuration has been public for years, currently the barrier to entry is substantial. One straightforward (but surprisingly time-consuming) improvement was to simply write a README for each cookbook &#8211; 77 in all. I <a href="https://github.com/openstreetmap/chef/pull/73">finished that project</a> last week.</p>
  <p>Unless you have administrator rights to OSMF hardware (and even I don&#8217;t have that!) you need to write the chef configuration &#8216;blind&#8217; &#8211; that is, you can propose a change but you can&#8217;t realistically test that it works before you make a pull request. That makes proposing changes close to impossible, so it&#8217;s not surprising that few non-administrators have ever contributed changes. I have experience with a few tools that can help, the most important being test-kitchen. This allows a developer to locally check that their changes work, and have the desired effect, before making a pull request, and also it allows the administrators to check that the PR works before deploying the updated cookbook. Both <a href="https://github.com/zerebubuth/">Matt</a> and I have been working on this recently, and <a href="https://github.com/openstreetmap/chef/pull/74">today I proposed a basic test-kitchen configuration</a>.</p>
  <p>This will only be the start of a long process, since eventually most of those 77 cookbooks will need test-kitchen configurations. Even in my initial attempts to test the <a href="https://github.com/openstreetmap/chef/tree/master/cookbooks/serverinfo">serverinfo cookbook</a> (that generates <a href="https://hardware.openstreetmap.org">hardware.openstreetmap.org</a>) I found a bunch of problems, some of which I haven&#8217;t yet figured out how to work around. There will be many more of these niggles found, but the goal is to allow future developers to improve each cookbook using only their own laptops.</p>
  <p>All of these are small steps on the long path to getting more people involved in the Operations Working Group. If you&#8217;re interested in helping, get stuck in to both <a href="https://github.com/openstreetmap/operations">our task list</a> and <a href="https://github.com/openstreetmap/chef">our chef repo</a>, and let me know when you get stuck.</p>


guid: http://blog.gravitystorm.co.uk/?p=815
url: https://blog.gravitystorm.co.uk/2016/03/23/coding-and-writing-with-the-atom-editor/
title: Coding and writing with the Atom editor
published: 2016-03-23 17:19:27
fetched: 2019-10-23 07:00:09.004492
created_at: 2019-06-09 18:11:40.721018
updated_at: 2019-10-23 07:00:09.085539
summary:
  Atom brands itself as &#8220;A hackable text editor for the 21st Century&#8221;. I first used Atom in August 2014 as an alternative to the Kate editor that comes with Ubuntu and which I had used for many years before that. I was drawn towards Atom by all the publicity around it on Hacker News, and [&#8230;]

content:
  <p><a href="https://atom.io/">Atom</a> brands itself as &#8220;A hackable text editor for the 21st Century&#8221;. I first used Atom in August 2014 as an alternative to the <a href="http://kate-editor.org/">Kate</a> editor that comes with Ubuntu and which I had used for many years before that. I was drawn towards Atom by all the publicity around it on <a href="https://news.ycombinator.com/">Hacker News</a>, and it seemed to have momentum and a few useful features that Kate was lacking. It didn&#8217;t take me long to find myself using Atom exclusively.</p>
  <p>Much is said online about Atom being slow to respond to key presses, or that it&#8217;s built in JavaScript, or that it uses a Webview component to render everything, or some other deeply technical topic, but none of those topics are worth discussing here.</p>
  <p>Instead, a key feature for me is that Atom is built on an extensive a plugin architecture, and ships with a bunch of useful plugins by default. There are <a href="https://atom.io/packages">hundreds more available to download</a>, but trawling through the list is a pain and there&#8217;s always a few gems buried in the long tail. To save you some hassle, here&#8217;s a list of the plugins that I use. I&#8217;ve divided them by the three main tasks for which I use Atom regularly &#8211; cartography, coding and writing.</p>
  <h2>Cartography</h2>
  <p><a href="https://atom.io/packages/language-carto">language-carto</a> &#8211; &#8220;Carto language highlight in Atom&#8221;. I only found out about this recently from <a href="https://github.com/mojodna/">Seth</a> but, as you can imagine, it&#8217;s very useful when writing CartoCSS styles! Finding out about this plugin by chance conversation is what sparked the idea of writing this post.</p>
  <p><a href="https://atom.io/packages/color-picker">color-picker</a> &#8211; &#8220;A Color Picker for Atom&#8221;. I use Atom to edit all my CartoCSS projects, so this seems like a good idea, but I haven&#8217;t actually used it much since installing. I prefer <a href="http://inkscape.org/">Inkscape</a>&#8216;s HSL colour picker tab when I&#8217;m doing cartography, and working in Inkscape lets me quickly mock up different features together to see if the colours balance.</p>
  <p><a href="https://atom.io/packages/pigments">pigments</a> &#8211; &#8220;A package to display colors in project and files&#8221;. This simply highlights colour definitions like #ab134f with the appropriate colour &#8211; almost trivial, but actually very useful when looking at long lists of colour definitions.</p>
  <h2>Development</h2>
  <p><a href="https://atom.io/packages/git-blame">git-blame</a> &#8211; &#8220;Toggle git-blame annotations in the gutter of atom editor&#8221;. I only started using this last week and it&#8217;s a bit rough around the edges, but it&#8217;s quicker to run git-blame within my editor than either running it in the terminal or clicking around on Bitbucket and Github. Linking from the blame annotations to the online diffs is a nice touch.</p>
  <p><a href="https://atom.io/packages/language-haml">language-haml</a> &#8211; &#8220;HAML package for Atom&#8221;. I was first introduced to <a href="http://haml.info/">HAML</a> when working on <a href="http://www.cyclescape.org/">Cyclescape</a> and instantly preferred it to ERB, so all my own Ruby on Rails applications use HAML too. This plugin adds syntax highlighing to Atom for HAML files.</p>
  <p><a href="https://atom.io/packages/linter">linter</a> &#8211; &#8220;A Base Linter with Cow Powers&#8221;, <a href="https://atom.io/packages/linter-rubocop">linter-rubocop</a> &#8211; &#8220;Lint <code>Ruby</code> on the fly, using rubocop&#8221;, and <a href="https://atom.io/packages/linter-haml">linter-haml</a> &#8211; &#8220;Atom linter plugin for HAML, using haml-lint&#8221;. I&#8217;m a big fan of linters, especially <a href="https://github.com/bbatsov/rubocop">rubocop</a> &#8211; there&#8217;s a separate blog post on this topic in the works. These three plugins allow me to see the warnings in Atom as I&#8217;m typing the code, rather than waiting until I save and I remember to run the tool separately. Unfortunately it means I get very easily distracted when contributing to projects that have outstanding rubocop warnings!</p>
  <p><a href="https://atom.io/packages/project-manager">project-manager</a> &#8211; &#8220;Project Manager for easy access and switching between projects in Atom&#8221;. I used this for a while but I found it a bit clunky. Since I&#8217;m bound to have a terminal open, I find it quicker to navigate to the project directory and launch atom from there, instead of clicking around through user interfaces.</p>
  <h2>Writing</h2>
  <p><a href="https://atom.io/packages/Zen">Zen</a> &#8211; &#8220;distraction free writing&#8221;. This is great when I&#8217;m writing prose, rather than code. Almost all my writing now is in Markdown (mainly for <a href="http://jekyllrb.com/">Jekyll</a> sites, again, a topic for another post) and Atom comes with effective markdown syntax highlighting built-in. I&#8217;m using this plugin to help draft this post!</p>
  <p>That&#8217;s it! Perhaps you&#8217;ve now found an Atom plugin that you didn&#8217;t know existed, or been inspired to download Atom and try it out. If you have any suggestions for plugins that I might find useful, please let me know!</p>


guid: http://blog.gravitystorm.co.uk/?p=796
url: https://blog.gravitystorm.co.uk/2016/01/11/openstreetmap-as-global-infrastructure/
title: OpenStreetMap as Global Infrastructure
published: 2016-01-11 18:58:57
fetched: 2019-07-19 10:30:07.949157
created_at: 2019-06-09 18:11:40.723698
updated_at: 2019-07-19 10:30:08.022705
summary:
  Shortly after the SotM US Conference in New York last year I was at a café discussing the future of OpenStreetMap. I was trying to describe my thoughts on how OSM could be perceived in a few years time and coined the phrase &#8220;Global Infrastructure&#8221; to label my thoughts. So what do I mean by [&#8230;]

content:
  <p>Shortly after the SotM US Conference in New York last year I was at a café discussing the future of OpenStreetMap. I was trying to describe my thoughts on how OSM could be perceived in a few years time and coined the phrase &#8220;Global Infrastructure&#8221; to label my thoughts.</p>
  <p>So what do I mean by Global Infrastructure? I was thinking that over the next 5-10 years (bearing in mind OSM is already over 10 years old) that it would no longer be thought of as an interesting open source or open data &#8220;project&#8221;. It would no longer be something cool or unusual that you find out about, it&#8217;ll just be something that exists, that permeates the world and that people and organisations all over the globe will be using and depending on OpenStreetMap without much further thought. Like the PSTN &#8211; the global system of telephone networks that means you can call anyone else on the planet &#8211; OSM will become Global Infrastructure that powers millions of maps.</p>
  <p>We&#8217;re already at the stage where OSM is a routine and necessary part of the world. There are hundreds of non-tech organisations using OpenStreetMap every day, from the World Bank to local advocacy groups. I could try to make a list but I&#8217;d be here all day and I&#8217;d miss hundreds that I don&#8217;t even know about. It&#8217;s been at least five years since the various interesting uses and users of OpenStreetMap become much bigger than my own knowledge horizon! And this list of users will continue to grow and grow until the absence of OpenStreetMap will become inconceivable.</p>
  <p>So beyond a catchy label, what does the concept of Global Infrastructure mean in practice? Well, if nothing else, I use it to guide my thoughts for the OSMF Operations Working Group. We&#8217;re being depended on, and no longer by a few thousand hobbyists who can tolerate some rough edges here and there. We&#8217;re going to have to keep building and scaling the core OSM services, and yet at the same time fade into the background. In a couple of years, if not already, most people in the OpenStreetMap community won&#8217;t even consider how it all works.</p>
  <p>There&#8217;s a never-ending series of technical challenges that the Operations team faces, some of which I can&#8217;t help with directly. So instead I&#8217;m keeping my focus on scaling our team&#8217;s activities, whether that&#8217;s <a href="https://hardware.openstreetmap.org/">automating our server information pages</a>, systemising our capacity planning, <a href="https://gravitystorm.github.io/owg-log/">publicising our activities</a> or lowering the barriers to getting more people involved in the work that we do.</p>
  <p>2016 will also see us working on new approaches to running our infrastructure. Perhaps not many visible changes, but improved systems for internal event log management and adapting various services to use scalable object storage are two projects that are high on my list &#8211; as are the perennial tasks around scaling and ensuring availability of the core database.</p>
  <p>The challenge of moving OpenStreetMap beyond being &#8220;just a project&#8221; and into &#8220;Global Infrastructure&#8221; goes much wider than the work of the OWG. It will affect every part of the project, from our documentation to our communications to how we self-organise tens of thousands of new contributors. I hope we can set aside anything that we no longer need, keep up our momentum,  and seize the opportunities to build on what we have already achieved.</p>


guid: http://blog.gravitystorm.co.uk/?p=791
url: https://blog.gravitystorm.co.uk/2015/10/30/some-openstreetmap-futures/
title: Some OpenStreetMap Futures
updated: 2015-10-30 11:40:50
fetched: 2019-06-09 18:11:40.648840
created_at: 2019-06-09 18:11:40.726413
updated_at: 2019-06-09 18:11:40.726413
summary:
  Earlier this year I was invited to give two presentations at the State of the Map US 2015 conference, which was held in the United Nations Headquarters in NYC. What a venue! As well as an update on the OpenStreetMap Carto project I gave a presentation on what I see as some of the development [&#8230;]

content:
  <p>Earlier this year I was invited to give two presentations at the State of the Map US 2015 conference, which was held in the United Nations Headquarters in NYC. What a venue!</p>
  <p>As well as an update on the OpenStreetMap Carto project I gave a presentation on what I see as some of the development prospects for OSM over the next few years. Making predictions is hard, especially about the future, etc, but I gave it my best shot.</p>
  <iframe class='youtube-player' type='text/html' width='560' height='315' src='https://www.youtube.com/embed/b6yRH5Ics7I?version=3&#038;rel=1&#038;fs=1&#038;autohide=2&#038;showsearch=0&#038;showinfo=1&#038;iv_load_policy=1&#038;wmode=transparent' allowfullscreen='true' style='border:0;'></iframe>
  <p>I think it&#8217;ll be interesting to look back in a few years and see how much of what I discuss was prescient, and more interestingly, what topics I missed entirely!</p>
  <p>One of the sections that interested me most (and took by far the longest to prepare the slides for) is that looking at who our developers are, and how this changes over time. It&#8217;s just before the <a href="https://youtu.be/b6yRH5Ics7I?t=17m54s">18 minute mark in the video</a> if you want to have a look.</p>
  <p>Here&#8217;s a couple of the charts, which provide food for thought (bear in mind they were produced in June, so the 2015 numbers reflect only the first 6 months of the year):</p>
  <p><a href="https://gravitystorm.files.wordpress.com/2015/04/cohorts.png"><img data-attachment-id="793" data-permalink="https://blog.gravitystorm.co.uk/2015/10/30/some-openstreetmap-futures/cohorts/" data-orig-file="https://gravitystorm.files.wordpress.com/2015/04/cohorts.png" data-orig-size="567,319" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="OSM Developer Cohorts" data-image-description="" data-medium-file="https://gravitystorm.files.wordpress.com/2015/04/cohorts.png?w=300&#038;h=169" data-large-file="https://gravitystorm.files.wordpress.com/2015/04/cohorts.png?w=567" class="alignnone size-medium wp-image-793" src="https://gravitystorm.files.wordpress.com/2015/04/cohorts.png?w=300&#038;h=169" alt="OSM Developer Cohorts" width="300" height="169" srcset="https://gravitystorm.files.wordpress.com/2015/04/cohorts.png?w=300&amp;h=169 300w, https://gravitystorm.files.wordpress.com/2015/04/cohorts.png?w=128&amp;h=72 128w, https://gravitystorm.files.wordpress.com/2015/04/cohorts.png 567w" sizes="(max-width: 300px) 100vw, 300px" /></a><a href="https://gravitystorm.files.wordpress.com/2015/04/retention.png"><img data-attachment-id="792" data-permalink="https://blog.gravitystorm.co.uk/2015/10/30/some-openstreetmap-futures/retention/" data-orig-file="https://gravitystorm.files.wordpress.com/2015/04/retention.png" data-orig-size="800,536" data-comments-opened="1" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" data-image-title="OSM Developer Retention ratios" data-image-description="" data-medium-file="https://gravitystorm.files.wordpress.com/2015/04/retention.png?w=300&#038;h=201" data-large-file="https://gravitystorm.files.wordpress.com/2015/04/retention.png?w=800" class="alignnone size-medium wp-image-792" src="https://gravitystorm.files.wordpress.com/2015/04/retention.png?w=300&#038;h=201" alt="OSM Developer Retention ratios" width="300" height="201" srcset="https://gravitystorm.files.wordpress.com/2015/04/retention.png?w=300&amp;h=201 300w, https://gravitystorm.files.wordpress.com/2015/04/retention.png?w=600&amp;h=402 600w, https://gravitystorm.files.wordpress.com/2015/04/retention.png?w=128&amp;h=86 128w" sizes="(max-width: 300px) 100vw, 300px" /></a></p>
  <p>I think the future success of OpenStreetMap depends on improving these figures &#8211; we should be aiming to retain at least 40% of our developers after their first year of contributing. We&#8217;re clearly getting better at attracting new developers, but what do you think is stopping them from sticking around?</p>


guid: https://gagravarr.livejournal.com/147212.html
url: https://gagravarr.livejournal.com/147212.html
title: Apache CXF, JAX-RS, and injecting dependencies into your resources without Spring
published: 2014-04-11 17:38:10
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.320693
updated_at: 2019-10-27 00:00:11.120525
summary:
  I find myself (for <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a>) wanting to do some work on an Apache CXF powered, RAX-RS RESTful API. For a few reasons, we're not using Spring, and we're not using a configuration XML file. It's all defined and run in code.<br /><br />For this, we now need to pass in some dependencies to our Resource classes. Ask around, and everyone says "that's easy with Spring", but in this case we're not using Spring... After quite a bit of reading around, I've found the answer. What you need to do is use a <a href="http://cxf.apache.org/javadoc/latest-2.4.x/org/apache/cxf/jaxrs/lifecycle/SingletonResourceProvider.html" rel="nofollow">SingletonResourceProvider</a> to control what object gets returned when CXF wants your resource.<br /><br />Using the code below, you can run a simple server, and then fetch <i><a href='http://localhost:8765/test' rel='nofollow'>http://localhost:8765/test</a></i> from your web browser / curl etc. Because of the singleton, you get "I am a singleton!", and not "I was re-created, not a singleton" which you'd get if CXF created a new instance each time.<br /><br />So, if you find yourself needing to inject some dependencies into your CXF resource classes, and you're not in spring, you can use this pattern to create the instance with the dependencies passed in, then have those used.<br /><br /><pre>/*
   * Licensed to the Apache Software Foundation (ASF) under one or more
   * contributor license agreements.  See the NOTICE file distributed with
   * this work for additional information regarding copyright ownership.
   * The ASF licenses this file to You under the Apache License, Version 2.0
   * (the "License"); you may not use this file except in compliance with
   * the License.  You may obtain a copy of the License at
   *
   *     http://www.apache.org/licenses/LICENSE-2.0
   *
   * Unless required by applicable law or agreed to in writing, software
   * distributed under the License is distributed on an "AS IS" BASIS,
   * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   * See the License for the specific language governing permissions and
   * limitations under the License.
   */

  import java.util.ArrayList;
  import java.util.List;

  import javax.ws.rs.GET;
  import javax.ws.rs.Path;
  import javax.ws.rs.Produces;
  import javax.ws.rs.core.MediaType;
  import javax.ws.rs.core.Response;

  import org.apache.cxf.binding.BindingFactoryManager;
  import org.apache.cxf.jaxrs.JAXRSBindingFactory;
  import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
  import org.apache.cxf.jaxrs.lifecycle.ResourceProvider;
  import org.apache.cxf.jaxrs.lifecycle.SingletonResourceProvider;

  public class TestCLI {
    public static final int DEFAULT_PORT = 9998;
    public static final String DEFAULT_HOST = "localhost";
    
    @Path("/test")
    public static class TestResource {
        private String msg;
        public TestResource(String msg) {
            this.msg = msg;
        }
        public TestResource() {
            this("I was re-created, not a singleton");
        }
        @GET
        @Produces(MediaType.TEXT_PLAIN)
        public Response getStr() {
            return Response.ok(msg, MediaType.TEXT_PLAIN_TYPE).build();
        }
    }
    
    public static void main(String[] args) throws Exception {
        JAXRSServerFactoryBean sf = new JAXRSServerFactoryBean();
        sf.setResourceClasses(TestResource.class);

        List<resourceprovider> rProviders = new ArrayList<resourceprovider>();
        rProviders.add(new SingletonResourceProvider(new TestResource("I am a singleton!")));
        sf.setResourceProviders(rProviders);
        
        sf.setAddress("http://localhost:8765/");
        BindingFactoryManager manager = sf.getBus().getExtension(BindingFactoryManager.class);
        JAXRSBindingFactory factory = new JAXRSBindingFactory();
        factory.setBus(sf.getBus());
        manager.registerBindingFactory(JAXRSBindingFactory.JAXRS_BINDING_ID, factory);
        sf.create();
    }
  }</pre><br /><br />When run, we see what we hoped for!<br /><br /><pre>$ wget -q -O - http://localhost:8765/test
  I am a singleton!
  $</pre>


guid: https://gagravarr.livejournal.com/146974.html
url: https://gagravarr.livejournal.com/146974.html
title: GOTO Conference Aarhus - My Notes
published: 2013-10-08 14:03:35
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.324199
updated_at: 2019-10-27 00:00:11.123325
summary:
  Last week, I was at the GOTO Aarhus conference. My general overview of it is available <a href="http://gagravarr.livejournal.com/146703.html" rel="nofollow">here</a>. For my much less exciting notes, read on...<br /><br />Now, before I go onto my notes, a small apology. These are just basic notes and pointers on the things I either wanted to read up on later, or things I thought were important. They're not a complete summary of a session. GOTO had a lot of students there helping out, and some of them did an amazing job at note taking and summarising<br /><br /><a href="http://gotocon.com/aarhus-2013/presentation/There%20and%20Back%20Again%20-%20Software%20Security%20in%20the%2021st%20Century" rel="nofollow">There and Back Again - Software Security in the 21st Century</a> - Brian Chess<ul><li>To error is human</li><br /><li>#1 education</li><br /><li>#2 code review</li></ul><br /><a href="http://gotocon.com/aarhus-2013/presentation/Why%20Agile%20doesn%27t%20scale,%20and%20what%20you%20can%20do%20about%20it" rel="nofollow">Why Agile doesn't scale, and what you can do about it</a> - Dan North<ul><li>Local optimisations don't roll up</li><br /><li>Agile doesn't have an opinion</li><br /><li>Book Suggestion: Agile adoption patterns - Richard durnall</li><br /><li>Book Suggestion: The leprechauns of software engineering - ebook</li><br /><li>Book Suggestion: Beyond budgeting</li><br /><li>Contextual consistency</li><br /><li>Scaling is more than just small things bigger</li><br /><li>Guiding principles, strong leadership</li><br /><li>Crossing the chasm of credibility is hard</li></ul><br /><a href="http://gotocon.com/aarhus-2013/presentation/Why%20code%20in%20Node.js%20often%20gets%20rejected%20by%20SoundCloud" rel="nofollow">Why code in Node.js often gets rejected by SoundCloud</a><br />Inspired by this: I need to stop just reading little bits on <a href="http://www.scala-lang.org/" rel="nofollow">Scala</a> and <a href="http://clojure.org/" rel="nofollow">Closure</a>! Need to try some things in them<br /><br /><a href="http://en.wikipedia.org/wiki/Conway%27s_law" rel="nofollow">Connway's Law</a> - <i>organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations</i><br /><br />Something on data vis stuff (I forget which, sorry!)<ul><li><a href="http://piwik.org/" rel="nofollow">Piwik</a></li><br /><li><a href="https://github.com/etsy/statsd/" rel="nofollow">StatsD</a></li><br /><li><a href="http://graphite.wikidot.com/" rel="nofollow">Graphite</a></li><br /><li><a href="http://blog.pkhamre.com/2012/07/24/understanding-statsd-and-graphite/" rel="nofollow">StatsD + Graphite</a></li></ul><br />Data description language (dsl)<br />Then from that to neo4j or similar to actually query<br />Allows for easy changes to db format<br />Closure macros for DSLs<br />Small team wins over big team when lots of knowledge needs to be shared<br />Ola Bini / @olabini<br /><br />Jazz.net/DevOps<br />"Mocking without hangover"<br />"Hiding selenium in ui tests"<br /><br />How to think about parallel programming - not!<br />~= an introduction to functional programming<br />Tail calls<br /><br />javap - bytecode printer<br />Non capturing lambdas faster than inner classes<br />Capturing lambdas same as inner classes, but may improve later!<br /><br />GitHub/crista/exercises-in-programming-style<br />Code golf<br />Kingdom of Nouns<br /><br />Scott Murray - d3 tutorials for non programmers<br />D3 academic paper<br /><br />Risk Management<br />Architecture covers hard parts, critical parts, prototype then fix <br />Executable requirements and feedback<br />If you can't identify a consumer for it, you don't need to deliver it<br /><br />Flowcon<br />Put it in the mission statement<br />Consider social justice issues<br />Start with program committee - needs to be diverse from there on down<br />Doesn't mean lowering the bar! Plenty of top notch speakers of all kinds<br />Women tend to say no more - tend to be more busy<br />Need to do outreach, work hard to get women to submit<br />CFP alone not enough to increase diversity<br />Speakers isn't a hard problem, though other parts are<br /><br />Scala and Play<br />Syntax, naming, versioning and white space are fertile ground for bike shedding<br />"Marketing is a moral imperative"<br /><br />"Out of the tar pit" mark and Mosley<br />State spoils testing. We test it anyway, as what else can we do?<br />"Generative testing" <br />Oop - less helpful when constraints must be enforced across objects<br />Immutable state instead of mutable state makes reasoning easier<br />Complexity through control still exists in FP<br />"Java repl"<br />"Functional reactive programming" - functional for UIs<br /><br />"5 second rule" for presentation slides<br /><br />5k / 10k art - tiny sized web apps and mobile<br />Pursue simplicity - avoid cognative load<br /><br />Discussion with <a href="http://gotocon.com/aarhus-2013/speaker/Trisha+Gee" rel="nofollow">Trisha Gee</a><br /><a href="http://code.google.com/p/spock/" rel="nofollow">Spock</a> - groovy, bdd test tool<br /><br />Overall - I need to give more talks!


guid: https://gagravarr.livejournal.com/146703.html
url: https://gagravarr.livejournal.com/146703.html
title: GotoCon Aarhus 2013 round-up
published: 2013-10-08 13:53:25
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.327034
updated_at: 2019-10-27 00:00:11.125664
summary:
  Last week, I was lucky enough to attend <a href="http://gotocon.com/aarhus-2013/" rel="nofollow">GOTO Aarhus 2013</a>, the original and biggest of the GOTO Conferences. In short, it was excellent!<br /><br />During the course of the conference, I took notes of interesting things (with varying levels of detail and comprehensiveness...), which I've put in <a href="http://gagravarr.livejournal.com/146974.html" rel="nofollow">another post</a>. I started off with one post, but it got a bit big, so I've opted instead to split them up. This one's all about the conference itself. But, before we get to that...<br /><br />My first involvement with GOTO was a couple of years ago, when I attended + spoke at the first GOTO Amsterdam. I largely enjoyed that one, but it wasn't quite what I'd expected, both in terms of the audience and the things covered. Unfortunately, this expectations gap meant that my talk there wasn't quite the right one... While everyone who came really liked it and learnt a lot from it, sadly it didn't get the numbers it might have had. This time round, I knew what to expect, and knew the kind of audiences and speakers there, so was better prepared to make the most out of it! <br /><br />For those who've not made it to a GOTO Conference before, they cover a bit of a mixture of topics, but one that seems to work well together. You seem to learn what you need today, what you might need tomorrow, along with a fun bunch of people to speak to between sessions to discuss it all with. Everything is related to Software Development, but in a broad sense, not just pure tech in isolation. So, along with the talks on technology stuff, you also have project management, architecture, careers, technology stacks, general approaches and more. Some of the best talks weren't the most immediately obvious ones, so it's certainly worth trying out some tracks outside of your normal role.<br /><br />GOTO Aarhus ran with 6 tracks, so it's not a small event. I'd say that at least half the time, I wanted to attend 3 of the talks, so you're not short on interesting things to attend! Possibly my record was one slot where I quite fancied going to 5 out of the 6 talks on offer! The event took place in the Aarhus Musikhuset, the main concert hall, so for many of the sessions you were sat in nice tiered seating, in a medium or big room. It took me a little while to work out which room was where, but then I discovered a very nifty locator in the mobile app. That worked much better than I expected, and showed you where in the building you were, where the room was, and what direction to head in. It was the future arrived! One or two more signs might've been helpful though, as you do look a little bit silly navigating your way round a concert hall using your ipad...<br /><br />The conference ran over 3 days, and despite fitting a lot in, didn't feel too busy. Partly I think this was due to the 20 minute break between every (50 minute) session, which gave a chance to pause, have a drink, snack on something and re-charge. It saved a mad dash between sessions, and certainly helped me keep going to the end! The sponsors seemed to like it too, as there were more people about for them to chat to. The break area was sunny, and you could walk outside for fresh air, which made a big difference to energy levels vs being buried in some deep basement for 3 days!<br /><br />Despite there being a large number of people there, everyone seemed very friendly, I had lots of interesting chats with speakers and attendees alike. Possibly I should've got a tshirt saying "I'm not Danish", as almost everyone tried speaking Danish to me first, but everyone was happy and able to switch to English! As well as the danes, there were people from all over Scandinavia there, along with a smattering of other European countries, so there was a good mix.<br /><br />GOTO had a lot of students there helping out, and some of them did an amazing job at note taking and summarising! Check <a href="https://twitter.com/search?q=%23gotoaar" rel="nofollow">#gotoaar</a> for some, but the one that most caught my eye was <a href="http://programmingandthecity.wordpress.com/" rel="nofollow">Mathilde Hoeg</a>, who did some amazing summaries like these: <a href="http://programmingandthecity.wordpress.com/2013/09/30/whats-dan-north-wearing-today/" rel="nofollow"><img src="https://imgprx.livejournal.net/fc67c0532b171c0f15c7973d1ac40c628013fa7e/hzhOxq9rkscOpP-phDeCJG_aU1OcAf72Tjf96JJvR6z9pv0GJRSgjxmFTomS2WkqKZIqAXVaqcC8OAp3va50tZOje3EH51Rk5QWG6gIv9ViTNBToWz3iIXuw7Rd3F0BR9SK10NmsD6UFtQikYmWp8A"></a><a href="http://programmingandthecity.wordpress.com/2013/10/07/theres-turtles-all-the-way-down-musings-on-good-user-experience/" rel="nofollow"><img src="https://imgprx.livejournal.net/77a746af65b88e18ae8099230b351a3ea39b9587/hzhOxq9rkscOpP-phDeCJG_aU1OcAf72Tjf96JJvR6z9pv0GJRSgjxmFTomS2WkqKZIqAXVaqcC8OAp3va50tVBNuce1qBxCdWfVk_IURKaQbh2TppAVn3-VfR-5Wk78gV-_9couE3qXvFFkY-tOmg"></a><a href="http://programmingandthecity.wordpress.com/2013/10/06/reminder-do-things-that-scare-you/" rel="nofollow"><img src="https://imgprx.livejournal.net/6bb56f9ff6bdec00206e88d768415e16acbbea2b/hzhOxq9rkscOpP-phDeCJG_aU1OcAf72Tjf96JJvR6z9pv0GJRSgjxmFTomS2WkqKZIqAXVaqcC8OAp3va50tVBNuce1qBxCdWfVk_IURKZwQxmKoidBdAEgNUvunB9UEt7aGmiz7vFzdjdUfDIL0w"></a><br /><br /><br />For my much less exciting notes than those... Move onto <a href="http://gagravarr.livejournal.com/146974.html" rel="nofollow">part 2 here</a>.<br /><br />If you missed Aarhus, as well as next year's one, there's still time to make it to <a href="http://gotocon.com/berlin-2013/" rel="nofollow">GOTO Berlin</a> in a couple of weeks, which looks to have an excellent line-up too! See <a href="http://gotocon.com/berlin-2013/" rel="nofollow">here for details and signup</a>.


guid: https://gagravarr.livejournal.com/146494.html
url: https://gagravarr.livejournal.com/146494.html
title: Why might you want Alfresco's Cloud-Style Multi Tenancy on-premise?
published: 2013-09-18 23:24:59
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.329920
updated_at: 2019-10-27 00:00:11.128001
summary:
  In <a href="http://gagravarr.livejournal.com/146003.html" rel="nofollow">part 1</a> we looked at what Alfresco Multi-Tenancy (MT) is, and why you might want to use it. In <a href="http://gagravarr.livejournal.com/146320.html" rel="nofollow">part 2</a>, we looked at how collaboration makes a strict one-tenant-per-user model no longer work well, and saw how the Alfresco Cloud has solved this to permit efficient and effective collaboration.<br /><br /><br />Let's have a quick review of the situation, to remind ourselves how it works:<br /><img src="https://imgprx.livejournal.net/ee6e80332220562657b4175c5841f3b58a726d41/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhvxN9jWaJHbJjBBVAGO3L345BNcGsoRpZ31MLPnkzIkI" /><br />Each team / department / company / etc has full and complete control and administration over their tenant (as identified by their email domain). When they need to collaborate with someone from outside of their tenant, they invite them in for that one site. The person invited in retains full control of their own tenant, but gets the limited rights given to them in the sites in invited tenants. They can work with others on documents in those sites, collaborate, discuss, comment, but not pierce the veil of the security or protection<br /><br /><br />Now, that sort of thing can be done without the need of Multi-Tenancy. Alfresco is heavily customisable and extensible, and allows you to override or replace all sorts of parts. If you know what you're doing, it's perfectly possible to achieve the same thing in a single tenant setup. It's not a quick job though, and you'll need to do things like:<ul><li>Define your own way of recording which "virtual tenants" people can access, and what belongs to what</li><li>Override various People searches (services and webscripts) to check/filter by that</li><li>Override the Category/Tagging services, along with various webscripts, to give each "virtual tenant" their own tag cloud</li><li>Override the default permissions, to be more restrictive</li><li>And a few other bits too...</li></ul><br />OK, so it can be done, what's the problem? Well, a few things, including:<ul><li>The initial work takes a while (reviewing then coding), which means less other work</li><li>Upgrades are slow, as you often can't cleanly override the bits you want to, so you often have to port your code forward + re-review</li><li>Testing and Validating takes longer, as there are more "moving parts" (so to speak)</li><li>You have to write custom reports to list who has access to what, so that access reviews can be done, rather than just listing tenant users</li><li>You have to do the work, rather than getting it included in your Enterprise subscription!</li></ul><br /><br />I know for a fact that at least one company would prefer to use Cloud-Style MT on-premise, rather than achieving the same thing with permissions + customisations (mine!), but I strongly suspect we're not the only ones. In fact, I'd rather suspect that there are huge numbers of potential customers out there who stuck Alfresco off their "potential systems" list early on because of a lack of such support "out of the box".<br /><br />Let us consider a typical collaboration environment, which could easily be different departments within a large organisation, or small companies working together, or a mixture of big company departments and their outsourced providers helping:<br /><img src="https://imgprx.livejournal.net/ed83fa36f78865b04d13a98b859e3e376d5c7c1d/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhM90LKfz3zy1d-eue4_cz8SLvL5Ctbv2tDkd05cdQEuw" /><br />In this situation, Users 1 and 2 work for the main company, but invite in two different smaller agencies to help them out with different things. We could even imagine a case where Users 3 and 4 collaborate together, without the others!<br /><br />How about some simple cases of this setup?<ul><li>Two different departments use the same install of Alfresco, but want complete control of their own spaces. They need to use the same marketing agency, and sometimes both departments and the marketing agency need to work together on an advert</li><li>Three different departments each have their own intranet, but also all work on the internet site. They need to collaborate with external designers and web agencies on sites relating to their joint external website, as well as each department with just the agency for their respective intranets</li><li>Operations and Fixed Fee both have their own areas, but they sometimes need to work with Legal and Purchasing, who both have their own areas too</li></ul><br />I'm sure I could go on with ideas involving federated companies, or other cases with strong inter-departmental divisions but an increasing need to collaborate across them on certain things. Now, let us consider a few cases where there are many companies, who might all want to share one collaboration system, but currently Alfresco Enterprise won't fit their needs because of a lack of the Cloud Style Multi-Tenancy.<br /><br /><b>Business Incubator</b>. Needs to be common sites to submit quarterly reports, as well as for general discussions about the space. Some companies will collaborate with each other to win funding, so need shared sites. In other cases, companies are secretive about their partners, and don't want to leak information about who they are working with. Not only do sites need to be private, but they need independent Tags and User lists to prevent one company being able to find out things on another one by checking tags or doing people searches.<br /><br /><b>Pharma</b>. Pharma A have WonderDrug A, Pharma B have WonderDrug B, but both are also working together with Pharma C on WonderDrug Z. All 3 use the same DM CRO, A and C use the same PV provider, B and C the same MW provider while A do their MW in-house. On WonderDrug Z the three pharma companies want to share their data, but A doesn't want B or C to know who they use for their in-house MW to prevent poaching, no-one wants anyone else to know their tagging schemes, and the DM CRO shouldn't have access to everything even if they work with everyone. (You can do this setup with permissions, but as previously mentioned it has notable upgrade costs)<br /><br /><b>Banking</b>. The Legal and Compliance teams need strong separation between their areas and those of the rest of the business. IBD and Equities need to be able to prove that the Chinese Wall remains unbroken. All of them need to collaborate on the Charitable Works project, and some of all of them need to offer input on the re-tendering of the catering contract. Passwords need to change often, and the idea of having multiple logins won't go down well.<br /><br />I'm sure there are a number of other cases too, where the need for strict separation between Tenants is required most of the time, but controlled access between them to collaborate is needed in others. Those were just the three that first sprang to mind. In all 3 cases, unless you have a very deep knowledge of Alfresco, you would do an initial evaluation of the product then reject it as unsuitable, since the feature isn't there out of the box. How many lost deals are there out there from this feature not being present? How many possible deals never got into an Alfresco prospects list, because the lack of collaborative MT was spotted before anyone talked to sales or a partner? How many times have organisations really wanted to open-ness and community that comes with Alfresco, but had to go for a different solution because of this one missing feature? I'm going to say the answer to all of those is "rather a lot"<br /><br />The crazy thing about these missed sales, these missed opportunities, these skilled Alfresco customers having to roll the solution themselves, is it's not some impossible new feature that would take man-years for Alfresco to add in. This is a feature that Alfresco have already written, tested, documented, and supported for getting on for 2 years! This wouldn't be some new area to develop, this is something Alfresco already support, and effectively support twice!<br /><br />How much work would be involved in migrating this existing feature over from Cloud to Enterprise? Well, some parts of it are already done. As mentioned in <a href="http://ecmarchitect.com/archives/2013/09/15/3554" rel="nofollow">this 4.2 blog post</a>, the new Public API on 4.2 uses cloud-style URL patterns, eg <quote><a href='http://localhost:8080/alfresco/api/-default-/public/cmis/versions/1.1/atom' rel='nofollow'>http://localhost:8080/alfresco/api/-default-/public/cmis/versions/1.1/atom</a></quote><br />The <i>-default-</i> in the URL is an explicit tenant selection, which is the first step in the move away from the old-style implicit tenant selection. What else? Well, there would be a lot of testing involved - Alfresco Enterprise has a much wider set of supported stacks/platforms than Cloud. Coding/Merging wise, it wouldn't actually be too bad, certainly a lot easier than some of the other Cloud -&gt; Enterprise merges that have happened of late. Documentation would need updating too, and probably some compatibility layers that maintain implicit tenant selection on old-style URL patterns.<br /><br />All in all thought, it's not something that could be knocked out in a couple of i-days, but it's not a huge undertaking, much smaller than others recently. It would reduce support and maintenance costs for Alfresco, by reducing from 2 to 1 the number of MT platforms supported. It would greatly reduce the support/maintenance costs for those currently forced to implement their own system. Perhaps much more importantly, a huge number of new sales opportunities would arise, and arguably a number of whole new sectors would suddenly be able to consider using Alfresco. I may be a little biased, but I'd say that's well worth going for!


guid: https://gagravarr.livejournal.com/146320.html
url: https://gagravarr.livejournal.com/146320.html
title: Cloud-Style Alfresco Multi Tenancy
published: 2013-09-16 00:21:34
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.332915
updated_at: 2019-10-27 00:00:11.130496
summary:
  In the <a href="http://gagravarr.livejournal.com/146003.html" rel="nofollow">first part</a> of this Alfresco Multi-Tenancy series, I tried to cover <a href="http://gagravarr.livejournal.com/146003.html" rel="nofollow">what MT is, and how Alfresco implements it</a>, concentrating on the On-Premise / One / Community / Enterprise versions of Alfresco. In this part, I'll try to explain a little of what changed with the Cloud version of Alfresco.<br /><br /><hr /><br /><br />Let us consider again our situation from the end of part 1. Each team / department / company / etc has their own tenant within our single managed HA install of Alfresco. Each one has their own set of users, and are fully isolated from each other. Our setup looks a little like:<br /><img src="https://imgprx.livejournal.net/b9588e15835cd23c5be440404f02a055a01e29c0/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjh8N13skU5lUuKYC43C56Pow" /><br /><br />At this point, <i>User 1</i> asks <i>User 4</i> to help out with a document in <i>Repository 1</i>. Maybe it's a cross-team project, and they need to collaborate on the presentation to give to management. Maybe User 1 has been working a press release, and needs to get User 4 in from the PR team to sign it off. Maybe Users 1 and 2 are a company, and they've hired in freelancer User 4 to help with some work. There are many reasons, but what to do?<br /><br />Well, one option is to email the document over, or copy it on a USB stick or something. This wouldn't be much fun, creates a tracking nightmare, and hey - didn't you buy this Content Management thing to avoid that sort of nasty mess in the first place? Clearly this isn't what we want...<br /><br />Hmm, well, Alfresco supports <a href="http://wiki.alfresco.com/wiki/Replication_Service" rel="nofollow">Replication</a> and <a href="https://wiki.alfresco.com/wiki/Transfer_Service" rel="nofollow">Transfers</a>, could we maybe use that? Let's see how that would work:<br /><img src="https://imgprx.livejournal.net/4addf064531d2eb92e0599175c0cfd7e24533629/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhs8jaBGRvKn6P0f4p5wmG5Q" /><br /><br />The problem here is that we still have two copies. That means two lots of hassle, constant issues with working out which version to use, who has the latest changes, what filename to use, how to keep in sync, all that sort of thing. In short, you've recreated most of the problems that drove you to a Content Repository in the first place! <br /><br />Well, what instead if we gave <i>User 4</i> an account on our <i>Repository 1</i> tenant? Wouldn't that work?<br /><img src="https://imgprx.livejournal.net/982df97056e128b5165294da5dc614bcad8d077a/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhsQrTaOqecgdsEcU11wwi0Q" /><br /><br />Now, our first issue is how to let them log in. Normally, Alfresco uses the domain part of the username to identify the tenant, so we can't just let <i>jane.smith@pr.agency.example.com</i> log into <i>@company.example.org</i> as the domain is wrong. This means we either have to use non-emails (eg <i>jane@pr.agency</i> and <i>jim@company</i>), or we have to play with a load of settings, or both. Next up, what about poor Jane and where to go? If she logs into his normal account, she can't see this Press Release she's supposed to work on. Then she remembers and logs in with a different, unusual account, and accesses the Company tenant with the thing to work on. In here, she can't see her normal content, can't see her normal changes, can't easily access her stock content in her site, and is generally confused about where she is (it looks the same but isn't!). The upshot is user confusion, annoyance, people rarely logging into some systems due to the faff, and a lot of admin overhead (creating all those duplicate accounts). This just doesn't fit how people work today.<br /><br /><hr /><br /><br />At this point in our story - enter Alfresco Cloud! A triumph of vision over a lot of things... The Alfresco Cloud version is built on top of the regular Alfresco Enterprise codebase (ish...), with customisations to both the underlying services, and the UI. Part of what it changes / introduces is a brand new model of Multi-Tenancy.<br /><br />When someone talks about Alfresco Cloud-Style Multi-Tenancy, or Cloud MT, or something like that, there's one key bit they're probably referring to. Well, two related bits. But first, two screenshots:<br /><img src="https://imgprx.livejournal.net/29b9ecf4e182aefb3c066f2fb9f09a519f552904/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhuXfiAUmqHwck4pzHayajPg" /><img src="https://imgprx.livejournal.net/6a20a3f8a43cce0306bf79ef56db7b16313f6ee9/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhHBFccy0uqKffJnc7bj9IFQ" /><br /><br />Leaving aside for a moment that one is using a largely-stock Alfresco 4.1 theme, and the other is using a the very snazy new <a href="http://docs.alfresco.com/4.2/index.jsp" rel="nofollow">4.2</a> one, there are two things to see. Firstly, the URL. In the local version, we have a common URL, with just a user's email in the final part of the URL for their dashboard. For the cloud one, we see the domain (tenant) included in the URL. Secondly, on the local one, there's no way to pick a tenant. On the cloud one, we see our tenant listed at the top. Clicking that...<br /><img src="https://imgprx.livejournal.net/18dec3cef4e5e10dc1a95defc2b012ac5a299206/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhtEGVyIlvzmKDrzMbqSYmYwKe_LlFK5ED_vGlBe7x0FM" /><br />We then have the option to access additional (secondary) tenants to which we have been invited to. Not all tenants on the system, mind you, just the ones to which access has been granted. Also, not all resources within those secondary tenants, only sites to which we have been invited. (In your home tenant, you can access all public sites by default, in secondary tenants you can only access sites to which you have explicitly been granted access, and no others)<br /><br />The practical upshot of all this - modern style collaboration can work, and can work well! We can largely maintain the required "silos" between data, each group (tenant) can manage things for their own team / department / company / etc, but they can invite specific people to collaborate with them on one specific thing (and only that). The invited user can access that small subset using their existing login, without needing to do anything special, can see what's going on, can work on that document alongside their own ones, and it's all good!<br /><br /><hr /><br /><br />Before we end this post, a little bit on the technical side. (Note - I didn't write any of the Cloud-Style MT code, but I have been to talks on it)<br /><br />What exactly is involved in the cloud-style multi-tenancy, from a coding side? I'd say the main things are:<ul><li>Ability to specify the tenant of interest independent of the login (so you can pick an invited tenant)</li><br /><li>Store/check/retrieve/manage additional (secondary) tenants for a given user</li><br /><li>Different permissions when in an invited (secondary) tenant to your home one (can't see non-invited sites, search users etc)</li><br /><li>Cross-tenant invitations</li><br /><li>A whole bunch of tenant creation stuff, self sign-up, tenant management, billing, metrics etc (required to run the cloud scale service)</li></ul><br />I'll largely ignore the last one, as that's specific to running The Alfresco Cloud, as opposed to running something like the Alfresco Cloud. (This part is pretty much independent of the rest, and while important for the Alfresco Cloud, has no interest to anyone else.)<br /><br />So, what's the scope of these changes? The first one was actually the one that took much of the work. The WebScript framework on both end needed changes, these had to be fed down to the Tenant Service, and the URLs changed too. Since people can be in multiple tenants, you can't just auto-detect their active tenant from their username. Instead, the URLs hold the current tenant of interest, and change based on the tenant you're in. <br /><br />An extended Tenant Service was also needed, to hold the details of the additional / secondary / etc tenants for a user, allow querying of them, management of them etc. This part is quite small. Then, you need some logic that pulls out the requested tenant from the URL / webscripts, checks it's allowed for the user, and sets of the context for the request. You also need to change some permissions things, as someone who's only a guest in a tenant (accessing it as a secondary tenant) shouldn't be able to do everything as in their own tenant. We also need to have the support for inviting someone to a different tenant, so you can invite <i>User 4</i> to join a site in <i>Repository 1</i> with the tenant change happening inside the invite transaction stuff. Oh, and you need to maintain this whole set of Multi-Tenancy code alongside the existing On-Premise single MT code, bug fix and enhance both etc...<br /><br /><br />All of this is hard, tricky, and took a lot of work! However, it's also all done... Alfresco have tackled the problem, and come out victorious! Yey for the Alfresco Engineers! <br /><br />In the <a href="http://gagravarr.livejournal.com/146494.html" rel="nofollow">final part</a>, we'll look at why other people might want all of this lovely thing not in the Cloud


guid: https://gagravarr.livejournal.com/146003.html
url: https://gagravarr.livejournal.com/146003.html
title: Alfresco Multi Tenancy - An Introduction
published: 2013-09-15 22:40:42
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.335815
updated_at: 2019-10-27 00:00:11.132831
summary:
  Alfresco has supported Multi Tenancy for a number of years now. A little over a year ago, Alfresco added support to the Cloud version which expanded the MT support, but thus far hasn't brought those improvements back to the On-Premise (One) version. In this series of blog posts, I'll try to explain what the traditional Alfresco MT does (and doesn't) do, what new things the Cloud MT handles, and finally why I feel everyone would benefit from from the Cloud MT on-premise.<br /><br /><br />Right, so, Multi Tenancy, what is it? The <a href="http://docs.alfresco.com/4.1/index.jsp?topic=%2Fcom.alfresco.enterprise.doc%2Fconcepts%2Fmt-intro.html" rel="nofollow">Alfresco Enterprise Documentation covers part</a>, and the <a href="http://wiki.alfresco.com/wiki/Multi-Tenancy" rel="nofollow">Wiki has a lot of details</a>. Neither of them seem to cover much of the basics, which can cause some confusion (it certainly has when I've tried to explain it to others). I'll try to give a quick intro, feel free to skip the next few paragraphs if it's all known to you already!<br /><br /><hr /><br /><br />With a simple content store system, you typically have a single installation, which hosts a single set of content. One set of users, one set of files, everything in together. Works just fine for your team / department / company / etc. Something like:<br /><img src="https://imgprx.livejournal.net/3da7788b525b2292819ccb389b06290910d6bea9/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIvjQa-nwtIl7EiwYFncHqzECvMfVEoTCGnfFg6zQBgyBQ" /><br /><br />Your new system is great, and everyone wants in! You start adding accounts for a couple of people from another team / department / etc, so they can use your system. You create them their own sites to collaborate in, but it's still your system, and you retain full control. After a while, they decide they want their own system, which they themselves have full control over. To do this, they install their own brand new copy of the system, set it up, create their own set of users and sites, grant permissions, tell people the new URL etc. You now have two independent installations, each holding their own independent repository stores.<br /><img src="https://imgprx.livejournal.net/05a209d93132325d2aab6b5f4ed5f4777188cf82/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIvjQa-nwtIl7EiwYFncHqzEDuP4lTepxbG59KIhcNaMpQ" /><br /><br />Having two independent systems may seem great to start with, as each team / department / company etc has their own area. They can do whatever they want, they have full control over their own things, they don't have to worry about checking permissions to prevent someone in one team having access to another team's things, there's a nice big gap between them. Then, you start needing to upgrade, to apply customisations, to develop new in-house features. Suddenly, there's two different systems that both need to be updated, both need to be tested and validated. Two different systems need to be maintained, two different systems need two sets of resources, twice the monitoring, more IT work, more co-ordination. Trying to add some more teams / departments / companies / etc in is going to make this even worse. Federation / common interfaces / <a href="http://chemistry.apache.org/" rel="nofollow">CMIS</a> can help a little with the common access, but the work grows....<br /><br />At this point, some bright spark starts asking for Multi Tenancy. Instead of having one complete install for each individual group, why not have them share a common one, but have the system pretend to them that each one is independent? From a maintenance and resourcing perspective, you pretty much just have the one system to look after. From a user's perspective, they each have their own totally independent content / users / permissions / admins / etc, as before, but it's simpler and cheaper to run.<br /><img src="https://imgprx.livejournal.net/2486ac17eb15ddc7fa4719c108e3b7ee0e40b2ec/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIvQK9xAFbrppPYLDaEECexJB5iCUuQ1lYwqWLefsVvLQf1_bq1Hwv7mcFIg4FE0w30" /><br /><br />In this model, there's generally a "super-admin" somewhere who can create or delete new tenants within the common (multi-tenant) system, and who can reset admin passwords within each tenant. Otherwise, each sub-part of the system (each tenant) is independent of each other, doing their own things in blissful ignorance and fully protected from what's going on in the other tenants. The cost of hosting / maintaining is shared, so everyone wins on cost. It isn't quite a fully independent system, as things have to be the same until you have identified which tenant you belong to (perhaps by a URL, or more commonly based on your username on login), but for old-style working this was fine. <br /><br />This sort of multi-tenant model was very popular with hosting companies, and large companies with strong divisions. Each team / department / company had their own sub-area that they could work in, that was "theirs" which they had full control of, but there's only one installation. For big setups, you could also get extra resilience and availability, since rather than 3 teams each having their own one server, they could club together and get a HA cluster with 3 machines in it that they all shared.<br /><br /><hr /><br /><br />As previously mentioned, Alfresco has supported this style of Multi-Tenancy for quite some time. (I believe there was one big OEM deal that funded a lot of the initial work on it, but there had been plenty of call for it from the field before this). It used to be a little fiddly to set up, but as of 4.0.2 it is pretty simple to get going. There's information <a href="http://wiki.alfresco.com/wiki/Multi-Tenancy" rel="nofollow">on the wiki</a> and <a href="http://docs.alfresco.com/4.1/index.jsp?topic=%2Fcom.alfresco.enterprise.doc%2Fconcepts%2Fmt-intro.html" rel="nofollow">in the enterprise docs</a>. Let's have a go at enabling it for our Enterprise 4.1 install to see!<br /><br />First, log into the Repo (Explorer) interface as an admin. Next, go to the tenant admin console at <a href="http://localhost:8080/alfresco/faces/jsp/admin/tenantadmin-console.jsp" rel="nofollow">http://localhost:8080/alfresco/faces/jsp/admin/tenantadmin-console.jsp</a> (assuming a default install). You should see a very simple Explorer console / shell thingy to interact with:<br /><img src="https://imgprx.livejournal.net/7a3de78bab94613d20545590329e3736186d6a30/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIs3SVUZD5GVNZ9Et5sV6-ip0xiQdH3SadSlSj08OAw6n9BuMAVH-cLRFzMBbm1mbqE" /><br /><br />When we run <blockquote>show tenants</blockquote> we see that we have no tenants by default. Let's create one! Type in <blockquote>create example.org adminpass</blockquote> and wait while it churns away. (Your first tenant is a little slow, as a load of setup happens, but new ones tend to be fast). Next, head over to share. Instead of logging in as <i>admin</i> instead use <i>admin@example.org</i> and a password of <i>adminpass</i> (we set the domain and admin password at creation time). When we log in, we see an empty repository, without any of the sites we previously had<br /><img src="https://imgprx.livejournal.net/00fa7dbbc878f4d40db8ec678643197a36dfe021/hzhOxq9rkscOpP-phDeCJAw0buod8sE2IIm3Ny4mXIt_sui1AFci7JAF0lHQDnjhHkNDZ2UXaukKcew6_buahQ" /><br /><br />Create a new site here, and upload a few files. Then, log out, and log in with the normal admin account. Search for the site you created, but you won't find it. Alfresco keeps the different tenants independent, including the default tenant and the one you just created.<br /><br /><hr /><br /><br />Finally, before we look at the problems that modern styles of working pose to this, let's do a quick round-up of what the Alfresco On-Premise / Enterprise / One implementation of Multi-Tenacy does and doesn't do.<br /><br />You can use Multi-Tenancy with the following:<ul><li>Alfresco Explorer (JSF-based web client)<br /><li>Alfresco Share (since 3.2)<br /><li>WebDAV<br /><li>FTP<br /><li>Web Scripts<br /> <li>CMIS (since 3.2)<br /><li>tenant 'guest' authentication is allowed, but must be within the context of a tenant (such as 'guest@acme')</li></ul><br />Unfortunately though, the following are not supported/implemented/tested in a multi-tenant enterprise environment:<ul><li>    CIFS<br /><li>    WCM<br /><li>    RM<br /><li>    Portlets<br /><li>    LDAP, NTLM and authentication methods other than "alfresco"<br /><li>    Inbound Email<br /><li>    Content Replication<br /><li>    IMAP<br /><li>    SPP / VTI (SharePoint Protocol)<br /><li>    reloadable Share config (if using dynamic models)<br /><li>    GoogleDocs integration</li></ul><br />In <a href="http://gagravarr.livejournal.com/146320.html" rel="nofollow">Part 2</a>, we look at what the Alfresco Cloud offers in place of this.


guid: https://gagravarr.livejournal.com/145679.html
url: https://gagravarr.livejournal.com/145679.html
title: Enabling replication with OpenLDAP 2.4 on RHEL 6 with cn=config (olc....)
published: 2013-08-14 17:41:47
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.338714
updated_at: 2019-10-27 00:00:11.135183
summary:
  The last time I setup <a href="http://www.openldap.org/doc/admin24/replication.html#Syncrepl" rel="nofollow">replication on OpenLDAP</a>, it was on Debian with a single slapd.conf file. This time both machines were running Redhat Enterprise RHEL 6, using the <a href="http://www.zytrax.com/books/ldap/ch6/slapd-config.html" rel="nofollow">OLC / cn=config</a> style of configuration, with ldif files. I found quite a lot of information on how to setup <a href="http://www.openldap.org/doc/admin24/replication.html#Syncrepl" rel="nofollow">replication</a>, quite a lot on using ldap modify to change cn=config, but not a lot on using cn=config ldif files for replication. Having worked out which bits to follow and which to ignore, I thought I'd document it for others to hopefully learn from!<br /><br />This guide assumes you're familiar with OpenLDAP replication, want to use Syncrepl, are using RHEL 6 for your master and your slave(s), and have already got your directory working on the master, set permissions etc.<br /><br /><h2>Step 1 - Create a read only replication account</h2><br />Using your favourite tools, create a new account (objectclass = account) in your directory, with a suitable name (eg uid=replication or some such). Set a secure password on it, and make a note of that. Next, edit <i>/etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif</i> and grant read permissions to this user on all attributes. Any it can't see won't get replicated, so tweak your <i>olcAccess</i> statements as needed. A basic setup would look like:<br /><pre>olcAccess: {0}to attrs=userPassword
    by self =xw
    by dn.exact="uid=pwreset,dc=example,dc=org" =xw
    by dn.exact="uid=replicate,dc=example,dc=org" read
    by anonymous auth
    by * none
  olcAccess: {1}to *
    by anonymous auth
    by self write
    by dn.exact="uid=replicate,dc=example,dc=org" read
    by users read
    by * none</pre><br /><h2>Step 2 - Enable the syncprov module</h2><br />On all machines (master and slaves), create a new file <i>/etc/openldap/slapd.d/cn=config/</i> called <i>cn=module{0}.ldif</i> . Into it place:<br /><pre>dn: cn=module{0}
  objectClass: olcModuleList
  cn: module{0}
  olcModulePath: /usr/lib64/openldap
  olcModuleLoad: {0}back_bdb
  olcModuleLoad: {1}syncprov</pre>Note that if you're on a 32 bit system, you'll need <i>/usr/lib/openldap</i> not lib64. This file will trigger the loading of the syncprov module, and the bdb one if needed. If you want to add more modules later for other things, you can either add them to the ordered olcModuleLoad list, or add cn=module{1}.ldif and list them in there<br /><br /><h2>Step 3 - Turn on syncprov for each directory</h2><br />syncprov needs to be enabled for each directory, which in the default config would mean for <i>olcDatabase={2}bdb</i>, and possibly <i>olcDatabase={0}config</i> too. For now, I've opted to enable syncprov for both, but only pull the former, but I may change that in time.<br /><br />Firstly, create two new directories, <i>/etc/openldap/slapd.d/cn=config/olcDatabase={0}config/</i> and <i>/etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb/</i> - these directories should match with a similar .ldif file of the same name, just without the .ldif. Into each one, create a file <i>olcOverlay={0}syncprov.ldif</i> which will hold the syncprov config for each directory. The file should look something like (depending on your chosen settings):<br /><pre>dn: olcOverlay={0}syncprov
  objectClass: olcOverlayConfig
  objectClass: olcSyncProvConfig
  # 
  # Sync Setup for the main LDAP Database
  #
  olcOverlay: {0}syncprov
  # Sync Checkpoints every 20 changes or 1 hour
  olcSpCheckpoint: 20 60
  # Keep a fair number of operations in the log
  olcSpSessionlog: 1000</pre>Restart slapd on the master server, and ensure it starts without error.<br /><br /><h2>Step 4 - Configure the slave(s) to poll the master</h2><br />Finally, on each slave we need to configure the directory to pull from the master. This will use the syncprov module load we setup earlier in step 2, which needs to be done for each server!<br /><br />Edit your database config file, eg <i>/etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif</i> and add to the bottom of it an entry like:<br /><pre>olcSyncrepl: rid=135
    provider="ldaps://ldap-master.example.org:389/"
    type=refreshAndPersist
    retry="60 30 300 +"
    searchbase="dc=example,dc=org"
    bindmethod=simple
    binddn="uid=replicate,dc=example,dc=org"
    credentials=MYsecurePASSWORD</pre>The rid needs to be unique per slave, and needs to be a three digit number, I've found a suitable part of the IP address to be a good option to go for!<br /><br />On each slave, ensure there's an empty directory, then start slapd. Within a short while, a search should then show all the data from the master, and you're good to go!<br /><br />If you hit problems, try running <i>/usr/sbin/slapd -h  ldap:/// ldaps:/// ldapi:/// -u ldap -d 255</i> to start the server in debug mode with logging to the console. The logs can be a little cryptic, but with googling you ought to be able to work out what's wrong and fix!


guid: https://gagravarr.livejournal.com/145561.html
url: https://gagravarr.livejournal.com/145561.html
title: Nick - Authenticating to Alfresco Share using Apache Auth (eg for SSO)
published: 2013-03-13 17:06:36
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.341523
updated_at: 2019-10-27 00:00:11.137527
summary:
  This post carries on from my <a href="http://gagravarr.livejournal.com/145234.html" rel="nofollow">earlier one on Alfresco Repo SSO</a>, and builds on the information on <a href="http://gagravarr.livejournal.com/145144.html" rel="nofollow">how the Alfresco WebScript authentication methods work</a>.<br /><br />At this point, we can hopefully log into the Alfresco repository (Explorer or <code>/alfresco/wcservice</code>) automatically by authenticating to Apache. Our next step, before moving onto SSO in Share, is to also allow header based authentication to Alfresco. We need to do this because of how Share will talk to the Repo - it will know the user's username from Apache, but it won't have their password (and equally won't have their client cert if doing SSL auth). Share needs to be able to tell the repo who you are, and have that trusted. For that, we need to tweak our External Authentication SubSystem.<br /><br />Before doing that, be aware that by turning on header based auth, anyone who can make http requests to the repo could add in that header and impersonate anyone, so take great care! You should do one of use firewall / auth rules it so that only Share can, use something like an Apache module to strip out the header except from Share, or use SSL + client auth + proxyUserName.<br /><br />Currently, our <code>external-authentication.properties</code> file looks like:<pre>    external.authentication.defaultAdministratorUserNames=admin,nick
      external.authentication.enabled=true</pre>To support header based auth, we add two more lines, giving us:<pre>    external.authentication.defaultAdministratorUserNames=admin,nick
      external.authentication.enabled=true

      external.authentication.proxyUserName=
      external.authentication.proxyHeader=X-My-Custom-SSO-Header</pre>If you don't specify a proxyHeader, the default is <code>X-Alfresco-Remote-User</code> (that comes from <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/web-client/config/alfresco/subsystems/Authentication/external/external-filter.properties" rel="nofollow">here</a>).<br /><br />With that in place, from a machine that is allowed to talk to Alfresco with the header, try executing something like:<pre>    $ curl -X GET -L -H "X-My-Custom-SSO-Header: nick" http://localhost:8080/alfresco/ | less</pre>Take a look through the HTML, and check you see something like <code>Logout (nick)</code> and not <code>Login (guest)</code><br /><br /><br />Now, we can turn to Share. There is information on this in <a href="http://wiki.alfresco.com/wiki/Alfresco_Authentication_Subsystems#Alfresco_Share_using_external_SSO" rel="nofollow">the wiki</a> and <a href="http://docs.alfresco.com/4.1/topic/com.alfresco.enterprise.doc/tasks/auth-alfrescoexternal-sso.html" rel="nofollow">the Enterprise docs</a>, but they're a bit short on explanation to understand why you need to do things. <br /><br />First, two things we can safely ignore now. One of those is the <code>KeyStore</code> entry. That's used if you want Share to talk to the Repo over SSL with client Auth. You may well want to turn that on for production, but not for getting started. (If you do, you should set the SSL client cert username as your <i>proxyUserName</i>, then the Repo will only check for the <i>proxyHeader</i> HTTP Header for the username for requests from that user). Secondly, we're going to be using a HTTP Header to carry the username, so we don't need the <code>alfrescoCookie</code> connector. <br /><br />If you don't have one already, create a <code>alfresco/web-extension/share-config-custom.xml</code> file, typically either in a module, or in the Tomcat shared classes. If you're creating a fresh one, use <code>share-config-custom.xml.sample</code> as a basis (it has nearly the config in it you need, along with other bits you won't want now so remove / comment them out). The key bit for us is to add/modify a <i>Remote</i> config section. This is where you define how Share talks to other systems over http/https, especially to the repository (via the <i>alfresco</i> endpoint). Define a Remote section as follows:<pre>   &lt;config evaluator="string-compare" condition="Remote"&gt;
          &lt;remote&gt;
              &lt;connector&gt;
                 &lt;id&gt;alfrescoHeader&lt;/id&gt;
                 &lt;name&gt;Alfresco Connector&lt;/name&gt;
                 &lt;description&gt;Connects to an Alfresco instance using header and cookie-based authentication&lt;/description&gt;
                 &lt;class&gt;org.alfresco.web.site.servlet.SlingshotAlfrescoConnector&lt;/class&gt;
                 &lt;userheader&gt;X-My-Custom-SSO-Header&lt;/userHeader&gt;
              &lt;/connector&gt;

              &lt;endpoint&gt;
                  &lt;id&gt;alfresco&lt;/id&gt;
                  &lt;name&gt;Alfresco - user access&lt;/name&gt;
                  &lt;description&gt;Access to Alfresco Repository WebScripts that require user authentication&lt;/description&gt;
                  &lt;connector-id&gt;alfrescoHeader&lt;/connector-id&gt;
                  &lt;endpoint-url&gt;http://localhost:8080/alfresco/wcs&lt;/endpoint-url&gt;
                  &lt;identity&gt;user&lt;/identity&gt;
                  &lt;external-auth&gt;true&lt;&lt;/external-auth&gt;
              &lt;/endpoint&gt;

          &lt;/remote&gt;
      &lt;/config&gt;</pre>With that in place, if you go to Share in your browser via the Apache connection - <a href='http://local-alfresco/share/' rel='nofollow'>http://local-alfresco/share/</a> if you followed part 1 - you get an Apache basic auth prompt, then you should be automatically logged into Share with the user details you gave to Alfresco. SSO done!<br /><br />One other thing to try before we understand how it works (and you lock down the security!), we'll try curl again:<pre>$ curl -X GET -L -H "X-My-Custom-SSO-Header: nick" http://localhost:8080/share/ | less</pre>Look for something near the top showing <code>Alfresco.constants.USERNAME = "nick"</code> and not <i>guest</i> to confirm header based auth worked for Share.<br /><br /><br />So, what's going on, and how does it all work? At this point, you'll probably want to grab the <a href="http://www.springsource.org/extensions/se-surf" rel="nofollow">Spring Surf</a> source code if you want to dig around, as most of the logic isn't in Share itself, but the underlying Surf framework.<br /><br />Looking at the config, there are a few things to note. In the <code>alfresco</code> endpoint, we have <code>&lt;endpoint-url&gt;<a href='http://localhost:8080/alfresco/wcs' rel='nofollow'>http://localhost:8080/alfresco/wcs</a>&lt;/endpoint-url&gt;</code> - note that it has changed to the WCServices URL from the regular Services one. As seen in the <a href="http://gagravarr.livejournal.com/145144.html" rel="nofollow">how the Alfresco WebScript authentication methods work</a> blog post, this is so that we can use the Authentication subsystems via the filters to authenticate, while services just does basic auth + tickets. The next thing is <code>&lt;external-auth&gt;true&lt;/external-auth&gt;</code>, which tells Share that we're using External Authentication and it should hide a few bits. <a href="https://anonsvn.springframework.org/svn/se-surf/trunk/spring-surf/spring-surf/src/main/java/org/springframework/extensions/surf/site/AuthenticationUtil.java" rel="nofollow">AuthenticationUtil</a> exposes this via <code>isExternalAuthentication</code>, and a few bits of Share use that. Finally, the connector-id ties us back to the Header and some Java.<br /><br />In the connector section of the config, the route taken is that Share finds an endpoint with the id <i>alfresco</i>, then uses that to find the connector to use by the <i>connector-id</i> defined there. For us, that's our new <code>alfrescoHeader</code> one. That includes <code>&lt;userHeader&gt;X-My-Custom-SSO-Header&lt;/userHeader&gt;</code> which is the HTTP Header used by Share and the Repo to pass along the username to use. The other thing in there is the class that drives it all, <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/slingshot/source/java/org/alfresco/web/site/servlet/SlingshotAlfrescoConnector.java" rel="nofollow">SlingshotAlfrescoConnector</a>.<br /><br />At this point, I should perhaps point out a gotcha. The underlying External Authentication SubSystem supports <code>external.authentication.userIdPattern</code> to only use part of the Header as the username, but due to <a href="https://issues.alfresco.com/jira/browse/ALF-18393" rel="nofollow">ALF-18393</a> Share doesn't and it all goes a bit wrong... The option isn't (currently!) Share Compatible<br /><br />Looking some more at <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/slingshot/source/java/org/alfresco/web/site/servlet/SlingshotAlfrescoConnector.java" rel="nofollow">SlingshotAlfrescoConnector</a> which we defined above, there's one key method in there, <code>applyRequestHeaders</code>. This is the part where Share is tweaking the headers on any connections it makes to the Repo. In it, it checks to see if you authenticated without a password (i.e. with SSO). If you have, your username is sent to the repo in the header <code>X-Alfresco-Remote-User</code> and if you set a config entry <code>userHeader</code> then with that too as well. There's also <code>setConnectorSession</code> which feeds your custom header down the stack.<br /><br />So, if you want to send some extra headers to the Repo to prove to it you are really Share, and you don't want to do it with SSL client certs (for which there is an example in <code>share-config-custom.xml.sample</code>), then overriding <i>SlingshotAlfrescoConnector</i> + <i>applyRequestHeaders</i> + specifying that class instead is likely the way to go.<br /><br />Where does the username come from to pass along? That's within Spring Surf, in <a href="https://anonsvn.springframework.org/svn/se-surf/trunk/spring-surf/spring-surf/src/main/java/org/springframework/extensions/surf/support/AbstractUserFactory.java" rel="nofollow">AbstractUserFactory</a> inside <code>initialiseUser</code><br /><br />The final bit of magic is explaining how specifying our magic header gets passed through Share to the Repo. The solution to that is to be found in <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/slingshot/source/java/org/alfresco/web/site/servlet/SSOAuthenticationFilter.java" rel="nofollow">SSOAuthenticationFilter</a>. As well as handling Kerberos and NTML challenge stuff, it also does a bit of magic in the case of custom username headers. Specifically, in <code>wrapHeaderAuthenticatedRequest</code> it makes the value of the custom header available as the result of <code>getRemoteUser()</code>, which allows the rest of Surf to know who you are.<br /><br /><br />Phew, and we're done! You should now have a working Alfresco + Share SSO setup, which can work either through Apache with Apache handling the Authentication, or direct to Tomcat with special headers. We (hopefully) understand how it works, why, and have a testbed. At this point, we can tweak it for our final SSO setup, put in any security around it we need, and go live!


guid: https://gagravarr.livejournal.com/145234.html
url: https://gagravarr.livejournal.com/145234.html
title: Authenticating to Alfresco using Apache Auth (eg for SSO)
published: 2013-03-11 17:43:21
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.344193
updated_at: 2019-10-27 00:00:11.139951
summary:
  I'm currently setting up SSO (Single Sign On) for an Alfresco + Share install. I know I'm going to need to do a bit of customising, but I've struggled a bit with understanding what's going on with the various SSO tutorials. They give instructions, but not always a lot of explanation, which can be tricky if you know you want to go off piste. So, having <a href="http://gagravarr.livejournal.com/145144.html" rel="nofollow">reminded myself how the WebScript /services/ + /wcservices/ authentication works</a>, my next step is to get some basic Apache Authentication in place, and have Alfresco trust that.<br /><br />Why basic Apache Authentication? Well, it's the basis on which things like <a href="http://wiki.alfresco.com/wiki/Alfresco_With_mod_auth_cas" rel="nofollow">Alfresco + CAS</a> and <a href="http://wiki.alfresco.com/wiki/Alfresco_With_Shibboleth" rel="nofollow">Alfresco + Shibboleth</a> work, it can be used for SSL client cert auth, it's fairly easy to setup, I know most of the pieces already (always a bonus!). My aim in this - get Alfresco Explorer and the /wcservices/ WebScripts trusting Apache provided authentication. I'm not looking at Share, that's a follow-on post.<br /><br />So, what components do we need? Firstly, there's the <a href="http://wiki.alfresco.com/wiki/Alfresco_Authentication_Subsystems#REMOTE_USER_CGI_variable_based_external_authentication" rel="nofollow">Alfresco External Authentication Subsystem</a>. Next there's Tomcat AJP-13. There's Apache mod_jk to talk AJP-13. Finally, there's the basic Apache Authentication. Because I just want to test, I'm using a simple file based Apache auth setup. You likely won't use that in production, but it'll get you most of the way to a working SSO setup, and is easy to test with.<br /><br /><br />Our first thing to do is tell Alfresco that it can trust the remote authentication from Apache. (The process for this is similar to enabling Header based external Authentication). To do this, we need to enable + configure another <a href="http://wiki.alfresco.com/wiki/Alfresco_Authentication_Subsystems" rel="nofollow">Authentication SubSystem</a>, specifically the <a href="http://wiki.alfresco.com/wiki/Alfresco_Authentication_Subsystems#REMOTE_USER_CGI_variable_based_external_authentication" rel="nofollow">External Auth SubSystem</a>. Create a directory <code>alfresco/extension/subsystems/Authentication/external/external-apache</code> in your usual customisation spot (typically a module or the Tomcat Shared Classes). Into this new directory, create <code>external-authentication.properties</code>, and populate it with something like:<pre># Which users should be treated as an Admin?
  external.authentication.defaultAdministratorUserNames=admin,nick
  # Enable the External Authentication
  external.authentication.enabled=true</pre>That's not all though, you also need to add this new authentication subsystem to the active list. Somewhere, quite possibly in your <code>alfresco-global.properties</code> file you should have an <code>authentication.chain</code> line. Prepend this with <code>external-apache:external</code>, something like:<pre># This is what we used to have, LDAP + Built-in
  ##authentication.chain=ldap-main:ldap,alfrescoNtlm1:alfrescoNtlm

  # Try the External Auth if we can, otherwise the previous ones
  authentication.chain=external-apache:external,ldap-main:ldap,alfrescoNtlm1:alfrescoNtlm</pre>Restart Alfresco, and check that you can log in as before. You should also see (assuming the default logging settings) something like <code>INFO  [org.alfresco.repo.management.subsystems.ChildApplicationContextFactory] Starting &#39;Authentication&#39; subsystem, ID: [Authentication, managed, external-apache]</code> in your logs to show it was found and used.<br /><br /><br />Next up, we need to enable AJP-13 in Tomcat, and tell it to trust Apache supplied Authentication. I&#39;m assuming you have two Tomcats, one for Alfresco on port 8080, and one for Share on 8081. If you only have one, skip the Share/8081/8010 bits. Edit <code>[Tomcat Root]/conf/server.xml</code>; for both Tomcats. Around line 90, you should come across an AJP Connector section, which may be commented out. Uncomment it, set the Alfresco one to port 8009 (default), and the share one to 8010. Finally, add the attribute <code>tomcatAuthentication="false"</code> to the Connector. This should give you something on the Alfresco Tomcat looking like:<pre>   &lt;!-- Define an AJP 1.3 Connector on port 8009 --&gt;
      &lt;Connector port="8009" protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false" /&gt;</pre>And for Share it'll be:<pre>   &lt;!-- Define an AJP 1.3 Connector on port 8010 (Alternate Port) --&gt;
      &lt;Connector port="8010" protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false" /&gt;</pre>Restart your tomcats, and ensure they come up without error. Check you can telnet to localhost 8009 and 8010 (there's no welcome banner) to ensure the Tomcats are listening and no local firewalling breaks.<br /><br /><br />Half way!<br /><br /><br />If you haven't already, install a copy of Apache (latest 2.4, or perhaps even 2.5/2.6 if that's out by the time you read this!), and make sure you can get the welcome page up. If on a Unix, make sure you have the Apache developer tools package installed too, if you're doing a package install. If you can, install a packaged version of the <a href="http://tomcat.apache.org/connectors-doc/" rel="nofollow">Tomcat Connectors module</a>. Failing that, grab the <a href="http://tomcat.apache.org/download-connectors.cgi" rel="nofollow">source code</a> and <a href="http://tomcat.apache.org/connectors-doc/webserver_howto/apache.html#Installation" rel="nofollow">build + install the module</a><br /><br />There are two parts to configuring up mod_jk, the first is a JK specific file <a href="http://tomcat.apache.org/connectors-doc/reference/workers.html" rel="nofollow">worker.properties</a>, the second is the regular Apache config. For the former, create a new file somewhere sensible (but make sure it's not somewhere that Apache auto-loads config files from, or you'll get errors). FWIW, I created a new directory under <i>/etc/apache2/</i> of <i>conf.other</i> and created it as <code>tomcat-workers.properties</code> but it'll depend on your distro. Assuming you setup your Tomcat AJP as above, create the file with:<pre># For communicating with Tomcat via AJP13

  # We have two Tomcats, and we'd like status please
  worker.list=jkalfresco,jkshare,jkstatus

  # Enable the status worker
  worker.jkstatus.type=status

  # Alfresco is on 8080 (http) / 8009 (AJP)
  worker.jkalfresco.type=ajp13
  worker.jkalfresco.host=localhost
  worker.jkalfresco.port=8009
  worker.jkalfresco.connection_pool_timeout=600
  worker.jkalfresco.socket_keepalive=1

  # Share is on 8081 (http) / 8010 (AJP)
  worker.jkshare.type=ajp13
  worker.jkshare.host=localhost
  worker.jkshare.port=8010
  worker.jkshare.connection_pool_timeout=600
  worker.jkshare.socket_keepalive=1</pre><br />Now, define a new vhost to Apache, either by tacking it on the end of your httpd.conf, or putting it in a new file in conf.d, or in sites-enabled, or .... as appropriate for your setup. To start with, go for something like:<pre># Load the mod_jk module (your path to it may be different)
  LoadModule jk_module /usr/lib/apache2/modules/mod_jk.so

  # These paths may need changing for your setup too
  # Where the file we created above lives
  JkWorkersFile /etc/apache2/conf.other/tomcat-workers.properties
  # Where to put jk shared memory
  JkShmFile     /var/log/apache2/mod_jk.shm
  # Where to put jk logs
  JkLogFile     /var/log/apache2/mod_jk.log
  # Set the jk log level [debug/error/info]
  JkLogLevel    info
  # Select the timestamp log format
  JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "

  # Define the vhost that'll expose Tomcat
  &lt;VirtualHost *:80&gt;
  	ServerAdmin webmaster@localhost
  	ServerName local-alfresco

          JkMount /status     jkstatus
          JkMount /alfresco   jkalfresco
          JkMount /alfresco/* jkalfresco
          JkMount /share      jkshare
          JkMount /share/*    jkshare
  &lt;/VirtualHost&gt;</pre>The ServerName needs to be unique, so either make it your machine name (only vhost), or create a new alias for 127.0.0.1 in your <code>/etc/hosts</code> file and use that. Restart apache, and try visiting [server name]/status to check that mod_jk is loaded and running. Then, visit [server name]/alfresco and check that you get explorer as guest.<br /><br />Finally, before we consider about how it all fits together, we need to turn on Authentication in Apache, so it has some auth to pass to Tomcat. Full details on this are <a href="http://httpd.apache.org/docs/2.4/howto/auth.html" rel="nofollow">in the Apache docs</a>, but basically you'll want to use htpasswd to create a digest password file, something like:<pre>  $ # Create the file, add admin
    $ htpasswd -c /etc/apache2/conf.other/alfresco.htpasswd admin admin
    $ # Create another user (don't use these passwords in production!)
    $ htpasswd /etc/apache2/conf.other/alfresco.htpasswd nick password</pre><br />Now, add to our vhost some auth lines:<pre>        &lt;Location /&gt;
            AuthType Basic
            AuthName "Test Alfresco Access Auth"
            AuthBasicProvider file
            AuthUserFile /etc/apache2/conf.other/alfresco.htpasswd
            Require valid-user
          &lt;/Location&gt;</pre>Restart Apache, and re-visit [server name]/alfresco . You should be prompted for a password. Specify admin and admin (as given to htpasswd), and you should be taken to Explorer logged in as admin automatically. We're basically there!<br /><br /><br />At this point, make sure you're only using a test server, have suitable firewalling etc in place. We're just trying to get some basics in place to test with, so we have a working base before making our changes. More is needed to make this production...<br /><br /><br />So, how does this all work? Apache is doing the authentication, and passing that through to Tomcat. We can see that Tomcat sees the username by creating a small test jsp somewhere in the Alfresco webapp, containing:<pre>User: &lt;%=request.getRemoteUser()%&gt;</pre>Run that, and you'll see the username you gave to the Apache auth come through. That username is passed into the Servlet Request.<br /><br />What does Alfresco do with it? When we enabled our External Authentication SubSystem, we effectively pulled in the <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/web-client/config/alfresco/subsystems/Authentication/external/external-filter-context.xml" rel="nofollow">external-filter-context.xml context file</a>, along with partly overriding the default <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/web-client/config/alfresco/subsystems/Authentication/external/external-filter.properties" rel="nofollow">external-filter.properties</a>. The <i>webscriptAuthenticationFilter</i> isn't really doing that much for us, the key bit is the <code>remoteUserMapper</code>. This is configured up to use <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/web-client/source/java/org/alfresco/web/app/servlet/DefaultRemoteUserMapper.java" rel="nofollow">DefaultRemoteUserMapper</a><br /><br />When the request comes in to Alfresco, for either Explorer or /wcservices/ (but <i>not</i> /services/ as that's <a href="http://gagravarr.livejournal.com/145144.html" rel="nofollow">different</a>), the global <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/remote-api/source/java/org/alfresco/repo/webdav/auth/AuthenticationFilter.java" rel="nofollow">Authentication Filter</a> fires. In <code>doFilter</code> it calls <code>getSessionUser</code> which in turn checks for a Remote User Mapper. If one is found (and we've enabled one with our next Authentication SubSystem), <code>getRemoteUser</code> is called. If you look at that method in <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/web-client/source/java/org/alfresco/web/app/servlet/DefaultRemoteUserMapper.java" rel="nofollow">DefaultRemoteUserMapper</a>, we'll see it checks the Remote User on the request, and returns that. So, no login is needed at the Tomcat / Alfresco layer, because the value from Apache is trusted.<br /><br />We can also see how we could do http header auth at this point if we wanted, and also see some of the security issues which mean you shouldn't blindly enable this in production...!<br /><br /><br />So, we can now use our test Apache auth setup to be auto logged-in to Alfresco Explorer and wcservices. In <a href="http://gagravarr.livejournal.com/145561.html" rel="nofollow">the next part</a>, we'll see how we enable <a href="http://gagravarr.livejournal.com/145561.html" rel="nofollow">Share auto-login</a> with the same setup, and how that actually works under the hood.


guid: https://gagravarr.livejournal.com/145144.html
url: https://gagravarr.livejournal.com/145144.html
title: Alfresco WebScripts (Services) and Authentication
published: 2013-03-07 16:10:39
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.347013
updated_at: 2019-10-27 00:00:11.142360
summary:
  I'm currently doing some work on enabling SSO for Alfresco, and I've come across some slightly confused wiki pages and blog posts out there around how webscripts and authentication fits together. As I've done quite a bit of fighting with this in the past, I thought it might be worth trying to summarise what works and why.<br /><br />Firstly, WebScript URL prefixes. The Alfresco webscripts are available under three pairs of URL prefixes, also known as WebScript runtimes. As detailed <a href="http://wiki.alfresco.com/wiki/Web_Script_Runtimes#Pre-defined_web_script_URL_mappings" rel="nofollow">on the wiki</a>, the three are:<br /><ul><li><code>/alfresco/service/</code> or <code>/alfresco/s/</code> - HTTP Auth + Tickets, "apiServlet"</li><br /><li><code>/alfresco/wcservice/</code> or <code>/alfresco/wcs/</code> - Explorer Style Auth, "wcapiServlet"</li><br /><li><code>/alfresco/168service/</code> or <code>/alfresco/168s/</code> - JSR-168 Portlet Auth, "portalapiServlet"</li></ul><br />I'll skip the portlet stuff, as I don't use it, and concentrate on the other too. (Note that all three expose the same set of webscripts, it's just the authentication in front of them that changes)<br /><br /><br /><br />The first thing to note is that describing <code>/alfresco/service/</code> as using HTTP Auth misses a few bits out. There are in fact several ways to authenticate to it. One option, most commonly used when debugging and admining, is to use HTTP Basic Auth (but make sure you're using it over SSL in production!). If no other authentication is sent, and the webscript requires it, the webscript framework will prompt you for your username and password with basic auth. <br /><br />Another option is to use Alfresco Tickets, as <a href="http://wiki.alfresco.com/wiki/Web_Scripts#Authenticating" rel="nofollow">detailed in the wiki here</a>. You can get a ticket a few different ways, but posting JSON of the username and password to <code>/alfresco/service/api/login</code> is often the most common. The ticket can be supplied with a request via the <code>alf_ticket</code> URL parameter, or by sending HTTP basic auth with a username of <code>ROLE_TICKET</code> and the ticket as the basic auth password. Another option is to send the URL parameter of <code>guest=true</code> to explicitly request guest access.<br /><br />If you want to see how the authentication to <code>/alfresco/service/</code> works, the main class to look at is <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/remote-api/source/java/org/alfresco/repo/web/scripts/servlet/BasicHttpAuthenticatorFactory.java" rel="nofollow">BasicHttpAuthenticatorFactory</a><br /><br /><br /><br />When looking at <code>/alfresco/wcservice/</code> , most things will describe it as using Explorer authentication. That isn't quite the full story though, if you take a look at <code>web.xml</code> you'll see that there are actually two Authentication Filters that apply to requests through wcservice. The <code>Global Authentication Filter</code> is used (that's the one shared with Explorer), but before that a special <code>WebScript Authentication Filter</code> is also tried.<br /><br />The latter is passed off to a spring bean <code>webscriptAuthenticationFilter</code>. Not all authentication subsystems implement one of these, so often it'll do nothing. If the authentication filter declines, then it'll fall back to the default <code>Global Authentication Filter</code>, which ends up as the bean <code>globalAuthenticationFilter</code>, and from there the normal Explorer authentication kicks in.<br /><br /><br /><br />Quite a few of the instructions out there for doing SSO involve specifying some custom share config, to change the <code>alfresco</code> Remote Endpoint to use <code>/wcs</code> . If you see one of these, it will almost always be coupled with changes to your Authentication subsystem(s) to do something with the <code>webscriptAuthenticationFilter</code>. When setting this sort of thing up / debugging, be aware that you'll need to test with /wcs and not /service, and that normally you can't test with Explorer as that doesn't use the same filter!


guid: https://gagravarr.livejournal.com/144845.html
url: https://gagravarr.livejournal.com/144845.html
title: Overriding Alfresco webscript library ftl files
published: 2013-01-30 19:04:30
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.349502
updated_at: 2019-10-27 00:00:11.144702
summary:
  On the whole, Alfresco is pretty good for customising and overriding, and there's been a lot of work done in 4.2 to make it even easier to customise Share and Surf. Overriding the spring beans for Java backed webscripts <a href="http://gagravarr.livejournal.com/144503.html" rel="nofollow">can be done with a bit of fiddling</a>. One problem we have hit though is with overriding FTL files, especially library ones.<br /><br />I'll use the <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/remote-api/config/alfresco/templates/webscripts/org/alfresco/repository/workflow/" rel="nofollow">Workflow REST API</a> as an example here, as it's one we needed to change, but it applies to a lot of the webscripts too. A common JSON emitting Alfresco REST API has a FTL file somewhat like:<br /><br /><pre>&lt;#import "workflow.lib.ftl" as workflowLib /&gt;
  {
     "data":
     &lt;@workflowLib.taskJSON task=workflowTask detailed=true/&gt;
  }</pre>It imports a library file, then calls one of the functions in there. If you want to inject a few extra bits into the JSON structure, then your only option is to edit the library file, or clone it. For simple libraries with a single macro, you could just clone the whole thing and customise, but for a library with many macros that gets rather nasty especially when upgrading. <a href="http://freemarker.624813.n4.nabble.com/Freemarker-user-macro-override-td975847.html" rel="nofollow">FreeMarker macros get replaced when a new one is defined with the same name</a>, much like spring beans, so for our hypothetical macro file:<br /><br /><pre>&lt;#macro taskJSON task detailed=false&gt;
  &lt;#escape x as jsonUtils.encodeJSONString(x)&gt;
        {
           "id": "${task.id}",
            ....
        }
  &lt;/#escape&gt;
  &lt;/#macro&gt;

  &lt;#macro propertiesJSON properties&gt;
  &lt;#escape x as jsonUtils.encodeJSONString(x)&gt;
  {
  &lt;#list properties?keys as key&gt;
     "${key}":
     ...
  &lt;/#escape&gt;
  &lt;/#macro&gt;</pre>To change <i>taskJSON</i> to inject some extra bits from the model, we'd have to copy the whole file and customise. However, there is a way that makes things a little less evil, which relies on macros overwriting each other. We take the original file, and copy it to the same directory structure in our module, but with a new name. Instead of <i>workflow.lib.ftl</i> we might instead go for <i>workflow-original.lib.ftl</i>. Next, add a new comment to the top, documenting where it originally came from, what version it is taken from etc. Now, create a new <i>workflow.lib.ftl</i>, and copy into that the whole of the macro we want to override. Add in the extra few JSON clauses. At the top of the file, add some documentation saying what version you copied and pasted from, what file it came from, and what changes you made. Trust me, you'll be thankful you did this when you come to upgrade... Finally, at the top of the file, before your customised macro, add in an <i>include</i> (not import, include) of the renamed original. Your new lib, with the name of the old one, now looks something like:<br /><br /><pre>&lt;#-- Customised File --&gt;
  &lt;#-- Original: remote-api/config/alfresco/templates/webscripts/org/alfresco/repository/workflow/workflow.lib.ftl --&gt;
  &lt;#-- Alfresco Version: 4.1.1.3 --&gt;
  &lt#-- Modification: Added custom1 and custom2 properties to the JSON --&gt;

  &lt;#-- This pulls in all the macros we haven't needed to change --&gt;
  &lt;#include "workflow-original.lib.ftl"&gt;

  &lt;#-- Override and customise the one we do need to change --&gt;
  &lt;#macro taskJSON task detailed=false&gt;
  &lt;#escape x as jsonUtils.encodeJSONString(x)&gt;
        {
           "id": "${task.id}",
            ....
           "myCustom1": ${task.custom1},
           "myCustom1": ${task.custom2}
        }
  &lt;/#escape&gt;
  &lt;/#macro&gt;</pre>(It's probably best to put your custom json at the start or end, to make upgrading easier)


guid: https://gagravarr.livejournal.com/144503.html
url: https://gagravarr.livejournal.com/144503.html
title: Overriding built-in Java backed WebScripts in Alfresco
published: 2013-01-30 15:27:48
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.352105
updated_at: 2019-10-27 00:00:11.146953
summary:
  Generally, Alfresco makes it fairly easy for you to override the built-in core repository Services. There's <a href="http://wiki.alfresco.com/wiki/Overriding_Spring_Configuration" rel="nofollow">this wiki</a> page and <a href="http://wiki.alfresco.com/wiki/Developer_Runtime_Configuration" rel="nofollow">this one</a> to get you started, then once you know about Spring you can largely make it happen.<br /><br />The Alfresco spring context file load order for services etc is controlled by <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/repository/config/alfresco/application-context.xml" rel="nofollow">application-context.xml</a>. The last bean defined with a given name wins, so if Alfresco ships with a bean definition for <quote>fooService</quote> and you define your own one later with the same name, yours will be used. Handily, as shown in that context file, spring context files in /extensions/ and /modules/ are loaded late, so your modules can include override services.<br /><br /><br />The picture is not quite so rosy for overriding the built-in webscripts. Here, the key context file is <a href="http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/web-client/source/web/WEB-INF/web-application-context.xml" rel="nofollow">web-application-context.xml</a>. This loads <i>application-context.xml</i> which we saw above, and then it loads the web scripts. This means that the built-in webscritps are always loaded <i>after</i> modules, so your module can never override a built in webscript.<br /><br />However, at the bottom of that file we see the answer - we need to provide a single <i>alfresco/extension/custom-web-context.xml</i> which pulls in our module webscripts. That gets loaded last, so will win!<br /><br />For my current project, we have added the following file as <i>alfresco/extension/custom-web-context.xml</i><br /><br /><pre>&lt;?xml version='1.0' encoding='UTF-8'?&gt;
  &lt;!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'&gt;
  &lt;beans&gt;
      &lt;import resource="classpath*:alfresco/module/*/module-webscripts-context.xml" /&gt;
  &lt;/beans&gt;</pre>All of our Repository modules already had a <i>module-context.xml</i> file in <i>alfresco/module/&lt;name&gt;/</i>. Those that need to override webscripts can now also provide a second file in that directory, <i>module-webscripts-context.xml</i>, and any webscripts defined in there can be guaranteed to win and be used.<br /><br />(The new custom-web-context.xml file can be bundled in a jar, shipped in an existing AMP, packaged as a brand new AMP, or dropped in your classes override directory similar to your repository properties file - it all depends what'll be cleaner for your setup)


guid: https://gagravarr.livejournal.com/144366.html
url: https://gagravarr.livejournal.com/144366.html
title: Spring in a QName
published: 2013-01-30 15:11:38
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.354892
updated_at: 2019-10-27 00:00:11.149268
summary:
  <b><i>This post was originally written in September 2010, and was available at http://blogs.alfresco.com/wp/nickb/2010/09/28/spring-in-a-qname/ - I have reposted it here as that blog is no longer online.</i></b>

  <p>Within Alfresco, we make a lot of use of Qualified Names (QNames) for addressing and naming things. Generally, when configuring Alfresco through Spring or properties files, we can use the short form, eg</p>
  <p><code>&lt;bean id="coreBean" class="org.alfresco.some.thing.core"&gt;<br />
  &nbsp; &lt;property name="typeQName"&gt;<br />
  &nbsp; &nbsp; &lt;value&gt;cm:description&lt;/value&gt;<br />
  &nbsp; &lt;/property&gt;<br />
  &lt;/bean&gt;</code></p>
  <p>Within the bean, the NamespaceResolver is used to turn the friendly, short form (eg cm:description) into the full form (eg {http://www.alfresco.org/model/content/1.0}description).</p>
  <p>However, every so often you may find yourself trying to configure something with Spring that no-one ever expected you to be trying to do&#8230; In this situation, the string form isn&#8217;t accepted by the class, and only a real QName object may be sprung in.</p>
  <p>As it turns out, creating a real QName object from within Spring isn&#8217;t actually too hard to do. So, in case you ever find yourself needing to do it, the definition will look something like this:</p>
  <p><code>&lt;bean id="coreBean" class="org.alfresco.some.thing.core"&gt;<br />
  &nbsp; &lt;property name="typeQName"&gt;<br />
  &nbsp; &nbsp; &lt;value&gt;<br />
  &nbsp; &nbsp; &nbsp; &lt;bean class="org.alfresco.service.namespace.QName"<br />
  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;       factory-method="createQName"&gt;<br />
  &nbsp; &nbsp; &nbsp; &nbsp; &lt;constructor-arg value="http://www.alfresco.org/model/content/1.0" /&gt;<br />
  &nbsp; &nbsp; &nbsp; &nbsp; &lt;constructor-arg value="description"/&gt;<br />
  &nbsp; &nbsp; &nbsp; &lt;/bean&gt;<br />
  &nbsp; &nbsp; &lt;/value&gt;<br />
  &nbsp; &lt;/property&gt;<br />
  &lt;/bean&gt;</code></p>


guid: https://gagravarr.livejournal.com/144097.html
url: https://gagravarr.livejournal.com/144097.html
title: Adding new Apache Tika Plugins to Alfresco (Part 3)
published: 2013-01-30 15:09:04
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.359373
updated_at: 2019-10-27 00:00:11.151437
summary:
  <b><i>This post was originally written in October 2010, and was available at http://blogs.alfresco.com/wp/nickb/2010/10/27/adding-new-apache-tika-plugins-to-alfresco-part-3/ - I have reposted it here as that blog is no longer online.</i></b>

  <p>In <a href="http://gagravarr.livejournal.com/143490.html" rel="nofollow">Part 1</a> we saw what <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a> is and does, and in <a href="http://gagravarr.livejournal.com/143623.html" rel="nofollow">Part 2</a> we saw what it has brought to Alfresco. Now it&#8217;s time to look at adding new Tika Parsers, to support new file formats.</p>
  <div style="height:1.4em"> </div>
  <p>Firstly, why might you want to add a new parser? The most common reason is licensing &#8211; all the parsers that ship as standard with <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a> are Apache Licensed or similar, along with their dependencies, and so can be freely distributed and included in other projects. However, some file formats only have libraries that are available under GPL or proprietary licenses, and so these can&#8217;t be included in the standard Tika distribution.</p>
  <p>There is a list of available 3rd party parsers on the <a href="http://wiki.apache.org/tika/3rd%20party%20parser%20plugins" rel="nofollow">Tika 3rd Party Plugins</a> wiki page, currently made up of GPL licensed parsers + dependencies. If your format isn&#8217;t listed there, and you want to add it to Tika within Alfresco, then what to do?</p>
  <p>Firstly, you need to write / acquire a Tika Parser. Writing a Tika Parser is quite easy, as the <a href="http://tika.apache.org/0.7/parser_guide.html" rel="nofollow">5 minute parser guide</a> explains. There are basically two methods to implement:</p>
  <pre>Set getSupportedTypes(ParseContext context);</pre>
  <pre>void parse(InputStream stream, ContentHandler handler, Metadata metadata,
             ParseContext context) throws IOException, SAXException, TikaException;</pre>
  <p>The first allows you to indicate the file types your parser can handle. This is needed when registering the parser with the AutoDetectParser and similar, but isn&#8217;t needed if you select the parser explicitly. The second method is the one where you do the real work of outputting the contents and populating the metadata object.</p>
  <div style="height:1.4em"> </div>
  <p>To see this in action, let&#8217;s take a look at a simple &#8220;Hello World&#8221; Tika Parser:</p>
  <pre>package example;
  public class HelloWorldParser implements Parser {
    public Set getSupportedTypes(ParseContext context) {
      Set types = new HashSet();
      types.add(MediaType.parse("hello/world"));
      return types;
    }
    public void parse(InputStream stream, ContentHandler handler,
           Metadata metadata, ParseContext context) throws SAXException {
      XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata);
      xhtml.startDocument();
      xhtml.startElement("h1");
      xhtml.characters("Hello, World!");
      xhtml.endElement("h1");
      xhtml.endDocument();

      metadata.set("hello","world");
      metadata.set("title","Hello World!");
      metadata.set("custom1","Hello, Custom Metadata 1!");
      metadata.set("custom2","Hello, Custom Metadata 2!");
    }
  }</pre>
  <p>Before we can use this in Alfresco, we need to compile this against <i>tika-core.jar</i> (note &#8211; you may need to implement the parse method without a ParseContext object if you&#8217;re using an older version of Tika), and then wrap our classfile up in a jar. Once our jar is deployed into our application container (eg the shared lib of tomcat), we&#8217;re ready to configure it.</p>
  <p>For 3rd party parsers which provide the Tika service metadata files, if we don&#8217;t want to control the registration in Alfresco then we can simply allow the default Tika-Auto metadata and transformer classes to handle it. In our case, we want to register it explicitly. To do that, we&#8217;ll create a new extension spring context file, and populate it:</p>
  <pre>&lt;?xml version='1.0' encoding='UTF-8'?&gt;
  &lt;!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'&gt;
  &lt;beans&gt;
      &lt;bean id="extracter.MyCustomTika"
            class="org.alfresco.repo.content.metadata.TikaSpringConfiguredMetadataExtracter"
            parent="baseMetadataExtracter" &gt;
          &lt;!-- This is the name of our example parser compiled above --&gt;
          &lt;property name="tikaParserName"&gt;
             &lt;value&gt;example.HelloWorldParser&lt;/value&gt;
          &lt;/property&gt;

           &lt;!-- Use the default mappings from TikaSpringConfiguredMetadataExtracter.properties --&gt;
          &lt;property name="inheritDefaultMapping"&gt;
              &lt;value&gt;true&lt;/value&gt;
          &lt;/property&gt;
          &lt;!-- Map our extra keys to the content model --&gt;
          &lt;property name="mappingProperties"&gt;
              &lt;props&gt;
                  &lt;prop key="namespace.prefix.cm"&gt;http://www.alfresco.org/model/content/1.0&lt;/prop&gt;
                  &lt;prop key="custom1"&gt;cm:description&lt;/prop&gt;
                  &lt;prop key="custom2"&gt;cm:author&lt;/prop&gt;
              &lt;/props&gt;
          &lt;/property&gt;
      &lt;/bean&gt;

      &lt;bean id="transformer.MyCustomTika"
            class="org.alfresco.repo.content.transform.TikaSpringConfiguredContentTransformer"
            parent="baseContentTransformer"&gt;
          &lt;!-- Same as above --&gt;
          &lt;property name="tikaParserName"&gt;
             &lt;value&gt;example.HelloWorldParser&lt;/value&gt;
          &lt;/property&gt;
      &lt;/bean&gt;
  &lt;/beans&gt;
  </pre>
  <p>To test this, we&#8217;ll need a node with the special fake mimetype of &#8220;hello/world&#8221;, which is what our Tika Parser is configured to handle. We can do that with a snippet of JavaScript like this:</p>
  <pre>var doc = userhome.createFile("hello.world");
  doc.content = "This text will largely be ignored";
  doc.mimetype = "hello/world";
  </pre>
  <p>If we run the above JavaScript, we&#8217;ll get a node called &#8220;hello.world&#8221;. If we run the &#8220;extract common metadata fields&#8221; action on it, we&#8217;ll then see the metadata properties showing through. Then, if we transform it to text/html, then we see a text heading of &#8220;Hello, World&#8221;. Thus we have verified that our custom Tika parser has been wired into Alfresco, is available for text transformation, and can do metadata extraction including custom keys.</p>
  <p>More information on Tika and Alfresco is available <a href="http://wiki.alfresco.com/wiki/Content_Transformation_and_Metadata_Extraction_with_Apache_Tika" rel="nofollow">on the Alfresco wiki</a>.</p>


guid: https://gagravarr.livejournal.com/143623.html
url: https://gagravarr.livejournal.com/143623.html
title: Apache Tika powered updates to Alfresco (Part 2)
published: 2013-01-30 15:04:40
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.362181
updated_at: 2019-10-27 00:00:11.153717
summary:
  <b><i>This post was originally written in September 2010, and was available at http://blogs.alfresco.com/wp/nickb/2010/09/24/apache-tika-powered-updates-to-alfresco-part-2/ - I have reposted it here as that blog is no longer online.</i></b>

  <p>In <a href="http://gagravarr.livejournal.com/143490.html" rel="nofollow">Part 1</a>, we learnt a little about what <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a> is and does. In this part, we&#8217;ll see what new features using Tika gives us in Cheetah.</p>
  <div style="height:1.4em"> </div>
  <p>On the metadata side, Tika delivers three important things for us. These are support for a wider range of formats (see below), enhanced ease of adding custom parsers (you can just spring in a bean with the class name of your parser and you&#8217;re done), and consistent metadata.</p>
  <p>This last one is less of an issue for Alfresco users than for many other users, but is a real issue for extractor developers. Within Alfresco, we always map the raw properties onto ones in the content model, but this is handled at the extractor level. As such, it shouldn&#8217;t matter to the user if one document format has a &#8220;last author&#8221;, another &#8220;last editor&#8221; and a third &#8220;last edited by&#8221;, they&#8217;ll all turn up in the same property in Alfresco. However, the extractor writer has to know about this, to provide the mapping, and this makes writing an extractor harder, and increases the chance of error.</p>
  <p>Within Tika, there is a set list of common <a href="http://tika.apache.org/0.7/api/org/apache/tika/metadata/Metadata.html" rel="nofollow">metadata</a> keys, and each Tika parser internally maps its properties onto these. As such, when you receive your metadata back from Tika, it all looks the same no matter what file you got it from. If the metadata is a date, then Tika will also take care of converting it to a common format, so you don&#8217;t have to worry about parsing a dozen different date representations.</p>
  <p>Finally, because the metadata is in a common format, we can more easily map it to the content model. Thus, in Alfresco 3.4, we see most of the common extractors have a wider range of metadata mappings to the content model as standard. One big example of this is in the case of images &#8211; EXIF tags are now automatically extracted and mapped onto the content model, and if the image was geotagged, then the location of the image is also mapped onto the content model. </p>
  <p>Give it a try &#8211; upload a geotagged image to Alfresco share in 3.4, and see all the new metadata that shows up such as the location, camera, focal length and more!</p>
  <div style="height:1.4em"> </div>
  <p>In the past, most text extractors that were used in Alfresco only able to produce plain text. However, all the Tika parsers generate XHTML sax events, and sure are able to produce not only plain text, but also HTML and XHTML. Also, since Tika the XHTML is a true XML document, we can make use of XSLT to chain transformations.</p>
  <p>The immediate benefit then is that all plain text content transformers that are powered by Tika can deliver an HTML version at no effort. Thus, HTML versions of PDFs, Word Documents etc can now be requested. </p>
  <p>(At the moment, the HTML generated is very clean, but not always all that complex. The Tika community is gradually improving the markup generated to include more meaning, especially semantic information, and Alfresco is pleased to be involved in this effort)</p>
  <p>Does being able to generate XHTML help that much? I&#8217;d say yes! With the forthcoming WCM Quick Start, we&#8217;ll shortly be adding some features around HTML versions of some kinds of uploaded documents. Using Tika, we were able to implement this feature very quickly, allowing us to concentrate the developer time on enhancing Tika. Next up, for some cases we wanted a whole XHTML document, and others we only wanted the body content. Using Tika and the SAX handlers, it&#8217;s a one line change to toggle between the whole document, or just the body contents, by picking a different transform handler. Finally, the output is XHTML, so for demo&#8217;s we&#8217;ve been able to use XSLT and E4X (from within a script action) to effortlessly manipulate the content.</p>
  <div style="height:1.4em"> </div>
  <p>Finally, as mentioned, using Tika delivers us support for a large number of new file formats. The current list of files supported via Tika is:</p>
  <ul>
  <li>Audio &#8211; wav, riff, midi</li>
  <li>DWG (CAD files)</li>
  <li>Epub</li>
  <li>RSS and ATOM feeds</li>
  <li>True Type Fonts</li>
  <li>HTML</li>
  <li>Image &#8211; JPEG, PNG, Gif, TIFF and Bitmap (including EXIF information where found)</li>
  <li>iWork (Keynote, Pages etc)</li>
  <li>Mbox mail</li>
  <li>Microsoft Office &#8211; Word, PowerPoint, Excel, Visio, Outlook, Publisher, Works</li>
  <li>Microsoft Office OOXML &#8211; Word (docx), PowerPoint (pptx), Excel (xlsx)</li>
  <li>MP3 (ID3 v1 and v2)</li>
  <li>CDF (scientific data)</li>
  <li>Open Document Format</li>
  <li>PDF</li>
  <li>Zip and Tar archives</li>
  <li>RDF</li>
  <li>Plain Text</li>
  <li>FLV Video</li>
  <li>XML</li>
  <li>Java class files</li>
  </ul>
  <p>What&#8217;s more, generally it&#8217;s just a case of dropping new Tika jars into Alfresco with little/no configuration changes, so we can look forward to easy addition of new formats with each new Alfresco release as the Tika support grows!</p>
  <div style="height:1.4em"> </div>
  <p>In <a href="http://gagravarr.livejournal.com/144097.html" rel="nofollow">Part 3</a>, we will look at mapping between Tika&#8217;s common metadata, and the Alfresco content model.</p>
  <p>More information on Tika and Alfresco is available on the <a href="http://wiki.alfresco.com/wiki/Content_Transformation_and_Metadata_Extraction_with_Apache_Tika" rel="nofollow">Alfresco Wiki</a>. Tika will also be discussed at the <a href="http://www.alfresco.com/about/events/devcon/" rel="nofollow">Alfresco Developer Conferences</a> in Paris and New York later this year.</p>


guid: https://gagravarr.livejournal.com/143490.html
url: https://gagravarr.livejournal.com/143490.html
title: Apache Tika and Alfresco – Part 1
published: 2013-01-30 14:58:32
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.365307
updated_at: 2019-10-27 00:00:11.156487
summary:
  <b><i>This post was originally written in September 2010, and was available at <a href='http://blogs.alfresco.com/wp/nickb/2010/09/24/apache-tika-and-alfresco-part-1/' rel='nofollow'>http://blogs.alfresco.com/wp/nickb/2010/09/24/apache-tika-and-alfresco-part-1/</a> - I have reposted it here as that blog is no longer online.</i></b><br /><br />For the forthcoming Project Cheetah release, there have been a number of improvements to <a href="http://wiki.alfresco.com/wiki/Metadata_Extraction" rel="nofollow">Metadata Extraction</a> and <a href="http://wiki.alfresco.com/wiki/Content_Transformations" rel="nofollow">Content Transformations</a>. These improvements have been delivered by using <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a> to power many of the standard extractors and transformers.<br /><br />In this series of blog posts, we&#8217;ll be looking at what <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a> is and what it does, how it fits into Alfresco, what new features it has delivered, how you can customise how Tika works, and how you can add new Tika parsers to easily support new formats.<br /><br />The idea for <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a> was hatched in 2006, largely from people involved in <a href="http://lucene.apache.org/" rel="nofollow">Apache Lucene</a>, who were struggling to sensibly index all of their documents. The project went through the <a href="http://incubator.apache.org/" rel="nofollow">Apache Incubator</a>, and after a period of time as a Lucene sub-project, in 2010 became it&#8217;s own top level Apache project. Tika is used by people indexing content, spidering the web, doing NLP and text processing, as well as with content repositories.<br /><br />For all these use cases, the problems are largely the same. You start with a number of documents in a variety of formats. You wish to know what they are, and hence which libraries may be useful in processing them. You then want to get some consistent metadata out of them, and possibly a rich textual representation of the content. You also probably wanted all of this yesterday!<br /><br />(As a side note, Alfresco users have historically been in a more fortunate position than most when faced with these challenges, as the Metadata Extractor and Content Transformation services have handled most of these for you.)<br /><h3>What services does Tika provide then? </h3>Firstly, Tika offers <a href="http://tika.apache.org/0.7/detection.html" rel="nofollow">content and language detection</a>. Through this, you can pass Tika a piece of unknown content, and get back information on what kind of file it is (eg pdf, docx), along with what language the text is written in (eg utf-8 english). Within Alfresco we tend to already know this information, so as yet don&#8217;t make much use of detection.<br /><br />Secondly, through the <a href="http://tika.apache.org/0.7/parser.html" rel="nofollow">parser</a> system, Tika provides access to the metadata of the document. You can use Tika to find out the last author of a word file, the title of an HTML page, or even the location where a geo-tagged image was taken. In addition, Tika provides a consistent view across the different format&#8217;s metadata, mapping internally from document specific to general metadata entries. As such, you don&#8217;t need to know if a format uses &#8220;last author&#8221;, &#8220;last editor&#8221; or &#8220;last edited by&#8221;, Tika instead always provides the same information. We&#8217;ll see more on using Tika for metadata in part 2.<br /><br />Thirdly, through the <a href="http://tika.apache.org/0.7/parser.html" rel="nofollow">parsers</a>, Tika provides access to the textual content of files. The text is available as plain text, html and xhtml, with the latter offering options for onward transformations through SAX and XSLT to additional representations. This can be used for full text indexing, for web previews, and much more. Again in part 2 we&#8217;ll see how this is being used in Alfresco.<br /><br />Finally, Tika provides access to the embedded resources within files. This could be 2 images embedded in a word document, or an excel spreadsheet held within an powerpoint file, or even half a dozen PDFs contained within a zip file. This is quite a new Tika feature, and we&#8217;ll hopefully be making more use of it in the future. For now, it offers the adventurous a consistent way to get at resources inside other files.<br /><br /><br />In <a href="http://gagravarr.livejournal.com/143623.html" rel="nofollow">Part 2</a>, we&#8217;ll look at the new features and support that Tika delivers to Cheetah.<br /><br />More information on Tika and Alfresco is available on the <a href="http://wiki.alfresco.com/wiki/Content_Transformation_and_Metadata_Extraction_with_Apache_Tika" rel="nofollow">Alfresco Wiki</a>. Tika will also be discussed at the <a href="http://www.alfresco.com/about/events/devcon/" rel="nofollow">Alfresco Developer Conferences</a> in Paris and New York later this year.


guid: https://gagravarr.livejournal.com/143215.html
url: https://gagravarr.livejournal.com/143215.html
title: Oxford Digital Switchover and DVB-Apps
published: 2011-09-28 14:18:33
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.367870
updated_at: 2019-10-27 00:00:11.158838
summary:
  If you use Linux with the DVB-Apps to watch TV, and you're in Oxford, then as of today (28th September 2011) you'll need to re-tune. (Well, everyone in Oxford has to too!)<br /><br />One slight snag is that the command line scanning tools (like <i>scan</i>) take an initial tuning file, and the one that ships in Debian / Ubuntu / Hg isn't yet updated.<br /><br />Armed with the data from <a href="http://www.ukfree.tv/txdetail.php?a=SP567105" rel="nofollow">ukfree.tv</a> and some trial and error, I&#39;ve got what looks to be the correct new tuning file. Debian / Ubuntu users will want to place this in <i>/usr/share/dvb/dvb-t/uk-Oxford</i> (other users wherever you want!) and then run a scan. (You could also use w_scan, which iterates each frequency in turn, but scan with a start file is normally much quicker)<br /><br /><pre style="overflow:scroll; height:14em">
  # Post-Switchover

  # Local, C51, details TBA, not yet transmitting as of Nov 2011
  T 713833000 8MHz 2/3 NONE QAM64 2k 1/32 NONE

  # PSB1 BBC-A, C53+. Apparently 730.2 but actually looks to be 730.167
  T 730167000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

  # ArqB (COM6), C55, 746.0
  T 746000000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

  # PSB3 BBC-B, C57, 256QAM DVB-T2
  T 762000000 8MHz 2/3 NONE QAM256 8k 1/32 NONE

  # ArqA (COM5), C59-, Apparently 777.8 but actually looks to be 777.833
  T 777833000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

  # PSB2, D3+4, C60-, Apparently 785.0 but actually looks to be 785.833
  T 785833000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

  # SDN (COM4), C62, 802.0
  T 802000000 8MHz 2/3 NONE QAM64 8k 1/32 NONE
  </pre><br />Note - as of 2011-12-08 these changes have been accepted into the upstream Linux Media repo. Depending on the age of your distro / checkout / etc, you may or may not need to update to the above by hand.<br /><br />Having done that, you should be able to run something like <i>scan -v -a 0 uk-Oxford &gt; new-channels.conf</i> to build your new channels.conf file. The result should be something like this:<br /><br /><pre style="overflow:scroll; height:16em">
  1-2-1 Dating:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:14816
  301:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:901:951:7168
  303:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:19584
  4Music:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:101:102:25664
  5*:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6673:6674:12928
  5 USA:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6689:6690:12992
  Absolute Radio:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6082:14688
  ADULT Babestn2:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:23872
  ADULT Babestn:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27904
  ADULT Blue:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24128
  ADULT Filth:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15728
  ADULT PARTY:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24256
  ADULT redhotTV:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15680
  ADULT Section:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27584
  ADULT Section:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27616
  ADULT Section:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15200
  ADULT Section:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15424
  ADULT smileTV2:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27520
  ADULT smileTV3:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:22400
  Al Jazeera Eng:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27712
  BBC 6 Music:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:432:18048
  BBC 6 Music:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1602:5760
  BBC Asian Net.:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:435:18240
  BBC Asian Net.:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1902:5952
  BBC FOUR:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:16832
  BBC FOUR:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:4544
  BBC NEWS:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:501:502:4352
  BBC ONE:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:101:102:4171
  BBC Parliament:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:205:421:17024
  BBC Parliament:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:601:602:4736
  BBC R1X:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:434:18176
  BBC R1X:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1802:5888
  BBC R5L:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:430:17920
  BBC R5L:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1402:5632
  BBC R5SX:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:431:17984
  BBC R5SX:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1502:5696
  BBC Radio 1:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:436:18496
  BBC Radio 1:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1002:6720
  BBC Radio 2:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:437:18560
  BBC Radio 2:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1102:6784
  BBC Radio 3:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:438:18624
  BBC Radio 3:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1202:6848
  BBC Radio 4:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:439:18688
  BBC Radio 4:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1302:6912
  BBC Radio 4 Ex:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:433:18112
  BBC Radio 4 Ex:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1702:5824
  BBC Red Button:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:4416
  BBC THREE:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:4288
  BBC TWO:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:202:4287
  BBC World Sv.:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:440:18304
  BBC World Sv.:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:2002:6016
  bid tv:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6273:6274:14272
  Big Deal:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24000
  Capital FM:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6114:14752
  CBBC Channel:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:301:302:4608
  CBeebies:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:401:16960
  CBeebies:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:401:402:4672
  Challenge:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6785:6786:15520
  Channel 4+1:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:580:581:8452
  Channel 4:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:560:561:8384
  Channel 5:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:540:541:8500
  Channel Zero:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28224
  CITV:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6833:6834:16032
  COMMUNITY:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24064
  CONNECT:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28352
  Create & Craft:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2301:2302:24384
  Dave:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:401:402:22272
  Dave ja vu:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:23936
  Daystar:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24320
  E4+1:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:501:502:22336
  E4:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:570:571:8448
  ESPN:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6801:6802:16096
  Film4:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:701:702:27136
  Food Network:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:23040
  Gay Rabbit:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:8577
  Gems TV:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2401:2402:24448
  G.O.L.D.:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15552
  Heart:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6098:14720
  heat:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1201:26240
  Home:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:14976
  Ideal World:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:501:502:25920
  ITV1 +1:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:600:601:8362
  ITV1:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:520:521:8263
  ITV2 +1:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6529:6530:15952
  ITV2:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:530:531:8325
  ITV3:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6881:6882:16048
  ITV4:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:601:602:28032
  Kerrang!:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1301:26304
  Kiss:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1101:26176
  Magic:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1801:26624
  More 4:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:590:591:8442
  Pick TV+1:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:202:22244
  Pick TV:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:301:302:22208
  Premier Radio:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1601:28160
  price-drop tv:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:611:612:22464
  Q:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1901:26688
  QUEST:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6929:6930:14498
  QVC:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6049:6050:13120
  QVC Beauty:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28288
  Rabbit:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:8642
  Really:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2111:2112:23712
  Rocks & Co 1:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2311:2312:27840
  Russia Today:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2101:2102:27456
  Sky News:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:101:102:22080
  Sky Sports 1:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:202:403:20160
  Sky Sports 2:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:204:411:20224
  Sky Text:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28096
  Smash Hits!:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1501:27040
  SMOOTH RADIO:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1401:26368
  STREAM:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28384
  talkSPORT:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1101:22592
  Teletext Hols:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6354:14784
  Television X:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15232
  The Hits Radio:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1701:26560
  TOPUP Anytime1:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:16224
  TOPUP Anytime3:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:16288
  VISION:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28320
  VIVA:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:202:25728
  Yesterday:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:301:302:25792
  </pre><br /><br />Finally, you'll want to do a full rescan in MythTV, which is a faff... :(


guid: https://gagravarr.livejournal.com/143000.html
url: https://gagravarr.livejournal.com/143000.html
title: Upcoming Conferences
published: 2011-08-28 13:43:44
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.370761
updated_at: 2019-10-27 00:00:11.161313
summary:
  For anyone who doesn't hawkishly follow my <a href="http://lanyrd.com/profile/gagravarr/future/" rel="nofollow">Lanyrd upcoming events page</a>, here's a quick round-up of some of the conferences I'm speaking at / helping with in the near future.<br /><br />Coming up in just under a fortnight (mental note - must finish my talk...)  is <a href="http://www.transfersummit.com/" rel="nofollow">Transfer Summit</a>, 7-8 September. Handily located in Keble, so no need to leave Oxford, this conference is back for it's 2nd year covering open source, open development, innovation and collaboration. Last year featured loads of very interesting talks, and this year's looking promising too. If you're interested in open source, open development, academic to business transitions for software, or just a fun conference in Oxford, come join us!<br /><br />The weekend following Transfer Summit we have <a href="http://barcamp.org/BarCampApacheOxford" rel="nofollow">BarCampApacheOxford</a>, once again being held in the University Club. This year we're on Sunday 11th September, with a meal the night before. As usual, we'll have a wide range of sessions, including (but not limited to!) ones on the Apache Software Foundation and it's projects. We'll also likely have things on mapping, Django, Arduino, OpenOffice, Paper Aeroplanes and listening to Satellites, so it should be a great mix! If you want to come, sign up on the wiki ASAP, so you can ensure you get one of the spots that includes a free lunch :)<br /><br />A month later, I'm off to Amsterdam for a few days. On Thursday 13th and Friday 14th October I'll be at <a href="http://gotocon.com/amsterdam-2011/" rel="nofollow">GotoCon Amsterdam</a>, mostly going to the great sessions on offer, but also talking as well. There's a <a href="http://gotocon.com/amsterdam-2011/schedule/index.jsp" rel="nofollow">good mix</a> of sessions across lots of different areas of software development, and we've an Apache track on there too on the Friday. If you are thinking of coming, you can use the code <b>burc200</b> for 20% off the registration price.<br /><br />On Saturday 15th, the day after GotoCon, we're also doing an <a href="http://wiki.apache.org/apachecon/AmsterdamHackathon2011" rel="nofollow">Apache Hackathon + mini BarCamp</a> in Amsterdam. Hippo are kindly hosting us, and should be a fun chance to get Apache types from Europe together for a chat, some coding, and hopefully also share some knowledge amongst ourselves. It's open to not only Apache committers, but also anyone who contributes code or documentation to Apache (basically anyone on a dev list), sign up on the wiki and come along if you're in Amsterdam that weekend!<br /><br />Next up is the <a href="http://www.amiando.com/alfresco-devcon-san-diego-2011.html" rel="nofollow">Alfresco DevCon 2011 USA</a> in San Diego. Not sure yet what I'm speaking on (though I think I'm down for a few sessions...). It's a chance to learn about developing solutions with and on top of Alfresco, in the lovely surroundings of San Diego, so who wouldn't want to go :)<br /><br />Finally (for now, but possibly not for this year...) is <a href="http://na11.apachecon.com/" rel="nofollow">ApacheCon NA 2011</a> in Vancouver, 7-11 November (plus hopefully a BarCamp on the 12th). We've an amazing set of speakers and talks lined up as usual, and it's looking promising for a great conference. Early bird rates end soon, so sign up!


guid: https://gagravarr.livejournal.com/142691.html
url: https://gagravarr.livejournal.com/142691.html
title: Fringe 2011
published: 2011-08-27 17:32:19
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.374019
updated_at: 2019-10-27 00:00:11.163533
summary:
  Last weekend I was up at the Edinburgh Fringe, and once again I managed to see quite a few shows, with a few gems amongst them. I <a href="http://twitter.com/gagravarr" rel="nofollow">tweeted</a> reviews of most of them at the time (where wifi permitted!), but this is aiming to be a slightly more expanded set. Mostly to help with picking shows next year, it must be said, but there is a day or two of the fringe left for other people to use them!<br /><br /><b>Stupid People Will Be Offended</b> - Free standup by Mikey Mileos at Dropkick Murphy's. We happened across this show when we got to The Vaults early, and nipped over the road for a drink. Only caught the first half of the show, but for free standup it was pretty good. I'd say probably half the jokes worked, which beats some paid-for fringe standup shows I've seen! Could be one to watch for next year, if he can improve over the year. <b>3*</b><br /><br /><b>Sherica</b> - <a href="http://www.shredproductions.co.uk/" rel="nofollow">Shred Productions</a> at Paradise in the Vault. Possibly summed up by the line early on: <i>Sorry. Stop ... This isn’t me. I don’t visit prostitutes. For God’s sake – I read the Guardian!</i>. This dark and thought-provoking piece follows the downward spiral after a teacher falls for a sex worker who turns out to be the sister of one of his pupils. Add in some blackmail, a new academy school struggling with the change, and it's a good piece. Not as good as <a href="http://www.thepaperbirds.com/" rel="nofollow">Paper Birds</a> were when they tackled a similar subject with <i>In A Thousand Pieces</i>, but good none the less. <b>4*</b><br /><br /><b>What You Will</b> - Shrewsbury School at C. Their own musical telling of Shakespeare's Twelfth Night worked out really well, with some good songs but still keeping the original a lot. Good lunchtime fun, <b>4*</b><br /><br /><b>Lethal Injection</b> - <a href="http://www.five-one-productions.com/" rel="nofollow">Five One Productions</a> at C ECA. A compelling, minimalist production about 4 men on death row, and their guard. As their time approaches, we learn of their crimes and how they ended there, all for very different reasons. <b>4*</b><br /><br /><b>Sanctuary</b> - Little Room Productions at Paradise in the Vault. A new opera set just before the apocalypse, as a father and daughter hide. Then others try to seek sanctuary with them. Very very operatic, which sometimes worked, but often got in the way. <b>3*</b><br /><br /><b>Matt Kirshen</b> at Underbelly. Self-deprecating stories from his life, with the odd bit of politics in there as well. Funny, smart and enjoyable comedy. <b>4*</b><br /><br /><b>Wasted Love</b> - One Academy Productions at C. This musical tells the story of a relationship support group, which despite how it may sounds was excellent. Great singing, great storytelling, heartbreak into uplifting redemption then sometimes tumbling back down again. This is what new and different musicals at the fringe should be like, <b>5*</b><br /><br /><b>Principal Parts</b> - <a href="http://www.striptheatre.co.uk/" rel="nofollow">Strip Theatre</a> at C soco. Surrealist telling of a group trying to assassinate Archduke Franz Ferdinand in Sarajevo, despite their incompetences. Good fun, <b>4*</b><br /><br /><b>Sunday in the Park with George</b> - One Academy Productions again (twice in an afternoon!) at C. A full 2 hour production of Sondheim's musical about Georges Seurat's painting. A good (but not amazing) production of this good (but not amazing) musical. <b>4*</b><br /><br /><b>Sammy J and Randy: Ricketts Lane</b> at Udderbelly Pasture. On top form again, this time Sammy J and his friend Randy (the foul mouthed purple puppet) are battling tax avoidance amidst their domestic misadventures. Silly tales, word plays, long running gags (can I get a pwappwa new guinea, pwappwa new guinea!), and generally very very very funny. <b>5*</b><br /><br /><b>Belt Up's Outland</b> - <a href="http://www.beltuptheatre.com/" rel="nofollow">Belt Up</a> in C Soco once again. A smaller set of shows to last year, and in a small and intimate setting for this immersive theatre production. This time it's related to Lewis Carroll's worlds and life. Similar to last year's The Boy James, more static and less interactive than some of their have been. Generally good, but one or two bits felt a little too similar to some of their past shows. <b>4*</b><br /><br /><b>Isy Suttie</b> at Pleasance was another funny, well done telling of the tale of her friend's relationship. Cheery comedy, with some good songs in it too, generally feel-good with laughs. <b>4*</b><br /><br /><b>Horne Section</b> was late night music and banter in the Spiegel Tent. Quite a bit of live jazz, with some standup and silly games thrown in, not to mention the crazy but great Jazz/Morris Dancing Crossover version of a Beyonce song! Some good guests too the night we saw it. <b>4*</b><br /><br /><b>Street Dreams</b> - <a href="http://www.littlecauliflower.co.uk/" rel="nofollow">Little Cauliflower</a> at Underbelly. A puppetry show, but not really as you know it. Instead, an amazingly done piece of abstract storytelling, as an old man goes about his day pestered and chased by a variety of everyday objects. Sweet, funny, sad, brilliantly done and engaging. A little hard to describe, true, but an enchanting hour not to be missed! <b>5*</b><br /><br /><b>Which One's Fergal</b> by Garrett Millerick at Just the Tonic at the Caves. A series of strong character sketches, satirical but well done, with a great mix of laughs and cringes. Garrett weaves his way between God's gift to Advertising Execs, a Traffic Warden, a failed motivational Hip-Hop singer and more. Biting but funny, a solid <b>4*</b><br /><br /><b>Flanders and Swan</b> - Tim FitzHigham and Duncan Walsh Atkins at the Pleasance. An excellent hour of Flanders and Swan songs and tales, with just the right amount of sillyness (the music stand french horn anyone?). <b>4*</b><br /><br /><b>Seagull Effect</b> - <a href="http://www.idlemotion.co.uk/" rel="nofollow">Idle Motion</a> at Zoo Roxy. The Oxford based group were back again, after last year's great <i>The Vanishing Horizon</i>, this time with a tale of the 1987 hurricane. Once again they had a great mix of theatre, multimedia, and the odd bit of dance and physical theatre. The stories of different people interleave with archive footage and sound, with stories swirling together as the storm approaches. Great stuff, but not quite as good as last year. <b>4*</b><br /><br /><b>Belt Up's 20 minutes to 9</b> - back to C Soco for another <a href="http://www.beltuptheatre.com/" rel="nofollow">Belt Up</a> production. This one-woman immersive show takes us through recollections, stories and social observations of an aging apparently well-off woman. At times funny, others odd, almost always engaging, but you're never entirely sure quite what's going on, in that way that Belt Up can do well. Great stuff, but not quite as good as they have been before. <b>4*</b><br /><br /><b>Sammy J</b> with Potentially at Underbelly is last but not least. Back again with stories, songs, and silly anecdotes that slowly stitch together to reveal the ridiculous but true linked narrative. Cruise Ships, Possums, a mystery man with a death wish in a red g-string, children's stories and a hell of a lot of laughs fill the hour. On top form again, <b>5*</b><br /><br />Only 19 shows this year, in just over 3 days, but I'm sure we'll get the show count back up again next year!


guid: https://gagravarr.livejournal.com/142470.html
url: https://gagravarr.livejournal.com/142470.html
title: Streetcar - first experiences
published: 2011-08-13 17:31:34
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.377053
updated_at: 2019-10-27 00:00:11.166018
summary:
  A bit over a month ago, the friend who was letting me borrow their parking space moved out of Oxford, and the cost of renting a space was a bit insane given how little I drive at the moment. So, I sadly sold my car (it's found a good new home down in Devon!), and I decided to get by with just renting cars as needed.<br /><br />(I had been hoping that <a href="http://www.commonwheels.org.uk/" rel="nofollow">CommonWheels</a> would be able to get a space near me, and either adopt my car or get a new one to go there, but alas they couldn't get the council to give them a parking space)<br /><br />CommonWheels don't have any cars near me, but <a href="http://www.streetcar.co.uk/join/invitation/?code=A53F-5FF" rel="nofollow">Streetcar</a> have 3 within a 10 minute walk of my flat, so that seemed a good bet for getting a car when I needed one, especially as they offer rentals as short as an hour, and have a nifty online booking system. I managed to find a signup voucher with £40 of credit (no longer available sadly, but <a href="http://www.streetcar.co.uk/join/invitation/?code=A53F-5FF" rel="nofollow">this link</a> will get you £20), and I figured that the £40 of credit would help offset the ~£60 signup fee for the year. <br /><br />As they said it might take a few days for the license check, I signed up just after selling the car, as I figured I should probably try to be organised and get it all sorted before I actually wanted to book something. As it turned out, the license check was completed very quickly (a quick conference call with streetcar and the DVLA and it was sorted), so I was actually good to go sooner than expected. This did present a slight snag, as the £40 "welcome credit" I'd got turned out to only be valid for 30 days, and not for the year as I'd expected... Luckily the streetcar team are nice and friendly, so when I emailed them about this they extended the validity of the welcome credit by a few more weeks, so it was still valid when I came to want to use the car for the first time.<br /><br />A little over a month after signing up, I finally had a spare weekend with a vague excuse to do something needing a car, so I booked the nearest car to me for as long as my welcome credit would allow. The car nearest to me is one of their swisher cars, a BMW, so I only got 9.30am - 3pm for my credit, rather than until 4pm if I'd gone for the Golf that's a few minutes further away. Still, I figured a BMW might be fun to drive, and it's nearer! I booked the car online, confirmed with my friends when I'd pick them up, got a friendly confirmation text and read their helpful instructions online about the car.<br /><br />Roll on to this morning, and it was time to give it a whirl. I got to the car just after half 9, eventually found it (not in it's space as the booking said, but not too far away), figured out where to wave my streetcard to open it (magic box on the dashboard), and unlocked the car. In the glove box was the magic widget as expected, I tapped in my pin and it gave me the keys. Then it was onto the first challenge, checking for damage.<br /><br />When I do a normal hire car, they normally have a sheet with all the existing damage marked on it (certainly with Hertz, less so with some of the others). It's then a quick walk around the car checking that everything's on there, then you're away. With Streetcar it's not so simple. In the instruction booklet is the damage log, but each bit of damage has it's own entry, and for this car the damage log went over about 5 pages. So, as I walked round the car, at each thing I spotted I had to check each (small) entry over many pages to see if it was mentioned, then move onto the next scratch and start again. It was really rather tedious to not have a summary, but have to check each thing in turn. Then the bad news, there were a couple of scratches not in the log. This meant a call to Streetcar to let them know, on their 0845 number (so instead of the call being free with inclusive minutes, I had to pay 35p/minute to call them, grr...). The call to let them know about the damage was fairly quick, but did have a rather long recorded message first, and as well as getting a damage reference for the log book I was also able to alert them to the fact that the car hadn't been left very clean inside as it was supposed to be. Having written the details in the log, I could finally hop in, adjust the mirrors and be off. I'd say the process took about twice as long from arriving to driving off as it's taken me the last few times I've got a car from Hertz, which is a bit pesky.<br /><br />The car itself was fine, nice to drive, and after picking up my friends and running some errands we headed south to visit a National Trust property (<a href="http://www.nationaltrust.org.uk/main/w-basildonpark" rel="nofollow">Basildon Park</a> near Reading, worth a visit!). Around 2pm we headed back, figuring an hour should be fine.<br /><br />One of the conditions of hiring a streetcar, along with not leaving it a mess, is that you need to return it with at least 1/4 of a tank of fuel. The car had only just had over a quarter of a tank when I collected it, which meant I was the one standing when the music stopped. The cars all come with a fuel card, so all you need to do is swing by a petrol station, fill it up, charge it to the fuel card and you're away. Not too bad, but it does take a bit of time to do, or slightly longer if you're like me and tried to go to a petrol station that wasn't where I thought it was, doh! This meant I was cutting it a bit fine to return the car for 3pm after all, so we tried texting (my passenger rather than me I should add!) to try to extend the booking by 30 minutes. A rather nifty feature I must say, just a shame someone else had booked it for 3pm... At 2.58 when we were still a few minutes off making it to the carpark we rung up to say we'd be late, and were told that they'd let the next person know but we had a few minutes grace. On returning to the carpark at about 3.03 we found someone else had parked in the streetcar space (it has a sign and everything!), so we had to find another space and ring customer services once again to let them know where the car was. Just as we were leaving the carpark we spotted the next people getting to the car, sorry for making you a minute or two late whoever you were...!<br /><br />Finally at home I checked my account to see the cost. As we'd driven to near Reading and back, we'd clocked up 61 miles, of which only 20 were included in the hire. That means there was an extra charge of £9.43 (it's 23p/mile, as shown on their <a href="http://www.streetcar.co.uk/pricing/default.aspx" rel="nofollow">pricing page</a>). Having filled the tank up I know that's more than the cost of the diesel, but not too much more and they've a business to run. I'd also got a £3.62 "Unauthorized Extension" charge, I guess for being a few minutes late. Total cost was £52.93, or £12.93 after the credit. If I'd gone for the VW Polo that's a 10 minute walk from my flat (rather than the BMW at 5 minutes), it would only have been £41.81 for the 5.5 hours.<br /><br />Out of interest, I checked with Hertz about how much it'd cost to hire from them. They're just over a 10 minute walk from my flat (just past the Streetcar VW Polo), so if I was to go for the cheapest streetcar they're probably a fair comparison. On a weekend, they're only open on Saturday mornings, so not as convenient. However, to hire a car slightly bigger than a VW Polo from Saturday morning to Monday morning, their website quoted me £43.60 (plus any petrol used). For the same journey I'd just done, that would mean a cost of about £50, but instead of only having the car for 5.5 hours, I'd not have had to worry about being late back, and I'd have the car for all of tomorrow too. Streetcar want £105 for to have their polo for an equivalent 48 hours, that's with 40 miles included, which seems rather bad value. In the week Hertz are open a lot more, and they only want £22.40 for a 24 hour hire (plus petrol). Add in a bit of fuel, and any hire over about 3 hours looks to be cheaper to go with Hertz, which was a surprise to discover. That's not even taking account of the £60/year Streetcar fee.<br /><br />Overall, the process with Streetcar was ok, but averages out not quite as quick and easy as Hertz, but certainly better than other car hire places I've used. Cost wise, if you just want a car for an hour or two, it's handy and (fairly) quick and better value than a taxi. For me though, any hire over a few hours and I'm better off just going with Hertz for the day, as long as I'm not doing a last minute night or Sunday booking. It's a bit disappointing to discover this, but at least now I know. I'm sure I'll be using my Streetcar membership a lot more this coming year, just only for short trips or last minute things, otherwise you'll find me heading to Hertz for a longer hire, or on the train!


guid: https://gagravarr.livejournal.com/142108.html
url: https://gagravarr.livejournal.com/142108.html
title: The Microsoft Binary File Format Validator
published: 2011-07-18 21:45:08
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.379723
updated_at: 2019-10-27 00:00:11.168896
summary:
  I first got involved in <a href="http://poi.apache.org/" rel="nofollow">Apache POI</a> back in November 2003 (or so a check of the archives tell me!), which is almost 8 years ago as a write this... Back then, documentation of the Microsoft Binary File Formats was pretty slim on the ground, and there was quite a bit of reverse engineering needed. If you'd told me then that in 2010 I'd be in Seattle at a Microsoft event on the public documentation on the file formats, I wouldn't have believed yet! Despite this, the <a href="http://msdn.microsoft.com/en-us/library/cc313105%28v=office.12%29.aspx" rel="nofollow">Documentation is public</a> (and has been for some time), and at the end of last year I did head out to Seattle for the <a href="http://blogs.msdn.com/b/officeinteroperability/archive/2010/10/25/office-binary-file-format-plugfest-2010-recap.aspx" rel="nofollow">Office Binary File Format Plugfest</a>.<br /><br />One of the neat things about the plugfest (beyond learning lots, meeting other people developing similar libraries to read the file formats, and reporting docs bugs in person to their authors) was the <a href="http://blogs.msdn.com/b/officeinteroperability/archive/2011/07/12/microsoft-office-binary-file-format-validator-is-now-available.aspx" rel="nofollow">Binary File Format Validator Tool</a>. One of the few frustrating things was that I couldn't share the tool with anyone else... Good news though, as it's now out of private beta, and you can go learn about it and grab a copy from <a href="http://blogs.msdn.com/b/officeinteroperability/archive/2011/07/12/microsoft-office-binary-file-format-validator-is-now-available.aspx" rel="nofollow">here</a>! (It's technically Windows only, but it seems to work just fine under Wine)<br /><br />Short term, I think one of the biggest uses for the BFF tool for POI will be for those dreaded bug reports of "I've got a file that seems to show a bug, but I can't let you have the file". Hopefully now the user will be able to run the BFF against their input file, and check it's valid. If it's not, then we know it's probably not our fault that it can't be read, and the user can go and speak to whoever wrote the software that generated it. In the case that the input's fine, the BFF can be run on the POI generated output. If this flag's a problem, then it'll hopefully come with the nifty details and docs references to help us fix it. No more "it doesn't work but I don't know why", and instead "option 5 should only be 0x3 when there are fewer than 32 entries, then it has to be 0x7, but POI has 0x3 hard coded". Link to the spec, a link to the problem code, and maybe even a patch too. Well, we can dream.... :)<br /><br />Longer term, we'll hopefully hook up the BFF to some sort of integration testing, to be run every once in a while, to alert us to when we're generating files with issues. That said, the BFF is stricted than Office is, so we'll need to fix up all the cases where Office lets us get away with something the spec doesn't technically allow, so we're warning free. This may take a little while to work through, so volunteers and patches are <a href="http://poi.apache.org/mailinglists.html" rel="nofollow">very welcome</a>, please <a href="http://poi.apache.org/guidelines.html" rel="nofollow">come help us</a>!


guid: https://gagravarr.livejournal.com/141954.html
url: https://gagravarr.livejournal.com/141954.html
title: BarCampApacheSydney
published: 2010-11-16 11:26:28
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.382247
updated_at: 2019-10-27 00:00:11.171304
summary:
  Having had a great time helping to organise a <a href="http://barcamp.org/" rel="nofollow">BarCamp</a> in Oxford in June as part of <a href="http://www.transfersummit.com/" rel="nofollow">Transfer Summit</a>, another in Atlanta as part of <a href="http://www.apachecon.com" rel="nofollow">ApacheCon</a>, plus a few at Apache Retreats, it shouldn't be a bit surprise to hear that I'm helping with another in December.<br /><br />What might be a bit different about this next on is that it'll be in Sydney! Yes, by popular demand, and thanks to several excellent and hard working volunteers on the ground, we're going to be having our first <a href="http://www.apache.org/" rel="nofollow">Apache</a> event in Australia!<br /><br /><a href="http://barcamp.org/BarCampApacheSydney" rel="nofollow">BarCampApacheSyndey</a> is to be held on <b>Saturday 11th December</b>, at the <a href="http://www.facilities.usyd.edu.au/fmo/services/venue.shtml" rel="nofollow">The Darling Centre</a> at the University of Sydney. In keeping with the Oxford BarCamps, we'll be doing a meal on the Friday night, then a full day's BarCamping on the Saturday.<br /><br />If you're going to be in (or near!) Syndey around the weekend of Saturday 11th December, you can get more details on the event, plus sign up, on the website: <a href="http://barcamp.org/BarCampApacheSydney" rel="nofollow">http://barcamp.org/BarCampApacheSydney</a>. See you then!


guid: https://gagravarr.livejournal.com/141704.html
url: https://gagravarr.livejournal.com/141704.html
title: Upcoming Conferences
published: 2010-09-21 15:00:29
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.384762
updated_at: 2019-10-27 00:00:11.173478
summary:
  This weekend I was at the Apache Retreat in Hursley, near Winchester. As you would have hoped from a weekend-long conference with Tipis and the ground floor of a former stately home, it was a great event. See the <a href="http://www.flickr.com/photos/zoeslattery/sets/72157624865092805/" rel="nofollow">photos</a> for yourself! Several people asked me about what other conferences I was doing this year, so I thought I'd do a quick write up.<br /><br /><br />First up is the <a href="http://www.alfresco.com/about/events/devcon/" rel="nofollow">Alfresco Developer Conference</a> in Paris, which is the 20th and 21st of October. It looks like I'm only going to be there for the Thursday now (owing to a Microsoft binary formats event), which is a shame as it's looking like a great two days. I'll be talking about two related Alfresco services, the <a href="http://wiki.alfresco.com/wiki/Content_Transformations" rel="nofollow">Content Transformation Service</a> and <a href="http://wiki.alfresco.com/wiki/Metadata_Extraction" rel="nofollow">Metadata Extractor Service</a>. I'll be looking at what these services do, how to make better use of them, and how to extend them. Quite a bit of this will be around the <a href="http://wiki.alfresco.com/wiki/Content_Transformation_and_Metadata_Extraction_with_Apache_Tika" rel="nofollow">new work</a> for 3.4 which uses <a href="http://tika.apache.org/" rel="nofollow">Apache Tika</a> to deliver support for a large number of new formats.<br /><br />For more information on my talk, you can watch the <a href="http://www.youtube.com/watch?v=HQ7KJZ7fSrc&amp;feature=player_embedded" rel="nofollow">video interview</a> I've done as part of the conference promotion. More information on the event is at <a href="http://www.alfresco.com/about/events/devcon/" rel="nofollow">http://www.alfresco.com/about/events/devcon/</a><br /><br /><br />Two weeks later is <a href="http://www.apachecon.com/" rel="nofollow">ApacheCon</a> in Atlanta. ApacheCon runs for the whole week, from Monday 1st to Friday 5th November (which means I can't be in New York for the 2nd Alfresco DevConf, why do these things always have to clash?). It's looking like a great conference, and I'm going to be quite busy there, talking, organising and attending! The Monday and Tuesday see the <a href="http://barcamp.org/" rel="nofollow">BarCamp</a> un-conference, which should feature loads of great sessions and discussions on new and existing Apache technologies.<br /><br />The main sessions run Wednesday - Friday, and here I'm running the "Content Technologies" track. I'm kicking off the track with an <a href="http://na.apachecon.com/c/acna2010/sessions/589" rel="nofollow">overview session</a>, which should give an introduction to a lot of the many and varied content related projects at Apache. Hopefully, it'll also whet the appetite of people for the rest of the track! The track will then continue with talks on various content areas, from the new CMIS standard + <a href="http://incubator.apache.org/chemistry/" rel="nofollow">Apache Chemistry</a> project which helps you build applications backed by content stores, through <a href="http://pdfbox.apache.org/" rel="nofollow">PDFBox</a> and <a href="http://poi.apache.org/" rel="nofollow">POI</a>, to how convert your application to be content driven. The chances are, your application deals with content somewhere, so come along and learn how Apache projects can help you!<br /><br /><br />Finally, I'm hoping to help with running an Apache BarCamp in Sydney in December. The Apache committers in Sydney are currently in the final stages of booking the venue, so hopefully this can be confirmed and announced shortly...


guid: https://gagravarr.livejournal.com/141316.html
url: https://gagravarr.livejournal.com/141316.html
title: Fringe 2010
published: 2010-09-05 21:37:02
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.387346
updated_at: 2019-10-27 00:00:11.175663
summary:
  <i>Note - This post has taken a little longer than planned in the writing....</i><br /><br />After a couple of days hiking near Glen Affric, I made it to the fringe for the Friday afternoon. My first few ideas for what to see were thwarted by shows having sold out, but by just gone 4pm I'd started my fringe trip for 2010!<br /><br /><b>Friday</b><br /><br />First up was <b>The Leeds Tealights</b> at Underbelly, which was a random picked based on the 4* reviews on the fliers. It was a 4 person sketch show, 3 guys and a girl, which set up the dynamic for a fair few of the skits. Most of the skits were only a few minutes long, so they got through a lot of material during the hour. A couple of them were absolute gems, most were fairly good for most of the time, and only a few fell completely flat... Overall, I'd give it <b>3*</b>, and it was a fun and lighthearted start to the fringe!<br /><br />Next up was the stunning <b>Tabu</b> out on Leith Walk. The non-central location allowed them to have a large big top as their venue, setup for standing rather than seating. It was billed as a circus show, but turned out to be so much more. Firstly, there was the live band, providing a musical backdrop to the show, often accompanied on vocals by the performers not doing acrobatics at that moment. Secondly, it wasn't just the (amazing!) acrobatics, the pieces changed and were woven together which each other, and the music, to provide a narrative for the show. It wasn't a static show either, instead the whole audience was moved about between pieces to create the appropriate gaps and spaces for the next set. Finally, the acrobatics themselves - stunning! Very varied, covering most major circus and acrobatic styles, stunningly well executed. It was a wider range of things than Circa (from last year), so not every piece ended up as developed as in that, but with the music and the thematic links between pieces, it was a wonderful performance. <b>5*</b><br /><br /><br /><b>Saturday</b><br /><br />My 2nd day at the fringe kicked off with <b>Shakespeare For Breakfast</b> at C. With free croissants, and being one of the few early starting plays, this is a usual favourite of ours. This year source of inspiration was King Lear, and their modern, silly take on it worked quite well. However, it's by no means the best production they've managed in all the years we've seen them... Good start to the day though, <b>3*</b><br />Was a zany version of King Lear, solid 3*, but last year was better<br /><br />This is Belt Up's third year at the fringe (2 years ago they did The Red Room, and last year The Squat as C Soco). They were back at C Soco again this year, with a series of immersive theatre set in The Room Above. The first of their shows that we saw was <b>The Boy James</b>, which was a very intimate and immersive show. A boy, his older friend/self, the joys of childhood adventures shared, and the loss of growing up. Amazing stuff, an excellent <b>5*</b><br /><br />Next up was another Belt Up show, <b>The Second Star to the Right</b>, carrying on with the JM Barrie theme. It was an almost hypnotic at times performance, with some lovely dance-like parts, as 4 actresses took us through bits of the Peter Pan story. <b>4*</b><br /><br /><b>Roam</b> at Zoo Southside was a piece of modern dance by a company we'd seen in previous years. Unfortunately, this year's show had no real narrative, but that did still leave us with some amazing dancing and great music! The dance was great as ever, <b>4*</b><br /><br /><b>Vanishing Horizon</b> at Zoo was largely back to theatre, though with some almost dance-like physical elements in places. A variety of interwoven stories covered the history of women aviators, family discovery, and the perils of trying to make a radio play... The stories fitted well together, and apart from a few iffy timings in the fictional bits (the historic factual parts seemed fine!), worked well. There was frequent, very impressive use of a large number of suitcases as props, and great physical parts using them to link everything together. Interesting and engaging theatre, <b>5*</b><br /><br />For a lighter note, we opted for <b>Amateur Transplants</b> again this year, and were rewarded as they were on top form! This year's show featured both of them, and focused more on the excellent shorter funny songs, rather than fewer longer works. As ever, it was dark and silly medic humour, but hilariously done! No-one left through being offended either... Back up to their previous <b>5*</b> standard again.<br /><br />Finally was <b>Geraldine Quinn</b> (<a href="http://twitter.com/geraldinequinn" rel="nofollow">@geraldinequinn</a>), for some musical Australian comedy. The show was about modern pop music, and featured some excellent, cutting songs and dance routines, exposing and ridiculing the faults of the pop industry. <b>4*</b><br /><br /><br /><b>Sunday</b><br /><br />Carrying on with Belt Up shows, we began Sunday with<b>Belt Up's Oddssey</b>. This was another excellent immersive piece, in one of their smaller venues, and powerfully done. However, while the re-setting into a dystopian future was good, it did sometimes make it tricky to entirely follow, so largely one for people who already know the original! <b>4*</b><br /><br /><b>Hunchback of Notere Damn</b> at Pleasance wasn't the only production of the story running, but was one of the good ones! Set in a smallish venue, it was a one man show, a powerful portrail of a poignant tale, <b>4*</b><br /><br />More Belt up, this time <b>Belt Up's Metamorphosis</b>, with their take on Kafka's tale. It was interesting staging, but I didn't find it as well translated as last year's "The Trial", so it ended up being that little bit to odd for my tastes... (I realise that Kafka isn't supposed to be normal, but still!) I rate it <b>3*</b>, but many others in my group gave it 5, take of that what you will!<br /><br />Over to Pleasance for our first of two boxing shows - <b>Beautiful Burnout</b>. This proved to be an excellent mix of story, monologues, group pieces, physical theatre and some dance-like fights. It did a great job of showing all sides of it, and worked well no matter what your knowledge or feelings on boxing were to start. An excellent <b>5*</b><br /><br />Another fun Australian musical comedy followed, in the form of <b>Sammy J</b>. This was our third time seeing him, and he remained as great as ever. True, you often question his sanity, but you laugh all the same! Songs and stories, probably almost all true, of his more recent past (last year's show did the earlier parts), and very funny. <b>5*</b><br /><br /><b>Putting It Together by Sondheim</b> was a late night almost-musical show. Rather than being a full Sondheim work, it was a collection of his songs around the themes of love and marriage. The songs were done well, but the pick'n'mix feel that went with the lack of any real narrative was the slight snag. <b>4*</b> <br /><br /><br /><b>Monday</b><br /><br />An early start, and for a full length play - <b>History Boys</b> at Greenside (again one of several productions). As one would have hoped from the reviews from previous years, this was a good story well told. The direction largely worked well, and it was a solid <b>4*</b> production.<br /><br />Then for something a bit darker - <b>Blackout</b> at Underbelly. This dark, depressing, excellent piece of physical theatre told of one young man's journey to mindless violence. Based on a real story and real interviews, it was dark but great. <b>4*</b><br /><br /><b>Pluck</b> were back again this year, relocated to the Gilded Balloon. Once again the trio impressed us with some crazy slapstick whilst managing a fairly decent rendition of a classical music concert at the same time! It wasn't quite as good as their previous show, but is was still good fun - <b>4*</b><br /><br /><b>Belt Up's Lorca is Dead</b> was I think their best show of the fringe. Belt Up are always surreal to some extent, but the chance to play the Paris set of surrealists, debating and retelling the life and death of Lorca really lets them into their own. Strange, but great! <b>5*</b><br /><br />More Belt Up followed, with <b>Belt Up's Atrium</b>. More great immersive stuff, weaving in and out of the different fantasies of a dying writer dictating his memoirs. Some great silly costumes too, but I must remember not to be a tiny bit more reserved when sat at the front, lest anyone have to see another one of my 5 second Kafka looks...! Good show though, <b>5*</b>  <br /><br /><br /><b>Tuesday</b><br /><br />Ovid's Metamorphoses at <a href='https://www.livejournal.com/rsearch/?tags=%23edfringe'>#edfringe</a>, told through the medium of 40s music, a few great bits with puppets, and some fun physical theatre! 4*<br /><br /><b>Others</b> by <a href="http://www.thepaperbirds.com/" rel="nofollow">Paper Birds</a>, who for the previous two years did the stunningly powerful and amazing In A Thousand Pieces. This year they'd sent letters, and then questionnaires to a number of women they felt to be interesting, and different `(other) to them. The play again featured three actresses, moderately minimal props, but this time was light-hearted for parts.  It was made up of the letters sent and received, the responses not received, the actress's ideas of what the responses might be, and the discussions that led from them. Parts showed the similarities between apparently different women, parts the unexpected differences, and parts the terrifying cluelessness of the actresses (I think, well, hope, largely hammed up!). In the end, it was a lighter piece than last year, but ended up not quite so powerful and thought-provoking. Still great stuff though, <b>4*</b><br /><br /><b>Hamlet! The Musical</b> - You'd probably expect this to be dreadful, but it actually turns out to be silly and quite good fun! <b>3*</b><br /><br /><b>NewsRevue</b> were on fairly good form once again. A few of the tory jokes seemed a bit rehashed from the 80s, but the Nick Clegg puppet worked well! Throw in some good characatures, and several Glee related songs, and you have a fun hour. <b>3*</b><br /><br />Finally the last show of the Fringe - <b>Shadow Boxing</b>. This was a one man show, about not only being a boxer, but the personal growth and discovery that lay around it, and with some good twists as it went along. Not as good as Beautiful Burnout, but a solid <b>4*</b> none the less.<br /><br /><br />And after 25 shows, that's it! During the course of the fringe, I <a href="http://twitter.com/gagravarr" rel="nofollow">tweeted</a> short reviews, which seemed to go well (based on in-person feedback from many of my friends, and from twitter comments), so I'll probably do that again next year. The above are hopefully longer and more thought out reviews, though as they vary in being written between 1 hour and 2 weeks after the event, they do vary somewhat...!


guid: https://gagravarr.livejournal.com/141140.html
url: https://gagravarr.livejournal.com/141140.html
title: Playing BBC HD and ITV HD on a DVBWorld DW2104
published: 2010-06-12 17:07:59
fetched: 2019-10-27 00:00:10.984140
created_at: 2019-06-09 18:11:41.390196
updated_at: 2019-10-27 00:00:11.178016
summary:
  Having got fed up with the ropey reception on Freeview, due to ongoing problems with my local transmitter (engineering works, fires, too windy for helicopters, the list of excuses is impressive...), I decided to pick up a DVB-S adapter for my mythtv box.<br /><br />The snag with having a very small, low power mythtv machine is that all the adapters need to be USB, which does limit the choices of linux compatible DVB-S adapters. However, a bit of browsing of the Linux TV wiki and some patience with ebay searches paid off, and I was able to pick up a <a href="http://www.linuxtv.org/wiki/index.php/DVBWorld_HD_2104_CA%2BCI_USB_Box" rel="nofollow">DVBWorld HD 2104</a>.<br /><br />With the help of <a href="http://chiana.homelinux.org/~marc/DVBWorld.html" rel="nofollow">this site</a>, I was able to get the firmware for it, and it was quickly up and running. I followed <a href="http://parker1.co.uk/mythtv_freesat.php" rel="nofollow">this guide</a> to get things running, which largely worked for the SD channels.<br /><br />However, despite the scan of the Astra 28.2E and Eurobird 1 satellites showing me both BBC HD and ITV HD, I was unable to get either working with MythTV or mplayer. Trying in mplayer I saw:<br /><pre>nick@minimyth:/dev/dvb/adapter1$ mplayer dvb://2@"BBC HD"
  MPlayer UNKNOWN-4.4.3 (C) 2000-2010 MPlayer Team
  Can't open joystick device /dev/input/js0: No such file or directory
  Can't init input joystick
  mplayer: could not connect to socket
  mplayer: No such file or directory
  Failed to open LIRC support. You will not be able to use your remote control.

  Playing dvb://2@BBC HD.
  dvb_tune Freq: 10847000
  Cache fill: 18.85% (1581056 bytes)   
  TS file format detected.
  VIDEO MPEG2(pid=5500) AUDIO MPA(pid=5502) NO SUBS (yet)!  PROGRAM N. 0</pre><br />But nothing played. Using szap myself to capture a bit of the stream myself, and trying with that, I got sound but no picture:<pre>mplayer /tmp/HD
  MPlayer UNKNOWN-4.4.3 (C) 2000-2010 MPlayer Team
  Can't open joystick device /dev/input/js0: No such file or directory
  Can't init input joystick
  mplayer: could not connect to socket
  mplayer: No such file or directory
  Failed to open LIRC support. You will not be able to use your remote control.

  Playing /tmp/HD.
  Cache fill:  0.00% (0 bytes)   
  TS file format detected.
  VIDEO MPEG2(pid=5500) AUDIO MPA(pid=5502) NO SUBS (yet)!  PROGRAM N. 0
  MPEG: FATAL: EOF while searching for sequence header.
  Video: Cannot read properties.
  ==========================================================================
  Opening audio decoder: [mp3lib] MPEG layer-2, layer-3
  AUDIO: 48000 Hz, 2 ch, s16le, 256.0 kbit/16.67% (ratio: 32000-&gt;192000)
  Selected audio codec: [mp3] afm: mp3lib (mp3lib MPEG layer-2, layer-3)
  ==========================================================================
  AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
  Video: no video
  Starting playback...</pre><br /><br />The key thing to spot here is that mplayer thinks it has MPEG2 video. However, both BBC HD and ITV HD are H264 when broadcast over Freesat! After some googling, it turns out that there's something up with how the dvb-s scan program outputs the channel lines. The easy option is to tell mplayer to use a workaround for this, by passing in an extra option to mplayer - <i>-demuxer lavf</i><br /><br />Thus, the easy way to play BBC HD on the DVBWorld card (my 2nd DVB adapter), the command is:<pre>mplayer -demuxer lavf dvb://2@"BBC HD"</pre><br />This largely seems to work fine, though the sound sometimes drifts which needs a quit and restart to work.<br /><br />However, it is possible to also hack channels.conf to contain the correct details to "just work(TM)". This seems a bit black magic, but you need to run mplayer with "-identify", pick the PMT_PID (via trial and error...), and add this into the channels.conf video entry with a plus. Thus, my channels.conf entries for the HD freesat channels are:<pre>BBC HD:10847:v:0:22000:5500+259:5502:6940
  ITV1 HD:10832:h:0:22000:2362+288:2369:10000
  ITV1 HD:10935:v:0:22000:513+289:641:3851</pre><br />By adding in the correct PMT PIDs, I can then just do <i>mplayer dvb://2@"BBC HD"</i> and it picks the correct video and starts playing! Still has sound drifts though, which I think might be slightly worse than with lavf but I've yet to double-blind test...<br /><br />Next up, time to make MythTV believe the stream is H264 too!


guid: https://blog.johnmckerrell.com/2017/11/03/year-notes-2015
url: https://blog.johnmckerrell.com/2017/11/03/year-notes-2015/
title: Year Notes: 2015
updated: 2017-11-03 19:50:04
published: 2017-11-03 19:50:04
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.918792
updated_at: 2019-06-09 18:11:41.918792
summary:
  Last year’s notes ended with a cliff hanger, would I take a permanent job at Axon Vibe or would I not? Well the year began straight away with a visit to head office. Flying out to Lucerne in Switzerland on the fifth of January. This was actually my wife’s first day back in work after being off for over a year’s maternity leave. It was unfortunate to have to leave her to sort out our daughter during her first week but I did at least get a later flight so that I could help get Evie to her first morning in nursery.

content:
  <p>Last year’s notes ended with a cliff hanger, would I take a permanent job at Axon Vibe or would I not? Well the year began straight away with a visit to head office. Flying out to Lucerne in Switzerland on the fifth of January. This was actually my wife’s first day back in work after being off for over a year’s maternity leave. It was unfortunate to have to leave her to sort out our daughter during her first week but I did at least get a later flight so that I could help get Evie to her first morning in nursery.</p>

  <p><img src="https://pbs.twimg.com/media/B6vO8QbCMAEF8KK.jpg" alt="View from the office terrace" /></p>

  <p>After much consideration I ultimately decided to take the job at Axon Vibe. I was intending to continue working for them so it’s not like I was looking for other clients, and the money was such that I wasn’t losing out by being a salaried worker. By this point I’d already agreed that I would have my daughter on Friday afternoons so I ultimately decide to work Monday - Thursday but to keep Friday for myself. That would allow me to keep working on my own apps or even doing small amounts of client work on Friday mornings. The view from the terrace (see above) certainly didn’t hurt either!</p>

  <p>I wasn’t going to mention too much about what we did at Axon Vibe that year as very little of what we did then has survived. We took some time to work out what our key offering would be, sometimes focussing on business to business applications, and at others trying to develop public facing apps. One interesting thing was that we took on a number of developers and a designer who all worked from DoES Liverpool. By the end of the year we actually had 4 people all working from Liverpool. That was great to see, even if I did find it a little strange sharing an office with colleagues! We also enjoyed a great week in October when we managed to persuade colleagues from Cirencester, Norwich and Edinburgh to visit while we worked on new developments.</p>

  <p>Way back in 2011 I’d actually taken over the running of a monthly event that would have geeks meeting up in a local bar on the last Tuesday of the month either just for a social meet-up or to hear someone talking about some technology or, well, anything that happened to interest them. GeekUp was a venerable event that started way back in <a href="http://geekup.org/events/1/">December 2005</a> in Manchester. Over time the event became more popular and drew in people from surrounding towns, Liverpool included. I attended my first event in <a href="http://geekup.org/events/20/">March 2017</a>, gave a talk on OpenStreetMap at the <a href="http://geekup.org/events/21/">following event</a> and then was glad to attend the <a href="http://geekup.org/events/22/">inaugural event in Liverpool</a> later that month.</p>

  <p>Running for so many years, GeekUp was such a great way for similarly minded folks in Liverpool to connect. Most of my friendships in Liverpool can be traced back to GeekUp, especially if you consider that I would never have co-founded DoES Liverpool if I hadn’t met Adrian McEwen there.  While I enjoyed attending GeekUp I found I wasn’t the best at running it. It was easier when we met in a bar as if there was no talk you could easily just chat and have a drink. It’s amazing how a simple schedule of “last Tuesday of the month” and perhaps the odd “Are you going to GeekUp?” on twitter would keep the event going quite well. GeekUp ran for four years meeting in 3345 (now “The Attic”) on Parr Street. On one of the last meetings there we found the room had been double booked (I’m sure with UKIP!) and it turned out we no longer had the booking at all. As we were in the process of setting up DoES Liverpool, and we were planning to host events there, we had a single meeting in Leaf on Bold Street, before moving the event to DoES.</p>

  <p>This change really affected the dynamic of the event, it was more difficult to get into the space and when you arrived you found a bland room of desks so it was quite different to meeting in a bar. We actually managed to continue for four more years but I struggled to get around to finding people to talk. Then in 2014 when I did manage to arrange and promote in advance a great line-up of speakers, I struggled to attract an audience! In the end I decided that life was complicated enough without the monthly stress of finding a speaker so 2015 was the year I brought the whole thing to an end. Although I don’t really get time to go out so much these days it does seem like Liverpool is missing a general geeky social meet like this now so it’s a shame there is no GeekUp, but there’s nothing saying someone couldn’t take it on again in the future!</p>

  <p><a href="https://twitter.com/mcknut/statuses/562163781703569408">I continued with my running this year too</a>. Living just 4km away from DoES Liverpool it’s actually an easy and fairly short run to get into the office and I’ve tried at times to make it my primary way of commuting. I haven’t managed it often but have a few times managed to <a href="https://twitter.com/mcknut/statuses/573894268856705025">run there and back for the four days I would be in the office</a>. No Half Marathon this year but I signed up for the Spring 10K around Sefton Park and managed to beat my personal best of around 46 minutes, I wasn’t too confident as I hadn’t done much speed training but was very happy when I blew almost 3 minutes off my record!</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Ok, last tweet on the subject, official results are in, 43:11 :-D <a href="http://t.co/Q9bULaos6D">http://t.co/Q9bULaos6D</a> <a href="http://t.co/o25JdXd2Yb">pic.twitter.com/o25JdXd2Yb</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/595134106230480896?ref_src=twsrc%5Etfw">May 4, 2015</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

  <p>I tried to take up <a href="https://twitter.com/mcknut/statuses/576422500130811904">gardening</a> as a hobby to brighten the place up. I even <a href="https://twitter.com/mcknut/statuses/582879965143810048">planted potatoes</a> so that we’d get greenery <em>and</em> useful, tasty potatoes. Though at times the potatoes seemed like they were trying to take over the DoES Liverpool meeting room we didn’t really have a very prosperous harvest and it felt like the time and effort could be better placed!</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">.<a href="https://twitter.com/DoESLiverpool?ref_src=twsrc%5Etfw">@DoESLiverpool</a> potato update: Growing pretty well! Should be harvesting in the next few weeks <a href="https://twitter.com/hashtag/weeknotes?src=hash&amp;ref_src=twsrc%5Etfw">#weeknotes</a> <a href="http://t.co/p1A51gXB3e">pic.twitter.com/p1A51gXB3e</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/605402268276559872?ref_src=twsrc%5Etfw">June 1, 2015</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

  <p>DoES Liverpool could still benefit from some more plant life but there definitely needs to be a plan for maintenance for this sort of thing! 🌻</p>

  <p>Hm.. anything else happen in 2015? Well we made this little announcement:</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Happy to announce that Evie is looking forward to being a big sister! All going well so far. <a href="https://twitter.com/hashtag/duedecember?src=hash&amp;ref_src=twsrc%5Etfw">#duedecember</a> <a href="http://t.co/iImTotOhFr">pic.twitter.com/iImTotOhFr</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/626479268462247937?ref_src=twsrc%5Etfw">July 29, 2015</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

  <p>As we settled into our new life with work and nursery and with Evie being such a good sleeper we decided that we might actually like to have another little person around the place. Funnily enough while Evie had actually come 10 days late Ethan actually came along exactly on his due date!</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I won on the premium bonds today but I think yesterday&#39;s prize was better! <a href="https://twitter.com/hashtag/itsaboy?src=hash&amp;ref_src=twsrc%5Etfw">#itsaboy</a> <a href="https://t.co/PXOmL0KpsH">pic.twitter.com/PXOmL0KpsH</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/672706695794188288?ref_src=twsrc%5Etfw">December 4, 2015</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

  <p>While Ethan seemed originally to be feeding okay we ended up having similar troubles with him losing too much weight and being harassed by mid-wives. After a week of problems we went along to an infant feeding clinic only to be told that Ethan had a tongue tie. A tiny piece of skin was stopping him from being able to move his tongue freely and causing him problems with feeding. Unfortunately our options were limited to waiting 2 months for an appointment in Alder Hey or trying to get it done in Chester Hospital. There was also the option of going private but we really didn’t think we should have to do that and also of the standards of care we’d receive. Being so close to Christmas we were quite concerned about whether we’d managed to get it done before the holidays so we were very happy when we got an appointment for the 23rd December. Poor Ethan ended up picking up a cold meaning the nurse almost couldn’t complete the tongue tie snip, and as it turned out had probably missed some as it really didn’t make much difference to his feeding. It seems ridiculous having to wait three weeks for something that should have been picked up and fixed while he was still in the hospital, and that the much vaunted and well-funded Alder Hey couldn’t do anything about it for months. Obviously we can’t be sure that dealing with it straight away would have reduced the problems, but it would have given Ethan a much better chance and given my poor stressed wife one less thing to worry about!</p>

  <p>I feel I should finish on a lighter note though so let’s back-track to October, we’d been invited to a Halloween party and I couldn’t think of what to go as. My wife spotted a Jack Skellington costume in the shop and came up with the great idea of combining it with one of my old Santa Dash costumes, resulting in this great result:</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Bit of recycling went on for my Halloween costume <a href="https://t.co/bPsG6YcrY0">pic.twitter.com/bPsG6YcrY0</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/660565811208261632?ref_src=twsrc%5Etfw">October 31, 2015</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>


guid: https://blog.johnmckerrell.com/2017/10/27/year-notes-2014
url: https://blog.johnmckerrell.com/2017/10/27/year-notes-2014/
title: Year Notes: 2014
updated: 2017-10-27 08:55:39
published: 2017-10-27 08:55:39
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.921865
updated_at: 2019-06-09 18:11:41.921865
summary:
  So obviously after the events at the tail end of 2013, this year was mostly spent dealing with the fact that we now had a tiny (not so tiny) baby to look after! Work-wise I’d stupidly taken on two new clients just before baby came along so while they were freelance clients who were aware of what was happening, it did mean I had some worries about making sure I could do work for them. In the end one of them tailed away to nothing within a few weeks of the new year and the other was just a week or so’s work that I managed to get in while baby was napping.

content:
  <p>So obviously after the events at the tail end of 2013, this year was mostly spent dealing with the fact that we now had a tiny (not so tiny) baby to look after! Work-wise I’d stupidly taken on two new clients just before baby came along so while they were freelance clients who were aware of what was happening, it did mean I had some worries about making sure I could do work for them. In the end one of them tailed away to nothing within a few weeks of the new year and the other was just a week or so’s work that I managed to get in while baby was napping.</p>

  <p>In retrospect the newborn stage is actually something of a calm before the storm as they do tend to sleep a lot. We had some issues around baby’s feeding and weight gain which the midwives dutifully freaked us out over but after a few months she did start putting on weight better. In fact once we weaned her onto food, which with perhaps rose-tinted hindsight went pretty well, she started putting on plenty of weight.</p>

  <p><img src="https://pbs.twimg.com/media/Bdsj5G8CYAABRRM.jpg" alt="mugged for milk" /></p>

  <p>The “week’s work” was quite an interesting iPad app and ultimately developed into a continuous 2 days a week. Without going into too much detail it was tourism based and involved having the app open while you were driving a car. It proved quite tricky to develop and test due in part to literally having to go out and drive in a car to get any useful test data. Also my clients were in London and there was times when they would upgrade the app, drive to central London, try to launch the app and find it insta-crashed. The fix was to delete and reinstall, not so easy for a multiple hundred MB app when you’d already gone to the location. With this and another client I’ll get to shortly I learned a lot about the benefits of automated testing, automated smoke testing and continuous integration. The testing tools at the time were not so good as they are now in Xcode but if I had a good method to simulate a drive, and to do smoke tests every time I committed code I could have avoided many of the problems I encountered.</p>

  <p>A few months into the year when I was mainly working on some updates to CamViewer and making further small changes to the iPad app I just mentioned I was approached by <a href="https://twitter.com/johnbfagan">John Fagan</a> who I used to work with at Multimap. He was wondering if I’d be interested in taking on a full time role at the company that he was working for - Axon Active. I’d not really been interested in full time roles but in fact as this was a foreign company I would be treated more like a contractor, and the money was pretty decent. Ultimately I told John no, only for him to suggest that I might work just a few days a week for them instead. When I mentioned these talks to the people I’d been engaging with on the other app they jumped at the chance to grab my other free days and so I ended up with 5 paid days a week split between Axon Active (3 days) and the other client.</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Miserable weather, awesome tunes! - Just posted a 12.40 km run - <a href="https://twitter.com/hashtag/RunKeeper?src=hash&amp;ref_src=twsrc%5Etfw">#RunKeeper</a> <a href="http://t.co/F0nJvhTmoD">http://t.co/F0nJvhTmoD</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/433677157651341313?ref_src=twsrc%5Etfw">February 12, 2014</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

  <p>Looking back on my tweets from the year I see multiple mentions of running. I’ve never really mentioned running on this blog which is quite bad because it’s become a regular part of my life. In fact in 2013 I ran my first marathon, a milestone I forgot to mention in the previous blog post! In 2014 I ran the same race again but found it much more tricky to fit in training around childcare, I mostly did best efforts at the training plan I was using but ended up with a decent result. While in 2013 I managed an awesome 1:40:00 for the half, I managed to follow it up in 2014 with a very respectable 1:43:25. In 2013 I ended up having real issues with my IT band, causing me to limp the last few miles. Annoyingly in 2014 I actually felt much better and had no issues such as this, but then faced a headwind for the same last miles! I could quite possibly have improved my PR if it wasn’t for that.</p>

  <p>With Axon Active I was working on a small project they’d been developing around taking in various items of data that would be made available on an iPhone, uploading them to a server and from there deriving information about patterns in the way you live and your future plans that we might be able to help you with. At the time we pulled in location and calendar events and would do things such as suggesting a place you might go to nearby for a quick lunch or let you know about travel options for your calendar appointments.</p>

  <p>Axon Active are a Swiss company but most of the people working for them were remote. At the time we had people in France, Edinburgh, London, Brighton, Manchester, even Russia! This allowed me to keep the flexibility of working from DoES Liverpool which was very handy. We would meet every 3 weeks in London for sprint planning and every 3 months for a trip to head office in Switzerland. This worked really well with the new baby, I could choose to work from home or from DoES Liverpool most of the time and the trips to London weren’t too tricky. Having to be away for a week wasn’t so great but it also wasn’t so often. We had plenty of support from my mother in law, Anne, so that made sure I was mostly not leaving my wife home alone with the kid. Also helped that my daughter got really quite good at sleeping from as early as 6 months!</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Enjoying the view at the Axon barbecue <a href="http://t.co/ZgonV9aqKF">pic.twitter.com/ZgonV9aqKF</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/461544777738379264?ref_src=twsrc%5Etfw">April 30, 2014</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

  <p>The project at Axon Active was initially just a side-project for an 8 person team but as the year progressed the company really saw the potential of what we were doing and it culminated in a new UK based company being formed at the very end of the year… but that’s really a story for 2015.</p>

  <p>Again looking back at my tweets I see that after some discussions with <a href="https://twitter.com/DefProc">Patrick Fenner</a> I had him and his wife <a href="https://twitter.com/RedVioletJen">Jen Fenner</a>, through their company <a href="https://www.deferredprocrastination.co.uk">Deferred Procrastination</a>, help me with a new design of the WhereDial. They actually engaged an old friend <a href="http://www.sophie-green.com">Sophie Green</a> to prepare some artwork for the device while Patrick and Jen looked at improving the functional design and the production method. They developed a way of using screen printing to allow much faster printing of designs onto the laser cut materials (ultimately screen printing would take seconds where laser engraving could take 15-20 minutes per piece). The resultant WhereDial looked really good and I was quite happy with the results. Unfortunately as I got busier with life and paid work I found I didn’t have the time to progress this so never got as far as selling the new versions. I’ve had them around my house and on my desk over the years and it’s interesting to note that while they do look good, the design is quite busy and the colours quite low contrast so it can be tricky to tell what’s happening from across a room. Something to keep in mind if I ever get around to developing the WhereDial again!</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">What do you think of the new look WhereDial? See it up close and get kits at <a href="https://twitter.com/hashtag/MakerFaireUK?src=hash&amp;ref_src=twsrc%5Etfw">#MakerFaireUK</a> <a href="https://twitter.com/DoESLiverpool?ref_src=twsrc%5Etfw">@DoESLiverpool</a> stand! <a href="http://t.co/SMaeAIll22">pic.twitter.com/SMaeAIll22</a></p>&mdash; WhereDial (@WhereDial) <a href="https://twitter.com/WhereDial/status/459941720008822784?ref_src=twsrc%5Etfw">April 26, 2014</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

  <p>The iPad app continued on for much of the year, we found it tricky getting the location based stuff working just how we wanted. We were trying to simulate something that a human would do and the clients had a particular level of quality in mind that was hard to replicate. There was also the need for a sat-nav component in the app, we didn’t want to call out to Apple Maps so would have needed to either build a sat nav ourselves or pull in a third party component. In the end this proved particularly difficult to find for iOS leading them to look at Android as an alternative. With my lack of interest in Android and continuing focus on the Axon Active role we ended up parting ways in Autumn. It was a really interesting app to work on but just had some difficulties that would really have required a lot more development resource than me working on my own. I didn’t hear too much from the clients once I handed the code over to their Android developer and I have a feeling that the project stalled around then.</p>

  <p>That’s probably it for 2014 really, once that app project stopped I found it useful having two days a week to do more work on CamViewer and also found the time to make some changes to the Chess Viewer app that I’d been working on intermittently. As 2014 drew to a close my wife started to think about returning to work and deciding what hours she wanted to do. Axon Active were also looking to setup the UK company and were offering people full time jobs, me included. I really wasn’t sure that I wanted to make that commitment, my stance hadn’t really changed from the beginning of the year, but I wasn’t really looking for any more work and the money was still pretty decent even as an employee so it was a hard decision to make. I didn’t make the decision until the first week of 2015 so I’ll make you wait to hear!</p>

  <p>(Let’s finish up with a photo of me ready for the 5k Santa Dash)</p>

  <blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Santa and his slightly concerned reindeer. <a href="http://t.co/FanlyQmHSv">pic.twitter.com/FanlyQmHSv</a></p>&mdash; John McKerrell (@mcknut) <a href="https://twitter.com/mcknut/status/541519342664687616?ref_src=twsrc%5Etfw">December 7, 2014</a></blockquote>
  <script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>


guid: https://blog.johnmckerrell.com/2017/10/20/year-notes-2013
url: https://blog.johnmckerrell.com/2017/10/20/year-notes-2013/
title: Year Notes: 2013
updated: 2017-10-20 08:34:39
published: 2017-10-20 08:34:39
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.924379
updated_at: 2019-06-09 18:11:41.924379
summary:
  I was tempted to skip this year as I had after all written a blog post that year already, but why not go crazy and write a second one, maybe the 2013 review would have been the first post of 2014 anyway?

content:
  <p>I was tempted to skip this year as I had after all written a blog post that year already, but why not go crazy and write a second one, maybe the 2013 review would have been the first post of 2014 anyway?</p>

  <p>Although I tend to write about geeky work related things on this blog it’s really my personal blog so I should definitely mention that the two major events of 2013 for me were personal rather than work related. We (my wife and I) moved house, and got pregnant! The house move wasn’t particularly planned and just came from noticing a house around the corner from ours that looked interesting, checking it out (awful) but then looking at a few more and deciding that actually we really could manage a nice upgrade, and might need to as we’d be needing more room soon.</p>

  <p>We were lucky that we could afford a lovely big victorian house, which looks nice and provides lots of room for kids and associated “stuff”, but isn’t the best when it comes to heating and having lots of little jobs that need doing. We actually didn’t move in for 3 months after buying it, but still even now have lots of jobs that need doing and some quite large bits of building work we’d like to do, if we could get around to it (making the cellar a usable space and extending the kitchen). Obviously having lots of jobs to do is standard for owning a house but this place definitely seems to take it to another level!</p>

  <p>Also back in 2013 I took on a summer student, <a href="https://twitter.com/ellcom">Elliot</a>. That was a great experience even if I hadn’t necessarily prepared well enough for having him around. I ended up giving him all sorts of different bits of work to do including upgrades to my CamViewer webcam viewing app and upgrades to the WhereDial. He actually did most of the work towards a Wi-Fi enabled webcam that would have used TP-Link mini Wi-Fi routers as much of the brains with a bespoke Arduino compatible circuit board controlling the motor. Unfortunately in the end I didn’t get around to productising that but hopefully he had some fun working on it and got some good experience. After that summer he went back to university to finish his degree then had no problem finding a job. I haven’t caught up with him in a while but I believe he’s still doing well and applying his great versatility to working on a variety of things from back-end server coding to mobile app dev.</p>

  <p>During the summer of 2013 I, with Adrian McEwen, Hakim Cassimally and Aine McGuire exhibited at Internet World trade show in Earls Court, London. That was quite an interesting experience. We were given a prime spot at the entrance to the show and used it to demonstrate a variety of IoT devices including the WhereDial and Bubblino, Hakim and Adrian also promoted their book <a href="https://www.amazon.co.uk/Designing-Internet-Things-Adrian-McEwen/dp/111843062X/">Designing the Internet of Things</a>. And because spending a week in London wasn’t enough we then spent the following weekend in Newcastle for Makerfaire!</p>

  <p><img src="https://dn3pm25xmtlyu.cloudfront.net/photos/large/762975627.jpg?Expires=1509698275&amp;Signature=UoGYIg1MrkxPG05857uuebnqTmObK73QDGZUSiNjkKRWmmzi9uFdFDZgAF3kXb2T2tNHMvpkQgkribWKv~BNgztro9Hq8wviOZ3Hsmz4f9UwUG49CqJGlrEr7AK~5l7DgfO-8qN3O9q0PLgF6l0ES1jYPHyauhh-vXCTJAViWSdMoOav8NmFwfNPtrZBUzulPATVZEWFLcAZhTAw-1bEoykC3QK5r-NAuPGr1kkrKHRNYOzOFdFbatwuNMjkfwufB3hagkN62OyBaDBglJsruaP6MtQLKYKuxp2Z3eCWLkUffLEx0YEh-oIkQrsX1PC~aXyimBgvmYqC3OY~LEmKDg__&amp;Key-Pair-Id=APKAJROXZ7FN26MABHYA" alt="Makerfaire 2013" /></p>

  <p>Wow, looking at my notes that was when my relationship with ExamsTutor ended. Unfortunately it didn’t end as well as I’d have liked, they simply decided they did not want to continue the relationship and largely cut off communication. I had owned the IP behind the apps so no further development occurred on those and it looks like they got removed from the app store in Apple’s great cull of 2016 (removing any app that hadn’t been updated in years). A shame to end that way as I’d enjoyed working with them but there wasn’t much I could do once they stopped replying to emails.</p>

  <p>My relationship with 7digital also ended this year, as I recall they were looking to take development in-house which was fair enough. I don’t think iOS was ever really a huge priority for them as Apple’s app store rules made it difficult for them to make any money from the app. I know they continued using my codebase for a few years after, it’s hard to tell if they still do, the app’s structure hasn’t changed too much but it’s quite likely it’s had some restructuring under the hood.</p>

  <p>As mentioned I (and Elliot) also continued working on CamViewer through that year. Interestingly looking at the <a href="https://camviewerip.com/changelog/">Changelog</a> that year seems to have been the start of me adding more functionality to the app. At the end of the year I added support for “HD” cameras that use RTSP by integrating a paid-for library. That was just in time for me to use cameras such as this as baby monitors, something I and my wife have both found really handy over the years.</p>

  <p>Pretty much the last thing that happened in this year, Evie was born!</p>

  <p><img src="https://farm6.staticflickr.com/5155/14392136644_d56952d6df_k.jpg" alt="Actually 6 months old in this picture" /></p>

  <p>So that’s a review of 2013 done, just 3 more years to cover (assuming I get this done quickly, considering I started this blog post in January (2017) that may not be the case!)</p>


guid: https://blog.johnmckerrell.com/2017/06/24/new-new-blog-new-old-blog
url: https://blog.johnmckerrell.com/2017/06/24/new-new-blog-new-old-blog/
title: New New Blog, New Old Blog
summary: So I’ve decided to replace my previous blog with a WordPress blog.
updated: 2017-06-24 12:12:58
published: 2017-06-24 12:12:58
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.926852
updated_at: 2019-06-09 18:11:41.926852
content:
  <blockquote>
    <p>So I’ve decided to replace my previous blog with a WordPress blog.</p>
  </blockquote>

  <p>That was the first line I put on my WordPress blog, which I’ve now replaced with Jekyll. The first post on that was in October 2006, so nearly 11 years ago!</p>

  <p>Over the past 11 years I’ve moved away from doing anything PHP, or anything server-side really. I have no particular need of the online editing that you get with WordPress or any of the other features really. I’m also trying to be quite paranoid about what daemons I run on my hosting server. I came across Jekyll (again) as part of a documentation project in my day job and was impressed by how easy it was to use so decided I would start migrating some of my own stuff over.</p>

  <p>My first migration was actually the website for my <a href="https://camviewerip.com/">CamViewer iOS app</a> which was already based on <a href="https://github.com/gollum/gollum">Gollum</a> the markdown wiki so should have been quite simple to migrate (actually it was still a bit of a faff due to different markdown versions).</p>

  <p>I hoped migrating Wordpress would also be easy as so many people use both technologies. As it turned out there was still plenty of work involved. I ended up using two migration tools. The main Jekyll importer didn’t seem to do a great job of pulling the HTML in but pulled all the comments across nicely so I ended up using <a href="https://github.com/thomasf/exitwp">exitwp</a> and writing a yaml copying tool to pull the comments from one to the other.</p>

  <p>This blog hasn’t actually got any comments functionality at the moment, I figure people can ping me <a href="https://twitter.com/mcknut">@mcknut</a> on Twitter if they want to make comments but I’ve copied the ones that were on the old site.</p>

  <blockquote>
    <p>My previous blog made use of whizzy fun modern technologies to allow me to host my entire site on <a href="http://base.google.com">Google Base</a>, <a href="http://pages.google.com">Google Pages</a> and <a href="http://del.icio.us">del.icio.us</a>. Unfortunately, because it was something I had just knocked up there was no comment support, and of course it did require Java and JavaScript to be enabled in the browser.</p>
  </blockquote>

  <p>That original blog was quite a weird thing, I tried to make it so that all the content was hosted online. Unfortunately as it was JavaScript based not much has been saved on the Internet Archive. It does seem like all the original posts are now lost but with any luck I’ve got the content… somewhere.</p>


guid: https://blog.johnmckerrell.com/2017/01/18/long-time-no-see
url: https://blog.johnmckerrell.com/2017/01/18/long-time-no-see/
title: Long time no see
updated: 2017-01-18 11:18:31
published: 2017-01-18 11:18:31
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.929222
updated_at: 2019-06-09 18:11:41.929222
summary:
  It’s been a very long time since I’ve written anything here. I thought I’d like to write up a small project I did recently, but then there would be a bit of back story, and then a bit more, and oh yes I haven’t blogged in 4 years so I should really do something about that.

content:
  <p>It’s been a very long time since I’ve written anything here. I thought I’d like to write up a small project I did recently, but then there would be a bit of back story, and then a bit more, and oh yes I haven’t blogged in 4 years so I should really do something about that.</p>

  <p>As such I’m going to try to write some year reviews. Not promising to make them in depth but it’ll give me a chance to look back and see what I’ve actually been doing all these years (actually it’s pretty obvious to me given that the last post was 4 years ago and a major thing happened just after that but here’s goes nothing..!)</p>


guid: https://blog.johnmckerrell.com/2013/01/07/adding-a-wiki-to-mapme-at-using-gollum
url: https://blog.johnmckerrell.com/2013/01/07/adding-a-wiki-to-mapme-at-using-gollum/
title: Adding a Wiki to MapMe.At using Gollum
updated: 2013-01-07 11:30:36
published: 2013-01-07 11:30:36
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.931635
updated_at: 2019-06-09 18:11:41.931635
summary:
  I recently added a wiki on my MapMe.At site and found it quite tricky to get working and difficult to find just the right information I needed so I thought I’d write it up.

content:
  <p>I recently added a wiki on my MapMe.At site and found it quite tricky to get working and difficult to find just the right information I needed so I thought I’d write it up.</p>

  <p>MapMe.At is still on Rails 2 which seemed to mean I couldn’t install Gollum as part of the site.</p>

  <p>I created a separate Rails 3 project that runs alongside MapMe.At and simply hosts Gollum, using instructions from here: <a href="http://stackoverflow.com/questions/13053704/how-to-properly-mount-githubs-gollum-wiki-inside-a-rails-app">http://stackoverflow.com/questions/13053704/how-to-properly-mount-githubs-gollum-wiki-inside-a-rails-app</a></p>

  <p>I wanted it to use the user information in MapMe.At’s session hash so switched MapMe.At to use activerecord based sessions and used information on here to make the rails 2 session load in rails 3: <a href="http://www.kadrmasconcepts.com/blog/2012/07/19/sharing-rails-sessions-with-php-coldfusion-and-more/">http://www.kadrmasconcepts.com/blog/2012/07/19/sharing-rails-sessions-with-php-coldfusion-and-more/</a></p>

  <p>I’m not actually using the rails2 session as the main session, I just load the information in. I have the following in <code class="highlighter-rouge">config/initializers/session_store.rb</code></p>

  <figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">module</span> <span class="nn">ActionController</span>
    <span class="k">module</span> <span class="nn">Flash</span>
      <span class="k">class</span> <span class="nc">FlashHash</span> <span class="o">&lt;</span> <span class="no">Hash</span>
        <span class="k">def</span> <span class="nf">method_missing</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="o">*</span><span class="n">a</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">b</span><span class="p">;)</span>
        <span class="k">end</span>
      <span class="k">end</span>
    <span class="k">end</span>
  <span class="k">end</span>

  <span class="no">MapmeAtWiki</span><span class="o">::</span><span class="no">Application</span><span class="p">.</span><span class="nf">config</span><span class="p">.</span><span class="nf">session_store</span> <span class="ss">:cookie_store</span><span class="p">,</span> <span class="ss">key: </span><span class="s1">'_mapme_at_wiki_session'</span></code></pre></figure>

  <p>Then in routes.rb:</p>

  <figure class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">App</span> <span class="o">&lt;</span> <span class="no">Precious</span><span class="o">::</span><span class="no">App</span>

    <span class="n">before</span> <span class="p">{</span> <span class="n">authenticate</span> <span class="p">}</span>

    <span class="n">helpers</span> <span class="k">do</span>
      <span class="k">def</span> <span class="nf">authenticate</span>
        <span class="n">oldsess</span> <span class="o">=</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">SessionStore</span><span class="o">::</span><span class="no">Session</span><span class="p">.</span><span class="nf">find_by_session_id</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="nf">cookies</span><span class="p">[</span><span class="s2">"_myapp_session"</span><span class="p">])</span>
        <span class="k">if</span> <span class="n">oldsess</span> <span class="n">and</span> <span class="n">oldsess</span><span class="p">.</span><span class="nf">data</span> <span class="n">and</span> <span class="n">oldsess</span><span class="p">.</span><span class="nf">data</span><span class="p">[</span><span class="ss">:user_id</span><span class="p">]</span>
          <span class="n">u</span> <span class="o">=</span> <span class="no">User</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="n">oldsess</span><span class="p">.</span><span class="nf">data</span><span class="p">[</span><span class="ss">:user_id</span><span class="p">])</span>
          <span class="n">email</span> <span class="o">=</span> <span class="s2">"</span><span class="si">#{</span><span class="n">u</span><span class="p">.</span><span class="nf">username</span><span class="si">}</span><span class="s2">@gitusers.mckerrell.net"</span>
          <span class="n">session</span><span class="p">[</span><span class="s2">"gollum.author"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">:name</span> <span class="o">=&gt;</span> <span class="n">u</span><span class="p">.</span><span class="nf">full_name</span><span class="p">,</span> <span class="ss">:email</span> <span class="o">=&gt;</span> <span class="n">email</span> <span class="p">}</span>
        <span class="k">else</span>
          <span class="n">response</span><span class="p">[</span><span class="s2">"Location"</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"http://mapme.at/me/login?postlogin=/wiki/"</span>
          <span class="kp">throw</span><span class="p">(</span><span class="ss">:halt</span><span class="p">,</span> <span class="p">[</span><span class="mi">302</span><span class="p">,</span> <span class="s2">"Found"</span><span class="p">])</span>
        <span class="k">end</span>
      <span class="k">end</span>
    <span class="k">end</span>

  <span class="k">end</span>

  <span class="no">MapmeAtWiki</span><span class="o">::</span><span class="no">Application</span><span class="p">.</span><span class="nf">routes</span><span class="p">.</span><span class="nf">draw</span> <span class="k">do</span>
    <span class="c1"># The priority is based upon order of creation:</span>
    <span class="c1"># first created -&gt; highest priority.</span>

    <span class="no">App</span><span class="p">.</span><span class="nf">set</span><span class="p">(</span><span class="ss">:gollum_path</span><span class="p">,</span> <span class="no">Rails</span><span class="p">.</span><span class="nf">root</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="s1">'db/wiki.git'</span><span class="p">).</span><span class="nf">to_s</span><span class="p">)</span>
    <span class="no">App</span><span class="p">.</span><span class="nf">set</span><span class="p">(</span><span class="ss">:default_markup</span><span class="p">,</span> <span class="ss">:markdown</span><span class="p">)</span> <span class="c1"># set your favorite markup language</span>
    <span class="no">App</span><span class="p">.</span><span class="nf">set</span><span class="p">(</span><span class="ss">:wiki_options</span><span class="p">,</span> <span class="p">{</span><span class="ss">:universal_toc</span> <span class="o">=&gt;</span> <span class="kp">false</span><span class="p">})</span>
    <span class="n">mount</span> <span class="no">App</span><span class="p">,</span> <span class="ss">at: </span><span class="s1">'wiki'</span>

  <span class="k">end</span></code></pre></figure>

  <p>I wanted a wiki on the site to allow my users to help out with documenting the site. Adding their own thoughts and experiences and perhaps fixing typos I might make. I’m not sure that’s really started happening yet but at least I have a nice interface for writing the documentation myself!</p>


guid: https://blog.johnmckerrell.com/2012/12/02/wheredial-ready-to-ship
url: https://blog.johnmckerrell.com/2012/12/02/wheredial-ready-to-ship/
title: WhereDial Ready to Ship
updated: 2012-12-02 19:43:27
published: 2012-12-02 19:43:27
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.934344
updated_at: 2019-06-09 18:11:41.934344
summary:
  A good few years ago I blogged about making a clock that showed location, similar to the clock that Weasley family had in the Harry Potter books. Well now you can buy one! I’ve spent the last year working on the design and getting the hardware ready. Take a look at the photos below and head over to the website for more information on the WhereDial!

content:
  <p>A good few years ago I blogged about making <a href="http://blog.johnmckerrell.com/2009/06/01/hacking-location-into-hardware/">a clock that showed location</a>, similar to the clock that Weasley family had in the Harry Potter books. Well now you can buy one! I’ve spent the last year working on the design and getting the hardware ready. Take a look at the photos below and <a href="http://www.wheredial.com/">head over to the website for more information on the WhereDial!</a></p>

  <div id="gallery-1" class="gallery galleryid-11793 gallery-columns-3 gallery-size-thumbnail"><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8868.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8868-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8871.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8871-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8883.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8883-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><br style="clear: both" /><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8898.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8898-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8900.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8900-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8904.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8904-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><br style="clear: both" /><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8908.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8908-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8911.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8911-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8916.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8916-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><br style="clear: both" /><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8921.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8921-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8922.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8922-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8927.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8927-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><br style="clear: both" /><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8928.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8928-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8931.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8931-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8940.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8940-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><br style="clear: both" /><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8944.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8944-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8978.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/IMG_8978-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><dl class="gallery-item">
    <dt class="gallery-icon landscape">
      <a href="https://blog.johnmckerrell.com/uploads/2012/12/2012-11-26-17.22.10.jpg"><img width="150" height="150" src="https://blog.johnmckerrell.com/uploads/2012/12/2012-11-26-17.22.10-150x150.jpg" class="attachment-thumbnail size-thumbnail" alt="" /></a>
    </dt></dl><br style="clear: both" />
  </div>


guid: https://blog.johnmckerrell.com/2012/02/17/openstreetmap-at-social-media-cafe-liverpool
url: https://blog.johnmckerrell.com/2012/02/17/openstreetmap-at-social-media-cafe-liverpool/
title: OpenStreetMap at Social Media Cafe Liverpool
updated: 2012-02-17 16:49:56
published: 2012-02-17 16:49:56
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.936818
updated_at: 2019-06-09 18:11:41.936818
summary:
  I gave a talk last night at Social Media Cafe about OpenStreetMap. I actually haven’t been too involved in the OSM community of late so it was nice to get back into it a little bit. It was also good to find that a large portion of the audience was not already aware of OSM so it was nice to introduce it to people.

content:
  <p>I gave a talk last night at Social Media Cafe about OpenStreetMap. I actually haven’t been too involved in the OSM community of late so it was nice to get back into it a little bit. It was also good to find that a large portion of the audience was not already aware of OSM so it was nice to introduce it to people.</p>

  <p><a href="http://www.ustream.tv/recorded/20487500">You can find the video of Social Media Cafe on USTREAM.</a> The video will be chopped up soon at which point I’ll link to or embed my own talk here too.</p>

  <p>I ended up with 63 slides taking up about 100MB so I’m going to try not uploading it to Slideshare this time, instead I’m going to summarise the talk here.</p>

  <p><a href="/uploads/2012/02/s1.a.jpeg"><img src="/uploads/2012/02/s1.a-300x225.jpg" alt="OpenStreetMap - John McKerrell @mcknut - Freelance Software Developer" /></a></p>

  <h3 id="why-do-we-need-openstreetmap">Why do we need OpenStreetMap?</h3>

  <ul>
    <li>
      <p>Geodata historically isn’t</p>

      <ul>
        <li>
          <p>Current - things change so often maps quickly become outdated.</p>
        </li>
        <li>
          <p>Open - if you know the map is wrong, wouldn’t it be simpler to let you update it yourself?</p>
        </li>
        <li>
          <p>Free - You want me to pay how much for Ordnance Survey data?? Especially an issue when you’ve helped build the map.</p>
        </li>
      </ul>
    </li>
    <li>
      <p>Wiki is obvious next step</p>
    </li>
    <li>
      <p>It’s just <em>fun</em></p>
    </li>
  </ul>

  <h3 id="we-make-beautiful-maps">We make beautiful maps…</h3>

  <h3 id="which-we-give-away">…which we give away</h3>

  <p><a href="https://www.openstreetmap.org/#map=14/53.4053/-3.0089"><img src="/assets/osm-map-liverpool.png" alt="Beautiful OpenStreetMap map" /></a></p>

  <p>It’s not just Liverpool, or even the UK, in the talk I showed maps of the Hague, Washington, DC and Berlin. You can pan and zoom the map linked to above to browse the coverage.</p>

  <h3 id="some-quotes">Some Quotes</h3>

  <blockquote>
    <p>“It’s absolutely possible for a bunch of smart guys with the technology we have today to capture street networks, and potentially the major roads of the UK and minor roads”</p>
  </blockquote>

  <p>Ed Parsons, ex-CTO Ordnance Survey
  currently Geospatial Technologist for Google</p>

  <blockquote>
    <p>“If you don’t make [lower-resolution mapping data] publicly available, there will be people with their cars and GPS devices, driving around with their laptops .. They will be cataloguing every lane, and enjoying it, driving 4×4s behind your farm at the dead of night. There will, if necessary, be a grass-roots remapping.”</p>
  </blockquote>

  <p>Tim Berners-Lee</p>

  <blockquote>
    <p>“You could have a community capability where you took the GPS data of people driving around and started to see, oh, there’s a new road that we don’t have, a new route .. And so that data eventually should just come from the community with the right software infrastructure.”</p>
  </blockquote>

  <p>Bill Gates</p>

  <p>Some big names in technology who clearly think user-generated mapping data is a good idea.</p>

  <h3 id="isnt-google-free">Isn’t Google Free?</h3>

  <p>A lot of people ask the question “Why do we need OpenStreetMap when Google Maps is free?”</p>

  <h4 id="current">Current?</h4>

  <p><a href="/uploads/2012/02/s15.b.jpeg"><img src="/uploads/2012/02/s15.b-300x225.jpg" alt="Google Map of Liverpool showing places that are no longer pertinent" /></a></p>

  <p><a href="http://maps.google.co.uk/?ll=53.40407,-2.985835&amp;spn=0.010937,0.031693&amp;t=m&amp;z=16">http://maps.google.co.uk/?ll=53.40407,-2.985835&amp;spn;=0.010937,0.031693&amp;t;=m&amp;z;=16</a></p>

  <p>This picture shows a Google Map screenshot that I took on 16th February 2012. In the centre of the map you can see the Moat House Hotel. This was bulldozed in 2005 but still shows up on Google’s map. You’ll also see the <a href="http://en.wikipedia.org/wiki/Consulate_of_the_United_States,_Liverpool">Consulate of the United States in Liverpool</a>. This was also closed down some time ago. So you can see that Google Maps isn’t perfectly current (and, for the record, I have now reported these problems to Google).</p>

  <h4 id="open">Open?</h4>

  <p>Google have launched their own project to map the planet. <a href="http://www.google.com/mapmaker">Map Maker</a> allows people in many countries to edit the data of the map, adding roads and POIs in a similar way to OSM. Unfortunately Google doesn’t then provide full access to this data back to the people who have made it! Map tiles are generated and shapes of the data entered can be retrieved but the full detail of the data is kept by Google. The license offered by Google also restricts its use to non-commercial usage, stopping people who have put effort into creating the data from being able to derive an income from it.</p>

  <h4 id="free">Free?</h4>

  <p>Though Google’s mapping API is free to use initially they have recently introduced usage limits. Though they claim that this will only affect 0.35% of their customers, it has already affected a number of popular websites that simply can’t afford to pay what Google is requesting. Some examples will be given of these later.</p>

  <h4 id="google-support-osm">Google Support OSM</h4>

  <p>It would be unfair to talk about the bad parts of Google without mentioning the good. Google has regularly supported OSM through donations, sponsorship of mapping parties and support through their “Summer of Code” programme.</p>

  <h4 id="as-do-other-providers">As do other providers</h4>

  <p>It also wouldn’t be fair to paint Google as the only supporter, for example:</p>

  <ul>
    <li>
      <p><a href="http://www.mapquest.com/">Mapquest</a> sponsors and supports OSM efforts.</p>
    </li>
    <li>
      <p><a href="http://www.bing.com/maps/">Microsoft Bing Maps</a> sponsors and supports OSM efforts, even allowing their aerial imagery to be traced.</p>
    </li>
  </ul>

  <h3 id="workshops">Workshops</h3>

  <h4 id="or-map-as-party-mapping-parties">Or, Map as Party (Mapping Parties!)</h4>

  <p>The first mapping party was in the Isle of Wight. At the time the only “free” map data available was an Ordnance Survey map that had gone out of copyright:</p>

  <p><a href="/uploads/2012/02/s22.b.jpeg"><img src="/uploads/2012/02/s22.b-300x225.jpg" alt="Isle of Wight OS Map" /></a></p>

  <p>A group of people went to the island for a weekend and collected GPS traces of all the roads:</p>

  <p><a href="/uploads/2012/02/s23.b.jpeg"><img src="/uploads/2012/02/s23.b-300x225.jpg" alt="" /></a></p>

  <p>And from these made a great looking map:</p>

  <p><a href="/uploads/2012/02/s24.b.jpeg"><img src="/uploads/2012/02/s24.b-300x225.jpg" alt="" /></a></p>

  <p>We also held a mapping party in Liverpool in November 2007 which allowed us to essentially complete the map of the centre of Liverpool.</p>

  <p>That video shows the traces of everyone involved with the mapping party as they went around Liverpool and mapped the streets. It was built using <a href="http://wiki.openstreetmap.org/wiki/Party_render">the scripts referenced on this wiki page</a></p>

  <h3 id="editing-osm">Editing OSM</h3>

  <p>Visit <a href="http://www.openstreetmap.org/">openstreetmap.org</a> and sign up for an account. If you have GPS traces, upload them, don’t worry if you don’t as you’ll be able to help by editing existing data or tracing over aerial imagery.</p>

  <h4 id="data-model">Data Model</h4>

  <ul>
    <li>
      <p>Nodes</p>

      <ul>
        <li>Single point on the earth - Latitude and Longitude</li>
      </ul>
    </li>
    <li>
      <p>Way</p>

      <ul>
        <li>Ordered list of nodes which together make up a long line or an enclosed area</li>
      </ul>
    </li>
    <li>
      <p>Relation</p>

      <ul>
        <li>A method of relating multiple ways and nodes together, e.g. “turning from way A to way B using node C is not allowed”</li>
      </ul>
    </li>
    <li>
      <p>Tags</p>

      <ul>
        <li>
          <p>Nodes/Ways/Relations can have key=value pairs attached to describe their properties.</p>
        </li>
        <li>
          <p>Example node tags:</p>

          <ul>
            <li>
              <p>amenity=place_of_worship, religion=buddhist</p>
            </li>
            <li>
              <p>amenity=post_box</p>
            </li>
          </ul>
        </li>
        <li>
          <p>Example way tags:</p>

          <ul>
            <li>
              <p>highway=primary</p>
            </li>
            <li>
              <p>oneway=yes</p>
            </li>
          </ul>
        </li>
      </ul>
    </li>
  </ul>

  <p>An online flash editor is available (Potlatch) simply by clicking the “Edit” link when looking at any map on OSM. An offline editing desktop app built in Java is also available, <a href="http://wiki.openstreetmap.org/wiki/JOSM">JOSM</a></p>

  <p>There are hundreds of tags that you can use to describe almost any data, <a href="http://wiki.openstreetmap.org/">use the wiki</a> to find more information especially the <a href="http://wiki.openstreetmap.org/wiki/Map_Features">Map Features</a> page.</p>

  <h3 id="license">License</h3>

  <h4 id="cc-by-sa"><a href="http://creativecommons.org/licenses/by-sa/3.0/">CC-BY-SA</a></h4>

  <p>This license lets anyone use the OSM maps for free so long as you mention that the source was OpenStreetMap and you share what you produce under a similar license.</p>

  <p>Very soon the license will change from CC-BY-SA to <a href="http://wiki.openstreetmap.org/wiki/Open_Database_License">Open Database License</a> which offers similar freedoms with more suitable legal terminology. Do read into it if you think it will affect you.</p>

  <h3 id="osm-in-action">OSM in Action</h3>

  <p><a href="http://www.nestoria.co.uk/">Nestoria</a>, a popular property website, has long supported OSM. A few years ago they made use of OSM data by using the maps generated from the Isle of Wight mapping party to replace the non-existent data in Google Maps. More recently they have been affected by Google’s plans to charge for its map data and so they have switched fully to OpenStreetMap data and maps.</p>

  <p><a href="http://www.cyclestreets.net/">CycleStreets</a> is a great website for finding cycle routes. They offer a directions engine that gives detailed descriptions of routes, allowing you to pick between Balanced, Fastest and Shortest routes. They also offer lots more information and a database of photos to give more insight into a journey. The routes they recommend are ideal for keeping cyclists off the busy dangerous roads and onto the quieter safer more direct routes.</p>

  <p><a href="http://mapme.at/">mapme.at</a> is a website that I have built for tracking people’s location. People use it to track places that they visit and journeys that they take. I use it to track everywhere I ever go. <a href="http://twitter.com/amcewen">Adrian McEwen</a> wrote a script that puts the location of the Mersey Ferries into mapme.at and that’s what you can see in the map above.</p>

  <p>A few years ago I worked with <a href="http://itoworld.com">ITO World</a> to create some animations of my data. They created great animations which you can find <a href="http://vimeo.com/user1530776">on my vimeo account</a> but below is one showing every journey I took in January 2010 with each day being played concurrently.</p>

  <p><a href="http://vimeo.com/10636004">All travels in January 2010 run at once.</a> from <a href="http://vimeo.com/user1530776">John McKerrell</a> on <a href="http://vimeo.com">Vimeo</a>.</p>

  <p><a href="http://geocaching.com">Geocaching</a> is a popular pastime based around GPSes, treasure hunting and maps. Their website used Google Maps and they also had issues when Google started to charge. As a result they have switched to OpenStreetMap too.</p>

  <h3 id="mobile">Mobile</h3>

  <p>Lots of mobile apps are available to let you use and contribute to OpenStreetMap</p>

  <h4 id="android">Android</h4>

  <p>Not being a regular user of Android I can’t recommend any apps personally but there is <a href="http://wiki.openstreetmap.org/wiki/Android">a large list of OSM Android apps on the wiki</a> and I’ve selected the following based on features they claim to offer.</p>

  <p><a href="http://wiki.openstreetmap.org/wiki/GoPenS">gopens</a> and <a href="http://wiki.openstreetmap.org/wiki/MapDroyd">MapDroyd</a> both allow you to browse OpenStreetMap maps on the go and claim to offer offline support, allowing you to view maps even when you’re not connected to the internet.</p>

  <p><a href="http://wiki.openstreetmap.org/wiki/Skobbler_Navigation">Skobbler Navigation</a> provides a full Tom-Tom style satnav for navigating on the go, all based on OpenStreetMap data.</p>

  <p><a href="http://wiki.openstreetmap.org/wiki/Mapzen_POI_Collector">Mapzen POI Collector</a> is a handy way to collect POI data while out and about, or to edit existing data.</p>

  <h4 id="iphone">iPhone</h4>

  <p><a href="http://wiki.openstreetmap.org/wiki/Skobbler_Navigation">Skobbler Navigation</a> is also available for iPhone, again providing a full Tom-Tom style satnav for navigating on the go, all based on OpenStreetMap data.</p>

  <p><a href="http://wiki.openstreetmap.org/wiki/Navmii">NavFree</a> is another full satnav app based on OpenStreetMap data.</p>

  <p><a href="http://wiki.openstreetmap.org/wiki/Offmaps">Offmaps</a> is an OSM map viewer that allows you to download large chunks of map tiles in advance so that you have them, for instance, when you go on holiday. I would recommend the original Offmaps over Offmaps2 as I believe the latter restricts the data you can access.</p>

  <p><a href="http://wiki.openstreetmap.org/wiki/Mapzen_POI_Collector">Mapzen POI Collector</a> again is available for iPhone and is a handy way to collect POI data while out and about, or to edit existing data.</p>

  <h3 id="humanitarian">Humanitarian</h3>

  <p>OpenStreetMap has been heavily involved in Humanitarian efforts, these have resulted in the formation of <a href="http://hotosm.org/">HOT - the Humanitarian OpenStreetMap Team</a>. Projects have included <a href="http://wiki.openstreetmap.org/wiki/WikiProject_Gaza">mapping the Gaza Strip</a> and <a href="http://mapkibera.org">Map Kibera</a> a project to map the Kibera slum in Nairobi, Kenya. These projects have many benefits to the communities involved. Simply having map data helps the visibilities of important landmarks: water stations, Internet cafes, etc. Teaching the locals how to create the maps teaches valuable technical skills. Some people build on the data to provide commercial services to their neighbours, building businesses to support themselves and their families.</p>

  <p>A hugely influential demonstration of the impact of OpenStreetMap involvement in humanitarian efforts occurred after the massive earthquake that struck Haiti in 2010. Very shortly after the earthquake hit, the OSM community realised that the lack of geodata in what was essentially a third world country, would cause massive problems with aid workers going in to help after the earthquake. The community responded by tracing the aerial imagery that was already available to start to improve the data and later efforts included getting newer imagery, getting Haitian ex-pats to help with naming features and working with the aid agencies to add their data to the map. You can see some of the effects of these efforts from the video below that shows the edits that occurred in Haiti around the time of the earthquake.</p>

  <p><a href="http://vimeo.com/9182869">OpenStreetMap - Project Haiti</a> from <a href="http://vimeo.com/itoworld">ItoWorld</a> on <a href="http://vimeo.com">Vimeo</a>.</p>

  <h3 id="switch2osm">Switch2OSM</h3>

  <p>If all of this has piqued your interest then visit <a href="http://openstreetmap.org/">openstreetmap.org</a> to take a look at the map, sign up and get involved in editing. Find more information on the wiki at <a href="http://wiki.openstreetmap.org">wiki.openstreetmap.org</a> or <a href="http://switch2osm.org">find out how you can switch your website to OpenStreetMap at Switch2OSM.org</a></p>


guid: https://blog.johnmckerrell.com/2011/10/23/yahms-revisited-upgrading-to-xbee-with-wire-antenna
url: https://blog.johnmckerrell.com/2011/10/23/yahms-revisited-upgrading-to-xbee-with-wire-antenna/
title: YAHMS: Revisited, Upgrading to XBee with Wire Antenna
updated: 2011-10-23 19:15:17
published: 2011-10-23 19:15:17
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.939334
updated_at: 2019-06-09 18:11:41.939334
summary:
  A good few months ago I blogged about my YAHMS project and my YAHMS Base Station. It was a funny time to get the project complete, just in time for the summer, but it has ended up being useful. Being able to turn the hall light on when we come home in the dark is really useful and, with the summer we’ve had, the heating has gone on from time to time too.

content:
  <p>A good few months ago I blogged about my <a href="http://blog.johnmckerrell.com/category/arduino-2/">YAHMS project</a> and my <a href="http://blog.johnmckerrell.com/2011/05/09/yahms-base-station/">YAHMS Base Station</a>. It was a funny time to get the project complete, just in time for the summer, but it has ended up being useful. Being able to turn the hall light on when we come home in the dark is really useful and, with the summer we’ve had, the heating has gone on from time to time too.</p>

  <p>It hasn’t been without its problems, though fortunately they have been relatively few. Power ended up being an issue for the temperature probes, using four AA batteries would keep the probes running for about a week. This isn’t bad but would mean replacing them regularly, also when the voltage went low, the voltage detector didn’t seem to work and I would see rising temperatures. In the end I decided to use some wall wart power supplies, unfortunately that means the probes are less portable and also means that half of the circuit was unnecessary, as the wall warts never provide a low voltage.</p>

  <p>I’ve also had problems with the Base Station falling off the network after around 5-7 days of use. There was originally a problem with the DHCP lease expiring so I made some changes to make it re-request the lease when it was near to expiring. This didn’t entirely fix the problem so to be sure I switched to a fixed IP. This led to the 5-7 day uptime, I’m not really sure what is happening here, I’m guessing some memory usage problem whereby there are not enough resources left to create a new Client object but I haven’t been able to track it down. When the problem occurs the system still manages to turn the heating on and off at the right times so it isn’t entirely useless, but it does become unresponsive to new settings and won’t send back data either.</p>

  <p>Final problem, and the one I’m going to fix today, is that the XBee modules sometimes have issues transmitting data to the base station. I originally wanted to have one upstairs but couldn’t get that to work at all. I ended up putting that one in the conservatory to get “outside” temperatures but it still has issues quite often and has to be positioned right by the door to even sometimes work.</p>

  <p>A few months ago I was approached by someone from <a href="http://www.farnell.com/">Farnell</a> who wanted to know if I would like to receive hardware to review on my blog. The arrangement is that they will send me hardware in exchange for a review on my blog and <a href="http://uk.farnell.com/digi-international/xb24-awi-001/wire-antenna-module-xbee-1mw/dp/1337912">a link to the product on their website.</a> So, here goes…</p>

  <p><a href="/uploads/2011/10/Photo-01-10-2011-20-46-50.jpg"><img src="/uploads/2011/10/Photo-01-10-2011-20-46-50-300x185.jpg" alt="" /></a></p>

  <h3 id="xbee-wire-antenna-module-series-1"><a href="http://uk.farnell.com/digi-international/xb24-awi-001/wire-antenna-module-xbee-1mw/dp/1337912">XBee Wire Antenna Module (Series 1)</a></h3>

  <p>Previously I’ve used the chip antenna versions of the XBee series one modules. These were great and really simple to work with but when I tried to use them   in a network around my house I had real issues trying to get the signal to pass through walls and ceilings/floors. The best solution would likely be to replace all of the XBee modules with alternatives with better antennas or with more powerful radios. To replace all of the modules would be quite expensive so I’ve decided to go with replacing just the module on the base station and hope that it does a better job of receiving the signals sent out by the other modules. This may be foolish but I’ll give it a go and see what happens.</p>

  <p>To replace the module should be a pretty simple job, it’s the same format as the chip antenna module so physically I just have to switch it out for the new one. Before I can do that though requires that I send some settings to the XBee to configure its ID and PAN ID as mentioned in my post about the temperature probes. To communicate with the Xbee I’m just going to use an Arduino Uno and wire the Din/Dout pins on the XBee to the RX/TX pins on the Arduino. I then load a blank sketch onto the Arduino and use the Serial Monitor to communicate with the XBee. What I’ll be sending to it is:</p>

  <figure class="highlight"><pre><code class="language-text" data-lang="text">ATMY=0,ID=1234</code></pre></figure>

  <p>Which broken down means:</p>

  <ul>
    <li>
      <p>AT _ - Attention!_</p>
    </li>
    <li>
      <p>MY=0, _ - the ID of this unit_</p>
    </li>
    <li>
      <p>ID=1234, _ - the PAN ID for the network of XBees_</p>
    </li>
  </ul>

  <p>To send those settings open your Arduino software, choose the right serial port in Tools -&gt; Serial Port then open the serial monitor with Tools -&gt; Serial Monitor. Choose to send no line endings from the drop down at the bottom and send <code class="highlighter-rouge">+++</code> then as quick as you can manage change to Carriage Return in the drop down first and send the above AT command. After the <code class="highlighter-rouge">+++</code> you should see <code class="highlighter-rouge">OK</code> and after the AT command you should see <code class="highlighter-rouge">OKOK</code>.</p>

  <p>You must also send an <code class="highlighter-rouge">ATWR</code> command too to write these settings to the flash memory.</p>

  <p>Once I did that I started seeing random bits of binary data appearing in the Serial Monitor. Fortunately this was a good thing, it was the temperature data from my probes starting to show up!</p>

  <p>I popped the newly configured module into my YAHMs hardware and waited to see if it worked…</p>

  <p>As mentioned it was already receiving data so I started to see temperatures show up straight away. The conservatory probe was working straight away which seemed positive as it had issues before. When I took it upstairs it didn’t seem to work at first so I tried angling the box so that the chip antenna would be “pointing” at the base station. This seemed to do the trick and I started collecting temperature readings. Unfortunately as time has gone by I’ve found that it works less and less. I’ve now moved it closer to the base station which got me a few more readings but again it has stopped working. It looks like I’m gonna need a bigger <del>boat</del> aerial!</p>

  <p>Hopefully I’m going to be able to try out an XBee with an external aerial which should work well on the base station, by this point I’ll have a few spare XBees, including one with a wire aerial so I should be able to get a much bigger range of readings.</p>

  <p><a href="/uploads/2011/10/yahms-delayed.png"><img src="/uploads/2011/10/yahms-delayed.png" alt="" /></a></p>


guid: https://blog.johnmckerrell.com/2011/09/27/reversing-the-brain-drain
url: https://blog.johnmckerrell.com/2011/09/27/reversing-the-brain-drain/
title: Reversing the Brain Drain
updated: 2011-09-27 10:57:01
published: 2011-09-27 10:57:01
fetched: 2019-06-09 18:11:41.817266
created_at: 2019-06-09 18:11:41.941790
updated_at: 2019-06-09 18:11:41.941790
summary:
  An unfortunate tweet from a friend yesterday suggesting that he might have to leave Liverpool to get a job, led to a little exchange:

content:
  <p>An unfortunate tweet from a friend yesterday suggesting that he might have to leave Liverpool to get a job, led to a little exchange:</p>

  <p><a href="https://twitter.com/mcknut/status/116983278065553408"><img src="/uploads/2011/09/does-trying.png" alt="" /></a></p>

  <p>That got me to a realisation:</p>

  <p><a href="https://twitter.com/mcknut/status/116986285842903040"><img src="/uploads/2011/09/does-rescue.png" alt="Tweet text: &quot;Recent tweet just made me realise that at @DoESLiverpool we've already recovered/rescued 4 people from working outside of Liverpool&quot;" /></a></p>

  <p>Now that comment is certainly meant to be taken as tongue in cheek, but there’s some very real truth in it.  To start with the more tenuous examples. Paul Freeman (<a href="http://twitter.com/OddEssay">@OddEssay</a>) contracts for a company in Eccles, near Manchester. Since setting up DoES Liverpool Paul has been able to spend more time in Liverpool. Yes he’s still working for a company in Eccles but he’s now buying lunch from Liverpool companies and engaging with the Liverpool tech community more regularly.</p>

  <p>Let’s move onto Paul Kinlan (<a href="http://twitter.com/Paul_Kinlan">@Paul_Kinlan</a>). Paul is a Developer Advocate with Google. He’s based in their London office but also spends a lot of his time travelling around Europe and the world promoting Google Chrome and HTML5. His main reason for being here is to work with developers who are using Google products in the north of England, but by supporting DoES Liverpool and taking a desk with us Paul is now able to spend more time in Liverpool meaning he gets to spend more time with his family who are still based up here.</p>

  <p>Finally we have Andy Hughes (<a href="http://twitter.com/andyhughes86">@andyhughes86</a>) and Andy Powell (<a href="http://twitter.com/p0welly">@p0welly</a>). They work for a company in Manchester who had offices in Stockport. Andy and Andy are both developers who worked in an office full of sales people. These were people who spent most of their time on the phone trying to make sales. Not the best environment for a pair of developers who need to get their head down and concentrate. Their company was moving into an office in Central Manchester, a smaller office in a trendier area which was going to mean Andy and Andy would either be in closer quarters with their noisier colleagues, or have to find somewhere else to work. Fortunately they came across DoES Liverpool and came in for their free hot desk day (bringing donuts!) They liked what they saw and took two permanent desks with us. Andy H lives in Runcorn, Andy P lives in New Brighton. Andy H unfortunately still has a long bus ride (but we’ll come back to that) while Andy P now has a 20 minute commute! As it turns out, Andy H is now moving to Liverpool. I had a chat with him recently and asked if he was planning to move to Liverpool before they joined us at DoES Liverpool. He told me that while he really wanted to move here he had resigned himself to living in Manchester. So not only do we have two people who are regularly coming to Liverpool and supporting businesses in the city centre, we’ve also got someone who would have had to live elsewhere being able to live in Liverpool as he’d hoped, paying taxes and again supporting local businesses.</p>

  <p>So this blog post certainly isn’t meant to suggest that other places are bad. Generally at the moment though if you want to work for a digital company in Liverpool your choices are to join a digital agency or go it alone. If you don’t want to do this you’ll probably need to work elsewhere. I also recognise that in all of the examples given the person is still working for a company based outside Liverpool. It’s nice though to recognise that even at this early stage DoES is supporting local people and even changing their lives for the better. Hopefully in the future as people start building new businesses based out of DoES those businesses will expand and will start recruiting more and more people from the city who will no longer have to leave just to find a good job.</p>

  <hr />

  <p>We also got some more news coverage recently, on Friday <a href="http://www.liverpooldailypost.co.uk/ldpbusiness/business-local/2011/09/23/google-s-breakfast-session-in-liverpool-92534-29471167/">getting in the Daily Post for hosting the OpenLabs Developer Breakfast events</a>, and an article 2 weeks ago that we only heard about because someone came into DoES for a look around, as a result of seeing it in the <a href="http://www.metro.co.uk/news/875522-google-builds-web-support-for-1-500-firms-in-liverpool">Metro</a>. As it turns out that one was syndicated so we were mentioned <a href="http://www.google.co.uk/search?client=safari&amp;rls=en&amp;q=%22Google+staff+will+support+a+new+technology+hub%22&amp;ie=UTF-8&amp;oe=UTF-8&amp;redir_esc=&amp;ei=ifl8TtTeO4ep8QOehfCyAQ">all over the place</a>!</p>


guid: tag:blogger.com,1999:blog-179325926611299376.post-7294325339811575330
url: http://chris-osm.blogspot.com/2018/09/rights-of-way-and-fruit.html
title: Rights of Way and fruit
published: 2018-09-14 11:28:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.275927
updated_at: 2019-10-07 01:30:08.391084
summary:
  I been wanting to get back to doing some cycling, I've tried before and struggled with the hills. I live in a village where every route back home is up hill. I went to the very interesting and very exciting Fully Charged Live show, all about renewable energy and low emission vehicles, such as electric cars. One of the areas there was about electric bicycles and electric skateboards and mountain boards. My brother loved the electric boards - he's a kite surfer and snowboarder, so he took to it straight away. I really, really liked the e-bikes. When I got home I checked out a local bike shop, borrowed an e-bike, sailed up the hill home with ease and bought the mountain bike version. It is fantastic.<br /><br />I'm happy with how courteous and respectful most drivers are, but there's always a few idiots and it only takes one to cause a serious problem, so I like to get off the roads. I've been wandering the paths and tracks near home on my new bike and loving the fact that hills are now not obstacles but instead are fun.<br /><br />Today I followed a route I've walked and cycled for years and I thought I knew very well, but as usual, when you take a close look at familiar things new stuff emerges. Mapping makes you look closely.<br /><br />I have the file of Public Rights of Way that the East Riding of Yorkshire council have finally released under OGLv3. I am grateful to Robert Whittaker for his persistent and skillful chasing of the council. He got the data released under the open licence where I had failed.<br /><br />In general the data shows where rights of way are present and shows the footpaths, bridleways and byways as expected. Today I followed a route which used various rights of way and all were signed with the yellow and green arrows and some fingerboards all put up by the council. You would expect these signs on the ground to match the data in the file from the council, but no.<br /><br />There is a part of Elloughton Dale where the path drops down a steep slope to the road through the valley. You can then follow the road for a couple of hundred metres before turning off to climb the other side of the valley. The descent is through woods and is steep enough that steps with wooden risers have been cut to prevent erosion. This however is not the route I took, because there is a more direct route that comes out of the woods opposite the point where the ascent starts, thus it avoids the steps and there is no need to use the road except to cross it. When I first used this more direct route, probably forty years ago, it was unofficial and the land owner put up barbed wire to stop it being used, but after a while a gate appeared instead (he was probably fed up with replacing the wire) and this was the preferred route to take. More recently, at least twenty five years ago, public right of way signs appeared both at the roadside and at the point in the woods where the route down the steps and this newer route diverge.<br /><br />This new route is not part of the data the East Riding of Yorkshire council publish and it is not on their 'Walking the Riding' map they publish on their website. So which to believe, council signs on the ground or the data they reluctantly published? Maybe their reluctance to publish was partly because they know the data is out of date. I have used what I know on the ground to map the route, though I can't add a prow_ref tag because that's not displayed on any of the signs.<br /><br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; text-align: right;"><tbody><tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-K6dYvHpTfbM/W5uatKctewI/AAAAAAAADa8/DxrSPNW80pU69tXrsAR7wzfIEqe2Gjf8wCLcBGAs/s1600/IMG_20180914_102451.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://2.bp.blogspot.com/-K6dYvHpTfbM/W5uatKctewI/AAAAAAAADa8/DxrSPNW80pU69tXrsAR7wzfIEqe2Gjf8wCLcBGAs/s320/IMG_20180914_102451.jpg" width="240" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Hawthorn looking good</td></tr></tbody></table>It has rained recently, but the long, dry summer means that there's hardly any sign of damp on the<br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-pudfx02232A/W5uat3vl_KI/AAAAAAAADbA/x2v5_pWANfU4urmqA_OtTGAOVZPU7KRmgCLcBGAs/s1600/IMG_20180914_102818.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://3.bp.blogspot.com/-pudfx02232A/W5uat3vl_KI/AAAAAAAADbA/x2v5_pWANfU4urmqA_OtTGAOVZPU7KRmgCLcBGAs/s320/IMG_20180914_102818.jpg" width="240" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">No fruit on the brambles&nbsp;</td></tr></tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-NngsGNxXScc/W5uaswfrTvI/AAAAAAAADa0/Gy7f7u9UceYqowQENmjETeWYFn3OzfeAQCLcBGAs/s1600/IMG_20180914_102612.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://3.bp.blogspot.com/-NngsGNxXScc/W5uaswfrTvI/AAAAAAAADa0/Gy7f7u9UceYqowQENmjETeWYFn3OzfeAQCLcBGAs/s320/IMG_20180914_102612.jpg" width="240" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Elder looking shrivelled</td></tr></tbody></table>ground. In the woods the ground is dry and even dusty in places and very few muddy spots even in places that are usually muddy. All this dry weather has taken its toll on wild fruit, but some more so than others. Bullaces seem a bit small but are abundant, hawthorn looks to be a fairly average number and a good size. Elder has few berries and they are small and many shrivelled one. The big loser this year are brambles (blackberries). There is hardly a single fruit to be seen with all the remnants of flowers hardly showing as any berries at all. Any wildlife that usually feasts on the masses of brambles around the paths and tracks here will have a to find something else if they want to survive.<br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"></div>


guid: tag:blogger.com,1999:blog-179325926611299376.post-7982025667568608764
url: http://chris-osm.blogspot.com/2017/03/have-you-moved.html
title: Have you moved ...
published: 2017-03-07 22:38:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.278863
updated_at: 2019-10-07 01:30:08.393743
summary:
  ... or was it just your postcode?<br /><br />The new open version of GB postcodes has some strange anomalies in it. There are nearly 1.7 million postcodes, but there are over 45,000 that have moved location since the November 2016 version. Most that have moved have moved less than 20 metres but 1,520 have moved 500m or more. There are 15 that have moved more than 10km, with the furthest moving over 62km.<br /><br />The source of these open postcodes have changed, maybe the process that generates them has a problem in it. I'm going to look into this some more and try to get some information from Ordnance Survey too.<br /><br />Maybe this is normal but it feels odd to me.


guid: tag:blogger.com,1999:blog-179325926611299376.post-1467800650664439387
url: http://chris-osm.blogspot.com/2017/03/new-postcodes.html
title: New Postcodes
published: 2017-03-05 15:14:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.281339
updated_at: 2019-10-07 01:30:08.395921
summary:
  I have thought about a tool to see what postcodes have been added recently for a while, so now I've finally got around to looking at it. The UK Office of National Statistics provide all kind of open data under the Open Government Licence and one of these is a list of postcode centroids for Great Britain. The dataset includes the date a postcode was added.&nbsp;<div><br /></div><div>When an area is quite well mapped it needs to be kept up-to-date. A mapper may have visited somewhere to map it when it's not normally somewhere she might visit, so any new developments might not get mapped as it is built. These new developments all get new postcodes, so showing a map with all the recently added postcodes might help mappers find places that need another survey.</div><div><br /></div><div>I've put together a map that shows all of the postcodes added since the beginning of 2016. You can see it here&nbsp;<a href="http://pcdates.raggedred.net/#10/52.7159/-1.3149">http://pcdates.raggedred.net/#10/52.7159/-1.3149</a>. &nbsp;There would be quite a lot of markers in some areas so I use the excellent MarkerCluster plugin to make the markers more manageable. Each marker is a single centroid. If you click on it it shows a popup with the postcode and start date in it.&nbsp;</div><div><br /></div><div>If it is useful please let me know what you think. Would you like anything changed? If people like it I will maintain it as each new release of postcode data &nbsp;id available, which is currently four times each year.</div>


guid: tag:blogger.com,1999:blog-179325926611299376.post-4938906376465446899
url: http://chris-osm.blogspot.com/2017/03/postcode-changes.html
title: Postcode changes
published: 2017-03-04 12:25:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.283976
updated_at: 2019-10-07 01:30:08.398088
summary:
  I have provided a tile layer of postcode centroids based on the Codepoint Open postcodes since they were first published in 2010. I also provide a similar layer for postcodes based on the Office of National Statistics dataset called ONSPD. I have just updated the two layers with the recently updated data.<br /><br />The ONSPD dataset contains almost a million more records than the Codepoint Open one. Most of these extra codes are retired postcodes. There are also some entries for BT codes. These are Northern Irish postcodes which are not released under OGL, so I don't publish them - I wouldn't want them to be used as a source for Northern Irish postcodes in OSM as that would violate their copyright. There are some IM codes, but without any coordinates. These are Isle of Man codes. I don't believe they are released under an open licence so again I don't publish them, but having no coordinates makes them useless for this purpose anyway. <br /><br />The ONSPD dataset has a column for the start date and one for the end date. The earliest start date (just year and month) seems to be 1996-06 which most active postcodes have, the most recent start date is 2017-01.<br /><br />Edit: The spread of dates is much wider than I first thought, with the earliest is 1973-08.<br /><br />I think showing any newly added postcodes would be useful to help people see where there may be new developments that need surveying. Some new postcodes will just be yet another code on a building that already has many postcodes. These are places that get lots of post (after all that's what postcodes are for), often royal mail buildings where there are PO Box addresses.<br /><br />The recently published files have changed their formats (again). Now the Codepoint Open dataset has fixed length postcode fields, so the natural spacing is lost unless they are reformatted, which is very easy. In the past the list of postcodes from Codepoint Open exactly matched the processed list of postcodes from ONSPD, now they don't. I'm examining what it is that is different.<br /><br />Many of the centroids have moved a few metres. I can understand that a centroid might move if the street or section of street had some new houses (delivery points) added or removed, but that is not the cause of most of the changes. It seems the reason for the change is that the source of the centroids has changed. The data used to come from Address Point (which is now discontinued), now it comes from the Postal Address Location Feed of Geoplace. This is interesting because it removes the connection with Royal Mail and gets the postcode from local authority data. Local authorities are where addresses are created, though Royal Mail must have a part to play in adding the postcode. Getting the data from Geoplace may be one small step towards releasing UK address data as open data. I hope so.<br /><br />To see how to use the tile layers follow the links for <a href="http://codepoint.raggedred.net/" target="_blank">Codepoint Open</a> and <a href="http://onspd.raggedred.net/" target="_blank">ONSPD</a>.<br /><br />Edit:<br />I have discovered there are 111 postcodes in the Codepoint Open dataset that are not in the ONSPD dataset after I have processed it. These are 'large user' postcodes and they don't have useful coordinates. In the Codepoint Open dataset there is a field called positional quality indicator. 10 indicates the best positional quality, 90 indicates the worst. All 111 are 90. In future I'll ignore these entries.


guid: tag:blogger.com,1999:blog-179325926611299376.post-8944826039361377599
url: http://chris-osm.blogspot.com/2017/01/go-pokemon.html
title: Go Pokémon
published: 2017-01-26 15:43:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.287098
updated_at: 2019-10-07 01:30:08.400810
summary:
  I'm always keen to see new contributors to OSM. They bring details that interest them to OSM, be it their local area, their sport, their hobby, their work or whatever. Having an open tagging scheme that allows all kinds of objects to be mapped is part of the strength of OSM.<br /><br />I follow new mappers in my area and sometimes add comments to their changesets or send them a welcome message. Sometimes a new mapper adds one thing and never adds more, sometimes they add one type of thing over wide areas. One thing the East Riding of Yorkshire needs is more fine detail of rural footpaths and bridleways. A few new mappers appeared starting to add footways and I was encouraged that this may spread. Sadly I was wrong and what I hoped was something useful is turning into something less so.<br /><br />It seems that someone has discovered that some details in OSM may be being used in Pokémon Go, the virtual reality game by Niantic to find imaginary creatures. These imaginary creatures appear more near water and in parks near footpaths. The new mappers were adding footpaths in parks to encourage these beasties to appear. Their new footpaths were in good places and were welcome, but then word spread.<br /><br />People then started adding more parks and ponds. Some were real parks and ponds that were just missing from the map, but some were plonked anywhere, over existing buildings and roads, over schools or whatever. A few people realised that changing existing areas' tagging was easier, so suddenly a building becomes a park or a pond.<br /><br />It is not clear to me that Niantic are using OSM data, but if they are, adding phoney stuff won't work very well because it will be found and cleaned up quickly. If Niantic are using OSM data they will probably only incorporate changes to OSM data occasionally, if at all.<br /><br />It's great to see new mappers. Maybe a few will become more interested in mapping than Pokémon. All the real parks, paths and waterways that are being added are all helpful. Tidying up the junk is nuisance but maybe I'll spot other stuff to improve too.<br /><br />Back to the new users feed. Oh, there's another one ...


guid: tag:blogger.com,1999:blog-179325926611299376.post-7341319206121853848
url: http://chris-osm.blogspot.com/2016/12/leaflet-plugin.html
title: Leaflet plugin
published: 2016-12-12 14:05:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.289425
updated_at: 2019-10-07 01:30:08.402991
summary:
  <div class="tr_bq">I have been trying to replace a long-standing map based on the OpenLayers 2 library to LeafletJS. The main sticking point was that OpenLayers has a useful permalink function that changes the URL to include a query string that identifies the map location and which layers should be shown.&nbsp;</div><br />Leaflet has a neat plugin to <a href="https://github.com/mlevans/leaflet-hash">show the location as a hash</a> but that doesn't include the layers to show. After a bit of investigation I decided to try to write a Leaflet plugin to make all this easier.<br /><br />If a map has optional base layers and some overlays it often will use the Layers control to make the choices easy to select. The layers are not easily accessible from the map object as only the currently displayed layers, not the hidden ones, are available. The Layers control has access to all the layers, but that control is not accessible from the central map object. The only way to have access to all of the layers is from the Layers control, so a new control, inherited from the Layers control seemed to be the right answer. HashLayers control was developed.<br /><br />HashLayers does all that the Layers control does, of course, but adds the functionality of the URL hash. The hash looks like this:<br /><blockquote class="tr_bq"><span style="background-color: #eeeeee;">#zoom/longitude/latitude/layers</span></blockquote>This gets appended to the URL of the page that the map is on. Using the hash means the position and layers can be changed dynamically as the map is zoomed or scrolled around and as different layers are selected. Changing the hash value of a URL doesn't force a reload of the page and doesn't show up in browser history. <br /><br />The zoom is the current zoom level and longitude and latitude are the centre of the map. The layers is a string representing the base layers and any overlays. If a base layer is selected 'B' is shown, 'O' shows an available base layer not selected. The overlays are shown as a string of 'T' or 'F' (true or false) showing if they are shown or not. A typical hash might be:<br /><blockquote><span style="background-color: #eeeeee;">#12/-0.393105/53.775095/BOFFTF</span></blockquote>&nbsp;This has the first base layer visible, the second one not showing and four overlays available, but with only the third one visible. These selections will be reflected in the visible selection control too.<br /><br />If a map is created with the HashLayers control on it and a hash is passed in the URL, the map will centre on the location and zoom specified and show the requested layers. If no hash is passed then the default location and zoom will be used from the setView() or map options as normal. Layers will be shown as they were added to the map. A hash to reflect all this will be appended to the URL which gets updated with every change. There is no need for the OpenLayers permalink control.<br /><br />I now need to test it more carefully and decide if it is worth publishing as a Leaflet plugin.


guid: tag:blogger.com,1999:blog-179325926611299376.post-1259980533065737603
url: http://chris-osm.blogspot.com/2016/12/replacing-openlayers.html
title: Replacing OpenLayers
published: 2016-12-03 23:56:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.291810
updated_at: 2019-10-07 01:30:08.405214
summary:
  When OS released their Open Data I created a simple map that showed a few of their offerings as overlays on an OSM base layer <a href="http://oscompare.raggedred.net/">oscompare.raggedred.net</a>. At the time my maps were driven by OpenLayers. OpenLayers v2 works well but is a lot harder to use than LeafletJS. As soon as Leaflet arrived I started to use that and it has only got better as time has passed. There is OpenLayers v3 available I think, but I doubt I'll use it.<br /><br />OpenLayers has a permalink option that changes the URL in the browser to include a query string describing the location of the centre of the map and the zoom level. This lets you bookmark or send a link that will open the map in the right place. Leaflet has a plugin to create a hash on the URL that does a similar thing. &nbsp;It works well but there is a bit of a problem.<br /><br />The OpenLayers permalink adds a 'layers=' bit to the URL. An example would be BOTFF. This means the first base layer is shown and not the second one, then the first overlay is visible but not the second or third. This great, because not only does the URL make map show the right place, but the overlay you wanted to show will also appear too. Leaflet's hash plugin doesn't do this and that is a problem for me.<br /><br />I decided to replace the map showing the OS overlays using Leaflet. It was quick to do and works well, but I can't use it as the replacement for the OpenLayers map because Leaflet doesn't have a built-in way to show the layers that were selected. People link to my map, according to the logs, with the query string to select the place and layers. If I just replaced it with the Leaflet-based map their links would all fail. They couldn't create a new bookmark that includes the layers to show either. I decided to fix this.<br /><br />I looked into the documentation for the latest version of Leaflet (1.0.2 at the time of writing). The documentation is very useful. If there are overlays on a map a layers control can be added that allows the overlays to be selected. There can also be more than one base layer that the control allows you to select.<br /><br />I thought about a generic way to create a similar effect to the OpenLayers permalink. For a one-off map I can code up the layers to do as I need, but a generic version would be useful. I can get at the layers that a map is currently showing, but Leaflet's way of hiding a layer is to remove it from the map. The definition of that layer is then not accessible from the map object. I couldn't see how to find the list of layers including the ones not displayed. The layers control has a list of the layers it uses, which would be ideal, but I couldn't find any way to find a a reference to that from the map object. There doesn't seem to be any way to find any controls from the map object. A map method that returns a list of controls might be useful.<br /><br />So far I have created a bespoke version of the Leaflet map I wanted. It will accept an OpenLayers-style URL, display the map and create a Leaflet-style URL. I'm thinking about how to duplicate the OpenLayers feature more generically. I have ideas ...


guid: tag:blogger.com,1999:blog-179325926611299376.post-302540053856758893
url: http://chris-osm.blogspot.com/2016/11/a-map-in-app.html
title: A Map in an App
published: 2016-11-14 23:10:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.294141
updated_at: 2019-10-07 01:30:08.407340
summary:
  Maps come in all shapes and sizes. I've made maps for a few different uses and since Openstreetmap came along there's always been some part of OSM in my maps. I offered to make a map for <a href="http://www.clockenflap.com/">Clockenflap Music and Arts Festival</a> in Hong Kong, so I expected to use OSM as the base. What I've ended up with uses the tools to create and display the map: JOSM, osm2pgsql, Tilemill, Mapnik and Leaflet, but none of OSM's greatest asset, its data.<br /><br />Clockenflap, like most festivals, is largely a temporary venue. This year it's a new venue to Clockenflap on the <a href="http://www.openstreetmap.org/#map=17/22.28401/114.16299">Central Harbour-front Event Space</a>. I looked at the area on OSM and how to map the temporary stuff that the event will put up. There's seven stages, food stalls, bars, attractions, art installations and all the usual support stuff like toilets, first aid and entrance and exit gates. That's a lot of stuff to add to OSM temporarily. Worst of all, when I started to look at it the site design was still not fixed. I quickly decided that it needed to be stored locally and not uploaded to OSM at all.<br /><br />The map is embedded in an app, which I'm not writing. The idea is to run the map in a Webview with a JavaScript Leaflet-based display and it works well. The map has a base layer with all the services on it and layers of overlays with icons that can be clicked. In the apps (one iOS and one Android) the JavaScript in the Webview calls a native iOS or Android function when an icon is touched.&nbsp; So instead of a web popup the app can respond with a dialog or a more detailed response. For example, if a stage icon is touched the app can show what acts are on the stage now and on next.&nbsp; The reverse is true too. For example, if the user scrolls through the list of food suppliers in the app and chooses one, the map is called and shows where the food stall is on the site. I can't show the apps yet, but a working example of the map is <a href="http://clockenflap.raggedred.net/">here</a>.<br /><br />The festival team provided the icons and the look of the map. I think it looks very good. There are sponsors who need to see their logos and colour schemes and the map needs to reflect this, as well as being useful for getting around on a site that no one knows their way around because it is only a day or two old. <br /><br />A few things are quite different from most of the maps I've made before. Driving the map from a phone screen is a challenge.&nbsp; Trying to get specific icons on the map so they are big enough to read on smaller screen, but don't overlap too much is tricky. At the lowest zoom level (17) very little is shown by default. If all the overlays are show it is a jumble, there's just no choice.<br /><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="http://clockenflap.raggedred.net/" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="226" src="https://2.bp.blogspot.com/-Wmv7Vb1IYsU/WCo_zEFb46I/AAAAAAAAChQ/430qu4-sRo0YoDkVO4fp1dCautm03PyjQCLcB/s400/jumble.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Lots to fit on at low zoom if everything is turned on</td></tr></tbody></table><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody><tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-1_kQ3izeq-E/WCpBylk9ffI/AAAAAAAAChc/9q0bG-ZPWcMQ7UHi9QgLeFAt_nGkqSx3wCLcB/s1600/foodstalls.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://3.bp.blogspot.com/-1_kQ3izeq-E/WCpBylk9ffI/AAAAAAAAChc/9q0bG-ZPWcMQ7UHi9QgLeFAt_nGkqSx3wCLcB/s320/foodstalls.png" width="199" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Food stalls need room for their much longer real names </td></tr></tbody></table>By the time we get to the highest zoom level (22) there's lots of space in most places. I needed that zoom level to comfortably resolve the food stalls, which are 3m x 3m, with an icon and room for their name. I needed to display a lot of food supplier names close together and the names needed to respond to a touch (or click). I used the Leaflet plugin Leaflet.Label for this. If I had used Leaflet 1.0, rather than 0.7, I think I could have used built-in tool tips instead.<br /><br />There is an optical illusion when you zoom in on an icon, especially on a phone where you use pinch zoom. If you show the higher zoom level with the same size icon on it the icon seems to have shrunk. People expect pinch zoom to make things bigger, so I increase the size of the icons as each level zooms in. That is easy for the stuff on the base map, Tilemill makes that simple, but scaling the icons on overlays took a bit of code I'd not needed to use before. This involves resizing the icon and using setIcon() to use the newly sized icon in a marker. setIcon() is not documented in Leaflet 0.7, but works just the same.<br /><br />The standard Leaflet layer control is much too small for a mobile phone. I tweaked the size and position with CSS. I also discovered that you don't need a base layer on the control and you can force it to always be open. Both are perfectly plainly described in the really good documentation of Leaflet, I'd just not read that bit before. <br /><br />I control the markers on the map by describing them in geoJson files, one for each overlay. To position the many markers on the map I created a simple editor which uses the Leaflet.Draw plugin, which is really simple to use in code and produces a very useful editor.<br /><br />I have enjoyed creating this map and working with creative people who have an absolute deadline to work to. If you are in Hong Kong on 25 - 27th November enjoy Clockenflap and enjoy using my map to find all the stuff Clockenflap has to offer.


guid: tag:blogger.com,1999:blog-179325926611299376.post-5535131837123107308
url: http://chris-osm.blogspot.com/2016/02/drains-and-meadows.html
title: Drains and meadows
published: 2016-02-05 16:30:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.296527
updated_at: 2019-10-07 01:30:08.409624
summary:
  I mapper jumped into mapping the rural areas of East Yorkshire more than a year ago but, sadly, seems to have stopped. This mapper has added farmland and woods in much more detail than anything before and has drawn the outlines of lots of buildings in villages too. There are a couple of things that I've decided to change in any areas that I take a look at now. That is meadows and drains.<br /><br />Meadows are a special thing in Britain, with most having disappeared. They are not just areas of grassland but a very specific kind of management, involving no fertiliser, a careful cutting of the grass for hay more than once a year and some animal grazing, but not all year round. The timing of the hay cuts allows wild flowers to flourish as they have time to set seed before they are cut down. This makes a beautiful landscape that has all but vanished from the English countryside. The mapper has made the mistake that almost any green field in the aerial imagery is a meadow, so I'm checking as best I can on the ground and changing it to farmland unless it really is a meadow. The farmland is drawn with each field outlined which means I can try to add hedges when I see them. This too improves the quality of the rural mapping.<br /><br />The other problem is that every field-side ditch has been mapped as a drain. This is further worsened by another mapper doing the same thing in Holderness, the flat land to the east of Hull. There are some waterways I would call a drain there, but most are ditches in my view. A ride out doesn't cover much ground to correct this and imagery is ambiguous at best. So there's lots to check and much will change.<br /><br />The East Riding countryside is a great place to spend time; having an excuse to carefully explore and examine more of it in detail is no bad thing, so I'm not complaining.


guid: tag:blogger.com,1999:blog-179325926611299376.post-5291690938405807165
url: http://chris-osm.blogspot.com/2016/02/schools.html
title: Schools
published: 2016-02-02 12:08:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.299165
updated_at: 2019-10-07 01:30:08.411824
summary:
  People meet up in a few areas across Britain drawn together to share views and ideas about OSM. One of these is known as Mappa Mercia, meeting in the old Kingdom of Mercia, roughly the Midlands. They started a quarterly series of projects which encourages people to focus on improving specific areas, such as nature reserves. This quarter it is schools.<br /><br />People across Britain have taken to the suggestion of checking schools and improving the quality of recording schools. This is a simple idea that will greatly improve the quality of data in OSM, something that the quarterly projects have always aimed at. It's a great idea and something I support.<br /><br />There is a list of UK schools available as open data which Robert has used to show a map of how well the schools in the list are represented in OSM. It uses postcode areas as that is how the open data is structured. This is a very good indicator of areas that still need work.<br /><br />This is all useful, but then I think the project took a wrong turn. People were encouraged to add a hashtag to changeset comments. This has then been used to produce a list of people who have contributed the most changes to the project. I don't like this. OSM is not a competition and turning it into one encourages edits just to climb the list (I'm not accusing anyone of that, just that the pressure is created). We have seen that with the list of anomalies published between OS Locator road names and OSM road names. To push an area up the list people just use the OS Locator names without checking. I know that OS Locator contains errors and there is a mechanism to record these, but many areas have very few recorded errors, so erroneous OS data is now in OSM.<br /><br />I like the idea of focussing on improving specific things in OSM. It should encourage surveying. I would prefer it is not be seen as a competition.


guid: tag:blogger.com,1999:blog-179325926611299376.post-3467663473585363469
url: http://chris-osm.blogspot.com/2015/11/when-is-town-not-town.html
title: When is a town not a town?
published: 2015-11-27 16:43:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.301470
updated_at: 2019-10-07 01:30:08.414116
summary:
  I live a few miles away from a rapidly growing settlement called <a href="http://www.openstreetmap.org/#map=14/53.7333/-0.5720">Brough</a>. I see it mentioned from time to time as a town in the local press, but I've always thought of it as a large village. It has been a settlement for a long time, it was a town, Peturia, when Rome ruled this area.<br /><br />The idea of not knowing whether a place in England is a town or a village seems easy to resolve: does it have a town council or a parish council? Well in the case of Brough it is not quite so easy. The local council is a town council, but it covers both Elloughton and Brough and is called Elloughton-cum-Brough Town Council. It is not that uncommon to find a civil parish or town council with multiple settlements within its bounds, but the council is a town council, surely one of the settlements needs to be a town.<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-xlJaJbhV8zc/VliIHMH1IpI/AAAAAAAAB8o/A8kf_Bo-AEc/s1600/ell-cum-brough.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="318" src="http://1.bp.blogspot.com/-xlJaJbhV8zc/VliIHMH1IpI/AAAAAAAAB8o/A8kf_Bo-AEc/s320/ell-cum-brough.JPG" width="320" /></a></div>It is clear to me there are two settlements here. Road signs show both places sometimes in different directions, OS Locator shows the street names as streetname : Brough : Elloughton-cum-Brough and streetname : Elloughton : Elloughton-cum-Brough.<br /><br />OS Open Names shows both places as populatedPlace: village. So is it possible to have a town council presiding over an area with two villages and no towns in it?<br /><br />I've asked the Local Authority, who, after all, provide the data for OS Locator and OS Open Names, what they think.<br /><br />The answer is a political hot potato at the moment, as Elloughton-cum-Brough town council have recently decided to spend £4000 on a mayoral-style chain for the head of the council. It seems he can legitimately call himself mayor after the town council made the appropriate change in 2011. That's a lot of money to spend on what some call trinkets at a time when there are council cuts elsewhere.<br /><br />Edit:<br />The clerk to the Elloughton-cum-Brough has contacted me and kindly given me information about the process that resulted in the parish council becoming a town council. She also confirmed that both Brough and Elloughton are villages. That means that the town council area has two villages and no towns in it.<br /><br />


guid: tag:blogger.com,1999:blog-179325926611299376.post-1101028669053509699
url: http://chris-osm.blogspot.com/2015/09/extracting-building-heights-from-lidar.html
title: Extracting building heights from LIDAR
published: 2015-09-28 13:36:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.303927
updated_at: 2019-10-07 01:30:08.416401
summary:
  The UK Environment Agency have released some LIDAR data in Digital Elevation Model (DEM) format. It includes Digital Terrain Model (DTM) data and Digital Surface Model data. The DSM data includes buildings and trees while the DTM is processed to remove these so the underlying terrain is visible. Tim Waters asked if you subtract the DTM from the DSM would you be left with just building and tree heights? I'd started to look at this and it turns out building heights are extractable in this way.<br /><br />I have written a script to do the subtraction and create both the difference file and an SQL file to load the data into a postgresql table. I created a hill shading image from one of the difference files to see what features have been stripped from the DTM data. Here's a jpg version of it:<br /><br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-A8irnPsuREM/Vgk2NZO443I/AAAAAAAAB5k/fu5mrBI8HKc/s1600/sub.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://2.bp.blogspot.com/-A8irnPsuREM/Vgk2NZO443I/AAAAAAAAB5k/fu5mrBI8HKc/s400/sub.jpg" width="400" /></a></div><br />You can see that the buildings and some other features are all well defined. Phil Endecott suggested using the DSM data to create building outlines which could be traced in OSM. His images look good. I would suggest starting his process with this difference data as all the terrain detail has been removed so it may be even better.<br /><br />Once the SQL file of height data has been loaded into a Postgresql database, which has the PostGIS extension installed, we can then do some queries on it. I selected some OSM building polygons in the range of the loaded data and found which of the height points fell within each polygon. The highest of these is the highest point of the building above the surrounding terrain. I've written a script as a place-holder to extract all the heights for a rectangular area. Someone could extend this to make a file to upload to OSM to add the heights for every building in the defined area. I feel this is clearly an import and so the usual <a href="http://wiki.openstreetmap.org/wiki/Automated_Edits_code_of_conduct">OSM import process</a> needs to be gone through before the import takes place.<br /><br /><br />I think there is real merit in using this data to extract building heights, which are needed for the 3D images of city buildings.<br /><br />The two scripts are available here: <a href="http://www.raggedred.net/shared/heights.zip">http://www.raggedred.net/shared/heights.zip</a> The first (s-t.py) needs matching DSM and DTM files and outputs a difference file and the SQL to load into a database table. This will work for any of the resolutions published by the EA. The second, much less polished, script (match.py) defines a rectangle, extracts the OSM buildings for that rectangle and then finds the height data for each building. I wrote it as a script so it can be extended to create a data file for processing or uploading or so overlay tiles could be made from it. I loaded some OSM data with osm2pgsql (which would normally be used for rendering) and added the table for the heights data to the database. The SQL for the table is:<br /><blockquote class="tr_bq"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;"><br /></span><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">CREATE TABLE eadata<br />(<br />&nbsp; hid serial NOT NULL,<br />&nbsp; height double precision,<br />&nbsp; locn geometry(Point,27700),<br />&nbsp; CONSTRAINT eaheight_prim PRIMARY KEY (hid)<br />)<br />WITH (<br />&nbsp; OIDS=FALSE<br />);<br /><br />CREATE INDEX eadata_index<br />&nbsp; ON eadata<br />&nbsp; USING gist<br />&nbsp; (locn);</span></blockquote>The output SQL data can be loaded into this with the command:<br /><blockquote class="tr_bq">psql <dbname> -f <filename></filename></dbname></blockquote>where <dbname> and <filename> are whatever you used. The table name eadata and field names are hard-coded in s-t.py.&nbsp;</filename></dbname><br /><br />It is important to say that I would not use a rendering database to create the upload data from as some fields will be missing. osm2pgsql is a lossy process. You can use the OSM ID to extract a current version from the API or from Overpass to add the height data to. I used the rendering data for convenience as I already had it available and to satisfy myself that the process works. &nbsp; <br /><br />I hope this is useful to someone. Please feel free to ask for more information if I've not made anything clear.


guid: tag:blogger.com,1999:blog-179325926611299376.post-4364259102000178425
url: http://chris-osm.blogspot.com/2015/09/more-lidar-goodness.html
title: More LIDAR goodness
published: 2015-09-20 17:31:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.306290
updated_at: 2019-10-07 01:30:08.418838
summary:
  I looked into LIDAR data from the <a href="https://www.gov.uk/government/organisations/environment-agency">UK Environment Agency</a> some weeks ago. I needed it help a local group who are investigating flood mitigation options. The data was listed as being about £26,500 but we got a 100% discount if we used it for restricted, research use, so we could afford it! A few weeks after I'd used the height data for the group I got an email from the Environment Agency. They said the data was being made available as Open Data under the <a href="http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/">Open Government Licence</a>. So now I could use it for any other purposes at no cost. You can get the data from <a href="http://environment.data.gov.uk/ds/survey">http://environment.data.gov.uk/ds/survey</a><br /><br />I decided to make a detailed relief map of part of the area close to home. The data doesn't cover the whole country, only parts that are deemed at risk of flooding. All of Hull and the river Hull catchment area are included in this. I've only looked at my local area so other areas may vary.<br /><br />The data is downloaded as Ordnance Survey 10km grid tiles. There are 2m, 1m, 50cm and 25cm options and digital terrain model and digital surface model options too, so let's look at these options, but first a bit about LIDAR.<br /><br /><a href="https://en.wikipedia.org/wiki/Lidar">LIDAR</a> is a technology that uses laser light to measure a distance repeatedly over an area to create a 3D model of an area. If the LIDAR transceiver is mounted at a fixed point it can pan around to record a very detailed image in 3D of everything that can be seen from that point. It works very well in this way inside a building or a cave to make a very accurate model. The US Space Shuttle flew a mission to use a variant of LIDAR to record the height of the surface of the Earth from space. This is available as <a href="http://www2.jpl.nasa.gov/srtm/">SRTM</a>.<br /><br />More recently LIDAR equipment has been flown in aircraft. The difficulty of making useful measurements from an aircraft should not be underestimated. The only data LIDAR returns is distance to the target, so knowing PRECISELY where the aircraft is in 3D is the real problem. GPS is hopeless at altitude measurement and scarcely good enough for lateral location, barometric height measurements vary over time and location and inertial dead-reckoning accuracy falls off with time. A combination of all of these plus post-processing can result in useful data.<br /><br />The Environment Agency LIDAR distance options specify the distance between the sample points, the 2m option having less detail than the 25cm option. The area that these options cover varies with the highest detail covering the smallest area. I chose the 50cm option as it covered the area I wanted at the highest level of detail. The detail does make for larger datasets and more processing needed to do anything with it.<br /><br />Clearly the LIDAR measures the distance to the first object it encounters from the aircraft, so it measures tree tops, building roofs and even vehicles. This is known as the digital surface model. This is often a composite from multiple images, as this data is, to compensate for location inaccuracies and to help remove things like vehicles. To get a useful model of the real landscape, without trees and buildings, the data is post processed to create the digital terrain model. This is the data I have used.<br /><br />The OGL data was different from the data the Environment Agency originally supplied. The original data was in smaller grid squares and the height was rounded to the nearest centimetre. The OGL data is in bigger squares which makes it a bit easier to process but seems to use 18 decimal places of a metre, which is smaller than the diameter of an atom.<br /><br />I wanted to create a relief map and make contours from the data and, not for the first time, <a href="http://www.gdal.org/">GDAL</a> had the tools. The data uses the UK Ordnance Survey projection, known as OSGB36 or ESPG:27700, so to use any OSM data with it I would need to reproject to WGS84 or EPSG:4326.<br /><br />To make a relief map I used gdaldem with the hillshade option on each of the datafiles. These need to be joined together to make a larger image, so the option -compute-edges is also needed. The complete command is:<br /><blockquote class="tr_bq"><span style="font-family: inherit;">gdaldem hillshade -compute_edges infile relieftiff/outfile.tif</span></blockquote>The output is geoTIFF files which can be merged into a single (large) geoTIFF with the command<br /><blockquote class="tr_bq">gdal_merge.py -o big.tif *.tif</blockquote>This creates a geoTIFF file which has the image of the relief in a TIFF image and also has the locations of the edges in the original OS projection.<br /><br />The next step is to use gdalwarp to reproject the large tiff file to one in the WGS84 projection. The command describes the source and target projection and filenames. There are significant missing pieces in the large TIFF as the available data was not rectangular. The -srcnodata 0 and -dstalpha makes missing data transparent rather than black.<br /><blockquote class="tr_bq">gdalwarp -s_srs epsg:27700 -t_srs epsg:4326 -srcnodata 0 -dstalpha big.tif bigr.tif</blockquote>The new TIFF file is what we want to see, but now it needs turning into tiles to be displayed on a slippy map. I decided that zoom level 13 to 18 would give a useful display. To make these tiles I used gdal2tiles.py, specifying the reprojected TIFF image, the zoom levels and the folder to put the tiles into.<br /><blockquote class="tr_bq">gdal2tiles.py -z13-19 bigrt.tif tiles</blockquote>This makes a set of tiles in the TMS format in the specified folder, in this case tiles.<br /><br />Another way to visualise the LIDAR data is contours. I decided to create a set of overlay tiles that are transparent except for the contours. These can have a different density of contours at each zoom level. I chose the smallest contour step to be shown at the highest zoom level to be 0.2 metres. The GDAL tool for the job is gdal_contour which makes a shapefile with a linestring for each contour. The command is<br /><blockquote class="tr_bq">gdal_contour -i 0.2 -a height infile outshapefile.shp</blockquote>The resulting shapefile needs to be reprojected to WGS84. The tool to reproject shapefiles is ogr2ogr<br /><blockquote class="tr_bq">ogr2ogr -t_srs epsg:4326 -s_srs epsg:27700 outshape.shp new.shp</blockquote>I decided to use Mapnik to make the contour overlay tiles. Mapnik can use shapefiles but specifying the long list of shapefiles created above would be a problem so I loaded the shapefiles into a postgresql table in a database with PostGIS enabled. Postgresql comes with shp2pgsql to do this:<br /><blockquote class="tr_bq">shp2pgsql -a -g way new.shp eacontours &gt; new.sql</blockquote>This makes SQL to load the shapefile into eacontours table, putting the geometry in the field called way. To load this into a database called ukcontours which already has the postgis extension installed the command is<br /><blockquote class="tr_bq">psql -d ukcontours -f new.sql</blockquote>I then designed the overlay with Tilemill to create the transparent tiles with more contours at higher zoom levels.<br /><br />You can see the results at <a href="http://relief.raggedred.net/">http://relief.raggedred.net</a>. I added a water overlay and a roads overlay (thanks to MapQuest for the roads) to help position the relief imagery.


guid: tag:blogger.com,1999:blog-179325926611299376.post-2585971643263278746
url: http://chris-osm.blogspot.com/2015/08/strapped.html
title: Strapped
published: 2015-08-11 17:14:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.308733
updated_at: 2019-10-07 01:30:08.421083
summary:
  Hull is a low-lying city, as are some of the towns and villages that surround it. I have blogged before about the serious flooding in the area in 2007, just before I started mapping in OSM. What a lot of people won't know, even locals, is that flooding is a regular occurrence in some parts of the area. This flooding is on a much smaller scale than happened in 2007, but it still causes misery for anyone whose house is flooded. The Cottingham Flood Action Group have tried to understand what causes this regular flooding and then campaign to fix the causes. I have helped a bit by providing maps and overlaying data from various sources. Some of these sources are not open so I don't want to publish them here.<br /><br />One source of contention is riparian ownership. This can apply when a water course runs through your land and you may then be required to maintain it. For example a ditch carrying surface water away may need to be cleaned out so water can actually flow in the ditch and not back up causing a flood elsewhere.<br /><br />The East Riding of Yorkshire council is strapped for cash. They are pushing people to maintain ditches and in some cases ditches in culverts to prevent flooding, claiming these people are riparian owners of the ditches. To be a riparian owner the watercourse must be wholly within the property or form the boundary of the property. If the watercourse forms a boundary, then the property owner owns the watercourse to the mid line.<br /><br />I have matched the Land Registry Inspire dataset to my surveys of a few of these and it clearly shows that the ditch is many metres outside of the privately owned land and well within the area of the public highway. This means that no matter how strapped the council is for cash, they must maintain the ditch or culvert. Of course, my survey probably isn't enough to convince the council, but it certainly should prompt people to get more evidence.<br /><br />Don't believe your council. If they claim you are responsible for something unusual, make them prove it and make sure you get enough facts to stand up to them. The council may be right, but they be just trying to get you to pay for something they are responsible for.


guid: tag:blogger.com,1999:blog-179325926611299376.post-5614757858870541163
url: http://chris-osm.blogspot.com/2015/06/lidar.html
title: Lidar
published: 2015-06-25 16:52:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.311233
updated_at: 2019-10-07 01:30:08.423310
summary:
  Today I received two DVDs with Lidar data on them. They cover Hull and some of the surrounding area data showing centimetre height records at 50cm spacing. It shows DTM, so no buildings or trees, though DSM is available too.<br /><br />I've just finished making a replacement window for the garage - the old one was completely rotten - so I fitted it this afternoon while the weather was fine. Now that's done I can turn to this detailed Lidar data.


guid: tag:blogger.com,1999:blog-179325926611299376.post-2388406161652696025
url: http://chris-osm.blogspot.com/2015/06/when-does-postcode-start.html
title: When does a postcode start?
published: 2015-06-06 19:48:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.313424
updated_at: 2019-10-07 01:30:08.425410
summary:
  I have mapped a fairly large area, Hull and most of East Yorkshire. When there was nothing on the map I just added what I found but now there is a fairly complete road network. Keeping that up to date can be hard work - how do you know what has been added since the last time I looked? OS Locator is really valuable for this as new roads appear on there as OS maps them. It would be better to know developments are starting and get in early and I think there may be a way: postcodes.<br /><br />New postcodes are allocated all the time. Around a thousand are created every month in GB. The Office of National Statistics publish a postcode list based on Codepoint Open data but with some extra stuff in there. One thing is the month the postcode was set up. Looking at the recent ones may let us find places that need surveying.<br /><br />I've created a map to visualise these: <a href="http://pcage.raggedred.net/">http://pcage.raggedred.net</a> <br /><br />I hope this proves useful.


guid: tag:blogger.com,1999:blog-179325926611299376.post-4393561707405936524
url: http://chris-osm.blogspot.com/2015/06/new-postcode-layer.html
title: New postcode layer
published: 2015-06-03 19:39:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.315710
updated_at: 2019-10-07 01:30:08.427543
summary:
  I've just finished extracting the data from a full download of OS Open Names and setting up a layer to see the new postcodes. The postcode data is only a centroid for each postcode area. <br /><br />I maintain a layer of postcodes from the Office of National Statistics, one from Codepoint Open and now Open Names. The Codepoint Open and ONS postcodes are in the same locations, though Codepoint Open data is a bit more up-to-date right now. The OS Open Names postcode locations are now sub-metre, so I expected that the centroid might be in a slightly different location, but some are substantially different as you can see here.<br /><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-Qm2kVEPyiCI/VW9Uw803IPI/AAAAAAAAB30/HB90N1eRZUU/s1600/postcodes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-Qm2kVEPyiCI/VW9Uw803IPI/AAAAAAAAB30/HB90N1eRZUU/s1600/postcodes.png" /></a></div>&nbsp;The Codepoint Open are in red, the OS Open Names are in magenta.<br /><br />I don't store any of these tiles, I render them on the fly. Real tiles are often stored in a hierarchy of folders in the form of<br /><blockquote class="tr_bq"><span style="font-family: &quot;Courier New&quot;,Courier,monospace;">tilename-base/zoom/xposition/yposition.png</span></blockquote>Since I don't store any of these, when someone requests a tile they would normally get a "Not found (404)" error. I trap these errors and use the folder hierarchy to extract the postcode centroids from a database for the zoom, x and y area of the tile.&nbsp; This is used to render a tile with a transparent background and centroid markers on it. I do this because I didn't want to use the considerable disk space the real tiles would take up, especially as I create to to zoom level 21 to make editing easier with the layer switched on.<br /><br />I also extracted the road name and place name data, reprojected all of the location data from OS grid references to lon and lat and stored them for later use. If anyone wants to see this I'll happily pass it on, just ask.<br /><br />If you want to see the new postcodes you can see the new layer on <a href="http://oscompare.raggedred.net/?zoom=13&amp;lat=53.74575&amp;lon=-0.48529&amp;layers=0B0FFTFT">oscompare</a>. Make sure you open the layer selection (blue+white +) and select what you want to see.  You need to zoom in to see the postcodes.


guid: tag:blogger.com,1999:blog-179325926611299376.post-3999719127448457403
url: http://chris-osm.blogspot.com/2015/06/twenty-by-twenty.html
title: Twenty by twenty
published: 2015-06-02 16:49:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.318128
updated_at: 2019-10-07 01:30:08.429767
summary:
  It seems that there are not missing sections in the OS Open Names downloads from OS OpenData. The sections are 20x20 km squares.


guid: tag:blogger.com,1999:blog-179325926611299376.post-2990157551890454167
url: http://chris-osm.blogspot.com/2015/06/os-open-names.html
title: OS Open Names
published: 2015-06-01 21:09:00
fetched: 2019-10-07 01:30:08.317240
created_at: 2019-06-09 18:11:42.320429
updated_at: 2019-10-07 01:30:08.431929
summary:
  As I noted in the previous post Ordnance Survey have said that OS Locator open data is being withdrawn and being replaced by OS Open Names. The file is a strange mixture of three types of data jumbled up. So I needed to separate them into something useful. The three types are postcode centroids, place names and road names. All of the locations are in the projection that Ordnance Survey use, OSGB36 or EPSG:27700. The great thing about the values used in this projection is that they are measured in metres from a fixed point. The previous OS open data, such as OS Locator or CodePoint Open (postcode centroids) use this projection too of course which fixes a location to a square metre. OS Open Names however has a decimal component so the location is now specified to the nearest millimetre - more accurate than OSM works to. The east-west specification in OS parlance is know as eastings and the north-south specification is known as northings. Eastings and northings can be converted to longitude and latitude for use in OSM using, for example, GDAL libraries.<br /><br />The postcode centroids are very simple records. It needs the postcode and the easting and northing for the location of the centroid. Comparing a few OS Open Names centroids with Codepoint Open records the centroids are in slightly different locations. <br /><br />OS Locator lists named roads with its name, a centroid and a bounding box for the road. There is also a hierarchy of place, borough, county or unitary authority.&nbsp; All of this is in OS Open Names and the increased millimetre accuracy is there too.<br /><br />I've not used the gazetteer of place names, but the name and location data are available as you would expect.<br /><br />All of the data types have some URIs in the files too for many of the fields. Many that I have tried to open are dead links, but some show the hierarchy of data. I'm not sure why this is useful. <br /><br />I think I'm going to write a routine to unzip and process all of the OS Open Names data. I'll load the data into a database, reproject it for OSM and make the processed data available if anyone wants it.<br /><br />First, however, OS need to supply the missing sections of their open data.


