Dec
23
2010

Order Tara's Bicycle Touring Cookbook Today!

Our Process: Automation & the GSDB

by Tyler

This entry is part of an ongoing series about how we've documented our adventure.


When we left on this trip, neither of us had an inkling that we'd be documenting our adventure in such exhaustive detail. The original idea was simple: a flickr account, and a nice template at Blogger for our journal—just a little writing and a few pictures sprinkled here and there. I didn't know how to use commas, and neither of us had a clue what an aperture was.

Fast forward a month and things were slowly getting out of hand. Sitting at a coffee shop in England, it dawned on me that the GPS tracks we'd been saving for who-knows-what could be used to show our progress on a Google map! Then, somewhere in Tunisia, we signed up for our dreaded Twitter account, and shortly after, we started taking field recordings too.

By this point, we were generating quite a lot of "stuff" every day: journals, photos, audio, tracklogs, expenses, tweets and more. As the data piled up, and our Excel files got messier, it became pretty obvious that we either needed to give it all up, or start building systems.

I don't want data entry to be a part of cycling around the world!

As luck would have it, the software platform I've been developing for the last few years lends itself very well to this sort of thing. Warning: technical content ahead.


I develop a generalized database management tool which can be highly customized to administer data for things like insurance policies and claims, union employee scheduling and payroll, trucking company compliance, event registration and billing, corporate asset management, customer service systems, employee training, project management, and much, much more.

Basically, I write systems to manage all kinds of super-boring data. It is actually a lot of fun, mainly due to the fact that I like organizing things, and also because my clients rule. In order to support them (and now, us), my laptop is running a full web development environment. This is a big part of what makes working on the road possible for me, as I can program without being connected to the internet.

I develop using Python / PHP (and Node.JS for the next generation of my platform), store our data in PostgreSQL, and host our sites with Apache. I manage my source code and deployment with SVN (I can't wait to migrate to Git when the free time presents itself).

We work on the GSDB (journaling) simultaneously by connecting our laptops with an ad-hoc wireless connection, a network cable with a crossover adapter, or just by being on the same wireless network. Tara edits her hosts file to point the "fake" domain gsdb.local to whatever my laptop's IP address currently is.


In October of 2009, after many hours of coding, and way too much time spent trying to make heads or tails of the mess we'd created, our days, tracks, expenses, pictures and recordings had been entered into my software. (Long after this, we eventually migrated our journal as well; more on that in another our process entry.)

When the dust settled and the kinks were ironed out, we launched what would become the engine room of our website, and what is now the hub of all of our projects together, the behind the scenes section of our website, which we've dubbed the GSDB (Going Slowly Database).

Here is one of the many things we use it for:

Converting Tracks to Travel Data

When we make camp or arrive at a hotel, the first thing I do is fire up my laptop and connect our GPS to it. Then, using Garmin's MapSource, I download the tracklog. This is, in very simple terms, the result of our GPS saving information about its current location to a file every second or so. The points in a tracklog form a long connect-the-dots which shows everywhere we've been.

GPS Tracks

With our tracklog at the ready, one of us creates a new day in the GSDB, and attaches the file to it. When we hit save, my system automatically uses the track data to enter our current location / altitude, distance traveled, total ascent / descent etc.

From turning the laptop on to completion, this entire process takes less than five minutes.

GSDB: Days

The track data we store in our database powers our google map, creating the markers that show each place we've slept, the tracks and elevation profiles between them, and the Show Map & Statistics link for our journal (which displays where we were on the date of the entry).

Next up: some example code for our google map, more about our custom journaling platform, and lots of details about how I'm working to automate everything in our documentation process that isn't writing or editing photos.


Previous Entry
-
Next Entry
-
G
Topics:

8 comments

As someone who spent the last three years as part of the MySQL documentation team I found this article absolutely fascinating! :)

I see you are using Postgres - I thought you were originally using MySQL - maybe a wrong assumption on my part.

I've used SVN for the last 4 or 5 years, and am also moving on from it. I'll probably be making the jump to Mercurial though (need to research more). I've used Bzr a bit and didn't really like it.

How do you find Python compared to PHP? I've used PHP for a while (and wrote some of the documentation for ext/mysqli, ext/mysqlnd). Which parts of the system do you use Python for? I will probably stay with what I know, but was curious on your use of Python.

Looking forward to further details.
Posted by Tony on March 2nd, 2011 at 9:59 AM
OK, so I'm only mildly geeky and I enjoyed your article. It brought back memories - and wishes that we had two computers while we were one the road. We used our GPS tracks to tag all our pictures - which has been very handy. Scott just did a post-process - ensuring that the time stamps on the camera matches the GPS, once both files were uploaded to the computer the GPS points could be automatically added to the photos (don't ask me the details, Scott did it). I love the database idea for storying all the source files - could definitely use that for my research projects!
Posted by Becky on March 2nd, 2011 at 5:02 PM
I stumbled upon your journal about a year ago while searching the net for inspirations for my 9th Swiss bike tour, and I'm following it ever since. I almost wrote a comment on several occasions, but somehow I was always too lazy to do that, shame on me. I could have written about how all my Romanian colleagues immediately agreed to your "you know you're in Romania when you are relieved to ride on a dirt road riddled with potholes" statement, or the fun I had reading about your troubles with Ingrid and Yves' german-speaking oven, because I have the same model and it does know English (and some twenty-odd other languages, too!), or how disappointed I was when there was noone at the top of Julierpass after I climbed it in an afternoon after a very late start from Tiefencastel, or how I followed your path to Madonna del Ghisallo on my tour's last day, except that I missed a turn and ended up taking a longer way up there with an _additional_ 200m climb, or that I spent so much time reading your journals that when I type "go" into Firefox' address bar, then it offers goingslowly.com before google.com... But all this was not enough, you had to mention that you're planning to migrate to Git to make me actually write a comment (;

Go ahead, it has fundamentally changed how I approach a new programming task, how I think about the evolution of source code, and my view about the purpose of revision control in general. How could I have ever lived without it?!

And I'm also glad that you will run my code, even when it's only a couple of lines (;
Posted by sz on March 3rd, 2011 at 12:22 AM
Tony--

Even though I don't use MySQL, thanks a million for contributing to that project; good documentation is so incredibly important for development! I made the switch to PostgreSQL many years ago, back in the 4x days when it was missing all kinds of important features like foreign keys, views, subselects etc.

On Python/PHP: I started web development using PHP when I was a really inexperienced programmer. As my knowledge and skills have improved over the years, so too has my disdain for the language. Some things that realy stick in my craw are the joke of an OO model and the hacked in namespacing implementation. There are lots of other things too; UTF8 handling is a nightmare, the memory footprint is huge, the standard library naming conventions are ridiculous, I could go on and on.

All of that said, there is no denying it is easy to get a lot of stuff done quickly using PHP. If that is your only definition of a good language, PHP is great. I've outgrown PHP in the sense that I've become a good enough programmer to see all the rough edges and be driven nuts by them.

Though I am migrating my business to Python, but I don't have a hybrid model in place. The next version of my platform will be Python (using Pyramid).
Posted by Tyler on March 6th, 2011 at 12:42 PM
sz--

Thank you so much for finally saying hello! I'm so pleased you've enjoyed reading along with us for these last years.

Tara and I now joke about how Romanian roads were like a US interstate compared to Mongolia :) Phew, climbing Madonna del Ghisallo on the last day of a tour? That is dedication. I can't believe you found another 200m of climbing on that hill!

Your vote of confidence on git is music to my ears, as is the siren song of rebasing. I can't wait.
Posted by Tyler on March 6th, 2011 at 12:52 PM
Becky--

What a great idea! What did you do if your tracks didn't have a perfectly matching timestamp? Fudge it by choosing the closest? Leave the photo without a geocode?
Posted by Tyler on March 7th, 2011 at 2:06 AM
> What a great idea! What did you do if your tracks didn't have a perfectly matching timestamp? Fudge it by choosing the closest? Leave the photo without a geocode?

If the track didn't have a perfectly matching timestamp, but there were points before and after, do a linear interpolation using the closest two points. That got about 95% of our photos, since we generally had our GPS turned on whenever we were moving.
In cases where we were stopped for the night (or first thing in the morning), we used the last/first point of the day. On the odd occasion where our GPS tracks didn't cover our location (i.e. Syria, since GPS was illegal there at the time), I used Google Maps to place the photos.
We used Geosetter http://www.geosetter.de/en/ for most geotagging, and the free version of GPS Trackmaker http://www.gpstm.com/dwlpage.php to simplify the tracks (remove extraneous points).
One of these days I really should write up our workflow like you've done... very nice!
Posted by Scott on March 7th, 2011 at 3:22 AM
here is how to set up Chromecast! ... 5 useful Chromecast Apps Download
you should check out.
Posted by Chromecast Apps Download on February 18th, 2017 at 5:10 AM
...and sign up for our newsletter!
Post a Comment
receive email for new comments
check this box to prove you are human

HTML allowed:<a><strong><b><i><u><em><strike>