Dropbox + Google Drive sync

I’ve just got a Google Drive invite, but I’m kind of attached to using Dropbox. But … I have subscription to Google Storage (a whooping 200GB as I store a lot of photo’s in Picasa). So not wanting to waste the new extra space on Google Drive and have a neat extra backup option I thought maybe I could get both services to sync against the same folder on my laptop… No it’s not as easy as going into the preferences of each app and choosing a folder to sync (although I’m sure you were able to do this in Dropbox previously!). Each app forces you to create a special named folder: ‘Dropbox’ for Dropbox and ‘Google Drive’ for Google Drive unsurprisingly. Now my first thought was to create a new folder called Cloud and then symlink each named folders to the new Cloud folder. Again no Google Drive must add some special attributes to the folder it creates which it checks for when it starts up. So eventually I did this:

  • Shutdown Google Drive and Dropbox
  • Open the Terminal
I’ve assumed that my Dropbox and Google Drive folders are saved in my user home directory (I think that’s the default for them anyway!)
cd ~ mv Google Drive Cloud
ln -s Cloud   Google Drive
cp -r Dropbox/* Cloud
mv Dropbox Dropbox.backup
ln -s Cloud Dropbox

And then finally restarted Dropbox and Google Drive and then waited for everything to happily sync! You’ll have a backup Drobox.backup folder which you can delete once you’re happy everything is syncing. Also you’ll should make sure that the contents of your Dropbox aren’t bigger than the capacity of your Google Drive (5GB by default) and vice versa.


NOTE: As mentioned in a comment below (and I probably should have mentioned above), you should only do this on one machine otherwise you might get into an infinite sync cycle! I generally just run this on my little old mac mini as home, which runs 24×7 so all my other installations of dropbox and google stay in sync where-ever i am!

Installing PHP 5.3, Apache2 and MySQL5 using MacPorts

I stopped using macports ages ago, if you aren’t using vagrant or something similar then you probably want to check out homebrew as an alternate mac package manager.

I’ve read a few posts on how to installed PHP on my mac using macports, but quite a few seem out of date or incomplete. I hope this is a guide on how I think it should be done from start to finish using easier to understand small steps.


28/08/09: Now works with Snow Leopard and this post has been updated reflect the latest variant changes to the PHP 5.3 install

03/09/09: Typo fixes and corrections from feedback

22/07/11: Updates for installing on OS-X 10.7 Lion

What? But my mac comes with PHP & Apache!

Well the first question should really be why use MacPorts instead of using the Apache/PHP that comes with OS-X. Basically the answer to that is that by default it’s not really setup for serious developer use, just the average end user who wants to play around with a script of hack together a small website. You could also use MAMP, but personally I don’t think it give me the level of control that I need as I try and develop with an environment as close to my production system that I can (i.e. a base unix environment). MAMP also hasn’t been updated for a while! Once you’re fully setup with MacPorts updating is a breeze and as the whole system has been compiled onto your machine in theory it should be a lot faster than a prebuilt universal binary.

Pre-Configure: Turn off the default web server

So the first thing you should do is to turn off Apache…

This is pretty simple, go to System Preferences, type Apache in it’s spotlight. It will highlight Sharing, but you just need to press return. Just make sure Web Sharing is disabled.

Setting up MacPorts

Install Xcode

Next you must make sure you’ve installed Apple’s Xcode Developer Tools (version 4.1 for Lion, version 3.1 or later for Leopard, or 2.4.1 or later for Tiger), found on the Mac App Store or at the Apple Developer Connection site or on your Mac OS X installation CDs/DVD. This basically installs a more complete unix developer base with compilers are other useful stuff needed for MacPorts.

Install MacPorts

After installing Xcode you can download and install MacPorts. The best way to install it is to use the DMG installer on the MacPorts site. On the other hand if you want to compile it from source then follow these steps:

Open up the Terminal

svn co http://svn.macports.org/repository/macports/trunk/base/
cd base
sudo make install

I recommend whichever way you install macports you should run this command to make sure everything is up to date:

sudo /opt/local/bin/port -v selfupdate

If nothing as gone wrong so far then you should have MacPorts fully setup. Only one more thing to do before installing PHP/MySQL and Apache

Update your system paths

Just so I don’t have to keep on typing /opt/local/bin all the time i’m going add that directory to my PATH. This information can be stored in your .profile which runs every time you start a new terminal. If you don’t have a .profile file use these steps to create one:

cd ~
echo 'export PATH=/opt/local/bin:/opt/local/sbin:$PATH' >> .profile

and that’s that. If you open a new terminal window (yes do that now) then those directories will be searched for terminal commands before the ones in the default path.

Setting up the LAMP stack

Enable SSL – if you want to!

Before installing PHP, i’m going to make sure that the version of curl installed is compiled with SSL enabled. If you don’t need or care about SSL then you can safely skip this step.

sudo port install curl +ssl

You should note that regardless if you run the command above first or one of the ones below then MacPorts will download a whole bunch or ports that are required to install that port or one of its dependencies. Also these commands may take sometime, so give it a while or go make a cup a tea.

Install MySQL

This one is fairly simple, well kind of…

sudo port install mysql5-server

Note: a lot of blogs recommend installing just the mysql5 port, but if you want the option to have MySQL start automatically at startup you’ll want to install the server. The mysql5-server port installs the mysql5 port anyway, so you can’t lose!

If you want to run MySQL at startup then run this command after the port finishes installing

sudo port load mysql5-server

Finally to make sure that the database if fully setup you must run this command (before starting up any mysql process!)

sudo -u _mysql mysql_install_db5

And make sure you pay attention to its instructions! Personally I didn’t do anything here as this is a development setup for me, so I prefer to keep my settings loose. Although if you want MySQL commands in your terminals path (like port) then run these commands and start a new terminal:

cd ~
echo 'export PATH=/opt/local/lib/mysql5/bin:$PATH' >> .profile

Create /opt/local/etc/mysql5/my.cnf, add the following to it and save

[mysqld_safe] socket = /tmp/mysql.sock

Also to remain compatible with other programs that may have been coded to look for the socket file in its original location then add this symbolic link:

sudo ln -s /tmp/mysql.sock /opt/local/var/run/mysql5/mysqld.sock

Install PHP & Apache

Ok we’re finally installing PHP with MySQL support and Apache (FastCGI is just being installed separately for use with some debugging and profiling tools).

sudo port install php5 +apache2 +pear +fastcgi php5-mysql +mysqlnd

Like MySQL if you want Apache to start automatically when you start your computer then run this command after the port installs (as shown while the ports are installing)

sudo port load apache2

Configure PHP

To customize php, copy /opt/local/etc/php5/php.ini-development (if this is a development server) or /opt/local/etc/php5/php.ini-production (if this is a production server) to /opt/local/etc/php5/php.ini and then make changes (see below).

i.e. for development:

cd /opt/local/etc/php5/
sudo cp php.ini-development php.ini

i.e. for production:

cd /opt/local/etc/php5/
sudo cp php.ini-production php.ini

To customize php edit (using your favourite editor) /opt/local/etc/php5/php.ini to configure it properly, make sure you set the correct timezone (for myself it’s this):

date.timezone = Europe/London

Configure Apache

If this is your first install, you need to activate PHP in your web server.To enable PHP in Apache, run

cd /opt/local/apache2/modules
sudo /opt/local/apache2/bin/apxs -a -e -n "php5" libphp5.so

Open /opt/local/apache2/conf/httpd.conf in your favourite text editor (it’s vi for me, but feel free to use pico, emacs or whatever!).

sudo vi /opt/local/apache2/conf/httpd.conf

If you want to change the default MacPorts Apache document root to match default Apple’s Apache (personally I didn’t, but I found this information on another blog – so it might be useful to some), change:

DocumentRoot "/opt/local/apache2/htdocs"


DocumentRoot "/Library/WebServer/Documents"

If you changed the DocumentRoot, change the Directory directive from:

<Directory "/opt/local/apache2/htdocs">


<Directory "/Library/WebServer/Documents">

For PHP to work correctly you must add index.php to the dir_module directive:

<IfModule dir_module>
    DirectoryIndex index.html index.php

Add a new mimetype so that Apache will direct files ending in .php to the PHP module for processing. Add the following within the <IfModule mime_module> block. Without this, all you’ll see is the text of your PHP scripts

AddType application/x-httpd-php .php
AddType application/x-httpd-php-source .phps

And finally, to enable user directories, uncomment:

Include conf/extra/httpd-userdir.conf

But if you are going to use virtual hosts (as described below) then uncomment this line in addition to (or instead of) the line above:

Include conf/extra/httpd-vhosts.conf

Save and close the httpd.conf file.

Advanced Post-Configuration

Configure Apache Virtual Hosts

Now unless you just want to access your new web server using http://localhost you might want to set up some virtual hosts for Apache. I personally only need one virtual host atm i.e. http://testing.example.com. Virtual hosts need to be configured in two places, the hosts file (/etc/hosts) and the Apache virtual hosts file (/opt/local/apache2/conf/extra/httpd-vhosts.conf).

Firstly add this line to /etc/hosts after the localhost has been defined i.e.    localhost    testing.example.com

Then you must edit the virtual hosts file /opt/local/apache2/conf/extra/httpd-vhosts.conf and add this configuration

<VirtualHost *:80>
    ServerAdmin webmaster@testing.example.com
    DocumentRoot "/opt/local/apache2/vhosts/testing.example.com"
    ServerName testing.example.com
    ServerAlias www.testing.example.com
    ErrorLog "logs/testing.example.com-error_log"
    CustomLog "logs/testing.example.com-access_log" common

The document root can be anything you want as long as apache has permissions to read that directory i.e. in this case you can:

sudo mkdir -p /opt/local/apache2/vhosts/testing.example.com
cd /opt/local/apache2/vhosts/
sudo chmod -R 755 testing.example.com

You can do this process for as many virtual hosts as you want. Also I personally keep all my websites stored in my user directory – yes bad on a server, but this is my personal development machine. Finally don’t forget to comment out or remove the example vhosts from http-vhost.conf or you’ll get warnings while starting up Apache, they are harmless to leave in, but it’s probably for the best to remove any potential errors

Installing extra PHP modules

There are plenty of extra PHP modules available from MacPorts, you can get a list of them by using this command. Although you won’t need any of the MySQL ones as you’ve already compiled it into PHP.

port search php5-

Here is the list of PHP5 ports I installed

sudo port install php5-openssl php5-curl php5-gd php5-iconv php5-http php5-mcrypt php5-xdebug

Updating MacPorts

MacPorts is simple to update, every few weeks I just run the command below to check for any outdated ports (i’ll also update this post with any changes to this process I discover)

sudo port outdated

to update all installed ports to the latests and greatest

sudo port upgrade outdated

That’s It

Now if you restart your computer and add code into the document root of your website (in this case http://testing.example.com) then you should have a fully working web server with PHP and MySQL. Ok it wasn’t easy, but for me it’s the best solution over the longer term

Slicehost move

Some content is missing from my previous blog posts at the moment as i’ve moved my hosting from Dreamhost to Slicehost. If you’ve used Slicehost then you may appreciate how time consuming it can be to install and build a web server from scratch, but I’m glad I made the move.

Also while moving my web hosting I shifted my email over to google apps, which is a massive pain if you’ve somehow managed to retain over 10 years worth of personal emails. On the plus side I’ve quickly rewritten my home page to use rails, which was surprising painless 🙂

Finally like a complete idiot I didn’t actually back up everything I thought I had so somethings have vanished into the ether, although I’m hoping I have a older (more complete) backup somewhere  🙁

Give me a little time to sort the broken links.



Just an update from my previous post. I’ve updated my little mashup to be a lot faster than it was and I still have a few optimizations in mind to speed it up even more. Also the site now handles any UK location (well the ones which google maps can geocode into a longitude and latitude) and finally I gave it a humorous name (well I found it funny).

Here’s the link again: UpYourAlley give the site a few seconds to load the markers and the same applies when you click on any marker.

Still there are a few little bugs and niggles that I need to sort out:

  • If more than one venue is at the same location then only one is currently shown.
  • The marker manager code I have isn’t as efficient and smart as I would like especially when scrolling around the map.
  • I would like to be able to filter (on a search) events by category i.e. movies, arts, gigs etc…, but that should be trivial.
  • More user feedback when loading data and handling failure when the users location cannot be geocoded.
  • Plus more, more and then some more…

For those who care about why it’s faster now…. I previously generating the data set completely in PHP on the server-side, but as each api request was made in series to PA the time taken to gather just the venue data took around 15 seconds and when factoring in the events data for each venue this grew to minutes. So when I rewrote the site (waiting for Top Gear at 9pm on sunday night – I thought it started at 8pm!) I set all the requests* to be sent asynchronous from the client using jquery and surprise surprise everything ran faster, although more bandwidth is used and more requests are made in total.

* The requests are still proxied through PHP to PA as I strip out and sort some data as well as hiding the API key that I use.

What’s happening near Balham

I’ve been allowed to play around with a new web service developed by some former colleagues at the Press Association (PA). The service gives you some access to PA’s entertainment listings database (think guardian guide, listings section of the Metro and most other papers etc. it all comes from the same source).

The data is organized into venues (your local arts center), titles (a movie on nationwide release) and events (a title and venue combination at a specific start/end times). Or for the more data minded venues can have zero to many titles, titles can have zero to many venues, events have an unique combination of one title, one venue and a date & time.

Anyway to quickly try out the new service I knocked up a quick mashup of what events are on in my local area each day. Firstly this mashup is massively slow as my host doesn’t have memcached installed for personal use *cough* Dreamhost *cough*. This is also partially due to the structure of the web service as I have to make a lot of requests to generate this mashup, but I’ll keep that feedback confidential for now as the web service isn’t complete nor open to external use yet.

Anyway here’s the link: http://listings.magudia.com (please give it a minute or two to load the data – yes really).

Once I get some time to play with it some more

  • I’ll open up the map to allow users to select their own local area
  • Have some dynamic marker loading on scrolling
  • Allow options to select which days to look for events (apart from today)
  • Filter by event types (only gigs, cinemas, comedy or combinations thereof).
  • Show more venue information
  • Optionally show more information about the event or title
  • Refactor, refactor, refactor…
For those interested the site uses jquery & the jmaps plugin to generate the map and gather data from a simple PHP proxy script which collates the data from PA into a JSON.
I can’t overstate how big a step forward this is for PA and hopefully we’ll see more of this from them in the future.
UPDATE: As always the moment you publish something is exactly the same moment you figure out how to fix the problem your having. I should be able to massively speed up the site, but I’m not going to do that today.


Testing out wordpress from iPhone. Is there a phrase for mobile blogging… Mologging? Moblogging? Or my favourite so far mogging 🙂

It’s also made me *finally* upgrade to the latest version of WordPress!

Virtualizing Balham

Last year I saw a online presentation from the TED conference about a Microsoft Labs project called Photosynth. When I saw the lecture I thought it was probably the most interesting thing I’ve seen come out of Microsoft in ages. This technology really impressed me as it gives anyone the ability to create a pseudo 3D image map of any space as long as you had enough pictures to generate it.

So now over a year after I first saw it Microsoft have released a version where anyone can create and upload their own Photosynth (I’m not too sure I like the name) and early this wet August Sunday morning I went out and give it a go near my local station. Anyone can view it providing that they’re using Windows preferably with IE7 or Firefox3 (no mac support yet, so I had to use vmware to view and create this – mac support is in the works apparently?!?).


After it finally generated and uploaded to the PhotoSynth website I was fairly happy with my results for a first try, but I wish I’d taken more photos from more points and from different distances to really show what I think this technology can do. Then again I did take over 120 photos, but I reckon to do it well I might need 4 or 5 times that! A few things I wish they could add are:

  • A better editor for more control over the end result as currently you have to upload the photos online and hope it all works.
  • The ability to add more photos over time to a current set of photos would allow me to gradually expand the detail and scope of the synth.
  • Geotagging of sythn’s so they could be placed onto a mapping service.

But over all if you like taking pictures then this is a new fun toy to play with…

Just another javascript flickr mashup…


Over the last few weeks i’ve moved over all my pictures from my php gallery to flickr, so now I really need to replace my hosted gallery with something better than a link to my flickr account. My solution to this was to create (what I think is) a nice little mashup of flickr hosted based pictures and a javascript visulisation of them. Rather than use one of the many flickr mashup’s out there I decided it was more fun to wrote my own…Simply i’ve taken 36 square thumbnails arranged in a 9×4 grid from flickr and used a combination of prototype, cube (with some modifications), lightbox2 with some javascript glue of my own to bind them together.

What does it do…

Well it’s easier to see it, but every second or two a randomly selected thumbnail in the 9×4 grid will rotate to a new picture from flickr and then this repeats forever. You can also click on any of the thumbnails in the grid to hightlight a picture and eventually i’ll provide a link to where that photo is hosted on flickr, once a figure out a better way to do it than to put the link into the title.

How’s it work…

Well i’m using a php script on my server to get a selection of photo details using phpFlickr; I decided to get 72 photo objects ever time I accessed to script, but on the initial load I display the first 36 onto the page. I used prototype to handle the Ajax request and json parsing, I guess I could I done this without prototype, but I couldn’t be bothered and it would have taken longer and lightbox2 requires it anyway! Then using a javascript image cube animation library I found this will animate a image rotation to a new picture from the remaining 36 pictures. Although I did have to modify the cube library a bit to use absolute positioning and setting some divs to be inline apart from the first thumbnail of the row. Once the page has runout of pictures from the initial server script load a new call is made to that script for another page of 72 picture objects from flickr … and so on … and so on … I also used lightbox2 to load up a larger version of the square thumbnail when you click on the thumbnail.

You didn’t take those pictures…

At the moment I’m just using the interesting pictures from flickr (they look better than mine), but once I fix the odd bug or few I’ll replace my hosted gallery to a nicer version of this mashup using my own pictures. Phew! PS Thanks firebug!

What next?

So I thinking of writing a new website project in Ruby (probably Rails), but I’m stuck on an actual idea. I know what I want to play with:

  • Languages: Ruby on Rails, JavaScript
  • DB: MySQL (as Dreamhost doesn’t have PostGres) 
  • Test frameworks: rSpec, Selenium
  • CI: CruiseControl.rb

At the moment I’m thinking of porting the lovely RoundCube mail project from PHP to Rails, it’s something that uses concepts that i’m fairly familiar with i.e. email protocols and I guess i’ve never been all that happy with any self-hosted web mail client, so I guess it’s time to put up or shut up and build my own. Obviously I want to refresh/update my Ruby and JavaScript skills and it’s associated agile components as well. Also as i’m using rSpec i’ll try and develop the project using BDD, but how i’m going to map business value priorities to a web email client could pose some interesting questions, although mapping behavior in tests might work better with the way I think … I think 😉 It’s not all that exciting, but if anyone has any other suggestions then just add a comment and i’ll consider it?

Simple Storage Service (You’ve Come a Long Way, Baby)

It’s taking a few weeks of cooking, but my Simple Storage Service is ready to come out of the oven to be eaten (and possibly spat out) by the world at large. Now there are still a couple of things on my TODO list, but nothing massive. Basically URL authentication of requests (needs some thought), postObject (I need to read the docs), virtual hosting of buckets (a lot of thought) and some tiny changes and bugs that i’ll fix over the next few days. So what has changed since my last post:

  • Anonymous requests can now be made where permission to do so has been set.
  • Authenticated/Alluser groups and ACL get and sets have been implemented.
  • All REST calls have been implemented (except postObject)*
  • Exception handling matches the S3 documentation (with some guess work)
  • The REST layer was completely rewritten using test driven development
  • phpDocumentator comments are being added to the code, so docs can be generated
  • I’ve created a web form to help you create new users to the service

So what’s next …. ? I guess I’ll polish what’s been completed so far and add some documentation to make it simpler to deploy. And as i’m off Snow boarding from Saturday i’ll wait to see what sort of feedback I get before getting started on the SOAP section which should be easier now that I’ve got a good testing setup + looking for a new job as i’ll be leaving mine soon! I’ve also found that the most popular php client for S3 (from a google search) is missing some useful functionality, so i’m pondering re-writing it and making several optimizations so it can stream downloads from S3 etc etc…

The best use for this software, apart from academic curiosity and mocking is probably a failover/backup service incase S3 goes down (which it has done). This would work best if you are CNAME record to map to s3.amazonaws.com as I believe that as this is under your DNS control it is fairly trivial to map it to another host.

Other than that I’ll write a blog on how to set it up using xampp on windows and macports on a mac (when I MacBook Pro arrives)…

You can checkout the latest code from here: http://svn.magudia.com/s3server

UPDATE: svn is broke since I moved to slicehost, you can download the code here: http://projects.magudia.com/s3server.zip

* As this service hasn’t been developed to work to meet Amazon’s data consistency model I implemented getBucketLocation, but essentially it does nothing. Although in theory I could use MySQL clustering to implement this I’m not going to unless someone wants to pay me and I also don’t have a global server network to play with 😉