General Technology « Quædam cuiusdam
Make BackupPC tweet its activities
Friday 14 January 2011 @ 10:48 pm

We’ve been using BackupPC as our home backup system for the last year or so, and it’s done a great job. It runs under Ubuntu on the home server (an Eee box) and backs up the various family laptops to a RAID array composed of two 1tb USB drives. For offsite backup, we now have a couple more USB drives that receive weekly archives from BackupPC and are swapped to and from my office. For the first time, I feel like we’re doing a decent job taking care of the family digital jewels.

Recently I’ve been looking at the content we create and store on external services. We’re all on Facebook, my daughter and I have five active or at least non-defunct blogs between us, I tweet. Why not archive all that as well? The pleasure I’ve been getting from reading my grandparents’ papers will, for our posterity, depend on this stuff. Not to be morbid, but read last week’s New York Times article “Cyberspace When You’re Dead“. Personal digital archiving is important, and we need to learn to do it properly. The Internet Archive is sponsoring its second conference on the subject in a few weeks.

I’ve started with Twitter. This has forced me, finally, to write some Python, in order to use the OAuth libraries necessary to authenticate to the Twitter API since they turned off basic authentication last summer. Jeff Miller’s tutorial, which uses the Tweepy module, got me going. Once you’ve got the authentication working, the demo script allows you to publish tweets from the command line, which is cool, but… Trying to think of a use for it, my mind went back to BackupPC and the need to monitor its actions and notice when backups aren’t working. It sends email notifications, but why not get them in the Twitter flow as well? It’s the perfect place for short notices.

So, our server now has its own Twitter account. BackupPC provides hooks for scripts to run at different stages of its various processes. To send a tweet after a backup job, add this to the configuration file for a host:

$Conf{DumpPostUserCmd} = "/home/backuppc/bin/post.py backuppc: $client $type: $xferOK"

This will send a tweet in the form “backuppc: cheetah full: 1″ (where cheetah is the host, the backup was full, and it completed successfully – 0 indicates a problem).

The script post.py is based on Jeff Miller’s example. I modified it to compose the tweet by concatenating all the command line arguments (to avoid the use of quotation marks, which I couldn’t get BackupPC to handle properly no matter how I escaped them).

#!/usr/bin/env python

import sys
import tweepy

CONSUMER_KEY = '*********************'
CONSUMER_SECRET = '*********************'
ACCESS_KEY = '*********************'
ACCESS_SECRET = '*********************'

auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth)

# drop first arg, which has name of script
args = sys.argv
args.pop(0)
tweet = ""
for arg in args:
        if tweet != "":
                tweet += " "
        tweet += arg

api.update_status(tweet)

All for the thrill of seeing tweets like this:




Home Zotero repository
Sunday 19 December 2010 @ 11:07 am

The bibliographic manager Zotero lets you sync your collection to a remote server, which is handy when you work on more than one machine. They provide space for free which is plenty for my citations; but to sync the attachments (pdfs, web page snapshots, etc.), the 100mb of free file storage isn’t enough: I’ve got 95mb of stuff already. You can, however, sync the attachments to any WebDAV-accessible storage. I have a home Ubuntu server with RAID storage for the family digital treasures (we run a backup service for all the machines in the house using the excellent BackupPC package), so it made sense to use that. Here’s a quick list of the steps involved.

  • Set up a Linux server. I’ve been using Ubuntu Netbook Edition on an Eee box for a couple of years with no complaints. Ours is pink; we call it “panther”. (You could probably do all this on a Mac or Windows machine that’s on all the time, if you prefer).
  • Buy a couple of large USB drives and set them up as RAID 1 (or buy more and get more redundancy). Software RAID is easy to set up in Ubuntu. Mount the RAID array at e.g. /raidarray and create the directory which the WebDAV service will use as its home, e.g. /raidarray/webdav.
  • Now that you’ve got some storage and a server, you need WebDAV. Install Apache, and add WebDAV capabilities to it. Make sure the Apache user has read/write permissions on the webdav directory.
  • One of the points of using your own server is that you can encrypt your network traffic using a self-signed certificate, avoiding the significant cost of a real one. Set up a self-signed certificate in Apache.
  • Now you need to make your server available on the Interwebs. If you use a cable or DSL service, you probably don’t have a fixed IP address, which makes you hard to find, since your provider may change your IP address frequently and unpredictably. The solution is dynamic DNS, which allows your to notify the DNS when its IP address changes. Set up a free account at DynDNS.com. You will also have to install a little package on your server that will check your current IP address periodically and report changes to DynDNS.
  • Set up your router to direct incoming traffic on your chosen port to your server. The settings are probably under the games and applications tab of your router config interface. Choose a high port number (make the script kiddies work).
  • Finally, put it all together: set up the virtual host in Apache that will serve your WebDAV repository. My virtual host configuration looks something like this, with service at https://mywebdav.dyndns.org:12345/webdav/ (not my real host), the RAID array mounted at /raidarray, and the certificate authority and other ancillary stuff in /var/www_dyndns:


<virtualhost *:12345>
ServerAdmin me@myemail.com
ServerName mywebdav.dyndns.org

# Indexes + Directory Root.
DirectoryIndex index.php index.html
DocumentRoot /raidarray/webdav/

<directory /raidarray/webdav/>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
</directory>

# SSL
SSLEngine on
SSLOptions +StrictRequire
SSLCertificateFile /var/www_dyndns/myCA/server_crt.pem
SSLCertificateKeyFile /var/www_dyndns/myCA/server_key.pem

# Logfiles
ErrorLog /var/log/apache2/mywebdav.dyndns.org/error.log
CustomLog /var/log/apache2/mywebdav.dyndns.org/access.log combined

# webdav service
<Directory /raidarray/webdav/>
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Order allow,deny
allow from all
</Directory>

Alias /webdav/ /raidarray/webdav/

<Location /webdav>
DAV On
AuthType Basic
AuthName "webdav"
AuthUserFile /var/www_dyndns/passwd.dav
Require valid-user
DavMinTimeout 600
</Location>
</virtualhost>

Restart Apache and you should be good to go. With your browser you can visit https://mywebdav.dyndns.org:12345/webdav and see the contents of your directory, after you accept the certificate and log in. You should also be able to mount that directory as a network drive from any client.

To use it for Zotero, create a directory named “zotero” in the “webdav” directory, and then go to the “Sync” tab in the Zotero preferences, turn on file syncing using WebDAV, and give it your address. Do the same in other machines you want to sync. As Zotero syncs, you’ll see the /raidarray/webdav/zotero directory fill up with .prop and .zip files.

Caveats and lessons learned:

  • I found that the process that determines the current IP address timed out a lot due to load on DynDNS’s servers. I got around this by installing a script on my own website (where this blog is hosted at Dreamhost) for it to visit instead. Instructions were easy to find.
  • Visiting your server using DynDNS from within your home network can be problematic. I couldn’t get it to work with my first router: requests would just hang. When that router died and I replaced it with a Cisco Valet, the problem went away.
  • Your server will be unreachable during the interval between a change of IP address by your provider and the next update to DynDNS. This hasn’t been a problem for me yet.
  • The self-signed certificate encrypts your traffic, but it creates a theoretical vulnerability to spoofing. Buy a real one if you’re worried. I wouldn’t use this to store sensitive personal information, though.
  • If you have problems, check the Apache logs. The first thing to check is permissions: can Apache read and write to the Zotero directory?



There is a giant floating Easter egg in the Quad
Monday 26 July 2010 @ 5:54 pm

Which makes sense, because I put it there. The giant floating gold-foil-wrapped Easter bunnies, on the other hand, make no sense at all.

Augmented reality is very weird.

I’ve been playing with Layar‘s Hoppala Augmentation service (with a hat tip to Fiacre O’Duinn at Library Bazaar). Via the web interface I placed an object in an augmented reality layer: an Easter egg in the demo “Hoppala goes Easter” layer. Then I installed the iPhone app, found the right layer (search “Easter”), and strolled through the Quad on my way to the LRT. It looked like this:

It was distinctly bizarre to look through the little window into an alternate world where this huge red egg floated along the path beside me. Freaksville. And people give you odd looks when you walk around the Quad holding your phone out at eye level. When you explain, though, that you’re watching the giant floating Easter egg, and that the floating chocolate bunnies aren’t your fault, people leave you alone.

This was my first experience using an augmented reality app with live GPS positioning. I must say, it’s disappointing in an entirely predictable way. GPS on a phone is rough, and the accuracy of your position changes from moment to moment. That means the egg moves around a lot as you walk, and its position in the middle of the Quad is only approximate, because the app’s sense of your position is only approximate. When building an augmented reality application, you’ll have to take into account that your users may not see your stuff exactly where you want them to, and your stuff might shift unexpectedly instead of smoothly panning and zooming to match the viewer’s movements. I’ll have to give up my fantasy of neatly overlaying an historical image over the corresponding contemporary view, like this set from the National Library of Ireland, where the juxtaposition of past and present is so immediate it makes your heart ache. Or maybe there are ways for the app to work around the limitations of a phone’s GPS.

Anyway, if you’re around U of A and you have an iPhone or Android phone, go to the Quad and check out the egg. And the bunnies. Maybe they do something if you click them. Do you think I should click one? Maybe I should go back and click one.




Older Posts »