Quædam cuiusdam

Getting Serious with Amazon Glacier

Wed, 29 Aug 2012 19:53:21 -0600

After playing around last weekend, I’m ready to push real content up into Amazon Glacier for long-term preservation. I’ve worked up an Ant script to do the work, and I’ve posted it on GitHub as pbinkley/glacier-ant-bagit.

Here’s what I’m doing:

work at the directory level: each Glacier archive will be a tar file containing the contents of a single directory.
package the content with Bagit so that it will be self-verifying when (if ever) I download it again.
keep metadata locally: at the moment the script maintains a CSV file, and eventually I’ll load that into a database. It also saves the bag manifest to provide a full list of files.

I’m using a new command-line interface to do the uploading: carlossg/glacier-cli. I liked glacierFreezer’s use of SimpleDB to store file metadata in Amazon’s cloud, but this package at the moment only connects to the us-east-1 zone, and I wanted to use something a little closer to home. The code of glacierFreezer isn’t currently open, either.

It’s ready to go, so I’ll let some a large job run overnight and see how it works out.

Update (next day)

Gary and Dave

The script is working on the Ubuntu family fileserver. Overnight it uploaded 70 directories with almost 3gb of stuff (the limiting factor being, obviously, our home internet connection). I fixed a couple of bugs; if you’ve taken a copy from Github, you should do a pull.

So: I’m on the hook for 3¢/month so far, in perpetuity, and 36¢ if I ever need to retrieve those directories (if I go over my 5%/month retrieval allowance). In exchange, the risk of loss of those old images has been reduced … by how much? How could you calculate that? I maintain four local copies already, three at home and one in my office, but a fire could easily bring me down to depending on a single spindle; and I’m getting older and will become increasingly likely to make mistakes with this stuff. Eventually these objects will become a digital inheritance. The more serious dangers they must survive to get through the next fifty years are probably those of human error, indifference, and absence of the right technical skills at the right moment. Multiplying copies onto different platforms, beyond the number required by a continuity-of-service calculation, makes sense: creating impediments to managing this stuff comprehensively reduces the potential consequences of massive failures of management. I hope.

Playing with Amazon Glacier

Sat, 25 Aug 2012 15:17:35 -0600

This week Amazon released its new digital preservation platform Glacier. It is similar to the S3 storage service, but optimized for long-term, low-access storage. You pay a penny per GB per month, and you accept that access will be slow (four hours or more) and expensive (12 cents/GB, with free access to 5% of your content each month). I’ve been storing family digital assets on S3 as a remote backup, and Glacier will save me a few bucks each month. I’ll only need to access the content if my local backups fail, so I can accept the barriers to access. And, of course, at work we’re interested in low-cost off-site replication. So, let’s check it out.

The initial offering from Amazon has a web management console and Java and .NET SDKs and a REST API, but no user-friendly client. Third parties are starting to release clients, though, and there’s enough there to work with. Within a few days there will be more.

Getting started is easy: just activate Glacier in your AWS account. The data model is simple: “vaults” contain “archives”, which as far as I’m concerned are simply files. You can create vaults through the web console and tie them into Amazon’s SNS notification service, but that’s as far as you can get; to upload a file you need client.

I started with the glacierFreezer command-line client, which is based on the Java SDK. It makes use of an Amazon SimpleDB domain to store information about your archives, so you need to create one for it first. Then gather your access key and secret key (from the “Security Credentials” tab in the web console), and run it:

java -jar glacierFreezer.jar <accessKey> <secretKey> <simpleDbDomainName> <vaultName> <fileName>

Up the file goes, and the results are stored by glacierFreezer in the SimpleDB domain:

<Item>
   <Name>readme-pbinkley.txt</Name>
   <Attribute>
      <Name>sizeBytes</Name>
      <Value>36</Value>
   </Attribute>
   <Attribute>
      <Name>archiveId</Name>
      <Value>E4lnahK_rbGeenbN07Yc3Myl3FLuJ6IhlmnLSHeAlYmfilRiiJ-3aCCs8C2lPgocUvmBIYpY2lIR1tWmVfXeji73WJrHKqIw9snU8ADWBkPO92Dp688E-mMyLCTMT-s1A7_D2bxxOQ</Value>
   </Attribute>
   <Attribute>
      <Name>fileName</Name>
      <Value>readme-pbinkley.txt</Value>
   </Attribute>
   <Attribute>
      <Name>localDateTime</Name>
      <Value>Thu Aug 23 12:36:42 MDT 2012</Value>
   </Attribute>
</Item>

That’s a lot of information you’re going to need to keep track of, because Glacier won’t keep track of it for you. If you want to be able to restore an archive to a local file with the same name as it had before you uploaded it, you need to remember the mapping of the archiveID to the fileName.

At this point I was blocked again, since glacierFreezer doesn’t yet have the functionality to do anything with an archive in a vault (give it a few days). The day after the upload, when Glacier had done its daily job of generating inventories, I could at least see in the web console that the vault had been populated:

We’ve got an archive! The file I uploaded was only a few bytes, so the 32kb size presumably represents the block size of Glacier’s file system.

This morning I looked again for Glacier clients, and found that the Node.js project node-awssum had added Glacier to the list of supported Amazon APIs. I’ve been meaning to play with Node.js for a while so I jumped on it. I installed Node.js and its package manager npm according to these instructions, then installed node-awssum (and the required package fmt) with a lovely simple

npm install fmt 
npm -d install awssum

The Glacier examples that come with node-awssum cover fetching vault descriptions and such, but not the job-oriented tasks that I need at this point. To fetch an inventory of a vault, or an archive from that vault, you need to initiate a job and wait for Glacier to let you know it’s done (which they say takes four hours). Not to worry, though, node-awssum is easy to work with. I copied one of the examples and created a script “inventory-retrieval.js” like this:

var fmt = require('fmt');
var awssum = require('awssum');
var amazon = awssum.load('amazon/amazon');
var Glacier = awssum.load('amazon/glacier').Glacier;

var accessKeyId = 'xxxxxxxxxxxxxxxx';
var secretAccessKey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
var awsAccountId = 'xxxxxxxxxxxxx'; // note: omit hyphens

var glacier = new Glacier({
'accessKeyId' : accessKeyId,
'secretAccessKey' : secretAccessKey,
'awsAccountId' : awsAccountId, // required
'region' : amazon.US_EAST_1
});

fmt.field('Region', glacier.region() );
fmt.field('EndPoint', glacier.host() );
fmt.field('AccessKeyId', glacier.accessKeyId().substr(0,3) + '...' );
fmt.field('SecretAccessKey', glacier.secretAccessKey().substr(0,3) + '...' );
fmt.field('AwsAccountId', glacier.awsAccountId() );

glacier.InitiateJob({ VaultName : 'test', Type: 'inventory-retrieval' }, function(err, data) {
fmt.msg("describing vault - expecting success");
fmt.dump(err, 'Error');
fmt.dump(data, 'Data');
});

I run that and get the following output:

$ node inventory-retrieval.js
Region : us-east-1
EndPoint : glacier.us-east-1.amazonaws.com
AccessKeyId : AKI...
SecretAccessKey : 3HO...
AwsAccountId : xxxxxxxxxxxxxxx
describing vault - expecting success
Error : null
Data : { StatusCode: 202,
Headers:
{ &#039;x-amzn-requestid&#039;: &#039;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&#039;,
location: &#039;/xxxxxxxxxxxxxxx/vaults/test/jobs/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&#039;,
&#039;x-amz-job-id&#039;: &#039;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&#039;,
&#039;content-type&#039;: &#039;application/json&#039;,
&#039;content-length&#039;: &#039;2&#039;,
date: &#039;Sat, 25 Aug 2012 18:36:32 GMT&#039; },
Body: &#039;&#039; }

So, I’ve successfully created the job. And now I wait, savoring the full experience of Glacier’s slow retrieval which is going to save me so much money compared to S3. I’ll update this post when I get Glacier’s notification that it’s complete. Meanwhile, I’ll contemplate David Rosenthal’s analysis of Glacier’s costs.

Update 1

The job took just over four hours; I didn’t get a notification (have to look into that, probably my misconfiguration) but the job description shows the time. The next step is retrieve the job output, and it turns out that node-awssum hasn’t finished this function: it generates the uri for a job description rather than the job output. The code was easy to patch so I was able to retrieve the inventory:

{
  VaultARN: 'arn:aws:glacier:us-east-1:xxxxxxxxxx:vaults/test',
  InventoryDate: '2012-08-24T13:54:52Z',
  ArchiveList:
  [ { 
    ArchiveId: 'E4lnahK_rbGeenbN07Yc3Myl3FLuJ6IhlmnLSHeAlYmfilRiiJ-3aCCs8C2lPgocUvmBIYpY2lIR1tWmVfXeji73WJrHKqIw9snU8ADWBkPO92Dp688E-mMyLCTMT-s1A7_D2bxxOQ',
    ArchiveDescription: 'Archived file readme-pbinkley.txt Thu Aug 23 12:36:41 MDT 2012',
    CreationDate: '2012-08-23T18:36:42Z',
    Size: 36,
    SHA256TreeHash: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
  } ] 
}

So it does have my original file name, but only in a text description. This response gives me the archive ID of my file (which I had anyway because glacierFreezer saved it for me - nice to see that they agree). Just to close the loop I’ve initiated the archive-retrieval job.

Update 2

I posted an issue about the problem with get-job-output in node-awssum. Heard a couple of hours later that it’s been fixed in master; tried it, it works. God I love open-source.

Update 3 (next day)

The archive retrieval job finished after 4 1/2 hours, and I can retrieve my file. I get a sha256 hash in the header to let me verify the content. For some reason Amazon doesn’t pay attention to my byte range request if I try to retrieve less than the whole file; perhaps it’s because the file is so short, just 36 bytes. I’ll try that again when I’ve uploaded something bigger. And it turns out the notifications were coming through to my email after all: dunno how I overlooked them. So all is good.

It’s easy to imagine a full-scale retrieval process that would manage the initiation of retrieval jobs, monitor the notification stream (which uses Amazon’s Simple Notification Service and can therefore push notifications using a variety of protocols), and fetch the output when it receives notification that a job is ready. Amazon says that output is available for at least 24 hours after the job completes, so you would want to manage the chunking of jobs in such a way as to avoid retrieving more content than you can download in a day, taking into account the somewhat convoluted calculations required to avoid overrunning your 5% monthly free download allowance.

I’m currently using S3 for offsite backup of my personal digital archive, and moving it to Glacier is a no-brainer. I don’t expect ever to retrieve this stuff, since I keep multiple local copies: it’s fire insurance. After a disaster, I’d be willing to pay the download costs to retrieve the family photos. In my professional role (where this is all speculative), I’d take David Rosenthal’s concerns seriously and avoid lock-in: as long as we’ve got local copies, we could move our content to a competitor of Amazon’s without incurring the retrieval costs.

Finally, my first experience with node.js has been great, and I’ll definitely be putting some time into learning more.

Contributing a Book to Internet Archive

Sun, 04 Mar 2012 17:27:23 -0700

Page 8, with photograph by Frances Binkley

Yesterday I ran across an interesting booklet in my grandparents’ papers; and noting that it was published before 1977 without a copyright notice and was therefore in the public domain in the US, and having an interest in user-contributed items for digitization projects at the moment, I thought I would scan it and contribute it to the Internet Archive. After a hasty reading of the FAQ I proceeded to mess up the process, so I thought I’d document it here.

The booklet is a prospectus for an art photography course offered by the New York photographer Rabinovitch (who went by his surname alone). My grandmother was a student of his in 1938, and a photograph of hers is on p.8, one of the ones I posted on my other blog. The booklet is in WorldCat (from the Getty) and Google Books, but hasn’t yet been digitized, as far as I found.

My item was a 32-page booklet with covers, so 36 page images in all. To upload a scanned book, you need each page in a separate image. I scanned two-page openings at 300dpi, 24 bit colour, saved as tiffs. I cropped each one to the two-page spread, then used ImageMagick to split each one into left and right pages:

convert -crop 50%x100% +repage Rabinovitch-00.tif output/Rabinovitch-00-%d.tif

(That produced output/Rabinovitch-00-1.tif and output/Rabinovitch-00-2.tif). Having tested it, I ran a quick awk job to generate a script to split all the images:

ls -1 *.tif | awk -F. '{ print "convert -crop 50%x100% +repage " $1 ".tif output/" $1 "-%d.tif" ;}' > split.sh

I had scanned the cover as a two-page spread, so I had to rename the image of the back cover to put it at the end of the sequence: filename sort-order seems to determine the order of the images in the final product. Finally (and this is the bit I missed the first time), I combined all the tiffs into a pdf:

convert Rabinovitch-*.tif rabinovitch.pdf

Now I had a 730mb pdf. Off to the Internet Archive Contribution page, where I logged in with the account I created a while ago when I uploaded a video. Click the “Upload” button at the upper right to get to the main uploading page, then click “Share” at the upper right, and select your pdf. A flash app uploads it; it took a couple of hours on my home connection. Meanwhile you have a form to fill in. It’s more oriented to a/v uploads than to books, and you don’t actually get to tell it what you’re uploading, but the process will figure it out in the end. You can fill in a title (which gets converted into an Internet Archive ID) and a description and an author, and that’s it for metadata. You can also apply a license. If you choose to dedicate your item to the public domain, you’ll be given the option in the next step to mark it as already in the public domain, which was appropriate for my item.

When the upload completed I submitted the form, and after a minute or so it told me my item was available and gave me a url. And it was available, but it hadn’t finished processing yet: all I could get was the original pdf that I had uploaded. There is a link on the upload page to a list of your unfinished tasks: go there, and you’ll see the processing task still running. Click to see a log of the ongoing processing (here’s mine, after completion). You can see which node in Internet Archive’s storage array received and processed the item, the job priority (-1: hey, I don’t mind), the generating of the xml (including the assignment of an ARK, replication of the files to another node with rsync, and finally the generation of the various derivatives: OCR, images, pdfs, etc., with the results being rsynced to the mirror node. The whole process took a little more than ten minutes.

The outcome is http://www.archive.org/details/RabinovitchProspectus1938. It’s in the Community Texts collection (formerly known as the Open Source Collection), and it has the full Internet Archive treatment. The photographs in the offset-print original have been nicely descreened, and look quite good in the book-viewer presentation. All it lacked was an Open Library page, so I activated my mad cataloguing skillz and created one.

I also have the option to edit this item; this brings up another form with more fields, including date, notes, and rights – things I couldn’t fill in from the upload screen. I can also modify the files if I like.

And here’s the result, all ready to help some future historian of the Depression-era New York photography scene decide whether they need a trip to the Getty:

A Walk with Love and Data

Wed, 26 Oct 2011 22:22:42 -0600

Last week I attended the annual Access conference, this time in Vancouver with the theme “The Library is Open”. It’s always an overstimulating conference, but this year I was made drunk with the confluence of fresh thoughts about things I care about more and more, the intersections of the presentations with things I’ve been thinking about in other contexts, and the engaging personalities. And there was beer. For a politically naive introvert, it was too much too quick. This posting is a preliminary and personal attempt sort it all out.

Tweet: @calvinmah On my way to #access2011 hotel to escort people to #hackfest

It started with Hackfest, where the geeks get together to spend a day coding up some interesting project from the list of submitted ideas, or from their own fertile and over-caffeinated brains. I had put in an idea for a personal digital archive, called “Spalatum” after the Roman fortress/palace that formed the matrix for the medieval city of Split in Croatia. When the Roman architects moved out and stopped maintaining the place, the locals moved in and kept the town going in their own way, subdividing the Roman buildings or putting up small houses against the old walls. Diocletian’s mausoleum became the cathedral. In the same way, I want a digital archive that can continue to be used and preserved by non-techie heirs after the geek who built it is gone. My idea wasn’t taken up by any of the groups (but I mean to get back to it). I worked on a Linked Open Data project, playing with the Silk Workbench application to enhance RDF data by matching fields with an external SPARQL-enabled source like dbpedia. It was fun and we learned a lot about Silk but didn’t get very far with our data.

That evening a bunch of the hackers went out to dinner at a Japanese/Korean place, then headed off to find some good beer. On the way we picked up Bess Sadler, who had (typically) found a medieval reenactment operation nearby and spent the last hour playing with swords.

Bess (photo: BigD)

Bess is a talented developer, and surrenders to laughter more completely than anyone I know: always a welcome addition. We ended up at the Steamworks Brewery and I talked with Bess and the dangerous-looking Nick Ruest, black locks spreading in all directions, gorgeously tattooed up one arm. We talked about working conditions at our various institutions. The passion that keeps us interested in the stuff we do isn’t always nourished in the workplace, unfortunately.

Ended up in the hospitality suite to taste a selection of craft beers, as prescribed by the magisterial man-mountain of San Diego.

@bohyunkim #access2011 keynote is excellent in describing where libraries are standing now - on the border b/w commons & increasing privatization

The conference proper opened the next morning with the keynote by Jon Beasley-Murray of UBC. He started with Borges’ vision of the infinite library and the fertility and sufficiency of the library as a public good. He developed this into a call to the barricades to defend the commons against corporate encroachment, holding that the domination of formerly open online space by closed systems like Apple’s and Facebook’s is precisely parallel to the early modern enclosures like the Highland clearances. (The displaced Highlanders came to Canada: what new lands will the refugees from Facebook colonize? Diaspora may be aptly named indeed.) He put it in Marxist terms: what the corporations are doing is “primitive accumulation”, which he preferred to call “accumulation through dispossession” (to avoid the implication that it only happened in the distant past). Their wealth derives directly from the resources which they have seize from the public domain for their own use. A roomful of librarians was on his side the whole way: information wants to be free, after all. He encouraged us to engage in “massive projects of common productivity” such as Wikipedia. I wondered wimpishly whether the leftist framing of the question would attach it to polarizing issues in right-leaning Alberta (where talk about protecting the commons from corporate exploitation is likely to be seen by the establishment as a coded attack on the oil industry), and asked a question about whether the idea could be framed in a non-partisan way. Beasley-Murray helpfully put it in terms of the university’s role in fostering social critique, regardless of party. “There are no saints here”: everyone is subject to critical investigation.

I was therefore thinking about democracy when the next presentation started. This was Jer Thorp’s dazzling tour through the data visualizations he does for the New York Times and some outside projects such as the 9/11 memorial. This talk was emotionally fraught for me, since it was the David Binkley Lecture, in honour of my brother Dave, who died in 2005. The idea was Gary Gibson’s, and each year the lecture is generously supported by Gibson Library Connections; my only contribution was to suggest that it be used to bring in a speaker from outside the library world. Dave was a regular at Access in the early days, and after I came into the library biz he and I overlapped at a few Accesses. (I treasure the time he came to a symposium in Edmonton on the future of the ILS, and he was being introduced as my brother for a change.)

Tweet: @cIRcle_UBC Cascade view of "But Will It Make You Happy?" - seeing the story (conversation) actually unfolding from different views #Access2011

The visualizations were dazzling and joyful. One was an aggregation of tweets that include the phrase “good morning”, represented as bouncing blocks on a rotating globe. You see the wave of pogoing cubes advance around the planet, following the sun, as people wake up and tweet. Another, more elaborate one was Cascade: a real-time tracker of social-media responses to NYT articles – not just tweets but also url-compressions and decompressions – represented in three dimensions. The most moving visualizations, though, were the ones that brought things down to the personal level. The first was a visualization of your movements based on the GPS data that iPhones were found to have been storing; the other was a timeline of NYT articles against which you could map the phases of your life (childhood, college, marriage, etc.), creating links to stories that were important to you. Thorp described the transition from irritation at another big-brother intrusion into his privacy with the iPhone scandal, to fascination with reliving a year of his life via the GPS traces. He called it “embodied history”. He and others set up a site where you can upload your trace (openpaths.cc) and make it available as a dataset to researchers if you choose. From the default position of “that’s too private to share” he moved to “under some circumstances, I’ll share that, even though it’s private”.

From Thorp’s talk and others over the next two days I learned of lots of exciting new tools that I want to master, or master more fully:

They all have to do with data normalization and visualization. They make me long to become a data wizard, able to make datasets dance the way Thorp does. The point, though, is to make data tell true stories. It took me back to palaeography classes with Father Leonard Boyle (later Prefect of the Vatican Library). The duty of the philologist, Father Boyle said, was to make an ancient document speak as fully and as truthfully as possible about the context in which it was created. There was an ethic involved: the relationship between me and a medieval scribe whose every pen-stroke I could trace but about whom I knew nothing else was real, and I owed the same respect for his or her personhood that I owed anyone I met on the street. The new technologies are just as capable of telling compelling lies as compelling truths, and they are therefore covered by the same scholarly and personal (and librarianly) ethic.

Real Face of White Australia

I felt a similar sense of responsibility in the “Real Face of White Australia” project, which I read about a couple of weeks ago on Tim Sherratt’s blog (note to future Access organizers: get this guy!) It starts from scans of immigration documents for mostly Chinese or Indian workers who came to Australia in the late 19th or early 20th century (the age of the “White Australia” policy) and were subject to restrictions on their travel. The documents include photographs; Sherratt’s inspiration was to use open-source facial recognition software to crop the faces out of the scanned documents and present them as a waterfall, with more faces appearing no matter how far you scroll down, each linked to the source document so you can find out about the individual. It zooms you from the macro level of political criticism of the racist policy down to the micro level of individual stories, and back again through the sheer accumulation of cases.

How seductive the tools are! Thorp showed us visualizations that cost him half an hour, which I would be obnoxiously proud to present after a month’s solid work. Inspired by Thorp and by a Hackfest project, I managed to make a little progress using Google Refine and Google Fusion to come up with a GIS visualization based on the 1926 volume of the Henderson city directory for Edmonton. Building on work Tricia Williams (now Jenkins) did for us some years ago, I had a dataset of names and street addresses roughly parsed from the raw OCR text of the directories. Out of 30,000 entries, 11,000 were parsable from the OCR. They were grouped by city block and projected onto the current Google map of Edmonton. At first a jumble of red dots show the blob of built-up areas; as you zoom in they resolve into individual dots, which in turn represent groups of identifiable individuals living on the same street: people with names and occupations, more or less truthfully represented in the Directory, and equally or less truthfully represented in my dataset, with all its OCR errors and parsing errors. The Murphy family, Arthur and Emily, are there, a year before the launching of the Famous Five appeal that established that women are “persons” under Canadian law.

That evening in my room I wanted to follow up on an email exchange with my younger brother with a link to one of our grandfather’s articles, from the book that was recently digitized by Internet Archive through the generosity of Rick Prelinger. I found that I couldn’t get access: the Selected Papers had been placed in the Lending Library collection and was flagged as unavailable for borrowing. Rick spotted my mournful tweet and worked his contacts at IA to have it fixed within a couple of hours.

Tweet: @pabinkley Drat: Internet Archive has put RCB's Selected Papers in lending library collection. Glad I downloaded text and images. #alwaysbackupthecloud

The piece I wanted was “History for a Democracy”, given as a closing keynote at the Minnesota Historical Society’s conference in 1937. Binkley criticizes the doctrinaire historical projects of the fascists and the communists:

Now it is the weakness of this kind of history – whether it be written for the church, the nation, the communist society, the fascist state, or even the federal democracy itself – that it stands at the mercy of objective criticism. The faithful following of the technique of historical investigation may at any time overturn elements of the story that stand as essentials in the use that is being made of it. Objective investigation may prove that the world was not created in 4004 B.C.; that the most important developments on the European scene were not the special experience of any one nation, but were shared in common by many peoples; and that the continuity alleged to be found in the life of a nation from the remote past to the present day is illusory or incidental. The communist interpretation of social evolution and political events may not be sustainable in the light of an objective criticism of the evidence, and the fascist or nazi interpretations may also go to pieces under criticism. Nor is the historical interpretation which has nourished the spirit of democracy immune. The bold conceptions of Freeman and Stubbs on early German democracy have already been relegated to the junk heap of discarded historical syntheses.

If we undertake deliberately to nourish our own institutions on a history of this kind, made to order for this purpose, we may find ourselves confronted with the tragic dilemma that the mission of our history cannot be served without abandoning the scientific historical method itself. And this would be particularly fatal to democracy, because democracy more than any other kind of government needs to sustain free investigation and criticism of everything. A myth that will not stand criticism must ultimately be protected by force. And an interpretation of history that one is not permitted to doubt and criticize becomes ipso facto an interpretation that one cannot sustain and prove.

In defending the role of history and free inquiry in a democracy, he zooms down to the personal level, both for the content and for the practice of history:

It took us several generations to build up the corpus of published material, to make the critical studies, to collect the bibliographies, to organize the knowledge from which our present historical writing is documented. Our Ph.D.’s move sure-footed through this material. If I want to work on the Clayton-Bulwer treaty, I know where to look for the material, and I can begin where the last scholar left off. But if I want to write the history of my family, or of the school district in which my son is going to school, I find nothing prepared for me. It will take us several generations to adapt and complete the documentary equipment for the writing of family and local history. It took us several generations also to train the army of scholars in the tradition of the craft. It may well take us several generations to train every man to be his own historian.

More and more, I think “democracy” is the word I’m looking for as the foundation of the values I want to buttress with my work.

Tweet: @shlew I love that #access2011 ended with inspiring talks by amazing women: @researchremix, @andreareimer, @eosadler.

The final day of the most powerful. First was Heather Piwowar, who is working to establish the evidence base for the benefits of the open-data model in scholarly communication. She grounded the need for peer review of scientific papers on the same principles as collaboration in open-source software development: 50% of published papers contain errors in their use of data, and 5-10% of those errors affect the research outcomes. Linus’ law applies: only if the data is available for review can those errors be corrected. The current model of on-request sharing doesn’t work: many researchers report that they’ve had requests denied, and the young and the productive are disproportionately represented in that group. On the other hand, the science done on the basis of reused datasets increases the return on the investment in the original research. Her current project is to track 1000 randomly-selected datasets (100 each from 10 repositories) through the published literature. The roles she proposes for libraries including hosting repositories and also advocating and educating among faculty, publishers and the public. It was great to see the solid evidence she’s gathering to back up our intuitions about the value of open access.

During the introduction of the next panel I was diverted by Brian Owen’s trivia question about the “Dukes of URL”, Dave’s band which played at the 1996 Access conference.

Nick (photo: BigD)

There was a correction from the floor: Owen had referred to 1996 as the first Access, but it was only the first in Vancouver, third of that ilk. I’ve wanted for a long time to assemble a history of the conference, and a couple of years ago I started gathering links and asking for materials; Art Rhyno sent me some early programs, which I still have. I let the project languish but now seemed like a good time to restart it. A wiki! No, a Google Doc! I started one and shared it out to a new Google Group “Access Conference History” (join us!), and lent only half an ear to the following session on Evergreen as I and Nick Ruest and others filled in the chronology of the conference since 1994, identified surviving conference websites and excavated others from the Wayback Machine, linked to Flickr sets and blog postings, and sketched in a few memories of specific conferences. Shared editing of a Google Doc is incredibly exciting for the immediate visibility of the collaboration: new text from other collaborators crawls across the screen in the paragraph above the one you’re writing, new coloured cursors appear as people join in. By the end of the session we had a solid framework.

The next session dealt with proprietary and open-source software. Bess Sadler described the Hydra community and what makes it work. The strength of the collaboration depends on building trust by, paradoxically, limiting the areas where trust is needed. If we have software tests, we know without trusting whether a given commit broke a given feature (and Hydra’s rule is “no code without tests”). Uncertainty is reduced, and the team can get on with building without wasting time and emotional energy assigning blame. I live with too much uncertainty (of my own devising, given my lousy documentation habits); I need to make Bess’s approach work in our environment.

Tweet: @eosadler Collection, analysis and policy response to data is gvmt's job, so recognize data as a vital public asset. #Access2011 #VancoverIsAmazing

The closing keynote was Andrea Reimer, a city councilor in Vancouver and sponsor of the city’s open data initiative. She’s in the middle of an election campaign and was a little late because a debate ran overtime; fortunately the conference program was running a little late as well.

Reimer’s talk took us back to the theme of democracy. Her principles: all people are equal; all people have the ability to reason. She allowed a 3% rule: in every hundred people, three are assholes who need to be worked around, but the rest, however much you disagree with them, are at least open to reason. From this it follows that people can make good decisions, and therefore democracy can work; but the public good depends on giving people good information on which to base those decisions. (This took me back to a photo Eric Hellman took at the Rally to Restore Sanity in Washington last year: a smiling woman carrying a sign that read “Librarians for Informed Opinions”. It’s the best brief statement I’ve seen of the role of libraries in a democracy. If you know who she was, please let me know!) Reimer described the process she and a few others initiated to start Vancouver’s open data initiative. It was the flagship for this movement among Canadian cities. Edmonton is now right up there among the leaders, I’m proud to say. I recently attended an organizational meeting for the Edmonton Pipelines project, which is bringing together GIS-based research in the Edmonton area, including the city’s open-data folks. They’re data pipelines: it’s about “maps and narratives for dense urban spaces”. The library is providing scans of historical maps, which we hope to have georeferenced and mounted in the Hypercities platform.

I was stirred to get up and ask another political question (what’s got into me?), or rather the same question: how can we carry this commitment across partisan boundaries in these polarized times – when the preponderance of dangerously uninformed opinions are on the right, across the divide from us? Reimer’s response was wonderful: I’m not religious, but if I had a religion, it would be the Age of Reason. Except for that 3%, we’re all open to rational argument and we just have to keep making the case on the best foundations we can.

We had heard that David Suzuki was to address the Occupy Vancouver camp a couple of blocks away at 1:00, so after the conference closed several of us hurried over to hear him. A general assembly was starting when we arrived, and it was necessary to pass a motion to let Suzuki speak. We were therefore introduced to the remarkable rules of order of the Occupy movement. When we arrived various people were introducing themselves from the stage, using the amplified public address system. They spoke in short spurts, and the crowd repeated their words. This turned out to be the “human microphone”, and it was used whenever anyone needed to be heard, either from the stage or from the crowd. People responded to the proceedings with a set of gestures, whether registering votes or just responding to what was being said. Jazz-hands means agreement, thumbs-down disagreement, hands above the head with fingertips touching a point of order, and arms folded means a block: the blocker is prepared to leave the movement if the motion is passed. This triggers further discussion and even dividing the meeting into break-out groups, until consensus (set at 90%) is reached.

We were enthralled; “This is great!” I said to Geoff Harder, “We should use it in Library Council meetings!” The motion to allow Suzuki to speak was blocked by someone off to the left. Through the human mic we learned the objection was that this was an egalitarian movement, and no one should be allowed priority of speech for mere celebrity. The motion was reworded: Suzuki wasn’t being allowed to speak first at the meeting, the meeting was being suspended to allow him to speak according to his invitation. Another vote, another block. “Yeah,” said Geoff, “this is just like Library Council.” “Are you prepared to leave the movement over this?”, asked the moderator, and after more discussion the block was withdrawn. The motion passed, and Suzuki was allowed to speak to us.

He gave a great twenty-minute speech. I recorded it on my phone (and as my version of the human mic I’ve posted it on Facebook). Suzuki covered all the themes you’d expect on such an occasion: pro-environmental, anti-corporate, anti-globalization, anti-consumerist, pro-democracy, pro-biodiversity, taking the long-term view. And he touched on information as a social good:

For five years the prime minister of Canada has never acknowledged the reality of human-induced climate change, or that Canada is the industrialized nation most vulnerable to its impact; and now he’s cutting back on scientists in Environment Canada and research on climate, so that we don’t have to listen to the facts. (6:15)

A myth that will not stand criticism must ultimately be protected by force, but often subterfuge will do. There were jazz-hands all around, mine included. It was the social vision Reimer had given us, of involvement, discussion, consensus, rationality, but stripped of the technology: just people.

After the speech we grabbed some lunch, and I split off to go to MacLeod’s books, the insanely jumbled second-hand bookstore at W. Pender and Richards. Just the place to settle your nerves after an overstimulating day. I was there for a couple of hours.

Then, still with time to kill, I went to the Vancouver Public Library, which looks like the Coliseum reoccupied by information freaks (I saw serried ranks of sticky notes stuck to a second-floor office window, planning – what? An event, a collection, a website?). I found a desk with an electrical outlet and tried to sort out the relics, digital and physical, of the week: tweets, photos by and of me and others, a few emails and Facebook updates, the video I took of Suzuki, a panorama I’d made with my phone and posted to Occipital (who owns them? How long will they take care of it?), the flyer from the Occupy meeting, the hotel bill, the books, the presents for my family in Edmonton.

All these zooms to visualize:

From the personal level to the community level and back again: as RCB proposed anchoring the project of history in expanding concentric circles rippling out from the individual, I want to see my digital work rooted in personal archiving practices that mesh with institutional and social movements
From the past to the present to the future: recovering old stories, preserving present experience so the stories it can tell will still be heard in when I’m no longer there to hear them
From information into action and back: guiding public policy with good information, in this age when we see how far wrong public policy can go when information is ignored or lost or hidden

I think those are the dimensions we work with.

Make BackupPC tweet its activities

Fri, 14 Jan 2011 23:48:42 -0700

We’ve been using BackupPC as our home backup system for the last year or so, and it’s done a great job. It runs under Ubuntu on the home server (an Eee box) and backs up the various family laptops to a RAID array composed of two 1tb USB drives. For offsite backup, we now have a couple more USB drives that receive weekly archives from BackupPC and are swapped to and from my office. For the first time, I feel like we’re doing a decent job taking care of the family digital jewels.

Recently I’ve been looking at the content we create and store on external services. We’re all on Facebook, my daughter and I have five active or at least non-defunct blogs between us, I tweet. Why not archive all that as well? The pleasure I’ve been getting from reading my grandparents’ papers will, for our posterity, depend on this stuff. Not to be morbid, but read last week’s New York Times article “Cyberspace When You’re Dead”. Personal digital archiving is important, and we need to learn to do it properly. The Internet Archive is sponsoring its second conference on the subject in a few weeks.

I’ve started with Twitter. This has forced me, finally, to write some Python, in order to use the OAuth libraries necessary to authenticate to the Twitter API since they turned off basic authentication last summer. Jeff Miller’s tutorial, which uses the Tweepy module, got me going. Once you’ve got the authentication working, the demo script allows you to publish tweets from the command line, which is cool, but… Trying to think of a use for it, my mind went back to BackupPC and the need to monitor its actions and notice when backups aren’t working. It sends email notifications, but why not get them in the Twitter flow as well? It’s the perfect place for short notices.

So, our server now has its own Twitter account. BackupPC provides hooks for scripts to run at different stages of its various processes. To send a tweet after a backup job, add this to the configuration file for a host:

$Conf{DumpPostUserCmd} = "/home/backuppc/bin/post.py backuppc: $client $type: $xferOK"

This will send a tweet in the form “backuppc: cheetah full: 1” (where cheetah is the host, the backup was full, and it completed successfully – 0 indicates a problem).

The script post.py is based on Jeff Miller’s example. I modified it to compose the tweet by concatenating all the command line arguments (to avoid the use of quotation marks, which I couldn’t get BackupPC to handle properly no matter how I escaped them).

#!/usr/bin/env python

import sys
import tweepy

CONSUMER_KEY = '*********************'
CONSUMER_SECRET = '*********************'
ACCESS_KEY = '*********************'
ACCESS_SECRET = '*********************'

auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth)

# drop first arg, which has name of script
args = sys.argv
args.pop(0)
tweet = ""
for arg in args:
	if tweet != "":
		tweet += " "
	tweet += arg

api.update_status(tweet)

All for the thrill of seeing tweets like this:

BackupPC tweet

Home Zotero repository

Sun, 19 Dec 2010 12:07:00 -0700

The bibliographic manager Zotero lets you sync your collection to a remote server, which is handy when you work on more than one machine. They provide space for free which is plenty for my citations; but to sync the attachments (pdfs, web page snapshots, etc.), the 100mb of free file storage isn’t enough: I’ve got 95mb of stuff already. You can, however, sync the attachments to any WebDAV-accessible storage. I have a home Ubuntu server with RAID storage for the family digital treasures (we run a backup service for all the machines in the house using the excellent BackupPC package), so it made sense to use that. Here’s a quick list of the steps involved.

Set up a Linux server. I’ve been using Ubuntu Netbook Edition on an Eee box for a couple of years with no complaints. Ours is pink; we call it “panther”. (You could probably do all this on a Mac or Windows machine that’s on all the time, if you prefer).
Buy a couple of large USB drives and set them up as RAID 1 (or buy more and get more redundancy). Software RAID is easy to set up in Ubuntu. Mount the RAID array at e.g. /raidarray and create the directory which the WebDAV service will use as its home, e.g. /raidarray/webdav.
Now that you’ve got some storage and a server, you need WebDAV. Install Apache, and add WebDAV capabilities to it. Make sure the Apache user has read/write permissions on the webdav directory.
One of the points of using your own server is that you can encrypt your network traffic using a self-signed certificate, avoiding the significant cost of a real one. Set up a self-signed certificate in Apache.
Now you need to make your server available on the Interwebs. If you use a cable or DSL service, you probably don’t have a fixed IP address, which makes you hard to find, since your provider may change your IP address frequently and unpredictably. The solution is dynamic DNS, which allows your server to notify the DNS when its IP address changes. Set up a free account at DynDNS.com. You will also have to install a little package on your server that will check your current IP address periodically and report changes to DynDNS.
Set up your router to direct incoming traffic on your chosen port to your server. The settings are probably under the games and applications tab of your router config interface. Choose a high port number (make the script kiddies work).
Finally, put it all together: set up the virtual host in Apache that will serve your WebDAV repository. My virtual host configuration looks something like this, with service at https://mywebdav.dyndns.org:12345/webdav/ (not my real host), the RAID array mounted at /raidarray, and the certificate authority and other ancillary stuff in /var/www_dyndns:

<virtualhost *:12345>
    ServerAdmin me@myemail.com
    ServerName  mywebdav.dyndns.org

    # Indexes + Directory Root.  
    DirectoryIndex index.php index.html  
    DocumentRoot /raidarray/webdav/

    <directory /raidarray/webdav/>  
    Options Indexes FollowSymLinks MultiViews  
    AllowOverride All  
    Order allow,deny  
    allow from all  
    </directory>

    # SSL  
    SSLEngine on  
    SSLOptions +StrictRequire  
    SSLCertificateFile /var/www_dyndns/myCA/server_crt.pem  
    SSLCertificateKeyFile /var/www_dyndns/myCA/server_key.pem

    # Logfiles  
    ErrorLog /var/log/apache2/mywebdav.dyndns.org/error.log  
    CustomLog /var/log/apache2/mywebdav.dyndns.org/access.log combined

    # webdav service  
    <Directory /raidarray/webdav/>  
    Options Indexes FollowSymLinks MultiViews  
    AllowOverride None  
    Order allow,deny  
    allow from all  
    </Directory>

    Alias /webdav/ /raidarray/webdav/

    <Location /webdav>  
    DAV On  
    AuthType Basic  
    AuthName "webdav"  
    AuthUserFile /var/www_dyndns/passwd.dav  
    Require valid-user  
    DavMinTimeout 600  
    </Location>  
</virtualhost>  

Restart Apache and you should be good to go. With your browser you can visit https://mywebdav.dyndns.org:12345/webdav and see the contents of your directory, after you accept the certificate and log in. You should also be able to mount that directory as a network drive from any client.

To use it for Zotero, create a directory named “zotero” in the “webdav” directory, and then go to the “Sync” tab in the Zotero preferences, turn on file syncing using WebDAV, and give it your address. Do the same in other machines you want to sync. As Zotero syncs, you’ll see the /raidarray/webdav/zotero directory fill up with .prop and .zip files.

Caveats and lessons learned:

I found that the process that determines the current IP address timed out a lot due to load on DynDNS’s servers. I got around this by installing a script on my own website (where this blog is hosted at Dreamhost) for it to visit instead. Instructions were easy to find.
Visiting your server using DynDNS from within your home network can be problematic. I couldn’t get it to work with my first router: requests would just hang. When that router died and I replaced it with a Cisco Valet, the problem went away.
Your server will be unreachable during the interval between a change of IP address by your provider and the next update to DynDNS. This hasn’t been a problem for me yet.
The self-signed certificate encrypts your traffic, but it creates a theoretical vulnerability to spoofing. Buy a real one if you’re worried. I wouldn’t use this to store sensitive personal information, though.
If you have problems, check the Apache logs. The first thing to check is permissions: can Apache read and write to the Zotero directory?

There is a giant floating Easter egg in the Quad

Mon, 26 Jul 2010 18:54:21 -0600

Which makes sense, because I put it there. The giant floating gold-foil-wrapped Easter bunnies, on the other hand, make no sense at all.

Augmented reality is very weird.

I’ve been playing with Layar’s Hoppala Augmentation service (with a hat tip to Fiacre O’Duinn at Library Bazaar). Via the web interface I placed an object in an augmented reality layer: an Easter egg in the demo “Hoppala goes Easter” layer. Then I installed the iPhone app, found the right layer (search “Easter”), and strolled through the Quad on my way to the LRT. It looked like this:

It was distinctly bizarre to look through the little window into an alternate world where this huge red egg floated along the path beside me. Freaksville. And people give you odd looks when you walk around the Quad holding your phone out at eye level. When you explain, though, that you’re watching the giant floating Easter egg, and that the floating chocolate bunnies aren’t your fault, people leave you alone.

This was my first experience using an augmented reality app with live GPS positioning. I must say, it’s disappointing in an entirely predictable way. GPS on a phone is rough, and the accuracy of your position changes from moment to moment. That means the egg moves around a lot as you walk, and its position in the middle of the Quad is only approximate, because the app’s sense of your position is only approximate. When building an augmented reality application, you’ll have to take into account that your users may not see your stuff exactly where you want them to, and your stuff might shift unexpectedly instead of smoothly panning and zooming to match the viewer’s movements. I’ll have to give up my fantasy of neatly overlaying an historical image over the corresponding contemporary view, like this set from the National Library of Ireland, where the juxtaposition of past and present is so immediate it makes your heart ache. Or maybe there are ways for the app to work around the limitations of a phone’s GPS.

Anyway, if you’re around U of A and you have an iPhone or Android phone, go to the Quad and check out the egg. And the bunnies. Maybe they do something if you click them. Do you think I should click one? Maybe I should go back and click one.

New Gizmo

Fri, 23 Jul 2010 21:14:30 -0600

Marshmallow models keyboard

Got me one of those Menotek floppy bluetooth keyboards, now that the iPhone supports it (in iOS4; kitteh not included). It takes some getting used to: I keep doubling letters. The iPhone’s autocorrection helps. We’ll see if this can be as useful as the old Palm folding keyboard, which was a wonderful keyboard; I could have transcribed War and Peace on it.

QR Codes Wherever I Want

Fri, 30 Apr 2010 20:49:07 -0600

The University of Alberta added WorldCat Local to its web offerings a while ago, but for my own library use I’ve gone on using our old OPAC by default, for no other reasons than familiarity and inertia. Now, though, I’ve found a solid advantage that WorldCat Local offers to my personal workflow: fixed record-level URLs.

I was trying to solve the age-old problem of capturing a call number so that it will be easy to consult when I get to the stacks to pick up the book. The OPAC record is on the screen of my workstation, my iPhone is on my belt: how to bridge the gap? Emailing it to myself is tedious, copying and pasting into a note and then syncing even more so. Taking a picture with the iPhone’s camera may get the call number but it’s hard to include enough of the citation to show which item this is, if I’m fetching more than one.

The solution I want, which was inspired by a tweet by Lorcan Dempsey, is a QR code that gives me the URL of the record. That way when I need it I’ll have the full citation, the call number, everything. QR codes don’t appear in WorldCat Local or in the OPAC (like Huddersfield), but there’s Greasemonkey for that, specifically the “QR Code for Everything!” script (and probably others, I didn’t explore). I can pop up a QR code for any page I visit in Firefox on my workstation, grab it into the iPhone using the free i-nigma app (or one of the other QR-reading apps) to snap a picture of the QR code, and then easily consult the full record in the stacks.

Capturing a QR Code in i-nigma (i-nigma is fast, making it very hard to get a screenshot)

A URL captured in i-nigma, with a “Go Online” button to take me there

The only problem with my default behavior is that the OPAC uses a session URL, which is meaningless once the session expires or when accessed from another device, so capturing it does me no good. WorldCat Local gives me a URL that doesn’t depend on the current session. That’s what I need: a cool URI that doesn’t change, at least across two devices and within the time-frame of my interest in a given book. I suppose I could customize the Greasemonkey script to use the OPAC’s permalink service before it generates the QR code, or we could enhance our unAPI service to provide a QR code as one of the options, but hell, WorldCat Local just works for this.

Embedding a SIMILE Timeline in Wordpress

Sun, 28 Mar 2010 13:26:41 -0600

I wanted to embed a SIMILE Timeline into a Wordpress posting, and it took longer than I expected to find a convenient way to do this without editing templates and such. There’s a nice plugin that does timelines for postings (WP SIMILE Timeline), but it doesn’t work for arbitrary timelines unconnected with the content of the blog itself. So, here’s a summary of a quick solution.

The requirement is to add some extra Javascript to a single posting (not for every posting, since that’s a fair amount of script to download needlessly), and to add onLoad and onResize handlers to the body tag. The easy solution: install the HiFi plugin, which does “Head & Foot Injection”. It allows you to specify some links you want to in the HTML head section, or at the end of the body section. It gives you a menu like this:

HiFi Screenshot

Those links will end up in the head for this posting.

The links are to the Timeline library, a local js file containing the configuration for the timeline we’re posting, and maybe a link to the css for that timeline. The local Javascript contains a call to a local XML file containing the events we ant to plot:

tl.loadXML("wp-content/2010/03/timeline-monet.xml", function(xml, url) {  
  eventSource.loadXML(xml, url); 
});

For convenience, I uploaded the js and xml as media files in the Wordpress dashboard; but it threw an error when it tried to save the xml, so I uploaded that manually. I use the year/month directory structure for my uploads (Settings / Miscellaneous / “Organize my uploads into month- and year-based folders”), so the uploaded timeline files end up in wp-content/2010/03.

The local Javascript contains functions onLoad and onResize, as in the Timeline examples. To get those to fire at the appropriate times, the Javascript ends by attaching them to the window’s onload and onresize events:

window.onload = onLoad; 
window.onresize = onResize;

Adding those two lines was the only change I had to make to the SIMILE sample code.

Finally, the posting itself needs a div with the appropriate id:

<div id="tl" style="height: 300px;"></div>

And that’s it. Here’s a demo, using the “Life of Monet” example from the SIMILE Timeline site.

[Demo removed after migrating this blog to Jekyll, 2017.]