already a blacklight 0.2

Bess has posted that 0.2 is out the door.

Within 24 hours of releasing Blacklight 0.1 she received a patch fixing a problem in one of the config files and augmenting the installation README. 0.2 is 0.1 but with the config file fixed, some better installation instructions, and exported from svn (as opposed to checked out).

She's ported the subversion repository over to rubyforge. The source is browseable here: http://blacklight.rubyforge.org/svn/ and you can see all your svn options here: http://rubyforge.org/scm/?group_id=5235

Blacklight 0.1 source is out there

Check out Bess's post for more details -- Blacklight 0.1 source is now available at http://rubyforge.org/projects/blacklight/.

American Literatures

American Literatures is a Mellon Foundation-funded project where five university presses—NYU, Fordham, Rutgers, Temple, and Virginia—have established an initiative designed to create new opportunities for publication in humanistic scholarship. The most innovative aspect of the program will be the establishment of a shared, centralized, external editorial service dedicated solely to managing the production of books in the initiative. This service will handle all copyediting, design, layout, and typesetting costs, and manage each title through to the point where it is ready for printing. The initiative has a web site and has announced on its "about page" (scroll down) in which areas each press is soliciting submissions.

I look forward to watching how this progresses for the UVA Press.

what we learned from our repository project

Earlier this week someone asked me what we had learned from our repository development project over the years. This is the first time that anyone has asked that so directly, as opposed to general discussions about assessment and process review and software optimization.

So, what did we learn? This is what I've come up with so far.

1. Have your media file standards in mind before you start. Of course standards will change (especially if you're talking about video objects) during a multi-year implementation project. But if you have standards identified before you start (and minimize the number of file standards that you'll be working with), you at least have some chance of making it easier to migrate and manage and preserve what you've got and to design a simpler architecture. We did this (in conjunction with an inventory of our existing digital assets) and it was key for us in developing our architecture and content models.

2. Know what the functional requirements of your interface will be before you start. We had developed functional spec documents and use cases, but two different stakeholders came back to us during the process with new requests that we couldn't ignore. In those two cases the newly identified functional requirements for our interface required that we change our deliverable files and our metadata standard. We had to go back and re-process tens of thousands and then over 100,000 objects to meet the functional need and have consistency across our objects.

3. Some aspect of your implementation technologies will change during the project. New technologies will become available for implementation during your project that are a better fit than what you planned to use. For example, we never initially identified Cocoon as part of our implementation, but it became a core part of our text disseminators.

4. Your project will never be "done." OK, we've got a production repository with three format types in full production and three in prototype. We've still got to figure out production workflows for those media types and there are more media types to consider. And, as a corollary to point 3, there are new technologies that we want to substitute for what we used. We're obviously going to switch to Lucene and Solr for our indexing. New indexing capabilities will absolutely bring about an interface change. There are also more open source web services applications available now than when we started in 2003. We can potentially employ XTF in place of some very complex TEI and EAD transformation and display disseminators that we developed locally. This is bringing about a discussion about simplifying our architecture -- fewer complex delivery disseminators to manage and develop and more handing off of datastreams to outside web services. Not that there aren't complexities there and a lot of re-development work, but it's a discussion worth having. We're talking a lot these days about simplifying what it takes for us to put new content models into production. The development of Fedora 3.0 will also have a huge effect.

EDIT 28 December 2007:

A couple of folks wrote and asked me why I didn't specify metadata standards. I mentioned the impact of interface design on metadata needs, but some additional reinforcement can't hurt. So ...

5. It shouldn't even need to be said that you should have your metadata standards identified before you start your development. We did. What we learned is that activities related to categories 2-4 will mean that you will have to make changes to what metadata you create and how you use it. For example, when changes were made to our interface design and functionality, we needed metadata formatted in a certain way for search results and some displays. We thought that we'd generate the metadata on-the-fly, but that turned out to be a lot of overhead so we decided to pre-generate the metadata needed: display name, sort name, display title, sort title, display date, and sort date. It isn't metadata we necessarily create during cataloging processes, but it's something we can generate during the conversion from its original form to UVA DescMeta XML. Another example is faceted browsing. To have the most sensible facets in our next interface, we need to break up post-coordinated subject strings or we'll have a facets for every variation. We thought about pre-generating this, but it turns out that Lucene can do this as part of the index building.

(http://digitaleccentric.blogspot.com/2007/12/adding-metadata-to-list-of-what-we.html)

developing a service vision for a repository

Dorothea rightly challenged me for not including a service vision in my post on repository goals and vision. I do have something like a vision, but I wouldn't say that it's quite where it needs to be yet. That said, I said I would post it, so I am.

What are the services needed around a repository?

• Identification and acquisition of valuable content
• You can't wait for content to come to you – research what’s going on in the departments and at the University, and initiate a dialog.
• Digital collections must also come from the Library and other University units – University Archives, Museums, etc.
• Consulting Services
• Advise on intellectual property and contract/licensing issues for scholarly output.
• Assistance in preparing files for deposit, creating or converting metadata, and in the actual deposit process.
• Access
• Easy-to-use discovery interface with full-text searching and browse.
• Instruction for community on how to find and use and cite content.
• Make the content shareable via Open Archives Initiative (OAI).
• Promotion and Marketing
• Build awareness of the high cost of scholarly journals, and that we are buying back our own institutional scholarship.
• Promote the value of building sustainable digital collections – preservation is more than just backing up files.
• Promote the goals of the Open Access movement, including managed, free online access and a focus on improved visibility and impact.
• Show faculty that they can build personal and community archives.
• Market repository building services that will enable the institution to build a body of digital content.
• Market the repository as a content resource and a venue that increases the visibility of the institution.

goals and vision for a repository

Last week I had the opportunity to have a lengthy conversation with some folks about our Repository. In doing so I was able to get at some really simplified statements about our activities.

Why a Repository?

• A growing body of the scholarly communications and research produced in our institutions exists solely in digital form.
• Valuable assets -- secondary or gray scholarship such as proceedings, white papers, presentations, working papers, and datasets -- are being lost or not reproduced.
• Numerous online digital collections and databases produced through research activity are not formally managed and are at risk.
• An institutional repository is needed as a trusted system to permanently archive, steward, and manage access to the intellectual work – both research and teaching – of a university.
• Open Access, Open Access, Open Access and Preservation, Preservation, Preservation.

What's the vision for a Repository?

• A new scholarly publishing paradigm: an outlet for the open distribution of scholarly output as part of the open access movement.
• A trusted digital repository for collections.
• A cumulative and perpetual archive for an institution.

What does success look like?

• Improved open access and visibility of digital scholarship and collections.
• Participation from a variety of units, departments, and disciplines at the institution.
• Usable process and standards for adding content.
• Content is actively added.
• Content is used: searched and cited and downloaded.
• There is a wide variety of content types.
• Simple counts are NOT a metric.

I really appreciate having the chance to formulate ideas like these that have nothing to do with the technology but everything to do with why we're doing what we do. I want to work this up into something more formal to share broadly.

LibX and OpenURL Referrer browser extensions

We launched our UVA Library LibX plugin for Firefox in June 2007, and its gotten some rave reviews from staff. Now that UVA has approved the rollout of Vista and IE 7 on its computers, we're testing the beta IE version of LibX. I understand we've supplied some feedback on installation and running on Vista.

When I saw the recent announcement of the availability of OCLC's OpenURL Referrer for IE, I paused a bit when considering who to send the annoucement to. LibX is the tool we promote with our users, it recognizes DOIs, ISBNs, ISSNs, and PubMed IDs, and supports COinS and OCLC xISBN, and works with our resolver and our catalog Virgo. We have our resolver working with Google Scholar.

In the end, I didn't forward the annoucement because we're trying to promote the use of LibX and I didn't want to dilute that message for our staff and users. The OpenURL Referrer is a very cool tool and a great use of the OCLC Resolver Registry so users don't have to know anything except the name of their institution to set it up. I'm just not sure if we need both, at least not right now.

I need to ask if we know how much use our LibX toolbar is getting.

killer digital libraries and archives

Yesterday the Online Education Database released a great list of "250+ Killer Digital Libraries and Archives." It lists sites by state, by type, has a focus on etexts, and is a remarkable compendium of digital resources.

Of course, the first thing I did was look for our digital collections on the list. The UVA Library hosts the wonderful Virginia Heritage resource, which brings together thousands of EAD finding aids for two dozen institutions across the state of Virginia. We have our Digital Collections, with more than 20,000 images, 10,000 texts, and almost 4,000 finding aids.

Nope. Not on the list.

Not surprisingly, our former Etext Center was on the list under etexts (the Center no longer exists as a unit and its texts are gradually being migrated). The Virginia Center for Digital History was there, as it should be with its groundbreaking projects and its great blog. IATH was there with its many innovative born-digital scholarly projects.

I sulked about this for a few minutes while thinking about the likely reason we weren't on the list -- for the past few years we've been talking nonstop about our Repository and Fedora and not about our collections. Now, we wanted and needed to talk about Fedora and our Repository because we we really trying new things and solving interesting problems with our development and participating in building a community around Fedora. But users don't care about how cool our Repository development is. They care about the collections in the Repository.

We've spent the last few months working at raising awareness about what we have. Our new Library home page now has a number of links to the digital collections. We have pages on how to find what you're looking for in our digital collections. We have feature pages for all of our collections in the Repository. We're making progress in migrating collections and making the Digital Collections site a central location where they're visible. We have an RSS Feed for additions to the collections. We now have a librarian and a unit dedicated to shepherding collection digitization through the process and working more closely with faculty. I hope the next time someone creates a list like this we'll be visible enough to be on it.

some weeks, you feel like you've just survived

This week was the first week of classes and it seemed more stressful than other first weeks. We put redirects in place for some directories for resources that we'd migrated from our former Etext Center collections to the Repository. We didn't give as much notice as we could have there and some folks were surprised. We also formally announced that our Etext and Geostat Centers no longer exist and are part of our Scholars' Lab. Those announcements required some redirects. Things then went briefly wrong with the redirects. There was a wrinkle in updating links in our catalog records -- in some cases we weren't migrating individual texts, instead pointing to LION, so where would the links go? And the redirects meant that records weren't going to as granular a location as they were before.

The increased load of the first week of classes caused some text delivery issues, but helped us find what appears to be a bug in Joost that was the cause of mysterious problems in the past that we now have worked around. Two tools the used to exchange data easily didn't anymore (but we found the cause immediately). An old assumption about what regions we included in our simple text search was proven false by some newly migrated texts and we had to make a mid-week change. One of the text sets that we migrated was missing what turned out to be a vital element from its styled delivery. We tried to be nimble in our responses, occasionally briefly breaking something else with a fix, but our amazing team worked hard to address everything quickly.

I know of two outstanding issues to resolve, then we're set until we start the process to completely replace our searching infrastructure and interface. We've got a prototype BlacklightDL almost where it needs to be to start seriously planning the swapout project. Another change management challenge ...

RomeReborn 1.0

As a technologist who trained as an archaeologist, I am excited by every virtual reconstruction of archaeological sites that I come across. I remember how impressed I was by the very early virtual Trajan's Forum project co-developed by UCLA's Urban Simulation Teams and the Getty, and UCLA's Virtual Los Angeles Project, both of which I first saw at the 1997 ACM Meeting. I was so impressed I invited Bill Jepson from UCLA to speak at the 1998 MCN Conference about virtual world building.

Today, the Comune di Roma celebrated the unveiling of RomeReborn 1.0. Bernie Frischer from UVA and Diane Favro from UCLA led an international team of archaeologists, architects, and computer modelers in assembling a huge recreation of Rome as it existed circa 320 AD under the Emperor Constantine, covering the area within the 13 miles of Aurelian Walls that encircled it. Virtual visitors can navigate through and around all the buildings and streets, including the Colosseum, the Senate House, and the Temple of Venus and Rome.

http://www.virginia.edu/uvatoday/newsRelease.php?id=2223

http://www.reuters.com/article/technologyNews/idUSL1114988420070611

http://chronicle.com/wiredcampus/article/2142/ancient-rome-restored-virtually

LibX at UVA

Some times things just come together the way you want them to.

A few weeks ago we started casually looking at LibX, following the announcements of beta tests at other institutions. We liked what we saw, but other projects got in the way of immediate follow up.

Last Tuesday one of our subject librarians mentioned that she'd seen a demo, and it \seemed time to get back to it. Jim Campbell pulled together what we needed and sent the configuration file off to Virginia Tech at 5:13, and received a test version UVa LibX toolbar back at 5:26. I had it installed and was searching at 5:29.

It was meant to be. A couple of other folks had seen the same LibX demo as the subject librarian who contacted us. They had decided that this was something we needed to look into, and there was some surprise when Jim sent out the announcement that we'd set up the prototype because we hadn't yet let folks know that we were working on it.

It all came together, and just nine days after setting up the test version (and making some config changes) it was announced that the UVa LibX toolbar was open for business at a Library-wide meeting. And we've arranged for the toolbar to be added to Firefox on all public Library machines for fall.

Quoting the email that Jim sent to the Library staff:

You can search Virgo as well as our ejournal list, WorldCat.org and Google Scholar.

LibX will also put the little orange Rotunda on pages from Amazon, the NYT Book Review, and other sources linking to a Virgo search.

If you highlight a term on a Web page and then right-click, you'll get a menu of search options.

But truly the coolest thing is highlighting the title in a citation on a Web page or in a PDF and dragging it onto that Scholar button. The LibX developers call it their magic button and a lot of the time it really does seem to work that way.

If it's an article we have electronic access to, you don't really even see Google or our Resolver -- you get to the article that quickly. It's a lot more efficient than going to our citation finder, typing or pasting in the info, being taken to a results screen, and then to the article (if not to the journal issue before that).

And the first time I saw a UVa symbol next to a book title in Amazon (David Wienberger's Everything in Miscellaneous) It was just plain cool.

updating the uva library web site

I want to point everyone to the new blog set up by our Communications department discussing the process for updating our Library web site.

It's "Radical Transparency" at http://uvalibwebdev.wordpress.com/.

Please take a look at what we've got going on. We're definitely looking for feedback!

blacklight exposed

I see that Bess has posted about her exciting Project Blacklight, where she worked with Erik Hatcher on developing an experimental interface for our MARC records. Props are also due to Chris Hoebeke for working with them on the data and mapping fields and subfields to the facets.

It's not a full replacement for our catalog yet, but it's exceptionally promising. There's a real buzz here about it, not just because it's cool, but because it started under the radar but has gone on to capture the attention of folks across the Library. Check it out.

catching up

I just realized that I haven't posted in entry in close to a month. I haven't disappeared from the face of the earth; rather, I've been nose-to-the-grindstone on some projects.

The Virginia Heritage Project went through an overhaul of some of its underlying functionality.

The text delivery in our Repository hasn't failed in quite some time.

Some milestone legacy text sets were migrated into the Repository collections.

We're planning logistics for our Google Book Search project activities.

Now I can re-surface and join the world again. I can catch up on what happened at code4lib 2007, and at the Users and Uses of Bibliographic Data Meeting, finish my contributions to the glossary that I'm supposed to be working on for the Fedora wiki, and think about what the future holds for us as customers with SirsiDynix's announcement of "Rome."

success is a double-edged sword

Since we launched our repository its success has revealed some server issues. We discovered previously undiscovered capacity issues that three very patient and experienced programmers and sys admins have been working very hard to troubleshoot. To allay fears, no, we had no problems with Fedora. But Fedora kept thinking that Tomcat/Cocoon wasn't responding, so Fedora would decline to complete its disseminations. The most puzzling thing was that our image delivery worked just fine, but our texts would fail. Actually, they'd work for a while and then fail. The texts are much larger and have much more complex disseminations and rendering, so we knew it was a capacity issue of some sort. The much increased number of such complex renderings was causing something somewhere to intermittently give up the ghoast.

We ended up doing a number of things: adding more retrys when requesting objects found through text search results. Moving Cocoon and Tomcat onto a newer box with upgraded versions and a lot more memory to allocate. Upping timeout limits in Tomcat and Apache. We found a log that kept filling up. There was a Cocoon STX bug that we needed to take into account in one text disseminator. We have one remaining mystery issue -- some Apache connections that aren't being released. It may be from Cocoon errors that aren't being properly identified as such and going into a wait state.

It's interesting what we never found in 2 years that we found in three weeks when more people started using the service.

unveiling of our repository

It seems like I've been working towards the unveiling of our Digital Collections Repository forever. participating in architecture planning. Collecting functional specifications. Coordinating production standards. Watching workflows come together. Watching teams coalesce. Implementation. Meeting with faculty for testing and feedback. Working with Library staff on testing and feedback. More implementation. More testing.

Finally, after 2 years of alpha and beta versions, we unveiled the Digital Collections Repository yesterday. We had been calling it a launch, but after two years it's really an unveiling.

There were a couple of glitches. High traffic caused issues with text rendering, seemingly due to server timeouts. Some updates hadn't gotten added to the index so a couple of faculty couldn't find some specific images. Expectations exceed some of our search capabilities (I can't index what I don't have in the metadata). But it's out there. We can rest on our laurels for a few days, then start planning the next release and the production and delivery of additional formats. And likely an entirely new indexing infrastructure.

Today, I got a message from a faculty member who I have never met. The selector for her department had emailed the announcement. She liked what she saw and wanted to know how to get her image collection selected and added. I am thrilled.

http://lib.virginia.edu/digital/collections/

screencasting

I want to give a shout-out to our Library's user education team, who have started screencasting. They've developed some great tutorials on our catalog, on RefWorks, setting up proxies, etc. Give it a look-see!

http://www.lib.virginia.edu/usered/tutorials.html

new del.icio.us for uva dl

For years I've been forwarding notices on articles, reports, sites, and what-not to to various lists and groups at the Library. My colleague Cyril pointed out the obvious to me the other day when he and Ronda was presenting a session for library staff on del.icio.us -- that while the email messages were useful, an annotated and tagged del.icio.us set would be even more useful (and more persistent than messages in folks' inboxes).

Given that it's the week before Christmas and work was winding down for the break, I went through three years of outgoing messages to particular internal email lists (yes, I'm a compulsive email hoarder) and created a del.icio.us set:

http://del.icio.us/uva_digital_library

I have a lot more to add, it needs some work on the tags, and I haven't created any bundles or set up any networks yet, but it's a start at pulling together things that I find useful and think my colleagues should know about.

google news

It's now official -- the UVA Library is joining the Google book scanning initiative.

Here's the UVA press release: http://www.virginia.edu/uvatoday/newsRelease.php?id=1053

And the identical Google press release: http://www.google.com/press/annc/books_uva.html

We're very excited here. Still feeling a bit overwhelmed as we get ready to think about the scale of the process, but excited nonetheless. It's not yet set when we're starting or what materials we're sending. It's going to make quite a change in our local digitization efforts.

discussion of blogs and rss

I led the second in our "Not for Geeks Only" discussion series last week on blogs and rss -- 40 people signed up! We really seemed to have struck a nerve , and our Library staff seem really pleased to be introduced to these topics in a personal, targeted way rather than wandering through every random page on the web trying to find what's relevant.

It's not comprehensive, but covers some blogs that I and my colleagues here use every day to help us do our jobs.

http://staff.lib.virginia.edu/HR/training/classdocs/blogs.ppt

Coming up in the future -- GoogleScholar and Google Book Search, flickr, del.icio.us/social tagging, and firefox extensions.