This last week, I had the pleasure of meeting with folks at the National Library of Medicine (NLM). This is one of the nation’s outstanding public institutions, responsible for PubMed; a comprehensive collection and licensing program in journals and texts; and unique, historically important and vital archival materials.

One of the NLM’s missions is to serve as a collection of record for the product of scientific research and (in their case) health sciences knowledge and exploration.

Historically, libraries fulfilled this aim by acquiring and carefully storing works in their physical form, paper, which is relatively costly to produce (compared to replicating digital copies), not easily distributable, and therefore rare. Of course, as our accumulated knowledge, research, speculations, and culture grew, achieving the purpose of what we know as the modern library required the construction of increasingly large buildings to house the printed material and the development of a highly trained staff to catalog and describe it.

With more of our research becoming an electronic record, there have been new efforts to understand the requirements for its necessary preservation. There have been some notable initiatives that are essentially joint ventures between the publishing and library communities to construct preservation repositories for e-journal content; these include the Mellon-supported Portico repository and the LOCKSS system. These collaborations share costs and include provisions for libraries to preserve access to these materials if their publishing partners exit the continued production and maintenance of scholarly literature.

It was disturbing to hear at the Library that increasingly, the largest publishers of STM (Scientific/Technical/Medical) material are asserting that they provide the repository of record for their journals, and they have little interest in participating or continuing to participate in library alliances to preserve this content. Essentially, these publishers are telling the entire public, “Trust us. We have an economic interest in this material, and we are its shepherds, and we will preserve it for you.”

If this perspective is becoming mainstreamed in this publishing community, it must be denounced as a disastrously shortsighted one. It is an abnegation of our responsibilities as a culture to permit the preservation of the world’s hard-earned record of knowledge and speculation to be commercialized to the exclusion of the greater interest of the commonweal; all members of a society must share the responsibility for preserving the record of our struggles and experience.

In this light, I am even more disturbed by the absence of any significant conversations between publishers and the library community to establish standards that permit the preservation of digital books and mixed media texts as these become increasingly dominant. This concern of mine is not new, and was expressed in conversations within the OeBF; I have an obligation to raise awareness within its successor, the IDPF, now that I am a board member. Given the inherently delicate and mutable nature of digital material, it is imperative that we begin serious dialogues on this issue to ensure that the cultural and scientific products of our society are adequately distributed among organizations whose primary mission is to preserve access to that record. Those organizations are libraries, not publishers. Libraries.

I must believe that the publishers of honor who have survived longest on this planet must recognize the responsibility to preserve is one they are obligated to share with public institutions, and their best partners by dint of a long, established history are university and government libraries. These publishers have a long, fine, proud record; they surely do not want it tarnished through an accidental loss either by themselves or their partners, an accident no less catastrophic for the lack of ill intent. Therefore, our old friends, let us talk anew.

More generally, and much more poignantly, I am profoundly concerned about a shift in the equilibrium between libraries, advertising and merchant companies such as Google and Amazon, and commercial publishers, as reflected in the correspondence establishing what we must consider to be a social contract. All of these actors, from different springs of motivations, seek access to increasingly networked, and therefore increasingly valuable, information.

Was it to be permitted by Mammon that libraries be able to hold print material for the public good only while its wide circulation and dissemination were frustrated by its very physical nature? And that now, when information may be freely embedded, fragmented, re-combined, and flashed across the network, there is so much to be gained that there is too much to be risked, by allowing, nay, by encouraging, public institutions to possess it except through ephemeral licensing?

Let me be clear: I do not ascribe evil purpose to commercial enterprise. They are after all, publishing or advertising companies, and in the pursuit of their legitimate corporate aims, they are undeniably adding to the public good. Rather, I will instead persist in proclaiming, “You are not doing enough. You are doing good, but not enough, no longer enough.” And it is all of our fault for that, all our inability to elevate this concern, to make it a crisis, for it is. Our societies might not fail for it, but they will be lesser for it.

I am aware there are many who counsel such thoughts be left at whispers and not be loudly spoken, at the risk of losing Commerce’s half loaf of the newly digital, thus newly useful, and so they defer in their asking for the full loaf of what should rest ultimately with the Public. And therefore I think it might well inevitably never do so. And certainly the half loaf will not rest unbaked by our rightful asking for the whole. If we do not risk the offence, and fail to take the chance of a necessary conversation, then we will have sundered some of our children’s chances.

Comments to comments, as part of that infrastructure is not well behaved, so here I am.

Dorothea,

There are some huge administrative issues.  Many of them exist as well at the institutional level, but solutions would be necessarily different in construction at the consortial HE level that I proposed.  I don't have a full suite of answers here, but clearly there have to be operational as well as governance and coordinating roles assigned to responsible parties.

Re: Amazon.  I think the larger question is again, would I trust any one provider without contractual guarantees?  A supercomputer center?  Not likely.  A large university?  Not necessarily.  So for any of these solutions, or their combination, one would have to seek SLAs and strategies for ensuring content viability, etc. 

Re: DSpace v. else.  It could be Fedora.  It could be CDL's Common Framework repository.  But it has to be something, and not many things, or you lose a lot of the goodness of scale.  HE tries to hard to accommodate distinctive advantages, at the loss of overall benefit.  I chose DSpace in my example, but whatever it is, we need to ride one horse and not a stable.  And if we did something like this, let's not spend a lot of time doing studies.

I agree with you on ORE.  And, I think realistically there will always be any number of institutional or even department scaled repositories.  That's probably an advantageous thing, and ORE and whatever else DLF's Aquifer can test and verify should also be supported.  I don't see it as black and white, one ring to rule them all, but otoh, I think we have to have that one ring as a strong and blessed option.

Jerry,

I used my community/individual terms somewhat sloppily, but I was thinking that the community served was some bounded subset of HE, like (I dunno) R1 institutions, and the service would be open to eligible users within them.  Of course we get into this huge thing about what is an eligible user, but let's say that this is institutionally determined for now. 

I think ideally this system would benefit from being flexible enough that sub- communities within this larger community - which you are right, would most likely be a consortium in organizational form - would create their own local admin superstructures.  Maybe the plant geneticists want to do something special, for example. 

There are clearly issues with types of data - again, not necessarily a unique problem to this model.  In fact, this was pointed out to me once when I presented an earlier version of this at SciFoo at Google.  The biggest problem, which would absolutely require gating, is the submission of out-of-scale files.  Most repositories are not going to handle petabyte files, nor would our networks, yet; additionally, a trillion 4k files are probably not happening with most storage systems, either.  Not to be flip, but these are technical issues that have to be administratively handled. 

The incentive problem is a tricky one, and in part I hope that scale, and clever design, would offer some positive feedback on use.  The ability to have a central store, with tagging, alerts, persistent linking, and other good things would I hope create an economy of services that would drive additional business.  This is one area where I think aggregation trumps many repositories not well coupled. 

Raymond,

DLF would be happy to participate, but we need someone interested enough to handle the engineering, etc.  

Several threads came together for me today in an undoubtedly useless thread, but I'll spin it out anyway. What's a blog for?

The various components are: Dave Winer's musing that there should be a generalized service for people to preserve their own content; thinking back to my work at the California Digital Library, where I hooked up CDL's Preservation Program with folks at Amazon's S3; a blog on Amazon's Elastic Compute Cloud (EC2) and its impact on something or other that I've completely forgotten; a conversation today with someone from UIUC who was talking about a model for repository (digital object archive) interoperation that I found myself not agreeing with philosophically, based on admittedly inadequate information; an email list proferring that Google may offer file-based storage for Google account users; an excellent review of what made YouTube successful as a service; and a touch of rye whiskey.

So at the end of this windy introduction, onto my evolving set of basic thoughts about preservation, and repositories.

1. I do not believe that libraries lend sufficient additional value to interpose themselves in the process of digital curation, compared to the individual/scholar who has either created or alternatively gathered the content, and is in the best position to evaluate its saving. Although libraries may well initiate very high value preservation acts, they should act in this fashion as privileged individuals, and not monopolize act of preservation.
2. Preservation should be initiated and controlled by the content owner or rights-validated gatherer, and content objects should not be gated or judged for exclusion except on the grossest factors (such as legality, rights issues, and so forth). Tagging, however, should be open to anyone within the eligible community. (Why heck, any god-fearing provider is able to take advantage of safe-harbor under DMCA.)
3. I think end-user facing services are far superior to institutionally-arbitrated ones. Further, consumer-facing services that live at the level of the network, not within the bounds of an institution, are likely to have more traction and more visibility. In other words, I think net-based apps can gain sufficient scale to make consumer-facing or community-facing applications superior to institutionally held ones across a range of critical factors such as performance and utility.
4. I think that academia is different enough from the general population's needs regarding storage in terms of content description, sharing, rights, intent of purpose, and potential or latent value to warrant their own community-based (higher-education) solution, which may not necessarily be distinct in form to a broader-based application.
5. I think it is quite sub-optimal (a nicer way of saying that something is stupid) for universities to continue creating their own individual, branded respositories when they don't talk to each other very well, and scaling is generally limited to the capabilities of the institution. Among other things, a continuing bias towards institutional solutions creates gross inequities in support capacity across the diversity of universities, in their size, focus, and aspirations.
6. I think P2P-based preservation strategies tend to possess significant technical administrative overhead to maintain peer-level coordination and cache consistency, compared to centrally-coordinated distributed solutions, and are probably at best an interim solution.
   
7. The availability and costs of network infrastructure, and our understanding of how to enact services across it, has advanced to the point that applications can scale easily without requiring the federation of institutional deployments.
   

So what I would propose is that a consortia of universities, perhaps led by their libraries, initiates adequate development to enable the deployment of a clustered instance of MIT's DSpace open-source respository system instantiated on Amazon's EC2 infrastructure. This network-based, community-oriented preservation repository would be directly available to all HE end-users with minimal gating and minimal content review. Obviously an adequate higher educational governance model would have to be created, and a means for adequate remuneration of the effort, which might range from institutional membership fees, or tiered service offerings which could be rescinded in case of payment lapse, obviating the development of free-riders.

Sure, this is probably silly, and it might not work. But it is no more silly than individual instances of DSpace or any other repository software; it solves the problem of repository inter-operation; it takes advantage of network-based positive feedback; it binds universities together in common purpose in a big way; it benefits harvestability and therefore content discovery through its aggregation; and it increases content visibility.

Google could do this themselves by gussying up Google Base and then bundling it as part of Google Apps for Higher Ed.

Maybe we should try it first?

A friend of mind attended RLG’s Discover to Delivery in New Contexts Conference in New York this past week, and sent me these (heavily paraphrased to protect the innocent) comments:

The conference covered a wide range of issues, and was very interesting. I think the consensus opinion at the close was that discovery has moved to the network layer and libraries should stop allocating their time and money trying to build better end-user UI, and concentrate instead on delivery, and their niche or customized services such as digitizing special collections, providing innovative end-user tools for managing information, and so forth.

This sounds like a great conference, and I would generally agree with these directions. In fact, they sound strikingly similar to those that a friend in the publishing industry has reached about trade publishers (which prompted me to relatedly inquire whether or not for certain classes of libraries, and certain types of bookstores, these organizations might merge in the future – but that’s another story).

These trends in libraries toward niches and communities of engagement suggest some tentative conclusions:

1) Libraries have been the archivists for around 350 years of active human historical documentation. The era of the production of cultural and scientific artifacts that are solely physical in nature is ending, and coincident with this, the existing role for libraries must change. As this record is digitized and described, we must develop new skills and services so our accumulated expertise can shift elsewhere. The future, as we know, is not only born digital, but born networked, and in great floods of interconnected data that will not stand well disposed for monopolistic, controlling efforts at curation and selection on behalf of an audience that has been empowered to develop its own expertise in information triage and review.

2) Libraries should begin to downsize, probably both physically and in staff. They should start shaping the production and licensing of digital books (by engaging directly with book publishers to help establish this channel, instead of letting it be defined without their input), and be smart about what material is worth cataloging, and to what extent. The majority of bibliographic metadata description and enhancement should be left to the national libraries and consortia such as OCLC, or in the hands of agencies and users submitting content to repositories. As discovery services move to the network there is less reason why libraries should maintain duplicative local data caches. It's time to start letting go of our technical and cataloging departments; many of these staff have valuable skills and are thirsting to make a contribution elsewhere; they see the world being transformed around them. We must echo this direction in the physical world: we must either re-allocate our under-utilized physical spaces, or tear them down. So much better a green lawn with benches and cherry trees than an under-utilized concrete monolith of a building.

3) We need to admit openly that the types of expertise we need in our future, smaller libraries are dramatically different than what we have heretofore valued. Indeed, change has been so fast, that some of our existing staff will not be able to be effectively re-directed. The salary savings can and should be re-allocated elsewhere, providing us a flexibility to invoke new initiatives that is now often beyond our grasp. In our future, smaller libraries, we must be network savvy, digital/analog amphibians. We need more the kind of graduates that our i-schools are producing (regardless of where they are produced), with a hefty dose of the public services and advocacy that are the highly valued morale heart of libraries. Engagement in the development of curricula for the skills for network driven information services must be an urgent priority.

4) A lot of the core battles that define how people are able to access and use information, and under what terms, are increasingly political engagements. Libraries will need to understand their worlds well enough to take clear positions, and assume an advocacy role for their users in an environment where the definition of privacy and fair use have been torn asunder and are in an active process of re-construction. Libraries have often seized this role: but now we must do so with more aggressiveness and assertiveness than ever before. Now is the time for our voices to be heard, and to help others to speak.

Just when you think there might be the slightest ever hope, the little-ist, teeny-ist hope that the RIAA might wind up getting a clue, they rush in to set us right.

Through Boing-Boing , a report that RIAA perceives they have a marvelous opportunity to teach college students the deepest morale values:

Yet this is about far more than the size of a particular slice of the pie. This is about a generation of music fans. College students used to be the music industry’s best customers. Now, finding a record store still in business anywhere near a campus is a difficult assignment at best. It’s not just the loss of current sales that concerns us, but the habits formed in college that will stay with these students for a lifetime. This is a teachable moment — an opportunity to educate these particular students about the importance of music in their lives and the importance of respecting and valuing music as intellectual property.

(From an Op-Ed in Inside Higher Education).

I just love this. I guess the most glaring stupidity is the conception that college students suddenly have new opportunities to do things they didn't before they walked on campus. But the more profound sadness is - as always - the complete inability to understand the huge transformation in how people engage with music and other content, and how much of a new opportunity there is for culture industries.

Libraries are beyond this, thankfully; even if they believe (sometimes, perhaps even often, justifiably) that they know better than students how to find information on-line, they don't sue them for what the students perceive as a simple, common practice.

DLF’s Aquifer Project has received new funding from the Mellon Foundation! Congratulations to everyone who has worked so hard on this project! (A project web status page will be developed in the next few weeks).

I get asked about Aquifer all the time, understandably, and because the issues are complex, I usually try to pass off with a good-natured “It’s great!” Which rarely works, of course. In truth (and in sum), I think Aquifer is one of the best tickets that libraries have to help obtain the collaborative experience and skills they desperately need to find new good purposes, and that may be the single most important aspect it possesses.

Significant inter-library and inter-university cooperation is really hard without additional external supports, and I think part of the re-invention of libraries has to address that. Finding appropriate models, and aims, for that collaboration is a huge challenge that we are just beginning to undertake on a community basis, and Aquifer is an early and shallow scratch in the dirt of this larger problem.

All of us know (in libraries) that it is no longer about "digital" libraries any more. We are talking about the whole library, and the name of my organization - "Digital Library Federation" - is a misnomer. A library cannot grapple only with digital; it must address its entire existing work flow, its assumptions, its role, its unique character, and how it works - at what levels and with what aims - with others.

I think one of the harder and longer-term questions is what types of organizations are most effective for libraries to get their work accomplished. OCLC/RLG, a close fraternal organization, and DLF will be increasingly working together because we can complement each other's strengths. But as a group, libraries need to think about why and how it is that OCLC can be effective: specifically, they pool (and shape) individual interests into a community statement, and they can build services directly through their own internal organization. DLF, which is in less of a position to do that because of our diminuitive staff size (2), must be more advocacy based, "start-up" oriented, and so forth. It must be more on the edge of future, and seeking investment areas of high potential leverage, recognizing that some of its efforts will fail. Aside from 1-2 aquifers, it is not, as we all recognize, in a position to drive deep, long collaborations itself; the administrative overhead is simply too high. DLF’s mission must be to encourage various forms of exploration -- to rush toward failure (and hopefully an intermittant, rewarding success).

Through the experiences we shall be gaining, and to achieve the aims that we must pursue, North American research libraries need to determine if we must create a wholly different kind of virtual organization, perhaps based on, or an enlargement of, OCLC/RLG, that actually enacts a large portion of the common interests of libraries, in addition to a DLF that is a little bit more edgy and explorative not just in technology but also in organizational relationships. Individual libraries, each to their own, are not going to be able to provide a lot of traditional library services (like d2d) in a world where scale trumps so much. They may well provide, as individual organizations, a set of other very interesting, assistive tools and services. Much of what we have done is going or gone, and we must be bold - and occasionally reckless - as we venture into new areas.

Development of new initiatives and technologies, in and of itself, must be directed towards directions that drive collaborative definitions of libraries. These are new forms of services and engagements, and there is much more at stake than the best new metadata schema or simply making content optimally available for harvesting.

I was listening to NPR yesterday, and heard a "Sound Clip" - a short segment of audio provided by a listener that "collects the sounds that fascinate our listeners." For some reason, that started me thinking about preservation.

I think I had always thought about the preservation of digital content items like Sounds Clips as a part of the process of preserving our culture - a snapshot, if you will. But the time that I have spent thinking about interactive media troubles that conception. Increasingly, we will live in an interconnected skein of rich multimedia, interactive services, and fully participative interfaces. People now, on the net, leap from semantically sensitive text to connected video to connected data display or analysis to writing, responding via Twitter, blogs, and community spaces like Facebook and MySpace. This has to mean something for digital preservation.

As a teenager said in the New York Magazine article, "Save everything ,"

If I don’t delete it, I’m still gonna be there. My generation is going to have all this history; we can document anything so easily.

What I have realized is that our conceptions of preservation are as outdated as our traditional conceptions of libraries themselves. Preservation cannot succeed if it is about capturing "things" - digital objects, or web sites. That's because we live in a sea of data, and what we need to capture, if anything, is something within the network: not objects, per se, but the flow, the pooling of the water that our data are on the network. This is a very different conception of digital objects - because it takes us away from objects and into the interactions and linkages between data - and I have frankly not thought through all that it should suggest for our efforts in preservation. Preservation must be different when we recognize that digital production is constantly happening, bound in networks, and made real through use.

Whatever preservation is, it might mean more of a focus on these things:

• Flow the preservation with the content, don’t dam it up to save it. What's worth preserving is not "the web site" or the "archive" but the interactions that people build among them.
• Collaborate to save aspects of the whole, but recognize you'll never have the whole.
• Don’t worry about fidelity; it shifts. All of the web is interactive. The "true" version will never exist. (It never really did; it always existed in multiple editions.)
• Engage with production. We can only preserve the living, and the best time and place to preserve is when the connections exist and work and demonstrate the utility of their social value.
• Recognize the value of the latent preservation increasingly inherent in the web.

Dave Winer speculated with a few of us the other day about what happens when a Flickr user dies; what happens to the account? A valid question indeed. But I think increasingly we need to realize that the data on the net is not going to vanish; like pools of water it might join other larger bodies, or might get cut off from network flow and slowly stagnate, but it will increasingly rarely vanish. Why? Because storage is cheap, and the value of having the data in the network will only increase.

These are incomplete and raw thoughts, but I think we need to shift our thinking about preservation to adjust with the way that our networked lives work, and move beyond focussing on preserving a static web more characteristic of our understandings and use 10 years ago.

 [P.S. Presentation I gave on these topics at Berkeley.]

I am honored by the journal Library Trends with a guest editorship for a special edition, very tentatively titled "Digital Books and the Impact on Libraries." The edition will focus on mass digitization efforts and their strategic impacts on library trends and workflow, with a focus more on policy than strictly technical issues. Topics will include not just discoverabiity and services integration, but also participative cultures, licensing and rights, privacy, and the unsolved question of preservation archives.

I will also use the opportunity to speculate on the longer term development of new story-telling techniques and mixed media narratives, with conjectures for how this type of work might impact publishing, scholarship, libraries, licensing, preservation, and search providers. As the Booksquare blog noted recently in an open letter to publishers from SXSWi, the trends are toward:

Multi-format storytelling. Cross-platform storytelling. Mixing words and sounds and pictures to extend the story beyond the book. Mixing fiction and reality in the blogosphere and beyond.

Although media-rich participative works are most commonly associated with fiction, publishers are already beginning to experiment with their integration into non-fiction volumes, and I think these will be valuable paths for academics and students as well.

I also want to acknowledge the generosity of Library Trends' publisher, The Johns Hopkins University Press , which has agreed to permit the resulting articles to appear online prior to their formal publication (currently scheduled for the Fall 2008).

I think Twitter is a very interesting concept; here are some recent descriptive posts from many-2-many, by Ross Mayfield, and Liz Lawley. 

So I've joined Twitter.  I'll be a low frequency poster, I think. 

But the fact that there is a lovelylicious Mac app - twitterific - might put me over the easy distraction edge ...

For greater social value and worth, see this more interesting than I originally thought article in First Monday, by danah boyd, " ... Writing community into being on social network sites."  It captures a lot of the normal Sturm und Drang of social networks. 

This last Friday, I was able to see some exciting new applications utilizing Google Earth and the Electronic Cultural Atlas Initiative work at UC Berkeley.  It struck me how the distribution of information through the network, combined with applications that centralize the ability to create clustered applications, like Google Earth, re-shape the landscape of information utility.  Nary a library in sight.  

I really am quite depressed about libraries.  They are wonderful community centers, computer labs, and archives for the ancient - paper things.  What they were - places that bought books and journals and then made them available on loan to others - is not going to be of much relevance for long.  They will be purchasing agents, houses for some types of services, perhaps.  Most everything else will live elsewhere, on the network.  

I think a lot of the work that libraries perform now is an investment in an object, like a book, or journal, as an object in-and-of-itself.  The largest amount of this work is sub-optimal for how people want to find and consume information-based objects.  Libraries catalog things, making them monuments for descriptive metadata, but the cathedrals are lost for people who can find their scripture online. The very paradigm of search that traditional metadata enables is of intrinsically low value for most users today.     

When I am awake too early in the morning thinking about such things, I occasionally thing back on work that I did in graduate studies in sociology.  I studied economic sociology, generally; specifically, what happens when industries are confronted with dramatic and profound shifts in how their business is conducted - either manufacturing, or product R&D.  

One of the things that I studied was the death of the traditional steel industry in the U.S.  Big Steel was threatened by a lot of things from the 1960s onwards, including the arrival of minimills, which were a dramatically different way of making steel.  But my studies of the death of steel suggested that they had held an opportunity to dramatically re-invent themselves, and were acutely aware of the changing environment in which they engaged themselves, and yet for a variety of complex reasons they choose not to invest in the re-creation of their industry. Their choice was not, to be clear, to emulate minimills, but to modernize and enter markets in related areas - high-end  fabrication, special purpose steel, ceramics and other high-heat and -pressure chemical and physical transformations.  These were roads largely not taken, in the U.S. 

And early in the morning hours, I think of libraries. 

Below I cite from one of my papers some descriptive information about minimills (think Web 2.0, and Google, and new paradigms for information discovery and delivery).  Folllowing that, I speculate on what Steel lost, and how.  As I have intimated, it is not my argument that minimills killed Big Steel's viability, any more than Google kills Libraries.  Certainly it was an important and critical threat in their market environment.  Rather, I argue that for a complex set of reasons, Big Steel chose not to make decisions which might have maintained their pre-eminence, and even while their world was changing irrevocably, they had held a gift, an opportunity, to help re-define what they were.  They let that moment slip through their hands.  

I think about libraries.  I wonder where libraries are in that moment, and if there are things -- and I think they will have to be dramatic things -- that will preserve their role, even grow their importance, in our societies -- to be something more than a nook in which to sit quietly, reading or watching information that rests elsewhere, and a pleasant place to spend time with friends.  Those are wonderful things.  But they are called "cafes."

On minimills -

 Independently owned minimills in the U.S. began springing up in quantity in the late 1960s, and they had captured a quarter of the American market by 1985.  Many of these plants were a result of backward integration by steel fabricators who realized huge savings by producing their own inputs. 

Their success would have been impossible before the 1960s when Big Steel controlled the market and was still relatively efficient.  But as new methods suitable for small production runs became available, a niche for minimills which utilized efficient techniques for local customers appeared.  The minimills out performed foreign competition by using scrap metal as input and by avoiding expensive transportation costs of finished products; they out-performed nearby integrated mills through superior production techniques and by supplying specialty products.

By the 1980s, minimills were increasingly direct competitors to the largest steel firms, expanding both their capacity and their line of goods.  Nevertheless, the Big Five [steel firms] did not attempt to emulate the minimills.  Not only would it involve prohibitive write-offs of very expensive investments in integrated mills, the construction and production technology was so different that they were not equipped to handle them.  Minimills built their facilities without bells or whistles and with an eye toward planned obsolescence.  Integrated producers did not know how to "build tight, build quick, and build cheap." 

The chairman of Florida Steel, which operated five minimills, described his plants in terms not applicable to integrated steel plants:

. . . A market or minimill is a concept . . . the concept of a steel producing facility with relatively low capital costs.  It is modern and efficient equipment that produces sufficient hot tons to be rolled into a tailor-made finished product and then sold in a closely defined market with maximum flexibility in pricing, production, equipment advances, and particularly total employee participation.  Each criterion is important but overall flexibility is the key.

On the last and lost chance -

The range of choices available [to Big Steel] in the 1950s and 1960s was unusually wide. Very rarely does an industry such as steel confront a unique moment where its cash flow is high enough and spending constraints low enough to allow long term flexibility in planning.  Steel was responsible for a large proportion of both the country's gross domestic product and of its exports.  Because of its critical role in American  industry, steel's decisions had tremendous ramifications for the entire economy. Any decision by steel executives which altered the industry's economic performance would impose a new reality on a vast constituency.

As early as 1950 and as late as 1965 steel companies could have invested billions of dollars to upgrade their facilities with technology that would enable them to compete internationally for the foreseeable future.  Since they were relatively healthy well into the 1960s they could have accomplished this largely without investment capital, and avoided an unreasonable debt burden which would them beholden to lenders.  

But consider the downside of this strategy.  As early as the mid 1950s, American steel confronted declining levels of plant utilization, declining stock values, and an acquisitive minded market.  Steel executives needed to increase dividends and overall profitability in the short term.  Modernization would produce the reverse effect, since the costs of reconstruction would mean reduced profits and dividends.  Price rises and diversification into quickly profitable industries, mainly through acquisitions, therefore became the favored course, with modernization postponed until the threat to stock prices was passed. But the recession overtook them before modernization could be seriously attempted.

The decision not to invest early enough in overhauling the technological foundation of American steel changed the environment in which steel operated; the entire topology within which they had to act was consequently altered.  Options which had been available in the 1960s had disappeared; the option which had seemed most unnecessary and most unlikely -- getting out of steel -- was the one they were forced to pursue.

In the 1950s and 1960s, therefore, power was inherent in the steel industry.  By the 1980s, the steel industry seemed to lack any power at all. Through is actions, steel had eradicated its own influence over the social fabric. We believe that the power which was invested in steel was a result of a complex of historical forces.  Big steel did not become powerful through its own efforts --  it became powerful due to its necessity in a rapidly growing economy heavily reliant on basic manufacturing industries.  But its loss of power was a consequence of two decades of decision-making mistakes.

Steel was careless with its gift.

A few days ago, I wrote an article in somewhat "poetic" language (as it has been described) which placed commentaries on certain aspects of the Library / Google relationship. I have learned a couple of things from this post:  

One is that my attempt to soften (not weaken, soften) a message resulted in language which was instead used as a leaden mallet to suit the hands and intent of the wielder.  The second, related lesson is that there are a large number of actors who would not mind seeing Google lessened in the world.  Neither of these was my intent, and so let me be clearer on a few points:

1.  I think Google is a company to be admired.  For whatever weaknesses they possess, they have a grand vision and have superbly executed it, even at some risk to themselves (and it embodies both commercial and non-commercial vision).  Google is filled with a large number of supremely brilliant and nice people.  It has been my pleasure to become acquainted with many Google staff, managers and engineers (not that this is a non-intersecting set ... ), many of them before they set foot on the 'Plex, and many only as Googlers.

2.  I believe the Google Books effort is, in its broadest conception, a wonderful thing, and I support it wholeheartedly.  When this scale of mass digitization first emerged as a possibility, I wavered.  My opinion quickly settled on strong advocacy for the intellectual aims of the project, and I have been a consistent and solid proponent of it since before the time I was quoted in Kevin Kelley's article in the New York Times Magazine, "Scan this book!" 

3.  My primary intent in the post is to suggest disappointment with libraries (not with Google).  I felt that there was much to be gained -- and I still feel very strongly so -- from union, collaboration, and sharing among libraries of the immense issues raised by this effort.  There are some who feel that libraries have not worked together well on anything past traditional shared cataloging; I am not quite that pessimistic, but otoh, I think libraries (and universities generally) have absymally coordinated their activities, and failed to do so at times which were arguably critical for higher education's aims and goals.  

My charge is senstitive to several issues: libraries need to avoid certain types of coordinated action just like any other actors. Certainly early on in Google-Library negotiations, there was intense uncertainty and a complex and not yet settled swirling miasma of speculation about the actions of search engines, libraries, publishers, authors, the law, and opinion.  It is perhaps unfair for me to criticize deeply some of the decisions made then.  But nonetheless, the overall mantra: Libraries must collaborate amongst themselves - is a paramount one for me.   

I want to also acknowledge with sympathy the perspective that many publishers have that question why the library should receive any copy of a non-public domain work.  This is a valid debate, with many deep and profound questions about ownership, the nature of rights, and the ability to monetize commodities and actions.   However, I happen to represent libraries, not publishers. I have a bias, and it is my responsibility to reflect it as articulately as possible. 

Simply put: I wish (in hindsight) that libraries - both in the library program and without - had seized more initiative and not only recognized the earth-shaking change afoot, but grappled hard to be a fundamental and defining part of that conversation, by engaging directly with publishers (who should often be seen as intellectual compatriots and commercial partners, not adversaries), and by opening up discussion for ourselves and our publics on  critical issues of rights, privacy, and the nature of scholarship.  Some of these conversations have sprung up, largely within the library community, but I think that libraries have not been as much publicly engaged as they could have been, and I see this as a critical part of their role.  

4.  One of my comments in the earlier post suggested that I felt that discussions between Google and libraries could have proceeded along paths more open and straightforward.  I acknowledge that, and I rest with it, but I also want to observe that a conversation can be held over this issue. To some degree, the possibility of having that dialogue has opened, and I look forward to it.  I am also confident that this will be an open and honest conversation,  as a sharing of perspectives, aims, and understandings. 

5. I conclude by urging libraries: let us engage deeply in these issues, not only amongst ourselves, and search engines, but with publishers and authors.  Let us break open this dialogue to better understand among the cacophony of voices all of the richness of our different perspectives, and struggle through the differences more openly and straightforwardly.  Only through this is any emergent consensus possible.  The alternative is that new understandings will be imposed on us.  Let us instead build the house we shall live in, together. 

I take it as an assumption that increasingly books will be born digital, maintained digitally, and over time, due to enhanced content options and linkages, their preferred format will be digital.  Digital books will be available for (traditional ebook) device download; enhanced viewing and linking on networked screens (irrespective of glass size); print on demand; text to speech; and perhaps other services.   More and more publishers are beginning to populate their own repositories, and sell/rent space within their repositories to others.  Random, Harper, Macmillan, and Ingram (MyiLibrary) are all beginning to experiment with versions of these repository functions.  The better ones permit full text search across content.  

Even for publishers - because branding in publishing is fungible - search across content pools is probably a market-enhancing and revenue optimizing strategy.  It does not take too much hind-quarters resting-based cogitation to suspect that a publisher+search engine deal that would provide a discovery service across the great majority of texts, with user selection being driven back behind the walls represented by publishers' content repositories, is an appealing concept. 

In this environment, which for libraries reprises the transition to digital journals but at a later, more intensively networked stage in the evolution of online resources, this could spell Bad News in several ways.  Here are two.

Let's assume it is essentially game over for discovery, that user-based search happens primarily external to libraries, and that libraries are forced to license access to digital texts, with a possible option to buy at a stipend.  (I suspect this stipend would be high, because it would sunder the possibility of future revenues for the publishers or aggregators.)  This is arguably bad enough in itself for libraries, which have already given their libraries away to Google. How would libraries even be able to record that they have licensed this material?  What metadata would they obtain?  Would a traditional cataloging record even be useful?  As with journals, one might imagine that libraries have summary  metadata; OCLC has a complete record for the digital entity; and the library enters license terms into an electronic resource management system.  That's it.  So libraries hold print records in full detail, and stub records (with no full-text) of digital items that they probably do not own, and are paying to access.  If I were a user, I wouldn't see much reason to use library services to do discovery, although I still might visit the library to obtain materials.  

There is a flip side to licensed content, which is preservation.  Currently, preservation of digital journal content is, ur, uncertain.  Few libraries actually mount copies of journal content on local servers.  Many contracts have "perpetual access" clauses which are in varying degrees undefined, and largely untested; in the last 10 years we have not experienced a major organizational publishing failure with loss of content.  It will happen.  There is also a Mellon funded project called Portico, which is a sterling effort, but with incomplete coverage.  Finally, there is a distributed system called LOCKSS, which has signed up a certain number of publishers, but again is incomplete.  I should credit publishers, who recognize that libraries - silly they - put a value on safeguarding cultural and information assets belonging to the society at large.  Of course, libraries have to pay for that frivolity.  

Could these scenarios be replicated for books?  Perhaps.  LOCKSS could sign up publishers to safeguard books; a Portico v2 could archive digital monograph-based material; contracts could contain perpetual access clauses.  But would this work?  Digital books will consume additional resources for storage and maintenance, and the library community has not discussed at any length this additional burden.  I wonder if libraries have the scale - or could even support the scale in others - to generate a workable preservation system.  Portico works on a subscription model, for example, on both sides (publishers and libraries); free rider and sustainability problems remain largely unsolved.  Critically, subscriptions for preservation access,  combined with new licenses for access to digital books, will strap libraries far beyond their current liabilities.  Buying paper books, after all, is a one time cost with the addition of physical maintenance; this is a fraction of the likely life cost of maintaining digital licenses.  The curves could be purty durn scary.  

And then there are the standards issues.  I might assume - naively - that publishers would rally around a common format for digitally prepared and published books, such as the IDPF's standards, which have been embraced by actors such as Adobe's digitaleditions software.  However maintenance of digital standards is a shroud whipped away by the wind with its owner dancing wildly and frantically behind it.  The costs of format maintenance, conversions, and so forth - particularly as digital books are naturally enriched through linkages to other content and services - will be staggering if not contained or modestly defined.   

The movement of book content to publisher controlled digital repositories, with historical texts digitized by Google, and with discovery interfaces necessarily driven to search providers with significant scale, service linkages, and distributed indexing capabilities - the future of libraries in bookland could be rather bleak.   It would be appropriate, I think, for libraries to convene to discuss what the best approach might be to some of these scenarios - first among themselves only, and second approaching publishers directly, as both market-driven actors and historical partners.  After all, not even a failing publisher wants to be known, even as their death drives their collection to ashes, as a publisher who squandered the thoughts and discoveries of the men and women whose work they were privileged to make available to others, once upon a time.  

Can we say it was a mistake? 
For it was a mistake.

The goal is undeniably grand, and good. 
The means have left much to be desired. 

We poisoned our hand before we played it.  We were approached singly, charmed in confidence, the stranger was beguiling, and we embraced.  For the love of selfish confidence, we spoke neither our fortune nor our misgivings with our neighbors or our friends.  We felt special, invited to loud weddings on far away islands of adventure; in the quiet we may wonder if we were given broken jewelry. 

We could have chosen to question the worth of the marriage under the terms offered.  We might have chosen to hold off the sequined cloaks of confidence that wrapped our suitor’s gifts.  The glamour of it!  Yet we knew there were other wives already, and we might have joined them in union, consulted openly, wondered what would be best for all instead of merely ourselves; but our concerns were narrow.  In our selfishness, and wrapped in the fears we were given, we re-wrote and redefined our aims, misplaced our responsibilities, allowed the light and glory of the ideal to suffuse its glow over the bargain’s deficits. 

Even individually, we might have said, this is not right.   None of us spoke openly; nor did I.  Private conversations spiced with the pleasure of secrecy are to be judged harshly and are not worth their echoes. Misgivings spoken as whispers are hollow if they could have been voiced out loud, and were not.  These were not.  Contract and Confidence are brittle prisons but proved effective; the glassy bars were steel to us, when they might have been shattered.

We might have chosen to be forthright.  We could have dealt with the issues before us and engaged with the publishers and writers ourselves.  Re-written the script given us, spoken with truth and openness, called for action, seized the defining of the future, acknowledged the company of commerce in the aim but broken with the duplicity and the conniving.  Striven for leadership and not fallen into compliance with only shadows of struggle for our gain. 

Can we say it?  The deals are not fair.  We were taken advantage of.  We are asked to be grateful for something wondrous where we could have achieved more for ourselves and demanded more from others.  We let this happen and we should not have.   Now we must count on the beneficence of others. We need speak of the bitterness, laugh at our own stupidity, and move forward. 

Let us re-write the rules for the future. 

Regardless of what is given us, whenever we might feel grateful for the generosity of the enriched, we must still own the defining of our expectations.  We will write a new dialogue.  There are bigger things yet to lose, and the losing, we have learned, is easy.   We have demonstrated that to ourselves; never before having had to worry about loss, never knowing it was really possible, we did not question the taking.  

It is not too late for us – libraries, museums, and non-profit presses – to reassert the validity of our aims, our missions, and our expectations.  Sunderings will yet arise.  They might come from outside our domain; they might come from within.  We might welcome the opportunities their arrival will provide. 

Individual privileges do not sum to a union.  We must speak hard truths together, or we speak only words falling to dust.  It is time for us to gather.  Wives, and ladies waiting, we must gather and speak.

[FN. Please also see a follow up posting, ... "A Reprise with Clarity."  20070309]. 

Molly Ivins is gone.  If you don't know who she is, then shame on you.

She's been gone a little while now, but I write this on the occasion of receiving a special issue of The Texas Observer, dedicated to her life, her spirit, and her writing.  Everyone needs to get a copy of this issue.  Right now.  Git.  it.  now.  

I am a resident of Texas and I always will be wherever the hell I happen to be living.  It's hard to explain unless you've grown up breathing  that air too, but Molly was exactly what the best liberals in Texas are.  Full of hell, full of fun, with lots of humor and an inability to tell anything if it ain't a story. Telling of the struggles that people have to live, and making sure that everyone learned of it.  

I never met Molly, but the confederacy is close enough that friends and family have; have worked with her, at the Observer; my aunt, a deacon at First Methodist in Austin, was chosen by Molly's family to say the last words to say for anyone.

May she rest, but let us not.  

I was fortunate enough to give a talk today at the I-School, at UC Berkeley.  The talk is entitled "Architects of Collaboration" and it involves early thoughts on what I think libraries have been, and should be, in the coming years.  [Uploaded: 2007-03-05].

 I have been very appreciative of having the I-School close by.  They are deeply intertwined with Silicon Valley, are filled with deeply inquisitive and kind souls, and they have offered themselves in very generous ways as my second home.  One of the things I will be doing at the DLF is creating a series of Digital Contests that focus on difficult, perplexing, or critical pieces of infrastructure, tools, or services for digital libraries in information retrieval, and that are of a nature to serve as I-School MA projects.  "Contest" in the sense of challenge, wherein achieving either a solution or a hard fought defeat are rewarding of equally high stature.  I look forward to working with the Berkeley I-School, and its compatriot I-Schools across the country, to establish these Contests as a source of prove-your-mettle intellectual engagements.  

 Perhaps not surprisingly, although I sometimes feel dangerously out in front of more traditional librarians, I am rarely too radical for the reference and special collections librarians that come to my talks.  This is heartening; there is a range of opinions within our institutions.  With the impatient, I can say openly "I am very pessimistic for libraries: here are ways that we must radically re-shape ourselves, quickly, or the whirlwind will destroy us."   We may realize together that there may indeed be no place for libraries in the future, unless we craft something new and breathtaking, emboldened by the righteousness of our efforts,  taking risks worthy of the blessed.  Among my audience there was open talk of uneven support for new initiatives as mundane as blogs or wikis, discouraged by library or university management too fearful of consequences to embrace them.  How can we move forward when faced with a lack of complete understanding? 

In part, we must rally together with those who understand the priority change must have in our work.  And in part, we must all say together, "Change or be destroyed."  A chorus is after all louder than solitary voices.  As my host Michael Buckland observed, there is clarity in the counsel of our fundamentals: making information available, ensuring open access, assisting others in discovery, creating user-empowering tools and services.   

 These are some of the strengths of our discipline -- there are others -- and they form the center from which we must advance.  For this is no time for introspection or withdrawal, but rather the best time for advance and action.  We must speak with our voices in the cacophony based on our diverse skills, experience and intuition, raising our flags high and addressing ourselves not amongst our neighbors only, but also to fields with whom we may speak of shared troubles, change, and transformation.  We have things to speak to publishers, journalists, and scholars, and we have as many things to learn from them.  Advancement in our field is not about how many articles we write in Library Journal, but how much we have changed the world of publishing; how much we have understood and commented on participant journalism; contributed to new story-telling narratives in mobile gaming; built new visions of learning in Second Life.  These are the libraries of our future. 

Our world is bigger than we have ever imagined it to be, and we must lay claim to new territories unimagined, and these claimings will define us.  We must reward ourselves for our collaborations, for our wider understandings, and neglect our attention on what is merely trivial.  I care not for one metadata schema vs. another, faceted browse here or topic maps there: give me an opinion on the changing definitions of privacy; let me hear who speaks for enduring access to centuries of accumulated history digitized by online advertising firms; let us discuss the best engagement with publishers that will protect author rights but honor use, building economies from information flow, not guardianship. 

These are the issues for our time, and we will find ourselves amongst them!