A five-pound sack

12 December 2014

Today I'm trying to put over twelve million items in a 16GB LibraryBox.

I mean, clearly that won't work. But what I want to do is put an index on the Box, so you can go through it and request an "inter-libary loan" wherein the next update goes out to the Internet Archive or Project Gutenberg or whatever and fetches it for the local collection. But daaaaaang, even the indexes are huge and unwieldly, and I haven't even tried to make them mobile-accessible yet.

Project Gutenberg's machine-readable index is a monster tree of RDF files. That's pretty parsable, and most of the information I can throw away (lots and lots of format metadata). Internet Archive is... a little more problematic.

Doesn't help, of course, that @textfiles just announced on Twitter that the Bay storm has knocked out IA power. So I'll stick with PG feeding the index for now. (It should help that @textfiles is also ~textfiles, but if you finger him, he's among the... do we have a name for people who haven't been back since the initial enthusiasm?)

Back to blog