Library of Congress Plan To Archive Every Tweet Ever Made

The US Library of Congress and Twitter are making steps towards archiving all tweets that have been made on the microblogging service since its creation in 2006. After rumours saying that the Library of Congress had cooled on the project, spokeswoman Jennifer Gavin confirmed that the Library was still very much interested in the endeavour and that figuring out the logistics of such a massive undertaking is taking longer than anticipated.

“We began receiving the material, portions of it, last year. We got that system down. Now we’re getting it almost daily,” Gavin said. “And of course, as I think is obvious to anyone who follows Twitter, it has ended up being a very large amount of material.”

When the LoC first announced their intentions in 2010, a pedestrian 50 million tweets were being made per day by Twitter uses. In 2012, that number has risen to 400 million, and the Library’s engineers are working in tandem with Twitter to figure out how to properly archive literal petabytes of data, while Twitter is also working on ways of allowing users to download their tweeting history.

“We’re working on a tool to let users export all of their tweets,” Twitter CEO Dick Costolo said in a meeting with reporters and editors at the New York Times on Monday. “You’ll be able to download a file of them.”

Although such a service is being offered by a number of third party sites, Twitter are still figuring out how to address the issue comprehensively, and with 140 million users, demand is still growing. Costello said, “It’s two different search problems. It’s a different way of architecting search, going through all tweets of all time. You can’t just put three engineers on it.”

The Library of Congress and Twitter are being vigilant about privacy in their efforts. Mr. Costolo could not say when the company would make the personal search tool or export tool available for public use, but he said he was confident that at some point users would be able to look through their entire history of messages posted to Twitter.

While any potential LoC database will only being accessible from the library, with a valid library card; it shall not be made available online, with Twitter saying that tweets could be used only “for internal library use, for non-commercial research, public display by the library itself, and preservation.” There will even be an six-month embargo on fresh tweets.