3lib.org

3lib.org, pronounced “freelib”, is a project by the Open Library Society, supported by the Open Bibliographic Data Working Group of the Open Knowledge Foundation. We improve the access to and the use of freely available scholarly metadata. Most of the records discussed here are already used in the Society’s AuthorClaim service. We are making the records available here for others to use them. The idea is to build over time a large freely available dataset for academic performance evaluation and data mining.

The datasets available here only concern documents, and occasionally, series of documents. Other projects of the society curate datasets about non-documents. They can be obtained at their respective site.

For the AuthorClaim collection of author profiles, use ftp://ftp.authorclaim.org.
For the ARIW collection of institution names and identifiers, use ftp://ariw.org/lib.

For reasons of efficiency, all data is available via anonymous ftp. If you prefer an rsync solution, contact me, Thomas Krichel.

Most datasets are encoded in a simple XML format called the Academic Metadata Format (AMF).

Datasets by format

native and near-native AMF datasets

Some datasets in AMF are full datasets. They are either native in AMF or they are converted with the intention to create a full representation of the original.

personal data from AuthorClaim. This data is hosted by AuthorClaim.
RePEc data from RePEc, converted from the native ReDIF format. This data is hosted by RePEc.

converted AMF datasets

Other datasets in AMF are converted with the intention to feed the data into AuthorClaim. AuthorClaim only needs four basic data elements: the title, handle, author name expressions and link to a web page with further information. I could do a fuller conversion. But I won’t do that until somebody tells me s/he needs it.

agris data converted from the record in the AgMES format, obtained from FOA’s ftp server for AGRIS.
arXiv data converted from OAI DC, obtained from arXiv via OAI-PMH.
A small set of CrossRef data, from automated queries made to the sigg engine for queries pertaining to authors registered with AuthorClaim.
Current Index of Statistics data, from a dump of Current Index of Statistics in 2008‒09. We are told that the data is no longer updated.
DBLP data from the DBLP dataset, as far as it is available in the DBLP compressed XML file.
E-LIS data. This data comes from the OAI-DC data in the OAI-PMH interface of E-LIS.
Euclid data. This data comes from the OAI-DC data in the OAI-PMH interface of project Euclid.
HAL data. This data comes from the OAI-HAL data in the OAI-PMH interface of HAL. OAI-HAL data contains more records than the OAI-DC dataset.
Acta Cryst E publications data from Acta Cryst E, a publication by the International Union of Crystallography.
open library data, post 1965 only. This based on the JSON file dated 2011‒08. That JSON files also contains pre-1965 data. Current versions of the file are available at that used to be available at http://openlibrary.org/data. The post-1965 filtering is based on a parsing of date information (which is quite good), we just decided that anything before 1965 is too old. ☺
SOLIS data. This is converted from a dump of SOLIS in 2010.
SPIRES data. This is converted from the SPIRES collection.

non-AMF datasets in native form

Since the amount if information required for AuthorClaim records is small, you may want to get some other information from the original sources. Here we list those that we think we can make available.

At this time, non are available because OKFN don’t give use enough disk space.

Unavailable datasets

PubMed. AuthorClaim contains a copy of PubMed, converted the AMF, for the data elements that are required. This is not redistributed because of licensing restrictions.

Tools

Oddities and issues

Some identifiers use a start "info:3lib". This is illegal. Processing software should remove the "3". References to items with the "3" will not contain it.

Contact

If you want to know more, write to me Thomas Krichel, at krichel at openlib dot org stating the subject “3lib”.

This web/ftp space sponsored by the Open Bibliographic Data Working Group of the Open Knowledge Foundation. It was sponsored by the Bibliographic Knowledge Network. This page was last updated on 2011‐12‐20.