Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
Advertisement
You can also search for this author in PubMed Google Scholar
You have full access to this article via your institution.
Technologist Carl Malamud.Credit: Smita Sharma
In a project that could unlock the world’s research papers for easier computerized analysis, an American technologist has released online a gigantic index of the words and short phrases contained in more than 100 million journal articles — including many paywalled papers.
The catalogue, which was released on 7 October and is free to use, holds tables of more than 355 billion words and sentence fragments listed next to the articles in which they appear. It is an effort to help scientists use software to glean insights from published work even if they have no legal access to the underlying papers, says its creator, Carl Malamud. He released the files under the auspices of Public Resource, a non-profit corporation in Sebastopol, California, that he founded.
Malamud says that because his index doesn’t contain the full text of articles, but only sentence snippets up to five words long, releasing it does not breach publishers’ copyright restrictions on the reuse of paywalled articles. However, one legal expert says that publishers might question the legality of how Malamud created the index in the first place.
Some researchers who have had early access to the index say it’s a major development in helping them to search the literature with software — a procedure known as text mining. Gitanjali Yadav, a computational biologist at the University of Cambridge, UK, who studies volatile organic compounds emitted by plants, says she aims to comb through Malamud’s index to produce analyses of the plant chemicals described in the world’s research papers. “There is no way for me — or anyone else — to experimentally analyse or measure the chemical fingerprint of each and every plant species on Earth. Much of the information we seek already exists, in published literature,” she says. But researchers are restricted by lack of access to many papers, Yadav adds.
Malamud’s General Index, as he calls it, aims to address the problems faced by researchers such as Yadav. Computer scientists already text mine papers to build databases of genes, drugs and chemicals found in the literature, and to explore papers’ content faster than a human could read. But they often note that publishers ultimately control the speed and scope of their work, and that scientists are restricted to mining only open-access papers, or those articles they (or their institutions) have subscriptions to. Some publishers have said that researchers looking to mine the text of paywalled papers need their authorization.
And although free search engines such as Google Scholar have — with publishers’ agreement — indexed the text of paywalled literature, they only allow users to search with certain types of text queries, and restrict automated searching. That doesn’t allow large-scale computerized analysis using more specialized searches, Malamud says.
Malamud’s project is his latest venture in a career spent releasing locked-up information for free access online — often in the face of legal challenges. He originally focused on publishing government-produced legal and financial information. But more recently he has turned his attention to opening up the scientific literature.
He began with a project to allow scientists to text mine — but not read — a giant store of research papers he’s holding on a server in India; an idea he says he’s still working on. The General Index now allows anyone to mine scientific works, but it doesn’t have its own web-search portal, so if scientists want to search it, they will have to download its files and develop their own programs. Malamud is hoping that users will make any search engines they create available to others.
In its compressed format, the catalogue totals almost 5 terabytes, and then expands to 38 terabytes. As well as sentence fragments, the files also include tables of nearly 20 billion keywords in the literature, and tables of a paper’s title, authors and DOI (article identifier), so that users can track down a full paper if they have access to read it.
Michael Carroll, a legal researcher at the American University Washington College of Law in Washington DC, says that distributing the index should be legal worldwide because the files do not copy enough of an underlying article to infringe the publisher’s copyright — although laws vary by country. “Copyright does not protect facts and ideas, and these results would be treated as communication of facts derived from the analysis of the copyrighted articles,” he says.
The only legal question, Carroll adds, is whether Malamud’s obtaining and copying of the underlying papers was done without breaching publishers’ terms. Malamud says that he did have to get copies of the 107 million articles referenced in the index to create it; he declined to say how, but emphasizes that researchers will not have access to the full texts of the papers, which are stored in a secured, undisclosed location in the United States.
“I am very confident that what I’m doing is legal. We are not doing this to provoke a lawsuit, we are doing it to advance science,” he says.
Nature contacted six publishers about the General Index for this article: all but one declined to comment. In a statement, Springer Nature said that the company supports open-research initiatives that use technology and algorithms to meet the needs of researchers. “We have seen some initiatives run into trouble, however, when the necessary rights have not been secured to enable their sustainability,” the statement added. (Springer Nature publishes this journal; Nature’s news team is editorially independent of its publisher.)
Another legal researcher, Arul George Scaria at Delhi’s National Law University, says that any publishers that tried to use copyright laws to prevent researchers from using the General Index “would eventually be disappointed”. The release of the index, Scaria says, is a “major development for the wealth of information it has unlocked from those 107 million journal articles”.
doi: https://doi.org/10.1038/d41586-021-02895-8
News & Views 03 NOV 21
Article 03 NOV 21
Career Guide 13 OCT 21
News 04 NOV 21
Nature Index 27 OCT 21
Career Column 21 OCT 21
Correspondence 02 NOV 21
Correspondence 02 NOV 21
News 02 NOV 21
Max Perutz Labs
Vienna, Austria
Max Perutz Labs
Vienna, Austria
Ludwig Maximilians University of Munich (LMU)
Munich, Bavaria, Germany
Vienna University of Technology (TU Wien)
Vienna, Austria
You have full access to this article via your institution.
An essential round-up of science news, opinion and analysis, delivered to your inbox every weekday.
Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.
Advanced search
© 2021 Springer Nature Limited
