Google Scholar is Filled with Junk Science

Google Scholar is the world’s largest and most-used academic search engine, yet it is increasingly becoming polluted with junk science, making it a potentially dangerous database for anyone doing serious research, from students to scientists.

The problem is that Google Scholar aims to be comprehensive, indexing articles from as many scholarly appearing journals as possible. On the surface, that goal seems noble, but a closer look reveals a major flaw in the strategy.

Because predatory publishers perform a fake or non-existent peer review, they have polluted the global scientific record with pseudo-science, a record that Google Scholar dutifully and perhaps blindly includes in its central index. Most predatory journals are included in Google Scholar. The database does not sufficiently screen for quality, in my opinion.

Google Scholar works well for known-item searches, for example, when you quickly need to locate a known article or a paper by a known author.

It performs poorly, on the other hand, at finding an article on a specific topic. It doesn’t use controlled vocabularies and includes junk science in its index. If you aren’t an expert, you are unable to separate out the junk science from the authentic science, and both are included one after another in Google Scholar search results. For those seeking the top scholarly literature on a given subject, the best resource is a focused, high-quality, curated database licensed by a library.

Junk Science

Predatory journals are also enabling the publication of much “activist science,” publishing articles that appear to be scientific but that could never pass peer review and be accepted and published in authentic journals. Activists publishing pseudo-scientific articles indexed in Google Scholar include:

o Those promoting hypotheses that mainstream science has found to be false, such as claiming that vaccines are the etiology of autism, or claiming that nuclear power is more dangerous than has been shown to be true

o Those denying hypotheses that mainstream science has found to be true, such as those denying that global warming is occurring

Additionally, people are using low quality scholarly journals to pursue personal theories or interests. These include:

o Those claiming far-fetched cosmological discoveries or theories that are impossible to prove or disprove

o Those publishing obvious pseudo-science, such as researchers documenting alien sightings

o Those using predatory journals to support a business interest, such as those promoting a new, unapproved medicine

o Those abusing the established taxonomy protocol to name species after themselves

The Future of Science

Science is cumulative, with new research building on findings already recorded in scholarly books and journals. When junk science is published bearing the imprimatur of science, later scientists may inadvertently use that work as the basis of their work, threatening the integrity of their results.

Google scholar does not sufficiently screen for quality and includes much junk science. To remain relevant and valuable. Google Scholar needs to limit the database to articles from authentic and respected scholarly publications and exclude articles from known publishers of junk science.

By: Jeffrey Beall
Follow on Twitter
Source: Scholarly Open Access

Comments:

T Anthony Howell says:

November 4, 2014 at 9:15 AM

While your mention of curated library databases are a solid recommendation, there are plenty of “junky papers” in many reputable “acceptance only” indeces, such as MEDLINE and PubMed Central.

I applaud Google Scholar’s ability to provide information to the public and scholarly community outside of these somewhat exclusive and manipulated database – not to mention the decades of skewing bio-medical evidence through the underreporting of negative trials and overreporting of positive trials.

Ultimately, the evaluation of the value and trustworthiness of the information needs to be performed by the consumer of that information.

conan the librarian says:

November 5, 2014 at 12:51 PM

While your point is well taken,the reputable sources you mention are at the very least suppose to adhere to some written acceptance policy. While Google Scholar is not ‘hindered’ with any written or professional policy – or do they have one?
In addition, while you cite the negatives of the ‘reputable’, do you want to muddy the waters by adding more negatives?
The consumer seeks these sources to become informed, albeit the info is not perfection, and not to evaluate the trustworthiness of sources.
Perfection will not be reached but if the attempt is not even made more negatives than positives will be present.

Nils says:

November 4, 2014 at 9:18 AM

I would never, ever, use Google Scholar to find articles on a specific topic – there are far better databases for that. The database is useful, though, to find articles by a specific author. It can also help finding articles citing one’s own work, that one might otherwise miss. However one has to keep in mind that the database includes many items which are not reviewed articles, such as proceedings, dissertations and slides of talks. Hence it would also be dangerous to use Google Scholar for bibliometric purposes.

Yurii says:

November 4, 2014 at 9:19 AM

Lack of proper curation in the majority of databases with automated or semi-automated information-gathering methods is a serious problem. Google Scholar is simply the most visible, and (hopefully) the least used in professional circles. It is more problematic when poorly curated information percolates into more specialized databases, that we use for hypothesis generation/testing. In the last couple of months I stumbled upon several rather egregious examples when poorly-vetted information affected interpretation of experimental results in several papers.

fredmurr says:

November 4, 2014 at 9:37 AM

This is a great blog, could you cite some examples of personal theories of the kind mentioned. I would love to showcase this to my students.

Joe Walder says:

November 4, 2014 at 9:51 AM

The librarians I deal with recommend using Google Scholar along with other databases. (I do research in the Earth sciences.) My anecdotal impression is that Google Scholar captures more articles relevant to me than do other search engines. The down side is that Google Scholar also sweeps up junk, by which I do not necessarily mean “junk science”, but simply irrelevant material. Keyword searches using any Google application are just prone to that.

My anecdotal impression is that Google Scholar gets many more “hits” on work by authors outside North America and Europe than do other search engines. Some of this stuff is junk, but some looks to me like decent work. For example, I have found good quality work published in a journal that is put out by an Indian professional society but published by a Western publisher. And I’ve found relevant papers in Chinese journals (I happen to be able to read some Chinese) published by universities or government labs. Commonly I’m just trying to “mine” papers like this for examples to add to some databases. And I’d say Google Scholar also does a much better job of finding papers in the Japanese literature (some of which is published in English) than do the other search engines that I’ve used.

Let’s be very careful not to conclude that scientists in countries that host junk journals are necessarily doing junk science. And let’s also be aware that seeing the name “Indian”, say, in the title of a journal does not mean it’s a junk journal.

Riaan Stals says:

November 4, 2014 at 11:08 AM

Hooray for this opinion piece!

Being a taxonomist by day, I have little problem finding [only] good stuff with Google Scholar. My institution, a national science council in South Africa, cannot afford an index-controlled, curated bibliographic database.

At night I am an avid reader of scientific matter in a wide spectrum, and it is then when the black spots in Google Scholar reveal themselves. Whereas taxonomy, by day, is rather clear-cut and professional experience allows the rapid detection of poor science (which has always been there in any case), it appears that the humanities and medical sciences served by Google Scholar suffer particularly badly from the inclusion of anything and everything that is published under the veil of science. And it is, of course, in my hobby interests where I do not have sufficient or professional insight to throw out the chaff.

At night I am in exactly the same situation as all tertiary students would be: misled by the Google Scholar stakes, unable to tell apart white and black and grey, but still entirely dependent upon its services.

Google Scholar is a blessing to the reading fraction of the developing world. But it may be time for much more stringent quality control. And can the service, perhaps with a controlled vocabulary and high curation costs, then still be offered free of charge?

J.J. says:

November 4, 2014 at 12:24 PM

The usual principle behind Google’s products is to index and search everything that is online. The only ‘curating’ is in the ordering of the results. I doubt that Google will make any efforts to stop returning bogus science.

It has always been clear to me that judging the quality of Google Scholar hits is on me. I think most sane researchers have realized this as well.

dzrlib says:

November 4, 2014 at 12:29 PM

In addition, Google Patents has a less than satisfactory search engine. It provides excellent PDF copies of US patents, but their scanning (unless recently improved) is like dragging the trash pond behind a suburban strip mall. Its numerous problems are well documented. http://patentlibrarian.blogspot.com/2008/09/comparison-of-free-patent-databases.html

MG says:

November 5, 2014 at 2:10 AM

As someone said before this is an interesting blog but I really would like to see some examples. I also noticed that there are a lot of non-academic articles in the subject-specific databases we usually recommend to our students, so it would be interesting to showcase the difference.

Neuroskeptic (@Neuro_Skeptic) says:

November 5, 2014 at 4:39 AM

Scholar would greatly benefit from the ability to order the search results by citations.

At a stroke this would make it easy to filter out 90% of the “junk”.

Implementing this would be very easy for Google so we can only assume that the lack of a citation ranking option is a deliberate choice, perhaps an attempt to avoid the “Matthew Effect” (cited papers get to the top of the ranking and get cited more.)

Neuroconscience says:

November 5, 2014 at 6:58 AM

While I understand the sentiment of this post, I think the message is wrong headed. As others have said, of course there needs to be some basic filtration method for citations. But at the same time, this points more to a need for stronger and more widespread post-publication peer review mechanisms. Pre-publication peer review is itself doing a pretty bad job of quality control; the answer isn’t a more restrictive publication mechanism but a better quality review mechanism.

Dave Langers says:

November 5, 2014 at 8:10 AM

I would precisely NOT use Google Scholar to search for particular papers by particular authors. If you have that much info, a database (mostly PubMed, in my case) works just fine.
Also, I don’t get the impression that Google Scholar is hugely infected with bogus results, or perhaps these are just so obvious that I ignore and hardly notice them. It is certainly not “unworkable”, and some common sense in interpreting the results of a search is always healthy (for curated databases too).
In contrast, what I find very useful about google is that its search engine is not overly sensitive to variations in spelling, or even synonyms perhaps, is my impression. It does not suddenly return no search results because of an ill-defined keyword. You can just type some stuff you may be interested in, and let google make sense of it. That makes it ideal – in my opinion – for searches where you are not entirely sure about what paper you are looking for. The more “exploratory” searches, say.
In summary, as with all literature, interpret sensibly, use the pros, circumvent the cons, and make the most of the tools you have available.

Dave Langers says:

November 5, 2014 at 8:19 AM

As an example, I keyed in a search query on what happens to be my field of interest: “methods to determine tonotopic organisation in human auditory cortex”. In PubMed I got only two papers, which were neither the best nor the most relevant. On Google Scholar I find an almost uncountable number of results, and the first few pages contain no junk whatsoever.
You might say that I don’ t formulate my query well enough for PubMed (to which I would not normally submit such a freeform expression, I admit), but that is precisely the advantage of Google Scholar: it doesn’t punish you for that.

Jeffrey Beall says:

November 5, 2014 at 9:03 AM

Excellent, then you can continue to use Google Scholar for your research.

Riaan Stals says:

November 5, 2014 at 10:26 AM

This is rather humorous. I entered “methods to determine tonotopic organisation in human auditory cortex” [note your own parentheses] into GS, with this result:

” Your search – … – did not match any articles.”

T Anthony Howell says:

November 5, 2014 at 8:24 AM

Google Scholar closely rivals specialized bibliographic databases.

See How readers discover content in scholarly journals – the results from a large scale reader survey – stm 2013—–in particular Slide 8 – Starting Points for Researching for Articles – Trend from 2005 to 2012.

yklein2 says:

November 5, 2014 at 10:53 AM

I use Google Scholar (the version licensed through my institution) on a daily basis. I use it in three ways: !, when I have basic information (author, year) and I need a complete cite; 2, when I need to find an e-journal copy for which our library has a license; and 3, to do key-word or topic searches Within my area of research, I take responsibility for separating the wheat from the chaff. (There is plenty of both.)

behalbiotech says:

November 5, 2014 at 10:13 AM

good topic, It’s funny to see some researchers even writing citations to their publications on the basis of google scholar. even claims were made that researchgate should show citations on the basis of google scholar. No doubt google scholar may give easy idea regarding papers published by a researcher but citations part is really bad. Further it brings out the papers/data published in predatory/fake journals, which is not reliable and should be avoided.

DS says:

November 5, 2014 at 10:49 AM

Peer-reviewed is not equivalent to good science. The author of this rant is making a big mistake. If you can’t tell the difference between junk or questionable science in a particular field then you are not an expert in that field and should stay away from that literature. Open post-publication review is the future.

Emmanuel says:

November 5, 2014 at 4:46 PM

I’m proudly Nigerian working in Ghana. To us here, Google Scholar is a blessing. It does not matter if some persons feel it “promotes” their so-called junk science! Let us be realistic here because not everything that is from Nigeria, India or Pakistan is a junk. Of course not everything from the United States, Canada, Australia and Western Europe is genuine. Please be fairer in your judgement. I rest my case.

Google – not! says:

November 5, 2014 at 7:56 PM

This is an important issue. I don’t know a single scientist who doesn’t use Google Scholar. When ding a scan for the latest literature, for example, to develop the introduction or discussion of a paper, most scientists would look towards a few main data-bases, most likely Thomson Reuters’ Web of Science, Elsevier’s Scopus/sciencedirect, Springer Science + Business Media’s SpringerLink, and possibly Wiley-Blackwell’s Wiley Online and even Taylor and Francis Online. To a much lower extent, DeGruyter Online, which has only recently started to make gains. But the spiders at Google and to a less extent those at Yahoo, are exceptional at tracking PDF files. In some cases, publishers may, for an additional fee, get their websites and files read more effectively by Google. This may be the case of Academic Journals, a Nigerian-based OA publisher listed on Beall’s list. The bottom line is, during a search for academic articles, most scientists will at least browse through Google Scholar, so this discussion is important, and relevant, even if many here offering comment appear to show vastly contrasting differences of opinion. One can also state that Google Scholar is important, precisely because it has the powerful tools to accumulate so much information, junk or not. So, I do agree that the fine line between quality and bad quality is beginning to become increasingly thinner, and that rubbish is mixed with great science when doing a search on Google Scholar, but perhaps, rather than lauding it as an academic data-base, perhaps we could be using it to our advantage instead. In effect, for the post-publication peer reviewer, it can be a most effective tool of detection. For example, it can allow us to identify the pseudo-academics that are trying to pass off their faulty work as veritable science. It can allow us to detect fraud, duplications, and plagiarism, simply because all the data is accumulated. And it allows the articles of the main publishers to be identified alongside the less famous ones, albeit 10 pages down the search list. So, I can appreciate Beall’s perspective, and I am also extremely concerned about the fact that Google Scholar could be giving the frauds and the pseudo-academics a pulpit from which to preach their fraud and rubbish, but it also gives those in pursuit of justice the perfect tool to catch, and expose, the very same frauds. It is the classical two-edged knife. However, for better or for worse, given its current positive aspects, which trump the negative ones, we would be worse off without Google Scholar. Let the predators use this tool, because it will be useful to bring about their downfall, too, in the long run.

Charles says:

November 13, 2014 at 5:15 AM

@ Google-not:

Totally agree with your perspective. Google scholar is more beneficial to science and academia as a whole than harmful. Although my institution provides access to Web of Science, its hardly ever the starting point of literature search. By comparison, Google scholar is extremely easy to use and is a great tool for conducting searches on topics about which you have little information at the initial stage. Additionally, for lots of researchers in institutions who can’t afford access to WoK/WoS, Google Scholar is an absolute lifesaver. I think researchers out there on a mission to save science from the junk by imposing stricter censorship are just paranoid. Let science and its method speak for itself!

Yamoato Shu says:

November 5, 2014 at 7:47 PM

There are many junk papers in SCI journals as well. Besides, how can you judge a new hypothesis by the one’s current knowledge?

Many Nobel winners did not get acknowledgment in the beginning, even the “good” journals did not accept their manuscripts. After hundred years their thesis then finally get proofed.

Google scholar is not doing anything wrong. If it is not there I assume our daily research progress would back to your slow age – walk 20 minutes to the library, use indexing card to find papers and read 10 papers in a whole day. I cannot go back to that old lifestyle. So I look forward and I face and accept the changes.

If you criticize on Google scholar, you will also criticize on the Internet, the Mobile techs, the Robot, the developing countries, all the new things which are changing our life. I agree that there are junk papers on Google scholar, but I am a researcher, I am a human, I can identify what is good or bad for me. It is not because someone “presents” me so many foods so that I have to eat them all. I can choose. Google is just like a marketplace, it collects and gives you choice, it even sort out a lot for you, what you need it to pick up what you want. This is not wrong with the technology.

I used to read junk papers in the library too. Now I prefer to read more junk papers on Google Scholar, as I get more good papers at the same time.

Google vs Google Scholar says:

November 6, 2014 at 10:11 AM

I think you make an excellent point. The same could also be said about Elsevier’s Scopus, which has started, in the past year or two, to attract several “flies” (aka the predatory OA publishers), to its collection. The fact that they get included on these supposedly prestigious data-bases indicates that their selection systems are highly porous, and unspecific. It also pollutes the data-base to the level that the legitimate journals lose their legitimate status. It is the urgent race to stay ahead of the pack and to beat the competition. Corporations like Reed-Elsevier and Thomson Reuters need to constantly show their share-holders that they are growing and that profits are increasing. When that upward trend drops, expect the downfall of two of publishing’s greatest (not in terms of quality, but in terms of size) establishments. It is their greed that will ultimately be their downfall. Ultimately, it is the pool of scientists who support these publishers (OA or not) or corporations who are to blame. That said, I put Beall’s hypothesis to a test, and considering that Google and Yahoo share similar style of spiders, I used Yahoo as my outlier. I entered the term “medicinal plant micropropagation PDF” (without the parentheses, of course) as my string of key words because I am interested in the tissue culture of medicinal plants and because I wanted to find some free papers on this topic, i.e., open access. On Yahoo, 3 of the 10 top hits were from publishers listed on Beall’s list, and only 1/10 each were from Springer and Elsevier, i.e., the predators trumped the traditional STM publishers. On Google (English, US), the situation was even worse. The very first hit was from a predatory OA journal [1], there were at least another 2 papers from publisher’s on Beall’s list, and another 3-4 PDF files or papers were of suspect quality, including one on ResearchGate, which I am also highly critical of. Only one of the first 10 hits was from a Springer journal. Finally, I checked Google Scholar (English, US), and found that all papers on the top page were from main-stream publishers, primarily Elsevier, Springer ad Wiley. I think this difference is important to note because many scientists do not necessarily use Google Scholar, they use Google. No matter what your opinion is on this topic, the fact that we are having this discussion is important enough.

GrZ says:

November 6, 2014 at 4:12 AM

With all the due respect, scientific publication is increasingly corrupted enterprise from the highest to lowest profile publishers. The ultimate goal of the biggest, richest,…the most “trustful” publishers is money, nothing else.
Except for health repercussion issues, science should be open, free, accessible, publishable, readable, available…to all.
In other word, authors should be able to publish their thoughts, hypothesis, reflections etc. to advance science without the hindrance of the so-called “peer-review”.
Again, peer-review should eventually be required only for papers dealing with ‘major’ or public health issues’, but not for other disciplines where contributions can help advancing knowledge.
The peer-review is becoming a dictatorship wall.
Peer-review is mostly harmful for science. It does more harmful than good.
How would it be acceptable that only 2-3 people (reviewers) out of millions others would determine the suitability or unsuitability of a paper?
If 2, 3, 4, 5, 6,…or 10 reviewers may find a paper unsuitable, there may be other 50, 100, 1000 people who might find it suitable.

Science or knowledge is not exclusive to a few people only and 2-3 reviewers are not a gauge of absolute reliability, otherwise how to explain the fact that many, many papers rejected by a given journal are accepted by another? This is the case of most scientific publications, which illustrate the ridiculous and contradictory character of the scientific publication industry.
If reviewers A, B in journal C have rejected a paper but reviewers X, Y in journal Z have accept it, doesn’t this mean that the peer-review is merely a matter of personal appreciation? It is not more than a subjective process.
People are different, their judgments follow. It is as simple as this.

Darwin, Mendel, Einstein,…etc. were publishing their works without peer-reviews.

Jeffrey Beall says:

November 6, 2014 at 4:40 AM

Wrong, wrong, wrong.

GrZ says:

November 6, 2014 at 4:21 AM

I’d also simply add that many journals have rejected many papers that have been proven to be great papers some years later; a proof that the peer-review process is biased.

Robin P Clarke says:

March 31, 2015 at 3:25 PM

The problem Jeffrey is that you start from the assumption that the whole system of selection of “proper” experts and expertise and so on is actually funtioning validly. There’s huge evidence that that is very far from the case (at least in medical matters), but if you insist on first assuming that it is, then inevitably you end up “proving” it true after all. You are too preoccupied with supposed indicators of genuineness (such as what university) when ultimately there are no really unscammable alternatives to just evaluating the particular document per se. You reckon to categorise some oa journals as “junk” and yet there has been plenty of real junk articles in the supposedly most esteemed journals. As per the words of Dr. Marcia Angell, the editor of the New England Journal of Medicine for 20 years:
“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.” (NY Review of Books, January 15, 2009)
And I could go on and on with more here.

researchontherocks says:

November 7, 2014 at 11:27 AM

I am a bit confused about the problem the author describes here. Scholar does give you the name of the journal/conference. You might avoid reading the paper if it is not published in so called “reputed” venue. What Google offers is high recall on natural language queries / keywords. It is up to the researcher to filter out the garbage (especially when you define garbage as “non reputed venue”).

Joe says:

November 7, 2014 at 3:47 PM

I do research in a US government agency rather than a university. We undergo a professional peer review every 4 years, at which time the review panel may recommend promotion (increase in civil service grade). I’ve served on review panels, and there is much more attention given to quality than quantity. My impression of the academic setting is different: bureaucratic bean counters just tally number of publications and amount of grants received. With this pressure on academicians to increase the number of their published papers, is it really any surprise that shoddy, pay-to-publish journals exist?

Unless there’s a radical change in the way academicians are evaluated by their universities, the shoddy, unethical, borderline criminal activities of “predatory” publishers are going to thrive.

JW says:

November 8, 2014 at 6:55 AM

Google Scholar indexes almost everything that Web and Science (as well as PubMed, Scopus, etc.) also indexes. So, Web of Science is merely a subset of Google Scholar. Web of Science offers almost nothing meaningful if your aim is to retrieve all relevant literature.

It is true that Google Scholar indexes quite some junk. But who decides what is junk science and what is not? Should this be the job of Google Scholar? The claim that Google Scholar is a “potentially dangerous database for anyone doing serious research” seems invalid to me. Why should Google provide the filter for the serious scientists? Can’t the serious scientists judge for themselves which paper is valid, and which is not? Isn’t this actually the job/obligation of scientists? If scientists want Google Scholar to filter out the junk for them, it means they apparently cannot decide for themselves which papers to include in their literature review.

If your goal is to do a literature review or meta-analysis, then the grey literature (e.g., government reports, conference proceedings, PhD/MSc student theses) need to be retrieved as well, otherwise you are contributing to publication bias. Web of Science misses all of this. People who only rely on Web of Science (PubMed etc) wrongly assume it is sufficient to rely on a subset of the overall scientific enterprise

It should be noted that Google Scholar indexes papers that appear online within a day or three; other indexing services take months or more.

Typhoon says:

November 8, 2014 at 6:18 PM

“Those denying hypotheses that mainstream science has found to be true, such as those denying that global warming is occurring”

Given the current lack of agreement between empirical observations and predictions by GCM simulations, leading to what has been called “global warming pause/hiatus” puzzle of the past 16 years or so, the sensitivity of the climate to CO2 levels is still very much an open question. The “global warming pause” problem is not fringe science, but has been discussed in such mainstream journals such as Nature.

“Those claiming far-fetched cosmological discoveries or theories that are impossible to prove or disprove”

In HEP, string theory has been has yet to make a testable prediction despite some 30 years of intense effort by the best and brightest in the field with a massive number of publications in leading physics journals.

In HEP/astrophysics, the so-far untestable multiverse hypothesis is currently very much in fashion among some leaders in the field

The point is that there are ideas being promoted in mainstream science that it could be argued easily satisfy your fringe criteria.

Google Scholar is a very useful tool for quickly locating papers in a particular field. Even better, it’s a free service. As with any source of information, caveat emptor always applies.

Bob Brown says:

November 14, 2014 at 4:23 PM

According to Aristotle, Galileo and Newton were junk science; according to Newton, Einstein was junk science, according to Einstein, Planck, Schrödinger , and Heisenberg were junk science. Never disparage what you do not understand, simply because your mind and your favorite, feel-good theories cannot explain it. Science peers into the unknown, not the established. Dare to forge ahead, and get out of the comfort zone. There is no such thing as an absurd theory, just as long as no theories get set in stone as to become dogmas.

Kane says:

November 11, 2014 at 3:00 PM

Just wanted to say that Google Scholar’s citation metrics are actually extremely useful for researchers working in certain fields. For example, I am a computer science researcher, focusing on topics such as systems security, web security, malware and program analysis. This is somewhat of an exception within computer science where conference proceedings (with papers typically up to 15-pages in double-column format) are the primary medium for top-tier publications, while journals are often considered easy & cheap targets, essentially graveyards for papers rejected at conferences. When I was a PhD student at a certain top university in Boston, my advisor wouldn’t let me publish in journals at all, he said it would be detrimental to my reputation 🙂 Well, I have to agree he was a bit too opinionated on this, but you get the idea. Some leading conferences in our field include ACM CCS, IEEE S&P, USENIX Security.

The problem is, many of the well-established databases do not index such conference proceedings at all. For instance, I have a high-impact paper published in 2011, cited 102 times so far according to Google Scholar. Going through the list, I can find 5-6 articles that are low-quality theses, or plain junk, but the rest are reputable research papers. When, I search for the same paper in ISI, I get a total of 3 citations listed, all from journals I’ve never heard of. What?

It looks like Google Scholar is the only reliable source of citation counts for security researchers at the moment.

Tyson Adams says:

November 13, 2014 at 2:17 PM

I hadn’t really noticed the problem until recently when someone challenged me to produce combination vaccine safety studies. Searching Google Scholar was no better than the Google search I performed. Most of the top searches were anti-vaxxer nonsense, but Scholar is meant to be better than the algorithms that bring nonsense up the rankings in the main Google search.

dikranmarsupial says:

November 14, 2014 at 7:38 AM

Google scholar gives you the citations for the papers it indexes, which is an indication of a paper’s acceptance by the research community. That is more than sufficient to weed out the junk. If you can’t find papers supporting some idea that are well cited, that is an indication that it is probably not a very good theory.

Marcus O. Muench says:

November 20, 2014 at 11:04 AM

I mostly agree with you that the ranking of returns by Google scholar by citations helps to place the best at the top. However, I have always felt that this also has some downsides. First, any new good papers that have not yet recieved citations may get missed unless one takes the time to limit the search to recent publications. Second, this feeds into a system that once papers get some traction they get all the citations whereas other papers that are just as relevant get ignored. This tends to overlap with the ‘quality’ of the journals in which the papers are published (which is the point of the whole discussion here). Nonetheless, I know many examples of very similar papers, were one was published in a very good journal and the other in a lower-impact journal, were the high impact journal paper gets all the attention. In an ideal world, both papers would be discovered by online searches, read and cited as appropriate. However, this does not always happen. The quality of the journal can be more important than the quality of the work when it comes to recognition, and Google scholars ranking to just adds to this bias.

The only solution that I can think of it just going back to basic good practices as a scientist. Search multiple databases and read as many articles as you can so that your citations are as accurate and inclusive as you can make them. Reviewers should also review the citations for mistakes and omissions.