From Clive Thompson of Wired Magazine
We’re often told that young people tend to be the most tech-savvy among us. But just how savvy are they? A group of researchers led by College of Charleston business professor Bing Pan tried to find out. Specifically, Pan wanted to know how skillful young folks are at online search. His team gathered a group of college students and asked them to look up the answers to a handful of questions. Perhaps not surprisingly, the students generally relied on the web pages at the top of Google’s results list.
But Pan pulled a trick: He changed the order of the results for some students. More often than not, those kids went for the bait and also used the (falsely) top-ranked pages. Pan grimly concluded that students aren’t assessing information sources on their own merit—they’re putting too much trust in the machine.
Now I’m a *huge* fan of Google Scholar. When I am writing a paper or grant, Google Scholar, my university library’s page, and my second brain DevonThink all stand at the ready. This is because I do much of my reading while in the process of writing. Writing exposes the holes in my understanding. So when I go to Google Scholar to find out what’s what, and I get the inevitable list of 3000 entries for, say, “thermal ecology ants”, which ones do I pay attention to?
It is not the first entry. I still, frankly, don’t grok the page rank in Google Scholar. But I can be reasonably certain that there will be entries on the first page worth paying attention to. I focus on three criteria in roughly the following order: who, title, and where.
- If I know and admire the work of the authors, I check it out (in this case, Cerda, Bestlemeyer, Porter, natch).
- If the title is interesting, I check it out (Thermal ecology of the neotropical army ant…), golly how did I *miss* that.
- If the work is published in a journal that I read regularly, I check it out (Ecology, JAE, Oecologia, Functional Ecology, natch).
To beginning grad students rules 2 and 3 work right off the bat. Rule 1, in contrast, takes some experience.
This is a huge thing. Science is now in an age of abundance, where the rules you use to filter through the literature glut are more important than ever.
What rules do all y’all use?
Survive and Thrive in Grad School 
The big advantage of ISI over google is that ISI can rank by citations, so the most cited articles float to the top. Of course, this puts newer articles at a disadvantage, but ordering by citation is a great way to get a very quick overview of a new topic almost instantly.
@ben weird, I was going to say that I find the number of citations from google scholar really informative, especially when I’m in unfamiliar territory.
I also love “cited by” to give me the bigger picture and lead me to articles I might otherwise have missed (checking who else cites articles that I rely on).
I second Lizzie’s comment–“Cited by” has opened whole worlds to me!
I tend to look much further down the list–I generally don’t feel like I’ve found the gold in Google Scholar until I’ve cleared a couple hundred hits–but then that may be a function of having research interests that have few results but share keywords with many other topics.
Your (and Pan’s) point is well-taken about over-reliance on “the machine,” but I do think it is to some degree appropriate. There are enough research scientists at Google that I have no doubt PageRank works somewhat differently on Scholar–that it has been fine-tuned to reflec academic research needs, rather than just being a publication- and patent-focused subset of Google itself. And while it would be unwise to put total faith in their algorithm (particularly when we can’t directly access it and assess its validity), it is one more piece of information that a buried scholar can use to sort and filter. Though not sufficient on its own, the fact that the top results have, by some third party’s measure(s), beaten out others is as good a way to pick a starting point as any other, at least in an unfamiliar topic or if one just needs a quick reference.
That said, I do like your criteria as well, although the very personal nature of them points to one other reason trusting GS’s PageRank may sometimes be useful: whatever biases are built in, they aren’t our own biases, which means they might break us out of our well-worn ruts through the literature.
scaffale vasagle
Google Scholar and the literature glut | Survive and Thrive in Grad School