Web behavior depends upon three interlocking communities: (1) authors whose web pages link to other pages; (2) search engines indexing and ranking those pages; and (3) information seekers whose queries and surfing reward authors and support search engines. Systematic suppression of controversial topics would indicate a flaw in the Web's ideology of openness and informativeness. This paper explores search engines' bias by asking: is a specific well known controversy revealed in a simple search? Experimental topics include: distance learning, Albert Einstein, St. John's Wort, female astronauts, and Belize. The experiments suggest simple queries tend to overly present the "sunny side" of these topics, with minimal controversy. A more "Objective Web" is analyzed where: (a) web page authors adopt research citation practices, (b) search engines balance organizational and analytic content, and (c) searchers practice more wary multi-searching.
The proposed United States Federal Communications Commission (FCC) ruling (FCC, 2003) opens the possibility for greater concentration of media ownership, possibly leading to fewer and more overlapping sources of information. How does the Web weigh into this picture? Harvard Digital Government researchers (Hindman and Cukier, 2003) argue the Internet is not really a counterbalance to traditional media bias because the Web itself is dominated by a few gatekeeper and winner-take-all link accumulating sites. Is the Web's apparent openness, diversity, and cost effective information dissemination an illusion?
Their empirical study (Hindman, 2003) investigates link characteristics of highly controversial topics such as gun control and abortion. Crawling 3 million pages confirmed a few sites accrue most of the inlinks, suggesting a normal surfer would be "pulled" toward those sites, rather than traveling to smaller and more diverse sites. Because search engines heavily use page links to rank results, searchers often start surfing at these same popular sites. This study also found website size and linking often mirrors well established traditional organizations, with only occasional Web-based newcomer groups. While a topic may be presented from many viewpoints and published cheaply, less popular sites are not necessarily easily accessible via search engines nor by surfing. Retrievability and visibility are quite different, but often confused in technical/political discussions.
Recognizing such bias is not an accusation of unfairness but rather is symptomatic of our growing understanding of complex Web technology operating at a scale of billions of pages and hundreds of millions of users. Although not precisely characterized, search engines collectively cover much less than the whole Web and individual search engines index different parts of the Web (Giles, 1999), further exposing alternative orderings by their ranking strategies. Indeed, those biases attract searchers to favor one engine over another.
Search engine bias has been mathematically characterized as deviation from norm or ideal (Mowshowitz and Kawaguchi, 2001), e.g. for benchmarking search engines on consumer decision-making queries such as "brand names of refrigerators". Such bias is termed indexical, versus concerns of propaganda and misinformation as content bias. Search engines are thus similar to media organizations, warning "too much consolidation [of either] limits the options for both information producers and information seekers". In practical terms, this study counters the widely held misconception of search engines behaving somewhat like objective and well-informed librarians (Sherman, 2002). Further political concerns (Introna and Nissenbaum, 2000) are voiced about the role of search engines in supporting or thwarting the inclusiveness ideology of the Web. A cause-effect description of search engine indexing practices illustrates certain political consequences. Indeed, the Web might be thought of as an economy of links (Walker, 2002), valued as both monetary and intellectual currency.
The commercial search engine optimization industry (WebMasterWorld, 2003) raises other bias issues tracking ever changing and unclear search engine strategies. Consumer surveys (ConsumerWebWatch, 2003) about paid placement in search results reinforces a common theme: the general public that is growing increasingly dependent upon search engine technology has relatively low understanding of how the technology works or their responsibilities for its proper use.
Beyond economics and politics, the Web also shares phenomena of complex, dynamic physical systems. Recent books (Huberman, 2001; Barabasi,2002) explore system theories based on Web-wide regularities of structure, growth dynamics, and patterns of authorship and reading. These studies show the Web having an overall "power law" feature, where size of sites, number of incoming, and number of outgoing links all follow a non-modal distribution (i.e., no average behavior). These laws describe a world where a few X have high Y following a rapidly descending curve with most X having low Y (X is a number of sites or pages, Y is a number of links or measure of traffic). A few sites receive a disproportionate amount of traffic and links, while most are effectively islands. Figure 2 and Figure 3 illustrate power curves.
These phenomena are further associated with "winner take all" market forces where the more links a site has, the larger its share as more pages join the Web. However,"winner take all" applies to a lesser extent (Pennock et al, 2002) when a topic is more uniformly interesting or a community is more competitive. Usability expertise (Neilsen, 2003) contends no adverse effect on controversial topics from this power law structure because: (a) topics rarely have overlapping top sites; (b) even if the top sites accrue the bulk of the links, normal searchers will ask more specialized questions that surface smaller sites; and (c) cheap advertising greatly diversifies the Web by exposing new or smaller sites.
The "search advice"
literature (Tyburski,
2003; Sherman and
Price, 2002) bases key
rules on this knowledge about Web
behavior:
use multiple
engines
(because no engine indexes even half the Web, each has biases), use
specialized search engines (much Web content is invisible, hard
to index), authenticate and evaluate the quality of web sites (many
sites are only self-promotion),
don't depend solely on Web searching (fee-based services and
traditional library materials may be better catalogued with higher
quality). A
disturbing analysis of critical thinking deficiencies of
college
students (Graham
and Metaxis, 2003) shows specific
examples of potentially harmful Web
content bias tricking unwary searchers. Furthermore, it's not reliable
to issue
retractions or corrections (Adams, 2003)
if the modifications are not
hyperlinked or reported with the original erroneous article.
Thus, an understanding of controversial topics on the Web must address three interlocking communities: (1) web page authors who bestow links on other pages, (2) search engines that partially rely on links for crawling and indexing pages and their ranking of search results, and (3) searchers (humans) who query for topics then make browsing selections from results serving both engines and authors. Certainly, a search engine could have corporate political biases of its own, e.g. by not crawling designated web sites or by eliminating pages with particular keywords. However, a search engine basically reflects the biases of web page authors, their institutions and objectives. Surfers and searchers reinforce both authors and engines.
In order to analyze controversy, we must formulate a more precise question, develop an experimental approach to collect and analyze data, then interpret the results. For example, the study (Mowshowitz and Kawaguchi, 2001) presents a mathematical model of bias, collects search engine query data, and offers an interactive website for exploring search engine bias.
Our starting assumption is: a controversial subtopic is revealed or suppressed to the degree its URLs are recognizable in the query for the broad topic. This paper uses the hypothesis a given, well known specific controversy will NOT be revealed in the top search results. A priori, we would expect that among the 1000s of URLs on a topic, few subtopics will be exposed in top search results. However, using this hypothesis, we might find some surprising refutations where, indeed, controversial subtopics are well represented in top search results. Or we might find absolutely no evidence of controversy in top search results, suggesting systematic bias. Or we might find a mix for different kinds of topics and subtopics. Each outcome leads us to ask about factors suppressing or revealing the particular controversy, which provides the main contribution of this paper.
The outline of each topic experiment is:
As we browsed Web pages, we made three levels of judgment:
- Select a broad topic and define the simple, perhaps naive, query a searcher might ask to start learning about the subject matter.
- Collect the top 50 URLs from each of 3 popular modern search engines and the top 100 from 2 meta-searchers (send the query to multiple engines and collate the results).
- Identify a controversial subtopic, then define a query using specific keywords for the controversy, and query the same engines in the same amounts.
- Browse the search results to identify pages revealing the controversy , especially those deeply into the controversy itself.
- Select a 2nd, more factual, somewhat related subtopic with specific keywords, but (to reduce experimenter exhaustion) simply assume all URLs are relevant to the topic.
- Compare the results of the subtropics to the simple query and to each other to identify overlapping URLs revealing the underlying controversy.
Figure 1 pictorially shows the various subtopics, with images denoting the three categories of web pages relative to a controversy. The areas we're particularly interested in are those overlapping the controversy subtopic with the main theme (results of the simple query) and the factual query. Each area represents the results of queries in the combined results from all search engines
More details of the experimental methodology are discussed in Appendix A. The reader of this paper should think of these experiments as prolonged browsing sessions aided by a URL calculator, note taking, and an iterative process. Appendix B provides a brief list of revealing URLs as examples, tables and graphs for the counts and computed percentages of URLs in simple, controversial, and factual queries (merged from all engines). The interested reader is invited to perform a search in their favorite engine for the simple query to get a feeling of the type of results expected (see Appendix A.2). However, search engine results are always changing and query settings may give different results from ours.
Table 1 provides a listing of the broad topics and identifies the selected controversial subtopic. Table 2 shows the factual subtopics and representative other subtopics, along with sample URLs from the 200+ URLs in each query group. Following are brief descriptions of each topic and subtopic for context with brief summaries of results and preliminary explanations.
| Broad Topic | Controversial Subtopic |
|---|---|
| Distance Learning | "Digital Diploma Mills", the trend toward commercialization of education, as characterized by technology historian David Noble. |
| Albert Einstein | Did Einstein's first wife, Mileva Maric, receive appropriate credit for scientific contributions to Einstein's early work? |
| Female Astronauts | Did the U.S. space program discriminate against the "Mercury 13", women pilots who passed preliminary astronaut screening tests? |
| St. John's Wort | Does this popular herbal remedy work effectively for depression and mood improvement? Recent medical trials (glaxo, Pfizer, lichter pharma) differ in their results, as reported by NCCAM, ACP-ASIM, and Nutrition Action Newsletter) |
| Belize | This small Central American country has a long, and ongoing, border dispute with Guatemala with deep historical roots in Spanish and British colonialism. |
| Broad Topic | Controversial | Related Factual | Other subtropics |
|---|---|---|---|
| Distance Learning (Sample) | "Digital Diploma Mills", David" Noble (Sample) | Formative Evaluation (Sample) | accreditation, faculty careers, intellectual property, web-based training, home schooling,, interactive media |
| Albert Einstein (Sample) | Einstein's first wife, Mileva Maric (Sample) | Einstein's pacifism and political activism (Sample) | religion, humanism, quotations, correspondence, relativity theory, cosmology, philosophy |
| Female Astronauts (Sample) | "Mercury 13" women pilots (Sample) | Astronaut selection criteria and procedures (Sample) | space shuttle, space station, Columbia, Challenger, Mir, mission control, astronaut training, astronaut qualification |
| St. John's Wort (Sample) | Effectiveness (Sample) | Dosage recommendations (Sample) | side effects, botany, depression, psychotherapy, vitamins, dietary supplements |
| Belize (Sample) | Border dispute with Guatemala (Sample). | Narcotics facts (compared with other countries) (Sample) | ecotourism, coral reefs, mayan ruins, British colonies, developing countries, technology |
Distance learning" (DL) refers to modes of instruction which are not same-time/same-place. The topic blends technology with instructional practice with the educational enterprise. The Web reflects these dimensions describing "how Web-based technology works and is used in DL", "how instructional practice benefits from technology", and "what (our) university/company's DL program offers". In 1997, a technology historian, David Noble, challenged the distance learning movement through a series of widely disseminated articles, inciting numerous debates. Faculty concerned by loss of control over their intellectual products, as well as contact with students. Active commercialization of DL was causing visible strife at certain universities among faculty, administration, and external interests. Noble coined the term "digital diploma mills" to refer to the new breed of university programs.
Our experiment found only 1 revealing page in the simple "distance learning" query: an annotated webliography by a library. The top search results are primarily organizational: DL trade associations, universities offering DL programs, as well as several explanatory pages on the nature of DL. Our 2nd subtopic is "formative evaluation", the standard terminology for educational methodology evaluation of technologies, materials, and experiences with them. We wondered if effectiveness and ongoing evaluation influenced David Noble or vice versa. However, the simple, controversy, and factual queries are nearly disjoint.
This topic might be described as suppressing controversy by organizational clout. The David Noble/Digital Diploma Mill subweb is linked around Noble's writings but perhaps its participating organizations are of less Web status (links, size) than the organizational ones. Readers interested in this particular controversy are also more likely to be inside the academic enterprise, rather than consumers of DL products. The DL "search engine personality" is to "help you find the DL program right for you", rather than "provide pro and con arguments for going the DL route". An information seeker might well assume distance learning is a fully accepted and well-paved alternative cheaper education, when issues of quality, accreditation, and durability of degrees are still unresolved. A few pages address other perceived negative aspects of DL, such as pure degree mills, faculty workload, rich vs poor, and intellectual property. This controversy seems to have abated since 1999.
Einstein's
biographies
discuss not only his role within normal science but also his life as a
political
activist immigrant during the W.W.II era. Because Einstein received
Nobel
recognition in 1921 and then
settled at the eminent Princeton Institute of Advanced
Study,
his opinions were widely sought and his pithy quotations broadly
disseminated. In the past two
decades, other facets of Einstein's personal
life have emerged, notably the extent to which his first wife, Mileva
Maric, might
have contributed
rather more than recognized in co-authorship of his
technical
work.
Numerous
family related facts appear frequently in biographies, including an
illegitimate
daughter lost to history, the failure of Maric's scientific career,
their
messy divorce (her settlement included his Nobel prize
money), and Einstein's estrangement from his children.
Compounding a benchmark of genius, Time Magazine's "Person of the
Century"
and widely quotable writings, there's no shortage of pages and web
sites about
Albert
Einstein.
We focus on the controversy: did Mileva Maric provide significant scientific support for Einstein's earlier work? Web authors reference books and articles about "our work", co-authorship listing on submitted papers, indications of Maric's superior math skills as well as her physics training. However, there is no definitive evidence one way or the other. This controversy was mentioned once in passing in a page from the simple "Albert Einstein" query. Maric is mentioned as student, sometimes colleague, but mostly unhappy wife and mother abandoned for a 2nd wife and family. Our factual topic is Einstein's pacifism, as expressed in this public writing and speaking. Pacifism, shows a greater number of its revealing URLs in the simple query with some Maric URLs are revealed in the pacifism query, primarily in pages with more substantive biographies or personality analyses.
One explanation for the absence of controversial content is simple: Albert Einstein is a prime target for term papers, an obligatory biography for science sites, and a fertile source of quotations. These top web pages are mostly bland, featuring the photogenic Einstein. Such pages essentially squeeze out not only controversy, but any deeper discussions of Einstein's life and work. The Web material about Einstein is somewhat organizational, with a few museums and affinity groups, plus the Einstein Page and a few other hubs. Many web pages are organizations named after Einstein (also influenced by a 2004 US presidential candidate, M.D. Einstein College of Medicine). Many pages contain little more Einstein content than a quotation, relevant or not. The Maric web has the unfortunate characteristic of lacking a central site for the various pages to link to.
Using our criteria
that a subtopic appear in the top results
for
the simple query, the Mileva Maric Einstein controversy is
suppressed by sheer volume of quotations, biographies, and passing
mentions.
Belize is a recently independent Central American country known for its barrier reef, jungles, and Mayan ruins. A former British colony, Belize has been caught in a long-standing (since 1821) territorial dispute with Guatemala. Recent international mediation has reduced the level of dispute to "confidence building" negotiations and free trade agreements. The settlement of this dispute remains a major current political issue and a significant one in Belizean history. We chose this controversy because it relates to certain aspects of tourism (visiting the border area) and provides a major contrast with the North American experience. Our factual subject is "narcotics", a recent problem for Belize as its location provides a transfer point from South to North America. Both topics delve into how a small country with few military resources deals with international issues.
The experiment found about 7% of the simple "Belize" queries controversy-revealing, typically pages like the CIA or encyclopedia country fact sheet. One search engine surfaced a Belizean government website on Guatemalan relations and the dispute. 3% of the Belize narcotics controversy were revealing, mentioned on pages about modern Belizean politics and country facts. Other controversies/disputes include dam placement, logging, elections, and human rights.
A simple "Belize" query returns mostly tourism information and some general country information, with relatively little depth in historical, political, or current events. Major Belizean newspapers are online but much of the political and cultural material comes from either U.S. or British sources. Our experiments suggest this topic as another controversy suppressed by organizational clout. As a one-time tourist (mainly for the reefs and ruins, in the 1990 pre-Web era), this author is now disappointed not to have had this cultural and historical background. Trading off a few tourism pages for more history would have presented a better rounded portrayal of "Belize" as a starting query.
Female
astronauts
serve as significant role models for young women through their career
participation
in science and engineering, feats in space, and competition in a still
male
dominated field. Most young people go through a "how do I become an
astronaut?"
career exploration period. Especially when the next step for manned
space
exploration is such a long one, promoters such as Dr. Sally Ride and
Dr.
Mae Jemison play important roles in maintaining interest in
space. This is a popular Web topic.
But there are
other persistent
voices from the beginning of the space era. A little known story about
the
early days of astronaut selection has received increasing attention
through
various space and women's organizations and recent books. When the
Mercury
space program started around 1960, concerns grew about human
performance and payload size, including astronaut weight and food and
oxygen consumables. A group of highly experienced
women
aviators were considered as astronaut candidates. They passed extensive
physical
and psychological testing, involving secrecy and disruption of their
careers and personal lives. Their testing success brought the issue to
U.S.
Senate hearings involving Mercury 7 astronauts and
NASA
officials as well as the group leaders, pilots Jerrie Cobb and Janey
Hart.
NASA and Vice president Johnson put an end to the testing for various
reasons,
including a perceived need for test pilot experience and public
relations
fear at loss of a female crew member. The Russian space program first
launched a woman into space in 1963. The shuttle need for mission
scientists led to the launch of Dr. Sally Ride in 1983. Senator
Glenn's 2nd (geriatric)
flight also brought
public
attention to the earlier "Mercury 13", some still seeking a space
flights.
This controversy is definitely visible in the results for the simple query "female astronauts", with about 16% of the top results being revealing, including 9% deeply describing the Mercury 13 episode and its protagonists. Mercury 13 headlines appear across CNN and other news sites, at aviation and space web sites, and in women oriented venues, notably the National Organization for Women and the ninety-nines (women pilots). This has been an easy cause for such groups to promote and continue in memory of four lost female astronauts in two space shuttle accidents, and recognition from Commander Eileen Collins. Numerous biographies and speaking engagements by women astronauts, NASA promotion of its internal women corps, and "first women" histories account for the dominant theme of the search results. Another factor revealing the controversy is timeliness coinciding with 40 and 20 year commemorations of Russian and American flights. However, the odds of finding Mercury 13 information are significantly less, 4%, with the factual query "astronaut selection", which largely rehashes the modern day criteria enumerated on NASA's website.
We judge the Mercury 13 topic as a controversy significantly contributing to and not suppressed within the broad topic of "female astronauts".
St. John's Wort (SJW) is a long used herbal remedy for depression, stress, anxiety, and related maladies. In the U.S., as an off-the-shelf drug, its effectiveness, side effects, and proper dosages are not regulated nor subjected to much clinical study. With increasing market competition and medical concern, the U.S. National Institutes of Health, following British and German studies, began clinical trials around 2000. These trials raised questions: is St. John's Wort effective for what its sellers claim? why does it work? what are the proper dosages? how does its effectiveness compare with other prescription drugs? and what are the known side effects and drug interactions?
We chose the controversy about ongoing clinical trials under NIMH (National Institute of Mental Health), with participating drug companies Glaxo, Pfizer, and Lichter Pharma. A page was judged to deeply reveal the controversy if it prominently addressed these clinical trials. This criterion omitted many pages of other clinical trials related to side effects (often HIV related), general descriptions of the herb plant, and pricing and product information. Our 2nd factual subtopic is "St. John's Wort dosage", related, of course, to effectiveness (how much is needed? how much is too much?) and other issues of measuring the amount of key ingredients.
Nearly 30% of the simple"St. John's Wort" query results raised the controversy to some significant degree. For the factual topic "SJW Dosage" about 16% controversy-raising. However, the number of unique pages is much lower from duplicates or syndicated articles: an NIH NCCAM (Alternative Medicine) fact sheet, a Nutrition Action newsletter study description, and an NIH warning of SJW ineffectiveness for major (opposed to minor) depression.
The top search results give significant prominence to this controversy. An astute reader who bypasses the pure product pages will have a good chance of running onto a page reporting on clinical trials, for example, a newsletter's succinct description with links to the on-going studies. However, a superficial reading of the product pages and many of the older (pre 2000) description pages could well lead to a false sense of acceptance of SJW as harmless, a solid alternative to therapy and medicine, and a relatively cheap and standardized product.
Each controversy was well represented on the Web in search results posed with the right query. However, we do not believe many searchers would be exposed to the controversies by search or surfing alone, without off-Web experience. We knew of Mercury 13 through a recently released book, of David Noble by regular reading of sources publishing his work, of Mileva Maric by vague recollection of a book review or PBS program. We discovered the controversies for St. John's Wort and Belize as added experimental topics.
Female Astronauts/Mercury 13 and St. John's Wort/Effectiveness showed the most prominence of their controversies. Mercury 13 is a new and interesting aspect of a somewhat stale topic, stimulated by current and anniversaries of space events and recently published books. St. John's Wort is addressed by commonly disseminated information regarding alternative medicines, with the force of U. S. Government advisories. The Einstein/Maric controversy was swept away by biographies from standard science resource sites. The Belize-Guatemala border dispute was buried a level down in Belizean history, which is dominated by the country's tourism industry. The Digital Diploma Mill controversy was dominated by the organizational clout of distance learning organizations and by its fading presence in the past 5 years.
Our experimental results are, not surprisingly, inconclusive. If all five topics had strong representation of controversies then we might consider search engines performing much like well-informed, unbiased professional subject experts. If all five had completely suppressed controversy, we might suspect a conspiracy to present only the pleasing or commercial sides of search subjects. Some ambiguity in the results arises from the nature and limits of the experiments themselves, but several factors of Web behavior warrant further study.
Appendix A.2 summarizes some limits of our experiments. messiness handling duplicates, dead URLs, and dud pages; subjectivity judging pages deep or revealing; incompleteness in the evaluations; unpredictability of query syntax and semantics; time sensitivity; and bias of engine choices. We believe any single experimental limitation made a difference of only a few URLs, but not a significant change in results. The most serious difficulty with the experimental methodology is assuming controversy suppression/revelation correlated with a single, specific, well known subject query and a simple query for the broader topic. Further examination of additional queries or a vaguer notion of "controversy, altercation, disagreement, dispute" are called for, but the current limited experiments still tell us much about how controversy is handled by search engines.
An interesting question is whether there is a clearly superior search engine among the ones used in the experiment, notably is Google the clear winner? Table B.4 shows a surprising consistency among search engines in their ability to reveal controversy Web pages. Google had a slight edge and AllTheWeb a slight lag, but for our purposes the engines are roughly equal in results. Also consistent is the overlap among engines, about 33%.
| |
Distance Learning/ Diploma Mills | Albert Einstein/ Mileva Maric | Female Astronauts/ Mercury 13 | St. John's Wort Effectiveness | Belize/ Guatemala Border |
|---|---|---|---|---|---|
| Suppressing Factors | |||||
| Organizational Clout | sellers, services, trade associations | prolific reference and science sites | NASA and big space sites | store sites | tourism industry |
| Poorly organized | academic talk | no Maric web site | |||
| Duplication, junk | trivial biographies, quotes | lists of women firsts, space history | chains of stores | lots of hotels | |
| Analytic Web secondary | newsgroups, ezines low readership | few in-depth biographies | only dissertations and long histories | ||
| Revealing Factors | |||||
| Promotion | debates, articles online | letters, biographies | books, 99s, NOW | NIH, syndication | Govt. agencies |
| Social Relevance | college costs, professors' worry | Serb history, feminism | role models, activism | safety, HIV | colonialism, mediations |
| Timeliness | Web commercialism debates | Time person of century, ads, books | Columbia, 20/40 years,. Glenn flight | clinical trials in progress | referendum on settlement |
| Media interest | "diploma mill" exposes | 1st wives club, love letters | character profiles, pilot feats, | nutrition, medicine news | daily stories |
Two broad communities of interest on the Web might be grouped as "organizational" and "analytic". Organizational communities are actual or virtual: companies, universities, trade associations, consortia, alliances, government agencies. Distance learning purveyors, science reference sites, NASA, medical advisors, and tourist associations have off-Web organizations, resources, and motivation to be big Web players. The "analytic" Web consists of online full-text (or tables of contents) journals, technical reports and preprints, opinion pages, bibliographies, and pages of links to these. Analytic literature is widely available on the Web, but is not as extensively interlinked nor represented by large dominant web sites. The Organizational Web tends to link to key web sites to an extent search engine ranking strategies naturally assume these organizations are what searchers are looking for first and foremost. And they're usually right.
Mercury 13 illustrates a cohesive controversy, with book web pages, online stories by protagonists, and broad reach into news web sites. In contrast, Mileva Maric shows how a few pages without a core website or organizing group does not change the primary perspective of its topic. The Distance Learning controversy is represented by many long articles distributed at limited-interest older publications without continuing interaction with DL purveyors.
The Einstein subtopic shares with Mercury 13 not only feminist interest, but also the issue of revisionist history on the Web. Social change interests some communities, but repels or is ignored by others. New views on a topic must coexist with older views (and the links to them). Do search engines take topic currency into consideration when ranking results? How could they? Page modification date, content creation date, and whether content is current are all different. With few pages yet a decade old, is it possible Web content is becoming increasingly outdated? Suppose an irrefutable and important fact were discovered about Albert Einstein? Would most of the 100s of Web biographies be updated? Probably not, although a few sites would feel the obligation. And, many of the important articles, such as Time's Person of the Century, should remain unchanged in archives. Likewise with St. John's Wort, clinical trials in progress will publish results in 2005 to share web space with pre-trials articles written around 1997. The Albert Einstein topic pointed out the crucial role for new content venues, such as Wikipedia (an open encyclopedia) which reflect more recent information as well as modern interests, and may be more frequently updated than traditional references.
Search engine strategies seek to provide credible, relevant answers to general questions, such as the ones we've posed for our experiments. Ranking highly the most linked to or heavily keyworded or largest sites on a topic makes sense. But beyond the "round up the usual suspects" strategy, what other search results are relevant? Searcher and surfer habits seem to be rewarding "more of the same" organization sites rather than less popular content from the Analytic Web. On the other hand, are search engine results mostly a reflection of what Web authors write about and how they link to each other? What about those pages clearly revealing of controversy? Which might serve as "role models" for analytically oriented authors who seek to get their pages into the top rankings? Our experimental results suggest social factors, timeliness, and media interest strongly influence the prominence of controversies.
Knowing that pages with controversial content may be found sometimes, but not consistently, suggests alternative strategies toward more prominence of controversial and analytic pages.:
Web page authors might adopt more rigorous linking practices to (a) reach a higher standard of objectivity and/or (b) exert greater influence on search engine ranking. Academic researchers learn early in their careers to pass stringent peer reviewing by addressing not only the data favoring their approach but also opposing views, contradictions, and unknowns. this practice adopted by web page authors, how would Web linking change? Search engine optimization experts now practice link exchange in recognition links have economic value. We simulate a more objective scenario below.
Many factors work against this change in linking practice. Most organizations invest heavily to promote themselves and certainly won't link to competitors. This is characteristic of the Organizational Web, but now consider the Analytic Web. Many safe sites (e.g. Yahoo or Google directories) are common hubs for links, leaving the information seeker to sort out the link's actual target. Well established authorities, often traditional real life organizations, are other safe and useful targets. It takes some courage to link to a lesser known or "lone genius" site. Linking to a less stable site with dead links or unpredictably changing pages is risky, even if its content is well written and provocative. As in writing thorough research paper, considerable extra thought must go into creating accurate and balanced references (links). We're still looking for the opposing links for this paper's novel topic. However, linking only to popular, but possibly bland or less relevant, sites reinforces their status in search rankings over more specialized sites.
Search engines might alter their ranking strategies to provide more openness to the "Analytic Web" (controversy + data + evaluation + detail). Search engine ranking strategies remain proprietary, often mysterious, in flux, and a challenge to searchers. It would be unreasonable, and probably undesirable, to urge search engine companies to adopt a more objective standard, e.g. 25% of rankings include controversies, 25% traditional organization web sites, and 50% objective pages, even if technically feasible. However, ranking might include more predictable and recognizable distribution of the organizational (institutions vs individuals) or analytic (glossy vs technical) subwebs, e.g. Teoma's "Link Collections from Experts and Enthusiasts".
Search
trainers and professional
searchers might alter their search practices. This paper shows how
five
different topics only weakly present controversies at the naive query
level.
Advice to query specifically for controversy belongs with advice to be
wary
of deceptive Web pages. Librarians have
long
advised the Web is often not the best source of analytic
information. Search pros recognize Web content must be authenticated
and
checked
from different sources before accepted. Mistrusting search engine
results
falls into this category. of advice, as do countermeasures for
balancing organizational and analytic Web content.
Engines,
authors, and searchers might advance toward alternative paradigms
for working the Web. Today's metaphor is "search". Another metaphor (Gerhart, 2000) is "Collect", emphasizing broader
search, filtering, and ranking to produce well-rounded, multiple-use
collections of URLs rather than a single-answer URL. Another paradigm
is explicit markup on concepts driven by ontologies (concept
specifications) in the Semantic Web (Berners-Lee,
200x).
Here's how the
current
Web appears. Figure 2 shows the number of inlinks for sites of the
"distance
learning" data graphed against the number of sites with this number of
inlinks.
Note
the area near the Y axis with few sites having large numbers of
inlinks,
leading to a long tail of the curve with most sites having 0 or 1
inlinks.
Figure 3 shows a similar curve for pages of "Albert Einstein". These
curves
apply for links among pages in the respective collections not the
data
about the Web as a whole, but sharing the same distributions as
fractals (Huberman, 2001; Barabasi,
2002).
Let's idealize a situation where many pages are objective; examples are starred in Appendix B.2. A purely objective page might hold a carefully selected list of links or provide multiple sides of a topic. Of course, most pages would be pro or con some issue, usually favoring their organization's position. Now, suppose we "require" all page authors identify and link to opposing and contrasting views as well as such relevant objective pages. Examples of this linking strategy include: Distance Learning pages linking to pages refuting its effectiveness, experimental results, as well as the negative business side of degrees; Einstein biographies linking to Maric biographies and providing more depth on the controversy; Female Astronaut lists also showing the Mercury 13 pilots (many do), plus pages analyzing NASA's acceptance of women; alternative medicine stores linking to NIH advisories, and revisions to knowledge as clinical trials advance; and Belize histories linking more to the Guatemala dispute coverage from both sides of the border.
Below is the
resulting graph when 10% of 500
URLs are objective and web pages distribute their links 50% to
supporting, 25% to opposing
views, and 25% to objective pages. Numbers of links and links among
pages are randomly generated, constraining pages to either both agree
or both disagree. This new graph, Figure
4, shows a redistributed power curve with many links to the
objective pages (near the Y axis) and a more even distribution of # of
inlinks with more pages getting inlinks. Figure 5 shows more
evenness in distribution of inlinks with a higher percentage of
objective pages. Of course, this simulated Objective Web differs from
Figures 1 and 2 not only in link distribution policy but also in number
of links.
The results are not surprising but visually emphatic, contrasting today's Web with a hypothetical web having a different standard of linking behavior. This model confirms that a more objective policy would exhibit an extreme change in some Web characteristics. Further effects would then appear in search engines ranking highly by inlinks, shifting preference to objective pages. The requirement to link more extensively and non-exclusively to agreeing or popular pages would distribute links more evenly across all pages.
The experimental methodology
introduced in this paper, while needing refining, raises
important
questions about search engine bias. A given topic is analyzed by
a combination of specified
searches (simple, controversy, factual), rating pages for
possible or deep revelation of controversy, then interpreting the
suppressing or revealing factors. The
dilemma of controversies is that the searcher
beginning to explore a topic doesn't know the search terms to
investigate a controversy unless it is revealed with reasonable
visibility, e.g. not #879 in search results, nor buried 3 links away
from result #30. The unpredictability of search engine indexing
and ranking algorithms exacerbates the difficulty of performing good
experiments. Nevertheless, these experiments
provide many insights into the nature of searching and authoring
for the Web:
No, search engines do not conspire to suppress controversy, but their strategies do lead to organizationally dominated search results depriving searchers of a richer experience and, sometimes, of essential decision-making information. These experiments suggest bias questions to force thinking about the Web in a more, well, controversial, light.
This appendix describes the details of search engine
queries (engines used, specific search terms), an analysis of
limits of the
experimental methodology, and a summary of the twURL URL analyzer used in
the experiments.
Simple: distance learning, Albert Einstein, female astronauts, St. John's Wort, BelizeQueries posed as Advanced Search, using phrase (vs all words) where possible. 100 URLs drawn from the results, then reduced to the top 50. For multi-searchers, top 30 drawn from each engine for Copernic, all for Profusion, as far as the multi-searchers would go, then the top 100 used.
Controversy: distance learning and David Noble, distance learning and digital diploma mills; Mileva Maric, Albert Einstein and Mileva; Mercury 13, Jerrie Cobb; St. John's Wort and {Glaxo, Lichter Pharma, Pfizer, NCCAM, ASP-ASIM, Nutrition Alert}; Belize and Guatemala dispute
Factual: distance learning and formative evaluation; Albert Einstein and pacifism; astronaut selection; St. John's Wort and dosage; Belize and narcotics
In summary, we have an outline of an experimental methodology no more valid nor reliable than its underlying technologies. A valid experiment is not biased away from producing the data needed. These experiments addressed bias by targeting recognizably deeper, then revealing, then other web pages. A reliable experiment would produce the same results for all samples taken and with different experimenters. This should be addressed simply by replicated and more exhaustive experiments. Thus, the results of these experiments should not be overly interpreted until the experimental methodology is refined and variations performed to address the above problems.
twURL is a Windows desktop tool for analyzing URL collections. URLs are extracted from saved search engine pages (Google, Teoma, AlltheWeb, Profusion) or exported files (Copernic) and merged into a URL base. Views of the URLs then include: Internet domains, # links, #sources, ranking by query and engine, keywords from a topic-specific controlled vocabulary. Once organized into Views, URLs are browsed in order within sub views, e.g. all URLs with keyword "Mileva" or URLs with most links in. Browsing consists of automatically loading the web page into Microsoft Internet Explorer in the order given in the current View. As URLs are browsed decisions may be made and recorded to identify Deep, Relevant, and Other pages. The data was imported into Microsoft Excel for further analysis, e.g. decisions per engine, overlaps among engines, decisions per query, etc. HTML reports of the entire collection and the controversy pages are available at http://www.twurl.com/Cotroversy/Data.
This table shows the results of queries from all engines. Revealing web pages suggest the existence of controversy, among other topics. Deep web pages provide details in full force regarding the controversy. A searcher might well miss a Revealing URL description but would definitely be presented the controversy in a Deep web page.
| #URLs | Revealing | Deep | %Deep | %Deep or Revealing |
||
| DISTANCE LEARNING (totals) | 661 | 152 | 100 | 15 | 38 | |
| controversy david noble | 281 | 147 | 100 | 36 | 89 | |
| factual formative evaluation | 179 | 1 | 0 | 0 | <1 |
|
| simple distance learning | 212 | 1 |
0 | 0 | <1 |
|
| ALBERT EINSTEIN (totals) | 759 | 110 | 4 | |
8 |
20 |
| controversy mileva maric | 504 | 91 | 58 | |
12 | 30 |
| factual albert einstein pacifist | 180 | 8 | 2 | |
1 | .6 |
| simple albert einstein | 171 | 1 | 0 | |
0 | <1 |
| BELIZE(totals) | 443 | 89 | 22 | 5 | 25 | |
| controversy belize guatemala dispute | 144 | 84 | 23 | 16 | 74 | |
| factual belize narcotics | 167 | 5 | 0 | 0 | 3 | |
| simple belize | 169 | 10 | 1 | 1 | 6 | |
| FEMALE ASTRONAUTS (totals) | 663 | 85 | 90 | 14 | 26 | |
| controversy mercury 13, Jerrie Cobb | 210 | 66 | 88 | 41 | 76 | |
| factual astronaut selection | 167 | 5 | 3 | 1 | 4 | |
| simple female astronauts | 349 | 25 | 32 | 9 | 16 | |
| ST JOHN's WORT(totals) | 948 | 106 | 145 | 15 | 27 | |
| controversy St. John's Wort effectiveness | 648 | 87 | 135 | 21 | 35 | |
| factual St. John's Wort dosage | 221 | 16 | 19 | 9 | 16 | |
| simple St. John's Wort | 267 | 48 | 29 | 11 | 29 |
| Topic | %Deep (Range) |
%Revealing or Deep (Range) |
Revealing or Deep Low/High Engine |
%Overlapping search/multi-search (Range) |
|---|---|---|---|---|
| Distance Learning | 17-21 | 33-49 | google/teoma | 35-46/62-74 |
| Albert Einstein | 6-11 | 14-21 | alltheweb/google | 26-33/49-82 |
| Female Astronauts | 16-24 | 29-37 | copernic/google | 27-34/56-84 |
| St. John's Wort | 15-22 | 25-35 | alltheweb/teoma | 27-34/66-95 |
| Belize | 4-8 | 20-32 | alltheweb/google, teoma | 32-35/66-85 |