Methodology

Methodology and Policies

1. Use of the Site:

The information presented on our sites is intended for the public, policy makers, the media, scholars, scientists, and students. It is not represented as science, but rather a compilation of the best Pro or Con responses we can find on each site’s core question and related issues.

2. Questions:

Our Questions were and are developed by researching the topics, contacting related experts and organizations, and getting feedback/ideas from readers and are intended to thoroughly cover the core question and related issues.

We continually encourage readers to send us more or better questions and responses that are more specific, more direct, and/or have better sources than those currently posted.

When we can’t figure out whether to ask a question in the positive or the negative, we’ve decided to flip a coin.

For example, the Chairman and Managing Editor flipped a coin on Jan. 18, 2008 to determine whether our question to the presidential candidates should be “Are illegal immigrants a net loss to the U.S. economy?” or “Are illegal immigrants a net gain to the U.S. economy?” We used the same method to figure out whether to ask “Should the federal government continue to arrest people for using medical marijuana in states where medical marijuana use is legal?” or whether we should stop arresting them. (The results of the two coin tosses were in favor of “net gain” on immigration and “stop arresting” on medical marijuana.)

3. Diversity of Responses:

As we work for a diversity of responses, if we receive two similar responses to a question from two equally credible sources (for example, both 3-stars), then we will generally give the posting to the respondent who has fewer other responses posted on our sites.

We will normally post up to 5 Pro and 5 Con comments per question. All comments should be relevant, responsive, clear, concise, and sourced.

We have less interest in political or emotional statements, but we sometimes post them if we feel they are especially interesting, relevant, or thought provoking.

4. Theoretical Credibility Ranking:

Evaluating the credibility of one person’s statements is difficult if not impossible, especially without knowing each person’s background, training, education, or work.

We have therefore built credibility ranking charts for each ProCon.org website to help you differentiate the theoretical credibility of the various sources on our sites. Most of the differences on the charts are because of the differences in the subject matter.

For example:

The Medical Marijuana site lists physicians as 4-star “KEY EXPERTS,” while other sites may not even have the “Key Expert” category.

Ambassadors or diplomats to the Middle East might be 3-star “Experts” in the Israeli-Palestinian conflict, but probably wouldn’t be in Medical Marijuana.

Therefore, to better understand the theoretical credibility of the contributors to each site, visit each site’s credibility chart.

The Credibility Ranking Chart was designed as a simple way to gauge the theoretical credibility of the responses received, although we note that sometimes, for example, a 1-star source () may be better informed or more credible than a 5-star () source.

Arguably, the credibility chart has been the most difficult part of our presentation, in part because we have tried to make an easy to use and useful chart of subjective and complicated questions.

The chart is based upon several basic premises, one being that the courts and many people equate a level of education and knowledge with theoretical credibility. The other is that ProCon.org doesn’t have near the resources to make a complex evaluation of the credibility of each contributor to our websites.

Some have questioned, for instance, why we have chosen to give the credibility of government facts and statistics our highest theoretically rating of five stars.

Our thinking is that government facts and statistics are generally reliable. However, what is less reliable, hence our lower rating, is when Government personnel attempt to quote from such facts out of context, or worse when they misuse those facts on purpose or by accident.

For example, we generally would give our highest rating–five stars, to a government report saying that there have been 52,850 killed in auto accidents in a given time period, but we would consider it, less credible for a government employee to say in a speech, “Fifty-thousand people died last year in auto accidents.” The government employee would probably receive one, three, or possible four stars, depending on the person’s education and position.

We usually don’t rank organizations anything other than 1-star because they are often dynamic and composed of a myriad of influences making a ranking difficult and subjective.

When we do rank organizations higher — such as The New York Times (which we rate as 2-star) — and that organization prints an editorial, a quote from that editorial would carry a 2-star rating. However, if that same organization quotes an individual who we believe should be rated a 1, 3 or 4 star, that quote would carry that 1, 3 or 4 star rating.

Those who request their name be withheld from their responses will be posted as anonymous.

5. Quotes from Individuals & Organizations:

The quotes listed in our biographies are responses to the core question posted on each topic’s homepage. We contacted the individuals and organizations’ principals (or spokespersons) or found a quote in a mainstream publication that answers the question. The quotes are dated so the reader can put them in an historical context.
The website researcher and Managing Editor may accurately apply individual and organization’s statements to either the Pro side or the Con side, despite the author’s personal opinion.
When someone has changed his/her views on a topic, we will retain the quote used on the website and mention the position change [in a bracketed red Editor’s Note] along with the date and source reflecting the change. In some cases, we will post a pro and a con statement reflecting both the current and prior positions if they are both deemed especially relevant to the question being asked.
We welcome anyone to offer responses for either the Pro or Con side, or both.

6. Seeking Clarification:

When we receive responses to our questions, we generally send an e-mail clarifying the questions we understood they were responding to and may ask clarifying follow-up questions in an effort to keep the comments on point.

For example, if a respondent claims “studies” say something, we will ask that source to clarify which studies.

We may ask respondents pointed and leading questions, or play “devil’s advocate” in an effort to clarify or simplify the responses, Pro or Con.

7. Editing Quotes:

“Exact Quotes” are shown in italics within double quotation marks.

Omissions are shown with ellipses (…); words added to quotes, usually for context, are shown with square brackets […].

8. Time/Resources:

When someone writes to suggest that we review particular studies and/or articles, we will normally ask for the exact comments they think are suitable for specific questions so we can more efficiently find and review them.

9. Bias:

While those involved in the site have biases like most people, we work hard to keep bias off the site. If you perceive bias on the site, let us know so we can review it, and importantly, have a chance to correct any bias.

Additionally, we have made some graphic design decisions, such as pro v. con, red v. green, left column v. right column, and other distinctions. While some may consider that some of these choices suggest bias to one side or the other (for example, that a column on the left suggests the political “left”), any such bias is unintended.

10. Editorial Commentary on Accuracy / Honesty

Although many people and organizations are occasionally careless or intentionally misleading with facts, data, and communications, we at ProCon.org believe that government and their officials should always disseminate accurate and truthful information (with the arguable exception of real national security needs).

The site’s Editor will therefore comment when we believe that information put out by government officials or organizations is false, misleading, or erroneous.

We will also comment in those few cases when the contributor believes that the information they are contributing should appear to others as having the opposite view as a plain reading of the material would seem to suggest.

We don’t comment on information that may be slanted, biased, or not clearly valid.

All comments by the site’s Editor will be noted in red in this format:
[Editor’s Note: The government report is based on the testimony of a physician whose license to practice medicine was revoked within six months of the report’s release. 12/30/05]

11. Editorial Discretion:

We reserve editorial discretion in determining what materials are put on and taken off the site. All policies are subject to exceptions to be approved by the Managing Editor or the President.

12. Comments Invited

We invite comments on how to make our Methodology & Policies better, clearer, and more evenhanded.

[On January 29, 2007 ProCon.org decided not to continue its Distinguished Contributor Award program.]