What to know about teaching evaluation skills
- Evaluating sources is not a skill students will be taught in one week or in one course - you're laying the foundation for deeper skills that will be honed in a student's specific discipline.
- You're teaching students the skills they need to evaluate viewpoints no matter where they find them - i.e. from vetted sources and from Twitter.
- You will need to revisit the teaching of these skills throughout the semester.
- Evaluating sources is intuitive to you - that is not the case for early researchers.
Class Discussion: What does it mean to be credible?
First, I recognize this is a lot of text for you, but it reflects what I have learned in my assessment of student work. Each section gives you examples and discussion questions for class that I have developed in direct response to what I see in RHE 306 students struggling with in their work.
Authority / expertise: someone’s education or job title can give them authority.
- The authority has to be relevant to the controversy.
- Ex. Is a medical doctor an authority? On what? Is he an authority on fracking? Is he an authority on the health effects fracking has on water supplies? Maybe, what kind of doctor is he? Is he a podiatrist?
Authority and credibility of a venue
- Where do you go for reliable information? What news sources do you trust? When you have a question about something, how do you choose whom to ask?
- Ex. You want to know more about water treatment in Austin. How safe is our drinking water? How is water treated? What chemicals are used? Are the chemicals safe? Whom would you ask?
- City of Austin Utilities can give you water safety reports that tell you the levels of toxins or metals in your water. Where is this info published? (on their website for their customers)
- A chemist could tell you more about the chemicals used to treat the water. Where is this info published?
- A medical doctor could tell you more about the health effects of those chemicals. Where is this info published?
- An engineer could tell you more about how water is transported in a city and how the facilities to treat the water are constructed. Where is this info published?
- If you answered an academic journal, yes that is correct. That is where a lot of this specialized research is first published.
- If you answered a newspaper, magazine or website, that is also correct. Journalists and reporters write about this research from academic journals for the general public, usually in language non-specialists can understand.
- A note about television news and comedy news programs:
- Comedy news programs such as Daily Show and John Oliver:
- Who writes for these shows? What is their background? Are they journalists? Politicians? No, they are comedy writers. They may be very smart and informed, but they are comedy writers.
- What is the purpose of these shows? To satire, to parody, to entertain.
- News programs and news channels (Fox, MSNBC, Bill O’Reilly, Rachel Maddow, PBS Newshour)
- Who writes for these programs and who is the host? What is their expertise?
- What is the purpose of these shows and channels? How do they make their money? How does advertising affect how news is presented?
- Who is the audience? Can you tell if the writer(s) or host is speaking to a particular point of view? A particular audience? How can you tell?
Credibility of a source based upon evidence presented by the author
- When do you need evidence or proof to trust a source?
- When an author presents evidence, does he make it easy for you to trace from where he got that evidence?
- What kinds of evidence are most convincing?
- The type of evidence needs to be relevant to the controversy
- Ex. How likely is it that Austinites will vote in favor of increased train lines?
- Public opinion polls could help you gauge attitudes towards public transportation in Austin
- Past voting records: How did people vote on this issue in the past?
- Exit polls, surveys and focus groups could give you insight into why people voted the way they did or how they will vote in the future. Who conducts these polls and where do they publish the results?
- Demographics: Who rides the bus? Where do people in Austin work? How far is their commute? This could be Census or American Community Survey information
- Ex. The plastic bag ban in Austin. Is it effective?
- Effective at what? How would you measure effectiveness? Less bags being manufactured? Less bags in landfills? Less bags littering the environment?
- Someone who works in Austin parks could tell you if there are less bags littering parks, water, etc.
- Someone who works for city sanitation could tell you if there are less bags in Austin landfills.
- Someone who works at a plastic bag manufacturer or distributer could tell you if their business has been impacted by the ban.
- Where is the above information published? For city services, you could try websites.
- Companies often release financial reports to shareholders and employees.
- Typically, a newspaper or magazine will have articles written by journalists who have investigated and synthesized this data or conducted interviews and report it all to the general public.
Identifying Bias and Perspective (Summary of Primary / Secondary Text)
Now your students should be able to recognize a viewpoint when they see one. The next step is to develop their skill at identifying what the bias and perspective is, even when it is nuanced or more hidden, or written by someone who could be viewed as an expert.
Things to consider:
- Author: Who is the author? What is their stake in making this argument?
- Audience and Publication: Who is this written for? Where is this published? What does that tell you, if anything, about the perspective?
- Alternative views: How, if at all, does the author recognize and address opposing viewpoints?
- Language: Does the author use language that indicates one side of a viewpoint?
Resources, Activities and Assignments:
Activity 1: Name that Bias!
This exercise is designed to get students thinking about author, audience and publication. Because they aren't reading an entire piece, they won't be able to consider alternative views or language. Set up a scenario for your students and give them a series of examples of viewpoints. Ask them to identify the bias and discuss how they came to that conclusion. You may use the fracking scenario below (and in the Powerpoint) or create your own.
Scenario: The Texas legislature is considering legislation to regulate fracking.
- President, Plains Exploration and Production Company (producers of natural gas), writes a letter to the editor of the Houston Chronicle about the number of employees in his company working in hydraulic fracturing.
- Professor of Geology at the University of Austin, and member of the board of the Plains Exploration and Production Company, publishes an article in the newsletter of America's Gas Alliance about studies denying a link between fracking and groundwater contamination.
- Head of the West Texas Cotton Farmers Association, is interviewed on NPR about how the drought and fracking has impacted farming.
- A West Texas rancher, writes a piece for Texas Monthly about fracking and its effect on livestock
- Head of the Texas Medical Association, blogs about the impact of the chemicals used in fracking on human health.
Activity 2: This approach is designed to get students further than the first approach by analyzing author, audience and publication, alternative views and language, through close reading of a viewpoint. This essentially models the evaluation piece of RS 3 and gives them practice before they try it on their own in RS 3. You can model the process in class, give students a take home assignment or make it a discussion.
- Modeling in class: Use your own scenario or the one provided below to model the process in class. If you choose to create your own scenario, you may want to find articles from conservative and liberal publications on the same topic and compare them. A list of publications by political viewpoint is available here.
- Scenario: Use this October 2012 op-ed from the New York Times (via LexisNexis) about doctor assisted suicide ("Four Myths about Doctor Assisted Suicide" by Ezekiel J Emanuel. 10/28/12). This topic allows you to discuss language (doctor assisted suicide versus euthanasia) and stakeholder viewpoints (doctors, patients, families, the poor).
- Remember to tell your students that uncovering bias is not a checklist activity - don't worry if the publication does not suggest bias, for example.
- Walk through the article with them discussing bias as it relates to the author, publication and language.
- In addition to looking up authors and publications with Google, you can also use Academic Search Complete or LexisNexis to further investigate an author and publication with your students. Search the author to see what else they’ve written (that is also held within the database); check for a brief biography of the author in the record itself or in the full text of the article; click on the source title to find out more about that source.
- Does the author recognize opposing viewpoints in the article? Does he deride or mock his opponents? Does he use dismissive language?
- Think about value-laden terms: What do you associate with the word euthanasia? Doctor-assisted suicide? Right to Die? Death with Dignity?
- Assignments or discussion: Find a few longer viewpoint articles from magazines or trade journals and ask your class to read them. Then discuss the articles in class, being sure to talk about all the considerations listed above. You may structure this as group work, too, and have students discuss in groups and then report out in order to lead to a larger classroom conversation. You may also use one of the following worksheets to structure an in-class activity or as a take home assignment for an assigned article or one they find and plan to use for their paper.
Your students should now be able to recognize the bias in a viewpoint piece and can now consider additional criteria when evaluating the credibility of a viewpoint. Many students may have discussed evaluating web sites in high school and have learned a checklist to apply to sources. There are a lot of models out there that discuss expertise of the author, accuracy, relevance, purpose/bias, scope, etc. See, for example, the CRAAP Test. While your students may have some knowledge of applying these criteria to websites they used in high school, keep in mind that the web sites they used in high school were supposed to be factual/informational, not viewpoints. Since they are actually looking for biased sources now, they need to learn how to evaluate the credibility of a viewpoint and discover that not all viewpoints are created equal. To teach this, work from the bias checklist you created as a class (see above) and add to it.
Things to consider:
- expertise and representativeness of author - what is the author's stake in making this argument? do they have expertise in the area to write an informed viewpoint? is this viewpoint representative - can this person speak credibly for a larger group?
- audience and publication - Who is this written for? In what source was it published? In addition to considering what it tells you about the perspective, consider the reputation of the publication.
- accuracy - in addition to considering whether alternative views are recognized and addressed, consider how they are refuted and how the argument the author is making is supported. What kind of evidence do they use to refute the alternative views and support their own? Is this evidence credible and accurate? Is the evidence from biased sources?
- currency - is this information current enough for your research? Do they use current enough evidence to back up their arguments?
Resources, Activities and Assignments:
Activity 1: Set up a scenario for your students and give them a series of examples of viewpoints, just as you did in the Evaluating Bias section. Explain to them that you are asking them to identify the bias of these sources and also asking them to determine whether or not these sources are credible viewpoints. As a class, identify the bias and discuss the credibility of each source, using the discussion to build a list of criteria they can use (see Things to Consider above). Keep in mind that in addition to bias, they will assess the expertise/representativeness of the author and reputation of the publication, and possibly superficially discuss accuracy. If you are in a hands-on classroom or your students have mobile devices with them, let them look up/Google publications to help determine their reputation. Feel free to use the scenario below or make up your own. You can use a variety of types of sources (magazine articles, web sites, blogs, etc) or multiple examples of the same type of source (all newspapers or all web sites, for example).
Scenario: Pretend Austin is considering requiring all homes to install at least one solar panel by the year 2018. This requirement would be enforced by only allowing people to sell or rent their houses after 2018 if they've installed the solar panel. Citizens will be able to vote for or against this at the polls in November.
- The head of Austin Energy writes a letter to the Austin American Statesman arguing that requiring solar panels will improve air quality in central Texas by reducing Austin's dependence on coal-generated electricity, especially in the hot summer months when there are more likely to be ozone action days.
- The head of the Austin Chapter of the American Association of Realtors is interviewed by the Austin Business Journal. She argues that we should not require solar panels because it will negatively impact the Austin economy and housing market because people won't be able to afford the upgrade and thus won't be able to put their houses on the market.
- A commissioner at the Texas Commission on Environmental Quality writes an article for the Texas Observer arguing that instead of requiring solar panels, interested homeowners should get an incentive to install them, because while solar panels will be good for the environment, they won't make enough of a difference in air quality to justify an actual requirement.
- A homeowner and parent writes a letter to the Austin American Statesman arguing for the requirement, saying it will boost the local economy by putting more money in the pockets of homeowners (they installed solar panels and now sell energy back to the city) and it will improve the health of central Texans by reducing ozone action days (their children have asthma and cannot go outside on ozone action days).
Activity 2 : This exercise is designed to get students further than the first approach by analyzing author, publication, accuracy and currency through close reading of a viewpoint. Find a few longer viewpoint articles from magazines, newspapers, trade journals and/or on the Web, and ask your class to read them. Then discuss the articles in class, being sure to talk about all the considerations listed above. You may structure this as group work, too, and have students discuss in groups and then report out to lead to a larger classroom conversation. This essentially models the evaluation piece of RS 5 and gives them practice before they try it on their own.
Determining and Evaluating Context
Why is it important?
It may be useful to incorporate a discussion and exploration of context into your teaching of source evaluation. Analyzing context is another tool students can use to evaluate a viewpoint and may allow them to get a more nuanced understanding.
How to talk about it in class:
We have discussed how to evaluate viewpoints by looking at bias, perspective and credibilty. You can do further evaluation of a viewpoint by determining and evaluating its context. To do so, try the following:
- Look at where it appeared in the newspaper. If something is at the top of the front page versus buried further in the paper, this provides a clue as to the bias/outlook of the news outlet toward that story. This translates to online news outlets as well. Is it on the homepage or deeper in the site? If it is on the homepage, is it up near the top or down near the bottom?
- Look at how a story is covered across news sources. Find a story in one paper and see how it is covered in other papers on the same day.
- Look deeply into a specific news source to determine its bias. Even if a source doesn't overtly have a bias (for example, it isn't representing itself as a liberal or conservative source), you can still determine a lot about its ideology by examining the demographics of its readership.
- If it does not come from a newspaper, what kind of source does it come from? A blog? A organization's website? Does the author have to go through an editor before being published? Does the website have an agenda?
- Find out more about the context of a web site by seeing who links to it and who has commented on it. While the number of comments or links doesn't indicate credibility or trustworthiness, it does tell you about the audience: who is reading it, what their political leanings are, etc.
Resources, Activities and Assignments:
- Resources: See the Determining and Evaluating Context section
- Activity 1: Ask your students to evaluate the context of an article/source they identified. Give them the Determining and Evaluating Context Guide and ask them to use the most appropriate method for their source. They can either discuss it in class, or turn in the Determining and Evaluating Context Worksheet.
- Activity 2: Choose a source and have the class determine the context of the source using some or all of the tools described in the Determining and Evaluating Context section of Resources/Databases. You can demo any/all of these tools and lead a discussion of the source you are evaluating as a class while doing so. Ask your students to tell you how the context of the source impacted their evaluation of that source.
- Activity 3: Demo some or all of the tools described in the Determining and Evaluating Context section of Resources/Databases. You can also lead a discussion of a specific source in conjunction with the demo to show how you can use context to evaluate a source. Then ask your students to do the same thing for a source they identified for their own papers. You can use the Determining and Evaluating Context Worksheet in conjunction with this exercise.