There is a television show called Ancient Aliens whose fundamental claim is that Earth was visited long ago by extraterrestrials who exerted wide-ranging influence over the development of society. Among other things, the show claims that Albert Einstein and Nicola Tesla either were, or were in communication with, extraterrestrials.
These claims are ridiculous. If someone brought up these ideas in conversation and provided the same arguments as the television show, they’d be laughed out of the room. But if one were flipping through channels and stumbled upon the actual show, they manage to look a lot more compelling. Why?
First, the show is on the History Channel, which still clings to some vestiges of credibility. Second, the format of the show is documentary-style. Interviews with alleged experts who have their meaningless credentials in the corner of the screen are accompanied by a narrator and music evoking mystery and intrigue.
In my math and physics classes, I’ve used the show as a teaching tool to examine what tricks people use when attempting to present pseudoscience or other outrageous claims as fact. But at its heart, the most compelling weapon at its disposal is its presentation as credible, its argument from a position of authority.
Appeal to authority is dangerous. Citation of an expert’s agreement is not sufficient in proving a claim. Arguments should be evaluated on their merits, not on who is speaking them.
It was therefore incredibly frustrating when, during multiple public comments of school committee meetings, scientists came on and made claims like this:
“The science is in, and we know that schools are 100% safe.”
This argument was preceded by a listing of the individual’s credentials, which included degrees and positions of authority. Yet even a student who is new to science would be skeptical of such a claim; assertions that a statement of high complexity is absolutely true or false are the least likely to be accurate.
At the beginning of this school year, there was much debate over whether schools were safe for teachers and students to return to. It was, understandably, a heated issue. There were very real costs, particularly in mental health, in keeping students out of school. People felt passionately about the topic, but this passion often clouded reason.
Studies that were pointed to in support of the position that schools were safe included those conducted among young people while school was not in session and those conducted in countries who had incredibly different policy responses to the United States, and even to Massachusetts (keep the bars and restaurants open!) These studies suggested what we know now: schools are safe to open for students, given proper precautions.
Unfortunately, the United States, Massachusetts, and Newton were not taking the necessary precautions. Massachusetts was fighting hard to keep things open that should’ve been closed, and doing so at the expense of schools. Newton, for a long time, refused to fix the dysfunctional ventilation or implement a surveillance testing program.
To go into a public forum, spout a series of credentials, and then make overly broad claims while ignoring the context for those claims is deeply irresponsible. Of course, if the public at large had a high degree of scientific literacy, this would be less of an issue. Voters would check these claims and realized that they were not applicable to Newton’s specific situation.
But, alas, that did not happen. Instead, the reopening debate was an emotional affair in which both sides claimed that science was on their side. It was made even worse by the district’s leadership, who encouraged belief in the lie that teachers were science deniers, too scared, or too lazy to return to the buildings.
The worrisome level of science literacy pervades leadership as well. Surveys can be powerful tools to learn how a group feels about a topic. They can also be utterly useless if those surveys are poorly written. Worst of all, they can be dangerous, intentionally worded in a way to elicit the desired response.
One of the best examples of this was the oft-repeated claim at the beginning of the year that 85% of families chose hybrid. As I discuss in more detail in a previous essay, that number came from data which in no way supported the claim that parents and school committee members made. Anyone who made that claim either didn’t understand where the data came from, misunderstood what the data said, or was deliberately attempting to mislead the public.
Another example of a poorly written question came from the Newton Teachers Association (NTA). A parent informed me that when they attended the NTA’s info session on the NTA’s reopening plan, they wished to ask a question. Before they could, they were asked to answer three questions:
1. Name, email, connection to NPS.
This is reasonable, but will certainly discourage individuals from asking questions, as some may prefer to remain anonymous.
2. Why do you oppose the Newton Public Schools’ current reopening plan?
This is an unbelievable question. For a group that was seeking to inform, win over members of the community, and clear up confusion, this question will ensure that only those who are already in agreement with the NTA get their voices heard. If you’re going to turn away everyone who either doesn’t have much info or simply disagrees with you, then what is the point of an informational panel?
3. Do you support the NTA’s reopening plan? Yes or No.
Again, why is this here? Remember that these questions were preconditions to simply asking a question. Presumably, someone is at this info session because they want to learn more! Why is there no option for “unsure?” Why not ask a more nuanced question on the way out, instead of as a prerequisite to engaging at all?
Surveys to members in the past have included similarly biased questions, presupposing support/opposition to an issue and asking why. The intentions were probably good; they wanted to be able to point to data that said people supported them. But the biased wording of the questions through which the data was collected render the data meaningless.
The problems are no less prevalent at the building level. I have messaged back and forth with fellow math teachers during presentations of survey results bemoaning the ambiguous/meaningless axes, unclear or leading questions, and bizarre sorting of data that had two overlapping categories. (Bucket 1: respondents answered A or B; Bucket 2: respondents answered B or C)
Surveys are incredibly difficult to write, as anyone who has studied political science will confirm. The order of the questions, changing individual words or phrases, and interpreting the data are just a few of the points at which a survey can be made meaningless. If you are going to conduct a survey on an important topic, consult somebody who knows how to write surveys. If you really are trying to figure out what people think on a topic, don’t you want to make sure you’re asking the right questions?
We live in a golden era of conspiracy theories. When the president of this country can go on national television and suggest injecting bleach to deal with COVID, we have a major problem. It is up to all of us to question everything – no matter who it’s coming from. Always check to see how the data was collected, what questions were asked, and whether a set of data collected in one time or location is applicable in another time or location. Better data collection, interpretation, and presentation will generate better, more accurate and democratic outcomes for everyone.
Comments
Post a Comment