Trusting scientists, questioning facts, and design

Trust me. I'm a scientist.

Ars Technica recently reported on a survey carried out jointly by Scientific American and Nature about the level of public trust in scientists and their opinions.  The results indicate clearly that people trust scientists.  Unfortunately, I'm not sure we can trust these results.

Don't get me wrong.  I am definitely a member of the species Homo Scientificus¹ - I know that science works, and that there are no known alternatives that can outperform science as a means of understanding reality.  But there's a problem with this survey that undermines the argument it makes.

Participants in the survey would have found out about it through one of the two publications that ran it.  It is fair to assume that most readers of Scientific American and Nature have an interest in science, and they are most likely to understand enough of it to see its virtue as a source of knowledge about reality.

This means that the sample used in the survey was biased.  Think of it this way: if the same survey were carried out via, say, the Online Christian Magazine, or the Christian Coalition, then I think the results would have been quite different.

It does not appear that this bias was seen as significant by its developers, who admit that the strong support for scientists shown by the results were not unexpected.  But that means you can't just generalize the results to some sound-bite like "people trust scientists."  That's not how Scientific American reported it, but that's how secondary sources, like Ars Technica, seemed to slant things.

Here's the argument a Luddite might use:

• the survey says people trust scientists;
• but the survey would have obviously sampled only those who read scientific publications;
• this indicates those who took the survey are predisposed to support science and scientists generally;
• since the sample is biased, the results cannot be trusted;
• therefore the survey actually proves the opposite of what it claims; and
• therefore the scientists who ran it cannot be trusted.

The Luddite is also probably sufficiently ignorant to not check the original source, and so would feel entirely justified in this ridiculous conclusion.

What does this have to do with design?  One of the fundamental principles of any good designing practise is to understand one's users.  In this case, while the direct users of the survey would be the community of those who already think well of scientists; there is also another, much larger, indirect user community - those who would hear about the survey through some intermediary source.  To those of us able to think in a designerly way, it would likely not have been a challenge to predict this unfortunate side-effect of the survey.  This means asking lots of (possibly uncomfortable) questions.

What's the point of conducting the survey?  Who's going to read about the results?  How can the results be misinterpreted?  How can misinterpretation be avoided?  How does one mitigate the undesirable effects of misinterpretation?  These are just a few of the possible questions that could be asked from a designerly point of view.

With that foreknowledge, perhaps the survey could have been run differently, or its results be written up more carefully.

This same kind of thing happened to me years ago.  One of my academic mentors showed me some pin buttons he'd had made to promote research activities among potential industry funders.  The buttons read: "Research makes cents."  He grinned, thinking he'd come up with something quite clever.  I said: "Right. Research makes only cents.  Not dollars; just cents."  He tossed the buttons.

I liked the buttons, but I was also able to put myself in the shoes of someone else, someone who doesn't "get it."  My mentor couldn't do that.  A good designer should have that talent.  (I should note, though, that having that talent doesn't necessarily make one a good designer.)

On a related matter, I note a correlation between the SciAm/Nature survey and another recent post of mine, in which I try to clarify the difference between science, the sciences, and scientists.  In this case, I would emphasize that whether we trust a given scientist does not bear on whether we trust scientists in general, nor does it bear on whether we trust science or the sciences.  Scientists will occasionally make mistakes, but in the long term, science still works.

---

1. Note to the gullible: there's no such thing as Homo Scientificus, wish though I might that there were.