Chris Herrod on: Being less biased than professional pollsters

I for one have rarely put much faith in polls–especially here in Utah.  Polls can be easily manipulated, especially if they don’t follow respected and generally accepted standards.   The National Council for Public Polls (NCPP) states:

Polls are not conducted for the good of the world. They are conducted for a reason – either to gain helpful information or to advance a particular cause.  [Link to NCPP here.]

Philosophically, I think polls are often used as a crutch.   When concerning core principles, a true leader should lead rather than stick their finger in the air in an attempt to simply get in front of the crowd. It takes courage to let people know how you feel BEFORE a poll is taken.   We need more leaders with the courage to espouse fundamental principles without being backed-up by a poll. But, that’s my personal opinion.

Do you ever get the feeling that the only reason we have elections is to find out if the polls were right? – Robert Orben, comedian and Pres. Gerald Ford speechwriter

Here in Utah, nobody seems to even question Dan Jones & Associates (a Cicero Group Company) – it’s almost heresy to do so. Interestingly enough, Dan Jones is married to the very vocal and opinionated (former) Democratic Senator, Pat Jones, who actively participated (while in office) in the running of the business and DJ&A’s focus groups.

The NCPP specifically states the first two questions asked by journalists regarding polls should be 1) who did the poll? and 2) who paid for the poll and why was it done?

Of course, there’s a financial incentive for the polling company to find results in favor of what the client wants. Reputable companies (which I believe Dan Jones & Associates is) try to mitigate the bias, but are not always successful. Even Dan Jones is fallible.

We live in a time when “Ph.D.” or any other title seems to give unquestionable authority to anyone who possesses it. However, just the opposite should be true, because good research is at the fingertips of almost everyone today. (In full disclosure, I may have a chip on my shoulder.   I’m a PhD flunky who got frustrated with the esoteric nature of the writing and ridiculousness of some of the research. Doctoral programs are often nothing more than a union card and equates to nothing less than academic hazing.   But, that’s a topic for another day).   The citing of good research (and the facts) should be the measure of validity and/or legitimacy—not simply a name or the reputation of a firm.

CASE and POINT

It doesn’t take a brain surgeon to find fault with the DJ&A’s poll. (The DJ&A poll is regularly cited by Utah Policy, Doug Wright of KSL’s Doug Wright Show [KSL 102.7 FM and 1160 AM], other media outlets, and proponents of Count My Vote.]

(Note to reader: Utah Policy was kind enough to give me the text, which I’m assuming is the poll question in its entirety.)

QUESTION #1

The recent Count My Vote initiative sought to change Utah’s party nomination process by allowing a candidate to gather signatures to get on the primary election ballot instead of having to go through the caucus convention system where convention delegates determine party candidates. Do you support or oppose the Count My Vote effort?

QUESTION #2

In the 2014 Legislature, a bill was passed as a compromise between some legislators and Count My Vote proponents. The compromise fulfilled the Count My Vote goal to provide an alternative way for candidates to get on the primary election ballot. It also preserved the caucus convention system as an option for candidates who wish to use that process. The bill opened primary elections to all voters, not just voters registered as political party members. Now the Utah Republican Party has filed a lawsuit against the bill, and some legislators want to repeal the bill or change it to give convention delegates more power in the nomination process. Do you favor or oppose maintaining the Count My Vote compromise bill as passed by the 2014 Legislature?

A fundamental principle in writing good poll questions is keeping the questions simple and free of biases (i.e leading words and phrases).  [Link to Qualtrics, Adobe Connect, or Survey Monkey here.]

Neither of these two poll questions are simple, and, both contain leading statements.

In Question #1, just the referencing “The Count My Vote Initiative” is a perfect example of a leading statement. It’s no secret that the term Count My Vote was focus group tested. To CMV’s credit, it’s actually a very good name—although deceiving. Question #1 referenced Count My Vote twice and Question #2 references it three times.

Once pollsters describe Count My Vote or SB54, bias is introduced into the question because what the CMV Initiative and SB54 actually do is up for interpretation. (See my previous blog on how Senator Bramble originally described SB54, and how he describes it now.) In Question #1, the Caucus/Convention System is described in a way that makes it seem restrictive or elite (I would assume this is on purpose). There’s no mention that anyone can become a delegate, or, how this system has ranked Utah #1 in the Nation in various categories, by various groups. (Unfortunately, many Utahns still don’t understand the Caucus/Convention System. It is to them what their neighbor, or Doug Wright, has told them it is. ) It’s biased to describe the CMV process in the body of the poll question and not describe the Caucus/Convention System equally, fairly, and honestly, too.

The polls are just being used as another tool of voter suppression. The polls are an attempt to not reflect public opinion, but to shape it. Yours. They want to depress the heck out of you. – Rush Limbaugh

In Question #2, leading terms are even more present, and the question farther from simplicity. The word “compromise” is used in the first sentence. Compromise has a favorable connotation; however, the political parties affected weren’t even involved. That’s quite a stretch from “compromise.” In the third sentence (Question #2), the pollster stated the CMV proposal (SB54) preserved the Caucus/Convention System. On the surface…perhaps. But, no one with credibility believes the Caucus/Convention System will be used (or viable) in a few years. One of Utah Policy’s own writers has stated that SB54 essentially neuters the Caucus/Convention System, so, it’s hardly unbiased to say SB54 preserves the Caucus/Convention System. In the fifth sentence, the pollster use the term “give more power.”   Using a term like “give more power” is a classic example of a leading statement (see previous sources cited).   And, the fact that we are even talking about a fifth sentence—and there’s even a sixth–shows how ridiculously far away from simplicity Question #2 went.

Now that I’ve done some research about legitimate and well-structured polling questions, I have to believe Dan Jones must be dreadfully embarrassed about Question #2, and hopes it doesn’t end up in a college textbook somewhere–as a bad example.

Had the pollster simply ask about SB54, or, “the CMV Initiative,” the results would have been more valid (unquestionably the “undecided” category would have been greater).  But, had the CMV poll been less leading, Doug Wright and the CMV folks wouldn’t have been able to use the questionable poll results as a drum-beating stick.  And, Rich McKewon calls the CMV movement a “noble cause.”

But, alas!  What else can we expect from the CMV folks?!  This is a perfect example of why so many people justly view polls with a healthy measure of skepticism.

– CHRIS HERROD | cf