PPP is prolific in its polling, with new polls released almost daily. I have linked to PPP's polls in the past. PPP has a good knack for finding headline-grabbing topics.
But since the November elections, I have noticed an almost non-stop release of polls which are presented by PPP as signaling bad news for Republican. It is as if on November 2 Republicans were the toast of the country, but on November 3 they were just toast.
Perhaps because PPP has a tendency to put a very political spin on its polling results, and because the spin has been doom and gloom for Republicans, I have begun receiving a steady stream of e-mails complaining that PPP's polling samples were skewed against Republicans.
I pretty much wrote off such complaints at first as the usual type of griping which happens when people don't like poll results. Rasmussen regularly is skewered in the left-blogosphere because the left-wing blogs don't like the results. If PPP's polling samples and methodology are legitimate, it is not fair to criticize the results. Maybe the world did end for Republicans on November 3.
But a poll released yesterday by PPP caused me finally to dig into PPP's sample data. In a poll which has received enormous attention and warmed the hearts of public sector union supporters, PPP found that in a "do over," current Wisconsin Governor Scott Walker would lose to the person he beat in November, Tom Barrett.
Here is part of how PPP's Tom Jensen described the results:
"We'll have our full poll on the Wisconsin conflict out tomorrow but here's the most interesting finding:if voters in the state could do it over today they'd support defeated Democratic nominee Tom Barrett over Scott Walker by a a 52-45 margin.
The difference between how folks would vote now and how they voted in November can almost all be attributed to shifts within union households. Voters who are not part of union households have barely shifted at all- they report having voted for Walker by 7 points last fall and they still say they would vote for Walker by a 4 point margin. But in households where there is a union member voters now say they'd go for Barrett by a 31 point margin, up quite a bit from the 14 point advantage they report having given him in November."But looking at the data behind the poll reveals that PPP used a sample skewed against Walker voters and towards union households.
The people answering the poll identified themselves as evenly split in how they voted last November, 47-47:
But the actual results in the election were in favor of Walker 52.3-46.5, as this chart from The New York Times shows.
So the sample used in the PPP poll seriously underweighted Walker voters by almost 6%.
Additionally, the PPP sample contained 32% union households:
But in Wisconsin, only 26% of households are union households, as this chart from the Milwaukee Journal Sentinel shows:
So the PPP sample overweighted union households by 6%.
Put together, the PPP voter sample used to show a huge swing since November appears to have accomplished the result by underweighting Walker voters and overweighting union households. This is particularly important since PPP claims (in the quote above) that it was a shift in union household voting patterns which was most prominent.
I posted a comment at PPP's blog last night asking Mr. Jensen for an explanation and pointing out the very issues raised in this post. As of this writing, there has been no response. If Jensen does respond, I will post the response.
Perhaps I'm missing something here. After all, I'm not a professional pollster, I'm just someone who read the data behind the headline. I understand that polling firms sometimes adjust data when the underlying sample is skewed; but there is no indication PPP has done that.
It is not fair to criticize PPP's results just because one doesn't like the results. But it is fair to examine the data behind the poll, which is something I will be doing with more frequency with PPP in the upcoming election cycle.
* * * *
You can help me. If you believe that there are substantive problems with PPP polls, please e-mail me with specifics. I'm not interested in name-calling or conspiracy theories. But if you can document sampling or other errors similar to what I have done above, I would love to hear from you.
Update: Allahpundit and Ed Morrissey at HotAir also have noticed sampling problems with a CBS/NY Times poll just released purporting to show strong union support.
PPP also released at 10 this morning more complete results of its Wisconsin polling, with some more headline-grabbing conclusions -- such as voters siding with the Senate Democrats -- but based on the same voter sample which underweighted Walker voters and overweighted union households, and accompanied by this disclaimer which was not on the results released yesterday (emphasis mine):
"PPP surveyed 768 Wisconsin voters from February 24th to 27th. The survey’s margin of error is +/-3.5%. Other factors, such as refusal to be interviewed and weighting, may introduce additional error that is more difficult to quantify."Update 2:10 p.m. - Jensen has not responded, but another person at PPP (Dustin Ingalls) responded to a series of questions posed by commenters, as follows (emphasis mine):
"Is this a poll of registered or likely voters?"
Registered.
"do you think there's any reason in particular regarding the difference of reported 2008/2010 #s from actual ones?"
Yeah, different voters turning out/responding to the poll. Last year's formula was quite simple: record GOP turnout, and a depressed Democratic base in Wisconsin meant one of the country's largest "enthusiasm gaps." 2012 will probably not be quite as historic a Democratic turnout as 2008 was, but much closer to 2008 than 2010. Our poll is of registered voters, but the more interested you are in politics and voting, the more likely you are to answer the phone and stay on the line for the whole poll, which mirrors their likelihood of voting. Registered voter polls for presidential elections, which have wider turnout than midterms, are closer barometers of the turnout than registered voter polls for a midterm, which see a much narrower and less predictable set of voters turn out, and tend to lean GOP anyway, particularly last year.
"Sample appears seriously skewed. 47% say voted for Walker, but reality of election result was 52.3% so seriously underweight Walker voters. Also, sample is 32% from union households, but only 26% in Wisconsin from union households, so overweight union households."
See what I said above. A lot of potential Barrett voters who will vote in 2012 (or would vote now) stayed home last fall and have been even further galvanized by this fiasco. Also, 26% and 32% aren't a whole lot different, but union voters are a pretty well organized bunch. They turn out at a higher rate than some non-union voters, meaning they'll make up a larger share of the electorate than their share of the overall population.
"Did you also poll the Republicans vs Obama and/or the Republican primary? Will we have those reslts later today, tomorrow or when?"
Dude, do you understand the concept of patience? We release one set of results each day during the week. That's been our M.O. for a long time.
March 1, 2011 1:11 PM
The 26% figure comes from 2010? Of course it'd be higher now, since union members have been stirred to action. It'd probably even be higher than in 2008 if an election were held now. If unions and the Democratic Party in the state can keep those workers angry and organized, that anger could probably cause an even bluer election in 2012 than 2008. If it subsides, you're still looking at an election closer in turnout to 2008 than 2010. Do Republicans think 2010 was the norm and that voters shifted to the right?Seems like a very weak response to me, as it attempts to excuse a clear skew in the sample by speculating that there would be higher turnout numbers only on one side if the election were held how. Well, maybe there would be higher turnout numbers on both sides. There is no evidence given to support such speculation. I hope that this does not represent the level of PPP analysis.
March 1, 2011 1:22 PM
--------------------------------------------
Follow me on Twitter, Facebook, and YouTube
Visit the Legal Insurrection Shop on CafePress!
Having closely observed the campaign, but NOT related to poll-science, I'll say this:
ReplyDeleteTom Barrett did NOT want to run for Governor, and it showed. It showed light red against white. It showed like gold against lead.
If you recall the GWHB campaign's last few weeks, Barrett made GWHB look like the Energizer Bunny.
So, no, I don't believe the poll. Scientific stuff to follow later.
This comment has been removed by the author.
ReplyDeleteNate Silver:
ReplyDelete"The table below presents results for the eight companies in FiveThirtyEight’s database that released at least 10 polls of 2010 gubernatorial and Senate contests into the public domain in the final three weeks of the campaign, and which were active in at least two states."
2010 Accuracy Analysis
538 NYT
More Nate Silver:
ReplyDeletePPP comes in thirteenth out of 62. PPP is not committed to NCPP or the AAPOR Transparency Initiative as of 6/1/10. "Pollsters that have made a commitment to transparency and disclosure have been shown to have superior results over the long-run"
The old 538 (pre-NYT)
"The following table lists the average accuracy of pollsters and polling aggregators based on 2010 Primary, Senate, Governor, and House races using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy. A minimum of nine polls/races were required to be included in the table."
ReplyDeleteAmerican Research Group
Since YouGov came in first in the American Research Group evaluation, I was looking around the YouGov website and found this:
ReplyDeletetoday.yougov.com
The only poll that matters was taken four months ago.
ReplyDeleteThe next poll that matters will be taken 20 months from now ... an eternity.
The. End.
Isn't this a classic example of begging the question?
ReplyDeleteThe poll is accurate because they assume that people now side with the unions thus their poll number showing people siding with the union is accurate.
so you're the one who conducted the Harp Seal / Union Boss / Teacher poll?
ReplyDeletedude.
First saw the dynamic with Gingrich in 90's. Commission a poll, word it and skew the samples to get the desired result (Gingrich unpopular), report your own cooked poll results as News (Gingrich unpopular), take another poll, similar method, report your cooked results as more news (Gingrich more unpopular), repeat, then run stories on most unpopular figure ever. Simple.
ReplyDeleteYou and Ed are doing the work I should be doing, LOL!
ReplyDeleteNice job, and linked, with fleebagger video as well: 'Update on New York Times Poll on Public Sector Unions'.
In Wisconsin, the full census is not as yet reported. But the latest Census numbers against which the demographics being used by the PPP Poll can be checked, indicates that as of mid-2009, there were 4,279,113 persons in the state 18 years of age or older, i.e., a fairly good snapshot of the entire voting age population of the State.
ReplyDeleteAll the Census categories (age 20 and above) add up to 4,110,055, which means (via a little simple math) that there were 169,059 persons who were 18 and 19 year old as of mid-2009.
The remaining available breakdowns, while not precisely consistent with the breakdowns in the PPP Poll can be approximated using an addition or subtraction of 1/10th of a 10-year category here, or splitting a ten year category there, etc., so that the percentages should produce fairly accurate ballpark estimates of the categories, and should add up to the overall estimate in the 2009 Census estimate.
At the link you cited in your post, PPP Poll shows that they surveyed four age categories, and the percentages of respondents from each of those categories, as follows:
18 to 29 ........ 10%
30 to 45 ........ 32%
46 to 65 ........ 38%
Older than 65.... 20%
Here are the actual numbers for each category, and the percentage of that Census number given for those 18 years old and older:
18 to 29 – (929,561) - 21.59%
30 to 45 - (1,215,554) - 28.8%
46 to 65 - (1,432,357) - 33.47%
Older than 65 – (701,641) - 16.4%
Given the fact that Pew reported extremely high voter turnout for young people (18 - 29) in the 2008 election nationwide, and that Wisconsin had the 4th highest turnout in the nation, these numbers seem incongruous with the response you received.
Why did they poll less than half of that young demographic? I thought they told you this result would mirror 2008 much more closely that 2010 did?
I understand that seniors have historically been more likely to vote that others, but the weight of those polled here seems broadly skewed toward those two upper categories, those 46 years old and older. That would include people much more likely focused on pending retirement benefits.
This seems to only reinforce what the percentages of factors like the high number of union households suggested in your post.
Incidentally, the CBS poll seems to have been skewed in an entirely different manner, at least as as relates to age.
I want to point out that the CNN exit polls showed that Barrett held a 63-37 edge with voters in a Union Household (26% of the vote). That is a 26% edge, not 14% as suggested by PPP.
ReplyDeleteMeaning... that he held about a 6.75 real percent edge in Union vote... whereas hypothetically he would hold about an 8% edge based on this poll. That is a difference of 1.25% - if you add that to Barret and subtract from Walker... you move the race from:
52.3 - 46.5
49.8 - 49.0
Walker would still have an advantage..