RAND vs. Gallup

RAND’s trend lines show Obama’s lead outside the margin of error (which is the grey area) at 95% confidence.

On October 27 RAND showed Obama up in the race for the Presidency by six points, 50.93% to 44.58%.   Since the second debate Obama has increased his lead dramatically.   If RAND is right, Obama is home free.

On that same day Gallup, the granddaddy of  all Pollsters, showed Romney up among likely voters 51% to 46%.    The race is tied among registered voters at 48% to 48%.

All other things being equal, it would be tempting to simply say that Gallup should be trusted.   As I’ll describe later, RAND is using a very different methodology than traditional pollsters, and the default conclusion is that RAND’s result is so off from other polls because of this methodological quirk.

Like Gallup, Rasmussen shows Romney opening up a true lead

There are two reasons I’m not going that route.   One is bias.  I support President Obama and if Gallup is right then the race is essentially over, Romney wins.   Yes, one can lose the popular vote and win the electoral college, but a 5% popular vote loss is too big a hurdle.  If Romney is ahead by that much, he’ll win enough states to coast to victory.

The second is that Gallup is as much an outlier as RAND.    While Rasmussen also shows Romney with a large lead (four points), most polls show the race even or trending slightly Obama.   IBD/TIPP has Obama up two, with a stable lead.   Reuters also shows Obama up two, while PPP has the race tied.   Moreover state polling shows consistent leads for Obama.  The leads are smaller for Rasmussen than other pollsters, but there is no way those state polls can be accurate if Gallup is right.   So there are objective reasons to doubt Gallup’s numbers.  (Whew!)

RAND’s “American Life Panel” is unique among polling operations this election cycle — I don’t know if this approach has ever been tried with so much time and effort.    From their website:

“Since July 5, 3,500 participants in the RAND American Life Panel (all U.S. citizens over the age of 18) have been invited to answer three questions every week:

  1. What is the percent chance that you will vote in the Presidential election?
  2. What is the percent chance that you will vote for Obama, Romney, someone else?
  3. What is the percent chance that Obama, Romney, someone else will win?”

Simply, they are not polling a new random sample every night, but keeping the same 3500 participants for the duration. They don’t screen them to see if they are likely voters, or even assign them to one or the other candidate.  They weigh their votes and likelihood of voting according to how they answer the questions.   500 are invited to participate every day on line, so each day’s result is from the previous seven days.  They weigh the results to match the demographics of likely voters.

There are reasons why this method might favor Obama, though Romney did hold the lead in July and early September.  All participants have internet connections, meaning that it could be pulling in more professionals than working class folk, especially among the white population.

Beyond that, people are also likely to self-report a higher likelihood of voting if they’re allowed to choose a probability.   Someone who probably won’t vote might say there’s a sixty percent chance that they will.   Independents may be more willing to split their preferences (e.g. 55% Romney 45% Obama) even though in a traditional poll when forced to choose they would pick who they’ll eventually vote for.    Using percentages given by the participants themselves may yield funky data.

RAND of course knows this, but points to benefits of this approach.   It should better capture trends, and may give real hints as to what the middle — undecideds or soft supporters of either candidate — are going to do.   If the numbers aren’t accurate at least the trend lines should be.

Only one poll really counts – and on Saturday Florida early voters got started on that one!

Gallup, on the other hand, uses traditional methods, but they’ve had their share of problems in the past.   In previous years Gallup’s tracking polls have shown rapid swings and have at times given unbelievable results.    Nate Silver analyzed this and suggests that the likely voter screen is the reason they are so far from the pack.   Gallup may be overly sensitive to enthusiasm or screening in a way that under counts support for Obama.   Gallup counters by noting they do weigh by demographic factors in their final numbers, expecting the 2012 voting population to be much like 2008.

There are other concerns about national polls — how geographically diverse is their sample?   If they over sample the South or the Midwest, voters of all demographic groups are more likely to be supporting Romney.   Gallup doesn’t give full answers, but insists their data and their likely voter screen is solid.    Gallup’s numbers have been more stable this election cycle, in part because they use a seven day rolling average.

Yet between the two are “the rest” – the majority of polls that show the race tied.   Then there are the swing state polls that show Obama’s leads stable, if smaller than last month.  The trend lines in Florida, Ohio, Virginia and North Carolina — states that Romney needs — look better for Obama than for Romney.

So what to make of all this?   Nate Silver has Obama’s odds of winning at about 75%.    Intrade’s traders put Obama’s odds at 63%.   Average poll trends in the swing states show Obama improving on lows before the third debate.

Predictwise spiked abit for Romney on the 24th, but since then has drifted in favor of Obama

In ten days we’ll know a lot more!   My own view is that RAND probably overstates Obama’s edge because if these 3500 people are polled every week, they are likely motivated to pay more attention to the race.    I won’t speculate on why doing that would help Obama, but it is the kind of factor that RAND can’t control for.   On the other hand, it’s feasible that the trend lines RAND shows could be accurate for undecided or “soft” support.   If that’s true, Obama may be leading by more than most people realize.

Still, Gallup may be right.   If the poll stays stable and Romney rolls to a surprising victory, then Gallup has the last laugh.  Romney supporters can’t count on that, but it’s better to be up five in Gallup than down.   However, I suspect Gallup’s methodology is generating poll results that inflate Romney’s lead.   They are not only an outlier (though joined recently by Rasmussen), but they have been an outlier for over a week, suggesting this is systemic rather than the result of funky data.

For now the good news is that partisans of each candidate have some polls to give them comfort in the emotional dash to the finish.   After all, if your guy is going to lose, it’s more fun to enjoy the last days  of the campaign believing he’ll win.

  1. #1 by SShiell on October 28, 2012 - 02:35

    Just for grins I thought I would provide a little more detail for you to chew on regarding the Gallup poll: The poll is made up of 9,424 likely voters — a large enough sample that the maximum statistical margin of error is one point.

    Does that mean it’s over? No – you can play with the polls all you want but the only poll that means anything is the one taken on 6 November.

    • #2 by Scott Erb on October 28, 2012 - 10:16

      Gallup must be taken seriously! If they are right, you will be very happy in nine days and I’ll be disappointed. Playing with the polls is like when a baseball fan plays with stats and scenarios during the pennant race. Yeah, it doesn’t affect the outcome, but it can be fun!

  2. #3 by lbwoodgate on October 29, 2012 - 08:33

    I missed this last week but here’s Paul Krugman’s take on Nate Silver 538 polling b log and Intrade, something you may have already reviewed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: