Thursday, March 24, 2016

Can Bernie Still Win? Post Idaho Utah and Arizona

Once again, I am not a Bernie Sanders supporter, but some friends talked me into looking at the data surrounding this primary, and I have found it fascinating. I spent a bit of time trying to figure out how to describe my feelings on Bernie Sanders performance Tuesday night, and "Held Serve" is the sports term that I think is most relevant.  

Bernie won two states by huge margins, but lost the biggest state (Arizona) by a significant margin.  In net, it turns out not to be a huge win, and doesn't fundamentally change the numbers for the rest of the race.


Other websites have detailed accounting of the Tuesday elections, including some craziness in Arizona, but the summary is this:  Bernie lost Arizona by more than expected, and won Idaho and Utah by much more than expected. Because Arizona is bigger than Utah and Idaho combined, Bernie performed slightly though not materially better than this blog's "Bernie Sanders Performance Improvement Plan."

After Tuesday night, Bernie sits in a similar position as he did before, no worse, and no better, really.  He picked up a few delegates over my initial posts, but not enough delegates to fundamentally change the race.  He still needs to win about 57% of remaining delegates to win.  

On the positive side for his followers, he over-performed polling in two western states, so they can likely argue that he has a chance in the rest of the west.  Here's what the delegate counts look like now, first the google view, and then our fair, sans super delegates view.


If this is the first time you've looked at this analysis, you can read the full methodology here.  Essentially, this analysis looks at the pledged delegates required to win (assuming supers will follow), and then uses a logistic function to calculate how much Bernie needs to outperform polling by in each remaining State in order to win.  

How is this helpful? In two ways really:
  • It allows us to quantify how much better than polls Bernie would have perform in order to win. (And the general plausibility of that performance, I generally think it's implausible at this point)   
  • It allows us to set intermediary targets for Bernie's improved performance, that let us know if his current performance is putting him on pace to win.  For instance, Bernie's target for Tuesday was 80 delegates, and he performed slightly better at 85.  On pace, but not good enough to change fundamentals the rest of the way. 

Here's the data, with polling and what Bernie needs to do going forward to have a shot.


I had a few open questions after all of this analysis that I wanted to address.  The first question was: can we project when Bernie will drop out?  To be honest there have been quite a few opinion and think pieces on this in the last few days, ranging from he should drop out now and get out of Clinton's way to, he should wait and see what happens at the convention (e.g. my very low probability scenario where super delegates try to take the party to the left when faced with a Trump opponent).

Right now the data shows that if Bernie doesn't think he has a reason to drop out now, then he won't drop out until after April 26th.  If he still is close after April 26th, then he probably won't drop out until the end of the primaries.  Some reasons for this:
  • Delegate Calendar: We're in a flat spot as far as the delegate calendar, there really aren't any significant races for the next month and the fundamentals underlying the delegate count really can't change until late April (see chart below).
  • Polling: Let's say Bernie is looking at polling to make a decision whether or not to drop out, the polling is pretty grim going forward.  That said, there are likely some in the Bernie camp that would still claim that polls are substantially biased against him (given results in Michigan et. al.), especially in caucuses.  As a side note, the biggest remaining contest (California) doesn't have recent, good polling data.  That would be helpful in both my analysis above, and in Bernie's decision.  Also, in national polls, Bernie appears to continue to close the gap (second chart below).

There's another thing about Bernie's campaign that is bothering me right now, and for lack of a better term, I'm calling it the Sanders Moral Hazard.  I think additional research into this area would be interesting, but here's generally how it lays out:
  • Candidates served by concentrated large donors, Super Pac's or the establishment are more beholden to the rational whims of said donors/institutions. Those donors, many from the business community, are used to pulling plugs on projects and are more dedicated to the party rather than individual candidates:  they may be more likely to pressure a candidate to drop when candidacy seems pointless.
  • Sanders (and candidates like him) have a lot of small donors and supporters, but no big donors to tell him to pull the plug on his candidacy.  Instead, they have a small, populist and emotionally motivated group of followers, that even in the face of defeat, want their leader to stay in.  There is no-one with an individualized motivation nor power to encourage Bernie to drop out.
  • The moral hazard here is this: Sanders has less incentive to drop out of the race because his actual risk (potentially spending money on a pointless campaign) is felt in such a diffuse way, rather than larger, rationally motivated supporters.
  • The irony (and potential harm) here is this: Sanders can stay in the race longer (and theoretically past the point of no-return) with smaller donors.  These small donors are the ones that *pay* for Bernie's risk, and are more likely to be poor/lower middle class voters.  In essence: The structure of Bernie's candidacy has the potential to hurt poor people.  


A few takeaway thoughts:
  • Sports Analogy: Sanders "Held Serve" on Tuesday in Idaho, Utah and Arizona.  
  • Positioning: Sanders is essentially in the same position going forward as he was prior to Tuesday, no worse, but not materially better.
  • Drop Out?: If the Sanders campaign sees no reason to drop out now, it's unlikely they will drop out in the next month.
  • Moral Hazard: Area for future research? There's a potential moral hazard in populist, poor-funded candidates having disincentives to drop-out of races at appropriate times.

Wednesday, March 16, 2016

Can Bernie Still Win: The final post?

For an update on this analysis, please see our most recent post, found here.

Our piece from last week on whether or not Bernie Sanders can still win the democratic nomination was massively popular, so we thought after last night's primaries we should update the analysis.  This post seeks to answer the question simply: Does Bernie still have a chance?  Please reference our prior post for methodology and meaning questions.  Here's a summary:
  • Bernie performed poorly last night, losing all five states.
  • More importantly, his performance was close to recent polling #'s, which he needs to beat significantly in order to win.
  • Going forward, Bernie will need to beat polling by an average of 15.2% to win the nomination. (logit(p) = 0.62)


First a look at Bernie's performance from last night. It was fairly dismal, but also on target with recent polling.  That means that he's not significantly outperforming polling numbers like our prior post found he would need to beat Hillary.  Here's last night's stats:

Now on to our friends at Google and how they are reporting the race.  We like the table they've added below the graphic!

And now our view, that shows where candidates need to get to win the nomination.  Hillary's lead is much more clear at this point.  (Please note this uses a very specific model for super delegate agency, whereas super delegates, in the end, follow the popular vote.)


We're going to use the same methodology that we used before, it's a bit technical but here's the gist of it:
We calculate the amount (using a mathematical logistic function) that Bernie needs to outperform polling by in order to win the nomination.
Once again for detailed method, look at our prior post. We have a "now unassigned" category this week, mainly for delegates from last night that haven't been assigned yet due to some complex party rules. Here's our output:

To win the nomination, Bernie now needs to capture 57.5% of outstanding delegates.  Our calculus shows that requires a logit improvement over current polling of 0.62.  So... what does that mean in not-crazy math terms:
Bernie will have to average (by-state, not weighted) beating current polling by 15.2% in order to win the nomination.
Some people will inevitably say that is doable given the Michigan results, but Michigan isn't representative of polling error in other States.  In fact, a quick look at recent results show that Hillary has out-performed Bernie about the same number of times that he outperforms her.  Side note: in Mississippi she outperformed polling by more than Bernie did in Michigan.  

Michigan is a true outlier, as a state with a lot of rigorous polling, where the pollsters ended up being quite wrong.  One last view at this, here's a new chart of how Bernie needs to improve polling by current percentage.


  • Bernie lost big last night, which put him even further behind in the delegate count.
  • It doesn't appear that he is continuing to significantly beat polling in each state.
  • He will need to beat current polling by 15.2% to win the nomination.

Wednesday, March 9, 2016

Can Bernie Still Win: The Bernie Sanders Performance Improvement Plan

For an update on this analysis, please see our most recent post, found here.

Another morning and another celebration from Bernie Sanders supporters on Facebook.  This time it seems fairly valid: Bernie won the Michigan primary, where he was trailing by 20% in the polls. A big win for Bernie, demonstrating the polls that show him down by 10% or more may be biased, and a big failure for public polling.  Also a win for snark against the media (whom many Sanders supporters consider biased towards Hillary).  Here's my favorite piece from Facebook: 

There's just one problem, and it's the same problem as Saturday. Bernie won Michigan by a slight margin, but lost big Mississippi (under-performing polls by 20%, hey, at least pollsters got it right on average.. there was a lot more polling in Michigan though..).  In net, Bernie still took a double-digit loss to Hillary, in the range of about 20 delegates.  Let's dig into the numbers a bit though.


I've shown my *fair view* (without super delegates) in the my prior posts (found here and here), so I won't spend too much time on them today.  Here's a view of Bernie winning the big state, yet losing the daily delegate count: 

And here's a view of the fair delegate count, once again showing Hillary expanding her lead, but with the majority of delegates needed for nomination still outstanding:

 Let's take stock of what we know (references findings of two prior posts):

  • Bernie is behind by a fairly significant margin and now needs to win (math) 54% of remaining delegates to win the nomination.
  • Bernie is behind in polling in aggregate and in most individual remaining states, so it seems unlikely if results follow polling that he will  catch up.
  • Bernie just massively outperformed polling in Michigan.  This could be due to a variety of issues, most likely that Bernie supporters are young, and young people are notoriously hard to accurately poll.  It may be indicative of an underlying bias against Bernie in polling for future states.
  • Since mid-summer, Bernie has been gaining polling share and continued to do so in January and February.
I put all this information together and realized the question:  Is there still a path to victory for Bernie?  Then I went to developing


(If you aren't a real nerd, you may want to just skip this)

From my past analysis I knew that Bernie needs 54% of remaining delegates to win; which means he also needs to outperform his current polling in the majority of states.  I put together a model to project the margin by which Bernie needs to beat polling in each state.  This method will also allow us to set to set targets along the way, and adjust future needed values as Bernie over and under performs to target.  

I analyzed Bernie's current polling by State, using RCP polling averages, but more heavily weighting recent values.  In states where polling wasn't available, I used polling in demographically and geographically similar states.
We know that Bernie has to beat polling, and if it was easy as figuring out the % he has to beat polling by in each state, (e.g. +12% in each state) this would all be quite simple algebra.  The problem here is that Bernie's potential to outperform varies by State. For instance, it's not reasonable to think that Bernie would pickup the same % in a state where he's currently only getting 20% of the vote as he would in a state where he's getting 45%.  

A sigmoid-type function fits both prior data, and makes a priori sense (less chance for variance at ends of the distribution, more in the middle).  I used a logistic function to calculate percent increase, holding each State to the same logit change over initial polling results,  Then I calculated the required  aggregate logit change to put Bernie ahead in the delegate count nationwide (value currently logit(p) =  0.51).  

Here's what that logit improvement correlates to in actual numbers (e.g. if he's currently polling at 45%, he needs to perform 58% in that State to be on track).


Back to non-nerd land, we calculated what Bernie needs to do mathematically to win the nomination.  If you think that polling is completely broken after Bernie's recent results you can call this his OBVIOUS PATH TO VICTORY.  If you think Bernie still has some work to do, you can call this the BERNIE SANDERS PERFORMANCE IMPROVEMENT PLAN.

A few notes on these numbers:  

  1. To "win" the overall model only requires Bernie to out-perform polling at half the rate he did in Michigan.  This may seem easy after the experience of Tuesday, but keep in mind: Michigan polling may just have been freakishly bad.
  2. The column "Post Change %" is the proportion of the popular vote Bernie needs in each State.
  3. I will continually update these numbers until the race is "over."
  4. We can create intermediary targets using these numbers, by summing earlier periods, such as "Bernie needs to get 343 total delegates on March 15th to remain on target."
  5. We can also set targets for individual races, such as "Bernie should win Ohio with 52.6 % of the vote to remain on target."


Some takeaway points:
  • Bernie's win was huge in Michigan, mostly because how huge the shift was against prior Michigan polling.
  • Polling may or may not be broken.  Obviously the polls were incorrect in terms of final voter behavior in Michigan, but we don't know how accurate they will be in other States.
  • Given that polling going forward may have issues,  we created a path to victory for Bernie, we will refine the model as more information comes in on the nature of polling bias and Bernie's per-state results.

Monday, March 7, 2016

Can Bernie Sanders Still Win? Part 2: Post Super Saturday

For an update on this analysis, please see our most recent post, found here.

After the weekend and our post on Friday, a lot of people pointed out that Bernie Sanders won big on (what CNN was calling) Super Saturday, so it appears he's moving in the right direction towards my March 15th drop-dead date!

I certainly could see how Bernie supporters would be excited about beating Hillary 2-1 in states on Super Saturday. Except for one fact:  Bernie still lost Super Saturday.  It was quickly clear that Bernie supporters weren't looking at the big picture, the final delegate count for the night.


Three primaries were held on Saturday, Kansas, Nebraska, and Louisiana.  Kansas and Nebraska are demographically similar Midwestern states with Bernie-favoring caucuses, whereas Louisiana was the outlier Southern primary State (with almost as many delegates as Kansas + Nebraska).

Here's a summary of what happened with the delegate count.  Notice that though Bernie saw small wins in the Midwestern States, he lost Louisiana by a huge margin, and thus lost the day.

Lucky for Bernie, there was another primary (this time in Maine, a Bernie-friendly New England State) where he won fairly easily.  Here's what the entire weekend looked like, with Bernie bringing home 51% of total delegates for the weekend (67-64).


That 51% victory sounds good for Bernie, but is that an adequate margin of victory?  First, let's look at how the press is reporting current aggregate primary election delegates: google is still showing super-delegates:

Super delegates, as we discussed before, may or may not actually vote for who they are currently supporting.  I recreated our "fair" view into the current state of the race.  I made a slight change from last time, and backed the super delegates out of the "to win" number, making the basic assumption that super delegates will, as they did in 2008, follow pledged delegate counts.

From this view, we can easily determine what Bernie needs to do from here to win, quick calculation: 2,899 pledged delegates left on the table, Bernie needs 1,550 to win.  Bernie needs to win 53.5% of remaining delegates to win the pledged delegate counts.

In essence, Bernie supporters may be happy about his performance on Super Saturday, but he needs to do quite a bit better than that to close the gap on Clinton.

A couple of quick notes on the current tone:
  • If Clinton can rack up big victories in a couple of states (Illinois, Michigan) this thing could be much closer to over very quickly.
  • The sentiment related to Bernie's "ghetto" comments from Sunday night's debate have been hugely negative (as well as shushing Clinton, which some perceived as misogynistic).  Those could have a negative impact with African American and women voters precisely in the two states he needs their support: Illinois and Michigan.


A few takeaways:
  • Bernie lost Super Saturday, despite winning two states, he is still behind in the delegate count.
  • For the weekend, Bernie won the delegate count, but only by 1%.
  • Looking forward, Bernie needs to win 53.5% of the remaining delegates-exceeding his performance over the weekend.

Friday, March 4, 2016

Can Bernie Still Win

For an update on this analysis, please see our most recent post, found here.

Though not a Bernie Sanders supporter, I seem to have a lot of friends and acquaintances who are. As the primary moves on, I've noticed the Bernie fans becoming increasingly disgruntled at the primary process, the democratic party establishment, Debbie Wasserman-Schultz, and generally the mainstream media.  This led to a telling Facebook message from an old college friend, with this general question:
The mainstream media seems to be writing off Bernie, but he's still in the race, so he still has a chance?  It also seems like the media is over-stating Hillary's lead by counting super-delegates that could change their votes, is that true?


The root of my friend's question revolves around a weird quirk in the way the Democratic party nominates candidates:  super-delegates.The definition of super-delegates are effectively this:
An unelected delegate who is free to support any candidate for the presidential nomination at the party's national convention.

Super-delegates have been a huge source of fear in the last couple of election cycles largely due to the uncertainty they create.  There was a theory in the 2008 election cycle that super delegates would nullify the people's will of nominating Barack Obama, and stick to party-favorite Hillary Clinton.  That obviously didn't happen, and when states started to swing to Obama, a good number of super delegates realigned their votes as well.  (Side note: 2383 delegates are needed to earn the nomination, there are 714 super delegates.)

The stats perspective is more interesting though: we have an outstanding number of delegates that will impact the nomination, and don't have a good way to estimate their allocation them because they don't follow state vote counts. We could ask them, but because they are humans, they tend to change their minds.

Here's where my college friend is right: the current vote counts on many websites are fairly misleading, because they are looking at current endorsements of super delegates, which (as we saw in 2008, somewhat) can change over time.  Also, they are excluding a number of "uncommitted" super delegates who may be more likely to be Bernie supporters waiting for him to show some progress (Clinton got a boost early by looking like the favorite all along).

Here's how google is currently reporting things:

And here's my more honest view of things, with pledged delegates only, as well as a top-line for how many delegates needed to win the nomination.

The point here, the initial views put out by the media are misleading, and there are still a lot of outstanding delegates out there.


Since this is still a competitive race, how can we evaluate Bernie's chance to win remaining states?  Let's start with some good news for Bernie fans, Clinton's polling lead (nationwide polls only) has been declining fairly steadily since the middle of last year, shown here:

That means that Sanders is picking up ground in the polls and velocity is with him, but there's still some bad news in the polls: he still trails Clinton by an average of 10% nationwide.

More bad news for Sanders is that the next two weeks of primaries don't look very promising for him.  Though state polling is fairly irregular, he doesn't have a polling lead in any of the next eleven states. Some of the States show huge Clinton leads, so it's relatively unlikely he will turn the overall delegate count around soon.  His performance will be interesting in Illinois and Michigan, as they may be telling in how he will perform in the rest of the Midwest and West (the former confederacy is clearly Clinton territory, Bernie performs better in his home-area, the northeast).  Here's a view into the next two weeks; I'm not willing to go beyond this at this point, because of recent movement in national polls.


We've established that Bernie is losing the election, though not by as much as the mainstream media have been reporting.  Also, we've seen that though he's gaining in polls, he probably won't make much delegate progress in the next round of primaries.  Is there still a path to victory for Bernie?  Maybe.  I can see two scenarios:

  • The Obama-Trending Scenario:  In this scenario two things have to happen.  First, Bernie has to continue eating into Clinton's polling lead, and overtake her, probably needing to lead polls regularly by March 15th. This is possible, but not likely.  Second, Bernie needs to get support of some super delegates who might be willing to change their vote.  For this scenario, the rational model of super-delegate is simply wanting to go along with party preference.  Thus by winning some of the later states AND putting together a coalition of super delegates (much like Obama did), he could possibly win.  (Probability: Probably less than 10%)
  • The Progressive-Trump Scenario: This scenario is based on a different rationality model of the super delegate, in this case a progressive rationalist.  Let's say, that as democratic party insiders the motivation is for super delegates is to get the most liberal person elected.  The common sense answer throughout the election has been that Clinton has a better shot at winning a general election than Bernie (yes, I've seen those other polls that say Bernie has a better chance, but head-to-head polls are junk for multiple reasons, message me if you want to discuss it).  In this case, Bernie keeps it close until the end, and doesn't drop out of the process. Then the Republicans (who's convention is first) nominate Donald Trump, who, consensus generally indicates, has no shot of beating either Hillary or Bernie.  Those rational super-delegates seeking the most liberal candidate now have less reason to choose Hillary over Bernie.  This scenario is a long shot (probability less than 3%) but does give you an idea of how rational Democrats may react to an increasingly likely Trump nomination.


What are the takeaways from all this?
  • My friend was correct that the mainstream media are currently over-estimating the Clinton lead, especially in light of what occurred in 2008.
  • Though Bernie has been gaining in the polls, he still has a lot of ground to makeup-and likely won't show any meaningful electoral progress over the next two weeks.  
  • There are two potential paths to Bernie winning, but he will need to quickly take a polling lead over Clinton and potentially get some help from Donald Trump to win.  He still is somewhat unlikely to win (15% at the high end).

Wednesday, March 2, 2016

Career Upsides, and Salary Growth by Entry Level Salary

Earlier today, while observing yet another fight on Twitter about Kansas employment numbers, I came in contact with an interesting data set.  The data, found here, contains employment numbers and salaries for various careers.  

One facet of the Twitter argument was the growth rate throughout careers, and whether entry level or median salaries were more relevant for low-wage employees.  I was initially very interested in the data, not because of the Twitter fight, but because of something else I've observed.

The observation: there is quite a bit of variation in career path and salary once someone starts a profession. Some professions (bank tellers, for instance) seem to top out their income only a few years into their career, whereas others (analysts, back-end finance managers) tend to see steady income growth throughout. 

Curious, I noticed the data had entry level salary, experienced salary, mean, median, etc.  I messed around with the data, and calculated the jobs with the best and worse career "upsides" as defined by ratio of experienced to entry level employee (with a minimum $50,000 differential to screen out some junk).  Here's a beautifully colored list of the best career upsides:

Why did I color it like this?  Because I noticed some trends, three categories of careers in these groupings:
  • GREEN-Highly educated professionals who gain skills/abilities (also, tenure for post-secondary teachers) as they move in their career.
  • PURPLE-Management type employees that can improve their lot as they move from low-level manager to middle management to director levels.
  • RED-Sales and other client facing roles that can build income by growing a book of business.

BTW, anyone else think it's absurd that they actually state a value for "Entry Level CEO?"

The list of low-income growth careers is much less interesting.  It's all service industry, many careers that are populated by high-school students.  These careers are also far different than our three groups above, in that expertise grows little over the career, there's no management (unless you transition), and there's little opportunity to grow a book of business. The best opportunity in these fields is to move to management, or a more skilled version of their current job.

One last thing I did, in relation to the initial Twitter fight, was to answer the question, how does wage growth throughout career vary by initial entry level wages.  Here's a chart (ratio of experienced:entry level salary, by entry level salary):

The answer isn't as clear as one might think, but two points can be made:
  • In some careers, people start with very low wages, but can increase those wages significantly with more experience.
  • There is a trend though, where the highest entry-wage careers also have the most potential salary growth.
Quick methodology note: These numbers represent point in time estimates of experienced versus entry level employees, and don't represent longitudinal data. Also, people tend to transition between job types, which can bias the numbers, for instance when someone moves from a high level manager to a chief executive (this analysis assumes you stay in a similar role).