…Okay, so the past few days I’ve been doing a deep dive into various commonly cited HR studies. And by “deep dive” I mean going beyond the abstract and really reading the whole study to assess how the researchers arrived at their findings/takeaways.
Interestingly, I’ve noticed that a surprisingly high number of these studies either:
A. Were conducted in very controlled conditions that are not all that reflective of the real world environment the results are supposed to apply to.
B. Have relatively narrow demographic populations, limiting how broadly the results of the study can be readily applied.
Here are some great examples of what I’m talking about:
Josh Bersin has an intriguing post up on Linkedin highlighting how human performance doesn’t follow a normal distribution; it actually follows a power-law distribution. As such, Bersin extrapolates that that the performance review process is flawed because it clusters most people as median performers (with a few high and low performers on either end of the distribution, respectively). It’s a very interesting premise, but let’s look into the details of the study (bold emphasis mine):
in 2011 and 2012 by Ernest O’Boyle Jr. and Herman Aguinis (633,263 researchers, entertainers, politicians, and athletes in a total of 198 samples). found that performance in 94 percent of these groups did not follow a normal distribution. Rather these groups fall into what is called a “Power Law” distribution.
You can read more about the study here, but it looks at the performance of researchers, entertainers, politicians and athletes to assess if human performance truly follows a bell curve. Even setting aside the methodology used to define exceptional performance here (which most of us in HR can tell you is a major question onto itself), can we really assume that performance outcomes in the study’s select fields can be extrapolated to be representative of other populations? Or put another way, even if these groups do follow a long tail distribution, who is to say that (as examples) Customer Service Reps or HR Managers or Financial Analysts or Insides Sales people etc. follow that same long tail distribution? …They very well might, but it’s a fairly big assumption to make, yes?
Let’s look at another example. This one is a 2012 CAHRS study on turnover. Here are some of its key findings:
• Organizations using high-involvement work practices have lower rates of quits, dismissals, and total turnover, which in turn leads to higher rates of customer satisfaction.
• Long-term investments in employees—such as the use of internal promotions , high relative pay, pensions, and full-time employment—lead to lower rates of quits, dismissals, and total turnover.
• HR practices that emphasize short-term performance such as intensive performance monitoring and commission-based pay—lead to higher rates of quits, dismissals, and total turnover.
As with the performance research, you can read more about this study here… but again it has some pretty powerful takeaways. Among them; more pay at risk leads to markedly higher turnover, less oversight/greater autonomy is correlated with lower turnover, and a greater number of internal promotions leads to lower turnover. Additionally, according to the study “quits” (i.e. resignations) impact organizations just as adversely as severances.
…Some of these are things that many of us see in our own organizations, but others – like the correlation of short-term incentives/pay at risk with turnover – may be less widely seen. Regardless, we should be cautious about applying the results of this study too broadly. Looking at the populations studied:
Data was taken from two nationally random surveys of customer service centers in 1998 and 2003. The 1998 data came from call centers in the telecommunications industry Data from 2003 was collected from call centers within all industries The surveys were completed by the senior manager at each center and were conducted by a university-based survey research institute
…This data is probably highly applicable to call-center employees in the telecommunications industry… but would a food service company be wise to rely heavily on the findings here when designing incentives programs or addressing turnover for, say, its Sales Managers? I don’t know that it would.
I want to emphasize that these studies (and others like them) are well conducted research with interesting takeaways. But I also think its likely a mistake to assume that results from studies conducted in heavily controlled environments and/or with narrowly defined populations apply too broadly to our own employee populations.
…Instead, it might be wiser for us to be conducting internal (field) studies that apply to our local populations. To this point, Talent Analytics CEO Greta Roberts has a great post up that speaks to the heart of what a good research analytics process looks like here. Specifically, Greta talks about the way that Google has grown employee engagement and productivity by leaps and bounds (while also lowering turnover) by making an organizational commitment to analyzing its population internally.
She goes on to highlight that this is an ongoing process. The organization conducts regular studies to understand what matters to their employees. The “what” changes based on the talent makeup, business environment, economy, and other variables. And only by contextualizing that “what” against the company’s ever shifting internal talent variables and external market forces are they able to implement engagement, performance and retention policies that truly reflect the wants and needs of their unique workforce.
To be sure, there many great organizations leading wonderful HR research efforts out there, and we’re of course all having impactful conversations right now on channels like #TChat and in the comments sections of fantastic HR blogs across the web. But it’s important to remember as we’re exposed to all of this information that it’s all unique to specific places and times and populations. And so everything we read and learn needs to be viewed through this lens.
…So let’s take and apply what studies we can directly to our workforces, while at the same time utilizing less directly applicable research insights as a starting point for conducting internal analyses that allow us to develop local best practices. It’s critical that in our zest to move the needle we don’t inadvertently sabotage our efforts by overgeneralizing.
…This was a long one. For those still here thanks for sticking with me. And as always, please share your thoughts in the comments section below.