Same Question, Different Answers: How It Happens

How can two studies researching the same question come up with two different answers? That was the dilemma that several media outlets recently had to confront, with the release of the results of two studies looking at the impact of the city of Seattle’s minimum wage ordinance, which raised the minimum wage rate for workers in that city. Even though the studies were looking at the same issue, they came up with results that contradicted each other.

The results of the first study indicated that the wage increase didn’t reduce overall job numbers or hours of work. Media: “Yay! Minimum wage increases are a good thing.” But then the results of the second study indicated that the wage increase caused declines in both numbers of jobs and amounts of work. Media: “Um…okay, maybe minimum wage increases aren’t that great.”

The fact that these studies had different results doesn’t mean that one study is right and the other is wrong, or that both studies are wrong and nobody really knows what happened. The studies are admittedly not easy reading – both use complex forms of economic analysis that, frankly, I wouldn’t try to explain because I would probably get them wrong. But we can still look at how the studies were designed and carried out to see if there are reasons why their results might differ.

Here’s a table that compares the major features of each of the two studies. I’ll explain below the table what the differences between the two studies might mean in terms of producing different results, so keep reading.

Reich, Allegretto, and Godoey (2017) Jardim, Long, Plotnick, van Inwegen, Vigdor, and Wething (2017)
Data source Quarterly Census of Employment and Wages [data collected by US federal Bureau of Labor Statistics] Washington State Employment Security Department labour market data
Data level County (King County, where Seattle is located) and city (Seattle) Data for Seattle extracted from data set including data from all of Washington state
Occupations included in data Food services Low-wage jobs in all industries; low-wage jobs in food services
Why occupation was chosen for study Occupation is highly likely to include low-wage workers.

However, it’s acknowledged that food services employment includes everything from fast-food to high-end restaurants.

Low-wage jobs are most likely to be affected by changes in minimum wage rates.

Tracking low-wage jobs rather than occupations reduces potential of data on higher wage workers affecting results.

Effects of a minimum wage change can be more specifically observed by looking at low-wage jobs in food services.

Limitations of data Data that are reported by employers may not distinguish between e.g. a business with multiple locations reporting as a single employer, and multiple locations of a business each reporting individually.

Data don’t include independent contractors (e.g. Uber drivers).

The way the data are categorized means that  data on food service workers who are paid more than minimum wage cannot be identified or removed.

Data do not include informal earnings or independent contractors.

Data don’t account for some reasons for variations in wages, e.g. workers in the same job having different amounts of experience or seniority.

Data don’t include information on wages for some kinds of contractors.

Measure of wages Average weekly wages Average hourly wage, calculated by dividing total reported wage earnings for the quarter by the total reported number of hours worked during the quarter
Source of information on employment Monthly reports Reports of hours worked and number of jobs in each quarter-year
Control group County-level data from other parts of Washington State and elsewhere in US; none of these areas had changes in their minimum wage rate during period covered by data County-level data from three counties in Washington State near but not adjoining Seattle
Period covered by control group data 6 years before first minimum wage increase in Seattle (April 2015) Third quarter of 2012 through third quarter of 2014
Findings Raising minimum wages up to $13/hr for low-paid workers did not cause disemployment [loss of jobs or hours of work] After the first increase in the minimum wage, a decline in jobs paying less than $11/hr

A decline in number of hours worked for jobs paying less than $13/hr

An increase in jobs paying more than $19/hr

A reduction in the number of jobs in low-wage food services, compared to the number of all restaurant jobs


Both studies used the same basic methodology. Both groups of researchers tried to answer the same questions: did the change in Seattle’s minimum wage rate cause changes in wage rates, or in the number of jobs, or the number of hours worked? And they both tried to answer that question by looking at the same kind of data: reports on job numbers, wages paid and hours worked in Seattle before and after the change in the minimum wage rate.

Both studies used secondary data, which are data that researchers do not collect themselves. Secondary data can be very useful when researchers likely wouldn’t have the time or the money to collect that data themselves. But researchers who use secondary data don’t get to choose how the data are collected, or what data are included or excluded. These studies both used secondary data collected by governments, but each study used a different data set collected by a different level of government. So one reason for the differences in the study results might be that the two studies didn’t use the same data – and both groups of researchers acknowledge that their data have limitations, which might also affect the results.

Both studies also used control groups as part of their methodology. A control group is a group that didn’t experience the change that is being investigated. So if there is no change in the hours and wage rates for the control group, but there is a change in work hours and/or wage rates in Seattle, it’s reasonable to assume that the change in Seattle is because of the change in the minimum wage.

The control groups in both of these studies are regions or areas outside Seattle where there was no change in the minimum wage rate. One of the studies (Reich et al) used a broader geographical range and a longer time period for its control group data than the other study (Jardim et al). Using  more data can help researchers to identify any events or trends that might have affected wage rates or hours of work; the researchers using the larger control group did some adjustments to their data to reduce the impact of those influences. But since each study used a different control group, another reason for the different results might be that the data from Seattle were being compared to two different sets of information.

Both studies also use the same basic assumption about employers’ economic choices. That assumption is that employers do not have an unlimited supply of money or resources, so if there is a increase in the cost of running their companies, the employers usually can’t get more resources – they have to decide how to re-allocate the resources they already have. Wages are a large part of most employers’ operating expenses, so if the minimum wage rate increases and employers want to keep their wage expenses the same, they usually have to reduce the number of employees, or reduce the overall number of hours that their employees work.

Both studies used the same indicators – job numbers or work hours – to assess whether Seattle’s higher minimum wage had any effect on employers’ economic choices. However, one study used monthly reports and one study used quarterly reports (every three months) – so although the data are covering some of the same information, there may be differences in how detailed the data are because one set of data was collected more often than the other.

But the researchers also had to decide not only how to measure any effects of an increased minimum wage; they also had to decide where to measure it, as in identifying the workers  most likely to be affected by the change. One study looked at food service workers, on the assumption that this sector of the economy has the most low-wage workers. But the other study looked at two groups of workers: low-wage workers in general, and low-wage workers in food services. The researchers conducting the other study were able to do this more detailed analysis because of the way their secondary data were organized.

And so to the results. One study’s results says that the increase in the minimum wage had a positive effect; the other study’s results say that the increase had a mostly negative effect. But – take a look at the specifics of the results. Although the directions of the results are different, in some ways the results are actually saying  the same thing – that the increase in the minimum wage may have a different effect at different levels of pay.

The pay rate at which the minimum wage increase appears to start making a difference is at $13 an hour. Workers making less than $13/hour lost jobs and/or work hours; workers making more than $13 an hour after the increase did not lose jobs or work hours. One study also found that more higher-paying jobs ($19/hour and up) were created after the minimum wage increase. This suggests that a couple of different things could be going on. Employers might have more flexibility in their budgets than was expected, so they could afford to pay higher wages without making any other changes. Or, instead of paying more to lower-wage workers, the employers shifted those workers’ tasks  or jobs to higher-paid workers, who might be able to do the same work better or in less time.

Nevertheless, the different impacts that both studies identify are really important findings. They show that a minimum wage increase might not only affect wages – it might also affect the number of jobs, or the number of working hours. They show that the impact of the minimum wage increase might be different at different levels of pay. And they also show that even if an increased minimum wage is intended to affect lower-paid workers, the increase might also affect workers making more than minimum wage.

Another important thing to look at in any research study is the generalizability of the results. Are the outcomes of this study, looking at a specific city, applicable to other cities or regions? This is a particularly critical issue for studies of minimum wage increases, because there are plenty of advocates on either side of that issue who are ready to leap on any evidence that supports their position. Both groups of researchers point out what happened in Seattle may not happen elsewhere, because:

  • The minimum wage increase in Seattle is being phased in through several small annual increases, rather than as a single larger increase. The impact of changes in the minimum wage elsewhere may depend on how the increase is structured or implemented.
  • The amount of the wage increase in Seattle is different for different kinds of employers. The amount varies depending on, for example, how many employees work for the employer; whether the employer pays medical benefits; and whether the employees receive tips from customers. The impact of minimum wage changes may be different in other places if the rate of change is the same for all employers.
  • The city of Seattle is in the centre of a large urban region. So employers in Seattle can avoid paying wage increases by moving their operations to one of the many nearby cities or communities. Employers in other regions may not have that option, so a minimum wage increase may have a different impact on their operations if they can’t go elsewhere.
  • There are several very large and extremely wealthy companies based in or near Seattle (e.g. Microsoft, Amazon, Boeing). These companies are not only large employers; they are also employers whose operations require a lot of highly skilled or highly specialized workers. This means the Seattle labour market is comparatively more stable and has more higher-paid jobs than labour markets in other cities. And it also means that higher wages in Seattle in general may force Seattle companies with lower-skill jobs to pay higher wages than they would elsewhere, if they want to attract and keep employees. So the impact of minimum wage changes may be different in labour markets with different characteristics.

After all this, do we really know any more about whether the minimum wage increase in Seattle is a good thing or a bad thing? Both groups of researchers point out that the full impact of the change may not emerge for several years, so more studies will be needed in the future to answer that question more accurately. There’s also impacts that may not be identified simply by looking at changes in wages and work hours; for example, workers who are paid more can afford to buy more goods and services, so a higher minimum wage may stimulate more economic activity in general.

But it’s not accurate to say that Seattle’s minimum wage increase is good because one study found positive effects, or to say that the increase is bad because one study found negative effects. One study very rarely proves anything, and comparing these two studies side by side shows how important it is to look at the details of studies before jumping to conclusions about their results. What looks like a small difference in how a study is designed or carried out can actually have a big impact on what that study finds.


    1. Hi Graham,
      The one that found the positive effects was by a group of economists affiliated with the Institute for Research on Labor and Employment at the University of California-Berkeley. The one that found the more negative effects was by researchers in the Evans School of Public Policy and Governance at the University of Washington in Seattle.

      1. Thanks for for that. I hope you didn’t hae to go to too much trouble.

        I just wondered who commissioned these groups to produce reports, or did they do it on their own initiative.

        I expect that it would be difficult to discover and probably not worth effort on a sunny autumn day. 🙂 🙂 🙂

      2. Hi Graham,
        Not a problem at all – I just looked at the title pages of the papers! But I’m glad you asked that question, because it’s always important to “consider the source” and whether there’s a potential bias because of the reasons for the project, or its backing. I suspect the Seattle minimum wage plan was an attractive subject for research because of the attention to minimum wage initiatives and similar plans, such as living wage policies – and also because the Seattle plan is markedly different in how it’s structured and implemented. So I imagine there are many people interested in whether this kind of minimum wage initiative has different effects than, say, one-time-only across the board increases.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s