Last year, some of the research produced by the Vancouver-based Fraser Institute received some serious criticism. The Institute claims its work is based on “careful, accurate, rigorous measurement”. But the International Labour Organization – an affiliate of the United Nations – released a report which outlined extensive calculation errors and questionable methodologies in the Institute’s Economic Freedom of the World database. And it was also discovered that data for the Institute’s “survey of mining companies” were being collected through a website that was open to anyone, regardless of whether they knew anything about mining.

You would think that criticism like this would make the Institute look a little more thoughtfully at how it conducts its studies. But judging by its new report, Comparing Government and Private Sector Compensation in British Columbia, the Institute isn’t being any more careful with its work. The research presented in this report has numerous problems that contradict the Institute’s claims of “rigorous” and “transparent” methodologies – and which make the results of the research unreliable, to say the least.

One of the basic principles of research is that the researcher should be able to explain why the research is needed: what problem the research solves, how the outcomes of the research contribute to knowledge about the subject, or how the research brings a different perspective to an issue. This explanation would seem to be particularly important for this report, since the Institute released a report on the same topic only two years ago. However, there is no credible explanation, other than having more recent data, of why a new study on the same topic is needed.

Both the 2013 and 2015 reports claim, in almost identical wording in the executive summary sections, that “as the BC government struggles with growing debt and looks for ways to restrain spending,” there is “heightened interest in how wages and non-wage benefits in the government sector compare with those in the private sector.” But there is no explanation of how this situation makes BC’s government different from any other municipal, regional, or federal government, or what about BC makes it an appropriate place to compare private and public sector compensation. The 2015 report adds the qualifier of “particularly in light of ongoing collective bargaining negotiations between the government and its public sector unions” – but these negotiations were “ongoing” in 2012 as well.  So it’s not clear why these negotiations are suddenly more relevant in 2015.

There are no sources cited to support the claim of “heightened interest” in comparing public and private sector compensation. A Google search tells me that these Canadian organizations have recently expressed concerns about differences between public and private sector compensation:

• The Independent Contractors and Businesses Association, in 2012
• The Canadian Taxpayers Federation, in 2013
• The Canadian Federation of Independent Business, also in 2013

Anyone who is familiar with the  Fraser Institute will quickly recognize that these organizations, more often than not, agree with the Institute on economic and social issues. For example, all of them have expressed support for the blatantly anti-union Bill C-377 in Canada’s Parliament. This group of mutually supportive organizations might share a “heightened interest” in private and public sector wage comparisons, but there is no evidence of widespread general interest in or concern about the subject.

Another basic principle of research is that the researcher should acknowledge the full range of perspectives and existing information on the issue in question. The Institute’s report includes a review of existing research comparing private and public sector compensation, and exploring the reasons for differences between the two. This review is highly selective. It does not cite any of the research indicating that there may not be significant differences in pay in the two sectors, such as the research cited in this discussion.  And this section also makes several assertions that are questionable: for example,

[The public and private sector environments] have distinct effects on unions and the threat of strikes. Since the public sector operates in a monopoly with no competitors, workers can threaten and undertake strikes that disrupt service in the public sector with almost no fear of losing customers or a contract. (p. 6)

This statement ignores such realities as many public sector workers being constrained in their ability to strike or being entirely forbidden from striking at all, and several large segments of the public sector, such as education, having private sector competition. The Fraser Institute should be particularly aware of the competitive education market, since it regularly publishes reports arguing that BC’s K-12 private schools are superior to its K-12 public schools.

Researchers should also acknowledge any limitations to the data they use in their analyses. The Institute’s comparison of public and private sector wage rates uses data from Statistics Canada’s Labour Force Survey. But the report makes no mention at all of recent controversies over the accuracy of the LFS data; for example, the fact that only 56,000 Canadian households (out of Statistics Canada’s estimated 12 million households) are surveyed to collect the data. The report states that its analyses are based on “custom” data from Statistics Canada – but, other than listing the variables used in its analyses, the report doesn’t specify what the “custom” data contained. It’s difficult, if not impossible, to replicate the report’s results without knowing the parameters of the relevant data.

The report states that the data set includes information from 68,966 employed individuals ages 15 and older. But there are other pieces of information about the data that are missing from the report. And those pieces are essential to assess the reliability and the validity of the data analysis and its results. Some of what’s missing:

• The characteristics of each variable, e.g. the number of responses or data points that were included in calculating the variable, and the mean, median, and standard distribution of the variable’s values.
• How many employees are represented in each category used in the analysis. A report comparing private and public sector wage rates should at least state how many public sector workers and how many private sector workers were included in the analysis.
• How the data were collected, e.g. whether the wage numbers are self-reported (a method which often introduces bias and inaccuracy into wage data) or are from some other source such as employer records.

The report notes that the LFS data do not distinguish between federal, provincial and municipal public sector employees. Not being able to separate data representing different public sector employers is a major problem for this report, if it is responding to the specific financial problems of BC’s provincial government. The BC government has no control over what municipalities or Canada’s federal government pay their employees. The report also mentions in passing that “When unionization is included in our model, the premium [the difference between public and private sector wages] is reduced to 3.6 percent” (page 12). Since Canada’s public sector has more unionized workers than the private sector, and since many public sector occupations are better-paid professional jobs requiring specialized training or education, this discrepancy is not surprising. But there’s no contextualization of that 3.6% figure – whether it’s exceptionally high or low, whether it has changed over time, or whether it’s comparable to what’s been found in other research.

However, there is also a much larger methodological problem with this analysis. The data used in the analysis may be so summarized that they obscure any truly meaningful differences between different types of workers. A footnote states that “The public sector wage premium within industry and occupation is not presented in the paper due to small sample sizes” (p. 12). Comparisons within industries or occupations would give much more useful results about public and private sector pay differences, because these would compare workers with common characteristics. When the information about the characteristics of the variables is missing, it’s difficult to tell whether the “sample sizes” within this data set actually are too small to conduct a valid statistical comparison. But it’s impossible to make any meaningful comparison of public and private sector wages without some assurance that what’s being compared actually has enough similarities for that comparison to be valid.

The report then goes on to discuss private and public sector differences in non-wage benefits, such as pensions, retirement funding, and paid sick time. The report acknowledges that “individual data on non-wage benefits such as pensions, vacation time, and health benefits are not readily available in Canada” (p. 14), and thus warns, quite rightly, that the analysis in this section does not provide definitive answers. But even with that caveat, the methodology in this section also has problems. For one thing, the analysis relies primarily on comparisons of averages. Averages for these kinds of data may not accurately reflect what’s actually going on in the workplace. There is also no mention of the reality that some public sector jobs, like firefighting and policing, involve dangerous and stressful work, and that may affect overall public sector absence rates and retirement ages. Without this kind of contextual information, simply showing that public sector workers are absent more often than private sector workers, or have different kinds of retirement benefits, really doesn’t prove anything meaningful. (And even with these limited data, statistical analyses such as t-tests could at least suggest whether the differences between averages are statistically significant.)

The report concludes, “Clearly, government sector workers in British Columbia enjoy higher wages, and more than likely higher non-wage benefits, than comparable work­ers in the private sector” (p. 21). The implicit theme of the report seems to be that higher wage rates in the public sector are a problem. But another equally valid way to frame the same issue of wage and benefit discrepancies would be to ask why private sector pay and benefits are not more comparable to the public sector. Taxpayers also fund private sector wages and benefits, through buying products and services from private sector employers. And better private sector benefits such as improved pension funding would ultimately save taxpayers money, because retirees with adequate income would be less likely to need taxpayer-supported social services. This perspective is completely missing from the discussion in the report.

The Institute says that it welcomes “informed” discussion of the issues it raises in its research. This report is not an “informed” contribution to the discussion of private and public sector wage and benefit differences. The report has major methodological problems, it is highly selective in its perspective on the issue, it doesn’t present important information about the data it analyzes, and it fails to acknowledge the limitations and context of its data. So it’s irresponsible for the Institute to suggest, based on this report, that “[A]n important way for governments in B.C. to better control spending is by ensuring public-sector compensation broadly reflects private-sector compensation for similar positions”. There may be credible evidence to support that argument, but it’s not in this report.