2015 was a really bad year for the Canadian Broadcasting Corporation (CBC). At the start of the year, business correspondent Amanda Lang was accused of being in a conflict of interest for her reporting on an issue involving the Royal Bank of Canada while having a personal relationship with a RBC executive. (Lang later left the CBC for a new job with Bloomberg TV.) Then radio host Jian Ghomeshi lost his job because of incidents that resulted in him being charged with one count of choking and five counts of sexual assault – and CBC management’s awkward handling of that situation led to the firing of two top executives. And then TV host Evan Solomon was fired after allegations that he exploited his work-related connections to sell high-priced artworks. (He found a new job on satellite radio and as a magazine columnist.)

There was one bright spot for the CBC in October when the decidedly anti-CBC Conservative party was defeated in the Canadian federal election. The potential for change in the CBC’s relationship with Canada’s federal government, which funds the CBC’s operations, was characterized by retired CBC journalist Linden MacIntyre as “the people that are the custodians of this publicly owned institution no longer seem[ing] to hate it” – but the CBC is still struggling with the fallout from the traumatic events that marred its reputation in the past year.

An external review of the CBC workplace was commissioned after the Ghomeshi debacle. The resulting report identified not only procedural problems in how the CBC handled allegations of employee misconduct, but also larger issues, such as a “host culture” that tolerated questionable behavior by high-profile talent, and unclear lines of authority that made it difficult to effectively address workplace problems. The report presented several recommendations, one of which was that the CBC hire a third party “to design and develop a comprehensive employee survey relating to workplace culture and respect in the workplace”.

The CBC acted on that recommendation, and the Gallup polling organization conducted an online survey of full- and part-time CBC employees in June and July. 52% of CBC’s 7600 employees completed the survey, which is an excellent response rate for a survey of this kind. The results of the survey, completed in October, were apparently only circulated internally. However, the Toronto Star newspaper obtained a copy of the findings from the survey. The results indicated, among other things, that nearly half of the respondents thought the CBC was a “psychologically unhealthy” workplace, and showed a 20% decline between 2012 and 2015 in the number of employees feeling proud to work at the CBC.

The day after the Star‘s story, the CBC issued its own statement about the survey. Its statement was positioned as presenting “the facts”, but it didn’t actually deny anything the Star said. So I think it’s fairly safe to assume that the Star‘s information about the survey is accurate. What’s less certain, however, is whether the results of the survey will actually help the CBC improve its workplace.

For a survey like this to be useful – that is, for it to provide reliable information as a basis for meaningful action – it needs to be designed to capture specific and consistent information. This consideration is especially important in collecting data on a multifaceted attitude such as employee engagement, because such attitudes can be affected by multiple factors in the workplace (e.g. pay, quality of supervision, job security, job tasks), and each of those can affect each person differently.

The Star story indicates that CBC management intends to provide each department with the survey results from the respondents in that department. That suggests the survey collected at least some demographic data from the respondents – at the very least, what part of the organization they worked in. It’s not clear if other demographic data were also collected, such as how long the respondent had worked at the CBC, what job they currently hold, their age, or their gender. These data can be extremely useful in identifying comparative issues – e.g. if women have different levels of employee engagement than men, that difference may point to something involving the role of gender in the organization’s culture. But there are other pieces of information in the Star’s story that also call into question how useful the survey results might be.

The CBC respondents were asked to rate a variety of statements on a five-point scale, with the choices on the scale ranging from “strongly disagree” to “strongly agree”. This is a fairly standard survey technique, but the potential problem with it is that different respondents may have different ideas about the meaning of each point on the scale, or different ideas about what separates the points on the scale – e.g. what distinguishes “strongly agree” from “somewhat agree”. If respondents don’t have the same idea of what the points on the scale represent, then respondents could be choosing the same point on the scale while having different feelings about the statement they are responding to. (There are several ways that survey designs can address this problem, such as attaching specific examples to each point to illustrate the differences between them, or assigning explanatory numerical values or percentages to each point (e.g. “somewhat agree” = “agree with this 75% to 50% of the time”).

A larger concern, though, is that several of the statements that respondents were asked to rate included more than one issue – e.g. “My supervisor, or someone at work, seems to care about me as a person”, and “Employer deals effectively with situations that may threaten or harm employees”. The problem with structuring survey questions this way is that if respondents feel differently about one part of the the statement than they do about the other – for example, if they think their supervisor doesn’t care about them but they think that their co-workers do – those two feelings are not going to be accurately captured by answering a single question that asks about both.

This problem can be avoided by structuring survey questions to have only one issue each. In the first example above, the statement “My supervisor, or someone at work, seems to care about me as a person” could be rephrased as two separate statements: “My supervisor seems to care about me as a person” and “Someone at work seems to care about me as a person”. Phrasing survey items this way can make the survey longer, which may discourage respondents with short attention spans or limited time – but it may actually be faster for respondents to answer more direct questions, rather than puzzling over how to best answer a question combining two things they feel differently about.

The overall results of the survey demonstrate that the CBC has some significant problems in its workplace culture – something which, hopefully, CBC management knew before the survey was conducted. But it’s unclear whether the survey results will give the CBC reliable information as a basis for the important workplace changes which it seems to desperately need. And that too is a problem, because in a workplace as demoralized as the CBC seems to be, ineffective or meaningless attempts at change might cause even more damage to employee engagement. Further data collection or change efforts need to be perceived as part of a genuine commitment to make things better.

It’s commendable that the CBC acted so quickly on the recommendation to conduct an employee engagement survey. If this survey is intended as a starting point, and if there are plans to conduct more specific data collection in the future, that too is commendable. Ideally, future surveys will have more specific questions that will potentially produce better information. But if the CBC plans to move ahead based on the results of this survey alone, it needs to proceed very carefully.