CUFOA

Canadian Unlicensed Firearms Owners Association
Association canadienne des propriétaires d’armes sans permis

Gary Mauser, Ph.D.
October 27, 2006

Stakeholders were selected by Ekos and asked a number of questions regarding "Firearms Program Legislative Amendments" in an Online Interview on behalf of the Conservative Government in 2006.

It is my professional opinion that this survey fails to meet the minimum requirements for properly designed research. My opinion is based upon my experience conducting survey research and having taught survey methodology at the university level for more than 30 years.

My comments are organized into three sections, goals, questions, and sampling.

The goals

In the light of comments made by both Prime Minister Stephen Harper and Public Safety Minister Stockwell Day about the importance of assessing the effectiveness of proposed gun laws(1), I'm surprised that the goal of this survey is to assess the support or acceptability of additional gun regulations. In my opinion, this survey violates the government's stated goal of identifying effective proposals.

The Ekos survey is superficial politics. It's only consistent with governing by polls, not principle. I do not need to remind you that this is something the Liberals were seen as doing in the last election and for which they were punished.

I would think the wiser goal would be to seek to improve public safety, and if this were to be achieved, the various groups in society will accept the result because they will see the benefits.

(1) On September 16, 2006, the Prime Minister Stephen Harper told, The House, a CBC Radio program, "Let's find out the facts and make sure that our actions fit the facts. A decade ago people ran out and created a gun registry that, in fact, didn't do anything to prevent these kinds of tragedies and di so at an enormous cost. We want to make sure that what we do is actually effective." On September 19, 2006, Public Safety Minister Stockwell Day said much the same thing in the House, "The fact remains that Canadian taxpayers have spent more than a billion dollars, yet we still do not have an effective system. But that is exactly what we want: an effective system that will work for all Canadians."

The questions

The most important goal in establishing policy is the cost effectiveness of an action, not whether it is 'acceptable to' or 'supported by' a stakeholder group. The questions in this poll are based upon the assumption that the New Conservative government intends to continue governing by polls in the manner of discredited former Liberal government. Asking respondents if they prefer additional regulations without specifying budgetary restrictions is not realistic. In the absence of budget information, it is too easy, since it is free, to prefer ômoreö of whatever the government is offering.

If the goal is to assess respondents' true preferences, then it is wiser to ask about 'trade offs'. For example, respondents could have been asked whether they would prefer additional screening for applicants for firearms permits or greater efforts to monitor violent criminals on probation or parole. Given realistic cost constraints, the police cannot do everything; an increase in one activity necessarily implies a reduction in another.

There are delicate legal issues involved in this survey. I do not believe that respondents will understand the full implications of their answers. If they do not, then that seriously degrades the usefulness of this survey to policy makers.

A relatively minor point is that no information was presented to the Advisory Committee about whether respondents can return to earlier screens after reading information provided later in the survey. If this is possible, then respondents could change their earlier answer to questions. Thus questions about æinitial perceptions' will be misleading because respondents may read ahead to learn more about the questions being asked.

A final egregious example stands out. The last question, on p 36, is exceptionally vague and misleading. This question attempts to get at an overall evaluation, by asking about ægreater firearms controls in Canada, but it is so misleading as to be meaningless. The problem lies in what is meant by the phrase, 'greater firearms controls'. There are two main possibilities:

Does 'greater firearms controls' mean - 'tougher sentences on gun crimes'?
Or does it mean 'stricter regulations'?

Thus, a positive answer, support for 'greater firearms controls', could be given by someone who wants either tougher sentences for gun crimes as well as someone who desires stricter regulations.

Thus, this question is misleading and meaningless.

Sampling Stakeholders

The concept of æstakeholders' as it is used in this survey is fundamentally flawed. Surveys are only useful if they can be shown to represent some target population. Otherwise, they can easily mislead policy makers. Unfortunately, Ekos has provided no valid justification for how stakeholders are selected.

The sampling methods used in this survey have two fundamental problems: no clear definition of what a 'stakeholder' is given, and there is no accepted methodology for sampling within classes of stakeholders.

This failure yields an ad hoc survey where the results will be shaped by unknown forces and will therefore be valueless to policy makers.

I was told Monday that the definition of a stakeholder was 'an organization that has an interest in firearms'. This is far too vague to be helpful. If so, then why were not movie makers or video game developers included?

What does æan interestÆ mean? It is by no means obvious why women's groups have an exceptional interest in firearms? Most victims of violence are men, as are most victims of firearm violence. Why were not men's victim groups included?

By uniquely focusing upon female victims, this category is pandering to political groups who are exploiting the firearms issue for their own policy purposes.

Thus, the categories of stakeholders have been arbitrarily defined and their member organizations chosen even more arbitrarily,

The results will not representative of the target populations, because the sampling methods are arbitrarily. While the final list of stakeholders has not been provided to the Firearms Advisory Committee, the preliminary list made available in August was heavily weighted towards groups that desire additional prohibitions or restrictions on firearms. An equally important problem is the difficulty of sampling within categories of stakeholders. Every category will have its own internal structure; some groups are very hierarchical, like the police, while others are more decentralized, like gun clubs and womenÆs shelters.

For illustration, consider the police. Which police officers should represent 'the police' The commanding officers? The middle ranks? The constables? A representative sampling? No information has been provided to the Firearms Advisory Committee about these methods. Thus Ekos could chose whoever is easiest or most politically compatible. If, as we are told, organizations were encouraged to take time to answer the questions so they can consult within their group, then surveys sent to the police will probably simply be passed to the commanding officer. If so, it would have been considerably cheaper just to ask the chiefs of police directly rather than waste so much more money in this fashion merely to arrive at the same results.

The high level off arbitrariness of the methodology means that many causal factors will be unknown to the civil servants who are ostensibly directing it. This makes the results problematic.

This arbitrariness of the results will reflect an unknown bias. The extent to which the survey company intended to shape the results will remain unknown. It is possible that the results will merely reflect the ignorance and unacknowledged biases of the survey researchers or the selection of the stakeholders.

This is not even an effective method of attempting to gauge political support within stakeholders. Again, consider the category of police. The sample of police interviewed in this sample cannot be said to represent all police in Canada. If not, then what good is this sample?

The survey methods will not yield representative samples of any of the stakeholder categories. This means that the survey results, based upon the sample, cannot be generalized to the stakeholder group. Nor a fortiori, can the results of entire set of stakeholders be generalized to the general public.

In conclusion, given these serious criticisms, I do not believe the results of this survey will be useful to policy makers.