Measuring Effective Crisis Management
Dec 15, 2007

To intervene effectively in human-made or natural disaster crises requires planning, implementation, and follow-through to ensure that goals are achieved and resources put to best use.

Damage from Katrina.

The international community has become seized with the notion of establishing relevant measures of effectiveness (MOE) for such interventions, as constituent States and their public’s demand increases accountability at all levels of operations. It is divining for effectiveness – using benchmarks, indicators, and metrics borrowed from various disciplines and professions, and applying them to reconstruction and stabilization as well as crisis management operations ­– to determine what, if any, activities are moving mandates forward and achieving effectiveness.

Notable examples where activities are being measured for effectiveness include efforts in Afghanistan and Iraq, as well as intervention activities in areas recovering from natural-disasters such as the Tsunami in Indonesia and Hurricane Katrina in the United States.

Although much has been learned through these interventions, with the development of any measurement system, important questions must be raised to net better results that support accountability at all levels.

With the 2010 Olympics on the cultural and economic horizon, it becomes a significant exercise to identify how Canada will respond to, and measure its effectiveness in the case of a potential security crisis. How will the country support coordination among its stakeholders to increase the effectiveness of managing a crisis while being accountable to an international public?

Impact Assessments
In 2005, I was asked by the Canadian Forces Joint Operations Group (CFJOG) to develop new systems for measuring the effectiveness of activities in Afghanistan prior to the deployment of the first Provincial Recon­struction Team in 2006. After undertaking an extensive research project, my research team soon discovered that assessment ­systems available in related fields and ­professions, such as humanitarianism and development, were being cobbled together and applied (in a somewhat uncoordinated fashion) to reconstruction, stabilization, and crisis management in general. These assessment systems employed various applications; under-utilized technologies; and lacked inter-operability. Additionally, ­systems on the market were not sophisticated enough to capture the inter-relationships between security, economy, culture, and recovery towards some type of societal normalcy, which is what the CF was after.

Calgary flooding.

From this, we developed a series of questions to inform the design of a wholly new inter-operable measurement framework that could consider a robust mix of qualitative and quantitative indicators useful to all stakeholders involved in complex crisis management. The first questions considered in the development process were:

  1. What system of measurement captures the qualitative and quantitative elements of crisis management activities?
  2. What types of crisis management performance measurement frameworks exist now that are employed by international community actors? Are they sufficient for capturing progress relevant to the Canadian Forces’ mandate in relevant crisis environments?
  3. If a measurement framework existed for purposes of tracking progress in crisis management, what would it include? How would it work? Who would use it? For what purposes?

Measuring the relationships between costs, time, activities, stakeholders, and impacts are critical areas that require a thoughtful approach when answering these questions. Using a measuring system that captures these relationships and connections is ­beneficial to meeting mandates, whether domestic or international. There is a danger that any type of crisis management activities have unintended consequences on recipient populations, and this is especially true in politically sensitive and unstable, high-risk environments such as Afghanistan. However, it is important to recognize that the same holds true in other circumstances, such as crisis management immediately following natural disasters or an attack on an international high-value target such as the  Olympic Games.

The most damaging results of any crisis are economic in nature. If economic cycles are not restored quickly in an immediate post-crisis environment, further accelerated systemic security breakdowns will occur. These, then, negatively affect more lives and livelihoods, and can neutralize costly crisis management efforts by causing a negative spiraling cycle for resources, personnel and funding. It is critical to stop this cycle before it begins. The aim must be to restore economic normalcy as quickly as possible after a crisis. Thus, it is important to inject economic and cultural benchmarks into measurement frameworks to capture the complex inter-relationships between security, economy, culture, and recovery to societal normalcy.

Ad-Hocery and Confusion
The basic aim of measuring effectiveness is to estimate the net effects or net outcomes of an intervention – without which complex large-scale projects/activities can fail. Net effects or net outcomes are those results attributable to the intervention, free and clear of the effects of other elements present in the situation under evaluation.  

Currently, there are no established systems of measuring impacts, progress, or effects of intervention activities on human populations in post-conflict or post-crisis environments. A myriad of ill-conceived ad hoc systems exist, however, that do not contribute to future intervention successes because they predominately cater to the political or economic advancement of those funding such intervention mandates.

Although measuring progress for donors putting up the money for the activities is indeed important – it is only a part of the equation of success. There is much confusion around the issue of measuring as an information-gathering exercise for use in information wars, compared to measuring to increase the effectiveness of interventions to assist civilian populations affected by crisis.

The Measurement of Effectiveness (MOE) systems available, especially those employing traditional military combat metrics, are limited in relation to quantifying cultural and social indicators. They require a recasting in order to capture the longer-term impacts upon people recovering from crises. Numbers of buildings reconstructed, kilometers of roads refurbished, and other tangible, quantitative results can be easily relayed to those requiring information. However, equally, if not more important, intangible, qualitative results are not as easily expressed. Consequently, they are often ignored despite their importance in crisis planning and management. Although counter-intuitive for many, to understand better the effectiveness of crisis management activities there is a need to include qualitative social and cultural narratives, as well as numbers and scores, in emerging measurement systems.

Consistency is vital if any MOE system is to be understood and recognized as truly useful. Most stakeholders’ analyses of crisis management, however, is neither systematic nor conclusive because it is often completed after the fact, and based upon ad hoc assessment systems. Crisis environments are significantly complex and the central challenge remains that they require a different data analysis procedure to measure the effectiveness of activities.

Most measuring systems oversimplify complex human conditions and tend to promote a false sense of confidence in those organizations managing crises. As well, analysis of impacts takes time and thoughtfulness – and these luxuries are rarely found in crisis environments. Moreover, effectiveness is difficult to measure because in some realms it may occur over a long time and most crisis management occurs in the short-term (3-weeks) to medium-term (5-year) time frame. The mere act of ­measuring outputs and probable outcomes simplistically reduces the human condition being observed, and is further complicated because identifying impacts and consequences is a subjective process. Further­more, cause-effect chains cannot be traced in linear fashion and reliable indicators and/or baseline data applicable to MOEs are often absent in crisis environments. These issues have led to a general inability of the international community to explain reconstruction activities, goals, and results to both recipient populations and tax-paying constituents back home. This has resulted in increased systemic failures of projects and activities, as well as examples of demoralizing corruption in the funding of complex, large-scale crisis management initiatives.

Keys to Crisis Management
The primary goal of crisis management is to see an affected civilian population safely through various traumas – environmental, security, economic, political – towards better-functioning, prosperous and secure societal norms. However, crisis management is not cheap. In recent cases, the 17-day Olympic Games required a hefty security price tag. Although winter Olympics have traditionally been less of a security risk than summer games, planning for crisis management is necessary in order to reduce potential threats. As domestic security falls under the direction of our national police, allocation of security ­budgets will be directed by Canada’s civilian policing structures. With support elements coming from the Canadian Forces and private security companies, the price tag for security at the 2010 Olympic Games may be prohibitive; nevertheless, it is imperative that the cost of a crisis management plan be coupled with an MOE framework and be built into the overall strategy for the Games.

Information flow is critical to coordination in managing a crisis. Dedicated information networks will facilitate speedy information dissemination to first responders in the event of a crisis during the Games. These networks need to work from the field to Ottawa, from Ottawa to the field, and out to the international public.

Training is critical to allow first responders to understand and recognize security threats. In addition, citizens are an important, often forgotten, aspect of crisis management. Citizens need to understand what is happening, what areas might be of concern, and how to report threats to first responders for analysis, processing, and reporting back to those who can respond. Proper budget allocation; information and communication networks; and advanced training are the keys to managing crises, coupled with an MOE framework that allows for adjustments and coordination of response activities.

Measuring for Effective Crisis Management
Interoperable systems of measurement create robust communication processes for relaying information from crisis environments to decision-makers. As the global peace and security field becomes ever more connected through new-use technologies and political agendas, it behooves all agencies involved in crisis management to use the same or similar forms of measurement systems such that information can be ­gathered, coordinated, and communicated understandably and seamlessly to relevant donors. In the end, my research team developed an advanced-technology measurement framework based on Effects-Based Operations logic that was tied to the objective of long-term sustainable peace and security through the assessment of cultural norms over time.

The framework is scaleable for various types of crises and for inter-operable use by all government and inter-governmental agencies, non-governmental organizations, the private sector and other stakeholders – for cross-analysis and public information campaign reporting. From the time the crisis begins and then management measures are implemented, the framework measures just how close we are, at any given period, to these agreed cultural norms that form the aim of our assistance.

As the international community continues to develop and test measurement frameworks, Canada is in a position to develop and apply a Canadian-made MOE approach to crisis management that enables all government departments, as well as the wider crisis management response community, to work in tandem in domestic and international theatres of operation, and to manage simultaneous crises.

It must be noted that a security and defence MOE framework alone is inadequate to address the real and perceived damage and pain experienced by civilian populations. Economic and cultural indicators must be integrated within the framework to measure true effectiveness. From a whole of Government , we must shift to a whole Canada approach to improve our response to potential crises. Concurrently we should develop and adopt an MOE for this whole of Canada approach. We would then stand well-equipped to lead the international community in the field and be better aligned with UN and NATO or other commitments in our own interests.

Sarah Jane Meharg, Ph.D, is the President of Peace and Conflict Planners and adjunct professor with the Department of Politics and Economics at the Royal Military College. She is the Senior Research Associate at the Pearson Peacekeeping Centre and is a leading post-conflict reconstruction theorist. She has extensive experience conducting field research in Bosnia and Herzegovina She may be reached at: or
© FrontLine Security 2007