|
PERFORMANCE INDICATORS FOR THE PUBLIC SECTOR
John R. Allen
Management Consultant
André Paradis
J. André Paradis & Associates
A. INTRODUCTION
The purpose of this paper is to describe a model that can guide public sector leaders in the development and utilization of performance indicators. The concepts described in this paper have been made use of in workshops held for federal, state, provincial and municipal government organizations and agencies within the United States of America and Canada. This paper, along with the handout material of the Consultants' workshops, serves as a reference as managers refine their performance indicators for use in operational planning and control, strategic planning, program review, performance appraisal and accountability to senior management, to elected officials and to the community.
B. CONTEXT OF PERFORMANCE INDICATORS - ACHIEVING RESULTS
Public sector programs exist in order to deliver services to the community, and public sector employees are entrusted with resources with which they can produce program results. Thus, there is one fundamental reason for developing and reporting indicators about program performance: the indicators must help management and staff to produce results. Too often performance indicators are seen as serving accountability purposes only, reviewing past performance in order to give rewards or punishments. However, the principle value of a formal set of performance indicators is that it forces us to think about how the program has succeeded so that we can plan for future successes.
Moreover, the environment of publicly funded services today demands that we collect and use indicators about program performance. Governments today are confronted by public pressures for more accountability, productivity and disclosure; these require them to come to grips with the results produced as well as the resources used. The demand for performance indicators also comes from the politicians who have limited knowledge about programs, but who still must make important decisions about them.
Performance indicators must not be seen as an information system isolated from the mainstream of public sector management. Rather they are a management tool serving a variety of management processes. Strategic planning, resource allocation, operational planning and control, program evaluation and managers' performance contracting are all processes that will not work effectively without performance indicators. Thus, performance indicators must clearly link to and support these processes.
C. WHAT DO WE NEED TO MEASURE? - AN ANALYTICAL FRAMEWORK
One of the pitfalls in developing performance indicators is to take the "smorgasbord" approach, i.e., create a list of all potential or currently collected indicators and choose the ones that seem to make sense. What typically arises is a list that is too large and which does not clearly link to a program's basic mission and its key objectives. If performance indicators are to help us to achieve results, and if they must therefore relate to management processes ranging from strategic planning to operational planning and performance contracting, we need to start at first principles to develop them. That means first, defining the program mission, then identifying its key objectives and finally selecting indicators by which performance can be planned, monitored and evaluated.
1. Define the Program Mission
The mission is a statement of the fundamental purpose of a program. All planning and evaluation should relate to the mission. The mission statement should address three questions:
- who is the primary client, or target group?
- why does the program exist in the first place?
- what must we do to serve the client and to satisfy the longer term purpose of the program?
2. Identify the Key Objectives
The objectives are statements of the specific results the program wishes to achieve in support of the mission. Thus they can be classified into three groups:
- client oriented - the benefits or impacts of the program from the client's point of view,
- strategic - the social, economic or cultural conditions that the program seeks to influence,
- operational - the work processes undertaken in conducting business.
3. Develop Performance Indicators
The indicators are the means by which success in achieving objectives can be planned, monitored and evaluated. Clearly, all three categories of objectives require performance indicators. The difference is the level of management to which they are reported and the uses to which they can be put.
- client oriented; these are the critical success factors in a program. A program seeks to affect strategic ends by delivering benefits to or having impacts upon specified clients, and the plan means little unless it results in benefits or impacts to clients. These indicators are therefore useful for virtually every management process and they should be of interest to all managers.
- strategic; they are of primary interest to politicians and senior staff. Too often strategic objectives are little more than statements of good intention. Indicators are needed at this level in order to trigger further strategic thinking and to guide the preparation and implementation of annual operational plans and budgets. Otherwise we are just gazing into a crystal ball.
- operational; these are needed to ensure success of the work plan. Unless the work plan is implemented properly, the program will not have the desired impact on clients and the strategic objectives will not be achieved. These are of primary interest to program managers for operational planning and control and for performance contracting. Typically only a few of the most important of these indicators would be reported to senior staff.
D. KEY CONCEPTS OF PERFORMANCE INDICATORS
1. Reporting
The principle behind reporting indicators is that managers should receive those indicators which are relevant to them or, information overload will occur and none of the indicators will be used. Thus an individual operating manager needs indicators about his/her own work plan and the client benefits/impacts to which the plan contributed. Senior staff would not receive operational indicators (except in a few special circumstances), because responsibility for these is properly delegated to program managers. They would receive indicators about client oriented and strategic objectives.
2. Effectiveness and Efficiency
Effectiveness can be defined as the extent to which the program achieves its objectives, and whether it is doing the right things. In either case, effectiveness is addressed by the previous framework. The indicators will reveal how well objectives are met, and the three categories ensure that effectiveness is seen not only in terms of work program development but also in terms of client benefit and the achievement of overall strategy. Efficiency indicators are those that relate output to input (usually in terms of dollars or time), and are needed to control cost.
3. Quality and Quantity
Too often, performance indicators are restricted to volume, but, a program may suffer if volume is increased while quality declines. In fact, quality is often more important than quantity in terms of client satisfaction. Therefore, indicators of quality (e.g. turnaround time, accessibility, error rate, etc.) should be developed as well as indicators of quantity.
4. Numerical and Narrative Expression
As anyone in the public sector knows, you cannot measure everything. We should try to use numerical indicators wherever possible. Some results however, defy measurement and sometimes measurement is just too expensive. A common pitfall of performance indicators in government is to ignore that which cannot be measured. This is a mistake because a non-quantifiable objective may be just as important to program success as a quantifiable one. Therefore, it too should be formally stated and reported. The indicator would be a narrative one (e.g., anecdote, expert opinion, interviews, etc.).
E. CRITERIA FOR SELECTING PERFORMANCE INDICATORS
The most important step in developing indicators is determining what needs to be measured. But once objectives are set, the selection of indicators should follow a set of standards, including;
1. Validity
Does the indicator reveal performance in achieving the objective? Indicators for operational objectives tend to be precise. It is often more difficult to get a true indicator for client oriented and strategic objectives. In such cases, proxy indicators or narratives might be needed. The resulting performance information might not be complete, but then few decisions in government can be made with complete information. Indicators can point the way, but they will not replace management experience and judgment.
2. Accuracy
Can the performance data be verified as reliable? After all, errors do occur and there is a possibility of "fudged" or sloppy data. Generally, the greater the accuracy, the greater the collection cost. The key is up-front quality control (e.g., clear instructions on definitions and completion of data collection forms) and an audit trail for random accuracy checks.
3. Clarity
Can a layman understand the indicator? This is important for indicators of client oriented and strategic objectives; these are used by politicians and senior staff who may not be familiar with the terminology of a program. Where special terminology is unavoidable, it is important to state the objective in lay terms even if the indicator is technical.
4. Timeliness
Can data be collected and reported in time to make it useful? It depends on the decision or management process the indicator supports. Indicators of strategic objectives may only require periodic data collection because strategic plans are not reviewed frequently. Data for operational objectives might be collected continuously to support the regular demands of operational planning and control and managers' performance appraisal.
5. Potential for Encouraging Undesirable Behavior
Behavior is elicited by indicators; that is what we want. Indicators however, can also cause people to behave in ways dysfunctional to the success of a program. The best way to avoid this is to ensure that all the key objectives, even the unmeasurable ones, are included and formally reviewed, and to ensure that we have indicators of quality as well as quantity.
6. Data Collection Cost
There is no rule of thumb about how much, performance indicators should cost. There are aspects of cost that we can control, such as breadth of data (record all phenomena or a sample only), frequency of collection and reporting, degree of accuracy and source of data (e.g., own records, client applications and reports, information from other jurisdictions and organizations, trained observer ratings, surveys and focus groups). Again, the point to consider is the importance of the decisions that the indicators support, and the fact to remember is that we will never have complete information no matter how much we spend.
F. SOURCES OF PERFORMANCE INFORMATION
Data collection is the heart of a performance indicator effort in government. Four sources of performance data are presented below;
1. Government Records
This refers to information the program collects on its own. After determining what you need to measure, the most obvious place to look for performance indicators is among data that a program currently collects. Sometimes, such data may be explicitly performance related and sometimes it is collected for other purposes - all it may require is repackaging. The experience of many, even most, government programs is that almost all performance data needed is already collected in some form or other.
If some required performance information is not currently collected, then an option is to have it collected by a program's own staff members. Such information would typically be about operational objectives or observed client behavior. To be effective with this method, clear, consistent definitions and procedures must be developed and tested with staff, training must be provided, an audit routine should be established and the findings should be reported back to staff. If possible, performance data should be incorporated onto records that are routinely kept (e.g., attendance sheets, activity logs, program cost records).
2. Information from Other Governments and Organizations
Data that you need and which is collected by others is attractive because it can cost little or nothing. The drawbacks are that you have no control over what is collected or how long it will be collected, the frequency and timing of collection and the accuracy of the data. Still, there are many sources of such data including other department or government operating programs, coordinating departments or programs, federal government funding programs which require yours or other programs to provide data, national, state or provincial polls and surveys, federal, state or provincial task forces and special studies, program evaluations, audits, commissions, questions to government leaders and special interest and lobby groups.
3. Client Applications and Reports
If a program provides funding to clients or to organizations which serve program clients, or if any other formal accountability relationship exists, client applications and reports can be a rich source of performance information.
However, clients will not like providing information to you any more than you like providing it to your own business department. Therefore the program must be clear about the data it needs and must avoid overloading clients with data gathering requirements. Data should be incorporated into existing forms such as grant applications, budget reports and progress reports if possible, and the information should be subject to audit. One of the benefits of collecting information in this way is that it can lead clients into evaluation of their own operations, and if they become more efficient and effective the program is more successful too.
4. Surveys
If you want to know whether clients are happy then you really ought to ask them. A survey is a method that samples the opinions, perception or satisfaction of a representative group of your clients, citizens or other interested parties about your programs. They are best suited for strategic planning or program evaluation because they typically do not provide information in a short time frame and they can be expensive. They do not need to be done by market research consultants, though. A program manager can design and administer a survey in-house, and most managers can eye ball the findings of a simple survey without needed complex statistical analysis.
G. USE OF PERFORMANCE INDICATORS
As stated earlier, the only reason for generating performance indicators is to help a program. This section describes five common uses of indicators;
1. Communication with External Parties
The framework set out in this paper is not new. It has been a feature of results reporting and program evaluation in public sector organizations for many years. This approach has benefits for any program in terms of its interaction with external parties that extend beyond accountability. All programs in government perform work; the real question in accountability, and indeed in resource allocation, is whether that work is resulting in any benefit or impact to people and whether it is contributing to a change in a social condition. The management and staff of a government program are committed to that program's goals.
External parties do not necessarily understand those goals and strategies. It is up to program management and staff to take every opportunity, including the submission of results reports, to show that it not only "does work", but also that it "does good", if it wants the support (including financial support) of external parties.
2. Strategic Planning
Strategic planning is the process by which an organization scans its environment, determines its basic mission, its overall long term objectives, the kind of benefits or impacts it wishes to have on its clients and the activities by which it will deliver them.
The framework presented in this paper addresses these basic features of strategic planning. The results information framework does more than structure thinking along strategic lines, however.
There are more good things that governments could do than they have the resources to undertake.
Strategic planning is the process of making fundamental decisions about rank order priorities. To provide guidance to managers, strategic plans must give direction about how the organization wants alternatives to be traded off.
In the public sector, this rank order of priorities can be difficult due to government's open nature. In the absence of a rank order of priorities, every objective appears to be a top priority, and this gives no useful guidance to managers and staff. If for political reasons, rank ordered priorities cannot be set, performance indicators are the next best thing. Indicators of performance in the strategic and client oriented objectives will allow targets to be set, actual performance to be monitored and evaluated, and subsequently will give managers and staff guidance about where they should be concentrating their efforts; those objectives which are being achieved do not need as much attention as those that are not. Thus, besides structuring thinking along strategic lines, the results information framework improves the implementation of the strategy.
3. Operational Planning and Control
Operational planning and control are those parts of the management cycle in which long range, or strategic aims are converted into a specific set of planning decisions, including statements of results expected and resources required, for a short term period, usually one year. It is the process which determines;
- what program results are to be achieved in support of strategic objectives,
- what activities are needed to achieve the results,
- how program performance will be monitored,
- what resources are required to implement the plans,
- who is responsible for implementation.
Operational planning has a shorter term focus than strategic planning, and deals more with choices around the actions that should be taken to implement strategic objectives. Therefore, the results that are relevant to this management process are the ones that we can influence over the short term - organizing the work processes so that the program's developed, out the door and into the clients' hands. Information on client benefits is essential in operational planning and control in order to guide the selection and design of projects and activities for the year, and it is essential for control to ensure that those work processes have the desired effect. It is possible to have a very busy year without making any difference to clients.
Thus, performance indicators are intended more for forward looking planning and control than for historical accountability. Even in accountability, the point of evaluating actual performance should not be simply rewards and sanctions. Rather past performance is worth evaluating mainly for what it tells us about what we should do in the future to maintain and improve performance.
A variance between planned and actual results does not constitute a judgment about whether a staff member, a manager or the program has performed well or poorly. Instead, it is a red flag that tells us that we should find out why performance varied and what we should do about it in the future.
Using performance indicators means formally integrating them into the planning and monitoring cycle of a program. In other words, there must be documents upon which objectives and results are planned, reviewed and modified on a regular basis, not simply reported once or twice a year to the business department or to a central agency. Indeed, it is becoming much more common for senior staff and politicians to ask for results information at any time during the year to ensure that indicators are being used internally by the program on a regular basis.
4. Program Evaluation
Program evaluation is another major use of results information. Program evaluation is a more formal and rigorous assessment of a program than is operational planning and control, and it takes place at much longer intervals. It is also much broader in scope, asking questions about the usefulness of and need for programs;
- is the program mission still relevant?
- how successful has the program been in achieving its strategic objectives over time?
- have the intended consequences occurred, and have there been any unintended consequences?
- are there program delivery alternatives?
The relevant results in program evaluation are those which are the closest reflection of how effective the program has been in achieving its mission; the client benefits/impacts and the strategic objectives. The results information framework recommended in this paper is not a substitute for program evaluation, but it can be of tremendous value in the management process.
Program evaluation can be quite expensive and the findings are often inconclusive. A major reason for this is that there is commonly a lack of good results information in many programs. However, results information gathered and used primarily for operational planning and control can help in program evaluation by;
- establishing a framework for analyzing the program that will speed the evaluation and that will have fostered an evaluation mindset among managers,
- developing a time series of performance data that will reduce the cost and strengthen the conclusions of program evaluation,
- identifying programs that need to be evaluated, so that limited program evaluation resources are focused on areas where they will do the most good, and not disrupting staff and program delivery in areas that are already performing well.
5. Performance Appraisal
The main value of performance indicators is in planning and improving program performance. Therefore, in implementing performance indicators, this should come before managers' or staff performance appraisal. Indicators can make an important contribution to this latter process.
The absence of performance indicators makes performance appraisal very difficult and largely ineffective. No targets can be set so individual performance cannot be objectively assessed. Performance appraisal regresses to a subjective opinion about personality traits (e.g., ability to get along well with others, creativity, leadership) and anecdotes about especially memorable events. This makes performance appraisal a personal relationship between the supervisor and the staff member rather than a business association with the organization, and it tends to focus staff attention on fire fighting and special projects rather than on solid, consistent performance. These conditions do little good for the supervisor, for staff, or for the organization, because what is really important is achieving results. Performance indicators can help by allowing targets to be set and objectively assessed, and by focusing individual effort on organizational goals.
| 