Office of Sustainable Development
Bureau for Africa
U.S. Agency for International Development
| AFR/SD | Africa Bureau’s Office of Sustainable Development |
| BCC | Behavior Change Communication |
| CBD | Community-based Distribution |
| GDP | Gross Domestic Product |
| GNP | Gross National Product |
| MOH | Ministry of Health |
| NGO | Non-governmental Organization |
| SIP | Sector Investment Program |
| USAID | United States Agency for International Development |
Acknowledgements
I would like to give my sincere thanks to the many people who contributed to this document. Thanks go to Abraham Bekele AFR/SD Health Economist, and Alex Ross, AFR/SD Health Policy Advisor, for their generous contributions to this document, especially in developing the Financial Sustainabilty and Enabling Environment sections, respectively; to the Human Resources Development Assistance Project, especially Paul Hopstock, Senior Associate, for his wisdom and invaluable contribution in developing the Institutional Capacity section; and to Roy Miller, Monitoring and Evaluation Coordinator of the LINKAGES Project, for his guidance and knack for asking the hard questions. My special thanks to the Center for International Health Information, especially to John Jerome and Lisa Nichols, for their technical input, for serving as the secretariat, and for pulling together the final product.
My sincere thanks to Hope Sukin, Chief, Human Resources Division, Africa Bureau, Office of Sustainable Development, for her interest in this document and her encouragement throughout the process.
I would also like to thank Katie McDonald (G/PHN), Tom Bossert (Harvard School of Public Health), Cathy Bowes (BHR/PVC), Karen Tietjen (AFR/SD), Karen Foreit (Policy Project), Dan Kraushaar (Management Sciences for Health), Oscar Picazo (The World Bank), and Rod Knight (Evaluation Project) for their invaluable comments, for which we are truly grateful.
And last but not least, my special thanks to Renuka Bery and the SARA Project staff for their invaluable assistance in editing, designing, and publishing this document.
Subhi Mehdi
Information, Monitoring, and Evaluation Advisor
Bureau for Africa
Office of Sustainable Development
A commonly accepted model of a health and family planning results framework is presented in Health and Family Planning Indicators: A Tool for Results Frameworks: Volume I, published by the Africa Bureau’s Office of Sustainable Development (AFR/SD). (See Figure 1.) The document defines sustainability as: “...the ability of host country entities (community, public and/or private) to assume responsibility for programs and/or outcomes without adversely affecting the ability to maintain or continue program objectives or outcomes.”
To build on this work, and in response to needs expressed by USAID field missions, AFR/SD developed this supplement to Volume I to measure sustainability in the health sector. This document is intended for use by program officers and program managers to help them define indicators of sustainability for their health and family planning programs. It is intended to be used as a resource document which can be referred to during program design and program assessment activities.
There are two types of sustainability indicators for health and family planning programs. The first type of indicator examines outcomes retrospectively long after program interventions have been completed. These indicators examine whether programs and health status have been sustained, and are often used in major impact evaluations. The second type of indicator examines aspects of ongoing programs and activities that can be used to predict future sustainability. These indicators are used for monitoring and process evaluation purposes. This document is primarily focused on the second type of indicator. Although some of the indicators in this document can be applied retrospectively, most assume the purpose is to examine aspects of current programs and activities that can be used to predict future sustainability.
To address measurement of sustainability, AFR/SD first developed a conceptual framework for Africa (see Figure 2). This sustainability model acknowledges the importance of sustaining health status and not merely a delivery system or an organization. The model is systemic; it takes into consideration all the elements that contribute to sustainability, and encompasses both supply and demand issues. The model also takes into account the socio-economic context of health systems as having a potentially significant bearing on the types and degrees of sustainable programs and outcomes that can be achieved.
This report includes a discussion of the framework elements followed by related indicators. Each indicator is presented with a more detailed Definition (as necessary), a Discussion section to clarify its use in relationship to sustainability, a proposed Data Source, and, a Reference to the original source. Appendix A is a glossary; Appendix B presents in summary form the complete list of indicators included in the document; and Appendix C provides a bibliography of related materials used to identify indicators and background information.
|
A Word of Caution...
The desired direction for change of many health status indicators (such as infant mortality or contraceptive prevalence) is obvious; that is, a successful child survival program will cause infant mortality to decline and a successful family planning program will bring about a rise in contraceptive prevalence. The same is not always true for indicators of sustainability. The desired direction for change for some of the sustainability indicators depends on the programmatic context. Consider, for example, the indicator “Government health expenditure as a percent of GDP.” In a country trying to privatize the delivery of basic health services while instituting cost-recovery measures, an appropriate target for this indicator might actually be a smaller percentage than the baseline, especially in a rapidly growing economy. Still, in most countries, the hopes are that governments will increase expenditures relatively more quickly for health than in other sectors. |
A. Sustainability of Health Status
Table of Contents
The frameworks in Figures 1 and 2 are based on the premise that family planning and health programs exist to improve the Health Status of a population, and strive to achieve that through Improved Use of Health Services, Improved Health Practices, or both. The highest level of objectives for any program, however, depends on what is deemed to be within the manageable interest of the operating units. These concepts form the basis of the model in Figure 1, intended for use in designing and monitoring family planning and health programs. These programs have four critical dimensions for achieving desired use and health status outcomes: Access, Quality, Demand, and Sustainability. Volume II begins where the model in Figure 1, Volume I leaves off; the sustainability framework expands the sustainability dimension within this earlier model.
It should be noted that there is a great deal of potential overlap between indicators of access, quality, and demand and indicators of sustainability. For example, indicators of the sustainability of demand are distinct from indicators of current demand. Similarly, indicators of sustainable institutional capacity are distinct from indicators of current institutional capacity, typically used as indicators of quality of services. Considerable effort is made, therefore, to distinguish between sustainability indicators and those related to access, quality, and demand.
When donor organizations speak about sustainability, the discourse moves rapidly to issues of local assumption of financial responsibility and maintaining the viability of service organizations. This document will do the same, with one significant difference: a focus on “the generation and/or preservation of demand for family planning and health services, and the willingness to practice healthy behaviors.”
Instances exist where health status is best preserved by eliminating the need for continued services (as in the eradication of smallpox and soon, polio, and when resources become available, to bring better water and sanitation to most people, and the need to practice oral rehydration therapy). Other instances exist where health status is best preserved by altering the situation that causes the health problem in the first place (as in market interventions to alter access to nutritious foods so that micronutrient distribution and supplementary feeding programs can be eliminated).
B. The Systems and Demand Dichotomy
Table of Contents
The sustainability framework distinguishes between Sustainability of Systems and Sustainability of Demand. Both are necessary to achieve a viable, self-reliant system that perpetuates and improves healthy outcomes.
The model identifies three components of system sustainability: Financial Sustainability, Institutional Capacity and Enabling Environment. Demand sustainability has two main components: Ability to Pay and Attitude. Each component has sub-elements, which can be viewed as program approaches or interventions. The sub-elements in this model are not an exhaustive list; rather, they represent current programs that address sustainability. The sub-elements are not presented in any hierarchical order, nor are there necessarily causal links between them.
The model is designed to capture the predominant aspects of sustainability as currently addressed in the context of USAID’s programs in Africa. ARF/SD recognizes that other models or programs might incorporate aspects of sustainability that are unique to their country and specific context.
C. External Factors: The Socioeconomic Environment
Table of Contents
The Socioeconomic Environment refers to a collection of contextual factors that contribute to sustainability and need to be taken into consideration when designing, implementing, and measuring the performance of a health program. A country’s level of economic development and corresponding availability of resources for supporting a health system are important initial factors in determining what levels of sustainability can be realistically achieved in a given time frame. For example, a very low income country will have fewer resources available for sustaining family planning and health programs than will a middle income country. Likewise, a country with a predominantly rural, dispersed population has to overcome barriers to sustainability that are greater than those in a more urbanized country. Socialist countries with little or no private sector development face different sustainability hurdles and issues than countries where private enterprise is more advanced. A country’s level of decentralization—particularly in the public health sector—is yet another consideration. A program’s maturity can determine which sustainability issues are addressed. Such aspects can affect the sustainability and demand of family planning and health systems, and need to be considered at the design phase and throughout implementation of family planning and health programs.
This document does not present an exhaustive list of indicators for measuring sustainability. Rather, it provides indicators that are already in use, indicators for which data may be more readily available, and that capture pertinent aspects of sustainability based on the developed model. The indicators are intended for use as measurement tools, and guidelines on their use are included in the discussion section for each indicator. Many factors, including specific country context, may influence which indicators would best serve as appropriate sustainability measures. In applying the indicators, program managers and program officers should consider the following questions:
(For further information on performance monitoring and the selection criteria for indicators, see Volume I, Health and Family Planning Indicators: A Tool for Results Frameworks.)
The indicators are grouped by component and sub-element. Several suggest disaggregation by sex to monitor progress toward gender equity. Many require the aggregation of data into a combined figure or score to capture an overall picture or program level result. This will vary by program. For example, USAID missions may decide to track data from numerous health centers in a target zone; these data will need to be further manipulated to obtain one final data point to represent the general status of that indicator in the zone. Furthermore, several indicators have not been validated. For example, those indicators that use a score based on a scale are proposed here for mission consideration.
Some indicators in this document are expressed as the “presence” (or “absence”) of a system, an individual, or a policy. In most cases, depending on the program context, it may be possible to define a stronger indicator. The underlying issue of importance for sustainability in most cases is measured in terms of progress in the use of a system, the effectiveness of an individual, and the implementation of a policy which goes beyond simple measurement of “presence.” When possible, this aspect of measuring systems, human resources, and policies is included in the discussion after each indicator.
A. Sustainability of Systems Indicators
Table of Contents
a. Resource Mobilization
Financial sustainability in health systems is: “having enough reliable funding to maintain current family planning and health services for a growing population and to cover the costs of raising quality and expanding availability to acceptable levels.1” To be sustainable, such funding should be generated from a country’s own resources. Financial sustainability assumes a minimum funding level commitment for family planning and health by the government; a “threshold” level of spending on family planning and health must be met before sustainability can be realistically considered. Local resources need to be available to replace donor funding as it declines or is withdrawn. This concept of a threshold is more fully developed in the World Bank’s 1993 World Development Report.
Under financial sustainability are the Public Sector Financing and Private Sector Financing categories, which refer to activities designed for financing health services. Public sector financing consists of the host government and donors’ financial inputs; private sector financing consists of those inputs from all other sources: out-of-pocket expenditures; non-governmental organizations (NGOs), private for-profit providers such as pharmacists, private practice clinicians, and traditional healers; and non-secular organizations such as religious mission-based clinics. (There also may be instances of mixed public and private sector funding, such as government support of private sector groups for service delivery.) The sub-elements, Resource Mobilization and Efficient Allocation and Use of Resources, affect both public and private sector resources in the health system.
Resource Mobilization encompasses the mechanisms that are used to generate financial and related resources for allocation in the health sector. Generally, governments institute policies or programs to mobilize these resources, but such efforts also may emerge through the private sector or through communities. A country’s national budget, generated through tax revenues, represents a principal means of resource mobilization for governments. In the health sector, more and more governments have begun to raise revenues through user fees. Insurance plans and private sector employer-supported health plans represent another such mechanism.
Discussion: The proportion of government expenditures on family planning and health serves as a barometer of the importance a government places on family planning and health issues. A strong positive correlation exists between gross national product (GNP See definition in glossary) per capita and government expenditures on health. Many industrialized countries contribute roughly 10% of their GNP toward health, while many less developed countries contribute close to 4% (1993 World Development Report, pp. 52–53). Identifying past trends in government spending is valuable for assessing future sustainability; therefore, to promote sustainability for most countries, this proportion should be increasing.
Data sources: Ministry of Finance (or equivalent), Ministry of Health
Reference: Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, p. 44.
Discussion: Similar to health expenditures, the proportion of the total health budget financed by donors provides a measure of a country’s dependency on donors to sustain family planning and health activities. As a country’s economy improves, there should be less dependency on donors for financial sustainability, therefore, this proportion should decrease over time. These indicators are also influenced by changes in the exchange rate or by inflation.
Data sources: Ministry of Finance (or equivalent), Ministry of Health
Reference: Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, p. 44.
Discussion: The World Bank (1993 World Development Report, pp. 66–67) estimates per capita health expenditures should be around $21 per annum (for a minimum package of services in all developing countries). Regardless of where this begins, monitoring of this indicator over time should show a progressive increase in expenditures (even after taking inflation into account).
Data sources: Ministry of Finance (or equivalent), Ministry of Health, World Bank European Community Public Expenditure Reviews
Reference: AFR/SD Working Group.
Discussion: Financing diversity increases the likelihood of sustainability because it reduces reliance on a single or small group of donors.
Data sources: Ministry of Health, Ministry of Finance, insurance providers, major private sector employers
Reference: AFR/SD Working Group.
Discussion: The degree of cost recovery is a measure of a health system’s sustainability. As more countries begin to devise cost recovery mechanisms, the proportion of costs recovered as a percent of all costs should increase. Cost recovery and control of the costs recovered are integral parts of sustaining the health system and decreasing dependency on donors.
Data sources: National health accounts, health center records (available through Bamako Initiative reviews)
Reference: AFR/SD Working Group.
Discussion: Cost recovery mechanisms are a form of sustainability in and of themselves. The ability of a service delivery point to retain the costs recovered (even if converted into supplies or salaries) is a measure of the decentralization of the health system. This also measures the degree of autonomy for the SDP.
Data Source: Regional (or equivalent) Health Office or Service Delivery Point statistics
Reference: AFR/SD Working Group.
Discussion: Budget is often allocated at the central level. The degree to which an SDP can allocate its own funds is a measure of decentralization and thus an indication of the sustainability of the health system as a whole.
Data Source: Regional (or equivalent) Health Office or Service Delivery Point statistics.
Reference: AFR/SD Working Group.
b. Efficient Allocation and Use of Resources
Efficient Allocation and Use of Resources must be practiced in order to achieve financial sustainability. Governments need to overcome inefficiencies and stop spending scarce resources on less effective aspects of family planning and health care. Instead, governments should focus resources on the services that do the most good for the greatest number of people at the lowest cost. Primary health care, as opposed to specialized health care provided through tertiary institutions, is a classic example of the way governments can provide services at low cost. Another example of ineffective use of resources is high spending levels on personnel costs, that may reduce the availability of funds for other critical purchases, such as drugs or educational materials. The transfer of resource allocation authority to the local level acknowledges that the organizations and individuals at the “front line” may know better community needs, and thus may be more likely to allocate resources efficiently.
Discussion: The proportion of the government’s health budget directed to primary care is an indication of the intent of the government to spend money on primary care as well as an indicator of political will to support the provision of primary care. See discussion under Indicator 2.
Data sources: Ministry of Finance (or equivalent), Ministry of Health
Reference: Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, pp. 44-45.
Discussion: The referenced text notes: “...these are the areas that governments tend to cut back when their budgets are severely constrained, and these are also sources of funding for many of the programs which have been developed by donor investments. Although these indicators can be used effectively to monitor system-wide sustainability, it is important to recognize that cross-country comparison can be highly misleading. Health systems differ significantly in how they define primary care....” This indicator used in conjunction with Indicator 1 above would further verify political support for primary care by monitoring actual public sector expenditures in this area.
Health expenditures are also likely to vary by the type of facility (primary as opposed to secondary or tertiary), with a disproportionate amount allocated to hospitals and higher level facilities. A similar indicator is “the percent of government health expenditure directed to preventive care or to maternal and child health (MCH) services.” Another could be “primary health care expenditure as a percent of recurrent costs.” (Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, pp. 33, 44.)
Data sources: Ministry of Finance (or equivalent), Ministry of Health
Reference: Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, pp. 44-45.
Discussion: The referenced document notes that this indicator is often applied to government systems to monitor the degree to which economic efficiency, and that jobs are often protected in times of hardship while other services are left to dwindle. Inefficiency is sometimes measured by high personnel expenditures and low expenditures on drugs and other services.
Data Source: Ministry of Health
Reference: Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, p. 36.
2. Institutional Capacity
Table of Contents
A key element of health and family planning delivery system sustainability is the institutional capacity of the organizations providing services. In this model, “institution” is used in its broadest sense and refers to any organization or group that provides health and/or family planning services. Such institutions include ministries of health, private clinics and hospitals, and community health groups.
It is important to distinguish between the current strength of institutions that provide services (which are indicators of the quality of services presented in Volume I, Health and Family Planning Indicators: A Tool for Results Frameworks) and those that are predictive of their future strength (indicators of the sustainability of services). Current institutional capacity is not necessarily predictive of future capacity because trained staff may leave or be promoted, equipment may break, be lost or stolen.
Institutions with well-developed systems are more likely to remain viable in the future than institutions without systems. Thus, the indicators of sustainability in this category refer to the presence of institutional systems that are likely to be used and sustained even when changes occur in institutional staffing. Current effective use of these systems is an indicator of the quality of services. Thus, for example, the presence of a system for assessing staff training needs is an indicator of institutional sustainability, while the current use of that system to help design staff training events is an indicator of quality. Sustainable systems contribute to sustainable quality, which contributes to the population’s satisfaction, positive attitude, and continued use of services.
Well-developed institutional systems usually include processes for regularly collecting information and mechanisms for using the information to make changes in approaches and activities. However, it is not enough to have systems for collecting and using information. It is also important to have clearly defined personnel assignments for implementing the data collection system and to use this information in management decisions.
The sustainable institutional capacity indicators are defined under four categories: Planning and Management, Human Resources, Information Systems, and Logistics Systems. There is a great deal of overlap among these categories. For example, all the systems included under human resources, information systems, and logistics might be considered part of the management system. Similarly, the systems for tracking logistical needs and commodities could also be included under information systems. The four categories are included, however, because they are frequently used in defining elements of institutional capacity. The indicators within each category are not presented in a sequential order (i.e., systems start with strategic plans, then personnel policies, then accounting systems, etc.) but there is a logical basis for the selection of the various elements that contribute to an effective institution (i.e., strategic plans, personnel policies, accounting systems, etc.). These indicators of sustainable institutional capacity can be used individually, or they can be combined to create an index of sustainable capacity. In general, institutions with more well-developed systems (and associated personnel assignments) will be more capable of providing sustained effective services than will institutions with fewer systems.
a. Planning and Management
These indicators examine the presence of systems for assessing the needs and desires of the client population, and for developing, revising, and monitoring the implementation of a strategic plan to address those needs and desires. The indicators are defined generically, and do not refer to any particular type or level of health or family planning institution. They should be applied to those institutions (government ministries, NGOs, etc.) that are the focus of the program’s institution-strengthening activities.
Discussion: Institutions that perform effective strategic planning have a greater chance of understanding and responding to changes in conditions that affect the organization, and are more able to apply effectively available resources to client needs and to generate demand for services. Strategic planning should include inputs from a range of institutional personnel because relevant planning information will be found throughout the institution, and because there should be ownership of the plan throughout the institution. A human resource plan should be included in the strategic plan to ensure that personnel needs (in terms of required skills) are considered in the institution’s planning. Strategic planning that includes human resource needs will help the institution avoid the possibility of management failure due to institutional loss of memory when personnel leave.
Data sources: Institution workplans, periodic reports, strategic planning document
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
Discussion: A strategic plan provides an institution with a description of the overall direction for its efforts and the strategies to be employed in achieving its goals. An institution also needs a more detailed operational plan which sets intermediate goals, defines the personnel and other resources which are required to achieve those goals, and provides a detailed budget. Such operational planning is typically conducted on a yearly basis, which allows for regular assessment of progress towards long-term goals, and for adjustments in approaches and methods. Institutions which perform regular operational planning are more able to adjust to changing conditions, and are more likely to effectively meet the needs of their clients.
Data Source: Institution workplans, periodic reports
Reference: AFR/SD Working Group
Discussion: All institutions providing health and family planning services to clients, even those financed completely through public sources, need to understand client needs and preferences in order to provide effective and useful services. Institutions with systems for regularly assessing client needs and preferences are more likely to generate demand for services, which in turn generates financial and/or political support for the institution.
Data sources: Institution workplans, periodic reports, needs assessment data collection instruments
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
Discussion: For the planning function to be sustained, there needs to be an individual or individuals whose job description explicitly includes planning. A written job description should define the specific activities and responsibilities associated with planning. Written and detailed job descriptions increase the likelihood that the function will be continued in the future, especially when staff leave. Including planning within a written job description also increases the likelihood that the function will be assessed as part of regular staff performance assessments.
Data sources: Written job descriptions, staff interviews to confirm the accuracy of the job descriptions, written staff performance assessments
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
b. Human Resources
These indicators examine institutional systems that are directed to recruiting, training, managing, and evaluating personnel. These functions are sometimes integrated within a single human resource component, and are sometimes separated into various units for personnel management and staff training. The indicators are generic and can be applied to institutions with different funding sources, services, and organizational structures.
Discussion: All institutions operate under a set of policies, rules, and regulations for personnel . This indicator measures the extent to which these policies are written, distributed, and consistently applied. Institutions with written policies, that are distributed and applied consistently tend to have fewer personnel problems, lower staff turnover and are thus better able to focus on client needs than institutions without these policies in force.
Data sources: Written personnel policies, rules, and regulations, staff interviews to assess the consistency with which rules are applied
Reference: Adapted from Family Planning Management Development Project (FPMD), Management and Organizational Sustainabilty Tool (MOST), Management Sciences for Health, September 1998.
Discussion: Institutions with informal systems for determining job assignments and responsibilities are highly reliant on the skills and abilities of current job occupants to assure that all necessary institutional activities are being completed. Institutions with detailed written job descriptions that are validated and regularly updated are more likely to continue to function effectively when job turnover occurs. Detailed job descriptions also increase the likelihood that staff performance assessments will be based on valid objective indicators of performance rather than on personal feelings and general reactions.
Data sources: Written job descriptions, staff interviews to confirm the accuracy of the job descriptions, written staff performance assessments
Reference: Adapted from Family Planning Management Development Project (FPMD), Management and Organizational Sustainability Tool (MOST), Management Sciences for Health, September 1998.
Discussion: Effective staff performance assessment systems will identify strengths and weaknesses in staff performance and will suggest areas where additional management support or staff training is needed to strengthen institutional operations. A regular staff performance assessment, if done constructively, serves to motivate staff and provides an objective means for rewarding effective performance. Institutions with regular staff performance assessments tend to have fewer personnel problems and lower staff turnover than institutions that do not, and thus are more able to provide effective services to clients on a sustainable basis.
Data sources: Written guidelines and forms for staff performance assessment, assessment schedules, staff interviews to confirm the application of the guidelines and assess the usefulness of the process
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
Discussion: The knowledge, skills, and abilities needed by staff are constantly evolving. New institutional functions and activities, new technical requirements for ongoing functions and activities, and staff turnover all affect staff duties. An institution that does not regularly assess staff training needs may lose its ability to provide the client services effectively. A regular training needs assessment will identify staff training needs and may cause an institution to reconfigure staff assignments to meet client needs more effectively.
Data sources: Written guidelines and forms for staff training needs assessment, assessment schedules, staff interviews to confirm the application of the guidelines, interviews with managers to assess the use of training needs assessment data
Reference: Adapted from Training Institutional Development Matrix, Development Associates, Inc., 1997.
Discussion: Training should transfer knowledge, skills, and abilities to staff to improve services to clients. In many cases, however, no test of the validity of this assumption exists. Effective staff training includes measures of whether knowledge, skills, and abilities were actually transferred (e.g., using pre- and post-tests); whether the knowledge transferred is relevant to the objectives and activities of the institution (e.g., through systematic review of training curricula and outcome measures); and whether the new knowledge was actually applied. An institution with a regular system for assessing these measures will gain more from staff training than an institution that does not, and thus will be better able to serve its clients.
Data sources: Written guidelines and forms for assessing the staff training effectiveness, assessment schedules, staff interviews to confirm the application of the guidelines, interviews with managers to assess training assessment data use
Reference: Adapted from Training Institutional Development Matrix, Development Associates, Inc., 1997.
Discussion: For important human resource functions to be sustained in an institution, an individual or individuals must have a job description that explicitly includes personnel and human resource development functions. A written job description will increase the likelihood that human resource activities will be continued, which will in turn improve the sustainability of the institution.
DData sources: Written job descriptions, interviews with staff to confirm their accuracy, written staff performance assessments
Reference: Adapted from Training Institutional Development Matrix, Development Associates, Inc., 1997.
c. Information Systems
These indicators examine the presence of specific Information Systems within an institution that provide management information on revenues, costs, clients, and services. Because institutions maintain separate systems for financial and program information, they are presented here under separate headings.
c.1. Financial Systems
Discussion: An institution cannot sustain operations if it does not have a system for tracking income/revenues and costs. To be most useful, the system should: (1) allow revenues and costs to be allocated to specific service cost categories; (2) generate data on trends in cash flow which can be used in making projections; and (3) allow tracking of expenditures against budgets. When combined with programmatic information, the system should be able to generate per-unit costs for the services offered by the institution. The information provided by financial systems is useful for regular program management and strategic planning. It would be desirable if the system could also track revenues earned and revenue foregone due to exemptions and waivers.
Data sources: Financial reports, cash flow analysis reports, unit cost summaries, financial staff interviews to assess system implementation, managers interviews on use of financial data use
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
Discussion: Financial information systems are only worthwhile if the information from them is used for institutional decision-making. The function is most likely to be sustained in an institution if the activities and responsibilities are included in a written job description of one or more managers. The use of financial data should, in turn, lead to more efficient resource use, and this should improve the quality and sustainability of activities.
Data sources: Written job descriptions, staff interviews to confirm the accuracy of job descriptions, written staff performance assessments
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
c.2. Programmatic Information Systems
Discussion: To function effectively, an institution needs reliable information on its clients and its services. An information system with well-defined methods for collecting, entering, reporting, and analyzing data increases the likelihood that program information will be used for institutional decision-making. This should lead to more effective and sustainable health and family planning services.
Data sources: Client and service summary reports, interviews with information system staff to assess the system, manager interviews on use of program data
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
Discussion: Program information should be used systematically to improve services to formulate policy. Use of data is likely to be institutionalized if responsibility for managing the data system and reviewing information from it is part of the written job description of one or more individuals.
Data sources: Written job descriptions, interviews with staff to confirm their accuracy, written staff performance assessments
Reference: Adapted from the Initiatives Project, Organizational Self-Assessment of Service Sustainability, JSI Research and Training Institute, January 1997.
d. Logistics Systems
These indicators examine the presence of systems in an institution that acquire, maintain, distribute, and replace the resources (vehicles, refrigeration equipment, etc.) and health and family planning services commodities.
Discussion: Institutions need up-to-date information on their resources to develop budgets and strategic plans. An effective system for assessing logistical needs would include a complete resource inventory, the age and condition of the resources, and when the resources should be replaced. Such a system lets managers plan major and minor purchases, and makes it less likely that the logistical network will have major failures or crises.
Data sources: Reports on logistical needs and resources, and interviews with managers on resource review systems.
Reference: Adapted from Family Planning Management Development Project (FPMD), Management and Organizational Sustainability Tool (MOST), Management Sciences for Health, September 1998.
Discussion: Most health and family planning institutions distribute commodities as part of their services. To function effectively and maintain stock, institutions need effective tracking and forecasting systems. Such systems need periodic inventories and a regular method for recording receipt and distribution of commodities. A system that records only receipt and distribution is not sufficient because commodities can be lost or stolen.
Data sources: Inventory and commodity tracking reports, interviews with information system staff to assess implementation, interviews with managers on use of commodity data
Reference: Adapted from Family Planning Management Development Project (FPMD), Management and Organizational Sustainability Tool (MOST), Management Sciences for Health, September 1998.
Discussion: Information about logistics and commodities needs to be used regularly to manage operations. Such use is likely to be sustained if one or more individuals has responsibility for managing logistical tracking systems and for using logistical and commodities data to guide management decisions.
Data sources: Written job descriptions, staff interviews to confirm the accuracy of the job descriptions, written staff performance assessments
Reference: Adapted from Family Planning Management Development Project (FPMD), Management and Organizational Sustainability Tool (MOST), Management Sciences for Health, September 1998.
3. Enabling Environment
Table of Contents
An Enabling Environment engenders sound policy, ensures coordination among donors and host governments to promote efficient and effective use of resources, and guarantees community participation and empowerment. Such an environment fosters sustainable family planning and health services. While the elements of an Enabling Environment are often considered in direct association with institutions and systems, they also apply to the sustainability of demand issues. For example, policies may contribute to improved services, altering attitudes towards services. Efforts to increase community involvement will foster greater community support for family planning, health services and healthy behaviors.
The indicators of a sustainable Enabling Environment are defined in three categories: Policy Process (policy formulation, implementation, and evaluation), Sector-Wide Approaches, and Community Empowerment. As with other aspects of this framework, the categories and their interpretations may overlap and defy clear demarcation. However, there is value in separating and measuring them to illustrate the system dynamics.
a. Policy Process
A policy is defined as: “... a stated plan or course of action designed to influence and determine decisions, actions, and other matters.” Whereas the content of specific policies will be important to sustain family planning and health programs and systems, an underlying functional Policy Process must be integrated with program management and planning. The policy process has three distinct phases: formulation, implementation, and monitoring and evaluation. This process is not just a function of the central government, it also applies to local governments and family planning and health service delivery institutions.
A broad consensus exists on the need for policies to guide health and family planning services delivery and financing. But how are those policies formulated? The roles of information, debate, and consensus, as well as the political environment in which these occur, are critical to policy development. But it is not enough merely to create a policy. Policies must be implemented, enforced and evaluated. Implementation depends largely on political will and action, as well as on good communication, clarity, and the presence of systems and human resources. Policy implementation also depends on the existence of a consistent legal, regulatory and government framework. These systems support policy implementation and enforcement procedures. Without them, policies may remain theoretical and non-functional.
The Policy Process has different stages, many of which overlap. Moreover, the policy process has been described as “messy,” with no clear rules or guidelines. However, some form of dialogue, informed decision-making, and broad-based involvement of people who will be affected by the decisions increase the chance that a system will become sustainable. The indicators in this section attempt to identify whether basic elements of the system are in place and functioning.
Discussion: A policy-making body implies having a regular system for carrying out policy-related tasks. All governmental institutions (particularly at the national level) need to be able to identify issues relevant to a sector, its programs, and its operations; the potential effect of a given issue; possible responses, pros and cons of those responses, and the potential impact of responses; and the resource and operational needs to implement a response. The Ministry of Health at the central level has a partial responsibility to develop policies for the sector in a given country. Institutions that have a system to identify and analyze issues, to generate policy options, and to make recommendations are likely to review complex issues more carefully, and thereby ensure a higher degree of sustainability. It is also important to review the ratio of national versus expatriate individuals conducting this work.
Data sources: Survey of various institutions, workplans, periodic reports, needs assessment data collection, and special (qualitative) studies
Reference: Adapted from the POLICY Project / PASCA Project, AIDS Policy Environment Score, Conceptual Framework, draft, The Futures Group International, May 1997.
Score on a scale:
1 = policy contains elements that undercut the program
2 = policy contradicts another policy
3 = policy delineates key elements of program without apparent conflict
4 = policy specifies key implementation needs (resources, etc.)
5 = policy contains feedback process (e.g., an evaluation plan) that leads to continuous evolution of policy
Discussion: For supportive implementation action to occur in a program area, the first step is the presence of a policy, often at the national level. Some policies are actually contradictory or are counter-productive. Thus, it is important to review a policy and assess whether it helps a program and contributes to its long-term sustainability.
Data sources: Program documents, qualitative analysis
Reference: Adapted from the POLICY Project / PASCA Project, AIDS Policy Environment Score, Conceptual Framework, Draft, The Futures Group International, May 1997.
Score on a scale:
1 = active opposition
2 = weak support
3 = neutral
4 = strong support (rhetoric)
5 = action to implement
Discussion: Effective policy implementation depends on a host of factors. Many need the support of political leaders and/or supervision. Support can be rhetorical or proactive. Support is an incentive for lower echelon personnel, including program managers, to implement the policy. Community and NGOs are also important advocates in this process. As a result, the indicator can be segmented by the specific parties providing the support: government officials, politicians, public opinion leaders, political parties, planning bureaucrats, religious organizations, private sector leaders, NGO leaders, media, and others.
Data sources: Public records (e.g., speeches, government records), media reports, special studies with key informants, qualitative analysis
Reference: Adapted from AIDS Policy Environment Score. Scaling proposed by AFR/SD Working Group.
Discussion: As a practical matter, when policies are implemented, programs and plans are either developed or are amended as a response to the policy and its elements. This indicator may also be presented as the number of programs with clearly stated policies.
Data sources: Program documents, plans, and special studies
Reference: Adapted from the POLICY Project / PASCA Project, AIDS Policy Environment Score, Conceptual Framework, Draft, The Futures Group International, May 1997.
Discussion: Regardless of a policy’s technical merit or the availability of human resources, a policy cannot be implemented without accompanying legal or regulatory statutes. (This element may be furthered by the presence of an effective democracy and governance program.)
Data sources: Legal and regulation text review
Reference: Adapted from the POLICY Project / PASCA Project, AIDS Policy Environment Score, Conceptual Framework, Draft, The Futures Group International, May 1997.
b. Sector-Wide Approaches
Other elements of the sustainability framework concern the effectiveness of an institution and specific financial aspects of service delivery. The role of donors in financing and leveraging national health care programs and systems, however, will remain a large issue for the foreseeable future. The evolution of Sector-Wide Approaches is an attempt to provide governments with more control over donor resources. The underlying principle of SWAPs is that governments develop a comprehensive approach for the country and for the relevant sector, such as a national health plan. This plan then becomes the basis on which donors coordinate their contributions. This approach is designed to empower the MOH to guide the sector’s development in that country. Joint donor planning and programming are becoming increasingly more important at lower levels of the health care system.
The focus on Sector-Wide Approaches recognizes that donors wield a high degree of influence over national policies and, in many cases provide large amounts of financial support. Donors presumably will not cease their support immediately or even in the short-term. This assumes that there are several ways in which donors work, or should work, with one another as well as with the host country government. As donors increase their coordination, and host governments exert control over priorities, sustainability of health and family planning systems should improve. This concept and term was recently adopted by host governments and donors in Africa.
In addition to donor and government relationships, another dimension of sector-wide approaches addresses cross-sectoral relationships critical to sustainable development. For example, programs for diarrheal disease control cannot be sustainable if water sources are not improved. Quantifying these linkages is difficult. As a proxy, Indicator 4 addresses the link between health programs and water and sanitation—an important issue in the health sector, particularly for child survival.
Score on a scale:
1 = no SIP or SWAP process
2 = disagreement over SIP/SWAP direction or no Ministry of Health (MOH) leadership
3 = SIP/SWAP developed, with MOH leadership
4 = agreement over direction, with MOH leadership, but no implementation
5 = SIP/SWAP led by MOH, being implemented
Discussion: The actual presence of a SIP program or SWAP involves a process whereby the national government and donors agree to a commonly derived direction for public health assistance. These processes are managed at the national level and involve multiple donors. Therefore, a SIP is usually a constructive sign of increasing donor-to-donor and donor-government collaborative discussions. Two alternative or additional indicators are the number of plans and programs in which joint donor and donor-government actions are included and the presence of a functioning inter-agency coordinating committee led by the MOH.
Data sources: Survey data, donor information, MOH
Reference: AFR/SD Working Group.
Score on a scale:
1 = no coordination
2 = donors meet periodically, no active collaboration
3 = joint donor and/or donor/local government plans
4 = joint programs launched
5 = donor funding leveraging each other’s programs and other local partners
Discussion: This indicator captures actual donor joint programs and plans in any given regions or subregions. The focus of this indicator is more on a micro level (compared with a SIP) in which a few donors at a sub-national level actively work together to implement programs, along with their government and NGO partners. Donors and governments working together at the implementation level will enhance the chances for long-term sustainability of programs and systems.
Data sources: Program data, Ministry of Health
Reference: AFR/SD Working Group.
Discussion: This indicator measures the emphasis donors place on supporting local talent, which increases the national capacity and hence, systems sustainability. A slight modification of this indicator is to review the percentage of technical assistance as part of all donor assistance. However, the value of this information is questionable other than to illustrate a decline over a period of time. SIPs may increase the requirement of using local resources, but this is unclear at this time.
Data sources: Donor information, program records
Reference: Adapted from Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, p. 42.
Discussion: Long-term sustainability of child survival programs, such as diarrheal disease control, relies on the existence of safe water and effective sanitation programs. It is important to monitor whether the MOH involves other relevant sectors to assure sustainability of health improvements.
Data Source: Review of program and resource alocation documents.
Reference: Environmental Health Project.
c. Community Empowerment
Sustainable system development cannot occur without some community involvement in decision-making and the community’s ability to act within the sector. This empowerment should be a goal in policy formulation and implementation, and is required for policies and practices to be sustained. Without community participation and action within the health sector, program and policy efforts will be stymied. Questions arise as to who “the community” refers to. This will be identified by the Mission as it examines the sub-system or program in question. A community may be viewed on a micro-level, as in the villages in the catchment area of a rural health center, or on a more macro level, as in a national pharmacists association. Although the definition of “community” will vary, involving others outside the formal health care system, particularly clients, is necessary to ensure the long-term sustainability for family planning and health care programs. Community involvement and support are also critical elements of demand generation and sustainability.
Score on a scale:
1 = policy dialogue not permitted by legal text or actual practice
2 = informal system of dialogue exists
3 = one-time survey of views initiated
4 = system for collecting views of many stakeholders established
5 = feedback loop to stakeholders exists
Discussion: An enabling environment for community empowerment and involvement in decision-making is evidenced by whether non-governmental groups are involved in policy formulation. The more the dialogue, the greater the potential for affected groups to support policy and its implementation.
Data sources: Qualitative survey of non-governmental sector’s involvement in policy formulation, presence of a public notification process of pending regulations and/or policies, and/or public forums for policy discussion.
Reference: Adapted from AIDS Policy Environment Score. Scaling proposed by AFR/SD Working Group.
Discussion: Community involvement in health sector decision-making, either at the programmatic level or at the policy level, does not happen automatically. There is typically resistance to such involvement. Hence, the existence of frameworks and operational structures that allow for community involvement is a positive step toward enhancing the potential for sustainability.
Data sources: Program records, Ministry of Health, district or regional health authorities, informed interviews
Reference: Adapted from Towards Well-Functioning Health Districts in South Africa: A Vision and Indicators for Assessing Progress, Centre for Health Policy, University of Witwatersrand, 1997.
Discussion: The greatest degree of community empowerment is the extent to which communities, especially the non-health sectors, are responsible for managing some or all of the affairs of the health sector. Although risks exist in this delegation of responsibility, the sustainability of family planning and health programs should increase as local communities take on more responsibility for the health sector. The types of authority granted to the community reflect evidence of community involvement. Local communities can be defined as appropriate to the USAID program and to the geographic zone in which it exists. It can refer to local government institutions, local health departments, and other structures at the local level that have decentralized responsibility for the health system.
This indicator does not seek an absolute level of responsibility at the local level. Rather, in an era of increasing decentralization, the local community (i.e., governmental structures or local health committees) must also be given certain authorities if it is tasked with larger responsibilities. This indicator assumes that, to date, health systems in Africa have been highly centralized. Alternative indicators that capture similar ideas include the percent of cost sharing revenues retained at the point of service, and the percent of the facility budget programmed at the facility level. (See discussion under Financial Sustainability.) Both of these indicators quantify the notion of authority captured in Indicator 3. A more detailed discussion of institutional systems, such as human resource management and accounting systems, is found under the section Institutional Capacity.
Data sources: Review of community and regional authorities, Ministry of Health, Ministry of Local Government, Ministry of Planning
Reference: AFR/SD Working Group.
Discussion: As a trend indicator, the information derived can illustrate a community’s perspective on the degree to which it feels empowered to participate in the health system’s decisions. It also captures how the degree of participation may alter the community’s view of the health system in general. Similarly, women’s involvement may not happen automatically, and disaggregating data by gender is one way to track progress in women’s participation.
Data sources: Special survey of limited samples across communities, to include different groups
Reference: AFR/SD Working Group.
Discussion: Sustainability of health programs is dependent partly on the support that the health sector receives from other sectors in the community, as well as the degree to which non-health sectors are involved in synergistic programming with the health sector. This two-way relationship should exist at all levels. Emphasis should be placed on documenting two key relationships: those that involve resource allocation decisions (such as Ministry of Finance and Planning; line ministries that allocate funds, e.g., for water and sanitation) and those that involve program decisions (e.g., water and sanitation, agriculture, and nutrition). Attention is given to committees and similar structures since they are easier to document than informal relationships.
Data Source: Sector reviews, local health authorities
Reference: AFR/SD Working Group and Environmental Health Project
B. Sustainability of Demand Indicators
Table of Contents
The concept of Demand appears in two places in the sustainability conceptual framework, first as a basic program component along with Access, Quality, and Sustainability, and second as a sub-component of Sustainability (see Figure 2). In the first context, the concern is with the creation of demand for family planning and health services. In the second context, the focus is on sustaining a sufficient level of demand to maintain family planning and health services and outcomes at acceptable levels. Sustaining demand in this way ultimately should be a function supported and carried out by host country organizations without external donor support.
Actual demand and sustainability of demand are different concepts and are therefore measured differently. With regards to measuring sustainability of demand simple measurement of actual demand is not sufficient. Instead, measuring how well demand is sustained as local resources replace donor support should be the focus. Several dimensions of demand sustainability should be monitored.
1. Ability to Pay
Table of Contents
An essential element of sustainability is that the people being served have the Ability to Pay for the family planning and health products and services they need to maintain good health. Ability to Pay is dependent on the socioeconomic environment (discussed earlier in External Factors); extreme poverty can mean that, unless products and services are almost free of charge, few people will have adequate access. For an equitable health system to be sustainable, knowledge of and access to family planning and health services and practices for the underserved and disadvantaged population need to be taken into account. Ability to Pay is also linked closely to Willingness to Pay; without sustained demand for services, people with means (even if created through the Protection Mechanisms noted below) may not seek services.
a. Protection mechanisms
Access is a critical factor in maintaining population-wide health outcomes. To ensure equitable access, appropriate Protection Mechanisms and/or safety nets need to be in place. Such mechanisms may include means-testing, exemptions, sliding scale fees, and community solidarity funds.
Discussion: Equity is defined mainly in terms of both geographic and economic access or as universal access to a package of basic or cost-effective family planning and health services. In some programs, income inequities are the main focus and indicators are used to measure subsidies received from government health systems. In all cases, equity or protection mechanisms are designed to protect the poor.
Data sources: Ministry of Health
Reference: Partnerships for Health Reform, Measuring Health System Performance: A Handbook of Indicators, September 1997, p. 20.
For demand to be sustainable on a long- term basis, knowledge of and favorable attitudes toward modern health care and health seeking behaviors need to be embedded in local cultures. If healthy behavior is highly valued—that is, deemed extremely important by a culture—then we can assume that people will continue to seek modern family planning and health services and will exhibit appropriate health seeking behavior after donor support is withdrawn. The key method for measuring sustained demand, therefore, is to measure people’s attitudes toward or values placed on healthy behaviors.
The measurement of values poses numerous technical problems, however. It is difficult to identify a tool to capture such an amorphous subject. Survey responses are potentially clouded by the tendency to provide the socially acceptable response; to tell the interviewer what is believed to be the desired or correct response. In addition, such surveys only provide a snapshot of values at a given point in time.
None of the alternatives to measuring values adequately addresses the persistence of demand, which is the essential, sustaining quality that programs seek to measure. Existing methods for measuring willingness to pay, for instance, including contingency valuation models (which attempt to determine the weight of the multiple factors that contribute to an individual’s decision-making), are useful for making programmatic decisions, but do not sufficiently capture the concept of sustained demand as a cultural value.
Assessing knowledge of healthy behaviors has been suggested as a proxy for demand, but just knowing about something does not necessarily indicate a favorable attitude toward it. The relationship between knowledge, attitudes, and behavior also may be variable between sub-sectors of family planning and health. For example, it is reasonable to assume that most people who understand the benefits of immunizing their children against childhood illnesses will have a relatively strong demand for immunization services. It does not necessarily follow that people who understand contraception can be expected to plan their families according to ideals of family size or spacing.
Measuring behavior rather than values is also insufficient because it is only feasible to observe behavior cross-sectionally. Behavior at one point in time, compared with one’s basic values or attitudes, is not predictive of future behavior. People’s professed values do not always correspond with their behavior; people often say they should do one thing, but do quite another. By assessing values, however, we at least obtain an indication of what a local culture believes should be important, and that is probably as close as one can come to predicting the sustainability of demand.
Beyond attempting to measure values and attitudes, to ensure demand sustainability there must be mechanisms in place to perpetuate demand in the population being served. This dimension is captured in the framework by Community Support and Behavior Change Communication (BCC). Willingness to Pay attempts to provide an indication of the value (measured in terms of financial, opportunity, or other costs) people place on the use of family planning and health services and the practice of healthy behaviors.
a. Community Support
Community Support is crucial in maintaining long-term demand for family planning and health services. Community mobilization and participation in management and support of the local health system increases the likelihood of sustainability. Social norms have a role in perpetuating demand for family planning and health services and healthy behaviors; these are fostered and maintained by active community involvement in family planning and health delivery. Such support promotes demand both for services at the health facility level and for behavioral changes within households and by individuals.
Discussion: Community support is crucial in maintaining long-term demand for family planning and health services. The existence of health committees in communities is a necessary, but not sufficient, condition for community support. These committees must be active and representative of the population (women in particular should be present on the committees). While measuring the activities of community health committees is not practical, one can gain at least some indication that they are functioning if they hold regular meetings. What constitutes “functional” should be further defined at the country level.
Data source: Community surveys
Reference: AFR/SD Working Group.
Discussion: While community-based distribution (CBD) is primarily a strategy for improving access to family planning, it may also be instrumental in building community support for certain kinds of other health services such as maternal health care. Even if the case for CBD having a positive effect on community support cannot be made conclusively, CBD may at least provide some indication of community acceptance of modern health practices. But CBD is not the ideal medium for the delivery of all kinds of family planning and health services, and one should not interpret the inclusion of this indicator in the sustainability model as a suggestion that CBD should be a key element of every country strategy.
Data source: Community surveys
Reference: AFR/SD Working Group.
Discussion: The essential idea in this indicator is to assess how important the population thinks it is to practice modern family planning and health behaviors. The indicator is presented in a general form to allow special modifications to fit different program strategies and monitoring concerns. To gain an indication of how the population views modern family planning and health practices, one could include multiple questions, each addressing a particular aspect of health behavior, then calculate a cumulative average of responses over the set of questions. One could also structure the questions to relate solely or primarily to those aspects of health behavior that the country program is specifically designed to promote.
A five-level Likert-type response scale set is also recommended for this indicator, with a range of “extremely important,” “somewhat important,” “uncertain,” “somewhat unimportant,” and “extremely unimportant.” Reporting only the percentage of respondents who say such family planning and health practices are “extremely important” is recommended, because this is the attitude sought to sustain demand for health and family planning services.
Analysis of the other responses, and of the composition and correlates of the population’s responses, can also provide useful information for program decision-making. For example, if there is a high percentage of respondents who say they “don’t know” or are neutral about the importance of these health behaviors, it would suggest that the communications mechanisms are inadequate. It may also prove analytically useful to include in the survey other value questions that pertain to other areas of life besides health; for example, food security and children’s education. These non-health value questions can provide a basis for comparing the relevant importance of health to other areas.
The measurement of values related to family planning and health services and behaviors could provide useful information if the results are disaggregated by the sex of the respondent. For example, BCC strategists might find it useful to know whether men or women tend to value health more, or whether there are differences by gender in the ranking of health issues.
Data sources: Household surveys
Reference: AFR/SD Working Group.
b. Behavior Change Communication
The objective is to measure the extent to which there exist stable, effective, locally-based and supported mechanisms for stimulating demand or for promoting behavior change for healthy practices. The presence of a local organization capable of designing, implementing, and evaluating a communication campaign on a health topic, for instance, increases the sustainability of Behavior Change Communication efforts.
Discussion: The first step toward a permanent mechanism for educating and persuading a population to follow modern health practices is the existence of a communication strategy with the population.
Data sources: Work plans, Ministry of Health
Reference: AFR/SD Working Group.
Discussion: The key aspect of this indicator is that the BCC program receives financial support from the government, and is not wholly dependent on donor support or the private sector. Public funding demonstrates the Ministry of Health’s commitment to educate the population and to motivate them to adopt modern family planning and health practices.
Data sources: Ministry of Health, Ministry of Planning
Reference: AFR/SD Working Group.
Discussion: In addition to a long-term BCC strategy and financial support for BCC, an implementation system is needed. This system could include the government, non-governmental organizations, and private sector firms. It is important that the means exist for carrying out the long-term BCC strategy. While this indicator is dichotomous in its basic form (because it simply measures the presence or absence of such a system), use of this indicator should be accompanied by some qualitative description and evaluation of the implementing system.
Data sources: Ministry of Health, Ministry of Planning, Country Agreements (between NGOs and governments)
Reference: AFR/SD Working Group.
c. Willingness to Pay/Use
Assuming that a sufficient proportion of the population to be served has the Ability to Pay for products and services, the question then becomes whether or not people are willing to pay for them. Are people willing to pay the opportunity costs involved in practicing healthy behaviors? Is the community willing to support the provision of family planning and health services, both financially and otherwise? These attitudes are important sustainability components.
In life-or-death situations, the demand for family planning and health services may be highly inelastic, or remain great regardless of the associated costs, but preventive care is another matter. People may not be willing to devote an adequate portion of their incomes to preventive health care unless they have learned to appreciate its value. They must also regard local family planning and health care providers as competent, courteous, and effective. A local health facility needs more than the surrounding community financial support to remain viable.
Healthy behaviors begin in the household and do not always require a financial investment to practice (e.g., breastfeeding). Nevertheless, they require a commitment that implies a positive attitude toward and understanding of the associated health impact. All these attitudinal factors can be influenced by Behavior Change Communication, which can increase support and improve attitudes to use and pay for family planning and health products and services when necessary. BCC also can play a critical role in perpetuating positive home health care behaviors.
Discussion: This indicator represents an attempt to apply the concept of demand elasticity for family planning and health services to the question of sustaining demand. Elasticity is the extent to which the demand for goods or services responds to changes in costs. The more elastic demand is, the more responsive it will be to changes in costs, decreasing as costs rise and increasing as costs fall. If demand is inelastic, it tends to remain constant as costs fluctuate or to change slowly in response to varying costs. The premise of this concept is that inelastic demand is indicative of sustainable demand. If people continue to seek family planning and health services regardless of varying costs for obtaining those services, then one can conclude that people think the services are essential, or at least very important.
Measuring demand elasticity using only monetary costs of services is likely to provide misleading or ambiguous data, because some segment of the population can be excluded because of inability, rather than unwillingness, to pay for services. Measuring other obstacles, such as accessibility, as costs for services is a more equitable measure.
The indicator in this model uses distance to a primary health care facility as a measure of cost in terms of time and energy required to travel to and from the facility. If the use rates for the two distances are similar, one could conclude that demand for family planning and health services is relatively inelastic. If they are dissimilar, this suggests that demand is elastic and therefore less sustainable. The question of what constitutes “similarity” of use rates needs to be worked out through research and analysis, but also needs to be determined locally.
The phrase “in selected communities” is important because it is recognized that it is not appropriate or valid to apply this measure to all communities. Physical infrastructure, size, and community composition need to be taken into account. A survey to measure the demand elasticity in this manner should use a sampling method that is restricted to only those communities where a reasonable comparison of use rates by distance from the facility can be expected.
Data sources: Household surveys, health facility records, Ministry of Planning
Reference: AFR/SD Working Group.
Discussion: Although using this indicator will be limited to those instances in which a program covers the period surrounding this policy change, the lack of a drop in use rates could signify clients’ willingness to pay for the same services they previously received for free. However, in most cases demand will drop initially then rise again as the population adjusts to paying fees for services. Therefore it will be important to track this indicator over time. This situation controls, to a degree, potential changes in quality (because it monitors only a brief time span). It may be confounded by the issue of ability to pay.
Data sources: Health facility records, Ministry of Health
Reference: AFR/SD Working Group.
| Capacity Building: | A set of activities and actions that assists the receiving institution or individual to enhance its ability, competence, and aptitude to plan, implement, and evaluate programs or policies. |
| Community-Based Distribution (CBD): | CBD programs rely on a network of trained volunteers or paid workers who reside and work in their communities and serve as a conduit for the sale or distribution of non-prescription family planning and health products, such as condoms or ORS packets. CBD workers often make home visits, provide health education and make referrals. Such programs increase access to health services, especially for communities distant from health facilities. |
| Community Empowerment: | Community empowerment refers to the level of involvement a community has in the decision-making process related to the goods and services available to its people. |
| Community Solidarity Funds: | Community solidarity funds are a protection mechanism to increase access to familyplanning and health services for the poor. Communities contribute money to a central fund to guarantee the availability of resources to cover health costs for indigent members of the population. Solidarity funds also may be used to help cover urgent care costs, such as the evacuation of emergency cases. |
| Contingency Valuation Models: | These models attempt to determine the weight of the multiple factors that contribute to an individual’s decision-making, or place a value on each of the contingencies that may influence behavior. Contingency valuation models attempt to predict behavior based on these factors. |
| Cross -subsidies: | Funds generated by one aspect of the health delivery system used to fund another program. For example, fees collected for curative services being used to fund preventive health education. |
| Decentralization: | A policy and planning process whereby a government shifts the authority, budgetary control, and responsibility for personnel within the public and private sectors in the nation. The degree and form of decentralization vary from country to country. “Rational” decentralization refers to a process which includes thorough planning that considers the positive, negative, and inintended consequences of various alternatives and continues to evaluate the evolution of a given policy and to respond to findings. It is the process of devolving planning, management, and evaluation authority and responsibility to lower levels of a system, such as from central government to district or provincial governments. |
| Efficiency: | This concept refers to the optimal utilization of resources and has the following three dimensions: allocative, technical, and economic. Each dimension is defined below. |
| Allocative Efficiency: | A health system is allocatively efficient when the marginal social benefit of the last unit of service it produces is equal to its marginal social cost. Alternatively stated a system is allocatively efficient if its resources are employed in those areas whose products/services provide relatively higher returns to the goals set by the system. For instance, investments in primary health care are believed to generate higher returns in terms of reduction in morbidity and mortality than investments in curative care. Therefore, it would be more efficient for governments to allocate resources to invest in primary health care than in curative care. |
| Technical Efficiency: | A system is technically efficient when it produces the maximum level of output/service for a given set and level of inputs. |
| Economic Efficiency: | A system is economically efficient when it uses input combinations which permit it to produce a given level of services at the lowest cost. |
| Effectiveness: | A system is effective if it is organized and its resources are arranged in such a way that it achieves its stated objectives. Cost-effectiveness refers to the process of achieving a stated objective in the least costly manner. |
| Elasticity: | Elasticity is the extent to which demand for goods or services responds to changes in costs. The more elastic demand is, the more responsive it will be to changes in costs, decreasing as costs rise and increasing as costs fall. If demand is inelastic, it tends to remain constant as costs fluctuate or change slowly in response to varying costs. |
| Enabling Environment: | The sum of conditions necessary to encourage policies and strategies to be developed, tested, adopted, implemented, and evaluated in constructive ways. This includes those conditions and actions that serve to reduce barriers to the review and adoption of policies, as well as those that proactively work to create positive conditions supporting the further evolution of the given policy and/or strategy. It is about an environment (and its component parts) that empowers or permits a policy or strategy to evolve. |
| Equity: | Comparable access to health services for all population groups. |
| Financial Sustainability: | Adequate reliable funding to maintain current family planning and health services which produce desired health status results for a growing population and which cover the costs of raising quality and expanding availability to acceptable levels. |
| Government Equalization Grants: | Grants which equalize resource allocation between regions or geographic areas (rural/urban). |
| Gross Domestic Product (GDP): | The total output of goods and services for final use produced by residents and non-residents in an economy, regardless of the allocation to domestic and foreign claims. It does not include deductions for depreciation of physical capital or depletion and degradation of natural resources. |
| Gross National Product (GNP): | GDP plus net factor income from abroad, less similar payments made to non-residents who contribute to the domestic economy. |
| Institution: | Any organization or group that provides health and/or family planning services. Such institutions include ministries of health, private clinics, hospitals, and community health groups. |
| Likert Scale: | A range of possible responses to a question, from one extreme to the other, giving a corresponding ordinal ranking of possible answers. |
| Means-testing: | The process of assessing ability to pay for a given set of goods or services based on income and/or wealth. |
| Policy: | A stated plan or course of action designed to influence and determine decisions, actions, and other matters. |
| Private Sector: | Inputs from all sources other than the public: from NGOs; private for-profit providers such as pharmacists, private practice clinicians, and traditional healers; and non-secular organizations such as religious, mission-based clinics. |
| Protection Mechanisms: | To ensure equitable access, appropriate protection mechanisms and/or safety nets need to be put in place, including means-testing, exemptions, sliding scale fees, and community solidarity funds. In some programs, income inequities are the main focus, and indicators are used to measure subsidies for services offered by government health systems. In all cases, equity or protection mechanisms are designed to protect the poor. |
| Public Sector: | Inputs from the host government and donors. |
| Sector-Wide Approaches: | Sector-wide approaches represent an increasing attempt to give governments more control over donor resources. Sector Investment Programs (SIPs) are one illustration. The presence of a SIP program implies a long process whereby the national government and donors agree to a commonly derived direction for public health assistance. Joint donor planning and programming is becoming increasingly important at lower levels of health care systems. |
| Sliding Scale Fees: | This is an example of a protection mechanism. Sliding scale fees represent the practice of adjusting fees for services based on certain criteria, typically the income level of the client. Such flexible fees represent a subsidy provided to increase access to services for the poor who might otherwise be unable to afford services. |
| Sustainability: | The ability of host country entities (community, public and/or private) to assume responsibility for programs and/or outcomes without adversely affecting the ability to maintain or continue program objectives or outcomes. In this document, the emphasis is on sustaining health status, and the sustainability definition encompasses the need for family planning and health programs to change and adapt based on constantly processed information about the setting and needs of the population served. |
A. Sustainability of Systems Indicators
1. Financial Sustainability
a. Resource Mobilization
Indicator 1: Government health expenditure as a percent of GDP
Indicator 2: Percent of total health expenditure financed by donors
Indicator 3: Total per capita expenditure on health
Indicator 4: Sources of financing for health and their relative shares of total expenditure
Indicator 5: Percent of total health expenditure recovered through various mechanisms of cost sharing
Indicator 6: Percent of cost sharing revenues retained at a point of service
Indicator 7: Percent of facility budget programmed at facility level
b. Efficient Allocation and Use of Resources
Indicator 1: Percent of government health budget allocated to primary care
Indicator 2: Percent of government health expenditures directed to primary care
Indicator 3: Personnel expenditure as a percent of total recurrent health expenditure
2. Institutional Capacity
a. Planning and Management
Indicator 1: Presence of a strategic plan that includes: (A) a mission statement for the institution; (B) strategies for the near term (e.g., 5 years), including budgets and priorities; (C) a vision for the institution for the long term (beyond 5 years); (D) a human resource plan defining staffing and training needs; and (E) evidence of participation in planning from a broad range of personnel within the institution
Indicator 2: Presence of a system for preparing yearly operational plans for the institution including operational goals, personnel requirements, and budgets
Indicator 3: Presence of a regular system for assessing the needs and preferences of clients and for adjusting services in response to identified changes
Indicator 4: Presence of a manager whose job description includes assessing clients’ needs and desires, for developing the strategic and operational plan, for revising the plan, and for assessing the operationalization of the plan
b. Human Resources
Indicator 1: Written personnel policies, rules, and regulations that are consistently applied
Indicator 2: Presence of detailed, accurate, and up-to-date job descriptions
Indicator 3: Presence of a system for regular staff performance assessment
Indicator 4: Presence of a system for the regular assessment of staff training needs
Indicator 5: Presence of a system for assessing the effectiveness of staff training in terms of the quality of the training provided, and the appropriateness of the training for institutional needs
Indicator 6: Presence of a manager whose job description includes reviewing and revising job descriptions, personnel rules and regulations and, assessing performance, training needs, and training outcomes
c. Information Systems
c.1. Financial Systems
Indicator 1: Presence of an accounting system that regularly provides income/revenue data and cash flow analysis based on specific service cost categories
Indicator 2: Presence of a manager whose job description includes reviewing financial data, analyzing unit costs, making financial projections, and tracking expenditures against budgets
c.2. Programmatic Information Systems
Indicator 3: Presence of an information system that provides reliable information on clients and services
Indicator 4: Presence of a manager whose job description includes managing the programmatic information system and for using information on clients and services for management and policy purposes
d. Logistics Systems
Indicator 1: Presence of a system for periodically reviewing the logistical needs and resources of the institution (offices, vehicles, etc.)
Indicator 2: Presence of a system for tracking commodities and forecasting needs, including a periodic inventory and regular reporting of receipt and distribution of commodities
Indicator 3: Presence of a manager whose job description includes periodic review of resource needs and tracking of commodities
3. Enabling Environment
a. Policy Process
Indicator 1: Presence of a policy-making body that conducts the necessary analysis of an issue and formulates a policy
Indicator 2: Presence of a national policy that supports the health program objective(s)
Score on a scale:
1 = Policy contains elements that undercut the program
2 = Policy contradicts with another policy
3 = Policy delineats key elements of program without apparent conflict
4 = Policy specifies key implementation needs (resources, etc.) 5 = Policy contains feedback process (e.g., an evaluation plan) that leads to continuous evolution of that policy
Indicator 3: The degree of support provided for a given policy
Score on a scale:
1 = Active opposition
2 = Weak support
3 = Neutral
4 = Strong support (rhetoric)
5 = Action to implement
Indicator 4: Specific strategies and program goals exist in response to a given policy
Indicator 5: Presence of a legal and regulatory framework for the given policy
b. Sector-Wide Approaches
Indicator 1: Existence of a Sector Investment Program (SIP), Sector-Wide Approach (SWAP) or similar program
Score on a scale: 1 = No SIP process
2 = Disagreement over SIP direction or no Ministry of Health (MOH) leadership
3 = SIP developed, with MOH leadership
4 = Agreement over direction, with MOH leadership, no implementation
5 = SIP led by MOH, being implemented
Indicator 2: Joint donor plans and programs in given regions
Score on a scale: 1 = No coordination
2 = Donors meet periodically, no active collaboration
3 = Joint donor and/or donor/local government plans
4 = Joint programs launched
5 = Donor funding leveraging each other’s programs and other local partners
Indicator 3: Ratio of technical assistance funded by donors that uses foreign versus national talent
Indicator 4: Presence of a cross-sectoral strategy for diarrheal disease control that includes water and sanitation components.
c. Community Empowerment
Indicator 1: Policy dialogue and formulation involves NGOs, community leaders, and representatives of the private sector and special interest groups
Score on a scale:
1 = Policy dialogue not permitted by legal text or actual practice
2 = Informal system of dialogue
3 = One-time survey of views initiated
4 = Established system for collecting views of many stakeholders
5 = Feedback loop to stakeholders exists
Indicator 2: Number of regions in which operate community decision-making structures operate to discuss health concerns and/or decide on program management issues
Indicator 3: Percentage of local communities with the following authorities: hiring/firing of staff, revenue generation and retention, and budget planning and implementation
Indicator 4: Community satisfaction with a) their level of input into health system decision-making, and b) the health system in general (disaggregated by men and women)
Indicator 5: a) Presence of health sector representation on non-health local/national government committees and/or b) presence of non-health sector representation on local/national health committees.
B. Sustainability of Demand Indicators
1. Ability to Pay
a. Protection mechanisms
Indicator 1: Existence of exemptions, waivers, or other protection mechanisms (cross-subsidies, government equalization grants).
2. Attitude
a. Community Support
Indicator 1: Percent of communities having functional local health committees that hold regular meetings
Indicator 2: Percent of communities having some form of community based distribution system
Indicator 3: Percent of the population (or men and women) that say it is “extremely important” [to practice preventive health behavior], e.g.:
(A) To have children fully immunized against childhood illness by their first birthday;
(B) For expectant mothers to make prenatal visits to a health facility;
(C) To space the births of their children;
(D) To use accepted preventive measures against IDS/STDs
b. Behavior Change Communication
Indicator 1: Existence of a long-term Behavior Change Communication (BCC) strategy for perpetuating demand for health and family planning services
Indicator 2: Presence of a Behavior Change Communication (BCC) program directed toward health and family planning services and behaviors that is at least partially funded by the host country government
Indicator 3: Presence of a government-endorsed organization responsible for implementation of a long-term strategy of Behavior Change Communication (BCC) for health and family planning services
c. Willingness to Pay/Use
Indicator 1: Difference between the percentage of households within 5 km. of a health facility using that facility and the percentage of households from 5.1 to 10 km. from the facility using that facility in selected communities
Indicator 2: Change in service delivery point use rates in the month prior to and following a change from no cost service delivery to a fee-for-service policy
Abt Associates Inc. Measuring Health System Performance: A Handbook of Indicators. Partnerships for Health Reform. September 1997.
Abt Associates Inc. Issue Briefs Topics 1-5. Health Financing and Sustainability Project, USAID.
BASICS. Guide on Use of Sustainability Indicators for Immunization. December 1996.
Bossert T., et al. Sustainability Strategy. University Research Corp. December 1990.
Brinkerhoff D. Enhancing the Capacity for Strategic Management of Policy Implementation in Developing Countries. Management Systems International. Implementing Policy Change Project. September 1996.
Brinkerhoff D, Goldsmith A. Analyzing the Sustainability of Development Institutions: Applications to the Agriculture and Health Sectors. University of Maryland International Development Management Center and University of Massachusetts College of Management. June 1990.
Child Survival Indicators Working Group. Inventory of Recommended Child Survival Indicators and "Core" Indicators. USAID. August 1996.
Crosby B. Organizational Dimensions to the Implementation of Policy Change. Management Systems International. Implementing Policy Change Project. September 1996.
Development Associates Inc. Training Institutional Development Matrix. 1997.
Durch J, Bailey L, Stoto M, ed. Improving Health in the Community (Summary). A Role for Performance Monitoring. Institute of Medicine. National Academy Press. 1997.
Esselman J. Sustainability, Sustainable Development and the Health Sector. USAID/CDIE. October 1994.
Fishstein P, Helfenbein S. Planning for Sustainability and Tools to Help. Management Sciences for Health. July 1995.
Foreit K, Lacey L. Toward an Analytical Framework for Sustainability of Family Planning and Health Services in Bangladesh. Options for Population Policy. August 18–September 14, 1991.
Gilson L, Balfour T, Goosen V. Towards Well-Functioning Health Districts in South Africa: A Vision and Indicators for Assessing Progress. University of Witwatersrand Centre for Health Policy. March 1997.
Goldsmith A, Gustafson D, Ingle M, Adoum C. Enhancing the Sustainability of Development Impact: Supplemental Guidance for AID Mission and Bureau Staff. July 1991.
JSI Research and Training Institute. Organizational Self-Assessment of Service Sustainability. Initiatives Project. January 1997.
IPPF/WHR. Working Statement on Sustainability.
Knight R, Tsui A, Mauldin W. Family Planning Sustainability at the Outcome and Program Levels: Constructing Indicators for USAID Strategic Planning. The University of North Carolina. The Evaluation Project. July 1997.
Levine R, Bennett J. Sustainability of Family Planning Programs and Organizations: Meeting Tomorrow's Challenges. Options for Population Policy. January 1995.
Management Sciences for Health. Management and Organizational Sustainability Tool (MOST), Family Planning Management Development Project (FPMD). September. 1998.
Management Sciences for Health. "Planning for Sustainability: Assessing the Management Capabilities of Your Organization". The Family Planning Manager. Family Planning Management Development Project (FPMD). Winter 1996/97.
McPake B, Kutzin, J. Methods for Evaluating Effects of Health Reforms. World Health Organization. July 1997.
Negrette C. Bogota for APH/A Project Design. PROFAMILIA. February 1995.
The Futures Group International. AIDS Policy Environment Score, Conceptual Framework, (draft) POLICY Project / PASCA Project. May 8, 1997.
Population Technical Assistance Project. Financial Sustainability Tool. (draft ) May 1995.
Russell Diane. Issue Paper 5: Theory and Practice in Sustainability and Sustainable Development. USAID/CDIE. May 1995.
Sclafani A. A Client-driven Model. JSI Research and Training Institute. Initiatives Project (via the World Wide Web). August 1997.
Trias, M. et al. Sustainability through Diversification and Cost Recovery—The Case of PROFAMILIA Colombia. June 1994.
Tsui A. An Update on the Development of the Sustainability Index. Memorandum. The University of North Carolina. The Evaluation Project. April 1996.
USAID. An SO4 Strategy for Presenting and Reporting on NGO/CBO Capacity Building Activities: A Concept Paper and Proposal to the IWG. April 1998.
USAID. Glossary of Terms. AFR/SD 1997.
USAID Africa Bureau's Office of Sustainable Development's (AFR/SD). Health and Family Planning Indicators: ATool for Results Frameworks. Volume I. July 1999.
USAID/CDIE. The Role of Evaluation in USAID. Performance Monitoring and Evaluation TIPS. 1997.
Williams, T. Sustainability: What it Means and How it Affects IPPF
Grant Recipients in Latin America. IPPF/WHR B Transition Project.