2019 Federal Standard of Excellence

Evaluation & Research

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY19?

Score
7
Administration for Children and Families (HHS)
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • ACF’s evaluation policy confirms ACF’s commitment to conducting evaluations and using evidence from evaluations to inform policy and practice. ACF seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. ACF established the policy in 2012 and published it in the Federal Register on August 29, 2014.
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • OPRE annually identifies questions relevant to the programs and policies of ACF and proposes a research and evaluation spending plan to the Assistant Secretary for Children and Families. This plan focuses on activities that the Office of Planning, Research, and Evaluation plans to conduct during the following fiscal year.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
2.4 Did the agency publicly release all completed program evaluations?
  • ACF’s evaluation policy requires that “ACF will release evaluation results regardless of findings…Evaluation reports will present comprehensive findings, including favorable, unfavorable, and null findings. ACF will release evaluation results timely – usually within two months of a report’s completion.” ACF has publicly released the findings of all completed evaluations to date. In 2018, OPRE released nearly 130 research publications. OPRE publications are publicly available on the OPRE website.
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • Coverage: ACF conducts research in areas where Congress has given authorization and appropriations. Programs for which ACF is able to conduct research and evaluation using dedicated funding include Temporary Assistance for Needy Families, Health Profession Opportunity Grants, Head Start, Child Care, Child Welfare, Home Visiting, Healthy Marriage and Responsible Fatherhood, Personal Responsibility Education Program, Sexual Risk Avoidance Education, Teen Pregnancy Prevention, Runaway and Homeless Youth, Family Violence Prevention Services, and Human Trafficking services. These programs represent approximately 85% of overall ACF spending.
  • Quality: ACF’s Evaluation Policy states that ACF is committed to using the most rigorous methods that are appropriate to the evaluation questions and feasible within budget and other constraints, and that rigor is necessary not only for impact evaluations, but also for implementation or process evaluations, descriptive studies, outcome evaluations, and formative evaluations; and in both qualitative and quantitative approaches.  
  • Methods: ACF uses a range of evaluation methods. ACF  conducts impact evaluations as well as implementation and process evaluations, cost analyses and cost benefit analyses, descriptive and exploratory studies, research syntheses, and more. ACF is committed to learning about and using the most scientifically advanced approaches to determining effectiveness and efficiency of ACF programs; to this end, OPRE annually organizes meetings of scientists and research experts to discuss critical topics in social science research methodology and how innovative methodologies can be applied to policy-relevant questions.
  • Effectiveness: ACF’s Evaluation Policy states that ACF will conduct relevant research and disseminate findings in ways that are accessible and useful to policymakers and practitioners. OPRE engages in ongoing collaboration with ACF program office staff and leadership to interpret research and evaluation findings and to identify their implications for programmatic and policy decisions such as ACF regulations and funding opportunity announcements. For example, when ACF’s Office of Head Start significantly revised its Program Performance Standards—the regulations that define the standards and minimum requirements for Head Start services—the revisions drew from decades of OPRE research and the recommendations of the OPRE-led Secretary’s Advisory Committee on Head Start Research and Evaluation. Similarly, ACF’s Office of Child Care drew from research and evaluation findings related to eligibility redetermination, continuity of subsidy use, use of funds dedicated to improving the quality of programs, and other information to inform the regulations accompanying the reauthorization of the Child Care and Development Block Grant.
  • Independence: ACF’s Evaluation Policy states that independence and objectivity are core principles of evaluation and that it is important to insulate evaluation functions from undue influence and from both the appearance and the reality of bias. To promote objectivity, ACF protects independence in the design, conduct, and analysis of evaluations. To this end, ACF conducts evaluations through the competitive award of grants and contracts to external experts who are free from conflicts of interest; and, the Deputy Assistant Secretary for Planning, Research, and Evaluation, a career civil servant, has authority to approve the design of evaluation projects and analysis plans; and has authority to approve, release, and disseminate evaluation reports.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
Score
7
Administration for Community Living
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • ACL’s evaluation policy confirms ACL’s commitment to conducting evaluations and using evidence from evaluations to inform policy and practice. ACL seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. The policy addresses each of these principles.
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • ACL is drafting an agency-wide evaluation plan that will be published on the ACL website once finalized. ACL’s evaluation plan is based on the results of developing an ACL-wide Learning Agenda. From the list of priority questions generated, the ones determined to be best addressed through evaluation were compiled into an agency-wide evaluation plan that briefly describes how evaluation research will be used to answer the relevant priority questions and also places the planned evaluations into the larger agency context. 
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • ACL has adopted a learning agenda approach, which involves annual reviews with each ACL center to support the generation and use of evaluation findings to inform agency strategies and decision making. The approach was implemented between September 2018 and December 2019. The result will be a written learning agenda that will guide ACL’s evaluation and performance management work based on broad input from ACL leadership and staff as well as external stakeholders.
2.4 Did the agency publicly release all completed program evaluations?
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • ACL selects evaluation contractors through a competitive acquisition process. Staff of the Office for Performance and Evaluation (OPE) oversee the evaluations and act as liaisons between program staff and evaluation contractors. While program staff are encouraged to write forwards for evaluation reports, they do not influence the findings. ACL’s evaluation policy states that ACL will conduct evaluations through the competitive award of grants and contracts to external experts who are free from conflicts of interest. Further, the Director of the Office of Planning and Evaluation has authority to approve the design of evaluation projects and analysis plans as well as to approve, release and disseminate evaluation reports. While all ACL programs are required to report on program results, over the past 10 years ACL has evaluated or is in the process of evaluating at least one program in 55% percent of its major budget lines. The programs least likely to be evaluated are resource centers and projects like State Councils on Developmental Disabilities (Councils) that are self-governing organizations charged with identifying the most pressing needs of people with developmental disabilities in their state or territory. Evaluation data have been used to improve programs through updating funding requirements and refocusing technical assistance.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
  • ACL typically funds evaluation design contracts, such as those for the Older Americans Act Title VI Tribal Grants Program evaluation and the Long Term Care Ombudsman Evaluation, that are used to determine the most rigorous evaluation approach that is feasible given the structure of a particular program. While the Tribal Grants Program and the Ombudsman program are full coverage programs, where comparison groups are not possible, ACL most frequently uses propensity score matching to identify comparison group members. This was the case for the Older Americans Act Nutrition Services Program and National Family Caregivers Support Program evaluations and the Wellness Prospective Evaluation Final Report conducted by CMS in partnership with ACL and published in January 2019. 
  • ACL is currently working with the Corporation for National and Community Service to design an evaluation of CNCS grantees serving older adults that are using evidence-based programs highlighted by ACL. The purpose of this joint work is to build the evidence base in the field and to improve the use of evidence-based programming by ACL and CNCS grantees. 
  • ACL’s NIDILRR funds the largest percentage of ACL’s RCTs. Of their 121 research projects being conducted by grantees, 19% (23/121) are employing a randomized clinical trial (RCT) or “true experimental” design. To ensure research quality, NIDILRR adheres to strict peer reviewer evaluation criteria that are used in the grant award process (see part (c) for details on rigor of research projects and part (d) for details on the design of research projects). In addition, ACL’s evaluation policy states that “In assessing the effects of programs or services, ACL evaluations will use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, high-quality quasi-experiments offer an alternative.”
Score
8
U.S. Agency for International Development
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • USAID has an agency-wide evaluation registry that collects information on all evaluations planned to commence within the next three years (as well as tracking ongoing and completed evaluations). Currently, this information is used internally and is not published. To meet the Evidence Act requirement, USAID will publish an agency-wide evaluation plan in the Agency’s Annual Performance Plan/Annual Performance Report in future years. 
  • In addition, USAID’s Office of Learning, Evaluation, and Research works with bureaus to develop internal annual Bureau Monitoring, Evaluation and Learning Plans that review evaluation quality and evidence building and use within each bureau, and identify challenges and priorities for the year ahead.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • USAID has an agency-wide learning agenda called the Self-Reliance Learning Agenda (SRLA). The SRLA prioritizes evidence needs related to the Agency’s mission to foster country self-reliance which covers all development program/sector areas, humanitarian assistance and resilience, and agency operations. This vision and mission is articulated in USAID’s Policy Framework and reorients the Agency’s programs, operations, and workforce around the vision of self-reliance or ending the need for foreign assistance.
  • USAID used a strongly consultative process for developing SRLA, as described in the SRLA Fact Sheet. First, the Agency compiled learning questions from a number of feedback processes to initially capture 260 questions which through consultations were reduced to the final to thirteen that represent the Agency’s priority learning needs related to Self-Reliance.
  • Now that the questions are published, USAID will partner with internal and external stakeholders to generate and gather evidence and facilitate the utilization of learning. These stakeholders include USAID’s implementing partners, other U.S. agencies, private coalitions and think tanks, researchers and academics, bilateral/multilateral organizations, and local actors and government in the countries in which it works.
2.4 Did the agency publicly release all completed program evaluations?
  • All final USAID evaluation reports are published on the Development Experience Clearinghouse (DEC), except for a small number of evaluations that receive a waiver to public disclosure (typically less than 5 percent of the total completed in a fiscal year). The process to seek a waiver to public disclosure is outlined in the document Limitations to Disclosure and Exemptions to Public Dissemination of USAID Evaluation Reports and includes exceptions for circumstances such as those when “public disclosure is likely to jeopardize the personal safety of U.S. personnel or recipients of U.S. resources.”
  • To increase awareness of available evaluation reports, USAID has created infographics showing the number and type of evaluations completed in FY2015, FY2016, and FY2017. These include short narratives that describe findings from selected evaluations and how that information informed decision-making. The information for FY2018 is being finalized.
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
  • USAID uses rigorous evaluation methods, including random control trials (i.e. assignment studies) and quasi-experimental methods for research and evaluation purposes. For example, in FY2018, USAID completed 21 impact evaluations, 12 of which used random control trials. 
  • The Development Innovation Ventures (DIV) program makes significant investments using randomized controlled trials to provide evidence of impact for pilot approaches to be considered for scaled funding. USAID is also experimenting with cash benchmarking—using household grants to benchmark traditional programming. USAID conducted five randomized control trials (RCT) of household grants or “cash lump sum” programs, and three RCTs of more traditional programs with household grant elements.
Score
9
Corporation for National and Community Service
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • CNCS has an evaluation policy that presents five key principles that govern the agency’s planning, conduct, and use of program evaluations: rigor, relevance, transparency, independence, and ethics.
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • In FY19, CNCS finalized and posted a five year, agency-wide strategic evaluation plan. The CNCS CEO’s goal is to use the plan to guide FY20 budget planning.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • CNCS uses the terms learning agenda, evaluation plan, and evidence-building plan synonymously. CNCS has a strategic evidence plan that includes an evergreen learning agenda. The plan will be reviewed and updated annually. While the agency is open to the feedback of external stakeholders, it has not engaged external stakeholders in the development of the evidence plan.
2.4 Did the agency publicly release all completed program evaluations?
  • All completed evaluation reports are posted to the Evidence Exchange, an electronic repository for evaluation studies and other reports. This virtual repository was launched in September 2015.
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • A comprehensive portfolio of research projects has been built to assess the extent to which CNCS is achieving its mission. As findings emerge, future studies are designed to continuously build the agency’s evidence base. R&E relies on scholarship in relevant fields of academic study; a variety of research and program evaluation approaches including field, experimental, and survey research; multiple data sources including internal and external administrative data; and different statistical analytic methods. CNCS relies on partnerships with universities and third party research firms to ensure independence and access to state of the art methodologies. CNCS supports its grantees with evaluation technical assistance and courses to ensure their evaluations are of the highest quality and requires grantees receiving $500,000 or more in annual funding to engage an external evaluator. These efforts have resulted in a robust body of evidence that national service allows: (1) national service participants to experience positive benefits, (2) nonprofit organizations to be strengthened, and (3) national service programs to effectively address local issues.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
  • CNCS uses the research design most appropriate for addressing the research question. When experimental or quasi-experimental designs are warranted, the agency uses them and encourages its grantees to use them, as noted in the agency evaluation policy: “CNCS is committed to using the most rigorous methods that are appropriate to the evaluation questions and feasible within statutory, budget and other constraints.” As of May 2019, CNCS has received 41 grantee evaluation reports that use experimental design and 66 that use quasi-experimental design.
Score
7
U.S. Department of Education
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • ED has a scientific integrity policy to ensure that all scientific activities (including research, development, testing, and evaluation) conducted and supported by ED are of the highest quality and integrity, can be trusted by the public, and contribute to sound decision-making. In January 2017, IES published “Evaluation Principles and Practices,” which describes the foundational principles that guide its evaluation studies and the key ways in which the principles are put into practice. That document is expected to serve as the foundation of ED’s formal evaluation policy, under development by the Evaluation Officer for consideration by the Evidence Leadership Group and, subsequently, senior ED leadership.
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • Since the passage of ESSA, IES has worked with partners across ED, including the Evidence Leadership Group, to prepare and submit to Congress a biennial, forward-looking evaluation plan covering all mandated and discretionary evaluations of education programs funded under ESSA (known as ED’s “8601 plan”). The plan is biennial, with the current plan covering FY18 and FY19. The process by which that plan is developed serves as the foundation for ED’s work on both its forthcoming Learning Agenda and Annual Evaluation Plan.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • To develop its draft Learning Agenda, ED has expanded the question generation and prioritization process from ESSA-funded programs, operated by the Office of Elementary and Secondary Education (OESE), to all programs operated by all of its programmatic principal offices. To help ensure alignment of the draft learning agenda to ED’s strategic plan, the Evidence Leadership Group has been expanded to include a member from ED’s Performance Improvement Office, and work has begun to ensure that evidence needs from Strategic Plan Goal Teams is actively solicited. ED anticipates seeking external stakeholder feedback on the draft Learning Agenda in the middle of 2020.
2.4 Did the agency publicly release all completed program evaluations?
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • To develop its draft Annual Evaluation Plan, ED will expand its current 8601 plan format to include the broader set of evidence activities implied by the Learning Agenda. This includes evaluation studies, as is typical for the 8601 plan, but also information about planned evidence building activities beyond evaluations, such as significant administrative data collections, improvements to ED’s performance monitoring activities, use of evidence in grant competitions, relevant policy studies, and advanced statistical and analytic activities (e.g., predictive modeling of student, borrower, and institutional behavior at Federal Student Aid).
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
  • The IES website includes a searchable database of evaluations, including those that use experimental, quasi-experimental, or regression discontinuity designs in order to determine impact. As of August 2019, IES has published 41 experimental studies, 1 quasi-experimental study, and 5 regression discontinuity studies. The What Works Clearinghouse lists studies by design. Currently, the WWC’s database includes 10,646 studies, including 1,115 that meet WWC standards for internal validity. Among them are randomized controlled trials (currently 747), quasi-experimental (currently 217), regression discontinuity (currently 4), and single case (currently 49).
Score
9
U.S. Dept. of Housing & Urban Development
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • PD&R has published a Program Evaluation Policy that establishes core principles and practices of PD&R’s evaluation and research activities. The six core principles are rigor, relevance, transparency, independence, ethics, and technical innovation.
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • A key feature of HUD’s Research Roadmap is a lengthy list of potential research and evaluation projects that feed into proposed evaluation plans that PD&R submits to Congress as part of its budget request. Actual research activities are substantially determined by Congressional funding and guidance.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • HUD’s Research Roadmap serves as the Department’s evidence-building plan and learning agenda. A new Roadmap is under development in FY19. HUD’s participatory process (see for example pp. 14–16 of Roadmap Update 2017) engages internal and external stakeholders to identify research questions and other evidence-building activities to support effective policy-making. Stakeholders include program partners in state and local governments and the private sector; researchers, academics; policy officials; and the general public frequently accessing HUDuser.gov web portal. Outreach mechanisms include email, web forums, conferences and webcasts, and targeted meetings
2.4 Did the agency publicly release all completed program evaluations?
  • PD&R’s Program Evaluation Policy requires the publishing and dissemination in a timely fashion all evaluations that meet standards of methodological rigor. Completed evaluations and research reports are posted on PD&R’s website, HUDUSER.gov. Additionally, the policy includes language in research and evaluation contracts that allows researchers to independently publish results, even without HUD approval, after not more than 6 months. 
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • PD&R is HUD’s independent evaluation office, with scope spanning all the Department’s program operations. PD&R’s Program Evaluation Policy defines six core principles for evaluation and research activities: rigor, relevance, transparency, independence, ethics, and technical innovation.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
Score
7
U.S. Department of Labor
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • DOL has an Evaluation Policy that formalizes the principles that govern all program evaluations in the Department, including methodological rigor, independence, transparency, ethics, and relevance. The policy represents a commitment to using evidence from evaluations to inform policy and practice.
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • The Chief Evaluation Office (CEO) develops, implements, and publicly releases an annual DOL evaluation plan. The evaluation plan is based on the agency learning agendas as well as the Department’s Strategic Plan priorities, statutory requirements for evaluations, and Secretarial and Administration priorities. The evaluation plan includes the studies CEO intends to undertake in the next year using set-aside dollars. Appropriations language requires the Chief Evaluation Officer to submit a plan to the U.S. Senate and House Committees on Appropriations outlining the evaluations that will be carried out by the Office using dollars transferred to CEO– the DOL evaluation plan serves that purpose. The evaluation plan outlines evaluations that CEO will use its budget to undertake. CEO also works with agencies to undertake evaluations and evidence building strategies to answer other questions of interest identified in learning agencies, but not undertaken directly by CEO.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • In FY19, the Department’s evaluation plan and learning agenda are a combined document. DOL will leverage its existing practices and infrastructure to develop the broad, four-year prospective research agenda, per the Evidence Act requirement. This new document will also reflect stakeholder engagement beyond the agency.
2.4 Did the agency publicly release all completed program evaluations?
  • All DOL program evaluation reports and findings funded by CEO are publicly released and posted on the complete reports section of the CEO website. DOL agencies, such as the Employment & Training Administration (ETA), also post and release their own research and evaluation reports.
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • DOL’s Evaluation Policy touches on the agency’s commitment to high-quality, methodologically rigorous research through funding independent research activities. Further, CEO staff have expertise in research and evaluation methods as well as in DOL programs and policies and the populations they serve. CEO also employs technical working groups on the majority of evaluation projects whose members have deep technical and subject matter expertise.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
  • DOL employs a full range of evaluation methods to answer key research questions of interest, including when appropriate, impact evaluations. Among DOL’s active portfolio of approximately 50 projects, the study type ranges from rigorous evidence syntheses to implementation studies to quasi-experimental outcome studies to impact studies. Examples of current DOL studies with a random assignment component include an evaluation of a Job Corps’ demonstration pilot, the Cascades Job Corps College and Career Academy. An example of a multi-arm randomized control trial is the Reemployment Eligibility Assessments evaluation, which assesses a range of strategies to reduce Unemployment Insurance duration and wage outcomes.
Score
5
Millennium Challenge Corporation
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • Every MCC investment must adhere to MCC’s rigorous Policy for Monitoring and Evaluation (M&E) that requires every MCC investment to contain a comprehensive M&E Plan, which includes two main components. The monitoring component lays out the methodology and process for assessing progress towards the investment’s objectives. The evaluation component identifies and describes the evaluations that will be conducted, the key evaluation questions and methodologies, and the data collection strategies that will be employed. Each country’s M&E Plan represents the evaluation plan and learning agenda for that country’s set of investments.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • For FY19, in an effort to advance MCC’s evidence base and respond to the Evidence Act, MCC is pursuing learning agendas at the country, sector, and agency level. At the country level, each country’s Monitoring and Evaluation Plan contains the context-specific learning that MCC hopes to advance during the life cycle of its investments. At the sector level, communities of practice around key MCC sectors have been formed to create, capture, and advance sector-level learning. In FY19, communities of practice around education,water, and sanitation will produce learning reports based on their findings to key research questions for the sector. Finally, at the agency level, MCC is embarking on an agency-wide learning agenda to better understand how MCC develops, implements, monitors, and evaluates the policy and institutional reforms (PIRs) it undertakes alongside capital investments. The PIR learning agenda is focused on better evidence for methodological guidance to economists and sector practices to support the expanded use of cost-benefit analysis in more cases of PIR that MCC supports. The purpose is to make investments in PIR more effective by meeting the same investment criteria as other interventions MCC considers for investment; to make assumptions and risks more explicit for all its investments that depend on improved policies or institutional performance; and to help inform the design of PIR programs to ensure that they have a high economic rate of return. In developing each of these learning agendas, MCC consults with internal staff, technical experts, partner country governments, beneficiaries, and MCC stakeholders.
2.4 Did the agency publicly release all completed program evaluations?
  • MCC publishes each independent evaluation of every project, underscoring the agency’s commitment to transparency, accountability, learning, and evidence-based decision-making. All independent evaluations and reports are publicly reported on the MCC Evaluation Catalog. As of August 2019, MCC has contracted or is planning 195 independent evaluations. To date, 109 Interim and Final Reports have been finalized and released to the public. 
  • In FY19, MCC also began to publish Evaluation Briefs that distill key findings and lessons learned from MCC’s independent evaluations. MCC will produce Evaluation Briefs for each evaluation moving forward, and is in the process of writing Evaluation Briefs for the backlog of all completed evaluations.
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • Once a compact or threshold program is in Implementation, Monitoring and Evaluation (M&E) resources are used to procure evaluation services from external independent evaluators to directly measure high-level outcomes to assess the attributable impact of all of MCC’s programs. MCC sees its independent evaluation portfolio as an integral tool to remain accountable to stakeholders and the general public, demonstrate programmatic results, and promote internal and external learning. Through the evidence generated by monitoring and evaluation, the M&E Managing Director, Chief Economist, and Vice President for the Department of Policy and Evaluation are able to continuously update estimates of expected impacts with actual impacts to inform future programmatic and policy decisions. In FY19, MCC began or continued comprehensive, independent evaluations for every compact or threshold project at MCC, a requirement stipulated in Section 7.5.1 of MCC’s Policy for M&E. All evaluation designs, data, reports, and summaries are available on MCC’s Evaluation Catalog.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
  • MCC employs rigorous, independent evaluation methodologies to measure the impact of its programming, evaluate the efficacy of program implementation, and determine lessons learned to inform future investments. As of August 2019, 37% of MCC’s evaluation portfolio consists of impact evaluations, and 63% consists of performance evaluations. All MCC impact evaluations use random assignment to determine which groups or individuals will receive an MCC intervention, which allows for a counterfactual and thus for attribution to MCC’s project, and best enables MCC to measure its impact in a fair and transparent way. Each evaluation is conducted according to the program’s Monitoring and Evaluation (M&E) Plan, in accordance with MCC’s Policy for M&E.
Score
6
Substance Abuse and Mental Health Services Administration
2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))
  • SAMHSA’s Evaluation Policy and Procedure (P&P) provides guidance across the agency regarding all program evaluations. Specifically, the Evaluation P&P describes the demand for rigor, compliance with ethical standards, and compliance with privacy requirements for all program evaluations conducted and funded by the agency.
2.2. Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))
  • The Evaluation P&P serves as the agency’s formal evaluation plan. The Evaluation P&P sets the framework for planning, monitoring, and disseminating findings from significant evaluations. The Evaluation P&P requires Centers to identify research questions and appropriately match the type of evaluation to the maturity of the program.
2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)
  • According to the Evaluation P&P (p. 1): “SAMHSA is actively working to develop a learning agenda to align its evaluation goals and activities with those of the Department of Health and Human Services (HHS).” As of September 2019, no public learning agenda is available on SAMHSA’s website. However, SAMHSA has posted a National Research Agenda on Homelessness.
2.4 Did the agency publicly release all completed program evaluations?
  • Results from significant evaluations are made available on SAMHSA’s evaluation website, a new step SAMHSA took with its newly-approved Evaluation P&P in the fall of 2017. As of September 2019, the evaluation website had one evaluation summary: a process evaluation of the Safe Schools/Healthy Students (SS/HS) State Program. No other evaluation reports or summaries are posted, including of any ongoing evaluation studies. However, a word search of SAMHSA’s publications for the term “evaluation” yielded 38 results, of which 10 are evaluation reports.
  • The following criteria is used to determine whether an evaluation is significant: (1) whether the evaluation was mandated by Congress; (2) whether there are high priority needs in states and communities; (3) whether the evaluation is for a new or congressionally-mandated program; (4) the extent to which the program is linked to key agency initiatives; (5) the level of funding; (6) the level of interest from internal and external stakeholders; and (7) the potential to inform practice, policy, and/or budgetary decision-making.
2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))
  • In 2017, SAMHSA formed a new workgroup, the Cross-Center Evaluation Review Board (CCERB). According to the Evaluation P&P (p. 2), the CCERB reviews and provides oversight of significant evaluation activities for SAMHSA, from contract planning to evaluation completion and at critical milestones, and is comprised of representatives from each of the centers, and Office of Tribal Affairs and Policy (OTAP) for cultural competency consultation, as necessary. CCERB staff provide support for program-specific and administration-wide evaluations.
2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?
  • SAMHSA does not list any completed evaluation reports on its evaluation website. Of the 10 evaluation reports found on the publications page, none appear to use experimental methods. According to the Evaluation P&P (p. 5): “evaluations should be rigorously designed to the fullest extent possible and include ‘…inferences about cause and effect [that are] well founded (internal validity), […] clarity about the populations, settings, or circumstances to which results can be generalized (external validity); and requires the use of measures that accurately capture the intended information (measurement reliability and validity).’”
Back to the Standard

Visit Results4America.org

Copyright 2024 Results for America. All rights reserved. Privacy Policy

Site by Wide Eye Creative