Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY19?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• The Director of the Office of Performance and Evaluation (OPE) serves as the Administration for Community Living (ACL) evaluation officer. OPE, which oversees the agency’s performance and evaluation work, has seven full time staff positions and in FY19 had a budget of approximately $11.5 million. The Director of OPE has the education, skill, and experience to meet the Evaluation Officer requirements listed in the Evidence Act and routinely gauges the coverage, quality, methods, consistency, effectiveness, independence, and balance of the portfolio of evaluations, policy research, and ongoing evaluation activities of the agency and assesses agency capacity to support the development and use of evaluation. The Director is also the designated ACL Performance Officer.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• As an operating division of a CFO Act Agency, U.S. Department of Health and Human Services, ACL is not required to have its own Chief Data Officer. But the Director of OPE has demonstrated training and experience in data management, governance, collection, analysis, protection, use, and dissemination and fulfills the aspects of this role which are relevant to ACL. These include coordinating with ACL’s CIO and Chief Privacy Officer on use, protection, dissemination, and generation of data to ensure that the data needs of the agency are met; ensuring that agency data conform with data management best practices; engaging agency employees, the public, and contractors in using public data assets; and encouraging collaborative approaches on improving data use.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, and other related officials in order to inform policy decisions and evaluate the agency’s major programs?

• As an operating division of the US Department of Health and Human Services, ACL is not required to have its own Chief Data Officer or Statistical Official. But ACL does have a governance structure to coordinate the use of evidence for program operations and policy making. The Office of Performance and Evaluation staff work with staff across ACL to define and implement ACL’s learning agenda and evaluation plan. This includes semi-annual meetings with ACL leadership and management staff and annual consultation with all program managers. In FY 2019 ACL funded a council to improve ACL’s data governance, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data that ACL collects through its evaluations, grant reporting, and other administrative data collections. ACL is also a member of the Interagency Committee on Disability Research and the Interagency Forum on Aging Related Statistics. These councils promote coordination of federal data, set federal research priorities, work closely with policy makers, and inform comprehensive government wide strategic plans for aging, disability, independent living, and rehabilitation research.

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY19?

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• ACL’s evaluation policy confirms ACL’s commitment to conducting evaluations and using evidence from evaluations to inform policy and practice. ACL seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. The policy addresses each of these principles.

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• ACL is drafting an agency-wide evaluation plan that will be published on the ACL website once finalized. ACL’s evaluation plan is based on the results of developing an ACL-wide Learning Agenda. From the list of priority questions generated, the ones determined to be best addressed through evaluation were compiled into an agency-wide evaluation plan that briefly describes how evaluation research will be used to answer the relevant priority questions and also places the planned evaluations into the larger agency context.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• ACL has adopted a learning agenda approach, which involves annual reviews with each ACL center to support the generation and use of evaluation findings to inform agency strategies and decision making. The approach was implemented between September 2018 and December 2019. The result will be a written learning agenda that will guide ACL’s evaluation and performance management work based on broad input from ACL leadership and staff as well as external stakeholders.

2.4 Did the agency publicly release all completed program evaluations?

• ACL releases all evaluation reports as well as interim information such as materials from presentations.

2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))

• ACL selects evaluation contractors through a competitive acquisition process. Staff of the Office for Performance and Evaluation (OPE) oversee the evaluations and act as liaisons between program staff and evaluation contractors. While program staff are encouraged to write forwards for evaluation reports, they do not influence the findings. ACL’s evaluation policy states that ACL will conduct evaluations through the competitive award of grants and contracts to external experts who are free from conflicts of interest. Further, the Director of the Office of Planning and Evaluation has authority to approve the design of evaluation projects and analysis plans as well as to approve, release and disseminate evaluation reports. While all ACL programs are required to report on program results, over the past 10 years ACL has evaluated or is in the process of evaluating at least one program in 55% percent of its major budget lines. The programs least likely to be evaluated are resource centers and projects like State Councils on Developmental Disabilities (Councils) that are self-governing organizations charged with identifying the most pressing needs of people with developmental disabilities in their state or territory. Evaluation data have been used to improve programs through updating funding requirements and refocusing technical assistance.

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• ACL typically funds evaluation design contracts, such as those for the Older Americans Act Title VI Tribal Grants Program evaluation and the Long Term Care Ombudsman Evaluation, that are used to determine the most rigorous evaluation approach that is feasible given the structure of a particular program. While the Tribal Grants Program and the Ombudsman program are full coverage programs, where comparison groups are not possible, ACL most frequently uses propensity score matching to identify comparison group members. This was the case for the Older Americans Act Nutrition Services Program and National Family Caregivers Support Program evaluations and the Wellness Prospective Evaluation Final Report conducted by CMS in partnership with ACL and published in January 2019.

• ACL is currently working with the Corporation for National and Community Service to design an evaluation of CNCS grantees serving older adults that are using evidence-based programs highlighted by ACL. The purpose of this joint work is to build the evidence base in the field and to improve the use of evidence-based programming by ACL and CNCS grantees.

• ACL’s NIDILRR funds the largest percentage of ACL’s RCTs. Of their 121 research projects being conducted by grantees, 19% (23/121) are employing a randomized clinical trial (RCT) or “true experimental” design. To ensure research quality, NIDILRR adheres to strict peer reviewer evaluation criteria that are used in the grant award process (see part (c) for details on rigor of research projects and part (d) for details on the design of research projects). In addition, ACL’s evaluation policy states that “In assessing the effects of programs or services, ACL evaluations will use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, high-quality quasi-experiments offer an alternative.”

Did the agency invest at least 1% of program funds in evaluations in FY19? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, rigorous evaluations, including random assignments)

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY19 budget.

• ACL invested $18.8 million on evaluations, evaluation technical assistance, and evaluation capacity-building, representing 0.85% of the agency’s $2.2 billion FY19 enacted budget.

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• ACL’s budget for evaluation was $18.8 million in FY19; there were no significant changes to the evaluation budget since the previous year. The bulk of ACL’s evaluation funds are based on a set-aside required in Title II, section 206 of the Older Americans Act, “From the total amount appropriated for each fiscal year to carry out title III, the Secretary may use such sums as may be necessary, but not to exceed 1⁄2 of 1 percent of such amount, for purposes of conducting evaluations under this section, either directly or through grants or contracts.” In addition, in 2017 ACL’s Office of Performance and Evaluation established a mechanism that allows ACL programs not covered by the OAA set-aside to transfer funds to OPE to be able to support evaluations of their programs. In 2017, 2018, and 2019 OPE added approximately $950,000, $1.7 million, and $3.2 million from these programs to its evaluation budget respectively.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• ACL provides technical assistance to grantees related to using evidence-based programs and building evidence. For example, the National Resource Center on Nutrition and Aging (NRC) provides research-based insight into different programs and approaches that deliver nutrition-related home- and community-based services (HCBS) administered through grants to the 56 states and territories. ACL’s Alzheimer’s Disease Supportive Services Program (ADSSP) grant program supports state efforts to expand the availability of community-level supportive services including the translation of evidence-based models into community-level practice in their programs. The 68 University Centers for Excellence in Developmental Disabilities Education, Research, and Service (UCEDDs) throughout the United States and its territories serve as liaisons between academia and the community. They fund model demonstrations to build evidence for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families.

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY19?
(Example: Performance stat systems, frequent outcomes-focused data-informed meetings)

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• ACL’s strategy focuses on five pillars: supporting families and caregivers, protecting rights and preventing abuse, connecting people to resources, expanding employment opportunities, and strengthening the aging and disability networks. These pillars provide structure and focus for ACL’s work. ACL’s outcomes measures are available, by program, in its annual Congressional Budget Justification, and include measures of program efficiency. As part of the U.S. Department of Health and Human Services Annual Performance Plan and Report, ACL reports on the following two Agency Priority Goals: (1) Increase the success rate of the Protection and Advocacy Program’s individual or systemic advocacy, thereby advancing individuals with developmental disabilities’ right to receive appropriate community based services, resulting in community integration and independence, and have other rights enforced, retained, restored and/or expanded; and (2) Improve dementia capability of long-term support systems to create dementia-friendly, livable communities (Lead Agency ACL).

4.2 Does the agency use data/evidence to improve outcomes and return on investment?

• ACL employs a program performance management strategy with multiple components. This includes coordination and collaboration with other agencies and organizations, enhanced partnerships between aging and disability networks, and senior leadership involvement in performance management. ACL has several recent and ongoing evaluation studies that examine the cost benefits of its programs in terms of health care savings (see Report to Congress: The Centers for Medicare & Medicaid Services’ Evaluation of Community-Based Wellness and Prevention Programs Under Section 4202 (b) of the Affordable Care Act which focused on several of ACL’s programs and Evaluation of the Effect of the Older Americans Act Title III-C Nutrition Services Program on Participants’ Health Care Utilization).

• The regional staff conduct annual reviews with the state units on aging to review the states’ work under their state plans on aging (which they develop under the Older Americans Act). While the forms used are for internal use only, states are asked to document their progress towards their approved goals, what performance indicators they use to measure their progress, and to report out on changes in program performance, targeting of priority populations, and program innovations for which they have received honors or recognition. There are also checks of how and whether states verify the quality of their performance data. ACL uses this information to inform TA directed to improve program operations, results, and return on investment. When making decisions about continued grant funding, NIDILRR uses a risk scale to determine whether the additional funding will be a good use of funds. NIDILRR’s long range plan also describes research as part of their new employment research agenda to continue development of return-on-investment models that can be used by Vocational Rehabilitation agencies to optimize the services they provide.

4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• As part of ACL’s performance strategy, OPE staff provide annual presentations on ACL Performance to ACL leadership. They also provide information to internal and external stakeholders about agency performance trends. OPE staff also hold annual meetings with ACL staff to report performance measure data and results, including discussing methods to incorporate performance and evaluation findings into funding and operational decision-making.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY19? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; open data policies; data-use policies)

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• As an operating division of a CFO Act Agency, the U.S. Department of Health and Human Services, ACL is not required to have its own strategic data plan and utilizes HHS’s data strategy. ACL provides public access to its programmatic data through a web based portal. In 2019, ACL created a council to improve ACL’s data governance, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data that ACL collects through its evaluation, grant reporting, and administrative performance measures. In addition, the council will help the Office of Performance and Evaluation meet its mission to provide and promote high quality, transparent information to support sound decision-making.

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• As part of its data restructuring efforts, ACL has created an internal inventory of its administrative data sets. In addition, these data sets/files are documented through the Privacy Impact Assessment and the Systems of Records Notice processes which are required under Titles II and III of the E-Government Act of 2002. The Act specifically requires that agencies evaluate systems that collect personally identifiable information (PII) and determine whether the privacy of that PII is adequately protected. Agencies perform this evaluation through a privacy impact assessment (PIA). One result is a complete listing of all data collections.

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c)) 

• In 2016, ACL implemented a Public Access Plan as a mechanism for compliance with the White House Office of Science and Technology Policy’s public access policy. The plan focused on making published results of ACL/NIDILRR-funded research more readily accessible to the public; making scientific data collected through ACL/NIDILRR-funded research more readily accessible to the public; and increasing the use of research results and scientific data to further advance scientific endeavors and other tangible applications. In March 2019, the ACL completed the ACL Data Restructuring (DR) Project to assess the data hosted on the Aging Integrated Database (AGID), and to develop and test a potential restructuring of the data in order to make it useful and usable for stakeholders. In 2019, ACL awarded a follow on contract to further integrate its datasets along the lines of conceptual linkages, and to better align the measures within ACL’s data collections across the agency. This work will consist of careful data documentation, building a data repository for aging datasets (with a capability for expansion for ACL’s disability datasets), aligning measures for conceptual linkages across ACL datasets, and reviewing datasets for potential topical navigation of the data. The ultimate goal is to expand ACL’s current public data portal (AGID) to allow users to examine ACL data across data sets, geographies, and years.

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)\

• As an operating division of the U.S. Department of Health and Human Services, ACL follows all departmental guidance regarding data privacy and security. This includes project-specific reviews by ACL’s Office of Information Resource Management (OIRM), which monitors all of ACL’s data collection activities to ensure the safety and security of ACL’s data assets.  In FY19, ACL awarded a contract to stand up a “Data Council” to enhance the quality, security, and statistical usability of the data ACL collects through its evaluation, grant reporting, and administrative data collections, and to develop effective data governance standards. In addition, each funding opportunity announcement states that “a data and safety monitoring board (DSMB) is required for all multi-site clinical trials involving interventions” (see for example The FOA for Disability and Rehabilitation Research Projects (DRRP): Assistive Technology to Promote Independence and Community Living (Development) HHS-2019-ACL-NIDILRR-DPGE-0355).

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• ACL staff provide technical assistance through presentations and ACL’s technical assistance resource centers to grantees, including state, tribal, and local governments. The resource centers providing technical assistance include: the National Resource Center on Nutrition and Aging (NRC), the Alzheimer’s Disease Supportive Services Program (ADSSP) and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service. This technical assistance includes annual workshops and presentations at the Title VI National Training and Technical Assistance Conference; training available through the ACL funded National Ombudsman Resource Center; and the Disability and Rehabilitation Research Program (DRRP), which funds capacity building for minority research entities.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY19? (Example: What Works Clearinghouses)

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• ACL defines evidence-based programs on its website. ACL’s National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) uses a stages of research framework (SORF) to classify and describe its funded grants and research projects within the grants. The four stages of SORF include: exploration and discovery, intervention development, intervention efficacy, and scale-up evaluation. Using SORF, NIDILRR gains insight into what is known and unknown about a problem; whether it is time to develop interventions to address a particular problem; whether it is time to test the efficacy of interventions; and whether it is time to scale-up interventions for broader use.

6.2 Did the agency have a common evidence framework for funding decisions?\

• The Older Americans Act requires the use of evidence-based programming in Title III-D-funded activities: Disease Prevention and Health Promotion Services. In response, ACL developed a definition of the term evidence-based, and created a website containing links to a range of resources for evidence-based programs. This is a common evidence framework used for Older Americans Act funded activities. For programs that are not legislatively required to use evidence-based models, through its funding process ACL requires all programs to provide clear justification and evidence (where available) that proposed projects will achieve their stated outcomes.

6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• ACL does not have a common evidence repository, but publishes intervention summaries of aging and disability evidence-based programs and practices. To support the use of evidence-based and evidence-informed programming, service providers can find out about evidence-based programs that serve people with dementia and their caregivers by consulting a white paper drafted with funds from ACL– Translating Innovation to Impact: Evidence-based interventions to support people with Alzheimer’s disease and their caregivers at home and in their communities. The Model Systems Knowledge Translation Center (MSKTC) has worked with NIDILRR’s Model Systems grantees to develop and publish a variety of evidence-based factsheets about living with spinal cord injury, traumatic brain injury, or burn injury.

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• ACL works through its resource centers to help grantees use evidence to drive improvements in outcomes for older adults and individuals with disabilities. For example, with funding from ACL, the National Resource Centers at National Center on Aging (NCOA), in collaboration with the Evidence-Based Leadership Council, led an innovative vetting process to increase the number of programs available to ACL’s aging network that meet the Title III-D evidence-based criteria. This process resulted in adding six new health promotion programs and three new programs for preventing falls. The Alzheimer’s Disease Supportive Services Program (ADSSP) funds competitive grants to expand the availability of evidence-based services that support persons with Alzheimer’s disease and related dementia (ADRD) and their family caregivers. Extensive evaluation of the National Chronic Disease Self-Management Education (CDSME) and Falls Prevention database helped generate important insights for potential new ACL applicants in preparing their applications using data-driven estimation procedures for participant and completion targets (see Guidance for Administration for Community Living 2019 Chronic Disease Self-Management Education Grant Applicants: Considerations for Estimating Participation and Completion Targets).

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY19?
(Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with rigorous evaluation requirements)

7.1 Did the agency engage leadership and staff in its innovation efforts?

• Agency leadership promote innovation through requiring program offices to explain, in their annual funding proposals, how the proposed use of funds will identify innovative practices. In FY16, ACL established the Elder Justice Innovation Grants program to support the development and advancement of emerging practices to prevent and respond to the abuse of older adults and adults with disabilities. ACL also partially funds a Forum on Aging, Disability, and Independence which engages staff to foster discussions about innovation for coordinating and integrating aging and disability stakeholders. The most recent publication from his group, released in 2019 is Artificial Intelligence Applications for Older Adults and People with Disabilities: Balancing Safety and Autonomy: Proceedings of a Workshop—in Brief.

7.2 Did the agency have policies that promote innovation?

• There are several funding streams that support innovation. The Older Americans Act, which funds ACL’s Administration on Aging, allows ACL to use up to 1% of its appropriations for nutrition innovation demonstrations designed to develop and implement evidence-based practices that enhance senior nutrition. One result is that, consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using $3.5 million to fund nutrition innovations and test ways to modernize how meals are provided to a changing senior population. One promising demonstration (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission), currently being carried out by the Georgia State University Research Foundation, is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.

• State Councils on Developmental Disabilities (SCDD) are charged with identifying and addressing the most pressing needs of people with developmental disabilities in their state and territory. Councils work with different groups in many ways, including educating communities to welcome people with developmental disabilities; funding projects to show new ways that people with disabilities can work, play, and learn; and seeking information from the public as well as state and national sources.

7.3 Did the agency have processes, structures, or programs to stimulate innovation?

• ACL uses innovation dollars provided under Title IV of the Older Americans Act as a means of testing new approaches to service delivery and developing replicable models that could then be embedded into core programs. In FY16, ACL established the Elder Justice Innovation Grants program to increase knowledge about effective prevention and intervention of abuse, neglect, and exploitation of older adults, native elders, adults with disabilities, people who self-neglect, and guardianship abuse. In FY18, ACL monitored the second year of 2-year grants awarded to 5 non-profit organizations in FY17, totaling $2.2 million. In FY18, ACL awarded grants under the Alzheimer’s Disease Programs to States and Communities HHS-2018-ACL-AOA-ADPI-0307 to pilot innovative  dementia-capable home- and community-based services (HCBS) programs to states and communities.

• NIDILRR participates in the Small Business Innovation Research program to improve the lives of people with disabilities through research and development of innovative products generated by small businesses, and to increase the commercial application of NIDILRR-supported research results and development products.

• The Engagement and Older Adults Resource Center, funded by ACL, provides technical assistance and serves as a repository for innovations designed to increase the aging network’s ability to tailor social engagement activities to meet the needs of older adults.

7.4 Did the agency evaluate its innovation efforts, including using rigorous methods?

• ACL has a number of model programs and demonstration grants that propose and test the use of innovative approaches. For example, ACL funded cooperative agreements for the development and testing of model approaches towards coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community (i.e. Living Well Grants). While the evaluation of this program is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.

Did the agency use evidence of effectiveness when allocating funds from its 5 largest competitive grant programs in FY19? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring for evidence; Pay for Success provisions)

8.1 What were the agency’s 5 largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY19, the 5 largest competitive grants programs are: 1) Independent Living ($116 million; eligible grantees: nonprofits having a 501(c)(3) status with the IRS, other than institutions of higher education); 2) National Institute on Disability, Independent Living, and Rehabilitation Research ($109 million; eligible grantees: States, Tribes, local governments, nonprofits, education institutions, selected small businesses); 3) Alzheimer’s Disease Program ($47 Million; eligible grantees: States, community-based service providers); 4) University Centers for Excellence in Developmental Disabilities (UCEDD) ($40 million; eligible grantees: selected universities); and 5) Medicare Improvements for Patients and Providers Act Programs (MIPPA) ($38 million; eligible grantees: States and Tribes).

8.2 Did the agency use evidence of effectiveness to allocate funds in 5 largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)

• Independent Living (IL) programs include a mix of formula and discretionary grants. The Centers for Independent Living (CILs) Program provides 354 discretionary grants to centers that are consumer-controlled, community-based, cross-disability, nonresidential, private nonprofit agencies who provide IL services. While not explicitly using the term “evidence-based” applicants must describe the rationale for using the proposed intervention, including factors such as: “‘lessons learned’ for similar projects previously tested in your community, or in other areas of the country; factors in the larger environment that have created the ‘right conditions’ for the intervention (e.g., existing social or economic factors that you’ll be able to take advantage of, etc.).” To continue receiving CIL program funding, eligible centers must provide evidence that they have previously had an impact on the goals and objectives for this funding.

• The Long-Range Plan of the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funding announcements require scientifically rigorous research (p3 and p15). The long range plan describes the ways in which “NIDILRR proposes to support competitions that build on prior investments that resulted in evidence of efficacy and effectiveness. These competitions will provide funding for further development and testing of practices and interventions in additional settings, or among new populations of people with disabilities. These efforts may support translational research to develop practical strategies for ensuring more widespread use of new evidence-based findings in the area of disability and rehabilitation research and development.”

• For Alzheimer’s Disease Programs Initiative (ADPI) funding, “If the applicant has held an ADSSP grant between 2011 and 2017, they must explain the work of their previous dementia systems project.” The funding announcement (page 12) requires that “All programs must identify, by name, and describe the dementia specific evidence-based or evidence informed intervention they propose to implement, how it fits into their proposed overall program and the intended beneficiaries.”

• UCEDDs are a nationwide network of independent but interlinked centers, representing an expansive national resource for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families. According to the funding opportunity announcement applications are also reviewed based on their description of current or previous evidence of relevant experience.

• MIPPA funds are awarded to State grantees and to the National Center for Benefits Outreach and Enrollment. To continue funding without restrictions, State grantees are required to submit state plans that ACL staff review for the specific strategies that grantees will employ to enhance efforts through statewide and local coalition building. The National Center applicants must describe the rationale for using the particular intervention, including factors such as evidence of intervention effectiveness. In 2019, the Center was awarded additional funding based on prior performance— specifically, assisting over 7.6 million individuals to identify over $29.6 billion in potential annual benefits.

8.3 Did the agency use its 5 largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• Grantees are required to provide evidence that they are meeting grant program outcomes as well as lessons learned about what works and how they overcame challenges faced. For example, the Rehabilitation Engineering Research Centers (RERC) Program for a RERC on Rehabilitation Strategies, Techniques, and Interventions is sponsored by NIDILRR. It engages in the systematic application of engineering sciences to design, develop, adapt, test, evaluate, apply, and distribute technological solutions to problems confronted by people with disabilities in functional areas such as mobility, communications, hearing, vision, and cognition. ACL’s Alzheimer’s Disease Programs to States and Communities (HHS-2018-ACL-AOA-ADPI-0307) is informed by the work and recommendations of National Alzheimer’s Project Act Advisory Committee. The Alzheimer’s Disease Programs Initiative is implemented with authority contained within Title IV of the Older Americans Act. It is designed to pilot dementia-capable home- and community-based services (HCBS) programs to states and communities, evaluate program outcomes, and use program outcome data to garner support to sustain successful initiatives beyond the federal program period.

8.4 Did the agency use evidence of effectiveness to allocate funds in any competitive grant program?

• ACL requires that grant officers attend training regarding ways to include information about evidence building into funding opportunity announcements. This includes information about text that can be included in funding announcements: 1) describing requirements for developing measurable outcomes; 2) explaining how the inclusion of evidence and evidence building plans can be used to score grant applications; and 3) instructing grant reviewers regarding rating applicants’ presentation of evidence and evidence building plans. The training was recorded and is available to all staff.

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• NIDILRR participates in the Small Business Innovation Research program to improve the lives of people with disabilities through research and development of innovative products generated by small businesses, and to increase the commercial application of NIDILRR-supported research results and development products.

• ACL’s Alzheimer’s Disease Supportive Services Program (ADSSP) encourages the translation of dementia specific interventions for use in communities. Examples include: the Savvy Caregiver (evidence-based) psychoeducational intervention focused on training family caregivers about the basic knowledge, skills, and attitudes needed to handle the challenges of caring for a family member with Alzheimer’s disease and to be an effective caregiver; Cuidando con Respeto (evidence-informed), Spanish version of the original Savvy Caregiver Program; and Savvy Caregiver Express (evidence-informed), a condensed version of the original Savvy Caregiver Program. ACL’s requirement for inclusion of dementia specific evidence-based interventions is demonstrated in the 2018 funding opportunity announcement entitled Alzheimer’s Disease Programs to States and Communities.

• ACL released a funding opportunity in 2019 titled Rehabilitation Research and Training Center (RRTC) on Improving Employment Outcomes for People with Psychiatric Disabilities. This is based on Individual Placement and Support (IPS), which is an evidence-based supported employment model that has been demonstrated to improve employment outcomes for people with psychiatric disabilities. In 2019, ACL will fund up to three projects as cooperative agreements for the development and testing of one or more model approaches of a coordinated and comprehensive system that includes two interrelated core components for enhancing and assuring the independence, integration, safety, health, and well-being of individuals living in the community. ACL made similar grants in 2017 and 2018 and is evaluating the outcomes of those grants. Initial results suggest that these models include a range of promising practices.

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• Funding opportunity announcements and grant reviews stress the need for strong performance measurement and evaluation.  ACL’s technical assistance centers— the National Resource Center on Nutrition and Aging (NRC), the Alzheimer’s Disease Supportive Services Program (ADSSP) and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service— promote the use and generation of evidence with ACL grantees. Grantees manuals also include information about the importance of and requirements for evaluation (see the Administration on Aging: Title VI Resource Manual). Staff of ACL’s Office of Performance and Evaluation make presentations regarding the importance of evidence with regional staff who are in frequent contact with State grantees and at grantee conferences (see ACL Track: The ACL Older Americans Act (OAA) Performance System – Crossing the Finish Line and ACL/CMS Track: Raising the Bar in Medicaid HCBS & Community Inclusion – Showcasing Transformation presented at the 2019 home- and community-based services (HCBS) conference; ACL Track: Assuring the Health & Welfare of Medicaid HCBS Beneficiaries: Federal Findings, Investments, & Promising Practices in Systems Change and ACL Track: Innovative Housing & Health & Human Services Collaborations: A Game-Changer in Supportive Housing & Community Living presented at the 2018 HCBS conference).

Did the agency use evidence of effectiveness when allocating funds from its 5 largest non-competitive grant programs in FY19?
(Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

9.1 What were the agency’s 5 largest non-competitive programs and their appropriation amounts (and were city, county, and/or state governments are eligible to receive funds from these programs)?

• Nutrition Services ($906 million; eligible grantees: states and territories); 2) Home and Community-Based Supportive Services ($345 million; eligible grantees: states and territories); 3) Family Caregiver Support Services ($181 million; eligible grantees: states and territories); 4) State Councils on Developmental Disabilities ($76 million; eligible grantees: states); and 5) Developmental Disabilities – Protection and Advocacy ($40 million; eligible grantees: states and territories).

9.2 Did the agency use evidence of effectiveness to allocate funds in largest 5 non-competitive grant programs? (e.g., Are evidence-based interventions/practices required or suggested? Is evidence a significant requirement?)

• States that receive Older Americans Act Home and Community-Based Supportive Services Title III-D funds are required to spend those funds on evidence-based programs to improve health and well-being, and reduce disease and injury. In order to receive funding, states must utilize programs that meet ACL’s definition of evidence-based or are defined as evidence-based by another HHS operating division.

9.3 Did the agency use its 5 largest non-competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• FY12 Congressional appropriations included an evidence-based requirement for the first time. OAA Title III-D funding may be used only for programs and activities demonstrated to be evidence-based. Consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using its 1% Nutrition authority to fund $3.5 million for nutrition innovations and to test ways to modernize how meals are provided to a changing senior population. One promising demonstration currently being carried out by the Georgia State University Research Foundation (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission) has drawn widespread attention is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.

9.4 Did the agency use evidence of effectiveness to allocate funds in any non-competitive grant program?

• The Older Americans Act state plans require grantees to provide information about past performance, including “information on the extent to which the area agency on aging met the objectives” related to “providing services to older individuals with greatest economic need, older individuals with greatest social need, and older individuals at risk for institutional placement.”

9.5 What are the agency’s 1-2 strongest examples of how non-competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Since 2017, ACL has awarded Innovations in Nutrition grants to 11 organizations to develop and expand evidence-based approaches to enhance the quality and effectiveness of nutrition programming. ACL is currently overseeing five grantees for innovative projects that will enhance the quality, effectiveness, and outcomes of nutrition services programs provided by the national aging services network. The grants total $1,197,205 for this year with a two-year project period. Through this grant program, ACL aims to identify innovative and promising practices that can be scaled across the country and to increase the use of evidence-informed practices within nutrition programs.

9.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• All funding opportunity announcements published by ACL include language about generating and reporting evidence about their progress towards the specific goals set for the funds. Grantee manuals include information about the importance of and requirements for evaluation (see the Administration on Aging: Title VI Resource Manual). The National Ombudsman Resource Center, funded by ACL, provides self-evaluation materials for Long-Term Care Ombudsman Programs (LTCOP)  funded under Title VII of the Older Americans Act.

In FY19, did the agency shift funds away from or within any practice, policy, or program that consistently failed to achieve desired outcomes?
(Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; incentivizing or urging grant applicants to stop using ineffective practices in funding announcements; proposing the elimination of ineffective programs through annual budget requests; incentivizing well-designed trials to fill specific knowledge gaps; supporting low-performing grantees through mentoring, improvement plans, and other forms of assistance; using rigorous evaluation results to shift funds away from a program)

10.1 Did the agency shift funds/resources away from ineffective practices or interventions used within programs or by grantees?

• ACL uses a quality review system (QRS) for developmental disability programs under ACL’s/Administration for Intellectual and Developmental Disabilities (AIDD). The QRS uses a three-tiered model to review program compliance, outcomes (i.e., evidence), and fiscal operations to review results to target and coordinate technical assistance. The first tier is an annual standardized review. The second tier is standardized, in-depth review involving a team of reviewers. These reviews are conducted on a periodic basis. Tier three is customized monitoring for programs that ACL has significant concerns in terms of compliance and performance. ACL continues development of a formula grant monitoring framework for Older Americans Act Title III and VII state formula grants. The framework combines assessments of grantee’s progress toward program goals and objectives with identification of risk or instances of fraud, waste and abuse. These reviews allow ACL, if warranted, to restrict grant funding based on findings of insufficient evidence of performance.

10.2 Did the agency shift funds/resources away from ineffective policies used within programs or by grantees?

• The Paralysis Resource Center State Pilot Program is part of an effort to ensure program efficiency and to test two approaches for making subawards to community-based organizations that provide long-term services and supports to people with paralysis, their families, and their support networks. Outcomes from the pilot will help ACL assess the most effective and efficient ways to make such subawards and will determine how ACL funds this effort going forward.

10.3 Did the agency shift funds/resources away from ineffective grantees?

• ACL is using a tool developed in FY 2018 to help grant officers more easily monitor the degree to which Chronic Disease Self-Management (CDSME) and Falls Prevention grantees are meeting their ACL approved program completion targets. ACL staff will use this information to either fully release or restrict grant funds over the life of the multi-year awards. Grant scores comprise the following three components:
  • Goal 1: Intervention Level – assesses progress towards meeting the target number of program completers (for CDSME grants) or participants (for Falls Prevention grants).
  • Goal 2: Sustainability Level – assesses progress towards establishing a sustainable program.

• Grantee Challenges – presents challenges identified by grantees and assesses the strategies proposed to resolve the challenges.

• Combined, these elements provide Project Officers and staff with an overall Grant Score and Status for each grant. The Grant Score provides a numeric indicator of grantee performance. The Grant Status indicates whether the grant is at “Low”, “Medium,” or “High” risk of eventually requiring a no-cost extension, based on the Grant Score. The tool also includes several features for designated administrative users to manage core information pertinent to supporting the manage grantee and grant performance data functionality.

• In addition, the Innovations in Nutrition Programs and Services grants will be monitored using a new tool that enables Project Officers to quickly assess grant status and proactively identify and engage with grantees that would benefit from technical assistance, to ensure that grants are completed on time.

10.4 Did the agency shift funds/resources away from ineffective programs? (e.g., eliminations or legislative language in budget requests)

• While much of ACL’s funding is based on formula grants, and therefore cannot be reallocated to other programs, ACL is working with GSA’s Office of Evaluation Sciences (OES) to test methods for improving outcomes for its congregate meals programs. Under the Older Americans Act, congregate meal sites are required to accept donations from meal recipients. But there has been a concern regarding how to balance the collection of funds that can be used towards meal service and making meal recipients that cannot afford to donate uncomfortable, thus suppressing attendance. This study, being conducted in FY19, will offer concrete evidence to improve program operations.

10.5 Did the agency shift funds/resources away from consistently ineffective products and services?

• While much of ACL’s funding is based on formula grants, and therefore cannot be reallocated to other programs, evaluation staff work closely with program staff to identify ways to translate evaluation findings into technical assistance and other types of program support. For example, based on early results from and evaluation of the Tribal Grant program, ACL has developed new program support materials to improve the delivery of Tribal Caregiver programs.