Agency Profile and Goals
In the year 2000, the National Science Foundation (NSF) commemorated its 50th anniversary as a pivotal force in advancing the nation's scientific and engineering research and education As the sole federal agency focused on supporting fundamental non-medical research and education, NSF plays a crucial role across all disciplines of science, mathematics, and engineering, catering to all educational levels.
The National Science Foundation (NSF) plays a unique role compared to other Federal agencies by focusing on a broad range of research and education initiatives rather than specific services or mission-oriented research in fields like energy, biomedicine, or space NSF provides funding through grants, contracts, and cooperative agreements to approximately 1,800 diverse institutions, including colleges, universities, K-12 schools, academic consortia, non-profit organizations, small businesses, and various research entities across the United States.
In FY 2000, approximately 95% of NSF's $3.9 billion budget supported research and education activities carried out by awardees at their home institutions These programs and activities directly engaged nearly
184,000 people, including researchers, educators, students, and other professionals.
To conduct the administrative work of the agency, NSF employed a scientific and engineering staff of approximately 1250 government employees, more than 120 visiting scientists and engineers, and over
190 contractors NSF staff administer the merit review and award process: they do not conduct the research nor do they operate the laboratories supported by NSF awards.
In FY 2000, NSF staff managed around 240,000 merit-based reviews and evaluated nearly 30,000 competitive proposals, leading to the processing of approximately 9,760 new awards and 6,680 continuing awards The selection of new awards was facilitated by approximately 50,000 external reviewers, who generously contributed tens of thousands of hours annually to assess proposals related to research and education.
NSF provides national leadership in improving science, mathematics, engineering and technology (SMET) education, and in broadening
The National Science Foundation (NSF) acts as a catalyst by investing federal funds to support innovative ideas and talented individuals in the pursuit of new knowledge and discoveries NSF is committed to fostering participation in the science, mathematics, engineering, and technology (SMET) fields, aiming to prepare a diverse and globally oriented workforce Through its initiatives, NSF significantly contributes to the development of the nation's future scientists and engineers.
For the past fifty years, the National Science Foundation (NSF) has been dedicated to advancing science, mathematics, and engineering research and education across the United States This commitment has led to groundbreaking innovations that have driven economic growth and improved the quality of life for Americans NSF-supported initiatives have significantly enhanced various sectors, including education, public safety, national defense, health, and the environment In this report, NSF expresses its passion for its mission and aims to foster a deeper understanding of its role within the federal system Looking ahead, NSF aspires to continue leading in scientific discovery and education, ensuring that future generations benefit from sustained health, prosperity, and an elevated quality of life.
The Foundation's main goal is to advance science by ensuring the United States leads in discovery, learning, and innovation in science, mathematics, and engineering It achieves this by providing merit-based awards to exceptional individuals and groups, collaborating with various educational and institutional partners across the nation NSF awards represent the nation's investment in those who drive the results of the research and innovation processes that the Foundation oversees.
To effectively serve the Nation, the National Science Foundation (NSF) developed the FY 2000 Annual Performance Plan, aligned with its Strategic Plan for FY 1997-2003 This plan is structured around three interconnected goal areas: Outcomes, Management, and Investment Process, ensuring a balanced and high-quality portfolio of awards.
NSF awards provide resources to enable, enhance, and secure the nation’s future
Performance Plans may be found on the NSF web site at http://www.nsf.gov/ od/gpra/start.htm.
The Outcome Goals of the Foundation focus on programmatic investments aligned with the agency's mission, encompassing all activities supported by NSF awards The Management and Investment Process Goals play a crucial role in achieving these Outcome Goals Since NSF's Outcome Goals are long-term and challenging to measure on an annual basis, they are assessed qualitatively by examining progress and trends over several years To evaluate advancement toward these goals, NSF compiles annual performance data across the agency and incorporates qualitative assessments from external experts Detailed descriptions of all goals and results achieved for FY 2000 can be found in Section V of this report.
The NSF's five Outcome Goals aim to enhance the impact of its research and education awards in science, mathematics, and engineering In FY 2000, a new goal was introduced to improve data quality measures for reporting Science Resource Studies (SRS) products Comprehensive details on these Outcome Goals, along with FY 2000 results and examples of achievements, can be found in Sections IV and V.A.
1 Discoveries at and across the Frontier of Science and Engineering;
2 Connections between Discoveries and their use in Service to Society;
3 A Diverse, Globally-oriented Workforce of Scientists and Engineers;
4 Improved achievement in Mathematics and Science Skills needed by All Americans; and
5 Timely and Relevant Information on the National and International Science and Engineering Enterprise.
Management Goals focus on enhancing the efficiency and effectiveness of administrative activities that support the mission of the NSF In FY 2000, two additional management goals were introduced: one aimed at improving electronic proposal processing and the other focused on promoting staff diversity For a detailed description of the Management Goals and their outcomes for FY 2000, refer to Sections IV and V.B.
Investment Process Goals focus on the means and strategies NSF uses to achieve its Outcome Goals and set performance targets for
The National Science Foundation (NSF) prioritizes long-term outcomes to highlight its value and impact on the American public Annually, NSF reviews nearly 30,000 proposals for research and education projects, utilizing a merit review process to identify the most promising initiatives However, the total funding requested in these proposals significantly exceeds NSF's annual budget, resulting in intense competition where only one in three new proposals receives support each year.
Each year, the National Science Foundation (NSF) evaluates the results reported by its awardees, focusing on the long-term impacts of its investments rather than just the annual progress of individual projects These outcomes serve as evidence of NSF’s effectiveness as an investment agent for the nation NSF emphasizes that strong performance in achieving Outcome Goals is crucial for enhancing the country's economic strength, security, and overall quality of life.
The Foundation consists of the National Science Board (NSB) and a Director who is also an ex officio member of the NSB The NSB includes 24 part-time members appointed by the President and confirmed by the U.S Senate, selected for their distinguished contributions across diverse fields such as science, mathematics, engineering, and education These members are tasked with representing the perspectives of the scientific and engineering communities throughout the nation, encompassing areas like basic and medical sciences, engineering, agriculture, education, research management, industry, and public affairs.
The National Science Board (NSB) enforces a six-year term limit for its members, prohibiting any individual from serving more than two consecutive terms The NSB plays a crucial role as both an advisor on national science policy to the President and Congress and as the governing body for the National Science Foundation (NSF).
The National Science Foundation (NSF) is led by a Deputy Director appointed by the President, with the U.S Senate's advice and consent, alongside seven Assistant Directors and two Office Directors Management of proposals and awards is overseen by nine programmatic organizations, which include seven disciplinary directorates—Biological Sciences, Computer and Information Science and Engineering, Engineering, Geosciences, Mathematical and Physical Sciences, Social, Behavioral and Economic Sciences, and Education and Human Resources—along with the Office of Polar Programs and the Office of Integrative Activities Over 1,250 dedicated NSF staff members contribute to these efforts.
Examples of Achievements Reported in FY 2000
In FY 2000, NSF proudly highlights significant research and education achievements reported by awardees, showcasing exciting discoveries that align with NSF’s mission These selected examples, supported by NSF funding, demonstrate the diverse accomplishments in various fields The report begins with notable projects from NSF’s Office of Legislative and Public Affairs (OLPA) and includes additional noteworthy achievements recognized by external committees For further insights, readers can refer to Section V.A, “Outcome Goals and Results for FY 2000,” and Section XIV, “Appendix of Additional Examples Illustrating Outcomes of NSF Investments.”
FY 2000 RESEARCH AND EDUCATION HIGHLIGHTS –
Examples Reported by OLPA (by Title)
Below are titles for NSF supported research discoveries and results that were reported as highlights by NSF’s Office of Legislative and Public Affairs (OLPA).
To explore the narratives tied to these titles and discover further examples, visit the NSF highlights at [www.nsf.gov/home/news.html](http://www.nsf.gov/home/news.html) This website offers comprehensive details and serves as a repository for additional recent newsworthy highlights and thrilling discoveries.
Astronomers Find Evidence for the First Planet Seen Orbiting a Pair of Stars
Scientists Report First Complete DNA Sequence of Plant Chromosomes
Earthquake Network Intended to Help Save Lives and Money
Bacteria May Thrive in Antarctic Lake
Report Shows Students Improving in Math and Science Preparation
Global Seismographic Network Establishes Internet Connection to Remote Africa
Astronomers Sight an Asteroid's Moon
Exploring the Far Frontiers of Sea and Space
Membrane Protein Research Yields New Insights into Inner Workings of the Cell
New England Experienced "Ice Age" El Niủo
Human-Computer Interaction Gets a Helping Hand, Eye, and Voice
Researchers Discover Evidence of Microscopic Life at the SouthPole
Three examples of NSF Highlights from the OLPA Exciting Findings are:
S CIENTISTS R EPORT F IRST C OMPLETE DNA S EQUENCE OF P LANT
On December 16, 1999, scientists announced the first complete DNA sequence of a plant chromosome for Arabidopsis thaliana in the journal Nature, marking a significant milestone in plant genetics This groundbreaking research offers valuable insights into chromosome structure, evolution, intracellular signaling, and disease resistance in plants The study, primarily funded by the National Science Foundation, along with contributions from the U.S Department of Agriculture and the U.S Department of Energy, highlights the collaborative international effort in advancing our understanding of plant genomes.
N EW E NGLAND EXPERIENCED “I CE -A GE ” EL N IẹO
The New England region underwent El Niủo-like climate changes during the Ice Age, NSF-supported researchers have found Scientists define
El Niño is a significant disruption of the ocean-atmosphere system in the tropical Pacific, impacting global weather patterns Recent findings indicate that during the later stages of the last Ice Age, there was a strong three-to-five-year cycle of El Niño activity, mirroring the frequency observed in today's climate.
T WELVE P IONEERING R ESEARCHERS R ECEIVE 1999 N ATIONAL
On January 31, 2000, President Clinton awarded the 1999 National Medal of Science to 12 esteemed researchers, including three Nobel Prize winners, recognizing their groundbreaking contributions and lifetime achievements This diverse group of Medal of Science recipients pioneered entirely new scientific fields like conservation biology and speech sciences, made significant discoveries regarding the ozone "hole," and validated theories linking technological progress to economic growth.
FY 2000 RESEARCH AND EDUCATION HIGHLIGHTS
E XAMPLES C ITED BY E XTERNAL E VALUATORS
The following examples highlight the significant results identified by external evaluators during the NSF's performance assessment process for FY 2000 These instances exemplify the positive impact of NSF investments on research and education, showcasing their broader contributions to the nation and its citizens Each example aligns with NSF's Outcome Goals, emphasizing their role in achieving key objectives For further insights into outcome achievements, additional examples can be found in Section V.A., titled "Findings from Program Assessments and Evaluations: Outcome Goals and Results for FY 2000."
Section XIV, as an Appendix to this report.
The integration of molecular biology and nano-fabrication has enabled the development of spinning molecular motors by assembling protein flagella on nickel posts Researchers have successfully attached these bacterial motors to a structured array of nanoscale metal posts, measuring their revolutions per second, horsepower, and efficiency This groundbreaking work marks a significant advancement in combining biological and mechanical components with engineered nanostructures, paving the way for innovative and powerful nanodevices The ultimate goal is to utilize these nanomotors in nanofactories to synthesize and deliver drugs directly to targeted tissues, thereby minimizing toxicity and enhancing the effectiveness of drug therapies.
Recent advancements in our understanding of the Early Universe, particularly through the study of Cosmic Microwave Background (CMB) radiation, are revolutionizing cosmology The CMB is believed to be a remnant from the Big Bang, which occurred approximately 12-15 billion years ago The Boomerang project employs balloon-borne microwave detectors to create detailed maps of this radiation while operating at an altitude of 120,000 feet over Antarctica These high-resolution maps have revealed that the universe's large-scale geometry is remarkably flat, marking a significant and fundamental discovery in the field.
A LIVE A FTER 250 M ILLION Y EARS : I SOLATION OF L IVE P ERMIAN
Recent interdisciplinary research supported by the NSF has revealed that ancient salt crystals from the Permian Salado Formation in Southeastern New Mexico contain viable microorganisms trapped in tiny fluid inclusions, dating back at least 250 million years These microorganisms were found in a salt crystal extracted from a dissolution pipe located 564 meters below the surface, ensuring the sample was taken from primary sedimentary beds Notably, two inclusions housed salt-tolerant microbes that form spores in response to concentrated brines, with one organism showing unique characteristics related to modern-day bacilli.
The discovery of viable microbes in the accretion ice of Lake Vostok, located 4 km beneath the East Antarctic ice sheet, marks the first evidence of a microbial community in this subglacial lake This finding garnered significant media attention, including a BBC documentary titled “The Lost World.” Research on Lake Vostok's ice suggests a large and diverse bacterial population within the lake, prompting plans for extensive interdisciplinary studies These investigations not only enhance our understanding of life on Earth but also provide valuable models for future interplanetary research.
Technology spin-offs from gravity research are emerging as experiments push the boundaries of our understanding of this weak force, which remains the least tested among all known natural forces Fourteen billion years after the Big Bang, advancements in laboratory measurements have achieved unprecedented accuracy, leading to significant technological innovations.
Development of high power solid-state lasers, up to 120 watts;
Development of pre-stabilized laser power amplifiers, in collaboration with industry;
Advances in large optics and metrology with sub-Angstrom smoothness and losses approaching one part per million;
Innovative software originally designed for huge numerical calculations in relativity was applied to oil exploration analysis
Research on risk sharing and financial markets has enhanced our understanding of key economic phenomena, particularly individual behavior in utilizing financial markets to manage income fluctuations, which has significant policy implications Studies of the 1990s currency crises revealed that the typical cause, rapid money supply expansion, did not apply to the 1998 Asian crisis A pivotal finding in this area is "Taylor's Rule," which serves as an effective monetary policy guideline It suggests that the Federal Reserve should adjust the federal funds rate based on the difference between actual and desired performance regarding price stability and full employment Specifically, the real federal funds rate should be set at 2% plus half the difference between actual and targeted inflation and half the percentage difference between actual and potential GDP, assuming a potential growth rate of 3.5% Consequently, the nominal funds rate should equal the targeted real funds rate plus actual inflation This approach is increasingly adopted by the Federal Reserve and other central banks to promote sustained economic growth while controlling inflation.
E DUCATION - I MPROVED A CHIEVEMENT IN M ATHEMATICS AND
This year, the NSF's focus on systemic reform and teacher education has resulted in significant progress, leading to improved academic performance among a diverse student body These investments have notably reduced achievement gaps between minority and majority students, demonstrating the positive impact of educational reforms.
In the first six years of the Miami-Dade Urban Systemic Initiative (USI), the median percentile scores on the Stanford-8 test for fourth-grade students demonstrated significant improvement, with African-American students rising from 26 to 40, Hispanic students increasing from 26 to 59, and White students showing a slight increase from 74 to 77, indicating meaningful progress in closing the achievement gap.
The Systemic Initiatives have also brought about substantial increases in the number of students taking more challenging science and mathematics courses in high school For example,
Over the past five years, advanced placement science enrollment in Los Angeles USI schools surged by 53%, significantly outpacing the 17% increase seen in non-USI schools Notably, African-American and Hispanic student enrollment in these advanced programs rose by an impressive 196% and 146%, respectively.
T RAINING W ORLD - CLASS S CIENTISTS IN M ODERN T ECHNOLOGIES
Types of Goals
The National Science Foundation (NSF) utilizes a combination of qualitative and quantitative objectives to assess its annual progress While NSF's Outcome Goals are primarily articulated in qualitative terms, the majority of its Management Goals and Investment Process Goals are defined using quantitative metrics.
Types of Assessments
NSF employs a variety of assessments and evaluations to ensure quality and process monitoring throughout the year Internal staff and senior management continually assess and evaluate programs and plans, while externally contracted groups review program progress against specific objectives Additionally, NSF utilizes Committees of Visitors (COVs) and Advisory Committees (ACs) composed of external experts to evaluate program practices, processes, and outcomes The Management and Investment Process Goals undergo internal reviews by NSF staff and are subject to third-party audits.
The National Science Foundation (NSF) evaluates program performance towards Outcome Goals through assessments conducted by Committees of Visitors (COVs) and Advisory Committees (ACs) Traditionally, each NSF program undergoes a quality and integrity assessment every three years, with COVs reviewing one-third of the portfolio annually, focusing on the previous three years of program performance COVs examine practices and processes, incorporating results assessments due to the Government Performance and Results Act (GPRA) As a result of GPRA, programs may be grouped for evaluation, leading to adjustments in COV schedules to better align with the GPRA framework.
In FY 2000, approximately 37% of the National Science Foundation's (NSF) 200 programs underwent evaluations by Committee of Visitors (COVs) to assess the quality of processes and progress towards NSF's Outcome Goals In the previous year, this figure increased to around 40% The evaluation of the remaining segments of NSF's portfolio by COVs is scheduled for FY.
2001 to complete the full three-year cycle of assessment of NSF’s programs under GPRA.
In FY 2000, around 250 members of the Committee of Visitors (COV) and 150 advisory committee members engaged in the performance assessment process, producing 64 reports that evaluated 78 of NSF's nearly 200 programs Electronic copies of these COV and advisory committee reports are expected to be accessible by December 2001.
NSF utilizes its internal data systems to effectively track and report on the progress made towards its quantitative Management Goals and Investment Process Goals, with the exception of one specific Investment Process Goal, known as Investment Goal 2.
2 In several instances, a single COV report evaluates more than one program. these goals and results are assessed and reviewed by internal management and staff rather than by external committees.
External group reviews are crucial for identifying issues, setting new goals, and redirecting efforts effectively Adjustments to programs and plans may be required due to challenges in achieving goals, the relevance of those goals, or difficulties in measurement As a result, some goal levels for FY 2001 and FY 2002 are being revised based on the outcomes from FY 1999 and FY 2000, alongside realistic expectations for future progress Such adjustments are managed through performance plans and internal management, while significant changes are addressed through revisions to the Strategic Plan, incorporating feedback from external Advisory Committees.
Assessment Processes
In FY 1998, the NSF implemented new reporting systems and procedures, along with guidelines and templates, to effectively collect and analyze both qualitative and quantitative data for performance reporting These systems are regularly updated based on feedback from prior experiences to ensure continuous improvement The standardized reporting templates facilitate uniformity and consistency in reporting qualitative goals across the agency, enhancing the assessment and evaluation process for measuring progress towards NSF's objectives.
The quantitative management and investment process involves a clear assessment strategy where the agency gathers pertinent data from internal systems and compares it against the fiscal year's performance targets Most quantitative objectives are assessed quarterly, with senior management reviewing the findings In FY 2000, the agency implemented a GPRA module to facilitate staff in monitoring their progress towards these quantitative goals throughout the year.
NSF programs are evaluated by external groups based on qualitative Outcome Goals This assessment primarily emphasizes the evaluation of Outcome Goals, with NSF collecting and maintaining performance data through reports from external Committees of Visitors (COVs) and Advisory Committees (ACs), adhering to the regulations set by the Federal Advisory Committee Act.
Assessment of goal achievement by external evaluators takes into account such factors as:
identified performance indicators for each performance goal;
the success to which NSF strategies and plans are implemented;
the level of resources invested;
external events beyond control of the agency; and
the agency’s capability to be flexible and respond rapidly to emerging opportunities
This portfolio assessment emphasizes evaluating the quality of previous investments, specifically their outputs and outcomes, while also considering the potential for the current package of awards to yield strong results in the future.
The performance assessment primarily focuses on past investments, evaluating whether NSF's outcomes for a fiscal year have met or significantly progressed towards specific performance goals COVs utilize their collective experience to determine what constitutes a "significant" achievement for a successful rating Additionally, COVs examine the quality of awarded grants and the integrity of the review process during the assessment period.
Each year, external Committees of Visitors (COVs) evaluate about one-third of the National Science Foundation's (NSF) programs, ensuring that all programs undergo a review every three years The insights from COV and Advisory Committee (AC) reports are merged with data from internal databases, allowing NSF management to compile a comprehensive performance report.
The flow-chart illustrates the comprehensive assessment process for Outcome Goals, detailing how data related to Management and Investment Process Goals is integrated into NSF’s Performance Report NSF staff create reports, evaluations, and studies to support the COVs in their assessments These COV reports inform NSF staff in the preparation of annual reports for each directorate/office, which are then reviewed and approved by the respective advisory committees (ACs) Each directorate/office is required to respond to COV recommendations, with responses subject to AC review Additionally, directorates/offices compile annual reports summarizing their fiscal year activities, which are also reviewed by the AC in October and November The process concludes with the AC report, highlighting the annual progress of each directorate toward NSF’s Outcome Goals.
Directorate / Office Advisory Committee s GPRA
Directora te/OfficeAnnualReportsCOV Reports
At the end of the fiscal year, each directorate provides all GPRA-related documentation, including COV and AC reports, annual reports, and responses to COV recommendations, to the Office of the Director Concurrently, NSF staff finalize and submit data pertinent to the Management Goals and Investment Process Goals to the Office of the Director.
NSF makes use of several stages in the proposal and award process to assess performance These include the following steps:
A PPLICANT AND G RANTEE I NFORMATION /M ERIT R EVIEW
Applicants and grantees must provide comprehensive information, including results from previous NSF support, details about available facilities and equipment, the location of proposed activities, biographical data on primary investigators, other funding sources, and NSF-specific certifications This information is required at application, award, and in annual and final project reports Awards are granted based on a merit review conducted by field experts, utilizing NSF’s merit review criteria and resource availability, while also considering the quality of prior results This information contributes to the data reviewed by COVs when evaluating program performance against Outcome Goals.
A NNUAL P ROGRESS R EVIEW BY P ROGRAM O FFICERS
Program officers conduct annual reviews of awards lasting over one year, typically before the award's anniversary and before releasing further funding The Principal Investigator (PI) submits an electronic annual progress report via FastLane, detailing significant achievements, progress from the previous year, plans for the upcoming year, and any potential issues affecting the project's timeline and budget Upon approval of this report, the administering program officer authorizes the release of funds for the following year These annual progress reports are also accessible to COVs for evaluation purposes.
P ROGRAM E VALUATION BY C OMMITTEES OF V ISITORS (COV S )
To maintain top-notch quality in processing and recommending award proposals, the NSF assembles committees of qualified external evaluators (COVs) to conduct a thorough review of each program every three years The selection of COV members adheres to strict guidelines to guarantee independence, comprehensive programmatic coverage, and balanced representation These committees consist of independent experts from academia, industry, government, and the public sector.
COVs have traditionally assessed the integrity and efficiency of the processes for proposal review With the implementation of GPRA in FY
In 1999, the National Science Foundation (NSF) implemented a retrospective assessment to evaluate the quality of its program results through outputs and outcomes NSF requests Committee of Visitors (COVs) to highlight significant annual achievements, assess the collective impact of projects, and outline expectations for future performance The management reviews COV recommendations, which play a crucial role in shaping NSF's evaluation of current programs and guiding future initiatives.
In FY 2000, COVs evaluated the success of NSF programs in meeting Outcome Goals 1-4.a and implementing merit review criteria (Investment Goal 2) using a standardized reporting template This template includes core questions focused on process, program management, and outcome quality COVs are required to justify their assessments and provide examples of success The aggregated results of these evaluations are used to assess NSF's overall progress toward achieving its Outcome Goals.
A typical COV comprises six to twelve members who evaluate one or more programs over two days In fiscal year 2000, around 250 COV members engaged in the performance assessment process, contributing to the Outcome Goal assessment results.
The 55 COV reports assess 78 of NSF's nearly 200 programs, with some reports evaluating multiple programs simultaneously A schedule of program evaluations can be found in Section XV Electronic versions of these COV reports are set to be released in December 2001.
D IRECTORATE /O FFICE A SSESSMENT BY A DVISORY C OMMITTEES (AC S )
How Assessments and Evaluations are Utilized
Summary Table of FY 2000 Performance Goals and Results
Agency Results for FY 2000
Overall, NSF was successful in achieving 64% – 18 of 28 – of its performance goals Results in this second year are consistent with those obtained in FY
In 1999, key areas for improvement were identified, including the utilization of merit review criteria by both reviewers and applicants, enhancing customer service by reducing proposal decision times, and boosting participation from under-represented groups The NSF remains committed to enhancing performance in these critical areas.
This year, NSF implemented a more stringent evaluation process for assessing goal achievement compared to the previous year Grading for the qualitative Outcome Goals was restricted to two options: successful or not successful, with a mandatory justification required for any successful ratings.
In FY 2000, the National Science Foundation (NSF) improved its performance characterization for stated goals by enhancing presentation and processes, as well as incorporating insights from external expert committees These changes were partly driven by feedback from the Administration, Congress, and the private sector regarding the quality of information, justification of ratings, and the effectiveness of NSF's two-tiered rating system To address these issues, NSF enlisted PricewaterhouseCoopers LLP to verify achievement data for most goals.
Investment Process Goals 7 out of 14 (50%); one goal did not apply
Table 1 provides a concise summary of the agency's aggregated performance results, outlining each goal along with its corresponding outcomes A detailed analysis of the results for each goal can be found in Section V.
In FY 2000, the NSF successfully achieved six out of eight Outcome Goals, representing a 75% success rate External evaluators confirmed the achievement of Outcome Goals 1 and 2, while noting improvements in NSF's performance for Goals 3 and 4.a, which were not fully met Additionally, NSF accomplished the quantitative Outcome Goals 4.b, 4.c, 5.a, and 5.b.
External evaluators have observed improvements in programs aimed at enhancing diversity since FY 1999, particularly in the participation of under-represented groups; however, participation levels still fall short of expectations They emphasize that the National Science Foundation (NSF) needs to focus more on increasing this participation and suggest that some NSF program portfolios should incorporate more "high risk" activities Additionally, common challenges such as rising workloads and delays in proposal processing could hinder program performance Further insights into Outcome Goal results and the NSF's strategies for addressing these challenges are discussed in Section V.A.
In FY 2000, the National Science Foundation (NSF) successfully achieved five out of its six management goals, representing an impressive 83% success rate Key performance enhancements were noted in the orientation and training of NSF staff utilizing FastLane, the electronic platform for proposal submission, review, and project reporting Additionally, there was a marked increase in the adoption of the electronic Project Reporting System by awardees for their project reporting needs.
In FY 2000, NSF faced challenges in achieving its management goal of developing the capability for electronic proposal submissions, primarily due to difficulties in establishing electronic signature protocols To address this, NSF plans to pilot ten full electronic review projects in FY 2001 to evaluate the effectiveness of these protocols Further details on the Management Goals can be found in Section V.B.
Results for Investment Process Goals
In FY 2000, the National Science Foundation (NSF) achieved seven out of its fifteen Investment Process Goals, while seven were not met, and one facility goal was not applicable due to the absence of completed construction projects Key areas for improvement include enhancing the use of new merit review criteria, identifying best practices for customer service training, allowing three months for proposal preparation, reducing decision-making time, increasing awards for new investigators, ensuring timely facility upgrades and construction, and minimizing unscheduled downtime to under 10% of total scheduled operating time Detailed discussions on these goals and NSF's strategies for addressing these challenges can be found in Section V.C.
Table 1 summarizes the outcomes for each goal outlined in the NSF FY 2000 Performance Plan, with further details and discussions available in Section V of this report.
Table 1 FY 2000 Performance Goals and Results
ANNUAL PERFORMANCE GOALS and RESULTS FOR OUTCOMES
Outcomes Performance Goals for Outcomes Results
NSF is judged successful when
Discoveries at and across the frontier of science and engineering
NSF awards facilitate significant discoveries and foster the development of new knowledge and techniques, often transcending traditional disciplinary boundaries These awards also create valuable connections across various fields, as evaluated by independent external experts.
FY 1997 and FY 1998 information indicated successful achievement.
Judged successful by external experts in all reports.
Reports by external experts indicate that in the aggregate NSF is successful in achieving this goal.
Connections between discoveries and their use in service to society
NSF award results are quickly accessible and can be utilized in education, policy development, and by other federal agencies or the private sector, as evaluated by independent external experts.
1997 and FY 1998 information indicated successful achievement.
FY 1999: Goal achieved Judged successful in the aggregate by external experts who noted improvements can be made in some programs.
Judged successful in the aggregate by external experts who noted improvements can be made in some programs, as in
In 1997 and FY 1998, the global workforce of scientists and engineers demonstrated excellence in research and education by employing modern technologies and integrating international standards This success is evident when academia, government, business, and industry acknowledge the quality of their work Additionally, there has been a notable increase in the participation of under-represented groups within the science and engineering workforce, as assessed by independent external experts.
Judged successful in most areas by external experts
FY 2000: Judged successful in a limited context– goal not fully achieved in the aggregate Most programs with specific responsibilities for these areas were judged successful
Improvements still needed in the same areas that were identified in FY 1999.
ANNUAL PERFORMANCE GOALS and RESULTS FOR OUTCOMES – continued
Outcomes Performance Goals for Outcomes Results
Improved achievement in mathematics and science skills needed by all
NSF awards facilitate the creation and implementation of effective educational models and practices that cater to the diverse needs of all students These initiatives ensure that teachers are well-trained in standards-based approaches, leading to enhanced student performance in schools and districts, as evaluated by independent external experts.
Baseline: Preliminary pilot efforts did not provide sufficient information to yield a valid baseline.
Judged successful in the aggregate by external experts for programs to which goal applies.
In FY 2000, the NSF was deemed successful in a specific context, although it acknowledged that the overall goal was not fully achieved Success was evident in programs that had clearly defined objectives aligned with this goal.
Over 80 percent of schools participating in a systemic initiative program will:
(1) implement a standards-based curriculum in science and mathematics;
(2) further professional development of the instructional workforce; and
(3) improve student achievement on a selected battery of tests, after three years of NSF support.
Through systemic initiatives and related teacher enhancement programs, NSF will provide intensive professional development experiences annually for at least 65,000 precollege teachers.
ANNUAL PERFORMANCE GOALS and RESULTS FOR OUTCOMES– continued
Outcome Performance Goals for Outcomes Results
Timely and relevant information on the national and international science and engineering enterprise.
Maintain FY 1999 gains in timeliness for an average of 486 days as the time interval between reference period (the time to which the data refer) and reporting of data
Table of FY 2000 Performance Goals and Results
Findings from Program Assessments and Evaluations
The findings from program assessments and evaluations conducted in FY 2000 are outlined in Section V, covering Outcome Goals in Section V.A., Management Goals in Section V.B., and Investment Process Goals in Section V.C Notably, the results for Management Goals and Investment Process Goals, except for Investment Process Goal 2, are compiled by NSF staff utilizing central data systems and are not evaluated in the COV assessment process.
The findings for Outcome Goals are derived from summarized judgments in the Committee of Visitor (COV) and Advisory Committee (AC) assessment reports, which aggregate general findings and goal ratings across the NSF These results are qualitatively summarized for Outcome Goals 1, 2, 3, and 4a in Section V.A Additionally, the COV and AC reports provide the basis for the results related to Investment Goal 2, detailed in Section V.C.
This article outlines each Outcome Goal with a brief introduction, annual performance goals, and indicators for the current fiscal year Following the Performance Indicators, aggregated results from the assessment process are presented, along with a discussion on performance and future plans Successful performance examples identified by COVs and ACs highlight the impact of NSF support, organized by goal and area of emphasis as detailed in the FY 2000 Performance Plan Notably, some Outcome results may pertain to multiple goals or areas of emphasis.
Independent contractors conduct studies and evaluations to tackle specific issues beyond the GPRA performance goals, complementing assessments from COVs and ACs The evaluations completed in FY 2000, detailed in Table 2 of Section V.D., are primarily not utilized in performance assessments, except for one COV report Nevertheless, the insights gained from these evaluations are valuable for programs to pinpoint issues and explore opportunities for future investments.
Outcome Goals and Results
The true value of NSF investments is assessed through long-term outcomes, which may stem from research or training conducted decades ago NSF’s Outcome Goals aim to advance science and engineering while enhancing the health, security, and quality of life for U.S citizens These goals emphasize the impact of NSF-funded research and education in science, mathematics, and engineering, aligning with NSF's mission A critical strategy for achieving these objectives is the implementation of rigorous merit review to allocate funds for initiatives that will have lasting effects on research and education.
Outcome Goals are defined qualitatively, and the progress of the National Science Foundation (NSF) in reaching these goals is evaluated through qualitative assessments of program outputs and outcomes This evaluation is conducted by external groups known as Committees of Visitors (COVs) and advisory committees (ACs) For further details on the role of COVs and ACs, please refer to Section III, “Assessment and Evaluation Process.”
In FY 2000, performance results from previous years' awards highlight notable achievements across various NSF-supported programs, Committees of Visitors, and Advisory Committees These selected examples showcase the diverse activities funded by NSF and illustrate the significant impact and success of its initiatives Each example is accompanied by a specific NSF grant number for verification purposes, providing insight into the breadth of NSF's contributions to research and development.
Summary of FY 2000 Results for Outcome Goals
External evaluators have consistently rated NSF’s programs as producing high-quality outputs and outcomes, indicating that these initiatives effectively further NSF’s mission to advance science and engineering The results from the second year of GPRA reporting align closely with those from the previous fiscal year.
In 1999, emerging trends were identified through the evaluation of various subsets of the NSF's program portfolio by different external evaluators each year This second year of reporting has proven crucial for NSF, as it highlights areas requiring attention and assists in pinpointing future improvement opportunities.
External evaluators in FY 2000 found that NSF programs successfully met Outcome Goals 1 and 2, while making limited progress on Outcome Goals 3 and 4.a, which we classify as not achieved However, the two quantitative sub-goals of Outcome Goal 4, along with both sub-goals of Outcome Goal 5, were successfully achieved Overall, we report that six out of eight Outcome Goals were met in FY 2000, following a complete achievement of all Outcome Goals in FY 1999.
In FY 2000, evaluators highlighted the same improvement areas as in FY 1999, particularly emphasizing the need for enhanced diversity While some reports show progress in the participation of under-represented groups, the overall numbers remain lower than anticipated for a meaningful impact Evaluators stress that the National Science Foundation (NSF) must focus more on increasing the participation of these groups to drive significant change.
To enhance program performance, it is essential to improve the balance of the portfolio by funding more high-risk proposals and to ensure that both of NSF’s merit review criteria are utilized effectively by applicants and reviewers Reports indicate that NSF Program Directors actively apply these criteria in their funding decisions; however, challenges such as increased workload and delays in proposal processing persist, hindering overall efficiency.
In FY 2000 NSF limited options for grading to either successful or not successful, and required clear justification for successful grades for qualitative measures An outside accounting firm verified
Discoveries at and Across the Frontier of Science and Engineering
The NSF fosters advanced research in science, engineering, and education, driving significant discoveries that enhance the nation's scientific and engineering capabilities These breakthroughs are crucial for developing innovative technologies that ultimately benefit society.
The pursuit of new knowledge, including innovative ideas, theories, tools, and methods, fosters a deeper understanding of complex problems and paves the way for economic growth This quest for discovery fuels the creativity and efforts of scientists and engineers, leading to innovations that significantly contribute to ongoing economic development and enhance the quality of life for all Americans.
NSF aims to achieve its objectives by backing the most promising research and education ideas, identified through a rigorous merit review process of competitive proposals This approach emphasizes innovation, creativity, and collaboration.
importance and quality of discoveries, new ideas, new tools, and new technologies;
interplay of disciplinary and interdisciplinary research; and
NSF's performance toward this Outcome Goal is successful when NSF awards lead to:
new knowledge and techniques, both expected and unexpected, within and across traditional disciplinary boundaries; and
identification of high-potential links across these boundaries as judged by independent external experts
External evaluators acknowledge that the significant impacts of research discoveries often take time to materialize, typically ranging from 3 to 10 years for effects to be seen in the private sector, and 15 to 20 years for broader implications to become evident.
Pilot projects used FY 1997 and FY 1998 information and expert judgment in performance assessments that indicated NSF was successful in meeting this goal
Comparison of actual performance with projected performance
Around one-third of the National Science Foundation's (NSF) program portfolio underwent evaluation by Committees of Visitors (COVs) to assess progress towards achieving specific outcome goals Detailed information regarding the evaluation process can be found in Section III, titled “Assessment and Evaluation Process.” In the fiscal year 2000, evaluators were tasked with determining the success of the programs in meeting these established goals.
The agency's results were compiled using reports from COVs and Advisory Committees, focusing exclusively on those with substantial comments and well-justified ratings All reports that assessed this goal indicated that the NSF successfully met its objectives in FY 2000, leading us to conclude that this goal has been achieved.
Annually, the NSF requests Committees of Visitors (COVs) to evaluate project support portfolios, focusing on identifying high-risk, multidisciplinary, or innovative activities They also assess the overall scientific quality and balance concerning these specific characteristics.
Management Goals and Results
NSF’s Management Goals focus on enhancing administrative, operational, and policy objectives to ensure excellence in agency performance For FY 2000, the six Management Goals prioritize staff training, staff diversity, and the integration of advanced technology into NSF operations Five of these goals build on previously established targets with stricter performance indicators, while the new goal emphasizes a commitment to fully incorporating technology into the Foundation's core activities.
Four factors are especially critical to successful management at NSF:
Key factors for establishing annual performance goals include electronic proposal submission and processing, staff diversity, staff technological training, Y2K compliance, and the utilization of electronic systems for project reporting The outcomes of these management goals, primarily measured quantitatively, are compiled and assessed by NSF staff, and are organized by performance area.
Summary of Results for Management Goals
In FY 2000, NSF successfully achieved five out of its six Management Goals, with notable improvements in the orientation and training of staff on FastLane, the electronic system for proposal submission and project reporting Additionally, there was an increase in the use of the electronic Project Reporting System among awardees However, the goal related to enhancing technological capabilities for electronic proposal submission and processing was not met due to complex challenges in establishing protocols for electronic signatures NSF conducted two pilot programs to address these issues.
FY 1997 FY 1998 FY 1999 FY 2000 FY 2001
FastLane is a comprehensive suite of electronic system modules designed to streamline all transactions and communications between the National Science Foundation (NSF) and its grantees through the Internet In development since 1994, FastLane significantly contributes to NSF's objective of establishing a paperless environment by the end of the project.
In FY 2001, the ambitious goal established in FY 1999 was sustained, with its target level of performance elevated based on real-time results and the actual performance observed in FY 1999 and FY 2000 This ongoing commitment reflects a continuous improvement approach to achieving higher standards.
In FY 2000, FastLane processed 25,160 proposals, representing 81% of the total 30,932 submissions This achievement was driven by a robust outreach strategy and the support of a dedicated Helpdesk, which handled over 35,000 assistance requests, with around 90% focusing on proposal preparation and submission.
Performance area: Electronic proposal submission
The research and education communities have collaborated with NSF staff to develop FastLane, a web-based interface for grantee institutions Each module of FastLane has seen an increase in usage, with the submission of full technical proposals being the most complex function Notably, NSF is the only federal research agency that accepts electronic proposals on a production basis, and as of FY 2001, electronic submission has become mandatory for NSF, with exceptions made only in special circumstances.
In FY 2000, NSF will receive and process at least 60% of full proposal submissions electronically through FastLane.
Percent of full proposal submissions received electronically throughFastLane.
In September 2000, the NSF Director announced Important Notice 126, emphasizing that starting October 1, 2000, all specified transactions with NSF must be conducted electronically through the FastLane system This notice is available for reference at the NSF website.
FastLane is gaining widespread adoption among external customers for proposal submissions, with a notable increase in program initiatives requiring FastLane submissions in FY 2000 Looking ahead, nearly all programs are expected to mandate FastLane submissions in FY 2001.
For FY 2001, the submission goal has been increased to 95% of full proposals, aligning with the requirements outlined in Important Notice 126 This target acknowledges that certain universities, colleges, and individuals with disabilities may face challenges in electronic submission due to transmission issues or lack of technical capability.
This goal was not achieved.
To improve operational efficiency, the NSF has implemented mandatory electronic submission for grant proposals Once submitted, these proposals are forwarded to the relevant office for administrative processing and undergo peer review, leading to the preparation of recommendations.
The National Science Foundation (NSF) processes around 30,000 grant proposals each year, making funding decisions and preparing award or declination letters Traditionally, NSF required paper submissions after electronic proposal submissions, but modernization efforts have been in progress for several years The aim is to transition to a fully electronic system for the entire internal review and award/decline process.
At the beginning of the year, the peer review process relied on paper for four key functions: communication between NSF and peer reviewers, the electronic panel review system, letters to principal investigators (PIs) regarding declined proposals, and the release of review results to PIs However, by year-end, NSF successfully eliminated most technological barriers to achieving a fully paperless process, with only one issue remaining—the need for an electronic signature for funding approval.
In FY 2000, two pilot projects for electronic signatures were launched, with evaluations taking place in FY 2001 to identify the most effective method for the agency and its clients However, several technological, financial, and legal challenges remain to be addressed before electronic signatures can be fully implemented.
Performance Area: Electronic proposal processing
The National Science Foundation (NSF) currently relies on paper processing for the review and management of proposals However, NSF aims to transition to a fully electronic processing system, ultimately eliminating the internal paper procedures currently in place.
Investment Process Goals and Results
The National Science Foundation (NSF) employs external merit review as a fundamental strategy to award funding for initiatives that significantly influence research and education in mathematics, science, and engineering This investment process centers around competitive merit evaluations conducted by external peers, guided by two established criteria.
The National Science Board emphasizes the importance of NSF's program staff, who consider both NSF priorities and external reviewer feedback when shaping their award portfolio A successful investment process relies on effective strategies and high-quality proposal and award procedures that align with Outcome Goals and fulfill customer expectations.
Summary of Results for Investment Process Goals
In FY 2000, the National Science Foundation (NSF) achieved seven out of its 15 Investment Process Goals, while seven goals were not met, and one goal was deemed inapplicable Key areas for improvement include enhancing the implementation of Merit Review Criteria by reviewers and program officers, ensuring new program announcements and solicitations are available at least three months before deadlines, reducing decision times to six months or less for 70% of proposals, and fostering transparency to increase awards for new investigators to 30% To ensure accuracy, NSF enlisted an external accounting firm to verify the data systems related to most Investment Process goals.
The Investment Process Goals of the NSF focus on enhancing the awards process through merit review and fostering an inclusive environment for new ideas and participants These goals set customer service standards and aim to improve proposal processing times They also encompass facilities oversight relevant to federal science and technology agencies Most goals include quantitative measures reviewed by NSF staff, while Investment Process Goal 2 is qualitatively evaluated by external experts Results are discussed in key performance areas, including Proposal and Award Processes, Customer Service, System Openness, Integration of Research and Education, Diversity, and Facilities Oversight.
Based on NSF’s original goal, which included merit reviewed projects as a percentage of all NSF funding, the
Foundation exceeded its goal of 90% for FY 2000 As in
In FY 1999, the NSF successfully allocated 95% of its funds to projects that underwent merit review, a standard that was upheld in FY 2000 Moving forward, this allocation strategy will be adjusted in FY 2001 in accordance with revised definitions from the OMB.
Percent of project funding subject to merit review
FY 1997 FY 1998 FY 1999 FY 2000 FY 2001
*N.B Based on old OMB definitions During FY 2000, the Office of Management and Budget revised the federal goal, stating that
70-90% of research and development funds should be awarded to merit reviewed projects Under the new definition, federally- funded research and development centers (FFRDCs) and merit-
Performance area: Proposal and Award Processes - Use of Merit Review
The NSF policy mandates that every funding or non-funding recommendation for a proposal must include at least three external merit reviews and a balanced discussion of those reviews, with an average of 5 to 9 reviews per proposal This merit review process, which evaluates the quality of the proposed project and its potential broader impact, is essential for NSF's funding decisions in research and education The Foundation emphasizes that a competitive merit review process, supported by peer evaluations, helps identify the most promising ideas from leading researchers and educators In FY 2000, over 29,400 competitive proposal decisions were made, with more than 46,000 external reviewers participating and over 8,700 serving as panelists.
At least 90% of NSF funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process
Percent of NSF funds allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.
**Based on the most recent definitions from OMB, the revised percent of project funding subject to merit
Largely successful, needs some improvement.
In FY 2000, the National Science Foundation (NSF) determined that scientific research subjected to competitive selection and internal program evaluation would not qualify as merit-reviewed Consequently, NSF has adjusted its target for FY 2001 to achieve an 85% merit review rate, reflecting this updated definition.
New criteria went into effect in early FY 1999 External expert judgment is used to assess performance The assessment process was used for the first time during
Performance area: Proposal and Award Processes - Implementation of Merit Review
The implementation of merit review criteria is crucial in the proposal selection process, ensuring that the most deserving projects receive support In FY 1998, the National Science Board revised the NSF merit review criteria to align with the NSF Strategic Plan, introducing two key criteria that evaluate a proposal’s quality and broader impact These criteria focus on assessing technical merit, creativity, educational impact, and societal benefits through expert evaluations The integration of both quality and impact assessments by expert reviewers and program staff is vital for achieving NSF's overarching goals.
External committees evaluate the NSF's progress in achieving its goals by assessing how reviewers and program officers apply the two merit review criteria The findings of this assessment are detailed below, utilizing the non-quantitative format permitted by the Act.
NSF’s performance in implementation of the new merit review criteria is successful when:
reviewers address the elements of both generic review criteria appropriate to the proposal at hand; and
when program officers take the information provided into account in their decisions on awards, as judged by external independent experts.
Use of merit review criteria by reviewers and program staff.
Achieving optimal performance in this goal necessitates that both merit review criteria are thoroughly addressed by reviewers and program staff However, external evaluators have determined that the NSF did not fully succeed in this endeavor.
In FY 2000, COVs evaluated 78 NSF programs to assess their success in meeting performance goals Of the 64 reports reviewed, 58 rated the programs based on merit review criteria The NSF was deemed successful in achieving its objectives in 20 instances.
In instances where NSF proposals were unsuccessful, reviewers often overlooked the second merit review criterion related to the broader impacts of the proposed activities, or applicants failed to adequately address these impacts in their submissions However, most Committee of Visitors (COV) assessments indicated that NSF staff effectively considered both merit review criteria in their decision-making process.
The two merit review criteria were not implemented until fiscal year 1998, meaning that the program assessments conducted by COVs in fiscal year 2000 included proposals reviewed prior to this implementation, specifically from fiscal year 1997 Consequently, it is important to recognize that the full utilization of these criteria by reviewers and staff during this assessment period cannot be expected.
The FY 2001 assessment will encompass proposals evaluated from FY 1998 onward, marking the inaugural review of the comprehensive implementation of the two criteria The complete utilization of these criteria is expected to be more evident in FY 2001.
In FY 1999, program achievement was categorized into two levels: successful and minimally effective, each with specific indicators Most reports during this period rated programs as successful in applying merit review criteria However, instances of limited success were often linked to a lack of comprehensive attention to both review criteria by reviewers and proposers Feedback from evaluators revealed that the descriptors for the minimally effective performance level did not enhance the evaluation process for these programs.
Table of Evaluations
Quality of the Reported Performance Information
In FY 1999 concerns were expressed by the General Accounting Office (GAO) with regard to the quality of reported performance information used by NSF.
In FY 2000, the National Science Foundation (NSF) engaged PricewaterhouseCoopers LLP (PwC) to independently verify and validate its GPRA performance data and the processes for data collection PwC's final reports indicated that NSF's reporting was sufficiently accurate, stating that any potential errors would not significantly alter the interpretation of the Foundation's success in achieving its performance goals.
“relies on sound business processes, system and application controls, and manual checks of system queries to confirm the accuracy of reported data.
We believe that these processes are valid and verifiable.”
KPMG LLP, an independent certified public accounting firm, was chosen by the NSF Inspector General to conduct an audit of NSF's FY 2000 financial statements The audit encompassed an evaluation of the data collection process related to NSF's GPRA goals Ultimately, NSF received an unqualified opinion, affirming that its principal financial statements were fairly presented in all material respects, with no reported material weaknesses in internal controls or noncompliance with laws and regulations.
All data inherently possess imperfections, necessitating a balanced approach to verification and validation procedures while acknowledging data limitations The National Science Foundation (NSF) recognizes the importance of enhancing data systems for performance information collection and maintenance, viewing this as an ongoing evolutionary process that improves over time NSF is confident in the quality of the data utilized to evaluate its progress towards performance goals and leverages insights from performance reporting to refine its policies and management practices The implementation of the Government Performance and Results Act (GPRA) has allowed NSF to systematically gather information and address issues in a more structured and focused manner than previously possible.
Basic research and education projects often take three to five years or longer to yield results, making it challenging to align reported outcomes with annual funding In some instances, the impact of NSF support may not be evident for two decades NSF conducts annual assessments of results submitted within the fiscal year through retrospective evaluations by external experts, covering approximately thirty percent of its total portfolio This approach aligns with NSF’s long-term Outcome Goals, which are not designed for quick achievement However, the current GPRA reporting format may not fully capture NSF's accomplishments or the overall value of its investments To illustrate the performance level related to these Outcome Goals, the article provides examples of achievements linked to NSF-supported awards, despite their seemingly anecdotal nature.
Data Verification and Validation Activities
For reporting goal achievment, all of NSF’s goals are aggregated across the agency To enable aggregation, NSF developed reporting templates in FY
In 1998, data modules were introduced to standardize data collection across the agency These modules and templates underwent revisions in FY 2000, incorporating insights gained from their implementation in FY 1999.
In 1999, the National Science Foundation (NSF) launched a Data Quality Program aimed at evaluating and enhancing data quality within the organization The NSF is committed to continuously refining its data collection methods and systems to improve identified areas, as resources and time permit.
During FY 1999 NSF staff implemented a Data Quality Project for the quantitative Investment Process and Management goals The objectives of the project were to:
1 Evaluate the quality of the data in the central databases.
2 Ensure the paper documents and the NSF central databases are synchronized.
3 Identify inconsistencies so that methods for correcting the cause of the inconsistencies can be developed.
4 Ascertain the causes of the data quality problems and develop systematic methods for correction.
6 Promulgate data quality policies and procedures NSF-wide.
In FY 2000, NSF increased the expected quality of information for the Outcome Goals in two ways:
1 NSF changed the two-level standard from successful/minimally effective to successful.
2 NSF required thorough justification for “grades.”
The NSF annually updates and revises guidelines and reporting procedures for collecting data related to Outcome Goals In FY 1999 and 2000, the Committee of Visitor (COV) guidelines were modified to align with GPRA-related reporting requirements, addressing a standardized set of questions for all programs reviewed within the fiscal year Additionally, reporting guidelines for Advisory Committees were established to ensure uniform and systematic data aggregation The implementation of these new procedures revealed areas for improvement, leading to revisions for FY 2000 reporting, with further updates planned for FY 2001 based on insights gained This experience has also informed the revisions of the FY 2001 and FY 2002 Performance Plans and goals, as well as the updated Strategic Plan.
In FY 2000, NSF established criteria to assess the acceptability and reliability of the qualitative information it utilizes, ensuring uniform quality in results Stricter definitions of success were implemented to evaluate the achievement of Outcome Goals Additionally, NSF employed a confidence limit to distinguish non-substantive information, excluding any data that fell outside this limit from consideration.
In FY 2000, the implementation of stricter definitions aimed to elevate performance expectations and consequently lowered the overall success rate for the NSF Despite this, the agency's performance remained consistent with FY 1999, showing emerging positive trends Notably, many issues highlighted by external groups in FY 1999 were also recognized in FY 2000, suggesting a surprising consistency in evaluations despite the involvement of different expert groups each year This continuity in findings reinforces the validity of the initial year's results, with a more comprehensive understanding anticipated as the third year's data becomes available.
NSF staff validate information from external sources to measure performance against Outcome Goals, which is further reviewed by external evaluators and tracked through grant numbers The agency relies on reports from COVs for performance assessment, compiling scores and comments to evaluate success in achieving these goals PwC reviewed this process and deemed NSF's approach to assessing performance based on qualitative measures as reasonable, confirming that NSF's results align with their own evaluations.
“overall conclusions regarding program success or lack thereof in respect to individual goals remained largely unchanged.”
NSF is expected to consistently engage external third parties to verify and validate the data utilized in performance goal reporting, contingent upon the availability of funds for this initiative.
Types and Sources of Performance Data and Information
Performance measurement relies on diverse data sources, many of which are external to the agency and beyond NSF's control This data is derived from administrative records, awardee reports, external committee evaluations, and internal data systems For more details, refer to the FY 2001 Performance Plan.
Quantitative data plays a crucial role in evaluating the Investment Process and Management Goals Typically, this data is gathered through internal data systems and is analyzed by staff on a quarterly basis to assess performance effectively.
Qualitative data for evaluating Outcome Goal performance is primarily supplied to the National Science Foundation (NSF) by external evaluators (COVs) towards the end of the fiscal year, and this information is subsequently reviewed by NSF senior management as it is received.
Data collection is contingent upon the specific type of information required and occurs year-round, culminating at the end of the fiscal year Reports are generated for each goal by designated staff responsible for reporting Senior management continuously reviews the collected data to assess performance, determine if targets require adjustment, and evaluate the utility of the information for the agency Additionally, data collection systems undergo regular monitoring and refinement to ensure effectiveness.
Data Limitations
Specific data limitation issues are discussed below The NSF FY 2001 Performance Plan contains additional information on data sources and limitations
In its second year of performance assessment, the NSF identified several data quality and limitation issues from the previous year To address these challenges during FY 2000, the agency focused on improving incomplete data collection systems linked to quantitative goals, particularly those concerning Facility Operations – Investment Goals 12-15 Additionally, efforts were made to enhance report templates to ensure that performance information from external groups is more comprehensive and consistent for qualitative Outcome Goals, as well as to provide clearer explanations for any missed goals.
To enhance the quality and value of performance data for Outcome Goals, significant steps have been implemented, including the introduction of improved reporting templates that require more comprehensive justifications from external committees regarding their ratings While there has been a noticeable improvement in the quality and consistency of COV reports for FY 2000, they still fall short of optimal standards In response, NSF has revised the COV reporting template guidelines for FY 2001 to further enhance consistency and completeness Additionally, NSF staff will collaborate more closely with COV members to ensure better reporting, facilitating the aggregation of qualitative information necessary for measuring progress towards the Outcome Goals.
In FY 1999, the National Science Foundation (NSF) utilized an alternative method to assess progress towards its Outcome Goals in research and education, employing a two-standard approach of "successful" or "minimally effective." However, this method proved to be largely unbeneficial and often confusing for evaluators Consequently, in FY 2000, NSF adopted a single standard—designating only "successful" as the benchmark—thereby raising the expected performance level for its Outcome Goals.
In its performance evaluations, the NSF requests COVs to report instances where data is insufficient or unavailable for assessing progress towards Outcome Goals Findings from FY 1999 and FY 2000 revealed that external evaluators often lacked adequate information to evaluate programs based on merit review criteria, to assess participation from under-represented groups, and to measure science and math skills among all Americans In response, NSF management is exploring strategies to enhance data availability for FY 2001 assessments However, challenges remain, particularly in obtaining comprehensive data on the voluntary participation of under-represented groups.
The National Science Foundation (NSF) is revising its goal wording to address challenges encountered in aggregating results from fiscal years 1999 and 2000 Specifically, Outcome Goal 3, which aims to enhance both diversity and a globally-oriented workforce, revealed mixed results While NSF successfully fostered a globally-oriented workforce across most programs, it fell short in achieving increased diversity As a result, NSF could not demonstrate overall success for this goal in FY 2000, despite some programs effectively targeting and realizing specific aspects of the objective.
In evaluating Outcome Goal 4, which aims to enhance math and science skills for all Americans, it was found that NSF programs that directly allocated funds towards this goal were successful However, external groups struggled to assess the success of programs not specifically aimed at this objective, leading to COV reports that lacked clear evidence of overall success Consequently, we cannot confirm successful performance for the agency regarding this goal.
In FY 2001, NSF introduced a new format for its goals, organizing the five Outcomes Goals into three categories, each with distinct indicators This structure aims to facilitate assessments by COVs, allowing for a focused evaluation of the relevant indicators pertaining to specific programs.
A limitation identified in Investment Process Goal 8, which focuses on maintaining openness in the system, highlighted occasional inaccuracies in identifying new Principal Investigators (PIs) To address this issue, measures were taken to improve the identification process within the NSF PI system Additionally, Investment Process Goals 12-15, related to facility oversight, saw a revision and implementation of the reporting system in the fiscal year.
In 2000, facility managers at universities utilized the NSF-developed system to report data supporting their operational goals A lack of consensus in FY 1999 regarding data definitions resulted in varied interpretations, a problem that was resolved in FY 2000 Despite this improvement, facility managers continue to gain experience in the collection and reporting of necessary information.
Quantitative Management Goal 3 focuses on Staff Diversity; however, a significant data limitation arises because applicants are not legally obligated to disclose their gender and ethnicity As a result, the accuracy of the data for this goal is questionable, and any objectives that rely on voluntary self-reporting may also suffer from incompleteness or inaccuracies.
In FY 1999, NSF faced challenges in understanding the reasons behind unmet goals To address this, in FY 2000, NSF revised its report templates to enhance information collection across its directorates and offices Each reporting unit was tasked with summarizing performance at a more granular level and providing explanations for any unmet agency goals, along with strategies for improvement This approach yielded more comprehensive insights into goal performance and highlighted key obstacles to achieving certain objectives The gathered information is now being utilized to formulate implementation strategies aimed at meeting targets in future fiscal years.
Other Issues – Timing
Major Management Challenges
The United States Senate Committee on Governmental Affairs performs annual oversight of federal agencies to enhance service delivery to citizens and taxpayers Collaborating with the Office of the Inspector General (OIG), the committee requests that each agency's OIG identify the top ten significant long-term management challenges they face.
In an FY 1999 report to the Senate Committee (letter dated 1 December
1998), the NSF OIG identified ten significant NSF management challenges:
NSF responded to the Senate Committee in a letter dated September 28,
1999, noting that the NSF IG had stated overall that the Foundation is well managed, and accordingly found these issues to be challenges, rather than managerial “deficiencies”
In FY 2000, the NSF Office of Inspector General (OIG) presented a report to the Senate Committee, highlighting ten critical management challenges faced by the National Science Foundation (NSF) The report emphasized concerns regarding the NSF's overall investment portfolio, underscoring the need for improved oversight and strategic management.
Major Management Challenges for FY 1999 and FY 2000
1 *Managing an effective merit review system
2 Capitalizing on NSF strengths when responding to increased expectations
3 Using the Government Performance and Results Act
4 Responding to the Chief Financial Officers Act
7 Sustaining high scholarship and integrity
8 Spending funds effectively and efficiently
9 Managing an effective system for cost sharing
10 Managing salaries and administrative resources
An * denotes areas included under goals in the FY 1999 and FY
In FY 2000, the Foundation successfully implemented 2000 Performance Plans, demonstrating a healthy operational status without major management deficiencies The Office of Inspector General (OIG) noted that the challenges faced by the National Science Foundation (NSF) remained consistent with those identified in FY 1999 However, the OIG acknowledged that Foundation management made ongoing progress in addressing these challenges throughout FY 2000.
In its FY 1999 Performance Report to Congress, the NSF did not address all major management challenges identified by the NSF OIG However, the OIG indicated to GAO that NSF is effectively responding to these challenges and does not need to include every issue in performance reports or future plans Despite this, the OIG raised concerns regarding effective oversight and the necessity for NSF to remain vigilant about emerging issues that could lead to problems In response, NSF senior management actively reviews and monitors these areas, incorporating select issues into their annual performance plans.
For FY 2001, the NSF OIG has identified 10 areas (see below) including some areas identified in previous years, and several new areas for NSF to monitor:
The NSF has established performance goals in its annual plans for FY 2001 and 2002, focusing on merit review processes, the utilization of FastLane, workforce training initiatives, and enhancing diversity within the scientific workforce Additionally, the NSF implements internal management controls to consistently monitor the effectiveness of its awards.
5 Management of Large Infrastructure Projects
8 Work Force Planning and Training
The NSF is dedicated to upholding the highest integrity standards to deliver exceptional outputs and outcomes This commitment encompasses various goals outlined in the FY 2001 performance plan, including administration, cost sharing by awardees, data security, quality, and project management The organization continuously strives to enhance its performance in these critical areas and beyond.
Transition to FY 2001 and Beyond
The NSF FY 2001 Performance Plan is based on NSF’s updated GPRA
The Strategic Plan for FY 2001 – 2006, finalized in September 2000, introduces new Strategic Outcomes that align with previous NSF GPRA documents The key areas of focus—developing People, enabling Ideas, and providing Tools—connect NSF's mission with its annual performance goals The goals outlined in the FY 2001 Performance Plan reflect insights gained from FY 1999 and FY 2000, addressing identified strengths and weaknesses while incorporating recommendations from the NSF Strategic Planning process.
Integration Group, and input from the research community, auditors,
Congressional groups and stakeholders have identified that the performance indicators for the Outcome Goals may be overly broad Further details regarding the annual performance goals and indicators related to these Outcome areas can be found in the NSF FY 2001 report.
To promote the progress of science, to advance the national health, prosperity, and welfare, and to secure the national defense
Development of a diverse, internationally competitive and globally engaged workforce of scientists, engineers, and well-prepared citizens.
Providing broadly accessible state-of-the-art information bases and shared research and education tools Ideas
Enabling discovery across the frontier of science and engineering, connected to learning, innovation and service to society.
Discoveries at and across the frontier of science and engineering.
The integration of scientific discoveries into societal applications is essential for progress A diverse and globally-minded workforce of scientists and engineers is crucial for driving innovation Enhancing mathematics and science skills among all Americans is necessary for achieving greater educational outcomes Additionally, providing timely and relevant information about the national and international science and engineering landscape is vital for informed decision-making and advancement.
Information on Use of Non-Federal Parties
This GPRA performance report was written and prepared solely by NSF staff.
Non-Federal external sources of information used in preparing this report include:
Reports from awardees demonstrating results.
Reports prepared by evaluators – Committees of Visitors and Advisory Committees – in assessing NSF programs for progress in achieving Outcome Goals.
Reports prepared by a consulting firm to assess the procedures the Foundation uses to collect, process, maintain, and report performance goals and measures.
Highlights or sources of examples shown as results may be provided by principal investigators who received support from NSF.
The National Science Foundation (NSF) relies on external committees to evaluate the advancement of its programs in relation to Outcome Goal achievement These external evaluators deliver detailed reports and feedback based on a specific template provided by NSF The Foundation utilizes these committee reports, such as the Committee of Visitors (COV) and Advisory Committee reports, to gain an independent assessment of its performance and to gauge progress toward its strategic objectives.
NSF commissioned PricewaterhouseCoopers LLP to perform an independent review of its performance data related to Outcome Goals, Management Goals, and Investment Process Goals This external assessment aimed to mitigate potential reporting bias inherent in self-assessments, thereby enhancing the credibility of NSF's performance reporting information and results.
Budget Information
NSF obligated $3.9 billion in FY 2000 Administrative support for theFoundation was approximately 5% of the total NSF budget.
Classified Appendices Not Available to the Public
Analysis of Tax Expenditures
Waivers of Administrative Requirements
Appendix of Additional Examples Illustrating Outcomes of NSF Investments
FY 2000 Examples of Achievements Cited by External Evaluators
Discoveries At and Across the Frontier of Science and Engineering
External evaluators highlighted several successful outcomes from NSF awards that align with Outcome Goal 1 These outcomes showcase significant discoveries and innovative techniques, both anticipated and surprising, that transcend traditional boundaries and foster valuable connections across diverse fields.
The NSF-supported outcomes from FY 2000 highlight a balance of innovative and interdisciplinary research, showcasing advancements in areas such as extreme environments, biocomplexity, and nanoscience Notably, many findings bridge the gaps between discoveries and new knowledge, emphasizing the interconnectedness of these fields The diverse range of awarded projects demonstrates significant potential for impactful contributions in these critical areas of research.
Turbulent flow of fluids plays a crucial role across various disciplines, including atmospheric sciences, combustion science, and fundamental fluid physics In engine design, the turbulence generated by fuel combustion significantly influences fuel efficiency and pollutant emissions Monitoring fluid flow characterized by high Reynolds Numbers is essential, as low values indicate smooth flow while high values signify chaotic behavior Recent advancements have utilized particle physics techniques and detectors to track small buoyant particles, effectively mapping complex turbulence patterns Additionally, groundbreaking studies have explored turbulence across a wide range of Reynolds Numbers using cryogenic helium at a critical temperature of 4.5 Kelvin, enhancing our understanding of fluid dynamics.
At 0 Kelvin, known as absolute zero, all atomic and molecular motion theoretically stops, making it colder than -273 degrees Celsius, with most materials solidifying at higher temperatures Liquid helium is preferred for ultra-high Reynolds number studies due to its unique physical properties, enabling scientists to generate turbulent flow at potentially the highest Reynolds numbers achievable on Earth within a cryostat Research on turbulence at these extremely low temperatures is expected to significantly influence the fields of aeronautical, chemical, and mechanical engineering.
The H2O project utilizes retired undersea telephone cables to create a seismological observatory on the seabed, located approximately 5 kilometers (about 4.6 miles) deep in the Pacific Ocean, between California and Hawaii This innovative adaptation will facilitate the detailed study and characterization of the Earth's mantle beneath the northeastern Pacific Ocean.
An international project in Antarctica has yielded significant insights into the long-term evolution of the region's climate, ice sheets, marine ecosystems, and geological features The documented changes in Antarctica provide crucial information to understand the complex shifts in global climate that occur over millions of years.
Recent findings indicate that the tundra regions on the Alaskan North Slope have transitioned from being greenhouse gas sinks to sources during winter, highlighting their potential impact on the global greenhouse gas budget Additionally, rapid and episodic algal blooms in the Ross Sea of Antarctica have been associated with changes in carbon export from the ocean surface, suggesting that these blooms could significantly influence atmospheric carbon dioxide concentrations when considered on a larger scale.
The National Science Foundation (NSF) has recently provided funding to advance the development of a small, long-range robotic aerosonde intended for cold regions Traditionally, such aircraft lack anti-icing capabilities to optimize weight and endurance In August 2000, successful flight tests were conducted over two weeks to evaluate new instruments, ice detectors, and anti-icing coatings These promising results will pave the way for further testing with additional miniature instruments, an enhanced aerosonde design, and a new catapult launch system.
An interesting example of research focused on Life in ExtremeEnvironments is the discovery of bacteria actively metabolizing at –
Recent findings reveal that life can survive at temperatures as low as -17°C in the snow at the South Pole, challenging previous assumptions about the lower temperature limits for life This remarkable resilience against harsh conditions, including intense ultraviolet radiation, extreme cold, and prolonged darkness, raises significant questions about the potential for life beyond Earth within our Solar System.
N EW T YPES OF O CEAN B ACTERIA THAT C ONVERT L IGHT I NTO
A newly discovered strain of bacteria utilizes bacteriorhodopsin, a light-harnessing protein previously found only in fungi and archaea, allowing it to thrive in nutrient-scarce environments like the open ocean Researchers supported by the NSF isolated DNA from seawater samples, identifying a unique gene responsible for bacteriorhodopsin and employing a fluorometer to detect bacteria that could utilize dim infrared light from deep-sea vents Although no bacteria were found at the vents, numerous positive signals emerged from surface waters These bacteria harness light through various mechanisms, enabling them to survive in challenging conditions and provide essential nourishment for higher trophic levels in the ocean ecosystem.
Collaborators supported by the NSF have created an innovative biosensor utilizing fiber optic technology to monitor microbial community structures and activities in coastal and estuarine waters and sediments This advanced biosensor is designed for future multi-assay capabilities, promising to enhance our understanding of biogeochemical processes in these vital ecosystems.
"relationships between cell abundance, gene expression, and actual microbial metabolic activities under different environmental conditions"
NSF-supported research groups have delivered invaluable insights into vertebrate evolution, particularly the transition from invertebrates By comparing gene expression in model organisms like mice and zebrafish, these labs have enhanced our understanding of brain, spinal cord, and neural crest development Furthermore, they have developed innovative techniques to manipulate gene expression in non-model systems, laying the groundwork for significant future studies in the field.
New nanoporous templates have been developed using the self-assembling nanoscale structures of cylindrical phase diblock copolymers This innovative method allows for the deposition of functional materials into the nanopores, resulting in highly dense arrays of magnetic cobalt nanowires These advanced nanostructures hold the potential to significantly enhance storage capacities, offering up to 200 times the storage of current commercial magnetic hard disks.
Connections between discoveries and their use in service to society
External evaluators highlighted successful outcomes from NSF awards that effectively linked discoveries to societal applications These results were quickly accessible and utilized in various contexts, including education, policy development, and by federal agencies and the private sector.
Ongoing wildfire research supported by the NSF aims to enhance the understanding of fire behavior and dynamics through collaboration with organizations such as the U.S Forest Service, Monash University, and Australia's Country Fire Authority By integrating models, developing instruments, and conducting field experiments in regions like the Northwest Territories of Canada and the western United States, researchers are uncovering critical insights into how fires spread This knowledge equips fire captains to strategically position their teams and equipment, ultimately improving safety for firefighters and potentially saving lives during wildfire incidents.
Schedule of Program Evaluations (FY 99, 00, 01)
The table outlines the meeting schedule for Committees of Visitors (COVs) during FY 1999, FY 2000, and tentative plans for FY 2001 It is important to note that the provided schedule and program titles are confirmed for FY 1999 and FY 2000, while those for FY 2001 are provisional Programs initiated in FY 2000 may not have a scheduled COV within this three-year timeframe, and new programs starting in FY 2001 will not be listed Additionally, Committee of Visitor reports will be accessible to the public online starting December 2001, with expectations for future reports to be available by July of subsequent years.
Committee-of-Visitor Schedule for FY 1999-2000-2001
Division (Program) FY FY FY
Advanced Computational Infrastructure and Research
Partnerships for Advanced Computational Infrastructure (PACI)
Division (Program) FY FY FY
Advanced Networking Infrastructure and Research
Special Projects in Networking Research X
Advanced Distributed Resources for Experiments X Instrumentation Grants for Research X X
Presidential Awards for Excellence in Mathematics X
Course, Curriculum, and Laboratory Improvement X
Division (Program) FY FY FY
The Louis Stokes Alliances for Minority Participation X
Program for Gender Equity in SMET X
Programs for Persons with Disabilities X
Alliances for Grad Ed and the Professoriate (AGEP) X
Research to Aid the Disabled X
Dynamic System Modeling, Sensing, and Control X X
Solid Mechanics and Materials Engineering X X
Network for Earthquake Engineering Simulation X
COV Schedule for FY 1999-2000-2001 (cont.)
Division (Program) FY FY FY
Grant Opportunities for Academic Liason w/Industry X
Electronics, Photonics, and Device Technologies X
Control, Networks, and Computational Intelligence X
Industry/Univ Coop Res Centers X
Combined Research-Curriculum Development X Supplement Support for Women, Minorities and Phys Disabled X
Geo-wide Education & Human Resources X
Division (Program) FY FY FY
Education, Human Resources & Special Programs X
National Astronomy & Ionosphere Center (NAIC) X
National Optical Astronomy Observatories (NOAO) X
National Radio Astronomy Observatory (NRAO) X
Division (Program) FY FY FY
Chemistry Research Instrumentation & Facilities (CRIF) X Inorg, Bioirnorg, & Organometallic Chemistry X
Solid-state Chemistry and Polymers X
Materials Research Sci & Eng Centers X
Atomic, Molecular, Optical and Plasma Physics X
COV Schedule for FY 1999-2000-2001 (cont.)
Division (Program) FY FY FY
Antarctic Ocean and Climate Systems X
Arctic Research Support and Logistics X
Social, Behavioral, and Economic Research
Decision, Risk, and Mgt Sciences X
Society Dimensions of Engineering, Science, and Technology X
Table of Acronyms
ABET Accreditation Board for Engineering and Technology
ACM Association for Computing Machinery
ACSI American Customer Satisfaction Index
ARPANET Advanced Research Projects Agency Network
BART Bay Area Rapid Transit
BGE Baltimore Gas and Electric?
CARA Center for Astrophysics Research in Antarctica
CAREER Faculty Early Career Development Program
CETP Collaboratives for Excellence in Teacher Preparation
CISE Computer and Information Science and Engineering
CREST Centers of Research Excellence in Science and Technology
CS&E Computer Science and Engineering
CTS Chemical and Transport Systems
DARPA Defense Advanced Research Projects Agency
DMII Design, Manufacture, & Industrial Innovation
ECS Electrical and Communications Systems
EEC Engineering Education and Centers
EHR Education and Human Resources
EIA Experimental and Integrative Activities
EPSCoR Experimental Program to Stimulate Competitive Research
ESIE Elementary, Secondary and Informal Education
FBI Federal Bureau of Investigation
FCSM Federal Committee of Statistical Methodology
FFRDC Federally-Funded Research and Development Center
GIST Geoscience Interactive Simulations for Teaching
GPRA Government Performance and Results Act (of 1993)
HBCU Historically Black Colleges and Universities
HHS Health and Human Services
I/UCRC Industry University Cooperative Research Centers
IBN Integrative Biology and Neuroscience
IEEE Institute of Electrical and Electronics Engineers
IGERT Integrative Graduate Education and Research Training
IIS Information and Intelligent Systems
IRIS Information, Robotics, and Intelligence Systems
LEARN Laboratory Experience in Atmospheric Research
MCB Molecular and Cellular Bioscience
MIT Massachusetts Institute of Technology
MPS Mathematical and Physical Sciences
MRI Major Research Instrumentation Program
NAEP National Assessment of Educational Process
NAPA National Academy of Public Administration
NASA National Aeronautics and Space Administration
NCAR National Center for Atmospheric Research
NIST National Institute for Science and Technologies
NOAA National Oceanic and Atmospheric Administration
NSFNET National Science Foundation Network
OIA Office of Integrative Activities
OIG Office of the Inspector General
OLPA Office of Legislative and Public Affairs
OMB Office of Management and Budget
ONR Office of Naval Research
OPP Office of Polar Programs
PACI Partnerships for Advanced Computational Infrastructure
PFSMETE Postdoctoral Fellowships in Science, Mathematics, Engineering and Technology Education