Many funding agencies ask for specific plans for formal evaluation or assessment of proposed education programs, to determine whether the education interventions that are developed are actually serving the students or other intended audience. This kind of formal program evaluation is becoming the norm not only in STEM education proposals, but also in research proposals or cohort training proposals, such as NSF REUs or NRTs.

 

In years past, program evaluation plans that were considered “acceptable” by NSF and other agency peer reviewers often consisted of simple pre- and post-surveys for students. The landscape has changed – this is no longer sufficient!

 

 

What Does a Strong Evaluation Plan Look Like?

 

At its core, a good evaluation plan should involve both of the following:

 

  • Formative Evaluation – collecting feedback during the program so that improvements to the program can be implemented in real time
  • Summative Evaluation – measuring the level of success or proficiency attained by your audience after they’ve completed the program.

 

What Should I Evaluate?

 

A strong evaluation plan includes examples of the kinds of evaluation questions you’ll ask, as well as what you’re aiming to learn by asking those questions. You should be aiming to assess these kinds of outcomes for your audience:

 

  • Cognitive Outcomes – Knowledge of scientific content, acquisition of science practice skills/laboratory skills, research self-efficacy
  • Affective Outcomes – Increased enjoyment, interest, motivation to study the subject material
  • Behavioral Outcomes – Examples could include matriculation into a science career or enrolling in graduate school

 

 

Who Should Perform the Evaluation?

 

To bring legitimacy to your evaluation plan, it’s best to employ an external evaluator who has demonstrated professional experience in designing and conducting program evaluations. Utilizing an external evaluator also helps you avoid conflict of interest and ensure that your evaluation plan is unbiased, since the evaluator is an objective, neutral third party.

 

 

Where Can I Find an External Evaluator?

 

Right here in the College! The Texas Institute for Discovery Education in Science (TIDES) focuses on designing and implementing assessment tools for STEM education programs. TIDES researchers are experts in program assessment, they know the STEM education literature and they know what the current challenges are in STEM education. Contact TIDES to talk more about your program assessment needs.

 

 

What are Some Literature Sources that I Can Cite in My Proposal?

 

Here are some literature sources that can help you learn more about designing good evaluation plans. You might want to cite these in your proposal:

 

The User-Friendly Handbook for Program Evaluation (2010):

This book provides specifics about the “whys” and “hows” of program evaluation. The original version of this handbook (2002) was published by NSF; although not published by NSF, they refer to the 2010 UFHB in several of their program solicitations.

 

Measuring STEM Teaching Practices (2012):

Download it here.

An NSF/AAAS report

 

NSF’s Common Guidelines for Education Research and Development (2013):

http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf13126

 

Principal Investigator's Guide, Chapter 5: Planning for Success: Supporting the Development of an Evaluation Plan:

http://www.informalscience.org/evaluation/pi-guide/chapter-5

 

FAQs about the NSF Common Guidelines:

http://www.nsf.gov/pubs/2013/nsf13127/nsf13127.jsp