New Guidelines for NSF Grant Submissions

On January 14th, 2013, new guidelines went into effect for NSF (National Science Foundation) grants submissions. The new Proposal and Award Policies and Procedures Guide contains several changes. For the full guidelines follow this link: Guide.  Educators and non-profits should review these changes carefully before submitting their NSF proposals.

One of the notable changes is in the merit review principles. The revised guide now includes three principles (previously two). The first principle addresses the NSF’s emphasis on high quality projects that have the ability to advance knowledge; the second principle is an emphasis on projects contributing to achieving societal goals. The third principle includes the following language: “Meaningful assessment and evaluation of NSF funded projects should be based on appropriate metrics…” (Chapter III, pg 1).

This is an important addition to the merit review principles, underscoring what we as evaluators focus on, namely that evaluation constitutes an essential component of research. This additional principle clearly demonstrates NSF’s commitment to research and proposals that address the potential outcomes and successes of the research.

For more information see the NSF published Dear Colleague Letter (October, 2012) that outlines all of the significant changes. Also, RIT Researcher News has published a summary that is another great resource for understanding all of the changes.

What’s Hot – Changes in Classroom Technology and its Assessment and Evaluation

With all the talk about how to increase learning in the classroom these days, it is easy to forget that any new approaches also need to be assessed for their effectiveness. For example, in both K12 and higher education, there are many discussions about new ways to impart information to students and therefore to increase learning. Let’s take the example of tablets. They are a hot item right now, both among consumers and education professionals. Not only are tablets seen as fun to use but they are currently outselling other computers. In some cases, tablets are becoming people’s primary computers. It is also very clear that young people can quickly be engaged with tablets, and they can be a very useful learning tool in the classroom.

However, it is important to consider how to evaluate whether or not a new technology is actually increasing learning outcomes. If educators are interested in incorporating these new technologies into the classroom, then it is equally important to start thinking about the evaluation process. For example, educators should carefully plan how they will implement the new technology, and plan on assessing learning before and after the introduction of the new technology. Some thought should also be given to how to measure an increase in learning. What is expected as the outcome of the implementation of new technology? What will be measured to actually know that students have increased learning?

We have seen quite a few grant proposals lately that deal with such new technology, but have found them to be lacking in evaluation. Perhaps the evaluation process is still unfamiliar to many. We as researchers and evaluators need to spend more time discussing the importance of evaluation in grant proposals, as well as the implementation of evaluations. We need to emphasize the importance of evaluating educational technology so we have the evidence to show what works and what does not.


Focus Groups: Part II

Comparing Types of Focus Groups

There are three types of focus groups in common use: face-to-face, telephone, and on-line.  All three types are used by researchers to get in-depth information about a program, product, or service.  The main difference among these methodologies is the location of the participants.

Unlike face-to-face focus groups, telephone and on-line focus groups operate with individuals who may not be in the same physical location as one another.  Telephone focus groups are usually conducted by having members call into a conference call line in which they can carry out a discussion with others in the group.  A typical on-line focus group is conducted with software that allows the group members to chat on-line with other members in real-time.  However, online groups have also evolved to allow participants to join in when convenient for them, thus not all participants are simultaneously present in the focus group.

Advantages and Disadvantages

Each type of focus group has its own advantages and disadvantages.  For example, a face-to-face focus group allows members of the group to not only hear other group members’ responses but also to see non-verbal cues and other visual cues that are not available on the telephone or in on-line focus groups.  However, face-to-face focus groups are restricted to groups of people that are in the same physical location.

Telephone and on-line focus groups do not have such restrictions.  If you wish to have a focus group made up of individuals from across a broad geographic area, telephone or on-line methods would be the appropriate choice. On-line focus groups may be a particularly good choice when asking sensitive questions because participants are able to remain fully anonymous.

One drawback to on-line focus groups is that group members must have internet access. While most Americans have internet access at home, some may have slower connections or no home internet access. In these cases, face-to-face focus groups may be more appropriate to gain responses from a broader sample.  Further, some researchers who use telephone focus groups claim that the ability to hear the tone of voice of other group members is a benefit that is not available to on-line focus groups.

Another consideration is cost. In some cases conducting on-line or telephone focus groups are lets costly than face-to-face focus groups because there are no travel or facility costs. However, the costs involved in focus groups are dependent on many factors. For a more detailed analysis of cost comparisons see the following article that compares on-line and face-to-face focus groups.

Choosing a focus group methodology will depend on many factors such as the types demographics of participants that you are seeking, where they are located, and their technology access and skill level. CMS Evaluation can help you decide on the appropriate method for your particular evaluation project.

For a more in-depth comparison of the three main types of focus groups, see the following article:

Focus Groups 101

37056_people_at_work_5Focus groups are a useful method for understanding the impact of a program on the people it serves.  Market researchers have been using focus groups for many years to test products, determine customer attitudes, likes or dislikes, and to gain a richer, more in-depth understanding of the topic at hand.

It seems that focus groups are also being used more frequently by social scientists as there is an increasing interest in hybrid methodologies, using a mix of both quantitative and qualitative methods to get more in-depth information from research participants on their attitudes and behaviors.


Focus groups are usually small groups of people ranging from about 5 to about 15 people.  The purpose of most focus groups is to facilitate a discussion among participants about a specific set of questions or ideas.  Focus groups are usually conducted by one or more facilitators who pose questions, probe for further responses, and moderate the discussion.  Focus group discussions are usually recorded by an audio and/or video recording device.  Because focus groups are dynamic discussions, participants have the opportunity to clarify and refine their expression of their own thoughts as they explain them to others in the group.


If you compare focus groups to other forms of research such as a quantitative survey, important advantages of focus groups become clear:

  • In-depth Information:  In a focus group, you can get in-depth information from each respondent.
  • Opportunity to Ask Follow-up Questions:  Focus group moderators are able to ask follow-up questions on the spot, often leading to valuable information and perspectives not anticipated in a closed-ended survey.
  • Opportunity for Respondents to Explain their Responses:  Unlike a paper and pencil survey that allows the respondent to only check agree or disagree on a form, focus group participants are given the opportunity to explain under what circumstances they agree or disagree with the idea being discussed. This also provides for richer data.
  • While surveys do allow additional information to be captured via the inclusion of open-ended questions, the discussion component of focus groups allows for a more dynamic development of ideas among the participants.


While focus groups are a rich source of data, there are disadvantages:

  • Time Intensive:  Focus groups are more time intensive than surveys and other research tools.  Because the data are primarily qualitative, time is needed to transcribe and code (make sense of) findings.
  • Cost:  Focus groups are more costly than other forms of research.  For example, focus group facilitators should go through significant training to prepare to moderate focus groups.
  • Small number of respondents can result in more biased sample: Focus groups are small groups that allow for a more manageable discussion among the group members.  Thus, the number of participants in focus group studies is smaller in comparison to other research methods.  As a result, there is a greater chance of obtaining a biased sample or a sample that is not representative of your broader client base.  It is extremely important to screen potential focus group participants on specific criteria to ensure participants are a good representation of the population under study.


For a more detailed summary of focus groups and the advantages and disadvantages, see this article on How Focus Groups Work.  Also, for a basic article on using focus groups to evaluate programs see the Instructional Assessment Resources (IAR) website.

In Part II of this post we will explore the different kinds of focus groups (Face-to-face, Telephone, On-line).

“Evidence-based”… What does it mean?

If you are an organization that is looking for funding sources, you may be aware that many federal funding organizations (e.g. Department of Health and Human Services) are increasingly interested in funding projects and programs that are evidence-based. Recently this term is being applied across many areas such as health, mental health, education, and the judicial system.


What does the term evidenced-based mean? While the definitions do vary somewhat depending on the area of application, all definitions refer to the use of scientific research as a way to inform service delivery, or programming or policy. The Substance Abuse and Mental Health Services Administration (SAMHSA), a division of the US Department of Health and Human Services, answers the question: What is evidence-based? as follows:

“In the health care field, evidence-based practice (or practices), also called EBP or EBPs, generally refers to approaches to prevention or treatment that are validated by some form of documented scientific evidence.” ….. “Evidence-based practice stands in contrast to approaches that are based on tradition, convention, belief, or anecdotal evidence.” (SAMHSA, What is Evidence Based?)

Pitfalls of Anecdotal Evidence

The SAMHSA definition points out a common problem; some organizations may continue to offer programs even though they have only anecdotal evidence that the program is accomplishing its goals. For example, anecdotal evidence could lead to the following hypothetical statement:

Demand for food bank baskets are a lot higher than one year ago; therefore more people must have fallen below the poverty level. In contrast, evidence-based research might result in the following statement: Our study shows that 25% of needy families in our community have stopped applying for food stamps and are instead turning to food banks for food, resulting in increased demand.

Organizations that conduct careful program evaluations will gain useful data that illuminate the reasons behind changes in client usage, or reasons why some programs are effective and others are not. This evidence will allow organizations to sustain and grow effective programs and services.

Evidence-Based Practice

This definition from The National Association of Social Workers (NASW) defines a broader process. They define evidence-based practice as:

“a process involving creating an answerable question based on a client or organizational need, locating the best available evidence to answer the question, evaluating the quality of the evidence as well as its applicability, applying the evidence, and evaluating the effectiveness and efficiency of the solution.” (NASW, Evidence Based Practice)

Finally, a succinct illustration of what the term evidence-based signifies comes from Jeffrey Pfeffer and Robert Sutton at Stanford University. One of their principles of evidence-based management is: “Be committed to “fact based” decision making — which means being committed to getting the best evidence and using it to guide actions.” (Pfeffer & Sutton, Evidence Based Management)

Evidence-Based Practice and Program Evaluation

The widespread emphasis on evidence-based research, practice, and policy is encouraging because it is this that we, as program evaluators, are consistently working toward: applying scientific methodology to assess the effectiveness of programs. In other words, we do the research that tells you what is working, how well it is working, and what is not working. A program evaluation consultant can help organizations look beyond their beliefs and anecdotal evidence and gain a broader understanding of the value of their programs and services.