Recently BusinessWeek magazine featured an article titled "The New Face of Philanthropy" on its cover, with the sub-headline, "today's donors are more ambitious, get more involved, and demand results." The newest class of funders wants nonprofit organizations to "produce measurable results—or risk losing their funding." As a relatively new and growing field, Youth Media has come under increasing pressure to both achieve and demonstrate appreciable progress. Indeed, the growing impetus for nonprofit groups to validate their effectiveness through independent evaluation is central to the long-term development of the Youth Media field itself.

Although it has long been a common buzzword in the nonprofit community, the term "evaluation" elicits a fair share of trepidation, misconception and dread. Though few people would disagree that examining the goals and effectiveness of nonprofit programs is necessary, the use of formal program evaluations in the Youth Media field are intermittent, at best.

Evaluation 101

Next month, Listen Up! will devote much of its annual workshop for youth video-makers to the subject of evaluation. Kim Sabo, a professional evaluation consultant who works with many non-profits, will be leading the workshops. "Even the word 'evaluation' is so confusing," she says. "People call what they are doing evaluation a lot of times and it's not. They do a survey or assessment, a one-strategy kind of thing. Or some groups say they don't do evaluation but you go into their program and they have 400 pounds of paper! People consistently collect an enormous amount of data about the program but they never aggregate it and put it into one form."

The Innovation Network, which supports organizations in conducting evaluations, defines program evaluation as "the systematic collection of information to answer important questions about activities, characteristics, and outcomes of a program. Evaluation stages include design, data collection, data analysis and interpretation, and reporting." Sabo adds, "the word systematic is extremely important because lots of people do collect data but not systematically." Carter McNamara, a professional evaluator for 20 years and co-founder of Authenticity Consulting, advises that "the most important thing when doing evaluation is to first answer the questions, 'What do I really want to be able to do as a result of this evaluation? What questions do I want to answer?' It's better to have a targeted goal for your evaluation activity." Most experts also stress that evaluation is an ongoing, cyclical activity that should be naturally incorporated into any program design.

Often, there's resistance among nonprofits to complete evaluations, which are at times treated as an arduous chore. One Program Director offered the common view that evaluations are "something to add-on at the end of a proposal. Just more work to say you're doing work." Sabo agrees that part of the problem is that the impetus is often external: "That's sometimes a problem in getting buy-in from the program staff. They think that it's just something they have to do for a funder." McNamara adds: "Evaluation is the same as learning—learning about what you're already doing and trying to improve it. It really helps to look at evaluation as a means to answer important questions in a reliable way and to help you make important decisions."

He emphasizes that "groups should always conduct evaluation; the question of how formal or informal it should be needs to be taken up on a case-by-case basis. Knowing what you're aiming for and being able to measure improvement or lack of improvement is really important in all development processes, youth development and skill development and organizational development. How will you know where you're getting if you don't look at it in a structured way?"

When looking at an evaluation of a Youth Media program, Robert Sherman, Program Director of Effective Citizenry at the Surdna Foundation stresses the importance of the two "core dimensions" of a program: the media product and the youth development involved. "That is something that I look for in a discussion with a prospective grantee. Looking to see that they are sensitive to both of those areas and thinking about them with some degree of clarity. But I would not want to put out a set of boxes to check off. A very mechanistic approach is not appropriate for the Youth Media field."

Who's the Evaluator?

After a program has decided that it wants to undertake an evaluation, is it always necessary to hire an outside consultant? Sherman advises, "You have to be really clear-eyed about your own capacity to evaluate and see what measurements are important to you. Then see whether you have internally the capacity to do it yourself or if you need to go outside. I would not advise a small startup group to use an outside evaluator. But I would advise a small startup group to be in touch with a more established group about the kinds of evaluations that they have undertaken." Sabo thinks that, since evaluation is often misunderstood as an onerous task, "You need someone to hold your feet to the fire, otherwise it always gets shoved to the back."

Whether a program decides to hire a consultant or go it alone, the most important thing, according to McNamara, is not to fear evaluation: "there's 20 percent of effort that generates a quick 80 percent in results—a little bit of effort often makes a lot of headway. If you have the choice of not doing an evaluation because you don't have an expert, or doing the evaluation yourself, I advise that you try it yourself. There's no such thing as a perfect evaluation, so keep it practical, focus on relevance, focus on utility."

Costs of Evaluation

Certainly, one of the questions to consider when embarking upon an evaluation is determining the real costs to the organization, in terms of people, money, and time. A common concern for groups is the lack of staff time. "It is time-consuming," Sabo says. "In youth programs it's even harder because young people come in after school hours so you only have them for a couple of hours a few times a week." Most evaluators agree that young people in a program should be a central part of any evaluation.

A related issue, of course, is allocation of funds to conduct an evaluation. Whether an evaluation is conducted internally or with the help of an outside evaluator, there is always a price. While many funders are increasingly asking for evaluations, not all readily fund the process. Therefore, in order to learn about a particular program's effectiveness, resources from that program must be diverted away for the evaluation. Balancing program goals, resources, and evaluation needs is imperative.

Sabo thinks part of the evaluation education process must include funders too. She says, "funders are also behind the times about evaluation. They are asking for evaluations so much of the time and not funding it. Funders want the accountability but they also don't know what they're asking of the program. Or they want particular outcomes without thinking about how they are connected to services. Getting everybody at the table to agree on the outcomes that we want to look at is incredibly important."

For his part, Sherman readily acknowledges the need for funding evaluations and feels that "funders who have experience with Youth Media and see the necessity of evaluation will be responsive to including, along with programmatic support, support for evaluations. That's a challenge for those of us working in this zone to try and build more and more philanthropic interest; the money for evaluation, hopefully, will follow." McNamara sees some progress as well: "Some funders are starting to realize that it's not enough just to confront a grantee with the question of 'How are you going to prove that you're reaching your outcomes?' They are partnering with the client organizations, to figure out together how the money is best spent. That's a wonderful breakthrough and I hope more organizations that give money can stop the confrontations around making organizations prove that they're achieving outcomes."

Evaluating Youth Media

As a foundation Program Director , Robert Sherman has seen many Youth Media evaluations that miss the mark. "A lot of evaluations start with a bean-counting approach," he says. "X number of young people came to x number of after school sessions and participated in y trainings. That's good data to have but it's not evaluation. It doesn't say whether the goal of the media program was reached or the goals for an individual Youth Media maker were reached. It just said that the program you had outlined in fact happened, or didn't happen. Often, that's as far as some groups get and it's just not far enough."

How does Youth Media rate on the evaluation scorecard? (See this month's YMR Spotlight for a profile of Youth Radio, which has done a lot of work on evaluation). Overall, "It's still spotty in the Youth Media field," according to Sherman. "Some of the larger and more established groups have been thinking about benchmarking their work while more startup efforts are not as sophisticated." Part of the problem is the inherent complexity of goals in a Youth Media program, which emphasize both youth development as well as the creation of a quality media product. "Different programs are at different points on the continuum of sophistication in doing that evaluation," he says. "The better groups, in fact, are looking at both of these dimensions and more startup efforts tend to focus on one or the other more."

How should the media product be included in the evaluation, and what exactly is being evaluated? Is it enough that young people have developed skills from creating a video or radio piece or should the impact of that piece be a part of the evaluation equation? Sabo knows this is a big issue for Youth Media groups. "One of the problems is trying to look at the impact of the media itself," she said. "We are generally pretty good at being able to talk about how the young people are impacted in the program, about the quality of the service delivery within the program. The harder thing that we've been struggling with is the impact of the media itself."

Sabo has had some success, however. "Some of the ways we've dealt with it is to talk about the distribution, the audiences that it has reached," she claims, with the caveat that this works best in a smaller distribution environment. "Programs that do community screenings or in a school can survey their audience," she offers. "Some groups use media as a particular point of advocacy, so this can be explored, especially if it's a local piece on advocacy because you have access to those people you're trying to communicate with and can survey or poll them. It's harder with national campaigns. You can know that the audience heard the message and thought it was a good message but not that it had any impact on their behavior."

For Surdna's Sherman, the distribution of a quality Youth Media product is "an important benchmark. To produce media that is not seen, read, heard or experienced doesn't get at the great benefit of what Youth Media has to offer -- which is young people communicating about issues that are important in their lives. The goal of making media and finding voice is to be heard."

However, distribution is not the only criterion with which Youth Media is being judged. Funders also want to see that the product has a measurable impact. "We don't have good metrics for the effectiveness of media, whether it be Youth Media or adult media," Sherman admits. "The idea of judging Youth Media by audience size, which is the conventional media evaluation tool, that's not really appropriate as the only evaluation focus. What is the impact on the listener or reader? Those ideas are very, very hard to measure." This obstacle is affecting the growth of the Youth Media field as a whole. "Funders are rightly cautious with their funds since we don't have good evaluation strategies for the impact on an audience. We have no real ways to capture—in a concrete way—what the results are of putting Youth Media out over the wires or over the airwaves. We're going to have to build a critical mass of Youth Media work that really is heard in the mainstream so that we can start to make some progress on that question. And we're going to have to figure out our own evaluation strategy for knowing when we've gotten there."

Presenting Findings

Though the process of evaluation itself can be invaluable, a final report or product is an important, tangible result that can be widely shared. Though Sabo issues updates on findings throughout the evaluation, she thinks it's important to find creative ways to present findings, so that they are usable and engaging: "Normally we issue a report at the end of the process, though I also have worked with groups to create a video as the final product. Or maybe a PowerPoint presentation, because people don't want to just put it on the shelf. It's better to show as a presentation to their board, the community, the program at-large or a funder. Videos are great for a Youth Media program because it's what they do a lot of times."

Growing Pains

Despite such uncertainties, the critical need for evaluation documentation in the Youth Media field is a sign of its growth. The days of a program relying on gut instinct, warm anecdotes, and compelling case studies alone is undoubtedly over. For the field to take its next steps, a significant effort must be made by youth media groups, funders, and the evaluation community to systematically document its successes, learn from its failures, and devise new strategies for analyzing its product and impact.

Related Links and Resources:

Innovation Network Evaluation Resources:
http://www.innonet.org/resources/eval_resources.cfm

Carter McNamara, Authenticity Consulting:
http://www.authenticityconsulting.com

Kim Sabo, Sabo Consulting:
http://web.gc.cuny.edu/che/kim.htm

Management Assistance Program Library on Evaluation:
http://www.managementhelp.org/evaluatn/evaluatn.htm

Surdna Foundation:
http://www.surdna.org