Introduction
The importance of being Earnest, stated Oscar Wilde in the title of his Trivial Comedy for Serious people (1), in which the protagonists maintain fictitious personæ in order to escape burdensome social obligations. The title of this play is basically based on a homophonous pun: “Ernest” is a masculine proper name, while “Earnest” means the virtue of steadfastness and seriousness, which translators know very well, because of the difficulty of translating in other languages.
Talking about culture production, we need to shift The importance of being Earnest in The importance of being evaluated. The longing of evaluation it is in fact, by now, intrinsic to any cultural project: sponsors, stakeholders, project managers are all aware of that; however, while the crucial relevance of evaluating is no more called into question, there are still many doubts concerning how an evaluation should be planned and fulfilled. The aim of this article is to give some simple, but practical and solid, guidelines to conceive a worthy analysis of cultural projects, of any kind.
The tasks of evaluation surveys are: to improve the practice of the projects and to raise standards, to show what happened as a result of the project and to find tangible proofs of success, to let participants feel the evaluation as something for their benefit and not just for funders and managers, last but not least, to prove and to stress that funding has been well used. Therefore evaluation has a bivalent role: inside the organization, it gives a feed-back concerning the activities, on the other side it is a way to legitimate the same activities, through tangible results, in the presence of the team managers. So within this framework, the presentation of those results, on a horizontal axis, gives to peers precious information for future projects, on a vertical axis it shows to managers how resources were used and if objectives were achieved and stresses the measures of success, while outwards it emphasizes to stakeholders the benefits and the impacts of the project and to sponsors and partners the effective and efficient use of resources, as well as the achievement of the objectives.
Guidelines to plan the evaluation
Before going on with the evaluation planning, it is important to have in mind that evaluating a cultural project is not at all like evaluating any other product or project, because it is not just a matter of demand and supply. Carrying out a cultural project means to deal with people and to deal not only with input and output, but also outcomes, intangible effects, and so on. In this framework an evaluation process should be planned in order to catch all this and to show it.
A very crucial point consists in collecting evidence: you need to collect evidence before, during and at the end of a project. Evidence collected before the project starts, shows the level of skills, knowledge and understanding, or personal and social development of people who will take part in the project. Without knowing this base line, it is difficult to show what has changed as a result of the project.
Here there are some phases to follow in order to catch all those evidences:
– Documenting the project
When planning, partners may also need to consider how the project is going to be documented. It means keeping a record of what happens during the project. It may consists of video, pictures, observation, writing, whatever necessary to document.
– Monitoring
Monitoring can describe routine collection of data, such as attendance figures, or checking of materials and equipment, to make sure they are of good quality. Some of the data you collect through monitoring may be used in evaluation. However, when you plan evaluation, you need to decide what evidence will be most useful, and whether or not additional or different data will be needed to find out if you have achieved the specific objectives of the project you are evaluating (2).
– Gathering qualitative and quantitative evidences
The evidence you collect is to help you judge if you have achieved the aims and objectives set during planning, and any unplanned outcomes, and is not just for keeping a record. It is likely that two kinds of evidence will be needed by partners to show what has been achieved, quantitative and qualitative evidence.
• Quantitative evidence produces data which enables you to measure numbers or percentages and statistics. It tends to deal with facts, such as the number of people taking part, or the cost per head of the project.
• Qualitative evidence shows people’s thoughts, opinions, ideas and feelings. Qualitative evidence may be more difficult to interpret, but is important for the evaluation of arts activities. Most usefully, qualitative information is collected from people with different points of views, who bring different perspectives. Qualitative evidence gives a sense of what really happened, allows judgement of the quality of the project, shows if and how people changed through the creative process and it is likely to reveal unexpected outcomes.
In a partnership project, partners should aim to collect a combination of quantitative and qualitative evidence. Sometimes funding can be dependent on quantitative evidence, such as certain levels of attendance or participation. Check always what is needed during planning. There are many ways of collecting evidence and it is important to choose those which could gather the evidence you need. Some of them are: comments’ boxes and books, displays of work or performance at the end of a project, drawings, charts and diagrams, ‘graffiti’ walls, interviews, observation, online websites, chat-rooms and email, participatory techniques, photography, questionnaires, small group discussions, tape and video recordings, written diaries, thumblr and blogs, etc…(3) A good evaluation should not be based just on one of those, but include many of them in order to be more rich and in-depth.
– Assembling, interpreting and reporting
You need to turn raw data into information, which can help partners to draw conclusions and take action if necessary. You also need to combine different kinds of evidence and, by doing so, it is important to find a proper way to emphasize results, i.e. using percentage, diagrams, quotation. Your evidence is likely to include many ‘quotable quotes’: used alone these may give a flavour of the project, and can often sum up unexpected outcomes, but they will be most convincing if you can provide some quantitative evidence and analysis to support them. Furthermore you should write a report to sum up the projects, the results and the final considerations. It may be addressed or may be used by the people who carried out the project, the team managers, the stakeholders, the sponsors and finally the community. For that reason it should be shaped according to the use. The report is crucial because it encompasses all the information concerning the projects: outputs, outcomes and measures of success included.
Four questions to plan your evaluation
It is by now evident the need of planning an evaluation system which can be useful to the comprehension of the entire project, to draw the qualitative level of the project, to stress strengthens, weaknesses and opportunities for future editions, participants feedbacks and the impact of the project.
Here there are some simple questions to keep in mind, in order to plan a solid and effective analysis:
– Which are the project tasks?
– To whom is the project addressed? How can we make the participants reactions and opinions visible?
– What is necessary to know about this project? What is crucial to value and measure?
– To whom is the final report addressed? Is it for an internal use or does it have external interlocutors? Which is the dossier objective?
Non conventional way of collecting and reporting data
As previously said, there are many ways to gather data and it is important to find the proper one to share the results with all the stakeholders.
In my experience at Fondazione Sandretto Re Rebaudengo (FSRR) (4) , a contemporary art museum in Turin, while evaluating art projects, I realized that questionnaires, focus groups, interviews and active observations were essential to evaluation, as they constituted the solid basis, however conceiving always new, innovative but still effective methods and ways to collect data and to show results, were able to make the difference. Let’s think about the potential of a long and detailed written report and compare it to the power of communication through new media and their effect. Of course, we can miss some details, but we can gain in attention and popularity. So it is important to keep on going with detailed evaluations, as it is crucial to find new strategies to collect what can still remain invisible or slightly perceivable.
Here there a couple of examples, introduced by a short description of each project. My intention is to find an effective and powerful way to collect results, but also to show them, not only to experts, like managers, partners and sponsors, but also to participants. It is thus important to note that they are not substituting any report, rather they can implement them and be more easily shared.
As mentioned before, the evaluation should be set up on data collected in different ways, to have many points of view and more details concerning the project itself and its impact.
My Modernikon: a project to make high school students closer to contemporary art
My Modernikon is a project carried out by the Educational Department of FSRR. It is a 8 meetings program for high school students (14 – 15 years old), at the end of which they are asked to introduce the show and the artworks exhibited there, with their own words, through their own experiences, to general public.
– Which are the project tasks?
The project’s aims are to enforce the relation between the teenagers and the contemporary art, trying to reach them through schools and to involve strongly motivated teachers in order to establish a stable and a fruitful collaboration with such institutes. Another task is to induce students to get in touch with contemporary art, by giving them the adequate tools to understand and to interpret artworks. Last but not least, the long-term objective to establish a stable connection with young audience, to increment their exposition to cultural products and to let them feel more comfortable, when adults, in getting in touch with contemporary art Institutions, in particular with FSRR.
– To whom is the project addressed? How can we make the participants reactions and opinions visible?
As said before, My Modernikon addresses to teenagers of Liceo Classico Cavour in Turin. To make their reactions and opinions evident, the Educational Department conceived a short video, visible at the following link on vimeo: http://vimeo.com/49009429, so basically accessible to anyone.
The video collects comments and opinions of students, teachers and Educational Department operators and gives a global view of the project. How much more powerful can be participants words rather than written transcriptions? This question moves the conceiving of his strategy.
– What is necessary to know about this project? What is crucial to value and measure?
It is of course important to find if the tasks are achieved, if teenagers feel comfortable at FSRR, if they would come back or would like to repeat such an experience, if they perceive or become aware of having the proper tools to talk to visitors about the show, about the artists and about contemporary art too. If they learn something or become more confident, furthermore if this project makes them interact with peers and class-mates.
– To whom is the final report addressed? Is it for an internal use or does it have external interlocutors? Which is the dossier objective?
The Educational Department intention is to create a product that could be sharable with all the stakeholders: partners, team managers, surrounding community, schools, students and their families, without forgetting to be detailed and able to give a complete overview of the project.
Workshops: a three days-fulltime workshop for students
The Educational Department of FSRR organizes three days full time workshop for groups of students coming from different schools and university, aged from 17 to 25 years old, based on the current exhibition of the moment. The idea is to let these groups live the museum space in a informal way, to induce them to get in touch with contemporary art and to go deep into contemporary creation. The program includes one day to visit the exhibition and to meet the curator, one day to work in small groups, focusing on a single artwork or a small group of artworks, trying to elaborate a thought, a project, an answer, a continuation…, the last day is to go on with the project and it ends with a final event when the public visits the show, as it is improved by the students contribution.
– Which are the project tasks?
The main aim of the project is to enforce a wider accessibility to art collection, by conceiving the exhibition spaces as place to dialogue and to think, as container but also as knowledge producer. The project aims to enforce the relation between teenagers and contemporary art and to favour dialogue and exchange. The second purpose is to induce participants to get in touch with contemporary art and to give them the adequate tools to understand contemporary artworks.
– To whom is the project addressed? How can we make the participants reactions and opinions visible?
As previously mentioned, this kind of workshop is addressed to students from different schools and classes, so the participants are not supposed to know each other, but at the end they are asked to work together. It is crucial to stress participants way of working together, their feeling, but also to show what they produced, especially because they are made for a one-night event, so any documentation and report is crucial to let people know what happened and how, either after the event.
– What is necessary to know about this project? What is crucial to value and measure?
Once again it is basically worthy to prove that aims are achieved and participants are supported in their creation, if they are induced to get in touch with the artworks, the art space and with other people. It is crucial to evaluate if the project improves their skills and supports creative thinking and reflection. The Educational Department carried out different tools, according to each specific workshop. One of them is this thumblr page: http://dipartimentoeducativo.tumblr.com.
It is conceived for a workshop made for the exhibition Press Play. Art and Media (5) and it is a webpage on which participants and Educational Department operators write, comment, explain, upload photos, video, recordings and interviews, in order to create a complete overview of the project and to give an idea of the continuation made by the students. The second example is again a video on vimeo website: http://vimeo.com/30912465 and it intends to show the development of the project until the final event and the relation with the public. It is referred to Glenn Brown solo show (6), a British painter born in 1966. Once again, how much more effective are images, photos, videos and people comments, rather than written reports? From this question arises the idea of creating a thumblr pages and the video.
– To whom is the final report addressed? Is it for an internal use or does it have external interlocutors? Which is the dossier objective?
The Educational Department intention is to create again a product to share with all the stakeholders. Detailed reports, data, charts, graphics and more detailed documents are produced for each of them, but still improved by those maybe simpler, but also powerful products.
Concluding remarks
Evaluation is a way to gather feedbacks, inducing also participants to analyse all the features of the project, in a way to support and implement the future activities. Also, evaluation stresses tangible proofs of success, quality and impacts on public in order to legitimate the roles of the operators in front of directors, managers and sponsors. Furthermore evaluation makes the decision-making process easier: in this framework it is important not to forget to show the results to all the stakeholders, shaping the data in an appropriate way, according to their focus.
To make an evaluation means to chart a course, to gather information, to stress results and impacts that can be hardly collected and showed elsewhere and that can be probably missed. It is important to let all the stakeholders get in touch with such results and to conceive an appropriate way to show them, but it is also crucial to gather data by using the proper tools. However, if it is stated that many Institutions and Organizations are well aware of the importance of “being evaluated”, it is true that just few of those activities are part of the Organization itself. Extraneous societies are often in charge to do that, or evaluation is sometimes supported and requested just for specific projects, without considering the whole activity.
The real challenge is to make evaluation part of daily activities, in order to benefit from that on a long term basis either.
Notes
(1) The Importance of being Earnest. A trivial Comedy for serious people is a paly of Oscar Wilde,first performed on 18 February 1895
(2) A guide to evaluating arts education projects, Art Council of England, 2004
(3) We will go into deep later on
(4) http://www.fsrr.org/?lang=en
(5) http://www.fsrr.org/mostre/press-play-larte-e-i-mezzi-dinformazione-2/?lang=en
(6) http://www.fsrr.org/mostre/glenn-brown/
Bibliography
Anderson D., Lucas K.B., and Ginns I.S. (2003), Theoretical perspectives on learning in an informal setting, in Journal of Research in Science teaching, 40(2)
Falk J. and Dierking L. (2000), Learning from museums: visitors experiences and the making of meaning. Walnut Creek, CA: AltaMira Press
Gadsby J. (2011), The Effect of Encouraging Emotional Value in Museum Experiences, Museological Review, University of Leicester, Issue 15
Griffin J. and Symington D. (1997), Moving from task-oriented to learning-oriented strategies on school excursion to museums, in Science education, 81(6)
Hein G. (1998), Learning in the museum, Routledge, London
Luckett H. (1982), Through children’s eyes: A fresh look at contemporary art. London: Arts Council of Great Britain
Piscitelli B., Everett M., Weier K. (2011), and QUT Museums collaborative, Enhancing young children’s museum experiences: a manual for museum staff
Soren B.J., Callen B., Chafe A., Grant L., Reitz T. (2004), Audience-Based Program Evaluation & Performance Measures, Museological Review, University of Leicester, Issue 11
Stevens D. (1985), “The social efficiency of Arts addiction” in Waits, C.R., W.S. Hendon and H. Horowitz (eds.), Government and Culture, Akron, Ohio, Association for Cultural Economics
Trimarchi M. (2002), Nomina sunt consequentia rerum: economic definition of culture, University of Catanzaro “Magna Grecia”, 2002
