1. Introduction
On the one hand, social network analysis is a technique increasingly applied to different research fields and, in particular, to social sciences. This methodology studies the society as a network of relationships between actors, where individuals and institutions realize themselves in relation to the others. On the other hand, culture is more and more the bulk of projects promoted by international organizations as a strategy of local development and poverty reduction since the 90’s (Pagiola, 1996; Cernea, 2001; Santagata and Bertacchini, 2011; OECD, 2005). The development of culture-based economic activities is indeed seen as an important pro-poor instrument. The paper aims to show how and why network analysis turns to be a precious and complementary tool to evaluate cultural projects targeting local development.
Cultural economics is indeed looking for new methods of investigation, beyond the traditional techniques such as financial and econometric analysis, enabling the mapping and measurement of cultural networks. In fact, the existing evaluation methods assess the impact of a cultural project promoting local development in a given period of time, usually coinciding with the duration of the project, but are not able to quantify the sustainability of cultural projects over time. In other words, they do not quantitatively estimate if the economic and social benefits created by a project can survive after its end.
Actually, the sustainability of a cultural project is closely related to the creation of new networks and social dynamics. Cultural networks provide the substrate in which ideas are conceived, implemented and sustained over time. Cultural sectors can be viewed as a network of actors who support the creative atmosphere and whose actions are closely interrelated and interdependent (Bertacchini and Santagata, 2012). For this reason, without the design, durability and dissemination of cultural networks, the sustainability of cultural projects could be hindered. At the same time, network analysis is arguably better suited to capture the complexity of local development. Economic variables describing local development, such as increased employment or entrepreneurial initiatives, may be easily incorporated as variables of the network nodes. Analogously, the creation of new companies and spin-offs may be captured through an expansion of the network. Crucially, identifying new connections developed between nodes or evaluating their increased strength may capture other more subtle and intangible dimensions of local development.
2. Overview of traditional project evaluation approaches
Traditional analysis put forward in the literature concerning monitoring and evaluation of project effectiveness generally distinguishes between evaluation of outcomes and evaluation of outputs. The former is defined in terms of the impact on the economy and society. The latter simply checks the correspondence between what has been planned and forecasted ex-ante and the effective achievement of the proposed objective.
A second distinction is methodological and refers to different approaches and tools developed to monitor and evaluate projects. Among the several approaches developed for project evaluation and monitoring, the most relevant and at stake for our study are: i) Performance indicators, ii) Cost-Benefit and Cost-Effectiveness Analysis, iii) Impact Evaluation.
These tools should not be seen as merely substitute, but more generally as complementary methodologies to evaluate outcomes and outputs, depending on many characteristics of the project.
Performance indicators are measures of inputs, processes, outputs, outcomes, and impacts. When supported with sound data collection and analysis (i.e. involving formal surveys), indicators enable analysts to track progress, demonstrate results, and take corrective action to improve the projects outputs and outcomes (World Bank, 2000). The main advantage of such an approach is to provide comparable and quantitative measures of the project relevant variables, so to facilitate benchmarking comparisons between different organizational units, districts, and over time. However, in many contexts, and especially with regard to projects in culture and development, performance indicators risk to be poorly defined or ill-suited because of the lack of data availability. For this reason, participation of key stakeholders in defining indicators is important because they are then more likely to understand and use indicators for management decision-making.
Cost-benefit and cost-effectiveness analysis are tools for assessing whether or not the costs of an activity can be justified by its outcomes and impacts. Cost-benefit analysis measures both inputs and outputs in monetary terms. Cost-effectiveness analysis estimates inputs in monetary terms and outcomes in non-monetary quantitative terms. These methodologies are particularly suited for taking decisions about the most efficient allocation of resources or for identifying projects that offer the highest rate of return on investment. At the same time, it is also recognized that requisite data for cost-benefit calculations may often not be available, and projected results may be highly dependent on the assumptions made. This is particularly evident in the context of cultural projects, where the results are often made up by intangible and less material outcomes and benefits to local communities (Throsby, 2010). In these cases, cost-effectiveness analysis can be a second-best option.
Impact evaluation is the systematic identification of the effects – positive or negative, intended or not – on individual households, institutions, and the environment caused by a given development activity such as a programme or a project (see Grun, 2006). Impact evaluation helps to better understand the extent to which activities reach a community and the magnitude of their effects on people’s welfare. Impact evaluation can range from large scale sample surveys in which project populations and control groups are compared before and after, and possibly at several points during the programme intervention; to small-scale rapid assessment and participatory appraisals where estimates of impact are obtained from combining group interviews, key informants, case studies and available secondary data. One of the main advantages of impact evaluation is that such an approach is able to provide estimates of the magnitude of outcomes and impacts for different demographic groups and regions over time.
However, impact evaluation may be less effective if difficulties emerge in identifying appropriate counter-factual evidence. For instance, evaluation design involving the collection of information on project and control groups at two or more points in time, provides the most rigorous statistical analysis of project impacts and the contribution of other factors. But in practice it is rarely possible to use these designs for reasons of cost, time, methodological or ethical constraints. Thus most impact evaluations use less expensive and less rigorous evaluation designs, which can be ultimately a problem.
All the presented approaches, although well refined, present some shortcomings in properly evaluating the effects of culture and development projects. First, they tend to rely on the quantitative dimension of the project outcomes, so neglecting more complex social and intangible dimensions of cultural enhancement and local development, such as the level of creative and cultural atmosphere developed by actors’ interaction and cooperation. Second, the use of economic multipliers takes into account the final effects but does not provide a sound methodology to capture the dynamics which lead to these effects, such as how the spillovers spread through the links between actors of the local system or how new businesses or projects are created as spin-off. Finally, and most importantly, these approaches only partly address with a proper methodology the sustainability of the project’ outcomes, which in many cases comes from the stability of stakeholder relations and interactions in the long term.
3. How and why network analysis can help
A network is defined as set of nodes, taken along with the relationships between them (see Freeman, 2006). In a cultural project, nodes are represented by all the actors involved in the project, i.e. the stakeholders. A node can be a distinct person, but can also represent firms, groups, institutions and organizations. Every time two actors interact, a direct link between them is created. Indirect links are created when two actors interact with a common third actor but not between them. The intensity of the relationships counts the number of times an actor interacts with another actor. We assume that the higher is the intensity of the relationship, the higher is the possibility that the actors will interact again in the future. Measuring the number of total relationships an actor has and their intensity tells us which actors have been at the center of the network, technically called hubs.
We can see a cultural project as a network in which stakeholders are the nodes. In this regard, network analysis can typify the structural and relational dimension of a project. Indeed, there are at least four sets of questions to which network analysis is able to answer:
1) Which have been the key-actors and the key-beneficiaries of the project? Are the key-beneficiaries coherent to the original aim of the project?
First, by studying how and how much the stakeholders have interacted with the project, we can individuate which have been the key-actors involved in the project. In particular, during the design, inception and implementation of a project, we can understand if the project has involved all the relevant local stakeholders or if, on the contrary, some important stakeholders have been marginalized. Moreover, by individuating the hubs, we can identify the actors who should be incentivized to guarantee the project sustainability for the future. If they will be properly incentivized to maintain their activities, the future existence of the network will be protected. Finally, for the mid-term and final evaluation of the project, by comparing the original aims and targets with the real beneficiaries of the project, we can evaluate the project effectiveness as well as its coherence.
2) Which network potentiality once the project will be over?
Once two actors have interacted, there is the potentiality that they will do the same in the future. For this reason, it is important not only to know whether two actors have interacted, but also to measure the number of times they did it (intensity of the relationship). As noticed before, the higher is the intensity of the relationship, the higher is the potentiality that the actors will interact again in the future. A desired outcome of a project and, especially, of a cultural project should be to improve local relationships and create the ground for future collaborations, sustaining the creative atmosphere. To this purpose, it is important that the stakeholders do not dialogue just with the project management, but also among them. By removing temporary actors from the network, we can understand which local network potentiality would remain once the project will be concluded.
3) Has the project created permanent networks?
The creation of permanent networks – such as associations, institutions, cooperatives etc. – is fundamental to further guarantee that the benefits created by the project will continue when the project will drop out. To verify the effective sustainability of a project, it is precious to have the possibility of observing and measuring the network some time after the end of the project, in order to understand what remains once the project is over, as well as if it has been able to create new contacts and new actors. In this regard, the greatest difficulty is the lack of data, so that a project should include since its beginning a programme of data collection to follow-up the network one or two years after the project dropped out.
4) Which actions and policies should be taken to improve the sustainability?
Network analysis can be useful for both mid-term and final evaluation. In the mid-term, we can verify if the project is creating networks and if they are potentially sustainable. By periodically monitoring the network creation and its future potentiality during the implementation phase, we can understand how to correct and improve them, as well as if targeted-groups are being involved in a network that is likely to survive in the future. In the light of this, the project management should be able to design intervention guidelines correcting the drawbacks. In the final evaluation, moreover, the network analysis could be a precious and complementary tool to individuate best practices and lessons learnt in terms of sustainability.
4. A simple example of empirical application
The proposed methodology is here applied to one of the 18 United Nations Joint Programmes (JP) financed by the MDG Achievement Fund in the area “Culture and Development” and aiming to contribute to the achievement of the Millennium Development Goals (UNESCO, 2012). The analyzed JP was implemented in Bosnia and Herzegovina, starting on May 2008 and ending on May 2011. The title of the programme was “Improving Cultural Understanding in Bosnia and Herzegovina”. The programme covered the cultural sectors of performance and celebration, arts and crafts, books and press, media, design, and tourism, and presented a large amount of objectives.
A programme is defined as a sum of projects aiming to the realization of common goals and it is composed by a series of actors. First of all, there is the JP management, that selects a number of partners usually represented by local NGOs, civil associations, and national and local authorities. In turn, the JP partners select their own sub-partners and, with them, design and implement the single projects composing the programme. Finally, each project creates a series of outputs benefitting the JP recipients. This structure is illustrated in figure 1.
Figure 1: The usual structure of a programme
Our purpose it to show through this example what network analysis can reveal about the programme sustainability over time. On the whole, the JP in Bosnia and Herzegovina presented the following structure:
– 19 JP partners (19 nodes)
– 53 sub-partners (53 nodes)
– 19 projects (19 nodes)
– around 40000 beneficiaries, for the sake of simplicity grouped by projects (19 nodes)
The representation of the JP network is reported in figure 2, where the size of each node is proportional to its centrality degree.
Figure 2: The network of the JP in Bosnia and Herzegovina
From figure 2, we can individuate which have been the main JP partners and their sub-partners, as well as the projects arisen from intensive collaborations. Here, we do not pay attention in listing and describing each node and its relevance since it falls outside our purpose, that is to provide a simple example of application rather than to fully evaluate the Joint Programme. As we can see, the JP has been able to create a dense network of relationships. If we imagine to drop out the JP and its own relationships from the Figure 2, we can observe that what remains is a large network composed by local stakeholders. We can say, then, that the JP has enable local actors to interact among them and has created a fecund field from which future partnerships may sprout and guarantee the sustainability of results over time.
Of course, what we state above is based on the expectations about future local partnerships rather than on concrete observations of the network some time after the end of the Joint Programme. Since follow-up data were not available, in order to show the potentiality deriving from applying network analysis we simulate two different scenarios: one positive and virtuous scenario, and one negative scenario.
Negative scenario
For this scenario, we create a situation in which the most part of the network has not survived one year after the end of the Joint Programme, revealing a low sustainability over time. In particular, we hypothesize that:
1) only the collaborations and the projects between two JP partners and their sub-partners have continued(1).
2) since we know from the original data that only 7 out of 19 projects have created permanent outputs (like associations, schools, touristic routes etc.), we assume that these projects have autonomously survived, but that all the related collaborations have been cancelled.
Positive scenario
For this scenario, we hypothesize a virtuous and sustainable situation where:
1) all the projects have been refinanced by national or local institutions, finally aware of the importance of culture for development thanks to the JP intervention
2) 5 JP partners(2) have enriched their collaborations by joining to new additional partners.
3) new collaborations have arisen among partners and across projects, so that 4 new projects have been created and new beneficiaries have taken advantage from their activities.
The resulting networks are represented in figure 3 and 4 respectively, where the size of each node is proportional to its centrality degree.
Figure 3: The network under the negative scenario
Figure 4: The network under the positive scenario
Now we have the possibility to compare both the scenarios with the original network created by the Joint Programme one year before. A graphical comparison is illustrated in Figure 5. Under the negative scenario, it is clear that only a little part of what the JP built has survived over time. This reveals a low sustainability. On the contrary, under the positive scenario, the network has continued to grow and enlarge, creating new collaborations, projects and beneficiaries, and reaching remarkable results in terms of sustainability.
What we showed was just a simulation of two possible and extreme scenarios to show how the methodology can be applied in the long term. When real follow-up data are available, in fact, network analysis is a powerful tool to track the network evolution and to quantitatively verify the sustainability of a project. Together with the network quantitative measures, a simple graphical comparison is able to tell much about it.
Figure 5: A comparison between the network during the JP and the network one year later (negative scenario on the left, positive scenario on the right)
5. Conclusions
Applying network analysis to cultural projects aiming to promote local development is an innovative approach to map and check their potential or real sustainability in the creation of networks as a fecund field on which building and sustaining a creative atmosphere.
To this purpose, it is important that a programme of data collection is set up since the beginning of a cultural project, so that the evolution of the network can be followed over time, even after the end of the project itself. In this way, the most precious contribution of network analysis relies on the possibility to compare the network created by the project with what remains of it some time after the project dropped out. If follow-up data show that the original network not only has been preserved but it has also expanded, we can state that the project has been successful in guaranteeing the sustainability of its results.
It is important to underline that the approach we propose do not want to be a substitute but rather a complementary tool for cultural project evaluation. Its added-value is to typify the structural and relational dimension of a project as well as to determine its sustainability. The rationale relies on the fact that a cultural project should be also able to create new networks capable to sustain the benefits after its end, to improve local relationships and to create the ground for future collaborations. Results emerging from network analysis can be used to design, monitor and reprogram on the way, as well as to capture and showcase tangible and intangible achievements of cultural projects.
Acknowledgment: This research would not be possible without the brilliant and pioneering view on Cultural Economics that Prof. Walter Santagata bequeathed to his scholars. He left us in August 2013 but his lessons will always remain in our minds. I want also to thank Dr Enrico Bertacchini for his precious suggestions and comments about the traditional evaluation techniques.
Notes
(1) In details, VIZART with its 6 sub-partners (Narodno pozorište Republike Srpske, Banja Luka; Kulturni centar Biha?, Biha?; Pozorište Prijedor, Prijedor; NGO Millenium, Srbac; Bosansko pozorište Tešanj, Tešanj; Scena Talija, Bijeljina) and EVPROPSKE with 10 out of its 19 sub-partners (Ju Gradska Galerija Biha? – Biha?; Demokratski Centar Nove Nade Biha? – Biha?; Ju Muzej Ii Zasjedanja Avnoja – Jajce; Omladinski Centar – Jajce – Jajce; Ju Pozorište Prijedor – Prijedor; Ju Muzej Kozare – Prijedor; Ug Društvo Urbanih Umjetnika Manifest – Prijedor; Ju Kulturni Centar Gradiška – Gradiška; Ju Centar Za Kulturu I Sport Srbac – Srbac; Omladinski Centar Bijeljina – Bijeljina).
(2) Fenix, Proni, Dobri Medo, Biosplus, and Viva.
References
Bertacchini Enrico, Santagata Walter (a cura di) (2012), Atmosfera creativa. Un modello di sviluppo sostenibile per il Piemonte fondato su cultura e creatività, Il Mulino, Collana “Progetto Alfieri”
Cernea Michael M. (2001), Cultural Heritage and Development: A Framework for Action in the Middle East and North Africa, World Bank, Washington (D.C.)
Freeman Linton (2006), The Development of Social Network Analysis, Vancouver Empirical Press
Grun Rebekka E. (2006), Monitoring and Evaluating Projects: A Step-by-Step Primer on Monitoring, Benchmarking, and Impact Evaluation, Health, Nutrition and Population Discussion Paper, World Bank, Washington (D.C.)
OECD (2005), Culture and Local Development, OECD Publishing
Pagiola Stefano (1996), Economic Analysis of Investment in Cultural Heritage: Insights from Environmental Economics, World Bank, Washington (D.C.), mimeo
Santagata Walter, Bertacchini Enrico (2011), Creative Atmosphere: Cultural Industries and Local Development, Working Paper No. 4/2011, Department of Economics “S. Cognetti de Martiis”, University of Torino
Throsby David (2010), The Economics of Cultural Policy, Cambridge University Press
UNESCO (2012), Knowledge Management for Culture and Development, Paris
World Bank (2000), Key Performance Indicator Handbook, Washington (D.C.)
