Evaluating the outcomes and processes of a research-action partnership : The need for continuous reflective evaluation

Programme evaluation and evaluation research have received considerable attention recently (e.g. Fazey et al. 2013; Rossi, Lipsey & Freeman 2003). The quality and significance of research programmes are traditionally evaluated against tangible, clearly measured outputs linked directly to the research itself, such as the number of peer-reviewed publications, number of citations, graduate training and other direct deliverables. Such systems of evaluation tend to suit research in well-defined disciplines but are potentially inappropriate for evaluating interdisciplinary or transdisciplinary research. Transdisciplinary (TD) research results tend to compete with the criterion of academic achievement in disciplinary research and are seldom evaluated in terms of their TD contribution (Bergmann et al. 2005). Given the pluralism of disciplines, research paradigms, and stakeholders’ expectations, interand transdisciplinary research programmes require a specific approach for evaluation (Klein 2008).


Introduction
Programme evaluation and evaluation research have received considerable attention recently (e.g.Fazey et al. 2013;Rossi, Lipsey & Freeman 2003).The quality and significance of research programmes are traditionally evaluated against tangible, clearly measured outputs linked directly to the research itself, such as the number of peer-reviewed publications, number of citations, graduate training and other direct deliverables.Such systems of evaluation tend to suit research in well-defined disciplines but are potentially inappropriate for evaluating interdisciplinary or transdisciplinary research.Transdisciplinary (TD) research results tend to compete with the criterion of academic achievement in disciplinary research and are seldom evaluated in terms of their TD contribution (Bergmann et al. 2005).Given the pluralism of disciplines, research paradigms, and stakeholders' expectations, inter-and transdisciplinary research programmes require a specific approach for evaluation (Klein 2008).
TD research, as defined in the sustainability sciences, is a research approach that addresses societal problems by means of interdisciplinary collaboration, and by transcending the boundary between science and society (Lang et al. 2012).This is achieved through collaboration between researchers and societal actors or practitioners, enabling mutual learning and co-production of knowledge Background: The KwaZulu-Natal Sandstone Sourveld (KZNSS) Research Programme is part of a collaborative, transdisciplinary research partnership between the University of KwaZulu-Natal and the eThekwini Municipality (EM), aimed at bridging the science-policy-practice gap.The research programme focuses on generating knowledge and capacity to support local land-use planning, management and policy development related to biodiversity and climate change issues.(Hirsch Hadorn et al. 2008;Jahn, Bergmann & Keil 2012;Lang et al. 2012).This approach implies a focus on the implementation of research outputs into policy and practice, and is an attempt to bridge the science-policy-practice gap (Knight et al. 2008;Van Kerkhoff 2014).The evaluation of TD research programmes therefore requires the assessment of both research outputs and practice outcomes (Bergmann et al. 2005); this raises challenges as implementation often takes longer than the lifespan of the research project, and because the various participants might perceive and assess the success of the research and the uptake of knowledge into practice differently (Roux et al. 2010).
Conventional evaluation of research programmes does not include use of research outputs for decision-making and practice.Evaluation is traditionally considered as a once-off activity, usually conducted at the end of the programme (Rossi et al. 2003).A single evaluation at the end of a programme means that the programme will not receive the benefit of continuous evaluation with the potential for amendment, or the opportunity to evaluate the process (i.e.how the program was conducted) in addition to the content or outcomes (i.e.what the program generated) (Dick 2003;Ferreyra & Beard 2007).In inter-or transdisciplinary research, attention needs to be given not only to the outcomes, but also the quality of the process (Klein 2008).Continuous evaluation explicitly addresses learning and accountability, and provides an opportunity for reflection by the participants in the programme (Van Ongevalle, Huyse & Van Petegem 2014).
There are several advantages to internalising evaluation and reflection activities within a team (Van Ongevalle et al. 2014), including that: (1) participants gain more ownership of, and accountability for, the process (Roux et al. 2010); (2) participants can adapt the way they work in an iterative manner throughout the life cycle of the project, rather than realising at the end of the process where they went wrong (Woodhill & Robbins 1998);and (3) participants are able to develop a more nuanced understanding of what success might look like (Roux et al. 2010).Evaluation and reflection activities are further enhanced as learning-focused activities when they are conducted in an engaged, participatory manner (Roux et al. 2010;Woodhill & Robbins 1998).Being able to learn from, and adapt to, complex and ever-changing social-ecological contexts is important for TD research partnerships to remain effective, relevant and responsive (Van Ongevalle et al. 2014).
In TD research, paying particular attention to learning is widely recognised as a critical step, which is often referred to as 'transdisciplinary learning' (Roux et al. 2010) or 'social learning' (Keen, Brown & Dyball 2005;Reed et al. 2010).Building participatory evaluation and reflection into TD research programmes is one way of making learning processes more explicit (Roux et al. 2010).Cundill, Roux and Parker (2015) point out that more attention needs to be paid to the social processes, such as learning and participatory reflection and evaluation, which support collaborative TD research initiatives.The present study is in part a response to such calls in the literature, and provides insights into such social processes as are experienced in a TD research programme, through its participatory reflection and evaluation activities.
The Durban Research Action Partnership (D'RAP) provides an opportunity to evaluate a TD research programme through a series of actor-oriented evaluation and reflection activities.D'RAP, a joint research partnership between a local university (the University of KwaZulu-Natal) and a local government department (the Environmental Planning and Climate Protection Department of eThekwini Municipality) was established with the intention to bridge the science-policypractice gap, provide knowledge to assist environmental decision-making and management, and build capacity of both organisations (Cockburn et al. 2016).In local government departments working on environmental management, biodiversity conservation and climate change adaptation, the shortage of human capacity and specialist skills has been recognised across South Africa (Funke & Nienaber 2012;Ivey, Geber & Nänni 2013;Wilhelm-Rechmann & Cowling 2011).Furthermore, the gap between research and action, or science and implementation, is recognised as a barrier to effective environmental management, biodiversity conservation and climate change adaptation in South Africa, and initiatives such as D'RAP are needed to address this hiatus (Knight et al. 2008;Reyers et al. 2010;Sitas et al. 2014).Cockburn et al. (2016) provided lessons for building a successful TD research partnership.They described the establishment of the D'RAP as a TD research programme for addressing the research-action gap, and shared lessons for building successful research-action partnerships.D'RAP is considered as an example of a TD research programme based on its interdisciplinary approach to addressing real-world problems through collaboration with practitioners and decision-makers, thus bridging the gap amongst disciplines as well as between science and society or practice (Lang et al. 2012).
In the current paper, we present an evaluation of the KwaZulu-Natal Sandstone Sourveld (KZNSS) Research Programme, which falls under the broader D'RAP, to reflect on its effectiveness at an early stage of the partnership.This research programme was the D'RAP's first programme, and there were initially few existing relationships between researchers and officials, and limited experience of engaging in these types of partnerships.Whilst recognising that it might be too early to assess the research impact on policy development and implementation practices, we share important lessons for establishing continuous learning and reflection processes in TD research programmes.We present a continuous, reflective evaluation based on the participants involved in D'RAP.We specifically aim to evaluate and reflect on (1) research outputs and outcomes; (2) the translation of the research into policy and practice; and (3) participants' perceptions of the outcomes and processes.We reflect on these participants' perceptions over the three years of the programme, draw on insights from the literature, and share lessons for evaluation and reflection processes in TD research partnerships.

Method Case study: The KZNSS research programme
The KZNSS research programme was officially initiated in May 2011 and ran for three years till June 2014.Research projects, conducted almost entirely by postgraduate students, started in January 2012 and the evaluation focused on the period from 2012 onwards.EThekwini Municipality (EM) provided funding of R1 500 000 for the duration of the three-year programme.Co-funding was leveraged and this amounted to an extra contribution of R2 240 000 over the three years.Funding was spent on student bursaries, project running costs, research assistants, and overall project co-ordination.
The programme focused research effort within the KZNSS ecosystem.This ecosystem is found only within the province of KwaZulu-Natal in South Africa, has very high species diversity and endemism, and has been identified provincially as critically endangered (Jewitt 2011).Owing to this conservation status, the municipality is mandated to conserve and manage it effectively (Boon et al. 2016:this issue).The research objectives of the programme were to: • assist EM in communicating the ecological and socioeconomic value of KZNSS • build capacity and human capital in the areas listed above • develop a learning organisation.

The need for a continuous planning, monitoring and evaluation approach
By the end of 2012, one year into the KZNSS programme, a lack of common understanding of each institution's research needs, framing, and how to conduct a collaborative TD research process was recognised.This triggered the need to evaluate the process and outcomes of the programme from the end of 2012 onwards (Figure 1).The co-ordination team comprised 6-10 members from both institutions, and was tasked with conducting an evaluation and reflection process.Various activities were conducted (Figure 2, activities 1-11) and followed a generic approach of evaluation, reflection, learning and adaption (Figure 3) (Roux et al. 2010;Van Ongevalle et al. 2014).Each activity was carried out at a different time in the programme period (2012-2014); however, activity 11, the special issue publication from 2016 (Figure 1), is also included as this was a direct output of the programme.For the purpose of clarity, the activities were arranged into three broad evaluation objectives: A. Research outputs and outcomes of the programme, including capacity building: assessing outputs (e.g. the number of graduates and research papers published) and outcomes (e.g.increased knowledge of the KZNSS).Traditionally, these include measures of scientific practice recognised by academic institutions (i.e.outputs for research).B. Translation of research into societal practice: assessing the suitability and the integration of research into EM environmental policy development and decision-making and management.This objective measures the translation and integration of research into action (i.e.outcomes for practice/implementation).

FIGURE 2:
Summary of all activities undertaken to evaluate the KZNSS Research Programme arranged according to the three main objectives of the evaluation process.The need to assess participants' perceptions of their research.

Event
Changes to project design and targeted approach to climate change projects.The activities were conducted in a participatory manner with all programme participants in order to embed reflective practices into the programme, and to begin building a learning organisation (Senge 1994), rather than a conventional programme solely focused on generating research outputs.Particular attention was paid to the principles of enabling leadership to create a suitable atmosphere for reflection and learning (Galuska 2014;Uhl-Bien, Marion & McKelvey 2007), which included learning-by-doing, allowing for mistakes, and creating opportunities for participants to question processes as they unfold.This was done by encouraging a flat, rather than a hierarchical structure, and encouraging students and junior academics to participate in meetings and discussions where decisions were made.Discussions took place in a manner that emphasised listening to and respecting diverse viewpoints.Furthermore, team-building activities such as excursions to project sites and social events were seen as opportunities for building relationships of trust and social capital amongst participants (Cheruvelil et al. 2014).The importance of building social capital and collaborative capacity in this case study is discussed elsewhere (Cockburn et al. 2016).
The approach that was followed to assess and report on each evaluation objective is presented below.Further details concerning the specific method of each activity (including time frame and participants) are given in Supplementary Material: Appendix 1.

Assessing research outputs and outcomes
The research outputs were assessed by standard measurements of quantitative scientific productivity (i.e.number of research projects, students trained and graduated, scientific outputs including publications and data).Specifically, a close-out report was compiled which detailed the scientific and collaborative management outcomes, as well as the human capital and social learning outcomes.Documenting the scientific outcomes involved recording the number of students who graduated through the KZNSS under each respective qualification (Honours -a separate 4th year of study following the 3-year Bachelor's degree, Master's and PhD).The programme addressed four different themes (biodiversity, ecosystem functioning, land use change and socio-economic changes); therefore each student project was classified accordingly.

Assessing the translation of research into policy and practice
An evaluation of the implementation of research into practice should focus on the extent to which data and knowledge generated by the KZNSS programme influenced policy development, decision-making and management.This is a lengthy process where the influence of research can be felt many years after the programme.Implementation of the KZNSS research programme into practice is still ongoing and only anecdotal evidence is available for assessment.
Here, the focus of the evaluation was on the extent to which the research could contribute to policy-development, decision-making and management.This was done by assessing the relevance of research in terms of policy/practice issues raised by eThekwini, alignment of research projects with EM needs, and any anecdotal evidence of translation of research into practice.
The programme first started with a very open approach, and research projects were initiated without a clear focus.
There was a tacit understanding that management guidelines would emerge once the research was conducted, following a conventional linear approach to knowledge dissemination.Following the realisation, at the end of 2012, that the research programme was not meeting the needs of the municipality, a research framework was developed (Figure 2, activity 1).This research framework provided a mutual understanding of research needs between the parties and a way to align research with decision-making products (Appendix 1).Formal proposal presentations, presented to the co-ordination committee, were required before new projects were accepted under the programme.Decisions on new projects were made by the co-ordination team (both UKZN and EM representatives).The municipality also provided research questions, in line with the research framework, based on needs and issues of concern to the municipality.These questions were used to guide research topics.
Students provided regular updates in the form of bi-annual (June and December) progress reports and presentations to principal investigators (PIs) and the co-ordination team.After presentations were given by students, discussions were held by the co-ordination team on the extent to which research projects were suitable for translation and integration into EM biodiversity and climate change actions.These group discussions were short (1-2 hours) and took the initial questions posed by the municipality and compared them with student research projects to assess which questions had been addressed.
As part of the students' progress reports, a section was included where students indicated the alignment of their project with the research objectives of the programme.This section encouraged reflection on how individual students perceived their project and project outputs to be of value and use to the municipality.This component of the research aligns with Cockburn et al. (2016) where an essential action for a successful partnership is to 'conduct research with implementation in mind'.

Assessing participants' perceptions
To assess the perceptions of participants in the research programme (including staff/PIs, students from the university and practitioners from the municipality), a mixed methods approach was taken, which included the following activities: an anonymous online questionnaire and follow-up questionnaire (Appendix 2) a year later, individual reflection cards, personal reflection on success and investment over time, and focus group discussions (Appendix 1).The results of the questionnaire and the reflections were collated and analysed using emergent content analysis (Creswell 2009).These activities assessed participants' perceptions of both the tangible and the intangible outcomes of the programme.

Evaluation of research outputs and capacity building
An objective of the KZNSS research programme was to build capacity and human capital.A total of 29 students were involved in the programme and 26 have graduated through UKZN, as detailed in Table 1.For a three-year programme with relatively limited funding, these figures indicated a high level of capacity building within UKZN, when compared with similar initiatives at the University.
Considering the number of projects completed, the resultant publications number was lower than initially hoped for, with only two publications in the ecosystem function theme (McPherson et al. 2016a(McPherson et al. , 2016b) ) and one across TD research (Cockburn et al. 2016) (Table 1).However, more than two years after the programme's funding period finished, 11 publications flowing from this programme have formed part of this journal's special issue.This highlights the substantial lag phase in producing standard research outputs.

Evaluation of translation of research into policy and practice
For the duration of the research programme, several activities were initiated to assess the alignment of the research projects against the research objectives and needs of eThekwini Municipality (Figure 2).This process was important for the municipality to redefine and clarify their research objectives/ questions, and for the programme to develop a coherent and reasonable research framework (see Cockburn et al. 2016).
The activities were also essential in identifying key gaps in research according to the municipality's research questions, and realigning the programme where necessary.Major research gaps in governance and climate change issues were identified; however, these were only partly addressed.
At the end of the funding period, whilst most research projects related to EM needs, the data and knowledge generated did not directly translate into practice or policy.This was largely because of issues of format, accessibility and usability of the information, which the municipality identified as barriers to implementation of the research into policy and practice.To address this challenge, the programme has now initiated a process of developing more integrated and implementable knowledge products, such as practitioner guidelines which synthesise the research on the KZNSS ecosystem.Whilst we were unable to assess the impact on formal and informal policy, anecdotal evidence (DR, personal communication) suggests that the partnership highlighted the importance for local government to link to science.Interestingly, the research programme yielded some unexpected results which indicate some success in bridging the research-practice-policy gap.For example, through interactions with international researchers and training organised by the programme, EM has now changed their practice around conservation planning and is now using a new software, Zonation (Moilanen 2007), introduced through the programme.This was only made possible through the research links between UKZN and the researchers who developed the Zonation software.With regards to practice, the KZNSS research programme has laid the foundation for interaction between researchers and practitioners around land use planning and policy, through the implementation of a new research programme to develop the eThekwini Municipality Strategic Environmental Assessment.The co-ordination team which was set up for the KZNSS programme has continued working together as new programmes have been added to the overall partnership, and meets on a regular basis.Two research co-ordinators have now been employed to manage the growing partnership.

Evaluation of participants' perceptions of outcomes and processes
Participants were asked to indicate the level of investment in the programme relative to the level of success they felt was achieved (Figure 4, Appendix 1).This was repeated for each year of the programme from 2012-2014.Some participants only joined the programme in 2013 which explains the increase in respondents over the years.In 2012, there were higher levels of perceived investment relative to success, and this was especially expressed by members of EM (Figure 4).By 2014, the perceived level of success had notably increased from 2012 (more data points in the upper left quadrant of the graph in Figure 4), reflecting a higher level of satisfaction.
The evaluation survey helped participants to reflect on their role and participation in the research programme, as indicated by 83% of respondents from 2014 (respondents included 4 EM staff, 9 UKZN staff, and 6 UKZN students).Knowledge generation and training of students were perceived to be the most successful aspects of the partnership.Respondents felt that the greatest factors contributing to the success of the programme were: (1) co-operation, collaboration and commitment of the participants and partner institutions; (2) grant funding and other support; and (3) good communication between the two institutions.The top three challenges that respondents experienced were (1) time constraints; (2) financial and logistical support; and (3) the need to work with different organisational cultures.
From 2013 to 2014, there was a decrease in the percentage of respondents who felt the programme had addressed many of the product-orientated outcomes such as assisting EM with decision making, developing monitoring tools, and communicating the value of the KZNSS (Figure 5a).In 2013, participants were more positive that the programme would address these needs but, by the end of the programme, participants realised that this had not been the case.On the contrary, participants perceived the success of the research programmes more favourably in 2014 with regards to 'soft' outcomes and the overall process (Figure 5b).Over 80% of respondents indicated that the programme had helped to increase their ability to work with diverse stakeholders, to develop new long-term work relationships, build new links between organisations, and build trust and mutual understanding among partners (Figure 5b).There was a distinct change in perceptions over time, with a decrease in the perceived success of addressing tangible outcomes and increase in the perceived success of the programme development (e.g., through better collaboration and the building of relationships and trust between partners).
In general, the programme appeared to facilitate TD research, and participants appreciated the organic, flexible nature of the programme and the open communication and exchange of ideas.The aspects that participants felt should be improved included an increase in administrative support and a broader research focus, and the leaders in the partnership were able to respond to these concerns and adapt the programme accordingly (within available resources).

Discussion
The process of evaluation and reflection of the KZNSS research programme has been rich in learnings relevant to the programme itself and the broader D'RAP partnership but also to other TD research groups.In particular, we discuss here the role of evaluation and reflection in such partnerships, the challenges of knowledge exchange, and the importance of paying attention to the ongoing research process.
The first key lesson which emerged from the present study was the importance of evaluation and reflection in a TD research partnership.After conducting a wide range of evaluation/reflection activities (Figure 2), the participants gained a richer and deeper understanding of the successes and the challenges of the partnership.Much of that learning would not have been possible without the evaluation and reflection process.For example, without this process, the impact of the programme could not be fully assessed, especially its importance in building the foundations for long-term research partnership between EM and UKZN (Figure 5b).A continuous process of evaluation, reflection, learning and adapting as well as flexibility with participants enabled the programme to readjust and improve as it was happening.
Evaluation and reflection are best done continuously and in a participatory manner to support learning and adaptation (Biggs et al. 2011).Building an atmosphere of reflection, learning and adaptation requires enabling leadership (Roux et al. 2010); it also requires additional resources such as time commitment (Klein 2008).For evaluation and reflection activities to be given appropriate attention, sufficient resources need to be explicitly allocated to this activity, possibly through the appointment of 'process champions' who can guide and facilitate such activities (Cockburn et al. 2016;Gray 2008).Lang et al. (2012) and Cockburn et al. (2016) provide several suggestions on guiding principles for TD research which can also help to inform a comprehensive evaluation of TD research partnerships.Through the process of reflective evaluation, we were able to identify the greatest success of the research programme in the less tangible outcomes of building social capital and collaborative capacity, which have laid a firm foundation for future working relationships between research and practice.As this evaluation provides an early assessment of a growing research partnership, we are able to take the learning into the partnership's future development.
The second key lesson that emerged from the present study was recognising the challenges in exchanging and integrating   knowledge.This is not a new finding but it is often underestimated by stakeholders involved in collaborative research programmes.The KZNSS research programme was only established in 2012 and work on integration is still taking place.As a result, the programme had relatively low levels of integration of research into action compared with the initial expectations of both the researchers and practitioners.A continuous reflective evaluation process, as described in the present study, helped to explicitly consider the translation of research into practice and paved the way for future knowledge integration.Integration of diverse knowledge types, and synthesis of knowledge into useful forms for practitioners and implementers, are widely recognised as significant challenges of TD and implementation-focused research (Pooley, Mendelsohn & Milner-Gulland 2014;Pullin et al. 2016;van Kerkhoff 2014), with some authors even considering integration to be the crux of inter-or transdisciplinarity (Klein 2008;Lang et al. 2012).
The present programme highlighted that the conventional linear approach to knowledge dissemination, initially followed during the first year, is not entirely suitable for researchaction partnerships (Lang et al. 2012;van Kerkhoff & Lebel 2006).For scientists to provide recommendations after the research and publication is complete, is not an effective TD practice.TD research seeks ongoing co-generation of knowledge, rather than researchers providing the results to the implementing agent at the end of a research process.One challenge resulting in this traditional 'trickle-down' of information is that, in reality, researchers and practitioners are still working in different organisations (i.e.what is often referred to as 'sitting in silos') (Pooley et al. 2014), and each comes with its own expectations in terms of conventional knowledge products such as peer-reviewed papers (UKZN) and practitioner-focused management guidelines, policy briefs and handbooks (EM).
Co-generating knowledge through TD research partnership can be a lengthy process (Roux et al. 2010).Appointing consultants to advise on action is an alternative method to obtain results timeously but this is at the expense of a formal peer-review process which validates the results.However, research partnerships such as D'RAP are critical to increase human capital and build long-term datasets to track global change which can hardly be achieved by appointing a consulting company.Such a partnership model between a university and local government should be encouraged as South Africa lacks appropriate human capacity in environmental fields and government departments (Funke & Nienaber 2012;Ivey et al. 2013), particularly in the areas of environmental management, biodiversity conservation and climate change adaptation (Wilhelm-Rechmann & Cowling 2011).TD research programmes should operate over longer time frames (e.g., at least 5-10 years).Such timeframes are typically not aligned with traditional research and practice timeframes (e.g.funding and degree cycles); however, they are necessary to allow sufficient time for building relationships and co-developing integrated knowledge (Klein 2008).Boundary organisations, such as D'RAP, and institutional champions (Franks 2010;Long, Cunningham & Braithwaite 2013) can provide long-term support and stability between funding periods.
The third key lesson which emerged from the present study was that a TD research partnership requires attention to the ongoing process (Klein 2008;Roux et al. 2010).The online questionnaire to assess the successes and challenges of the partnership revealed the importance of 'soft', less tangible outcomes (i.e.paying attention to the process of building relationships and not only the research products or outputs) (Figure 5a and 5b).Investing in process rather than product takes much time, effort and commitment in order to build relationships, understanding, and shared decision-making regarding the research programme (Cockburn et al. 2016).Investing in process is often overlooked in research-action partnerships and their evaluation because there is often no obvious product (Fazey et al. 2013).
The importance of these 'social factors' in TD research partnerships is widely recognised (Gray 2008;Klein 2008;Sitas et al. 2016), and TD research processes are considered 'social processes of knowledge production' (Spaapen, Dijstelbloem & Wamlink 2007).An important lesson from evaluating participants' perceptions was realising the value of less tangible outcomes.Most participants recognised that we had not yet achieved the more tangible, conventional research outputs of the programme but that we had achieved important networking outcomes such as building social capital and developing collaborative capacity (Figure 5b).The trust building and the laying of a foundation of effective working relationships between the partners was considered a key success factor in the subsequent development of three new research programmes in partnership with EM (Reforestation Research Programme, Global Environmental Change Research Programme, and Strategic Environmental Assessment).
Method: This was another mini reflection conducted at a full team meeting where students had presented their research to EM and UKZN staff.Each participant was given a blank graph (Figure A1) and had to position the relationship between investment (i.e.time, effort and project funds spent by the respondent or his/her institution) and the success of the programme (as perceived by the respondent/institution).This was done in 2014, so participants reflected back on 3 years of the programme (2012, 2013 and 2014).If a respondent indicated that a particular year fell below the diagonal dashed line, they perceived their investment into the programme to be greater that the success achieved.However, if the year in question was depicted above the diagonal line, the respondent perceived the success to be greater than the investment.The x and y axes provide a relative scale of increasing proportion as the level of success or investment.Results from this activity are presented in Figure 4.
Activity 9: Close-out workshop Aim: The purpose of this workshop was to reflect on successful and unsuccessful aspects of the partnership and explore future goals through an in-depth reflection process.Learning through this process was also incorporated in the planning for phase two.
Method: A 2-day workshop was held to reflect on the research that had been conducted, considering how it could inform practice, where the major gaps were, and how practice could inform research.Various activities were organised to facilitate the reflection and discussion process.
The reflection process was divided into three parts.The first was a paired verbal reflection with another workshop participant.The second was a 20-minute personal reflection, and the last was a feedback session to share thoughts and ideas.The following questions were suggested for each reflection: The close-out report can be accessed online: http://www.durban.gov.za/City_Services/development_planning_management/environmental_planning_climate_protection/Publications/Documents/KZNSS_Close_out_Report2011_2014.pdfActivity 11: Special issue publication Aim: To publish peer-reviewed articles of new knowledge generated through the research programme.
Method: Researchers were invited to voluntarily submit manuscripts to be included in a special issue of a peer-reviewed journal.A special issue compilation was decided on as all manuscripts touched on various aspects of the KZNSS.Where possible, authorship of papers included members from both institutions, adding to the aims of co-generation of knowledge.Guest editors also represented both EM and UKZN.The special issue includes 11 manuscripts adding to the knowledge of the KZNSS across various disciplines.

•
improve understanding of biodiversity, ecosystem functioning and ecosystem services • improve understanding of past, present and future land use changes • assess the effects of climate change • develop monitoring protocols in the face of climate change • address specific climate change adaptation challenges, including ecosystems-based adaptation • assist EM with decision-making for land use planning and policy

FIGURE 1 :
FIGURE 1: Timeline indicating when each evaluation and reflection activity was carried out and which objective each activity aimed to address.

FIGURE 3 :
FIGURE 3: Diagram depicting the continuous process of evaluation, reflection, learning and adaptation, as implemented in the KZNSS programme.

FIGURE 4 :
FIGURE 4:Participants' perception of overall success and investment over time.Investment refers to the amount of: time, effort and project funds allocated to the research programme.Success refers to the success of the programme as perceived by the participant.No unit of measure was used to determine investment or success, but rather participants were asked to gauge their response on a scale from low to high.

FIGURE 5 :
FIGURE 5: Change in participants' perceptions of (a) outcomes and (b) processes between 2013 and 2014.

TABLE 1 :
The number of research projects and scientific publications completed via the 3-year KZNSS research programme (2012-2014) and the number of students funded by eThekwini Municipality (EM) or other funding sources.