Abstract
Background: The economic and social impacts of the coronavirus disease 2019 (COVID-19) pandemic have hindered efficient service delivery especially at a local government level. The significance of impact evaluation within this context is critical and is highlighted by the emergence and current implementation of the National Evaluation Policy Framework.
Aim: The study attempts to explore the impact evaluation of COVID-19 on service delivery in South African municipalities.
Setting: The study was conducted at Nelson Mandela Bay Municipality (NMBM) and Buffalo City Metropolitan Municipality (BCMM).
Methods: Researchers used a mixed method approach to collect data. In-depth interview questions were used to collect data in one-on-one interviews with 13 municipal senior managers. The questionnaires were designed to meet the specific research objectives and were distributed to the municipal employees in different employment categories. A total of n = 191 people were sampled, questionnaires were despatched and 186 were returned, which gave a 97% response rate.
Results: The finding demonstrates that outcomes and impacts evaluations are not important in these municipalities and the absence of monitoring and evaluation (M&E) continually remains no one’s responsibility.
Conclusion: There is no confidence in these municipalities that monitors plan to improve service delivery and will work towards achieving the goals of the organisation beyond COVID-19.
Contribution: This study could potentially empower municipal managers and policymakers by identifying a variety of shortcomings and offering advice on their action planning.
Keywords: accountability; implementation, monitoring and evaluation; policies; programmes; transparency.
Introduction
The coronavirus disease 2019 (COVID-19) pandemic is having a devastating impact on the South African economy and the health and livelihoods of its citizens. Therefore, the economic and social impacts of the pandemic have hindered efficient service delivery, especially at the local government level (Hes 2020). Taking into account the situation at the Nelson Mandela Bay Municipality (NMBM) and Buffalo City Metropolitan Municipality (BCMM) where they had experienced a variety of direct effects and were among the seven metros that were declared as hotspots (Government Gazette No 43364). This means that these two municipalities had to cease some of the activities and comply with lockdown measures. These municipalities have been struggling with a series of challenges that include incessant poverty, unsafe drinking water, inadequate sanitation and the high unemployment rate of the majority of its residents (Eastern Cape Socio Economic Consultative Council 2021; Gxabuza & Nzewi 2021). This latter has triggered the study to examine how these two municipalities conduct the impact evaluation of COVID-19 on service delivery. The significance of impact evaluation within this context is critical and is highlighted by the emergence and current implementation of the National Evaluation Policy Framework (NEPF) (Mbava & Rabbie 2018). Impact evaluation is a new practice, but not much literature has been written about it in the context of municipalities (Wróblewska 2021). As a result, it is also true that improving impact evaluations is indeed a challenge. Therefore, this article aims to help meet that challenge by laying out a potential solution.
It was anticipated that municipalities will feel the impact of the pandemic far beyond the lockdown period. Therefore, municipalities have to act with unprecedented skill to be able to respond to the government’s changing regulations (South African Local Government Association [SALGA] 2020). The Department of Planning Monitoring and Evaluation (DPME) developed a short survey to realise issues and challenges faced by South African municipalities as a result of the COVID-19 pandemic in order to support municipalities and their communities. The survey results show that several municipalities find it increasingly difficult to deliver water, sanitation and electricity to communities (DPME 2022). This is, however, not only because of additional pressures brought about by the COVID-19 pandemic such as the additional strain. Unfortunately, these results did not explore the impact evaluation of the COVID-19 on service delivery. In this regard, the competence of municipal officials in management and accountability is key to addressing and reducing the impacts of COVID-19 on citizens (Ball 2020). Hence, the study attempts to explore the impact evaluation of COVID-19 on service delivery in South African municipalities.
Literature review
Theorectical and conceptual framework
As a theoretical foundation, this study is based on a Theory of Change (ToC) and Disruption Theory. Impact evaluations are viewed as a component of a well-developed ToC that clearly outlines the causal pathways through which a programme works to produce outputs and influence final outcomes (Gertler et al. 2016). Reflecting on the impact of COVID-19, recent research relies on disruptive theory and is more focused on practices and methods to manage and inspire disruptive innovations (Vaska et al. 2021). From the standpoint of service delivery during COVID-19, impact evaluation is a critical foundation (Thornton et al. 2022). Given the cause-and-effect focus of the research, a ToC is a critical underpinning of any impact evaluation (Helstrom & Helstrom 2018; Weiss 1998). This is complemented by Disruption Theory as it can be used to create a balanced portfolio of innovative initiatives (Raylor 2011). Constructing a ToC as one of the first steps in the evaluation design can assist in clarifying the research questions (Gertler et al. 2016). While an impact evaluation can show whether or not a programme is effective, it must also include an implementation evaluation to show how the ToC processes are being carried out (DPME 2014). Theory of Change continues to be a crucial method for managing evaluations during COVID-19 projects because of the variety of project and programme types that exist in municipalities (Weiss 1998). According to this ToC, activities and outputs will result in possible outcomes and the impacts (Gaines 2020; Patrizi & Patton 2010). This study draws discernment on the basis of these theories, as it dissects the the impact evaluation of COVID-19 on service delivery in NMBM and BCMM.
The ToC can be represented in the form of a logframe to demonstrate the link between the inputs, processes (actions), outputs, outcomes and impact of policies, programmes and projects (Woodhill 2007). The logframe incorporates indicators at different levels and baselines as part of the logical framework. This is followed by the key assumptions and risks that underlie the results chain and key outputs and related activities required to achieve the desired outcomes. A summary of the human and financial resources (inputs) needed to achieve the outcomes and impacts. The logframe should be continuously adjusted to accommodate changes in the environment (Uwizeyimana 2020). Disruption theory aligns ideas that are already developing in order to improve the municipal chances of succeeding.
The discourse of impact evaluation
Impact evaluation is one of many approaches, along with monitoring and other forms of evaluation that support evidence-based policy. Gertler et al. (2016) advise combining impact evaluations with monitoring and complementary evaluation approaches to get a complete picture of outcomes. This indicates that monitoring and evaluation (M&E) are at the heart of evidence-based policymaking (Thornton et al. 2022). Monitoring is an ongoing process that keeps track of what goes on within a programme and uses the information gathered to guide daily management and decision-making, programme implementation and other activities. Meanwhile, evaluations are used selectively to respond to specific questions related to design, implementation and outcomes (Imas & Rist 2009). Impact evaluation offers a fundumental set of tools that stakeholders can use to assess and enhance the quality, effectiveness and efficiency of policies and programmes at different stages of implementation or, to put it another way to concentrate on outcomes (Gertler et al. 2016). Finally, municipalities have a responsibility to inform the public about effectiveness of public programmes during COVID-19. Transparency and accountability can have a strong foundation built on evidence. A rigorous impact evaluation should provide credible evidence by establishing causal attribution using an appropriate counterfactual (Adams, Barnes & Pressey 2019; Ferraro 2009; White & Raitzer 2017).
Impact evaluation, like other types of evaluation, serves two purposes. The first purpose is accountability, which ensures that development actions result in development outcomes. The second is learning, which provides an evidence base for choosing and designing development interventions that are likely to be effective in promoting desired outcomes (Reed et al. 2021). The study by King et al. (2015) draws a distinction between ‘evaluation studies’, the latter of with ‘focus on immediate questions of what and how something works’. This distinction is a problematic. The fact is that, ‘impact evaluations are concerned with determining what works and why’ (International Labour Organisation 2020; Leeuw & Vaessen 2009; Owen & Rogers 1999). Planning in advance and specific objectives from practitioners are essential for a successful impact evaluation. Impact evaluation can be a crucial mechanism for reducing the risk of unanticipated negative outcomes that might otherwise go unnoticed by practitioners for years (Jensen 2011). It is impossible to determine the causal effects of development interventions without an impact evaluation. It is neither possible to hold development expenditures accountable nor to derive meaningful knowledge from development operations to improve development policies in the absence of understanding what effects have occurred as a result of development efforts (White & Raitzer 2017).
According to the World Bank’s Independent Evaluation Group (IEG), impact evaluations play a critical role in the global push for better evidence on results and development effectiveness (IEG 2005). They are particularly well suited to responding to questions regarding the effectiveness of development interventions. Well-designed impact evaluations also clarify why an intervention worked or did not work, which can vary over time and space. This will ensure that resources are allocated where they will have a greatest impact while also maintaining future public funding for international development is maintained (Leeuw & Vaessen 2009). Decision makers require more evidence about the impact and its causes. The pressures for this are already significant and will grow as resources for international development are increased. Without such evidence, the justification for aid and potential future funding sources might be questioned. If an impact evaluation reveals that a programme has not achieved desired results, it is imperative to determine whether it has been properly implemented and how the ToC is working in practice (DPME 2014).
The primary function of impact evaluations is to produce data on programme performance for use by government officials, programme managers, civil society organisations and other stakeholders (Gertler et al. 2016). In addition to setting and tracking national and international goals, it requires programme managers to increase accountability, determine how much money to budget for each programme and direct programme design and policy decisions. Impact evaluations generate pertinent information from the standpoint of accountability, they reveal knowledge about the (societal) effects of programmes that can be linked to the (financial) resources used to achieve these effects (IEG 2005). Its goal is to determine the extent to which observed and measured changes can be attributed to specific factors, such as policies, programmes or other interventions. Aside from the immediate benefits, impact evaluation assists in comprehending how the programmes overall impact service delivery beyond. Hence, the study aims to answer the fundamental question of how COVID-19 impacted service delivery in South African municipalities and how impact evaluations can empower institutions such as municipalities.
Institutional framework for impact evaluation in South African municipalities
In South Africa, impact evaluation is acknowledged in the NEPF. According to this framework, impact evaluation ‘is an approach that measures changes in outcomes and the well-being of target beneficiaries that are attributable to a specific intervention’ (NEPF 2011). The NEPF is a component of the Government-wide Monitoring and Evaluation System (GWMES) and regards impact evaluation as one of the main evaluation foci in the South African public sector (RSA 2007). The framework provides for an evaluation system to be implemented at a minimum across the entire government in order to ‘improve the effectiveness and impact of government by reflecting on what is working and what does not working and revising interventions to achieve those objectives’ (DPME 2014). This emphasises the importance of public sector impact evaluations in providing evidence of programme outcomes while also revealing the reasons for potential successes or failures (Mbava & Rabbie 2018).
Furthermore, the NEPF states that evaluations should address the needs of the target population and be ‘relevant in relation to the evolving needs and priorities of government’ (RSA 2011:7). The scholars further opine that specific interventions have been planned to ensure that objectives, inputs, outputs, outcomes and anticipated impacts of local government are in line with citizens needs (Kariuki & Reddy 2017). Municipalities in South African appear to be struggling to find ways to assess the outcomes and impacts of their projects and programmes. In addition, evidence suggests that existing impact evaluations have gaps and limitations, implying that evaluation methods and designs are not always appropriate for informing policymakers’ needs (Mbava & Rabbie 2018).
The impact of coronavirus disease 2019 on service delivery: The municipal experiences
The economic impact of lockdown has been devastating resulting in numerous job losses as long-standing businesses in the hospitality and tourism industry as well as small and informal businesses have closed (Jain et al. 2020; Spaull et al. 2020; Visagie & Turok 2021). The ability of the local government to provide services has been hampered by a significant decline in its revenue sources (DPME 2020b; Gumede 2020; PMG 2020a). As so many households lacked sufficient water, the country made providing water a top priority (Mkhize 2020; PMG 2020b). Additionally highlighted were the variations in municipal service delivery, particularly regarding water (Fani 2020). A few municipalities have been identified as working on the impact of COVID-19 (Nel & Lewis 2022). The authors also emphasise the importance of municipalities performing infrastructure provision and maintenance functions as well as service delivery.
It is apparent that the pandemic accentuated existing challenges regarding service delivery, as well as providing new challenges and opportunities. Consequently, service delivery has become highly politicised. There was a failure of governance during the COVID-19 pandemic (Narsiah 2021). Several instances concerning the failure of service delivery have been highlighted. The study of Mathiba (2020) tackles the topical issue of corruption in the public procurement process during the COVID-19 crisis in South Africa that is seen to have an impact on service delivery in municipalities. The author maintains that the unveiling of the multi-billion rand COVID-19 pandemic relief package by the South African Government in 2020 without a comprehensive regulatory framework on emergency procurement led to the massive corruption that took place during the emergency procurement process. The COVID-19 pandemic was then being classified as ‘a national disaster in provision of section 23(1)(b) of the Disaster Management Act, 2002 (Act No. 57 of 2002)’ (De Coning 2020). South African municipalities have been required to rapidly set up an emergency response to COVID-19 and to establish effective response systems to mitigate the severity of its impact.
A number of studies have pointed out serious socio-economic challenges such as poverty, unemployment, underdevelopment and corruption (Mathiba 2020; Ngcaweni 2020; Rachidi 2020; Tafeni & Mngomezulu 2020). These challenges are likely to have escalated to serious levels because of COVID-19’s severe budget cuts and regulations that prevented any form of work except those classified as essential services. The COVID-19 pandemic, like all pandemics, has a spatial dimension that needs to be managed (McCoy 2020). However, it has negatively affected service delivery by municipalities. The COVID-19 pandemic in South Africa has brought to the fore systemic weaknesses in the quality of service delivery such as water and sanitation services, housing and healthcare in various communities across the country (Mahlala & Netswera 2020). It was highlighted by the DPME (2020:10) that additional pressures were brought about by the COVID-19 pandemic. Audits have shown that local government not only struggles with managing finances but is also plagued by non-delivery, fraud, a lack of accountability and the inability to effectively manage key projects and programmes (Auditor-General 2020:9).
Mahlala and Netswera (2020) assert that the South African Government has an essential obligation to monitor and evaluate the public service and public institutions. The public service should deliver services that a society requires to maintain and improve the welfare of its citizens. This means all municipalities are obliged to provide municipal services to all members of society in a most transparent and equitable manner (Tafeni & Mngomezulu 2020). As a result, municipalities are monitored by national and provincial governments in relation to their performance in discharging their development and service delivery responsibilities (Constitution of the Republic of South Africa 1996). Moreover, service delivery can be regarded as the paramount function of any government (Kwandayi, Makanyeza & Ikobe 2013). Section 197(1) of the Constitution states that there is a public service for the Republic that must function and be structured in terms of national legislation. Municipalities must loyally execute the lawful policies of the government of the day. Nevertheless, municipalities on their own are unable to respond to the service demands because of backlogs fuelled by COVID-19 implications. The existence of M&E provides local government with an opportunity to be introspective in terms of the policy impact and processes that enhance service delivery (Sebake & Mkhonza 2020). This means that the management of COVID-19-related requires changes in conditions of service in the public sector (Pietersen 2020). The author provides a robust reflection on the disruptive elements of the COVID-19 pandemic on the conditions of service delivery in the national, provincial and local government spheres. South African municipalities can learn from the experiences and lessons of OECD countries in terms of how they have responded to the COVID-19 pandemic and the impact of service delivery at municipalities. For example, the studies of Rachidi (2020) have revealed the impacts of the pandemic on human resources. The author argues that management needs to take into account the well-being of employees as this is critical to maintaining organisational productivity. The author contends that being considerate of employees’ well-being would require management to employ innovative strategies and be open to continuous change. Furthermore, Rachidi (2020) recommends that during and after the pandemic, management should work to create employee friendly work environments that provide emotional support to employees, as all these factors are critical in maintaining employee productivity during and after a public health crisis such as COVID-19.
Methodology approach
The study employed a mixed-methods approach. This approach requires the quantitative and qualitative data to be collected concurrently in a convergent core design and the results are merged together (Creswell & Creswell 2018). This approach offers completeness, complementarity, resolution of complex issues, enrichment and results quality (Leedy & Ormrod 2021). Hence the study used quantitative methods for impact evaluations to measure and quantify the effects of COVID-19 and used qualitative methods to provide valuable insights into the processes, mechanisms and contextual factors that contribute to the observed impacts.
In-depth interview questions were utilised to collect data and targeted 22 municipal senior managers in one-on-one interviews who are heads of directorates or departments. The researchers used their judgements to select the municipal departments that have continued to provide essential services to communities during the COVID-19 lockdown, such as waste removal, water, sanitation, electricity and disaster management, as well as safety and wellness. Out of 22 municipal senior managers, only 13 managers were interviewed. The researcher reached the degree of saturation (Guest, Namey & Chen 2020). The questionnaires were structured according to the specific research objectives and distributed to the municipal employees in different employment categories. The structured questionnaires were created using a 1 to 5 Likert scale, with 1 indicating strongly disagree, 2 indicating disagree, 3 indicating undecided, 4 indicating agree and 5 indicating strongly agree. A total sample of n = 191 was sampled, according to the Sekaran (2003) sampling table.
The targeted respondents were solely responsible for COVID-19 activities or operations. In total, 191 questionnaires were despatched and 186 were returned, which gave a 97% response rate. The International Business Machines Statistical Package for the Social Sciences (IBM SPSS) version 27.0 was used to analyse quantitative data (Bandalos 2018) and NVIVO version 12 to analyse the qualitative data. The Cronbach’s co-efficiency alpha values were 0.888, and the total number of items was 13 (0.888 N = 13) and was used to determine the reliability of the individual dimensions, with a reliability coefficient indicating a high degree of acceptability, as well as consistent scoring for the different categories of this research (Taber 2018).
Descriptive statistics were used to summarise data in an organised manner by describing the relationship between variables in a sample or population (Kaur, Stoltzfus & Yellapu 2018). Descriptive statistics include types of variables (nominal, ordinal, interval and ratio) as well as measures of frequency, central tendency, dispersion/variation and position. As descriptive statistics condense data into a simpler summary, they enable municipal decision-makers to assess specific populations in a more manageable form. The standard deviation was made to discover how respondents responded to questions vary or deviate from the mean (Quinn & Keough 2002).
The thematic analysis was used to analyse qualitative data from the interview schedules and also one open-ended question from questionnaires that eventually provided an opportunity to identify themes used for the discussions in this study (Charmaz 2009). The research instruments were informed by the research study’s objectives and attempts to gather elaborative answers to the research questions. Interviews and questionnaires were crafted using research tools theory (Quad 2016).
Ethical considerations
The researchers were granted ethical clearance by the Nelson Mandela University’s Research Ethics Committee (No. H/21/HUM/PML–001). to conduct research in the NMBM and BCMM. During the collection of data, the researcher was guided by the Protection of Personal Information Act, 2013 (Act No 41 of 2013) to maintain the anonymity and the confidentiality of the respondents throughout the study. This Act aims to protect the personal information processed by public and private bodies including local authorities (Netshakhuma 2020).
Results and discussion
The dimension to investigate the impact of COVID-19 on service delivery during the pandemic was established to address the objective of the study. This dimension deals with the responses of employees when asked to respond to the opinion statement to assess the impact of COVID-19 on service delivery during the pandemic. Table 1 summarises the scoring patterns.
| TABLE 1: The impact of coronavirus disease 2019 on service delivery during the pandemic. |
Table 1 reveals that 40% of respondents agreed that the municipality continuously conducts impact evaluations of COVID-19 compared with 24% who disagreed and 26% who were undecided. Despite the high percentage of respondents who agreed, the high percentage of disagreement and undecided is noticed. This finding is in support of the World Health Organization (WHO 2020), which developed M&E systems and suggested that the impact of the COVID-19 pandemic must be monitored and evaluated.
The mean score values displayed in the sub-dimension – that the municipality continuously conducts impact evaluations of COVID-19 – reflect that on a scale from 1 to 5, the respondents were between 30 538 and the standard deviation was 115 656. This indicates that a high proportion of municipal employees ranged from ‘agreed’ to ‘undecided’ on statements relating to this sub-dimension. These averages reflect more room for improvement regarding the municipality continuously conducting impact evaluations of COVID-19.
The qualitative findings were confirmed by a male senior manager in the health and public safety services directorate, who had this to say:
‘I’m thinking of the projects of the institution, which are in the city manager’s office and whether [they] are progressing or not progressing on that is measured by the city manager’s office. So it is [the] city manager’s office [which] would be able to see … the impact, but it is an assumption; it has not been tested, but I remember that there was a total shutdown. I know then from our side here, that people were not at work, even the service delivery departments were not at work up until such time that they came and the regulations talks [about] essential services. Then [the] institution [has] to identify those that are called essential services in terms of COVID-19, [and those that are] not essential services in terms of [the] Labour Relations Act.’
This finding is consistent with the study conducted by White (2009) who found that there is confusion concerning two rather different understandings of what impact evaluation might involve. Impact evaluation points out the importance of the strategy or initiative to assess the relevance of the goals (Awasthi 2020). It is noticed that the COVID-19 pandemic had an impact on citizens and municipalities received special funding. As a result, the public wants to be prioritised and is starting to demand more service than at any other time.
Table 1 demonstrates that 49% of respondents agreed that these municipalities do analyse the impact of how COVID-19 has affected the municipal finances, compared with 18% who disagreed and 33% who were undecided. This finding is in support of the study by McCoy (2020) who concurs that COVID-19 has reduced revenue and many economic activities and caused exceptional health interventions. In addition to this, municipalities have experienced a variety of direct effects as a result of the COVID-19 pandemic. Moreover, the impact of the pandemic in South Africa, particularly concerning budget cuts, has negatively affected service delivery by municipalities (Tafeni & Mngomezulu 2020).
The mean score values displayed in the sub-dimension – that municipalities analyse the impact of how COVID-19 has affected the municipal finances – reflect that on a scale from 1 to 5, the respondents were between 33 548 and the standard deviation was 101 497. This indicates that a high proportion of municipal employees ranged from ‘agreed’ to ‘undecided’ on statements relating to this sub-dimension. These averages reflect a significant gap regarding municipalities analysing the impact of how COVID-19 has affected the municipal finances.
The qualitative findings were confirmed by a male senior manager in the economic development directorate:
‘The problem, though, with the onset of COVID is that we’ve had to find ourselves having to shift in terms of what it is that we’re focusing on and shift in terms of budget. Remember, everything that you do in the municipality, it’s backed by your budget. So we had to shift and move budgets around from our own. Remember, we did receive an additional budget we [had] to use from our own internal operating budgets to cater to the requirements of COVID-19. So that is what we had to do. We find that some of our operating budgets ended up suffering.’
There is insufficient empirical or scientific evidence that municipalities do analyse the impact of how COVID-19 has affected the municipal finances. The study of Mathiba (2020) tackles the topical issue of corruption in the public procurement process during the COVID-19 crisis in South Africa, which impacted the service delivery in municipalities. As a result of COVID-19, the timing for the IDP, BEPP and budget approvals had to be postponed for a month. This was to resolve concerns surrounding enforcement and the processing of details that would affect strategic preparation and align with current meeting and attendance practices (NMBM 2020).
Table 1 shows that 43% of respondents agreed that the municipality makes a follow-up analysis of the impact of projects in specific communities affected by the COVID-19 crisis, compared with 23% who agreed and 34% who were undecided. The mean score values displayed in the sub-dimension – that the municipality makes a follow-up analysis of the impact of projects in specific communities affected by the COVID-19 crisis – reflect that on a scale from 1 to 5, the respondents were between 32 097 and the standard deviation was 106 751. This indicates that a high proportion of municipal employees ranged from ‘agree’ to ‘undecided’ on statements relating to this sub-dimension. This average reflects a significant gap in these municipalities.
This finding is consistent with this study’s finding that confirms the report published by the South African Monitoring and Evaluation Association (SAMEA 2013), concluding that South African municipalities are seemingly still facing challenges in finding ways on how to assess the outcomes and impact of municipal projects and programmes. Furthermore, the municipalities are still focusing on outputs rather than on outcomes, which is supposed to be the key focus of M&E. The study revealed that municipal officials have limited space or capacity to measure outcomes and the impact of projects.
Table 1 illustrates that 42% of respondents agreed that the municipality monitors have plans to improve service delivery and work towards achieving its goals beyond COVID-19, compared with 23% who disagreed and 36% who were undecided. This finding is consistent with the study conducted by Sebake and Mkhonza (2020) who consider M&E to be tools to monitor projects and evaluate service delivery successes and failures in South African municipalities.
The mean score values displayed in the sub-dimension – that municipality monitors have plans to improve the service delivery and work towards achieving its goals beyond COVID-19 – reflect that on a scale from 1 to 5, the respondents were between 32 151 and the standard deviation was 101 182. This indicates that a high proportion of municipal employees ranged from ‘agreed’ to ‘undecided’ on statements relating to this sub-dimension. These averages reflect a significant shortfall in these municipalities. This finding supports the empirical finding of the research study by Mantzaris and Ngcamu (2020) who concur that challenges arising from the procurement of services and equipment during the COVID-19 pandemic are because of service delivery inefficiencies.
Table 1 demonstrates that 36% of respondents agreed that the municipality has rendered good-quality service delivery to communities during the COVID-19 crisis, compared with 34% who disagreed and 30% who were undecided. This is consistent with the study conducted by Fraser and Morkel (2020) who conclude that municipalities in South Africa have been in the spotlight for years for their shockingly inadequate service delivery to residents.
The mean score values displayed in the sub-dimension – that the municipality has rendered good-quality service delivery to the communities during the COVID-19 crisis – reflect that on a scale from 1 to 5, the respondents were between 30 376 and the standard deviation was 114 074. This indicates that a high proportion of municipal employees disagreed and others were undecided on statements relating to this sub-dimension. These averages reflect that municipal employees do not have confidence in the services they deliver to communities. This finding confirms the results of the study conducted by Fraser and Morkel (2020) who concur that the recurrent service delivery protests could be an indication that the levels of performance of government, as evaluated according to citizen satisfaction, remain wanting.
Table 1 presents a low 39% of respondents who agree that the municipality always assesses the outcomes and impact of projects and programmes, compared to 27% who disagreed and 34% who were undecided. This finding is consistent with the study conducted by SAMEA (2013), which reported that South African municipalities appear to still be facing challenges in finding ways on how to assess the outcomes and impact of their projects and programmes.
The mean score values displayed in sub-dimension – that municipality always assesses the outcomes and impact of projects and programmes – reflect that on a scale from 1 to 5, the respondents were between 31 237 and the standard deviation was 105 563. This indicates that a high proportion of municipal employees ranged from ‘agreed’ to ‘undecided’ on statements relating to this sub-dimension. These averages reflect that municipal employees have low confidence of the evaluation of outcomes and the impact of their projects and programmes.
This finding is supportive of the study conducted by numerous authors such as Mackay (2006), Xue et al. (2013) and Mkama (2017). These authors concur that there is a need for a system that focuses on the core aspects of impact management and outcomes that are aligned with programme goals. Outcomes and impacts are important result levels and the levels of achievement must continually remain on the radar of development work managers.
The results and discussion in this article show the Disruptive theory and ToC; log frame processes inform the impact evaluation of COVID-19 on service delivery in South African municipalities. Taking into account the assumption that the South African municipalities have enough resources and conditions in municipalities are favourable to implement impact evaluation appropriately. When the ToC is developed strategies that are required to reach the long-term goal are defined and assumptions are articulated based on best practices (Nkonki-Mandleni 2020). For instance, amicable understanding between the employer and employees needs to be informed of impact evaluation of COVID-19 on service delivery and employees be well trained on how the impact evaluation should be implemented. Municipal employees also need to be educated on the benefits of its use for both the organisation and employees. In addition, all stakeholders in the two selected municipalities should be made aware of their roles and responsibilities. To this end, they should effectively execute those roles and responsibilities without any interference in order to meet the service delivery objectives. Lastly, the policy landscape or environment in the two selected municipalities need to be conducive to supporting the implementation of impact evaluation.
Conclusion
This dimension investigated the impact of COVID-19 on service delivery during the pandemic. The study arrived at the conclusion that employees are not confident that the municipality continuously conducts impact evaluations of COVID-19. These municipalities have not shown commitment to analyse the impacts of how COVID-19 has affected the municipal finances. In these organisations, employees are not confident that municipalities make a follow-up analysis on the impact of projects in specific communities affected by the COVID-19 crisis. This finding demonstrates that outcomes and impact evaluations are not important in these municipalities and the absence of M&E continually remains no one’s responsibility. Again, there is no confidence in these municipalities that monitors plan to improve service delivery and will work towards achieving the goals of the organisation beyond COVID-19. In addition, it is troubling that employees do not believe that the municipality has not always rendered good-quality service delivery to communities during the COVID-19 crisis. This finding demonstrates the leadership failure in these municipalities to set the tone of what should have been proper and appropriate during the COVID-19 pandemic. This problem will not be resolved without addressing the quality of leadership and those in charge of these municipalities.
These findings tackle the topical issue that have had an impact on service delivery in municipalities during the COVID-19 crisis in South Africa. The quest for effective M&E tools needs to be further explored as the COVID-19 pandemic is still a threat. Furthermore, the municipalities could undertake to minimise the gaps and limits in its assessment of the pandemic and to reduce the risks associated with the pandemic.
Recommendations
The following recommendations are derived from the findings of the study on this dimension. The municipal manager should start focusing on outcomes, which is supposed to be the key focus of M&E systems, and change the attitudes of line managers not seeing impact evaluations on COVID-19 as an extended function of their job. Monitoring and evaluation managers in these municipalities need to assist on the political side of M&E in terms of addressing or implementing the outcomes and impact of projects. Human resources should provide training for leadership in order to operate in a manner consistent with the principles of section 195 of South Africa’s Constitution. This study further recommends that future research should explore an analysis of the impacts of M&E systems on COVID-19 projects in communities within these municipalities.
Acknowledgements
Competing interests
The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.
Authors’ contributions
B.Y., S.B.N. and S.P. contributed equally to this work.
Funding information
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Data availability
The data that support the findings of this study are available on request from the corresponding author, B.Y., upon reasonable request.
Disclaimer
The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.
References
Adams, V.M., Barnes, M. & Pressey, R.L., 2019, ‘Shortfalls in conservation evidence: Moving from ecological effects of interventions to policy evaluation’, One Earth 1(1), 62–75. https://doi.org/10.1016/j.oneear.2019.08.017
Awasthi, S., 2020, A robust monitoring & evaluation system can prove effective during pandemics, viewed 24 July 2020, from https://www.outlookindia.com/website/story/opinion-a-robust-monitoring-evaluation-system-can-prove-effective-during-pandemics/354701.
Auditor-General, 2020, ‘Second Special Report on Financial Management of Government’s Covid-19 Initiatives’, AGSA, Pretoria, viewed 26 July 2022, from https://www.agsa.co.za/Portals/0/Reports/Special%20Reports/Covid19%20Special%20report/Second%20special%20report%20on%20financial%20management%20of%20government’s%20Covid19%20inititatives%20-%20FINAL%20PDF%20(interactive).pdf.
Ball, I., 2020, ‘Reflections on public financial management in the Covid-19 pandemic’, Journal of Accounting & Organizational Change 16(4), 655–662. https://doi.org/10.1108/JAOC-10-2020-0160
Bandalos, D.L., 2018, Measurement theory and applications for the social sciences, Guilford Press, London.
Charmaz, K., 2009, Grounded theory: The SAGE encyclopedia of social science research methods, Sage, Philadelphia, PA.
Creswell, J.W. & Creswell, J.D., 2018, Research design: Quantitative and mixed method approaches, 5th edn., Sage, Los Angeles, CA.
De Coning, R., 2020, ‘South Africa’s national response to COVID-19, with specific reference to sub-level governance in provinces’, Africa Journal of Public Sector Development and Governance 3(1), 74–89. https://doi.org/10.55390/ajpsdg.2020.3.1.4
Department of Planning, Monitoring and Evaluation (DPME), 2014, Guideline on impact evaluation, evaluation guideline no 2.2.13, viewed 28 December 2020, from https://evaluations.dpme.gov.za/images/gallery/Guideline%202.2.13%20Impact%20Evaluation%2014%2003%2020%20%20(3).pdf.
Department of Planning, Monitoring and Evaluation (DPME), 2020a, Progress in the national effort to contain the covid-19 pandemic, Department of the Planning, Monitoring and Evaluation, Pretoria, viewed 28 December 2020, from https://www.dpme.gov.za/Pages/default.aspx.
Department of Planning, Monitoring and Evaluation (DPME), 2020b, The impact of COVID-19 on South African municipalities: Survey results, Pretoria, viewed 08 November 2021, from https://www.dpme.gov.za/publications/Reports%20and%20Other%20Information%20Products/DPME_COVID-19%20Municipal%20Survey%20Results_June%202020.pdf.
Department of Planning, Monitoring and Evaluation (DPME), 2022, Integrating a transformative equity criterion into evaluations for promoting transformative systemic change, viewed 16 July 2022, from https://evaluations.dpme.gov.za/pages/guidelines-other-resources.
Eastern Cape Socio Economic Consultative Council, 2021, Economic review of the Eastern Cape, viewed 19 March 2022, from https://www.ecsecc.org/datarepository/documents/ecsecc-q4-reportmar-221_1k4M6.pdf.
Fani, N., 2020, Municipal response to Covid-19 II, presentation at connective cities virtual event II, Provision of Municipal Services, viewed 10 July 2021, from https://www.connective-cities.net/en/details/veranstaltung/municipal-response-to-covid-19-pandemic-ii.
Ferraro, P.J., 2009, ‘Counterfactual thinking and impact evaluation in environmental policy’, New Directions for Evaluation 2009(122), 75–84. https://doi.org/10.1002/ev.297
Fraser, D.I. & Morkel, C., 2020, ‘State of monitoring and evaluation in Anglophone Africa: Centre for Learning on Evaluation and Results in Anglophone Africa’s reflections’, African Evaluation Journal 8(1), a505. https://doi.org/10.4102/aej.v8i1.505
Gaines, C., 2020, Monitoring and evaluation tools, Sopact, viewed 03 May 2020, from https://www.sopact.com/monitoring-and-evaluation-tools.
Gertler, P.J., Martinez, S., Premand, P., Rawlings, L.B. & Vermeersch, C.M.J., 2016, Impact evaluation in practice, 2nd ed., Inter-American Development Bank and World Bank, Washington, DC.
Guest, G., Namey, E. & Chen, M., 2020, ‘A simple method to assess and report thematic saturation in qualitative research’, PLoS One 15(5), e0232076. https://doi.org/10.1371/journal.pone.0232076
Gumede, W., 2020, Municipalities have to clean up their acts. If the don’t post COVID SA will explode, Wits University Opinion Piece, 23 Septermber 2020, Sunday Times, Johannesburg.
Gxabuza, F.G. & Nzewi, O., 2021, ‘Findings on the role of cooperatives in local economic development in Buffalo City Metropolitan Municipality: A participatory monitoring and evaluation approach’, Africa’s Public Service Delivery and Performance Review 9(1), a387. https://doi.org/10.4102/apsdpr.v9i1.387
Hes, D., 2020, Public-private partnerships essential for economic recovery. The impact of Covid-19 on the business of local municipalities, viewed 10 October 2022, from www.businessmediamags.co.za.
Helstrom, T. & Helstrom, C., 2018, ‘Achieving impact: Impact evaluations and narrative simplification’, Prometheus 35(3), 215–230. https://doi.org/10.1080/08109028.2018.1522829
Imas, L.G.M. & Rist, R.C., 2009, The road to results: Designing and conducting effective development evaluations, World Bank, Washington, DC.
Independent Evaluation Group (IEG), 2005, OED and impact evaluation: A discussion note, Operations Evaluation Department, World Bank, Washington, DC.
International Labour Organisation (ILO), 2020, COVID-19 and the world of work: Country policy responses, viewed 10 October 2022, from https://www.ilo.org/budapest/WCMS_741143/lang--en/index.htm.
Jain, R., Budlender, J., Zizzamia, R. & Bassier, I., 2020, The labour market and povert impacts of COVID-19 in South Africa, National Income Dynamics Study – Coronavirus Rapid Mobile Survey (CRAM), viewed from https://cramsurvey.org/reports/.
Jensen, E., 2011, ‘The problems with science communication evaluation’, Journal of Science Communication 13(01), C04, viewed 9 November 2022, https://doi.org/10.22323/2.13010304
Kaur, P., Stoltzfus, J. & Yellapu, V., 2018, ‘Descriptive statistics’, International Journal of Academic Medicine 4, 60–63. https://doi.org/10.4103/IJAM.IJAM_7_18
King, H., Steiner, K., Hobson, M.A.R. & Clipson, H., 2015, ‘Highlighting the value of evidence-based evaluation: Pushing back on demands for “impact”’, Journal of Science Communication 14(02), A02. https://doi.org/10.22323/2.14020202
Kariuki, P. & Reddy, P., 2017, Operationalising an effective monitoring and evaluation system for local government: Considerations for best practice, viewed 16 January 2020, from https://creativecommons.org/licenses/by/4.0/.
Kwandayi, C., Makanyeza, H. & Ikobe, B., 2013, ‘Strategies to improve service delivery in local authorities’, International Journal of Information Technology and Business Management 15(1), 1–11.
Leedy, P.D. & Ormrod, J.E., 2021, Practical research: Planning and design, 12th edn., Pearson Education, Harlow.
Leeuw, F. & Vaessen, J., 2009, Impact evaluations and development: NONIE guidance on impact evaluation, The World Bank, Washington, DC.
Mackay, K., 2006, Institutionalization of monitoring and evaluation systems to improve public sector management, Evaluation Capacity Development Working Paper Series, No. 15, The World Bank, Washington, DC.
Mahlala, M. & Netswera, F.G., 2020, ‘Steering public service administration in South Africa through COVID-19 and related corruption guided by the triple challenges of governance’, Journal of Public Administration 55(31), 480–489.
Mantzaris, E. & Ngcamu, B.S., 2020, ‘Municipal corruption in the era of a COVID-19 pandemic: Four South African case studies’, Journal of Public Administration 55(3/1), 461–479.
Mathiba, G. 2020, ‘Corruption, public sector procurement and COVID-19 in South Africa: Negotiating the new normal’, Journal of Public Administration 55(4), 642–661.
Mbava, N.P. & Rabbie, B., 2018, ‘Strengthening impact evaluations in the South African National evaluation system’, Administratio Publica 26(4), 74–98.
McCoy, D., 2020, What exactly is the government’s coronavirus strategy?, Queen Mary Global Policy Institute, Queen Mary University of London, London, viewed 23 March 2021, from https://www.qmul.ac.uk/media/news/2020/pr/what-exactly-isthegovernmentscoronavirus-strategy.html.
Mkama, W., 2017, Evolution of natural resource programs performance management in Tanzania: A review of monitoring, evaluation and learning (MEL) aspects and policy recommendations, Tanzania Natural Resource Forum, Arusha, viewed 21 February 2021, from https://www.tnrf.org/files/pubs/project_performance_mgt_tanzania.pdf.
Mkhize, Z., 2020, ‘The COVID-19 pandemic in South Africa: EARLY lessons for public governance’, in P. Joice, F. Maron & P.S. Reddy (eds.), Good public governance, IIAS Public Governance Series, Brussels. The International Institute of Administrative Sciences, 79–83. https://doi.org/10.46996/pgs
Narsiah, S., 2021, ‘Local government and the politics of service delivery in South Africa in the context of COVID-19’, Democracy Development Programme 1, 189–212.
Nel, V. & Lewis, M., 2022, ‘The South African Local Government and municipal planning responses to COVID-19’, in N. Silver (ed.), Local government and the COVID-19 pandemic, pp. 667–686, Springer, Cham.
Nelson Mandela Bay Municipality (NMBM), 2020, Annual report for 2018/2019 financial year, viewed 03 July 2020, from https://www.nelsonmandelabay.gov.za/DataRepository/Documents/nmbmdraft-2018-19-annual-report_5Sq85.pdf.
Netshakhuma, N.S., 2020, ‘Assessment of a South Africa national consultative workshop on the Protection of Personal Information Act (POPIA)’, Global Knowledge, Memory and Communication 69(1/2), 58–74. https://doi.org/10.1108/GKMC-02-2019-0026
Ngcaweni, B., 2020, ‘Leaping forward: Preliminary thoughts on COVID-19 and shifts in public administration’, Journal of Public Administration 55(4), 608–620.
Nkonki-Mandleni, B., 2020, ‘Monitoring and evaluation for university–community impact in driving transformation agenda’, South African Journal of Higher Education 37(1), 151–165. https://doi.org/10.20853/37-1-5679
Owen, J. & Rogers, P.J., 1999, ‘Impact evaluation’, in J.M. Owen & P.J. Rogers (eds.), Program evaluation, pp. 263–308, Sage, London.
Parliamentary Monitoring Group (PMG), 2020a, Role of local government in combating spread of COVID-19 virus: SALGA briefing, viewed 04 October 2021, from https://pmg.org.za/page/RoleoflocalgovernmentincombatingspreadofCOVID-19virusSALGA20%briefing.
Parliamentary Monitoring Group (PMG), 2020b, COVID-19 initiatives on water, sanitation & de-densification: Briefing by Minister and DHS, Department of Human Settlement Water and Sanitation, viewed 04 October 2021, from https://pmg.org.za/committee-meeting/30104.
Patrizi, P.A. & Patton, Q.M., 2010, ‘Strategy as the focus for evaluation’, New Directions for Evaluation 2010(128), 5–28. https://doi.org/10.1002/ev.343
Pietersen, J.M., 2020, ‘Possible implications of COVID-19 for the conditions of service in the public sector’, Journal of Public Administration 55(3/1), 435–449. https://doi.org/10.55390/ajpsdg.2020.3.1.2
Quad, A.D.J.P., 2016, ‘Research tools: Interviews & questionnaires’, in Research methodology in education, viewed 20 October 2021, from https://lled500.trubox.ca/2016/225.
Quinn, G.P. & Keough, M.K., 2002, Experimental design and data analysis for biologists, Cambridge University Press, Cambridge.
Rachidi, M.F., 2020, ‘Implications of the COVID-19 pandemic on management of human capital’, Journal of Public Administration 54(4), 668–700.
Raylor, M.E., 2011, ‘Disruption theory as a predictor of innovation success/failure’, Strategy & Leadership 39(4), 27–30. https://doi.org/10.1108/10878571111147378
Reed, M.S., Ferre, M., Martin-Ortega, J., Blanchec, R., Lawford-Rolfed, R., Dallimerb, M. et al., 2021, ‘Evaluating impact from research: A methodological framework’, Research Policy 50(4), 104147. https://doi.org/10.1016/j.respol.2020.104147
Republic of South Africa, 1996, The Constitution of the Republic of South Africa, 1996 (Act No. 106 of 1996), Government Printers, Pretoria.
Republic of South Africa, 2007, Policy framework for the Government-Wide Monitoring and Evaluation System, (GWMES), National Treasury, Pretoria, viewed 22 June 2021, from https://www.dpme.gov.za/publications/Policy%20Framework/Policy%20Framework%20for%20the%20GWME%20system.pdf.
Republic of South Africa, 2011, National evaluation policy framework, The Presidency, Department of Performance Monitoring and Evaluation, Pretoria, viewed 22 June 2021, from https://www.gov.za/sites/default/files/Evaluation_Policy_Framework%20approved%2011%2011%2023a.pdf.
Sebake, B.K. & Mukonza, R.M., 2020, ‘Integrated development plan, monitoring and evaluation in the City of Tshwane: A confluence question for optimising service delivery’, Journal of Public Administration 55(3), 342–352.
Sekaran, U., 2003, Research methods for business: A skill-building approach, 4th edn., John Wiley & Sons, New York, NY.
South African Local Government Association, 2020, The municipal support and intervention model, COGTA, viewed 12 December 2020, from www.salga.org.za.
South African Monitoring and Evaluation Association (SAMEA), 2013, 4th Biennial SAMEA Conference: September conference materials, viewed 18 July 2020, from http://www.samea.org.za/index.php?module=pagesetter&type=user&func=viewpub&tid=4&pid=59.
Spaull, N., Ardigton, C., Bassier, I., Bhorat, H., Bridgman, G., Brophy, T. et al., 2020, NIDS-CRAM Wave 1 systhesis report: Overview and findings, viewed 25 July 2022, from https://cramsurvey.org/report/.
Taber, K.S., 2018, ‘The use of Cronbach’s alpha when developing and reporting research instruments in science education’, Research in Science Education 48, 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Tafeni, Z. & Mngomezulu, B.R., 2020, ‘A perspective on the budgetary impact of COVID-19 in government departments with specific reference to the Department of Cooperative Governance & Traditional Affairs, Eastern Cape province, South Africa’, Journal of Public Administration 54(4), 599–607.
Thornton, J.A., Harding, W.R., Slawski, T.M. & Lin, H., 2022, ‘Monitoring and evaluation: The foundation for lake and reservoir management’, Earth 3(1), 115–124. https://doi.org/10.3390/earth3010008
Uwizeyimana, D.E., 2020, ‘The logframe as a monitoring and evaluation tool for government interventions in a chaotic and complex environment’, Africa’s Public Service Delivery and Performance Review 8(1), a328. https://doi.org/10.4102/apsdpr.v8i1.328.
Vaska, S., Massaro, M., Bagarotto, E.M. & Dal Mas, F., 2021, ‘The digital transformation of business model innovation: A structured literature review’, Frontiers in Psychology 11, 539363. https://doi.org/10.3389/fpsyg.2020.539363
Visagie, J. & Turok, I., 2021, ‘Rural-urban inequalities amplified by COVID-19: Evidence from South Africa’, Area Development and Policy 6(1), 50–62. https://doi.org/10.1080/23792949.2020.1851143
Weiss, R.S., 1998, ‘A taxonomy of relationships’, Journal of Social and Personal Relationships 15(5), 671–683. https://doi.org/10.1177/0265407598155006
White, H. & Raitzer, D.A., 2017, Impact evaluation of development interventions: A practical guide, Asian Development Bank, Philippines.
White, P., 2009, ‘Stedman’s Medical Dictionary for the health professions and nursing’, Reference Reviews, 23(1), 47–48.
Woodhill, J., 2007, ‘M&E as learning: Rethinking the dominant paradigm’, in J. De Graaff, J. Cameron, S. Sombatpanit, C. Pieri & J. Woodhill (eds.), Monitoring and evaluation of soil conservation and watershed development projects, viewed 09 January 2020, from http://www.capfida.mg/km/atelier/wageningen/download.
World Health Organization, 2020, Considerations for mass gatherings in the context of COVID-19: Annex: Considerations in adjusting public health and social measures in the context of COVID-19, viewed 20 December 2020, from https://www.who.int/publications/i/item/considerations-for-mass-gatherings-inthe-context-ofcovid-19-annex-considerations-in-adjusting-public-health-andsocial-measures-in-the-contextof-covid-19.
Wróblewska, M.N., 2021, ‘Research impact evaluation and academic discourse’, Humanities and Social Sciences Communications 8(1), 1–12. https://doi.org/10.1057/s41599-021-00727-8
Xue, Y., Turner, J.R., Lecoeuvre, L. & Anbari, F., 2013, ‘Using results-based monitoring and evaluation to deliver results on key infrastructure projects in China’, Global Business Perspectives 1(2), 85–105. https://doi.org/10.1007/s40196-012-0002-3
|
Hide
Show all
Email this article (Login required)
