A Critical Analysis of Regional Program Evaluation Practice in Kazakhstan

In recent years the development of public program evaluation has received growing attention in Kazakhstan. An institutional and legal base for program evaluation has been established. However, the examination of literature has evidenced that there have been rudimentary attempts to interrogate evaluation practice, particularly at the regional level. It is still not well known how effective or valuable it is. It is imperative to run a diagnostic and assess the evaluation system to answer this question. This article aims to evaluate the quality of regional program evaluation practice in Kazakhstan. It applies a meta-evaluation tool to understand the extent to which such practice complies with three fundamental and recognized evaluation standards: namely, value, validity, and utility. As a sample, the study used evaluation reports conducted by regional Audit commissions. This research is the first attempt to apply established evaluation standards to the Kazakhstani context. Therefore, it was assumed that some discrepancies with the standards may occur. Having confirmed this hypothesis, the findings indicate that regional program evaluation falls far short of these standards. The paper identified many conceptual and methodological problems, which seriously compromise the validity and soundness of evaluation practice. It is expected that it will stimulate discussion in academic and subject matter expert circles. Furthermore, having identified key areas for improvement, the study may help reform the evaluation field and contribute to better policy- and decisionmaking, thus saving taxpayers’ money and improving people’s wellbeing. In the end, the research put forward several recommendations for strengthening evaluation practice.

1. Legal information system of Regulatory Legal Acts of the Republic of Kazakhstan (Adilet), (2020). https://adilet.zan.kz/

2. Akorda, (2020). The Address of the President of the President of the Republic of Kazakhstan at the government meeting [akorda. kz]. Official website of the President of Kazakhstan; 2020. [updated January 24, 2020; cited November 2, 2021]. Available: https://www.akorda.kz/ru/speeches/internal_political_affairs/in_speeches_and_addresses/vystuplenie-prezidenta-kasym-zhomarta-tokaevana-rasshirennom-zasedanii-pravitelstva-respublikikazahstan.

3. Alibekova, B., Akimova, B. & Ahmetova A. (2019). The formation and current state of audit commissions in the system of state audit bodies]. Statistics, accounting and audit [Statistika, uchet i audit], 1(72), 16-21. (in Russ)

4. Alibekova, G., Tayova, G. & Ilmaliyev, J. (2018). Evaluation issues of support programs of commercialization of scientific development in Kazakhstan. Complex use of mineral resources [Kompleksnoe ispolzovanie mineralnogo syrya], 4(307), 181-191. (in Russ)

5. Aymagambetov, E. & Tyngisheva, A. (2019). Modern state of the healthcare system in Karaganda region. Karaganda University Bulletin [Vestnik KarGU], 2, 11-17. (in Russ)

6. Baimbetov, M. (2019). Implementation evaluation of state programs in the Republic of Kazakhstan. Bulletin of the Omsk Regional Institute [Vestnik Omskogo regionalnogo instituta], 3, 43-46. (in Russ)

7. Bopiyeva, Z. & Kazakova, T. (2009). Kazakhstan’s experience in impact evaluation of state programs. Vestnik Instituta zakonodatelstva Respubliki Kazahstan [Bulletin of Legislature Institute of the Republic of Kazakhstan], 2(14), 20-27. (in Russ)

8. Bopiyeva, Z. & Kazakova. T. (2009). Ways of improving effectiveness of state programs]. Bulletin of the Legislation Institute of the Republic of Kazakhstan [Vestnik Instituta zakonodatelstva Respubliki Kazahstan], 4(16), 160-164. (in Russ)

9. Chelimsky, E. (1985). Comparing and contrasting auditing and evaluation: some notes on their relationship. Evaluation Review, 9(4), 483-503. https://doi.org/10.1177/0193841X8500900406.

10. Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Sage.

11. Dosayeva, A. (2019). Principles, approaches and features of state audit. Bulletin of Turan University [Vestnik Universiteta Turan], 4(84), 149-154. (in Russ).

12. Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Evaluation, 1995(68), 15-32. https://doi.org/10.1002/ev.1017.

13. Gayfutdinova, V. (2017). Why development programs don’t work in Kazakhstan [Pochemu v Kazahstane ne rabotayut programmyi razvitiya], [updated November 8, 2017; cited December 10, 2021]. Available: https://forbes.kz//process/expertise/pochemu_v_rk_ne_rabotayut_programmyi_razvitiya_ot_zarubejnyih_konsultantov/ (in Russ)

14. Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The qualitative report, 8(4), 597-607. https://doi.org/10.46743/2160-3715/2003.1870

15. Ho, J. Y., & Hendi, A. S. (2018). Recent trends in life expectancy across high income countries: retrospective observational study. Bmj, 362. https://doi.org/10.1136/bmj.k3622

16. Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards: a guide for evaluators and evaluation users.

17. Kaldiyarov, D. & Turgambekova, Z. (2019). Effective methods for designing and evaluation state program in the social sphere. Modern Science [Sovremennaya nauka], 5(1), 107-109. (in Russ)

18. Kari, A. (2015). Methodological problems of implementation evaluation of strategic documents and programs in public management of economy in the Republic of Kazakhstan. Economics: the strategy and practice [Ekonomika: strategiya i praktika], 3(35). (in Russ).

19. Martens, K. S. (2018). Rubrics in program evaluation. Evaluation Journal of Australasia, 18(1), 21-44. https://doi.org/10.1177/1035719X17753961

20. Murzaliyeva, A. & Karshalova, A. (2018). State effectiveness audit of implementation of healthcare budget programs. Economic systems management: electronic journal [Upravleniyeekonomicheskimi sistemami: elektronnyi jurnal], 4(110), 32. (in Russ)

21. Nygmetov, K. (2014). Strategy and program evaluation in the system of state planning in the Republic of Kazakhstan. Economics and management [Ekonomika i Upravleniye], 34(547), 95-102. (in Russ)

22. The Organization for Economic Cooperation and Development (OECD). OECD Ilibrary. (2021) [updated February 13, 2021; cited March 30, 2022]. Available: https://www.oecd-ilibrary.org/

23. Open Government, Magenta Book. Central Government guidance on evaluation, (2020). [updated 2020; cited March 30, 2022]. Available: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/879438/HMT_Magenta_Book.pdf

24. Pokidayev, D. (2020). Who will be conducting evaluation of effectiveness of state programs [Kto zaymetsya otsenkoy effektivnosti kazahstanskih gosprogramm], [updated February 7, 2020; cited December 2021]. Available: https://kursiv.kz/news/obschestvo/2020-02/kto-zaymetsya-ocenkoyeffektivnosti-kazakhstanskikh-gosprogramm. (in Russ)

25. Pritvorova, T. & Bektleyeva, D. (2017). Social effect evaluation methods in active labour market projects in Kazakhstan. Herald of Omsk University. Economics Series [Vestnik Omskogo universiteta. Seriya Ekonomika], 3(59), 201-210. (in Russ)

26. Rakhmatullayeva, D., Bobkov, V. & Jatkanbayev, E. (2015). Modelling of social effects of direct foreign investments in regions of Kazakhstan. Economy of Region [Ekonomika regiona], 2(42), 285-300. (in Russ)

27. Scott-Little, C., Hamann, M. S., & Jurs, S. G. (2002). Evaluations of after-school programs: A metaevaluation of methodologies and narrative synthesis of findings. American Journal of Evaluation, 23(4), 387-419. https://doi.org/10.1177/109821400202300403.

28. Scriven, M. (1991). Evaluation thesaurus (4th ed.). California: Sage Publications.

29. Scriven, M. (1995). The logic of evaluation and evaluation practice. New Directions for Evaluation, 1995(68), 49-70. https://doi.org/10.1002/ev.1019.

30. Shakirova, G., Sitenko, D. & Vasa, L. (2019). Effectiveness audit of public budget expenditures in the process of state audit. Karaganda University Bulletin [Vestnik KarGU], 7562, 263-274. (in Russ).

31. Stufflebeam, D. L., & Coryn, C. L. (2014). Evaluation theory, models, and applications (Vol. 50). John Wiley & Sons.

32. Vechkinzova, Y. (2008). Evaluation of implementation of state development programs in the Republic of Kazakhstan. Journal of New Economy [Izvestiya UrGEU], 1(20), 105-111. (in Russ)

33. Yessimova, S. (2009). Impact evaluation of state programs in the Republic of Kazakhstan. University Scientific papers [Universiteski naukovi zapiski], 3, 257-260. (in Russ)

34. Youker, B. & Ingraham, A. (2014). Goal-free evaluation: an orientation for foundations’ evaluations. The Foundation Review, 5(4). 51-61. https://doi.org/10.9707/1944-5660.1182