Mission impossible dph day 2012
Download as PPTX, PDF
1 like211 views
This document discusses capacity building and ways to measure its effects. It notes that capacity building is complex and not fully captured by current measurement instruments. Several frameworks for understanding capacity building are presented, including one that views it as a hierarchy with technical tools and skills at the bottom and broader systemic and institutional changes at the top. Different methods for evaluating capacity building are also discussed, including outcome mapping and most significant change technique. The document argues that evaluation should look not just at what activities are done but also what influence they have in driving change. It suggests introducing a more mixed evaluation approach using both quantitative and qualitative methods to better reflect the complexity of capacity building work.
Mission impossible dph day 2012
- 1. Measuring capacity building: mission impossible? Tine Verdonck, B辿n辿dicte Fonteneau (HIVA) and Bart Criel 1
- 2. The subject of our discussions Academic trias Research Teaching Service Delivery Service delivery: multiform Consultancy work overseas Variety of services delivered to (Belgian) general public, civil society organisations, public health authorities (cfr. steunpuntfunctie DGD) Capacity building via Institutional Collaborations (ICs) 2
- 3. Purpose of session Share views on place & importance of Capacity Building (CB) as academic activity Reflect on ways/methods how to measure its effect(s). Hypothesis: complexity of CB not properly captured by instruments currently used 3
- 4. Programme Intro (Bart): 10 Some data on our ICs (Tine): 10 Quick reactions: 5 Outcome Mapping (B辿n辿dicte): 20 Quick reactions: 5 Most Significant Change (Tine): 5 Discussion on sample of MSC stories collected (Bart and Tine): 30 Final thoughts (Bart): 5 4
- 5. Some history ITM from 1990 till today 1990: a small ENOV / URESP 2012: a big Public Health Training (ICHD): main project Department Critical mass Training (MPH HSMP-MDC): Research: mainly action- still important, but less research geared to change, central few publications Research: explosion in Service delivery (incl. capacity publications, action-research building) central in culture of marginalized unit: Service delivery (incl. capacity Societal relevance building) Relevance for decision- Has objectively strongly makers increased But less prominent in departmental culture and policies than in past 5
- 6. A contradiction in current ITM policy & discourse? De facto, we have never done as much CB as today But ITM discourse vv outside world: ITM is different from other academic institutions Internally: service delivery/capacity buidling is not a priority, not a core task, to be phased out, we are not an NGO 6
- 7. Evaluation / recognition academic work at ITM as of today Research (publications) +++, Teaching +/++, SD/CB +/- Our (departmental) weaknesses Is there consensus within department on importance of SD/CB as academic activity part enti竪re ? Voluntaristic discourse, but no instrumentarium / set of indicators to measure effect of SD/CB 7
- 8. Capacity Building ITM 2020 Working Group CB 8
- 9. What is capacity building about? Something to strive for Interwoven with research & training Long-term process 9
- 10. Dimensions of capacity building Scientific Managerial Institutional capacity capacity capacity Capacity Building 10 Adapted from Patrick Develtere, HIVA/KUL
- 11. Capacity building: now and in the future From rich to poor North TODAY From knowledgeable to less knowledgeable A finite process South South South FUTURE Alliances between equal partners Mutually beneficial North North Long lasting 11
- 12. Our message Powerful justification of ITM involvement in capacity building Ethical reasons: solidarity Pragmatic / opportunistic reasons: ITM needs strong partners & fields to do research Technical reasons: ITMs expertise in capacity building, health and health care (excellence?) ITM and capacity building after 2020? Probably (still) the case, here and there, but with highly variable outlook depending on context 12
- 13. A framework to analyse Capacity Building 13
- 14. Capacity Building Potter C & Brough R (2004) Systemic Capacity Building: a hierarchy of needs Health Policy & Planning; 19(5): 336-345 Definition? There is a need for CB is an as unuseful diagnosis as the patient is feeling unwell Consultant-speak Over-pompous synonym for training -> Lack of clarity 14
- 15. Capacity building (follow.) Potter & Brough (2004) Enable programme execution independent of changes of personalities, technologies, social structures and resource crises Develop sustainable and robust systems 15
- 16. Systems Capacity: a hierarchy of needs Potter & Brough (2004) Tools enable Skills effective require use of Staff & Infrastructure Structures, Systems and Roles Capacity Pyramid 16
- 17. Systems Capacity: a hierarchy of needs Potter & Brough (2004) Easier and more technical Tools Skills Staff & Infrastructure Harder and more socio- Structures, Systems and Roles cultural Time to implement change 17
- 18. Indicators of the Framework Agreement
- 19. Project cycle management - indicators How do we verify success? Describe the objectives in operationally measurable terms (quantity, quality, time) Basis of monitoring and evaluation system
- 20. Our indicators Framework Agreement Subprogramme: Institutional Collaboration Period: 2008-2010 Total 381; objectives 57 and results 314 Per project; range 10-61 and mean 25
- 21. Domain of FA indicators Research Training Services/GRIPP/Networking Management/Supporting services PI from Department Public Health PI from other Department
- 22. + output in numbers People Master and PhD scholarships awarded and degrees obtained Short course scholarships awarded and certificates issued Exchanges and missions ... Publications Publications in international peer-reviewed journals; other publications Theses (master and PhD) ... Products CD-ROM, digital atlas, web-based course, online-modules, website Patent: application or obtained; test/method developed ...
- 23. Publications Framework Agreement Sources of references: reports of FA 1, 2, and 3 Including only PubMed publications Removing duplicates 729 publications
- 24. Number of publications per year
- 25. Nr of publ. per year according to the affiliation address of the first author
- 26. Tag cloud of key words
- 27. Questions raised Relevance in partner countries? Who determines research agenda? Who does what? What happens if collaboration stops? Are publications a good indicator of capacity building?
- 28. Level of FA indicators in hierarchy Theoretical model Framework agreement (IC; FA3-I) Easier and more technical Tools Skills Staff & Infrastructure Harder and more socio- Structures, Systems and Roles cultural Time to implement change
- 29. Formulation of FA indicators: examples of activity done Quantit辿 et qualit辿 des ressources documentaires acquises par le centre de documentation Routine testing for second line drugs standardised (2012) Gene mutations related to resistance investigated (2013) Work-based health systems fellowship programme fully established One dissemination workshop/year for local MOH staff
- 30. Formulation of FA indicators: examples of changing influence Publications describing the positive impact of measures recommended by XXX to other national institutions Policy guidelines take into account results of studies conducted at XXX XXX attracts a cadre of nationally renowned faculty Level of satisfaction of the health service managers with alumni performance and initiatives has increased Le nombre de cadres gestionnaires form辿s XXX et exer巽ant des fonctions de gestion de haut niveau dans le syst竪me de sant辿
- 31. Formulation of FA indicators Activity done Changing influence
- 33. Most significant change An experiment
- 34. Steps 1. Collecting stories (done) 2. Selecting most significant of stories (now) 3. Feeding back results of selection (later)
- 35. Most Significant Change Technique What? Making sense of complex information; participatory; qualitative When to use? Complex participatory programmes; emergent outcomes; social change When not to use? Accountability reports, public relations, to capture expected change Characteristics? Identifies values; bridges cultures; analysis > data collection; rich picture
- 36. Best illustration of what we want to achieve? 1. PhD student learned to analyse data 2. Invitations to participate in consultancy & reflection 3. Launch of new training programme 4. PhD student involved in grant writing 5. Positive feedback on newsletter 6. PhD student submitted PhD proposal 7. Organization of new training programme 8. Inauguration alumni association 9. We got access to data on births and deaths 10. PhD student discussion about draft manuscript
- 37. MSC: group discussion Reading out of 10 stories recently collected Discussion Which story is a good / the best illustration of what we wish to achieve with CB? Why? Is this currently captured by the current evaluation instrumentarium that is used? 37
- 38. 38
- 39. Final thoughts 39
- 40. Systems Capacity: a hierarchy of needs Other PCM/ methods Easier and Logframes more (OM, MSC) technical Tools quant Skills Staff & Infrastructure Harder and more socio- Structures, Systems and Roles qualit cultural Time to implement change 40
- 41. Way forward? Evaluation questions: measure not only what we do (=production), but also what we induce (=change in processes) More room for soft things: values, trust, affect, self- confidence, As a department, familiarise with other methods that complement PCM/Logframe Introduce this enriched evaluation framework in forthcoming FA4 41
- 42. 42
- 43. Statements put forward to Director 5 statements on Service Delivery / Capacity Building Scaling 1 (strongly disagree) to 5 (strongly agree) by A & B 43
- 44. A: 3B: 3 Stelling 1. Dienstverlening aan ontwikkelingslanden - waaronder onze activiteiten mbt institutionele samenwerking en steunpuntfunctie - zit ingebakken in het DNA van het ITG en haar personeel. Het was/is bepalend in de keuze van velen om op het ITG te komen werken 44
- 45. A: 1B: 1 Stelling 2. Gegeven stelling 1, zou het van slecht personeelsbeleid getuigen om deze activiteiten niet als volwaardig te erkennen en te valoriseren. 45
- 46. A: 4B: 4 Stelling 3. De gangbare methodieken om dienstverlening/institutionele samenwerking te meten / evalueren slagen er onvoldoende in om de complexiteit en de (eventuele) impact van deze activiteit te capteren. 46
- 47. A: 4B: 5 Stelling 4. Het ITG moet als ambitie hebben om naast een "Centre of Excellence" in onderzoek (en onderwijs), ook een Centre of Excellence te zijn in het domein van de dienstverlening/ institutionele samenwerking. 47
- 48. A: 2B: 1 Stelling 5. De huidige druk in onze samenleving op academische instellingen om steeds meer en sneller meetbare outputs (productie) te bereiken in hun onderzoek, gaat ten koste van de maatschappelijke dienstverlenende rol van deze instellingen. 48
Editor's Notes
- #20: How would we know whether what has been planned actually happened?. Specific to the objective it is supposed to measure; Measurable (quantitatively or qualitatively); Available at an acceptable cost; Relevant to the information needs of managers; Time-bound when will the target be achieved