The Internal Learning System, Assessing Impact While Empowering Internal Learning
1 like952 views
AEA Conference Presentation, October 2012 Originally developed by: Dr. Helzi Noponen Presented by: Anna Kiel Martin Katrina Mitchell
More Related Content
Similar to The Internal Learning System, Assessing Impact While Empowering Internal Learning (20)
The Internal Learning System, Assessing Impact While Empowering Internal Learning
- 1. Self Help Group Promotion, PRADAN. Photo source: Helzi Noponen THE INTERNAL LEARNING SYSTEM ASSESSING IMPACT WHILE EMPOWERING INTERNAL LEARNING Anna Kiel Martin American Evaluators Association National Conference, Minneapolis Katrina Mitchell October 26, 2012
- 2. WHAT IS ILS? ≒ SIMPLE ≒ Quantitative ≒ Evaluation + ≒ Non-extractive ≒ Highly participatory ≒ Pictorial/accessible ≒ Customized ≒ Embedded in the program ≒ Multi-year
- 3. HOW DID ILS DEVELOP? Helzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt
- 4. WHY USE ILS?
- 5. EXERCISE: HOW DOES THIS WORK? You have been given a sample page. Identify what the purpose of the page is and how you would interact with it as a participant. How would you use it to assess impact or empower learning?
- 6. EVALUATION Sanitation Social Problems
- 7. LEARNING Suicide is Not a Solution ?
- 8. PLANNING Livelihood Production Plan Problem Sorting ? HPWDS
- 9. PLANNING Strategic Interests Plan ?
- 10. WHATS IN THE ILS TOOLBOX? Women own the ILS book. It is their record. Though illiterate, they can read it. No one has ever before suggested they can do such work. They take the responsibility seriously. It is like a green light for them to dare to think about achieving a better life. Each indicator picture has an implied program value attached to it. Women absorb this. It gives permission or space to discuss sensitive empowerment and wider social issues and the courage to attempt change Helzi Noponen
- 11. HOW DO YOU KNOW YOUR CURRENT SITUATION? General Health Care Practices Evaluation formats: ≒ Yes/no questions ≒ Quantity questions ≒ Multiple choice questions Woman goes to doctor when needed. Safe water practices ≒ Ranking and/or scaling questions Knows oral re-hydration therapy. Washes hands after latrine and before food handling. Woman herself uses mosquito net. Hygiene around menses
- 12. HOW DO YOU IMAGINE A BETTER FUTURE? Learning formats: Suicide is Not a Solution ≒ Bad Scene / Good Scenes ≒ Cautionary Cartoons / Illustrated Folk Tales ≒ Dream Scenes ≒ Panorama Scenes ≒ 6 panel Stories (photo novella, picture parade, comic strip) ?
- 13. HOW WILL YOU CREATE A BETTER FUTURE? Plan for Meeting Strategic Interests Planning formats: ≒ Problem Sorting Exercises think discuss husband family help friend help get comfort ≒ Goal Setting Exercises ≒ Priority Choosing Exercises other action get service ≒ Planning Formats budgeting meet officials work with men trees, loan repayment plans write officials research elite help campaign 0 1 2 3 cluster help SHG help other SHG help protest action exposure visit speak out training
- 14. HOW DO YOU WEAVE IT TOGETHER? Planning Learning Evaluation
- 15. EXERCISE: HOW DO YOU You are going to a pot-luck and have to bring a dish. What mental steps do you go through from the invitation to arriving at the right place, at the right time, with the right food to share?
- 16. PARTICIPANT: HOW DOES ILS CHANGE BEHAVIOR? 1. Collect 2. Assess 3. Analyze 5 Tasks 4. Plan or alter 5. Document, share, reinforce
- 17. PARTICIPANT: HOW DO I USE THIS TO IMPROVE MY SITUATION? Dalit women in Varanasi displaying their job cards Photo source: Leena Patel/UN Women (seeking permission)
- 18. HOW DOES A PARTICIPANT INTERACT WITH ILS? ≒ 5 tasks of ILS ≒ Ownership of diary/ data participatory ≒ Census they all have one ≒ Community-building, decreases isolation, empowers (story telling)
- 19. WHO INTERACTS WITH ILS? PARTICIPANT PROGRAM STAFF EVALUATOR AGENCY/ORGANIZATION Program participants with diaries/workbooks PRADAN, Photo source: Helzi Noponen Field promoter with PRADAN, Photo source: PRADAN Helzi Noponen in Bihar, 2006. Photo credit: Valerie Arendt Image source: PRADAN website
- 20. HOW DOES A PROGRAM MANAGER INTERACT WITH ILS? ≒ Facilitation in a group setting ≒ Reveals conditions/ behaviors which arent readily visible ≒ Highlights patterns ≒ Demands radical accountability ≒ Encourages reflective practice and innovation
- 21. HOW DOES AN EVALUATOR INTERACT WITH ILS? ≒ How to (steps to ILS): ≒ Design ≒ Implementation ≒ Measurement ≒ Commitment to participatory techniques ≒ Effective creation and implementation mirrors the tool; lots of learning and adapting
- 22. HOW DOES THE AGENCY INTERACT WITH ILS? ≒ Impact assessment ≒ Can quantify hard-to-measure social change outcomes ≒ Use with outside stakeholders for funding and advocacy ≒ Commitment to participatory techniques, accountability and innovation/learning
- 23. HOW WOULD YOU USE IT? EARLY CHILDHOOD HEALTH CARE BUSINESS DEVELOPMENT HOMELESS / YOUTH Stock photo Maternal / neo natal health care. Photo courtesy of Neighborhood House Recent immigrants. Photo courtesy of Neighborhood House Stock photo
- 24. IS ILS RIGHT FOR MY PROJECT / ORGANIZATION? YES NO I want QUANTITATIVE data I need QUALITATIVE data Learning is important I just need to assess Participant input is important Summative only Formative and summative Outcome only Process and outcome data Short term or one-off evaluation Longitudinal, long-term Highly customized USE TRADITIONAL M&E or EXTRACTIVE PIA
- 25. HOW WOULD YOU CONSTRUCT SOMETHING LIKE THIS .as it was originally meant to be used?
- 26. WHAT SHOULD I CONSIDER WHEN DESIGNING AN ILS DIARY? Research design ≒ Identify your needs and analysis plan before you start Survey design ≒ Both formative and summative ≒ Types of questions ≒ Level of detail ≒ Relevant to all participants Diary design ≒ Pictorialize lots of considerations! ≒ Create modules ≒ Start simple. Build up to complexity and risk.
- 28. HOW DO YOU TURN PICTURES AND PENCIL LINES INTO DATA? PARTICIPANT DIARY INTERVIEWER DIARY The semi-extractive Land Utilization Land Utilization 99=not farmer (skip page) interview event: Grows crop? 0=No; 1=Yes number of yearly harvests? average yield? base mid point end point ≒ Side-by-side number of harvests # grain harvests optimum yield base mid point end point ≒ Sharing base mid end grain yield 5 optimum yield base mid point end point ≒ Transparency base mid end # pulses harvests base mid point end point above 75% yield pulses yield 4 ≒ Verification base mid point end point above 75% yield # oilseeds harvest ≒ Human Subjects 50% - 75% yield base mid end base mid point end point Protocol oilseeds yield 3 base mid point end point 50% - 75% yield # vegetables harvest base mid end base mid point end point 2 25% - 50% yield vegetables yield 25% - 50% yield base mid point end point # fruits harvest Include sugarcane base mid point end point 1 base mid end Less than 25% yield Less than 25% yield fruits yield 20 20
- 29. WHAT ARE ITS STRENGTHS? Traditional M&E Participatory Impact Assessment Extractive METHOD Empowering Donors/policy AUDIENCE Clients Judging PURPOSE Learning External ORIENTATION Internal From Simonowitz, 2001
- 30. COPYRIGHT, ATTRIBUTION AND USAGE A note on copyright, attribution, and appropriate usage. The Internal Learning System was developed by Dr. Helzi Noponen in partnership with Ford Foundation India and Imp-Act, and implemented with a number of organizations including PRADAN, the Handloom Weavers Development Society (HLWDS), ASA, the High-range Plantation Workers Development Society (HPWDS), PLAN Honduras, and others. Many of the illustrations contained in this presentation come from diaries/workbooks created by Dr. Noponen for these organizations. Parts of this presentation were adapted from presentations and materials created by Dr. Noponen. Anna and Katrina are currently working with Dr. Noponens family and colleagues to preserve and extend the legacy of ILS. Please consult with us for attribution and usage information prior to distributing or re-using these materials.
- 31. Muhammad Yunus, Nobel Prize Winner, with Helzi Noponen, 2008. Photo source: Gary Smaby HELZI NOPONEN, 1953-2012 Innovative mind, open heart. helzinoponen.wordpress.com
- 32. THANK YOU! KATRINA MITCHELL Research and Strategy for Good genius@katrinamitchell.com 612-269-2891 ANNA MARTIN Thinking Evaluation anna@thinkeval.com 612-805-6460 internallearningsystem.com
- 33. Building a Diary | Components The diaries are composed of a series of themed modules. The modules together form a development curriculum that is a logical whole. Each module utilizes a variety of different approaches, or components, for meeting program goals and evaluation. The three types of components are for learning, planning and evaluation. Every module does not have all three, but most include at least some evaluation exercises. Usually a learning exercise introduces the module. Learning Planning Evaluation Heavily facilitated, typically in a group setting, by Planning tools allow for us to take on greater Some of the evaluation is for impact assessment the program officer or 鍖eld agent. These exercises complexity in our lives than if we try to keep it all in (proving) perspective, many of them help the make up the curriculum of the program; they are our heads. The planning exercises in the diary participant to assess their own life and circum- messages from the program/agency to its constitu- open up a whole new world for participants. For stances in a meaningful way. Many have never ents about the intervention. They tend to be many, it is the 鍖rst time they have purposefully thought in these terms before. Simply writing problem-solving or strengths-based. The purpose created and acted on a plan. These plans help down and quantifying thier life and experience is to inspire participants on how life could be, or participants move from reactive/crisis orientation to can provoke new ways of doing things, new ways how a problem could turn out well. They are not a proactive/future orientation. of thinking. prescriptive but are conversation starters. It is important that the participant takes an active Yes/no questions Bad Scene / Good Scenes role in their own planning. The diary is a tool to Quantity questions Cautionary Cartoons / Illustrated Folk Tales identify, prioritize and then strategize about the Multiple choice questions Dream Scene problems and possibilities in their life before taking Ranking and/or scaling questions Panorama Scenes (e.g. Gender Benders) action. 6 panel Stories (photo novella, picture parade, comic strip) Planning components also allow for accountability -- both the participant to themself and the partici- pant to the group, vice versa. Problem Sorting Exercises Goal Setting Exercises Priority Choosing Exercises Planning Formats Image source: Helzi Noponen Image source: Helzi Noponen ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell. ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com
- 34. Building a Diary | 5 Tasks of ILS The core of the ILS strategy is to engage the participants in a 鍖ve-step process that carries over and informs how they will think about their life. We all do these things, but the diary makes this process explicit, in order to empower the participants with choice. 1. Collecting Data Participants own the ILS diary. It is their record. The diary promotes re鍖ection on their current situation, in a struc- tured and relevant format. Each indicator picture has an implied program value attached to it. The process of carefully marking and keeping the diary acts as a green light for them to dare to think about achieving a better life. 2. Assessing Data / Change Participants, though illiterate, can read their own diary and that of others. Individually, and in the context of a group, they gain an understanding of their changing life and livelihood situation. Patterns emerge, the ability to set goals, prioritize problems and solutions and plan for the future are all facilitated through the use of the diary. 3. Analysing Causes of Change / Troubleshoot Once a goal is set, or a pattern emerges, participants use their data and assessment of the situation to identify potential causes and reasons. Asking why? and pursuing an answer calls forth the wisdom and skills of each participant, empowering them to step into a position of primary actor in their lives. 4. Plan or Alter Strategies, Training With an understanding of the forces at play, participants make intentional changes to their strategies and behavior for achieving their goals. They begin to track and assess data again, to understand the impact of this new plan of actiongaining greater self-awareness, con鍖- dence and better results in the process. 5. Documenting, sharing and reinforcing values Sharing between participants can be a wonderful source of inspiration, support and power; discus- sion is an oft over-looked learning tool. The diary acts as a testimony to individual experiences, which are often duplicated between group members. These recorded similarities give permission to discuss sensitive wider social issues and inequitable structures and the courage to attempt change. Collective action to summon needed resources and services is bolstered by the diaries written record. The source for all images on this page: Helzi Noponen ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell. ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com
- 35. Why ILS? | Going beyond proving While ILS can help prove program effectiveness, and can be used for internal accountability, its real strength lies in its ability to support improving. It is the learning and planning components of ILS coupled together with evaluation that make it a powerful tool which is integral to the program design. Program Impact Assessment Objectives PROVING IMPROVING / LEARNING Strengthen Self Proving to Stakeholder advocacy program Relationships participants and staff EMPOWERMENT Planning Proving to funders + organizational Staff Participant leadership learning learning Re鍖ection + Assessment Traditional Impact Assessment Participatory Impact Assessment (ILS, and others) Traditional impact assessment has an external focus (to the donors and Participatory Impact Assessment (PIA) is internally focused, aimed at improving policy makers) and a more quantitative and objective approach. It tends to outcomes through the empowerment of the program staff and participants. PIA falls be extractive, the information gathered is primarily used outside of the along a spectrum of participatory methods used to gather data from extractive formats community and little, if any, feedback is provided to the community in terms to fully engaging participants in evaluating their own progress and the program.2 While they can understand. The purpose of the assessment is to prove effective- some focus on the pictorial aspect of ILS as a way to gather information from a ness and efficiency rather than to empower the community, participants or low-literacy population, the true strength of ILS is in its abiltity to empower learning and program staff to learn and improve.1 inspire action.3 1. Marisol Estrella and John Gaventa, Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review,as summarized in Anton Simonawitz, Making impact assessment more participatory June 2000 2. Anton Simonawitz, Making impact assessment more participatory June 2000 3. For a discussion of how ILS differs from PRA techniques using pictures see Noponen, H. (2002) The Internal Learning Systema tool for micro-鍖nance and livelihoods interventions. Development Bulletin 57: 106110. ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell. ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com
- 36. Building a diary | How to ILS (the basics) ILS is not just about creating a pictorial diary. It is based on well-grounded research design and starts with the creation of a simple survey instrument using indicators derived from thorough background research on the community and the program. Initial research is often done with a participatory approach, though a survey could be designed in collaboration with program staff who have sufficient 鍖eld expertise. In both cases it is critical to do at least one 鍖eld test before implementation. In order to make a diary that is interesting and relevant to all participants, it should contain very few skip patterns. The survey should be designed with questions that can be answered pictorially such as status or yes/no, simple quantities, multiple choice and scale ratings. Each indicator should relate to a learning component. Do not include survey questions that are only of interest to program staff or for external proving. CREATING A DIARY PUTTING IT TO USE Train program staff Conduct background research to develop appropriate indicators and determine barriers to success Print and mark diaries with ID and date Distribute census wide Draft a simple survey instrument From identi鍖ed barriers and program including demographics and key goals conceptualize learning and indicators of program success planning components For baseline make 鍖rst marks in red Pictorialize the survey (Hire an artist to create drawings) ANALYSIS & REPORTING Likely, issues arise with: Develop research design Draft layout and assemble learning, planning and evaluation components a. Layout of a question on a page causes into modules that form a logical whole. confusion Train interviewers! b. Individual pictures are not understood at all, Train 1-2 staff in facilitation or interpreted differently across the group Select sample based on design c. A particular question or learning component (panel or cross-sectional) causes discomfort, stone-walling or con鍖ict FIELD TEST!! from the group Semi-extractive side-by-side d. Particular subject matter or issues were interview event. Participatory! repeat Debrief / sharing with overlooked as necessary participants and staff e. One of the choices of a multiple choice Data coding and entry question is incorrect or missing Revise* f. Facilitation of a particular learning component Analysis should occur differently or at a different time. Create facilitation guide Report to all stakeholders ILS was developed by Helzi Noponen. This document created by Anna Martin and Katrina Mitchell. ILS | Internal Learning System To reproduce or use please contact us at internallearningsystem.com. www.internallearningsystem.com