�ݺ�ߣ

Editor's Notes

• #2: Hi everyone, I’m going to give a relatively quick update on some of the work I’ve been doing with regards to evidencing change and the impact of JISC funded projects before running a short exercise to help gather everybody’s input and feedback on what I’ve got to say.
• #3: So in terms of an agenda I thought the following would suffice. I’d talk for approximately 20 minutes based upon my experience in this area so far and highlight my suggested approach for gathering evidence of change and impact from JISC funded projects.We’ll then break into groups at which point you’ll have the chance to discuss my thoughts, focusing on: any negatives, positives, and what might need to change for this to work in your context.Finally, I’d like each group to feed their thoughts back to the wider group before we openly discuss everything. I’ll explain more about the group discussion when we get to it.If at any point anyone has a question they’d like to ask or any comments please feel free contribute. If I know I’m going to answer the question later in my talk we’ll park it. If not I’ll try to provide an answer there and then.
• #4: So, a brief warning. I need to highlight the fact that I’m not an expert in evaluation, or measuring impact and benefits. I have worked with a lot of JISC projects over the past couple of years and am beginning to build up a good picture of how JISC’s processes work in reality.And although it seems like I might be proposing a new way of doing something here; in my mind, none of this is new. It’s just moved higher up on our list of priorities and we’re beginning to give it the attention that it so badly needs.
• #5: Over the past couple of years I’ve found myself being pulled in a number of different directions by different parts of JISC that are all attempting to do pretty much the same thing. The problem I’ve faced is that everyone has a slightly different way of thinking, and interpretation on how it should be done. In a nutshell, we’re trying to improve the sector by taking risks on it’s behalf and sharing knowledge around that whether it’s to innovate, learn from our mistakes our prevent people reinventing the wheel; to ensure the future of JISC we need to demonstrate what impact we’re having in a compelling and well evidenced way. Sounds simple but in reality it’s extremely difficult, for a number of reasons.
• #6: More often than not, especially in my experience, we’re funding projects that are being managed by ‘accidental project managers’. And, for me that’s fantastic, because they will know their area of focus inside out. The problem is that they tend to lack the necessary administrative skills for capturing relevant and essential information about the journey of their project. You could argue that its more a question of focus than skill but lets not get into that; and I’m not just talking about budgets, risk, issues—I’m talking about a project log, or project diary if you’d like, that tells a real story of change and links to evidence of that change. We usually get the stuff they think we want to hear rather than what we really want to hear.Communication is an issue on two different levels. The first being the way that we communicate what we want from projects. We need to be absolutely clear on what’s expected of our projects and they need to know this when a call goes out. I think we’ve improved this recently but at times I still don’t think the process is very clear. Some projects require a bit of hand holding and we don’t yet provide that level of detail. We just expect they can handle it, they’re educated people – they should be able to interpret what we want from them from a 40 page funding call. The truth is they can’t, it is difficult, and we need to change that and stop burying our head in the sand.The second communication issue is around dissemination. I think JISC has produced some fantastic outputs but for some reason or other, they’re often put on a shelf and forgotten about. That needs to change and it also needs to be planned more effectively as a part of each programme. We also need better information about what works and what doesn’t work in terms of dissemination but perhaps that’s a topic we can take up in the next session.
• #7: Some of you might recognise this. I think it’s quite a useful diagram of Innovation at work. You can see the whole process in action from requirements gathering all the way through to ‘solutions, advice and guidance’.
• #8: What I’m talking about today relates more directly with these two circles, creating, updating, and testing stages of the cycle. The next two circles, I think relate more directly to the work of Comms and Marketing and Services. JISC Advance are certainly looking at how they can address the issue of impact and benefits at this level and I believe this afternoons session might touch upon it too.Both areas require some form of requirements gathering as that gives us a baseline picture. We started here. If we changed X we could achieve Y. Using that baseline information and idea of what we set out to do we can begin to tell a story of change and highlight what impacts have been achieved along the way. So, rather than talk around the subject lets explore the approach I’m proposing for evidencing change and impact at a project level.[Provide everybody with the hand outs]I’d really appreciate it if everyone could treat this as draft as it’s very much still under development. I did watermark it but for some reason it didn’t print and I didn’t want to waste the paper.
• #9: It focuses on three steps. Looking at and documenting the current state more accurately. Evidencing any change and then the telling of a projects’ impact story/or stories.The first section of the hand out focuses on why we should be doing this which feeds from some of Simon’s work on the OUT Team’s evaluation framework and also some background research on what project’s themselves might get out of this. The second page highlights the main template to be used by projects in this process. You’ll probably be surprised to find that this is extremely simple stuff, albeit usually quite difficult to answer. It has to be simple because projects are under huge amounts of pressure to deliver at a number of levels, whilst having to deal with the day job too. You might argue they should prioritise their workload more appropriately but at the end of the day, they care about their work and they want to help improve it so we should be helping them. That’s why we’re here, that’s why we spend so much on programme support activities.So, step one of this approach focuses on capturing information about the current state of an organisation before and during the early stages of a project. If you look at the template you can see there are lots of questions in there to help a project really think through what it is they’re trying to achieve. A lot of this will be in their project plan, however if you’ve read through any number of bids you’ll find the majority of projects aren’t actually very good at doing this. They’re usually unclear and provide no real evidence of the problem space they’re trying to improve.I should say at this point that we had some great feedback from a group of critical friends that we’re working with at the moment. They mentioned that they get the best out of projects when interviewing them, or questioning what it is they’re doing so the template has been developed in that way. It’s a line of question that attempts to glean appropriate information from them in order to evidence change and its impact.
• #10: If at this stage a project is still struggling to think through it’s expected benefits we’ve used a matrix exercise to help them brainstorm ideas. [Go to example benefits]We ran this with a group of projects from the Transformations and Course Data programmes looking to apply Enterprise Architecture, and they came up with the following in 5 minutes. You might argue whether or not some of these things are really impacts or benefits but it’s pretty impressive to list all of these things in just five minutes.And although I haven’t added any other tools into this document yet I think there are a whole host available that we could pull together for this.
• #11: The second stage of the process focuses on what actual change was achieved. Where possible it highlights where JISC inputs have been used and begins to get projects thinking about their narrative and story. If they’ve really thought about the first section and identified useful indicators they should find this pretty straight forward.
• #12: And then the third step is about creating a digital story. Projects might like to aggregate the narrative surrounding each row and develop an overarching story of their project, or they might like to develop multiple stories based upon their entries within the template. Either way, it’s up to them depending on how comfortable they feel with the process and how much time they can apply to this.As a minimum we’d expect at least an overarching story which can be added to their case study. At this point it’s probably worth saying that this all seems like a lot to be asking for, but we already do when asking projects for case studies. At the moment every project is expected to fill in a case study template in a support wiki that we’ve set up for them.[Go To Project Support Wiki]Within that case study template there is a section on benefits. I’d expect the Evidencing Change table to be added here along with a projects story. Then it can also be used as a standalone product as well.
• #13: I’m currently employed by JISC infoNet and we are responsible for helping to promote the effective strategic planning, implementation and management of information and learning technology. We do this through online guides which we’ve branded infoKits. Each infoKit is developed through background research but also by taking the findings from JISC Programmes i.e. real examples from within the sector. So, outputs from the FSD programme contributed to a number of infoKits—Enterprise Architecture; Cloud Computing; Shared Services; and so on. A major issue we had was that projects didn’t give us any real evidence and therefore it was very difficult to write anything about them. We need concrete findings, things we’re happy to promote. We need things we can shout about and this process will help them to do that.
• #14: In terms of next steps, we’re beginning to apply this to both the Transformations Programme and the BCE A2ROI projects. The Transformations Critical Friends are extremely happy with this proposal as is the lead evaluator who are all in place to help get the best out of our projects.Another JISC area I’m involved with is Emerging Practices, so trying to quickly meet the needs of JISC funded projects as and when its required. Our current focus is around Enterprise Architecture but also the application of Measurement Tools and Communication. The benefits and impact guidance we’ve been working on will be developed as a part of the measurement tools work and a Digital Storytelling workshop will be offered as part of the communications strand.