狠狠撸

ChiYan Lam, MEd
CES 2013
Insights on Developmental
Evaluator Roles during
Innovation: A Case Study
@chiyanlam
June 11, 2013
Assessment and Evaluation Group, Queen’s University
狠狠撸s available now at www.chiyanlam.com
1Monday, 10 June, 13

Acknowledgements

Impersonal
Hard to Focus
Passive

?Learn from peers
?Re?ect on prior
experiences
?Meaning-making
?Active construction
of knowledge

Assessment Pilot Initiative
? Barriers to Teaching and Learning: $, time,
space
? Interested in integrating Web technologies
to Teacher Education to open up possibilities
? The thinking was that assessment learning
requires learners to actively engage with
peers and challenge their own
experiences and conceptions of
assessment.
6

Dilemma
? Barriers to Teaching and Learning: $, time, space.
? PRACTICUM = out of sight, out of touch.
? Instructors became interested in integrating Web
technologies to Teacher Education to open up
possibilities
? The thinking was that assessment learning
requires learners to actively engage with peers
and challenge their own experiences and
conceptions of assessment.
7

So what happened...?
? 22 teacher candidates participated in a hybrid, blended
learning pilot. They tweeted about their own
experiences around trying to put into practice
contemporary notions of assessment.
? Guided by the script:“Think Tweet Share”
? Developmental evaluation guided this exploration,
between the instructors, evaluator, and teacher candidates
as a collective in this participatory learning experience.
? DE became integrated; Program became agile and
responsive by design
8

Research Purpose
to learn about the capacity of developmental
evaluation to support innovation
development.
(from nothing to something)
10

Research Question
What insights, if any, can be drawn from this
development about the roles and the
responsibilities of the developmental
evaluator?
11

Developmental Evaluation
in 2011
? takes on a responsive,
collaborative, adaptive
orientation to evaluation
? complexity concepts
? systems thinking
? social innovation

DE supports innovation development to
guide adaptation to emergent and
dynamic realities in complex
environments
DE brings to innovation and adaptation
the processes of:
? asking evaluative questions
? applying evaluation logic
? gathering and reporting eval
data to inform support project/
program/product, and/or
organizational development in real
time.Thus, feedback is rapid.
Evaluator works collaboratively with
social innovators to conceptualize,
design, and test new approaches in
long-term, ongoing process of
adaptation, intentional change and
development.
Primary functions of evaluator:
? elucidate the innovation and
adaptation processes
? track their implications and results
? facilitate ongoing, real-time
data-based decision-making
in the developmental process.
(Patton, 1994, 2011)

DE supports innovation development
to guide adaptation to emergent and
dynamic realities in complex
environments
DE brings to innovation and adaptation
the processes of:
? asking evaluative questions
? applying evaluation logic
? gathering and reporting eval data
to inform support project/
program/product, and/or
organizational development in real
time.Thus, feedback is rapid.
Evaluator works collaboratively with
social innovators to conceptualize,
design, and test new approaches in
long-term, ongoing process of
adaptation, intentional change and
development.
Primary functions of evaluator:
? elucidate the innovation and
adaptation processes
? track their implications and results
? facilitate ongoing, real-time data-
based decision-making in the
developmental process.
(Patton, 1994, 2011)
Improvement
vs
Development .

Developmental
Evaluation
is
reality
testing.

Practicing DE
? Adapative to context, agile in methods,
responsive to needs
? evaluative thinking - critical thinking
? bricoleur
? “purpose-and-relationship-driven not
[research] method driven”(Patton, 2011, p.
288)

Five Purposes and Uses
1. Ongoing development in adapting program, strategy,
policy, etc.
2. Adapting effective principles to a local context
3. Developing a rapid response
4. Preformative development of a potentially broad-
impact, scalable innovation
5. Major systems change and cross-scale developmental
evaluation
17
(Patton, 2011, p. 194)
Five Uses of DE

Evaluator Roles
18
Neutral, detached social scientist
Judge
Program Advocate
Consultant
Change agent
Technical expert
Facilitator of organizational inquiry
Support Decision Making
Collaborator
Empower othersText
Auditor

Role Conceptualization
? Depends on...
? Evaluation methods
? Evaluation models
? evaluator relationship with
stakeholders
? activities!? (Skolits, Morrow, & Burr,
2009)
19

Why Roles Matter
? Roles we adopt determine behaviours and
degree of contribution within an eval.
? Has implications for:
? how we conduct evaluation (processes)
? what is considered acceptable behaviour
(practices)
? training
20

DE Roles - Bricoleur
? Patton, 1994, 2011
? Collaborator
? Evaluator
? Support decision-making
? Documentarian
? Elucidate innovation and adaptation...
? .....?
21

DE Roles - DE Primer
? Coach to the organization, and to the
innovation process
? Collaborator
22

DE Roles - DE 201
? Analyst (strategic thinking, patter
recognition)
? Relationship Builder
? Servant Leader
? Facilitator of Learning
23
(Dozois, Langlois & Blanchet-Cohen, 2010)

Method & Methodology
? Qualitative Case Study
? understanding the intricacies into the
phenomenon and the context
? Case is a “speci?c, unique, bounded
system” (Stake, 2005, p. 436).
? Understanding the system’s activity, and its
function and interactions.
? Qualitative research to describe, understand, and
infer meaning.
24

Data Sources
? Three pillars of data
1. Program development records
2. Development Artifacts
3. Interviews with clients on the signi?cance
of various DE episodes
25

Data Analysis
1. Reconstructing evidentiary base
2. Identifying developmental episodes
3. Coding for developmental moments
4. Time-series analysis
26

Findings - DE Roles
during Innovation
1. Manager
2. Facilitator of learning
3. Evaluator
4. Innovation thinker
28

DE Role - Manager
? Manage the project
? resources, personnel
? Manage the progress
? timeline, deadlines,
milestones
? Rally the troop
? keep on task
http://civilengineerlink.com/wp-content/uploads/2011/02/project-managers.jpg

DE Role - Evaluator
? Report on usage
? shortened reporting cycle
? monitor the conversation
and report on substantive
themes
? facilitates data-based
decision-making
? Systematic inquiry - render
information (e.g. via lit review)
to inform program decisions
http://freshspectrum.com/freshspectrum-in-the-wild-12-illustrations/

DE Role - Facilitate
Learning
? Meaning-making of data
? made dif?cult due to no
prior framework
? had to be appropriate given
context
? To evaluate requires deep
familiarization with a particular
subject --> makes you a quasi-
expert.
? creates opportunity to help
develop capacity in others

DE Role - Innovation
Thinker
? Innovation is like
entrepreneurship. Good
entrepreneur has good
business acumen. Knows
the ‘business’
? Evaluators working in
social innovation needs
to know innovation .
? Process of innovation is
often non-linear.
http://seec.schulich.yorku.ca/ContentResources/LibraryFiles/innovations.jpg

ThankYou!
Let’s Connect!
@chiyanlam
chi.lam@QueensU.ca

狠狠撸

Ces13 roles pdf

More Related Content

Ces13 roles pdf