際際滷

際際滷Share a Scribd company logo

How evaluations change with open data

A
alinepennisi

The paper aims at exploring the consequences of the gradually increasing availability of Open Data for evaluation as we know it. Using concepts from the literature on evaluation and democracy, it contends that new technologies both require a new behavior by evaluators and open up possibilities in the very framework in which evaluation is done. The pressure to open up data changes the way governments and public sector offices conceptualize, produce, and disseminate data. Responding to this demand requires that internal procedures change in fundamental, still partially unexplored ways. Issues arise also for citizens seeking information. They face a rapid growth of internet-based sources, which both creates opportunities for research and difficulties in assessing data quality, credibility, and usability. It also implies that public interventions--be they programmes, projects, or services--are open to public scrutiny of a new, more informed type. It increasingly involves expert, non-expert, and differently-expert scrutiny. It is highly unlikely that Open Data will ever provide all--or even most--information needed for an evaluation. There is a risk that, in addition to opening up new research avenues and framing new evaluation questions by new actors, the availability of great masses of data on public policies obscures the need to directly observe effects and to build credible theories about phenomena. The very existence of open data, and the possibilities they open up to public scrutiny call into question the role of internal and external evaluators. This is even more so when thinking of the opportunities opened by the ability to conjure collective intelligence in evaluation processes--using concepts already developed in the participation tradition. The paper explores these themes based on an on-going research project. The two authors are involved in the Open Data movement in Italy and will advance their research during the next months through their work, research on existing literature, and holding workshops (e.g. within the Sapienza Seminar on Classic Evaluation Theorists). 10th EES Biennial Conference

1 of 20
Download to read offline
Swarm Intelligence or Lost in the Crowd. Open Data, Public Scrutiny of Public Action, and How Evaluations Change Aline Pennisi e Laura Tagle EES Annual Conference Helsinki, October 2012
Outline State of the art
the increasing use of ICT in all domains Increasingly citizens/businesses are making their own information on the internet and consuming information made by others The internet is increasing the value of information created by government There is economic and social use of all this information . and new opportunities to evaluate government policies
what open data is A piece of content or data is open if anyone is free to use, reuse, and redistribute it without restrictions from copyright, patents or other mechanisms of control. collected by governments while performing its tasks (open government data) created and shared by users Restrictions: technological or legal features Focus: how open data changes the way the public sector relates to the external world and works internally
how open data and ICT affect the way the public sector works Dissemination of raw administrative data (while dissemination of statistics is not a novelty ...) and accessibility to anyone Interaction with a huge number of external bodies (citizens, businesses, other public authorities) on a much wider range of possibilities, with fewer filters The connection/integration between your data and the data produced by other parties ... public but also non-public Crowd-sourcing and collaborative data collection/validation The increased uncertainty about official/truth, given a less marked border between "certified" information and not The possibility that others provide a service that was previously the sole prerogative of the Government / public sector (in competition, replacement or in addition)
Open government and e-government Overlapping yet not the same Different approach to use of ICT Both share similar limitations Digital divides (territorial, age, socio-economic) New vulnerabilities
E- government Stress on technology: use of ICT to provide services Allows, but not necessarily requires: new services collaboration openness Scrutiny Frustratingly interacting with an automated system from home not intrinsically better than walking to the counter to frustratingly interact with a human
Open government Stress on activities Data as public good: push for publication of data changes internal functioning of government Openness relates to Possible collaboration: accomodates or invites interactions with other public and private entities New services: no fixed bundle of services, but invites and accomodates creative use of existing public goods (among which data) to provide new services Multi-centric, democratic governance Scrutiny of government action
Minnesota Tagged Fish http://www.dnr.state.mn.us/fisheries/tagged_fish_reporting/index.htm l 9
FixMYStreet http://www.fixmystreet.com/ 10
OpenCoesione http://www.opencoesione.gov.it/ 11
first claim on benefits of open data / ICT efficiency: with open data the role of Government/public administration in informing the citizens of public affairs and in providing utilities can be largely reduced (e.g., Robinson & Yu, 2010) o other (private intermediaries, profit or not-) can do this better, because they are on the "users side and having to compete every day in the market they need to continuously innovate and do it better o others can do this at lower cost, extrapolating data from more and more numerous sources, and giving it more meaning (through research, representation, processing, data update---beyond simple data delivery) But some questions are left unresolved: who is responsible for ensuring these services and making sure they are fair/ for everyone? are we really willing to pay "public" services supplied by third parties at a cost? does everyone really mean everyone? How many people are digitally literate?
second claim on benefits of open data / ICT democracy: Open data allows you to re-invent the relationship between rulers and ruled, between public service providers and citizens in favor of the latter (e.g., Tim O'Reilly, 2009; Maier-Rabler & Huber, 2011) o Government/public administrations can offer an unbiased platform where all actors interact (government as a platform ... from Donald Kettle's "vending machine" to Eric Raymond's "Bazaar") o everyone can monitor the actual conduct of the Government/Administration But some questions are left unresolved: It is not clear whether we have enough or the right data to determine accountability If there isnt a first authentic/explicit interpretation of data will we actually know which are the objectives of policies? If there is no official info who will lead the citizen in choosing between alternatives? Which groups are more active on the internet? How are vulnerable groups/territories reaching out on the internet?
third claim on the benefits of open data / ICT effectiveness: Open data improves the services provided and supports better policy decisions (e.g.,) o It reduces the information asymmetry and citizens/businesses can more easily report about the situation on the ground o allows you to rebuild confidence between the parties and share goals, and then to generate co-operative behaviors between policy makers and policy beneficiaries o dissemination, extreme detail and speed of information facilitates coordination among the various policy actors o it can provide valuable data for informing the choices of citizens and economic operators o You can create a stronger pressure towards results however: is voice enough to identify policies "that work? how to deal with moral hazard? How to select among all the information available?
opportunities for evaluation o increased availability of (often free!) data on the programs, projects or services that you want to evaluate o possibility to have many more reviews from many more evaluators (since they all have the same chances to have access to crucial data) o open government data is changing the way to collect and make available data on social phenomena (e.g., urban decor, environment, crime) o and it pushes the observation of social phenomena towards more detailed territorial level (using more administrative archives and fewer samples/estimates) MORE ?
challenges for evaluations Input data overwhelm the discourse already Government-produced information on programs, projects and services are not necessarily sufficient for the assessment of effects .. Open government data can only concern secondary data (created or collected or archived by public sector for its purposes) Difficulties in discerning sense, reliability, biases in primary datanot exactly new issues, but with NEW technical features and at a (potentially) much larger scale Judgment criteria multiply exponentially Speaking truth to power has a new meaning: main client is not anymore public sector but the public Independence becomes a must: reputation vis--vis the masses, not only the policy or professional community Quality control not only Steering Groups or advisory groups but the collective intelligence Evaluation needs primary data
challenges for evaluators Skills Demand: will open data stimulate demand for evaluation? Or will the availability of multiple analyses fulfill the need for evaluationin a time of shrinking resources? Status
expert, non-expert, and differently- expert scrutiny o how is the data going to be interpreted now that everyone is legitimated to do it? o Myriads of analyses and new evaluators Who needs to pay an evaluator when there are thousands of self- appointed analysts? Where do we acquire the new skills?
New frontiers opportunities opened by the ability to conjure collective intelligence in evaluation processes--using concepts already developed in the participation tradition evaluators also now have new tools to gather information, which can change the relationship with what is observed or evaluated Evaluation as expert knowledge, as an application of methods by evaluators, questioned. Clearer what has always been there: evaluation is one among many practices of public scrutiny, New, different expertise come into the play-field People with no voice & power can have their say.
Which one are we going for?

More Related Content

How evaluations change with open data

  • 1. Swarm Intelligence or Lost in the Crowd. Open Data, Public Scrutiny of Public Action, and How Evaluations Change Aline Pennisi e Laura Tagle EES Annual Conference Helsinki, October 2012
  • 2. Outline State of the art
  • 3. the increasing use of ICT in all domains Increasingly citizens/businesses are making their own information on the internet and consuming information made by others The internet is increasing the value of information created by government There is economic and social use of all this information . and new opportunities to evaluate government policies
  • 4. what open data is A piece of content or data is open if anyone is free to use, reuse, and redistribute it without restrictions from copyright, patents or other mechanisms of control. collected by governments while performing its tasks (open government data) created and shared by users Restrictions: technological or legal features Focus: how open data changes the way the public sector relates to the external world and works internally
  • 5. how open data and ICT affect the way the public sector works Dissemination of raw administrative data (while dissemination of statistics is not a novelty ...) and accessibility to anyone Interaction with a huge number of external bodies (citizens, businesses, other public authorities) on a much wider range of possibilities, with fewer filters The connection/integration between your data and the data produced by other parties ... public but also non-public Crowd-sourcing and collaborative data collection/validation The increased uncertainty about official/truth, given a less marked border between "certified" information and not The possibility that others provide a service that was previously the sole prerogative of the Government / public sector (in competition, replacement or in addition)
  • 6. Open government and e-government Overlapping yet not the same Different approach to use of ICT Both share similar limitations Digital divides (territorial, age, socio-economic) New vulnerabilities
  • 7. E- government Stress on technology: use of ICT to provide services Allows, but not necessarily requires: new services collaboration openness Scrutiny Frustratingly interacting with an automated system from home not intrinsically better than walking to the counter to frustratingly interact with a human
  • 8. Open government Stress on activities Data as public good: push for publication of data changes internal functioning of government Openness relates to Possible collaboration: accomodates or invites interactions with other public and private entities New services: no fixed bundle of services, but invites and accomodates creative use of existing public goods (among which data) to provide new services Multi-centric, democratic governance Scrutiny of government action
  • 12. first claim on benefits of open data / ICT efficiency: with open data the role of Government/public administration in informing the citizens of public affairs and in providing utilities can be largely reduced (e.g., Robinson & Yu, 2010) o other (private intermediaries, profit or not-) can do this better, because they are on the "users side and having to compete every day in the market they need to continuously innovate and do it better o others can do this at lower cost, extrapolating data from more and more numerous sources, and giving it more meaning (through research, representation, processing, data update---beyond simple data delivery) But some questions are left unresolved: who is responsible for ensuring these services and making sure they are fair/ for everyone? are we really willing to pay "public" services supplied by third parties at a cost? does everyone really mean everyone? How many people are digitally literate?
  • 13. second claim on benefits of open data / ICT democracy: Open data allows you to re-invent the relationship between rulers and ruled, between public service providers and citizens in favor of the latter (e.g., Tim O'Reilly, 2009; Maier-Rabler & Huber, 2011) o Government/public administrations can offer an unbiased platform where all actors interact (government as a platform ... from Donald Kettle's "vending machine" to Eric Raymond's "Bazaar") o everyone can monitor the actual conduct of the Government/Administration But some questions are left unresolved: It is not clear whether we have enough or the right data to determine accountability If there isnt a first authentic/explicit interpretation of data will we actually know which are the objectives of policies? If there is no official info who will lead the citizen in choosing between alternatives? Which groups are more active on the internet? How are vulnerable groups/territories reaching out on the internet?
  • 14. third claim on the benefits of open data / ICT effectiveness: Open data improves the services provided and supports better policy decisions (e.g.,) o It reduces the information asymmetry and citizens/businesses can more easily report about the situation on the ground o allows you to rebuild confidence between the parties and share goals, and then to generate co-operative behaviors between policy makers and policy beneficiaries o dissemination, extreme detail and speed of information facilitates coordination among the various policy actors o it can provide valuable data for informing the choices of citizens and economic operators o You can create a stronger pressure towards results however: is voice enough to identify policies "that work? how to deal with moral hazard? How to select among all the information available?
  • 15. opportunities for evaluation o increased availability of (often free!) data on the programs, projects or services that you want to evaluate o possibility to have many more reviews from many more evaluators (since they all have the same chances to have access to crucial data) o open government data is changing the way to collect and make available data on social phenomena (e.g., urban decor, environment, crime) o and it pushes the observation of social phenomena towards more detailed territorial level (using more administrative archives and fewer samples/estimates) MORE ?
  • 16. challenges for evaluations Input data overwhelm the discourse already Government-produced information on programs, projects and services are not necessarily sufficient for the assessment of effects .. Open government data can only concern secondary data (created or collected or archived by public sector for its purposes) Difficulties in discerning sense, reliability, biases in primary datanot exactly new issues, but with NEW technical features and at a (potentially) much larger scale Judgment criteria multiply exponentially Speaking truth to power has a new meaning: main client is not anymore public sector but the public Independence becomes a must: reputation vis--vis the masses, not only the policy or professional community Quality control not only Steering Groups or advisory groups but the collective intelligence Evaluation needs primary data
  • 17. challenges for evaluators Skills Demand: will open data stimulate demand for evaluation? Or will the availability of multiple analyses fulfill the need for evaluationin a time of shrinking resources? Status
  • 18. expert, non-expert, and differently- expert scrutiny o how is the data going to be interpreted now that everyone is legitimated to do it? o Myriads of analyses and new evaluators Who needs to pay an evaluator when there are thousands of self- appointed analysts? Where do we acquire the new skills?
  • 19. New frontiers opportunities opened by the ability to conjure collective intelligence in evaluation processes--using concepts already developed in the participation tradition evaluators also now have new tools to gather information, which can change the relationship with what is observed or evaluated Evaluation as expert knowledge, as an application of methods by evaluators, questioned. Clearer what has always been there: evaluation is one among many practices of public scrutiny, New, different expertise come into the play-field People with no voice & power can have their say.
  • 20. Which one are we going for?