Sharing the Open Data Retrospective

We commissioned Dalberg, a global strategic advisory firm, to conduct an Open Data Retrospective to explore: ‘how and why did the open data field evolve globally over the past decade?’ as well as ‘where is the field today?’ With the concurrent release of the report “The State of Open Data” – led by IDRC and Open Data for Development initiative – we thought this would be a great time to make public the report we’d commissioned. 

You can see Dalberg’s open data report here, and its affiliated data here. Please note, this presentation is a modification of the report. Several sections and slides have been removed for brevity and/or confidentiality. Therefore, some details about particular organisations and strategies are not included in this deck.

Dalberg’s report covers the trajectory of the open data field and characterised it as: inception (pre-2008), systematisation (2009-2010), expansion (2011-2015), and reevaluation (2016-2018). This characterisation varies by region and sector, but generally captures the evolution of the open data movement. 

During the inception phase, we watched initial open data champions - largely in the UK and US - begin to launch policies, portals and practices. In the systematisation phase, new platforms and initiatives started to take shape, such as the Open Government Partnership, which aims to empower people and encourage governments to be open and responsive. The open data movement spread into new geographies and sectors during the expansion phase, seeing new open data initiatives in Latin America, and parts of Africa, and across sectors such as elections, education, crime and health. And – triggered in part by significant data and privacy events such as the Snowden leaks as well as growing issues around data and digital rights – the field is now going through a reevaluation phase, as it looks to navigate these growing issues and increased awareness of data privacy, as well as changing political climates (particularly in the UK and US). 

During the last decade, there has been demonstrable progress in building the ecosystem as policies, norms, and standards have been introduced and a dramatic increase in the amount of data that is released and used. A few countries now have open data laws or laws that include open data standards (for example, Germany’s open data law enabling free access to government data), while technical standards have been widely adopted, such as the Open Contracting Data Standard. The field has also built methods to monitor progress, such as the Open Data Barometer.

The release and use of data have achieved results in fighting corruption, reducing procurement costs, enabling consumers to understand their options, and creating businesses fueled by new data sources. For example:[2]

• In 2014, the Ukrainian government created ProZorro, an open source, open data e-procurement system at a cost of around $4.7 million USD. To date, ProZorro has saved over $350 million USD on planned government spending and increased the number of companies bidding for contracts by 50% 
• A Tu Servicio, a Uruguayan website that provides easy-to-use government health data, helps citizens make data-based decisions about their health service providers. Within one month, the website received 35,000 visits, equivalent to 1% of Uruguay’s population, compared to fewer than 500 downloads in the year before the launch
• Transport for London (TfL), the government body responsible for transport in the capital, decided to release 63 key datasets largely as an experiment without a strong business case supporting the decision. This has unlocked £130 million for the London economy via growth of businesses using TfL data, and better transport services and commute time saved. Today, over 600 applications use TfL data and reach four million people. 

As impact continues to unfold – and in some cases, where impact has fallen short or been negative or nonexistent – so does the discussion around how open data should now evolve. 

For instance, many early open data advocates did not give sufficient attention to the risks and costs of opening data – or propose mitigation strategies to address them. These debates gain momentum as we learn more about how abuses of data use and new threats to data privacy impact citizens and institutions around the world.

Cases of opening data without having the appropriate precautions have evidenced open data’s risk. For instance, TfL made its bike-sharing data publicly available, which unintentionally made it possible to track the journeys and whereabouts of a single cyclist. One could glean from the dataset where an individual likely worked, lived, and socialised. 

Furthermore, there have been a number of ‘white elephant’ open data efforts and portals, where open data has been released without a clear purpose, without meeting its potential, or without having the appropriate precautions, risking harming perceptions about the value of the field. It is important to open data better, not just to open more data. 

Relatedly, an important discussion has emerged around purpose-driven release (i.e. opening data for a clear goal) versus free-market-driven release (i.e. releasing as much as possible). We see that the open data projects that have most effectively delivered impact are those that are focused on solving problems that people care about. The balance must be having clearly a defined purpose that aligns data release to real problems, yet without cutting off serendipitous uses. 

The open data field has matured from 2008-2018, but it requires additional efforts to secure sustainable release and use of quality data over the long term. The field also faces important external threats that undermine the open data paradigm, from political shifts, including the rise of authoritarian  governments, to growing concerns about data privacy, such as the recent Cambridge Analytica scandal and the misuse of data in elections campaigns around the world. 

The Open Data Retrospective reflected many of the trends and findings that we have observed both through our own work and that of our investee organisations in our portfolio. For instance, as the field evolved, so our own strategy evolved too, from Open Data (2014) to Data Governance (2016), and to Data & Digital Rights (2018-2022). 

Dalberg’s report highlights a number of implications for the future of the open data ecosystem and indeed Luminate’s work too. In particular, the assessment presented four possible future scenarios:

1. Open data is sustained as a field in and of itself
2. Sector-specific data efforts dominate
3. Energy shifts to other trends and open data is deprioritised
4. A new frame ties many data-related topics together, including open data. 

Our own future most closely resembles #4 above: a new frame that ties many data-related topics together, including open data. This approach has been informed by the last ten years in the space and is in response to the more recent shifting landscape. As Luminate, we are expanding the support we have provided to the open data field for the past decade by also supporting organisations and advocacy that ensure people’s rights are upheld online, and in the design and use of technology. Our work includes a focus on digital threats to democracy, and data rights. We’ve already begun doing this through grants to Privacy International, Data & Society, AI Now, Upturn, and Digital Freedom Fund. We encourage readers to explore our Data & Digital Rights strategy (2018-2022), which we’ve published as part of our commitment to be transparent and accountable.

Our goal is to support people and communities to exercise their human rights in the design and use of data and technology. Observations underlying our strategy are: 

1. Openness and privacy arise from the same impulse for control and agency, for example the ability to act independently, with freedom of choice 
2. Data and tech are not neutral; they reflect existing concentrations of power
3. The race is on to shape the norms and standards that will underpin the governance of data.  

Dalberg’s report confirmed that Luminate is not the only organisation in this field in the ‘re-evaluation’ phase. In fact, we are in good company. As for our portfolio, the Open Data Institute once focused much more exclusively on ‘openness’ created a helpful data spectrum to indicate which types of data should be closed, shared, or open, respectively and has also been looking at models of data sharing and data access, including data trusts. GovLab, which has focused on open data impact and demand, is increasingly looking at data stewardship, and data collaboratives as a form of data governance. And the Web Foundation, which was heavily involved in open data work through initiatives such as its Open Data Barometer, has now shifted strategy to focus more on digital equality and data rights. 

We look forward to what the next decade of open data brings, and we welcome your feedback on the report!