AidData 3.0: Engine for Growth

August 30, 2017 Data Management Systems and MEL
Josh Powell
Innovation, Open Data

In 2003, an undergraduate student at the College of William and Mary set out to write his honors thesis on the efforts of international development organizations to help communities adapt to and mitigate the effects of global climate change. His research quickly hit a roadblock: existing public data on foreign aid flows was not precise enough to enable his analysis. From that challenge, the idea for AidData.org was born: build the world’s most comprehensive and granular database of aid information, enabling better development research.

Today, AidData.org is recognized far beyond the academic realm as a leading information and data source on development programming, including comprehensive information on aid projects, geocoded data, cutting edge working papers and research briefs, and an influential blog. Bridging the gap between academia and the policy community required us to rethink how to make our data accessible and usable for users’ diverse needs.

As AidData entered into a five-year partnership with Higher Education Solutions Network (HESN) in October 2012, its audience continued rapidly expanding in development policy-makers and practitioners. We recognized that AidData.org would need to evolve in step with our user community to ensure that all users could access and use the wealth of geocoded aid information produced under the HESN award. With HESN support, the AidData team worked for 10 months to complete a wide set of internal consultations and external engagements to define user requirements and build a new, flexible, and powerful technology platform.
The finished platform — AidData 3.0 — manages data flows, publication, and visualization of millions of records, enables real-time release of news and opinion content, and facilitates publication of research papers and briefs. Its launch in October 2013 represented a significant milestone in establishing AidData as a “one stop shop” for development data and research, and demonstrated the ability to efficiently code and process large volumes of data. Additionally, a GIS portal for geocoded data, total resource tracking, interactive dashboards, and multi-source (AidData, IATI, OECD) search functionality empower users to access the development data they need.

The launch event during Global Transparency Week was attended by more than 125 participants. It featured Dr. Homi Kharas of the Brookings Institution and Dr. Alex Deghan, then-Director of the USAID Office of Science and Technology. From 2013-2014, AidData’s internal data management tools were developed and integrated with the external web portal — and from 2014-2016, AidData.org gained 200,000 unique users.

Tools and Technology
One of AidData 3.0’s key goals was to expand beyond traditional development data to include total resource tracking: tracking remittance flows, foreign direct investment, private philanthropy, and development assistance together in one user-friendly portal. Supporting the collection, standardization, classification, and visualization of these heterogeneous data types and sources required several technical innovations. Development Gateway (DG), an AidData Center for Development Policy (ACDP) institutional partner, used several new technologies to do so:
  • NoSQL databases and querying of big data (MongoDB and ElasticSearch) to manage a larger, broader database

  • Customized and flexible workflow management for creating new coding and data management workflows, enabling data from multiple sources

  • Leveraging Esri GIS tools, together with Esri open source projects, for fast mapping and geocoding in the GIS map tool

  • New Javascript frameworks and libraries (Backbone.js, Boostrap.js, HighCharts, etc.), and approaches to combining content management (Drupal), visualizations, and maps in one portal — which powers AidData 3.0’s interactive dashboards and GIS

  • A modular, API-driven approach to separating data management, data publication, data querying, and data visualization functions

The lessons learned in deploying new technologies for web GIS mapping, big data management, and workflow flexibility to build AidData 3.0 have been applied by the ACDP partners to build a wide range of data visualization and analysis tools. For example, DG used lessons learned from AidData.org’s GIS module in the creation of the MONITOR GIS tool built for USAID/Colombia, and used a similar MongoDB database design in archiving more than 100,000 Development Experience Clearinghouse (DEC) documents for USAID’s Global Development Lab through HESN.

DG has also leveraged AidData 3.0 innovations to power a wide range of data-driven development projects, such as using the NoSQL database tool to improve performance of web portals (including the Open Contracting Explorer and Corruption Risk Dashboard in Vietnam), implementing Drupal to power the Extractive Industries Transparency Initiative web portal, and leveraging advancements in geospatial data visualization in the AidData 3.0 GIS to develop the Aid Management Platform GIS portal and more than a dozen additional GIS portals (including that of the Complex Emergencies and Political Stability in Asia (CEPSA) program, built in partnership between ACDP members University of Texas at Austin and DG).

aiddata

Figure 1: CEPSA Nepal GIS Map

How AidData.org is Used

For the first time, AidData.org’s dashboards and GIS modules provided users with the ability to analyze and explore AidData’s 1 million-plus records representing $40 trillion in development financing in real time without the use of statistical software. This made its data accessible to practitioners and policy-makers without the time or capacity to download and analyze raw data.

USAID Staff: AidData.org’s GIS features are frequently used by USAID Geocenter team members to analyze and download geocoded data as they prepare for global visits to USAID Missions:

“The data collected and published by AidData is a great starting point for when I work with a mission that lacks GIS capacity. Checking the AidData website is one of the first things I do when I receive a request from a country I haven’t worked in before. The ability to immediately show how mapping development interventions can provide a highly valuable visual resource for decision-making makes GIS integration a no-brainer.”

USAID GeoCenter Team Member

AidData Staff: AidData.org staff uses the internal tools to gather, clean, geocode, and publish data to the AidData.org web portal. By combining all of these functions in one suite capable of managing millions of records, time and effort required for data management has been significantly reduced. Further, the AidData.org internal data management tools were used to generate the 2016 launch of the 3.0 research release, and have minimized the time required to download and republish the latest OECD data on development activities.

“The AidData internal software is a critical, catalyzing tool that has enabled efficient, accurate data management across the organization and beyond. It is a tool that has saved time for regular data management tasks such as importing new sources of data (such as the OECD CRS data, annual report data for donors outside the DAC, etc). It has also enabled value-add work that would have been effectively impossible without it, including performing double-blind and arbitration coding to assign sub-national locations to development finance activities and to assign activity codes to capture the full range of activities within each development project. The internal software will provide a solid foundation for AidData’s work now and well into the future.”

Brooke Russell (AidData Assistant Data Manager)

Researchers: Researchers use the web portal to access the research release, download geocoded datasets, and disseminate their research through the web portal and blog. Since 2013, more than 250 scholarly articles have cited the AidData.org data in their publication.

Other Users: AidData.org has been accessed more than 225,000 times since its launch in 2013, with tens of thousands of data and research downloads used for various purposes.

Share This Post

Related from our library

Developing Data Systems: Five Issues IREX and DG Explored at Festival de Datos

IREX and Development Gateway: An IREX Venture participated in Festival de Datos from November 7-9, 2023. In this blog, Philip Davidovich, Annie Kilroy, Josh Powell, and Tom Orrell explore five key issues discussed at Festival de Datos on advancing data systems and how IREX and DG are meeting these challenges.

January 17, 2024 Data Management Systems and MEL
DG’s Open Contracting Portal Designated as a Digital Public Good

Digital Public Goods Alliance designated DG’s Open Contracting Portal as a digital public good in September 2022. The Portal provides procurement analytics that can be used to improve procurement efficiency and, in turn, reduce corruption and increase impact.

December 6, 2022 Open Contracting and Procurement Analytics, Process & Tools
An aerial view of Bogotá, Colombia on the cover of the Data Value Project’s white paper. Photo credit- Bergslay:Pixabay.
Challenging Pessimists—and Optimists—to Reimagine Data and Power

Josh Powell and Jenna Slotin reflect on the Data Values Project and building a movement for change in data for development.

May 10, 2022 Global Data Policy