Introduction

 This literature review of almost one hundred publications frames The GovLab and Nesta’s Report, “Using Collective Intelligence to Solve Public Problems” within the existing academic literature on collective intelligence. Through this analysis, we seek to explain how public institutions can work with groups to solve public problems.

Collective intelligence (CI) has become a sprawling topic of inter- and transdisciplinary study. A wide range of fields have become interested in collective intelligence, the study of which has been bolstered by the creation of an annual ACM conference on Collective Intelligence that brings together scholars from both the computer and social sciences as well as an older conference on Computer-Supported Cooperative Work and Social Computing that has been going on since 1986.[1]

Many disciplines have been concerned for a long time with going beyond an examination of individual intelligence to looking at group intelligence and studying how groups and teams perform. From armies to orchestras, social psychologists like Richard Hackman have long examined how groups form, behave and cohere.[2] They study how trust emerges in groups and teams and the psychology of group collaboration.[3]With the rise of the Internet and the commercial World Wide Web a quarter-century ago, some of that focus has shifted to looking at groups and communities online whether on social media platforms or video games.[4]

In operations research and management sciences, for example, the core questions around what is sometimes also called crowd science as well as collective intelligence often address how collective intelligence functions within business organizations, such as how companies make use of a distributed “crowd” to innovate, looking at the relationship between a group of customers, suppliers or employees and business processes.[5] Studying collective intelligence has become central to understanding how firms produce value.

In social sciences, there are a variety of studies. Some collective intelligence scholars look at how to design online communities for political ends or to create new forms of democratic institutions.[6] There are those who look, for example, at the way online groups translate into activism and others that look at how online groups produce political polarization.[7] Others are interested in how new forms of peer production and bottom-up organization compare to hierarchical bureaucracy, markets and other forms of social organization.[8]

In computer science and human-computer interaction (HCI), those who study collective intelligence may focus on the affordances of platforms and how different socio-technical environments give rise to more or less productive group performance.[9] They may use big data -- analyzing thousands of data points from an online community -- to draw inferences about how groups behave online and model collective behavior.

For a good disciplinary overview of collective intelligence, see Tom Malone and Michael Bernstein’s Handbookof Collective Intelligence as well as numerous resources available at the MIT Center for Collective Intelligence, the Nesta Centre for Collective Intelligence Design and on websites that focus on how groups work together like the P2P Foundation. The GovLab also provides a robust repository of literature on collective intelligence and governance.

For a good historical overview of the study of collective intelligence, see Chao Yu, Yueting Chai, and Yi Liu’s literature review on collective intelligence.[10]

Given the existence of these excellent resources that present the literature with a chronological or disciplinary focus, our literature review uses a different lens. We present key readings drawn from diverse literatures within the framework of the nine design questions outlined in our report: “Using Collective Intelligence to Solve Public Problems”. These questions collectively describe the process for launching a CI project. The literature review illuminates debates in the field over these design choices.

What is collective intelligence?

The term “collective intelligence” (CI) is but one of many terms used to describe how groups work together aided by machines to create or analyze information, deliberate, and make decisions. Collective intelligence is often conflated with other terms including engagement, swarm intelligence, wisdom of crowds, crowd science or terms that refer to dialogic or decisionmaking methods such as crowdsourcing, collaboration, and co-creation.  In one systematic review of 9,418 scholarly articles on collective intelligence published since 2000, Shweta Suran and her coauthors note that while the concept exists in sociology, psychology, biology, management, economics, computer science, and many other disciplines, CI commonly refers to collaborative problem solving and decisionmaking.[11]

In the Handbook of Collective Intelligence, Director of the MIT Center for Collective Intelligence Thomas W. Malone and Stanford University computer scientist Michael S. Bernstein define the concept as “groups of individuals acting collectively in ways that seem intelligent.”[12] The authors intentionally choose a broad definition which recognizes the various forms and definitions of intelligence. For Malone and Bernstein, the size of the group is less important than the simple fact that members of the group work together to accomplish a common action.

Some have linked CI to the use of technology to facilitate problem solving, decisionmaking, or other related tasks. For example, in a short animated video titled “What is collective intelligence?” Kathy Peach - Co-head for Nesta’s Centre for Collective Intelligence Design - contends that CI emerges “when people work together with the help of technology to share ideas and gather data”.[13]

Others take a broader definition. In his 2017 book Big Mind: How Collective Intelligence Can Change Our WorldGeoff Mulgan (who served as Nesta’s Chief Executive from 2011 - 2019), likens collective intelligence to the decision-making process within the human brain, where a group of individuals neurons “only become useful when they’re connected to billions of other neurons”, much like how humans link up to solve problems. For Mulgan, CI is not defined by a particular use of technology, but technology can accelerate the process and create greater impacts.[14]

Collective Intelligence Design Guide

In our work, we identify 9 questions that organizations must consider when designing a CI project. These are all ways of asking whether an organization has the readiness to leverage the distributed knowledge of a group. John Prpic calls this  “crowd capital.” Crowd capital is the structure, content and process by which an organization expands its knowledge through distributed participation.[15] These nine questions reflect those factors that are observed in practice but also discussed in the literature and that determine the success of collective intelligence projects. To support our discussion in the Report of practical experiences and case studies, we outline supporting academic literature for each of the nine topics. Keep in mind that there is no one right answer to each of these questions. Rather, organizations need to grapple with them to determine whether to use collective intelligence intelligence.

 1. Can you articulate the project’s goals?

Research bears out the focus on being able to articulate the purpose of the CI exercise for the eventual success of the project.

A meta-analysis of citizen science projects by Technion - Israel Institute of Technology found that the most successful initiatives had a clear goal. While most had a scientific goal, which “is the essence of a Citizen Science project”, others also had educational or social goals.[16] Likewise, a 2016 report by the US Government Accountability Office (GAO) surveyed open innovation exercises across federal agencies strategies (including projects related to crowdsourcing and citizen Science, idea generation, open data collaboration, open dialogue, and prize competitions) in order to identify best practices. The report found that successful projects had clearly defined goals and performance measures up front, and aligned the chosen strategy with the agency’s capabilities.[17]

The same is true for civic engagement projects.[18] In a literature review on stakeholder participation in marine conservation, Brianne Leigh Kessler found that identifying goals around shared themes created shared value among stakeholders and informed the design of the participation mechanism later on.[19]

The goals have to be attractive as well as clear. If problem solvers do not feel that they can contribute a meaningful solution, or the solution is too simple to attract the right problem solvers, the contest might not lead to the expected outcomes.[20] Drawing on the Nesta’s past challenge prizes as well as those conducted by other foundations and governments, a 2017 practice guide from the Nesta Centre for Challenge Prizes  found that choosing a challenge prize that fits the goals of the initiative, that will create incentives for participation, and that will accelerate progress towards creating implementable solutions to the defined problem are key decisions that go into designing successful open innovation competitions.[21]

A related practice is to align the CI initiative with an institution’s larger strategic goals. To give one example, in a 2018 blog post, Geoff Mulgan uses a well-established and concrete set of aims - the Sustainable Development Goals (SDGs) - to describe how the United Nations Development Programme (UNDP) can use collective intelligence to become a platform for international problem solving. Introducing digital platforms, design skills, and other CI tools to the UNDP’s modus operandi can make it more likely the SDGs are achieved - a clear alignment with the UN’s long-term global strategy.[22]

2. Can you identify the right participants?

While some CI efforts are open to all, many engage a smaller cadre of experts. Thus, project owners must answer key questions about who the right people are, how many people to include, what is the impact of size on performance, whether to engage experts or laypersons, and whether to select participants or allow them to self-select.

There are conflicting hypotheses about how group size influences group performance. While some who focus on the law of large numbers or Condorcet’s jury theorem posit that the accuracy of decisions should increase as the group adds more members, more recent studies have found that the interaction between size and decision-making is more complex.[23]

A literature review by Chao Yu, Yueting Chai, and Yi Liu of the National Engineering Laboratory for E-commerce Technologies at Tsinghua University in Beijing found that “the number of members in a group does not convey the more the better.” The authors reviewed several experiments and found that while in some cases larger groups generated more unique and better quality ideas, in part due to greater cognitive diversity, if groups are too large they can become unwieldy, leading to challenges in facilitating information exchange between members. There is therefore a need to constrain the size of the group to only what is needed to complete the task.[24]

Ofra Amir’s 2018 study, which compared problem-solving in nominal groups (those where members solved a problem on their own, then convened to select the correct answer) of varying sizes came to a similar conclusion. Amir found that at a certain point, adding an additional team member ceased to increase the likelihood of identifying the correct answer, particularly for complex problems, and even became detrimental as it introduced more incorrect solutions from which the group chose.[25]

Social psychologist Richard Hackman comes to the same conclusion: “Another fallacy is that bigger teams are better than small ones because they have more resources to draw upon. A colleague and I once did some research showing that as a team gets bigger, the number of links that need to be managed among members goes up at an accelerating, almost exponential rate. It’s managing the links between members that gets teams into trouble. My rule of thumb is no double digits.”[26]

In a 2011 study, Kevin J. Boudreau, Nicola Lacetera, and Karim R. Lakhani, found that for open innovation contests dealing with low uncertainty problems, increasing the number of competitors reduced the overall quality of solutions produced, but increased the quality of the best solutions produced. By contrast, for high uncertainty problems, both the overall quality of solutions and the best solutions increased as the number of competitors increased. Thus, a totally open prize-backed challenge appears to be a more effective approach for dealing with high uncertainty problems.[27]

There are also practical reasons for limiting the size of the group. In a 2017 Nesta report entitled Digital Democracy: The Tools Transforming Political Engagement, Julie Simon argues that in many cases, “it is more important to engage a smaller number of people rather than the public at large.” This may be because of the skills or expertise involved, the time commitment, or to bring in a plurality of views on the topic at hand.[28]

Going beyond issues of size, Beth Simone Noveck has written extensively on the benefits of expert engagement as distinct from an “open call” to the general public. The Governance Lab has created a series of case studieson employee-based collective intelligence processes in the public sector again showcasing expert engagement.[29]

There is also debate within the literature about the merits of top-down selection vs. self-selection. Proponents of sortition argue that random sampling is needed to ensure that participants mirror the population they are meant to represent, a consideration that is particularly relevant for democratic CI exercises. In his book Against Elections: The Case for Democracy, historian and author David Van Reybrouk argues that democratic institutions have experienced a decline in legitimacy and effectiveness because the electoral process has come to represent the aristocracy rather than the will of the people. Van Reybrouk advocates that institutions adopt various forms of citizen assemblies (also called mini-publics), bodies where citizens selected through sortition would debate and vote on laws.[30] Likewise, John Gastil and Erik Olin Wright’s 2019 book Legislature by Lot: Transformative Designs for Deliberative Governance advocates for sortition as a demcratic reform that will increase civic innovation, combat corruption, and improve accountability.[31]

Others argue that letting people follow their passions by self-selecting is key to producing useful results. While Roslyn Fuller sees a place for sortition within the democratic reform process, in her book In Defence of Democracy she argues that using online platforms to enable mass participation, as countries around the world are already doing, would be a more effective means of representing the population’s viewpoints than a shift entirely to direct democracy.[32] Likewise in a 2014 article, Tanja Aitamurto argues that crowdsourcing, by definition, is an open call and that unequal representation is an unavoidable natural consequence. Thus, even in democratic exercises, curating representative samples would compromise the value of the crowdsourcing process.[33]

To be sure, crowdsourcing can also be used as a means of locating expertise. In a 2013 conference paper, Alessandro Bozzon found that crowdsourcing efforts can find domain specific experts and match expert candidates to the tasks that need to be carried out through social networking platforms like Twitter, Facebook, and LinkedIn.[34]

3. Can you reach the participants you have identified?

While much of the literature tends to focus on questions of incentives, a subset of the discussion that we discuss below, the practitioner literature, in particular, addresses strategies for outreach and awareness building. This step is often overlooked, an error which can lead projects to engage only the “usual suspects” of participation.[35] In an article about lessons learned for open innovation challenges, Sara Holoubek of Luminary Labs writes that good marketing is essential. Since the “solvers” who contribute most to such challenges often sit outside of the normal problem-solving ecosystem, organizations should expand their communications strategies to reach communities outside of their typical network.[36]

Julie Firmstone and Stephen Coleman’s study of public engagement efforts by local governments in the United States and Australia, found that “a small number of participants contribute a disproportionate share of ideas and opinions.” The authors propose design recommendations, such as building clarity and transparency in messaging, that could help reach a larger audience thus engage a larger crowd.[37]

Past studies have noted that while online platforms have a great potential to improve public engagement, public institutions must create the cohesive infrastructure and strategy for two-way communication to occur. Helen K. Liu’s 2017 study of the digital communication strategy of the City Council of Leeds, UK found that a disconnect between how the press team and project team framed their public outreach strategy made it difficult for participants to effectively engage in the initiative. Liu concludes that more effective digital communications strategies can improve public engagement by allowing public institutions to tap social networks as a source of public opinion, allow members of the public to organize online communities, and increase the diversity of types of communication between government and the public.[38]

Authors have also noted the need to engage both online and offline communities to ensure a greater diversity of participants. In “Governing with Collective Intelligence”, Saunders and Mulgan recommend that institutions “Remember that there is a world beyond the internet”. As underserved populations such as the elderly, young, and impoverished often lack access to digital platforms, institutions should “combine online methods with more traditional forms of engagement” to avoid amplifying existing biases in public engagement.[39]

For more information on marketing to and recruiting participants for public engagement initiatives, see Eric Gordon’s  2016 “Accelerating Public Engagement” roadmap and Mozilla’s “How to recruit?” toolkit.[40]

4. Who is the right owner?

Choosing the owner is a question of whether the initiative should be managed top-down or bottom-up. The answer is both and the combination is a key factor to success, according to many researchers.

Daren Brabham explains: “Crowdsourcing, Brabham tells us, is an online, distributed problem solving and production model that leverages the collective intelligence of online communities for specific purposes set forth by a crowdsourcing organization—corporate, government, or volunteer. Uniquely, it combines a bottom-up, open, creative process with top-down organizational goals.”[41] Daren Brabham argues elsewhere that “crowdsourcing entails a mix of top down, traditional, hierarchical management process and a bottom-up, open process involving an online community.” Therefore, the organization and crowd jointly share control.[42]

Having an institutional owner with the resources to manage the project is a key indicator of both effectiveness and longevity.[43]

A 2013 literature review by Kjetil Holgeid and Mark Thompson found that large IT projects in the public sector tended to fail due to leadership-related issues, such as a lack of management skill, more so than challenges related to technology. Such failures are highly visible and can lead to projects overrunning their schedule and timeline, but can be avoided if the project owner plans for challenges in advance and is adaptive when issues arise.[44] Top-down management approaches should also offer adaptive incentives in order to stay relevant. As motivations for participation vary greatly and may change over time, an adaptive top-down approach can adapt the incentives it offers to sustain participation over time.[45]

Another challenge for collective intelligence projects is governance, namely sharing ownership and responsibility for the project among partners.[46] This is an issue treated throughout the literature on public-private partnerships, which frequently overlaps with CI. A 2011 working paper from the Overseas Development Institute notes that public engagement efforts in the scientific community, “if undertaken appropriately, are time-, energy- and resource-intensive” efforts that are prone to delays and complications. Moreover, discussion with sponsors, organisers, and advisers arise at each stage of the public engagement effort, underscoring that “clear processes for planning, along with clearly outlined roles and responsibilities of different actors, facilitate an efficient management process.”[47]

A 2006 article published in the Australian Journal of Public Administration came to a similar conclusion. This study of a state-level multi-stakeholder public engagement effort in Australia found that a department’s new approach, which focused on collaborations between civil society and private sector, “impeded the ability of the project to facilitate participation by diverse stakeholders in the decision-making process” because the owners did not clearly articulate how each partner’s work would fit together. Lucia Boxelaar and co-authors argue that as the project owner, public administrators should continuously adapt their strategies to create more inclusive public engagement environments that allow diverse stakeholders to participate.[48]

For an overview of the literature on partnerships, see Amelia Clarke and Andrew Crane’s literature review “Cross-Sector Partnerships for Systemic Change: Systematized Literature Review and Agenda for Further Research”, and John D. Donahue and Richard A. Zeckhauser’s book Collaborative Governance: Private Roles for Public Goals in Turbulent Times.[49]

5. Can you select the incentives that will encourage participation?

Designing incentives to promote participation is a well-studied area within the field of behavioral psychology and other fields. Scholars differentiate between intrinsic motivation (where behavior is inspired by one’s own desire to achieve an outcome) and extrinsic motivation (where an external influence shapes behavior).[50] So, the major question for CI initiatives is choosing between intrinsic and extrinsic incentives.[51] The literature is almost equally divided between those arguing the merits of one over the other.[52]

Proponents of extrinsic incentives contend that participants will exert effort proportional to the reward they expect for their performance.[53]

Many have studied how extrinsic motivators such as cash prizes can create incentives for public participation - an approach known as a prize-backed challenge. McKinsey & Company’s 2009 practice guides, which draws learning from 12 prize-backed challenges run by nonprofits, found that successful open innovation competitions incentivize participation by linking the prize to a larger goal, investing resources in the design of the prize, and celebrating the winner.[54] Others have hypothesized that extrinsic incentives provide a “trophy” value in addition to any monetary value they may provide.[55]

Framing the task as part of a larger project can make extrinsic motivators more effective. In a study of financial incentives offered to workers on Amazon’s Mechanical Turk (an online marketplace for crowdsourcing human intelligence tasks) Winter Mason and Duncan J. Watts found that increasing compensation offered to workers increased the quantity of work performed but not the quality. However, changing the incentive mechanism from a piece-rate system to a quota system did improve work quality. In other words, offering workers an incentive to complete a larger project, rather than multiple incentives to complete many smaller tasks, increased the quality of work performed even though the total compensation and amount of work were the same.[56]

On the other hand, many have found that offering extrinsic incentives undermines the value of intrinsic motivation. Edward L. Deci, Richard Koestner,and  Richard M. Ryan’s large meta-analysis of 128 studies found that offering rewards causes participants to feel that the activity is less valuable. The authors also noted that offering extrinsic rewards also introduces surveillance, monitoring, and competition, which have negative effects on intrinsic motivation.[57]

Likewise, others have found that people tend to participate when they know that their participation will be relevant and used and have argued that intrinsic motivators are more powerful.[58] An unpublished dissertation conducted by a group of international academics studied participant motivations in two long-lasting citizen science initiatives from Finland and Japan.[59] Victoria Palacin and co-authors found that the projects sustained engagement by linking the work of the crowd to meaningful outcomes, and in some cases, by creating tasks that participants enjoyed doing. Creating a transparent process that links the group’s work to their own intrinsic desires to do good and enjoy their work can thus act as an incentive.[60]

Scholars have also found that simply offering a prize is not enough to incent participation. In 2012, Elisabeth Costa and David Halpern of the UK Behavioural Insights Team conducted a randomized controlled trial where households who registered to vote were entered into a lottery to win a cash prize. The study found that households in low-income areas of London were more likely to register to vote, but households in wealthier neighbourhoods experienced no effect.[61] Therefore, there is a need to match the incentives to the intended participants.

In the middle ground, some have proposed offering a combination of intrinsic and extrinsic incentives. In “The Collective Intelligence Genome”, Thomas W. Malone defines a “genome” of  16 building blocks that characterize CI projects in the private sector. Each of these design decisions informs either 1) what is being done 2) who is doing it 3) why they are doing it or 4) how they are doing it. Malone stresses that CI exercises either impart the incentives of money, love or glory. In other words, in the absence of financial incentives (Money), project owners can appeal to participants’ enjoyment of the activity, the opportunity to socialize, or the contribution they are making to a larger cause (Love), or to the desire for their contributions to be recognized by their peers (Glory).[62]

Still others have studied how a third variable, other than the incentive itself, affects the quality of participation.

The design of the platform, for instance, may be a motivator for participation. A 2019 report by the UK Behavioural Insights Team studied how design features like “swipes, likes, and streaks” on websites and social media platforms encourage user engagement, an approach companies often use to exploit consumers by driving up page views, purchases, and sharing of “fake news”.[63] While the report looked at manipulative techniques designed by companies to harm consumers, a public institution could use the same underlying concepts from behavioral psychology to encourage user engagement in a collective intelligence initiative.

In the same vein, a 2014 field experiment conducted by researchers at Northeastern University and Carnegie Mellon University examined both prizes and platform design as incentives for collaboration. Professors Christoph Riedl and Anita Williams Woolley analyzed how incentives, working strategies, and platform design affected a team’s performance in a virtual problem-solving challenge. The study found that while team-based cash incentives improved overall performance, they did not lead to greater collaboration within teams, and as such, “incentives should be accompanied by other tools and structures that encourage collaboration”, such as online platforms that facilitate information sharing.[64]

6. Can you define the tasks?

Crowds can perform a wide range of tasks from predicting the value of future financial assets, to developing software, to creating ontologies. Their performance across these tasks is similarly varied.[65] Richard Hackman developed a typology for understanding the multiplicity of tasks that people acting in groups undertake.[66]Groups do different things. Track teams perform “disjunctive tasks” where performance is measured by the best member. There are “conjunctive tasks” like a roped together group of climbers, who perform at the level of the least competent member. “Additive” tasks like a tug of war or many crowdsourcing projects are the sum of the parts. “Compensatory” tasks are those specific kinds of crowdsourcing projects that average the results of member contributions like a prediction market. “Complementary” tasks involve the assembly of smaller parts into a whole such as we find on Wikipedia. When we begin to take apart the different ways that groups function, (Hackman studied all of these), we begin to gain insight into how to design ways for groups to participate in different tasks of governance.

In his book Infotopia: How Many Minds Produce Knowledge, Cass R. Sunstein writes that groups are good at finding, assimilating and analyzing large quantities of information. Thus, some of the most successful examples of collective intelligence are efforts where large crowds aggregate and synthesize large amounts of information (such as online wikis, or open source software) and use the information collected to draw insights and inform action (such as prediction markets). However, Sunstein contends that deliberations have failed to effectively use the collective intelligence of participants because individuals too often do not disclose all that they know, for instance because they hold a minority opinion. Therefore, tasks should be designed to incent participants to share all their information about the problem at hand.[67]

In a 2010 literature review, Karan Girotra, Christian Terwiesch, and Karl T. Ulrich found that groups are better at idea generation than idea evaluation.[68] Eric Bonabeau speculates that this is because individuals “are far weaker explorers than evaluators.” For Bonabeau, individuals find it easier to recognize patterns and organize information (tasks used for evaluation) than to seek out a diversity of inputs of viewpoints (tasks used for idea generation).[69]

Daren C. Brabham’s 2013 report, published by the IBM Center for The Business of Government, identifies four types of crowdsourcing approaches used by government, distinguished by the task the crowd performs: Knowledge Discovery and Management (collecting information), Distributed Human Intelligence Tasking (performing “micro-tasks”), Broadcast Search (open innovation competitions), and Peer-Vetted Creative Production (idea generation and solution identification, performed by a crowd of peers). Public institutions should choose the task depending on whether the problem is an information management or ideation problem, and whether the needed information is internal or external to the institution.[70]

The tasks should be designed to match the abilities and interests of participants to the work they perform. In an excerpt from her 2009 book Wiki Government: How Technology Can Make Government Better, Democracy Stronger, and Citizens More Powerful, Beth Simone Noveck uses the example of a patent examiner to illustrate how traditional institutions have failed to accomplish this. Noveck argues that public servants often fail to make the right decisions because the bureaucracy leaves them to act as experts in fields outside of their areas of expertise. Public institutions can “build the bridge between institutions and network” by creating mechanisms that allow self-selected experts (and regular people with the right know-how) to participate in tasks that they find interesting and meaningful.[71]

The same is true for lawmaking projects based around collective intelligence - a practice The GovLab calls “crowdlaw”. In a 2018 essay, Beth Simone Noveck describes how governments are using new technologies to incorporate the collective intelligence of citizens into the lawmaking process. Noveck examines such initiatives across five stages of the lawmaking process. The author found that the most successful crowdlaw efforts are those whose task design matches the unique information needs of that stage to the skills, interests, and abilities of the participants.[72]

Regardless of the type of task, project owners should define discrete tasks that participants can easily complete. In an analysis of crowdsourcing contests, Haichao Zheng writes that organizers “should not design tasks that are ill structured and vaguely defined.” Zheng found that the most successful tasks were designed to be highly autonomous (participants could complete them on their own), explicitly specified, low-complexity, and required a variety of skills rather than tacit knowledge.[73] 

7. Can you design the process?

Another consideration in the literature is how to design a process that participants can reasonably understand, and that the institution can manage. Designing the process is a question of coordinating shared resources, relationships between participants, synchronization of tasks, the interrelation between tasks, group decision-making, and communication.[74] In our report, for example, we distinguish between crowdsourcing, collaboration and co-creation processes.

Mirroring the management structure of the initiative at large, this coordination can come from a combination of top-down and bottom-up forces. For an example of the former approach, Amy Kates and Jay R. Galbraith propose the “Star Decision Model” -  a framework for designing an organization based on five types of policies: Strategy, Structure, Processes, Rewards, and People. The design of the model emphasizes that management can balance each of these factors to avoid the pitfalls of central decision-making.[75]

On the other hand, Lee Rainie and Barry Wellman describe a kind of “choreography and exertion” that occurs within loose social networks. They write that members organically form specialized relationships to complete tasks together.[76] As a CI project combines institutional management strategies with distributed work and communication, it may use a combination of both of these approaches to coordinate among both the crowd and organization.

Clear and specific instructions are very important. With regard to task-oriented outcomes, Tara Behrend and co-authors show that when average participants are asked to perform technical tasks with specific instructions and detailed job classifications, their performance is equal to or better than the performance of experts.[77]

In a 2013 Harvard Business Review article, Kevin J. Boudreau and Karim R. Lakhani describe four types of workflows that put the work of the crowd to good use: contests, collaborative communities, complementors, and labor markets. They write that, when properly designed and deployed, these collaborations often outperform a corporation's own internal innovation unit.[78]

Careful planning is also needed to mitigate the risks associated with crowdsourcing. In a 2018 study, Joseph D. Tucker identifies risks at each of the six stages of running a crowdsourcing contest: organizing, soliciting, promoting, judging, recognizing, and sharing. By planning in advance, organizers can anticipate risks (such as a lack of community representation) and take measures to address them (such as forming a local steering committee).[79]

Nesta has written about these considerations before in the “The Collective Intelligence Design Playbook.” This toolkit provides “tools, tactics and methods” for practitioners to design and deliver a collective intelligence project. The playbook identifies four functions of collective intelligence: understand problems; seek solutions; decide and act; and learn and adapt. The toolkit provides worksheets and exercises that help to think through the challenges presented by each type of CI project and to generate solutions.[80]

8. Can you evaluate inputs?

A further consideration in the literature is how to create institutional readiness to evaluate and make sense of input from the group. After all, a network of distributed participants can end up generating a large volume of information. Much of the literature focuses on judging and evaluation criteria or platforms.

Collecting a large number of inputs from participants without the infrastructure to process them can leave project managers overwhelmed, making it difficult to collaborate and pull insights from the group’s work.

In examining a collaborative lawmaking experiment in Finland, a 2016 study by Tanja Aitamurto observed differences between a small expert group and a large, distributed crowd in terms of cognitive diversity, size, idea generation and evaluation method. These incongruities eventually led to an ineffective collaboration between the two groups, as their approaches were too dissimilar. The author concluded that “we need to develop better mechanisms for harnessing the crowd’s collective intelligence to policymaking”, with an emphasis on the need for “better systems for analyzing and synthesizing that input.”[81]

Some have suggested that artificial intelligence (AI) could aid in collective intelligence by helping to process large amounts of information. In a 2018 paper, Stefaan G. Verhulst, of The GovLab, posits that, if implemented effectively, AI capabilities “can at least partially automate and generally improve the processes of gauging, analyzing and even acting upon the inputs received from participants in CI initiatives”.[82] Analyzing the high volumes of citizen input collected on online platforms is sometimes better suited to the tools of natural language processing than the skills of public servants.

Likewise the 2020 report “The Future of Minds and Machines: How artificial intelligence can enhance collective intelligence,” written by Aleks Berditchevskaia and Peter Baeck of Nesta, identifies various ways AI interacts with and adds value to collective intelligence efforts. Most notably, AI can aid in the efficiency of data processing, optimise and prioritise processes, and help structure and organise knowledge, which all have great potential for aiding humans in evaluating inputs as part of a collective intelligence approach to problem solving.[83]

An important aspect of any open innovation competition is choosing judging criteria and judges. Clear judging criteria helps participants focus their work on usable and relevant solutions. Likewise, selecting experts as judges helps build legitimacy and is a good way to include experts who are not competing in the process. Choosing the criteria and judges early in the process helps build transparency and accountability.[84] For open innovation competitions run by US Federal Government agencies, these considerations are governed by the America COMPETES Act regulation.

Even if the open innovation exercise is not a competition, it is important to set impact metrics and collect the data to measure them. This is necessary to gauge the success of the initiative, including the quality of solution, the level of innovation achieved, and the impacts of the chosen solutions, and the effectiveness of the prize mechanism (including its cost-effectiveness). For more information about choosing impact and evaluation metrics, see Sections 3.4 and 3.7 of The MITRE Corporation’s report A Proposed Logic Model for Managing Open Innovation Competitions.[85]

9. Can you use what the group creates?

Finally, using what the group creates -- not just evaluating -- brings us back to questions of institutional readiness to use collective intelligence and asks what the institution needs to do in order to use information or decisions made by “the crowd.”

Public institutions must assess whether they are ready to use citizen engagement, or they risk squandering the work of participants. In the book The Social Media President: Barack Obama and the Politics of Digital Engagement, James E. Katz describes how the Citizen’s Briefing Book, a public engagement initiative developed by President Obama’s Transition Team, failed to use the inputs of citizens to inform policy. The Transition Team did not plan in advance how the administration would use citizens’ policy ideas, or even what the purpose of the initiative was beyond demonstrating an interest in open government. As a result, the Briefing Book “was an opportunity that did not end up realizing its full potential.”[86]

Likewise, the book Civic Tech in the Global South, edited by Tiago Peixoto and Micah L. Sifry, recounts numerous instances where public engagement tools provide citizens a “voice”, but lack the “teeth”, or the ability to go beyond transparency and translate citizens’ inputs into action. Thus, technology is not a panacea so long as institutions remain unwilling or unable to provide a meaningful response to citizen engagement.[87]

A similar challenge has been observed outside of the public sector. To give one example, Caroline Gottschalk Druschke and Carrie E. Seltzer’s 2013 reflection on a citizen science pilot project notes that while the initiative achieved its data collection aims, it did not effectively equip citizen scientists with the knowledge and confidence to engage in scientific efforts outside of the study - a goal of the initiative -, and thus failed to make participants “feel that this research—their research—matters.” The authors write that future projects can accomplish this goal by engaging with participants before, during, and after the initiative in order to more effectively address their needs and concerns.[88]

Organizations can avoid these risks by evaluating their readiness prior to launching a CI initiative. The Open Innovation Readiness Assessment Model (OIRAM) is one method  organizations can use to enhance readiness for an open innovation initiative. The model considers internal characteristics (such as knowledge management, organizational characteristics, and strategy) and external characteristics (such as collaboration approach, aim, and network retention). These considerations are needed to mitigate the risks of a CI effort failing.[89]

As Ulrich Lichtenthaler writes in a 2011 article, “Without a successful implementation process, the potential benefits of open innovation strategies will not materialize, and a company may lose some of its technological competitive advantage.”[90]

To address these issues, the crowd should be actively involved in evaluating and refining the process. In a 2016 paper from The World Bank, Tiago Peixoto and Jonathan Fox write that ICT-enabled citizen voice initiatives in the Global South have not influenced the willingness of decision makers to respond to users’ feedback, but they have influenced their capability to do so. In other words, “where senior managers are already committed to learning from feedback and using it to bolster their capacity to get agencies to respond, ICT platforms can make a big difference.”[91]

Some have proposed models to make this feedback easier to collect and use. In a 2017 study of crowdsourcing platforms for microtasking work, Mark E. Whiting noted that the distributed nature of crowd work makes it difficult to accurately evaluate the reputation of crowdsourcing platforms. Whiting proposes a “guild” model where members of the crowd would one another’s work, and in doing so involve crowd members in redesigning their platforms based on their peers’ feedback.[92]

Those initiatives that are successful should publicize their success to make the crowd know that their work is relevant. Gwanhoo Lee and Young Hoon Kwak’s 2011 paper, which used the experience of US Federal Government agencies to develop a series of guidelines for implementing open government initiatives, came to this conclusion. The authors emphasize the importance of linking the outcomes of such initiatives to change both within and outside the organization. Lee and Kwak conclude that project leaders should widely publicize the progress of the initiative at each stage, and ultimately should “recognize, celebrate, and advertise success stories of public engagement to facilitate cultural change towards openness, transparency, sharing, and collaboration.”[93]

To bring this discussion full circle, scholars and practitioners have emphasized the importance of linking the group’s work to institutions’ larger aims as a means of making collective intelligence more effective. A well-designed project can transform how the public bureaucracy operates, a strategy Marco Steinberg describes as a “strategic design challenge.” Steinberg argues that the solution to working within linear bureaucracies, which focus on administering existing services rather than innovation, is about “disrupting people without disrupting institutions.”[94] Likewise, in his book How to Run A Government: So that Citizens Benefit and Taxpayers Don't Go Crazy Michael Barber writes that “The widely held view that the way to bring about change is to ‘win hearts and minds’ and then proceed is largely a myth. In fact, the reverse is true: you need to proceed, and if you do so well - a very big and important ‘if’ - hearts and minds will follow.”[95]

Conclusion

In this literature review of eight dozen articles and books, we provide a cross-disciplinary look at “collective intelligence literature” from a range of fields in order to explain the concept of collective intelligence and how it is being used to solve problems.

Google N-Gram Viewer Showing the Dramatic Rise in the Use of “Collective Intelligence”

The literature refutes the notion that groups engage only in mob rule. Rather, our findings demonstrate that productive collaboration is possible. This has deep implications both for restoring faith in participatory democracy and for reinvigorating the practices of citizen engagement.

Harvard scholar Mark. H Moore writes that the way the public sector creates public value is by undertaking work in the middle of the “strategic triangle”, where legal and political support is on one vertex, and administrative and operational feasibility are on the others.

Like Tom Malone’s concept of the Collective Intelligence genome, our nine-part structure endeavors to find that sweet spot by describing the components of an effective collective intelligence project. Those components -- from the design of the process to the delineation of the tasks to using the group’s input -- derive from extensive literature on how groups work online and off and are bolstered by observations from the original qualitative research conducted for this accompanying report. Our hope is that this report and this literature review and analysis not only answer the question - how do I create an effective collective intelligence project -- but make the urgent case for doing so whenever possible to bolster the effectiveness and legitimacy of decisionmaking.