Virtually all innovation processes include generating and selecting opportunities or ideas. When a movie studio creates a new feature film, it typically considers several hundred plot summaries, a few of which are selected for further development. When a company develops the branding and identity for a new product, it creates dozens or hundreds of alternatives, and picks the best of these for testing and refinement. When a consumer goods firm develops a new product, it typically considers many alternative concepts before selecting the few it will develop further. Generating the ideas that feed subsequent development processes thus plays a critical role in innovation.
The success of idea generation in innovation usually depends on the quality of the best opportunity identified. For most innovation challenges, an organization would prefer 99 bad ideas and 1 outstanding idea to 100 merely good ideas. In the world of innovation, the extremes are what matter, not the average or the norm (Dahan and Mendelson (2001), Terwiesch and Loch (2004) Terwiesch and Ulrich (2009)). This objective is very different from that in, for example, manufacturing, where most firms would prefer to produce 100 units with good quality over making 1 unit with exceptional quality followed by 99 that have to be scrapped. When generating ideas, an organization makes choices by intention or default about its creative problem solving process.
In this paper, we investigate two commonly suggested organizational structures for groups generating ideas. In the first, individuals work together as a team in the same time and space. The team approach is widely used in organizations (Sutton and Hargadon (1996)). Despite its wide usage, hundreds of experimental studies have criticized team processes as relatively ineffective (cf. Diehl and Stroebe (1987), Diehl and Stroebe (1991)). In the second approach, individuals work independently for some fraction of the allotted time (sometimes called a nominal group) and then work together. This hybrid process has been suggested and studied in the prior literature as a way of effectively combining the merits of individual and team approaches. (cf. Robbins and Judge (2006), Paulus, Brown et al. (1996), Stroebe and Diehl (1994)). These studies find that the hybrid approach leads to more ideas and to higher satisfaction with the process among participants. While the two approaches have been compared for many years (Pugh (1981)) the prescriptions in the literature conflict.
The existing idea generation literature (often called the brainstorming literature) exhibits three gaps with respect to idea generation in innovation management. First, most papers focus on the number of ideas generated, as opposed to their quality, with the tacit assumption that more ideas will lead to better ideas. Second, the few papers that look at the quality of ideas look at the average quality of ideas as opposed to looking at the quality of the best ideas. Third, the focus of the existing literature is entirely on the creation process, and ignores the selection processes that groups use to pick the most promising ideas for further exploration. Given our focus on the use of idea generation in innovation, our metric for effectiveness is the quality of the ideas selected as the best.
Building on prior work on innovation tournaments and on extreme value theory applied to innovation, we build theory that relates organizational phenomena to four different variables that govern the underlying statistical process of idea generation and selection: (1) the average quality of ideas generated, (2) the number of ideas generated, (3) the variance in the quality of ideas generated, and (4) the ability of the group to discern the quality of the ideas. Each of these variables affects the quality of the best ideas produced by a team or a hybrid group. We report on a laboratory experiment, which compares the two group structures with respect to each of these four variables individually, and which measures their collective impact on the quality of the best idea. An accurate measurement of idea quality is central to our work. While most prior research has relied on the subjective evaluation of idea quality by one or two research assistants, we use two alternative approaches: a web-based quality evaluation tool which collects about 20 ratings per idea and a purchase-intent survey which captures about 40 consumer opinions about their intent to purchase a product based on the idea. Our framework, with its emphasis on the importance of the best idea, and our novel experimental set-up let us make the following three contributions.
1. We find evidence that the best idea generated in the hybrid structure is better than the best idea generated by a team. This result is driven by the fact that the hybrid structure results in about three times as many ideas per unit of time and that these ideas have significantly higher average quality.
2. We find that the hybrid structure is better at identifying the best ideas from the set of ideas it previously generated. However, we also find that both team and hybrid structures are, in absolute terms, weak in their ability to assess the quality of ideas.
3. We show that idea generation in teams is more likely to lead to ideas that build on each other. However, in contrast to the common wisdom articulated by many proponents of team brainstorming, we show that such build-up does not lead to better idea quality. In fact, we find that ideas that build on a previous idea are worse not better, on average.
The remainder of this paper is organized as follows. We review the relevant literature in Section 2. We then develop our theoretical framework in Section 3 and state our main hypothesis. Section 4 describes the experiment and Section 5 describes our performance measures. Section 6 reports our main results. Section 7 looks at the role of build-up in groups, and Section 8 contains concluding remarks.
The role of organizational processes in idea generation has been examined in the social psychology literature and in the innovation management literature. The social psychology literature has examined the idea generation process in detail in what is often called the brainstorming literature. The innovation management literature has focused on innovation outcomes and organizational forms. The social psychology literature mostly originates with Osborne’s 1957 book, Applied Imagination (Osborne (1957)), which introduces the term brainstorming. Osborne argued that working in teams leads to multiple creative stimuli and to interaction among participants, resulting in a highly effective process. His argument spawned many experiments. Diehl and Stroebe (1987) and Mullen, Johnson et al. (1991) provide a detailed overview of this literature.
These studies experimentally examined groups generating ideas as teams or as individuals. In terms of performance metrics, the literature focuses on the average quality of the ideas generated, the number of ideas generated, and measures that combined the two such as the total quality produced. Quality ratings for ideas generated are typically provided through evaluations by research assistants. For example, in Diehl, M., and W. Stroebe (1987), the ideas were rated by one research assistant and a second assistant was used to verify the reliability. The research has unequivocally found that the number of ideas generated (i.e., productivity) is significantly higher when individuals work by themselves and the average quality of ideas is no different between individual and team processes. (All of these studies normalize for total person-time invested to control for differences in the numbers of participants and the duration of the activity.) Given the focus of these studies on productivity and average quality, they conclude that team processes are inferior to individual processes.
However, this main conclusion is in stark contrast with Osborne’s hypothesis, empirical work that examines metrics such as variance of the quality distribution (Singh and Fleming (2009)), and to anecdotal evidence that team idea generation processes (i.e., brainstorming) are widely used in organizations. In line with the social psychology literature we also conduct experiments. However, in contrast to this literature, we examine idea generation in the specific context of generating ideas in response to an innovation challenge. Given the focus on innovation, we are concerned with the quality of the best ideas resulting from the idea generation process, not with the average quality. Furthermore, we depart from this literature by employing a novel method of evaluating idea quality based on a large panel of independent raters and on a purchase-intent survey conducted with subjects from the target market segments.
To resolve the contrast between the social psychology literature and anecdotal evidence about the practices of real organizations, Sutton and Hargadon (1996) conducted a field-based observational study of the product design consulting firm IDEO. They found that contextual differences between the lab and the real world (e.g., the nature of problems addressed) may explain the contrast between practice and the laboratory findings. More recently, Kavadias and Sommer (2007) take an innovative approach to this paradox.
They show analytically that the specific nature of the problem and group diversity matters, and they conjecture that the experimental evidence may be an artifact of exploring simple idea generation problems which are not representative of real situations. The role of organizational structure in the idea generation process has also been examined with large-scale empirical studies, most notably by Singh and Fleming (2009), who use patent data to study differences in quality variance between inventors who work by themselves and those who collaborate. They examine the impact of collaboration on both the upper and lower tails of the quality distribution. Quality is measured as the number of citations received by the patent. Taylor and Greve (2006) also examine average quality and variance of creative output in the comic book industry. They measure quality as the collector-market value of a comic. Singh and Fleming (2009) find an asymmetric impact of collaboration, in that collaboration reduces the number of bad ideas and increase the number of very good ideas.