Organizations are often faced with a “problem of plenty” when they have too many ideas but are unable to effectively evaluate these ideas. A large number of ideas are sometimes garnered by organizations as outcomes of crowdsourced contests, suggestion boxes, or brainstorming sessions. A well-known illustration of this phenomenon was demonstrated in BP's crowdsourcing challenge, which generated an overwhelming 43,000 ideas. Organizers of the challenge struggled with evaluating the ideas and selecting ideas that were beneficial to the organization. Moreover, using generative AI tools in the ideation process typically results in a large number of ideas such as large numbers of product design concepts being generated automatically using to text-to-image models.
While having a large number of ideas is a mostly good problem to have, it can also present significant difficulties for organizations. Specifically, it becomes increasingly challenging to identify the top ideas in a fair and transparent manner. It also becomes challenging to integrate or combine useful ideas in a meaningful way.
One common solution is using crowds to evaluate ideas. In such cases, the crowd of evaluators can like or vote on ideas. The ideas that receive the highest votes or likes increase their chance of being recognized as the top idea. In some cases, this is an elegant and feasible solution. In other situations, the use of crowds can result in highly biased evaluations. Some evaluators may be individuals who have contributed ideas or are connected to such individuals. Some ideators may actively canvass votes, which can again impact the accuracy of the evaluations.
Common Challenges in Idea Evaluation
Evaluating and selecting the most promising idea from a large pool of ideas can be a daunting task for organizations. There are several key challenges associated with idea evaluation and selection, which can have a significant impact on the overall effectiveness of the process.
Firstly, when dealing with a high number of ideas, it is common to encounter many "junk ideas." These ideas can take various forms, including trivial, far-fetched, or irrelevant ideas. These ideas can distract evaluators and dilute the pool of truly valuable and innovative ideas. To eliminate these junk ideas, it may be helpful to use automated approaches similar to spam filters. One could consider how semantically similar an idea is to the challenge statement or to other ideas. Ideas that are extremely different, maybe junk or spam ideas. However, one challenge with automated approaches to identifying junk ideas is that ideas that very novel ideas may appear like junk ideas.
Another challenge with idea evaluation is the lack of independence in the evaluation process. For instance, the novelty of ideas is a key factor in evaluating their potential value. However, if two ideas that are novel, but similar to each other, are presented to an evaluator consecutively, they may find the first idea that is evaluated to be novel but not the second. To minimize this effect, it may be helpful for an evaluator to first scan all the ideas before making an assessment.
Additionally, an organization may be interested in ideas that may be complementary and may need to be combined with each other to fully realize their potential. This requires an evaluation process that not only assesses the independent potential of individual ideas but also their compatibility with other ideas and their ability to work together in a cohesive manner.
Finally, there is often a tradeoff between the novelty and feasibility of ideas. While it may be tempting to prioritize innovative and unconventional ideas, these ideas may also be less feasible to implement. On the other hand, more practical and feasible ideas may lack the excitement and potential impact of more novel ideas. Organizations must find a balance between these two factors to effectively evaluate and select the most promising ideas. However, given the negative correlation between idea novelty and feasibility, simply combining the novelty and feasibility scores can eliminate both highly novel ideas as well as easily implementable ideas. In such a situation, it may be helpful to create three separate evaluation buckets for ideas, highly novel ideas, highly feasible ideas, and ideas that are both novel and feasible.
Six Unconventional Idea Evaluation Methods
Here we present six approaches for evaluating a large number of ideas when there are limited evaluators available.
Evaluation by Elimination: In this strategy, the goal is to eliminate at least one idea at a time. The objective is to eliminate an idea that is clearly not a top or winning idea, rather than necessarily the worst idea.
Evaluation by Partitioning: In this strategy, ideas are divided into two groups: top ideas and bottom ideas. In each pass, the goal is to reduce the number of ideas to be evaluated by approximately half.
Moving Window Evaluation: In this approach, only the top idea inside a moving window is selected. For instance, if there are 100 ideas in total, each window may contain five ideas. In the first five ideas, the top idea is selected, then the window advances by five ideas, and the top idea is selected again. Once all the ideas have been evaluated at least once, the process repeats with ideas that have been selected in the previous round. The moving window can also be used to eliminate ideas.
Evaluation using Overlaps: In this approach, an idea moves to the next stage only if more than one evaluation agrees that it is a top idea. Similarly, an idea is eliminated only if multiple evaluations agree that it is a poor idea. These evaluations can be the same evaluator who goes through all ideas and ranks them multiple times, or by independent evaluators.
Evaluation by Binning: In this approach, the ideas are segregated into three bins: a bin for ideas with the potential to be a top idea, a bin for ideas that are clearly not going to be a top idea and should be eliminated, and a bin for ideas that the evaluator is unsure about. The middle bin is then taken up again, and the process is repeated with more time spent per idea in every subsequent round.
1-2-3-Tap Evaluation: In this approach, the evaluator assigns 1, 2, or 3 points or “taps” to an idea based on its perceived level of quality, with the number of taps representing the evaluator's beliefs, not a score. Ideas that do not receive any taps are eliminated, and the process is repeated with the remaining ideas. Ideas that receive 1 tap in two rounds or receive no taps in the second round are also eliminated. Equal amounts of time are allocated for each evaluation round.
Organizations are excited to see a large number of ideas as part of idea contests and innovation challenges. However, the task of identifying the top ideas to take to the next stage of the idea management process is not an easy task. Evaluators are required to scan through hundreds if not thousands of ideas. Some of the methods we have identified could potentially help in simplifying the idea evaluation process.
- Acar, O. A. (2019). Why crowdsourcing often leads to bad ideas. Harvard Business Review.
- Dean, D. L., Hender, J., Rodgers, T., & Santanen, E. (2006). Identifying good ideas: Constructs and scales for idea evaluation. Journal of Association for Information Systems, 7(10), 646-699.
- Licuanan, B. F., Dailey, L. R., & Mumford, M. D. (2007). Idea evaluation: Error in evaluating highly original ideas. The Journal of Creative Behavior, 41(1), 1-27.