Skip to main content

Six Unconventional Methods for Evaluating Large Number of Ideas

Idea Evaluation Methods to Screen Large Number of Ideas

By Tojin T. Eapen

Organizations are often faced with a “problem of plenty” when they have too many ideas but are unable to effectively evaluate these ideas. A large number of ideas are sometimes garnered by organizations as outcomes of crowdsourced contests, suggestion boxes, or brainstorming sessions. A well-known illustration of this phenomenon was demonstrated in BP's crowdsourcing challenge, which generated an overwhelming 43,000 ideas. Organizers of the challenge struggled with evaluating the ideas and selecting ideas that were beneficial to the organization. Moreover, using generative AI tools in the ideation process typically results in a large number of ideas such as large numbers of product design concepts being generated automatically using to text-to-image models. 

While having a large number of ideas is a mostly good problem to have, it can also present significant difficulties for organizations. Specifically, it becomes increasingly challenging to identify the top ideas in a fair and transparent manner. It also becomes challenging to integrate or combine useful ideas in a meaningful way. 

One common solution is using crowds to evaluate ideas. In such cases, the crowd of evaluators can like or vote on ideas. The ideas that receive the highest votes or likes increase their chance of being recognized as the top idea. In some cases, this is an elegant and feasible solution. In other situations, the use of crowds can result in highly biased evaluations. Some evaluators may be individuals who have contributed ideas or are connected to such individuals. Some ideators may actively canvass votes, which can again impact the accuracy of the evaluations.

Common Challenges in Idea Evaluation

Idea Evaluation Methods to Screen Large Number of Ideas

Evaluating and selecting the most promising idea from a large pool of ideas can be a daunting task for organizations. There are several key challenges associated with idea evaluation and selection, which can have a significant impact on the overall effectiveness of the process.

Firstly, when dealing with a high number of ideas, it is common to encounter many "junk ideas." These ideas can take various forms, including trivial, far-fetched, or irrelevant ideas. These ideas can distract evaluators and dilute the pool of truly valuable and innovative ideas. To eliminate these junk ideas, it may be helpful to use automated approaches similar to spam filters. One could consider how semantically similar an idea is to the challenge statement or to other ideas. Ideas that are extremely different, maybe junk or spam ideas. However, one challenge with automated approaches to identifying junk ideas is that ideas that very novel ideas may appear like junk ideas. 

Another challenge with idea evaluation is the lack of independence in the evaluation process. For instance, the novelty of ideas is a key factor in evaluating their potential value. However, if two ideas that are novel, but similar to each other, are presented to an evaluator consecutively, they may find the first idea that is evaluated to be novel but not the second. To minimize this effect, it may be helpful for an evaluator to first scan all the ideas before making an assessment.

Additionally, an organization may be interested in ideas that may be complementary and may need to be combined with each other to fully realize their potential. This requires an evaluation process that not only assesses the independent potential of individual ideas but also their compatibility with other ideas and their ability to work together in a cohesive manner.

Finally, there is often a tradeoff between the novelty and feasibility of ideas. While it may be tempting to prioritize innovative and unconventional ideas, these ideas may also be less feasible to implement. On the other hand, more practical and feasible ideas may lack the excitement and potential impact of more novel ideas. Organizations must find a balance between these two factors to effectively evaluate and select the most promising ideas. However, given the negative correlation between idea novelty and feasibility, simply combining the novelty and feasibility scores can eliminate both highly novel ideas as well as easily implementable ideas. In such a situation, it may be helpful to create three separate evaluation buckets for ideas, highly novel ideas, highly feasible ideas, and ideas that are both novel and feasible.

Six Unconventional Idea Evaluation Methods

Idea Evaluation Methods to Screen Large Number of Ideas

Here we present six approaches for evaluating a large number of ideas when there are limited evaluators available. 

Evaluation by Elimination: In this strategy, the goal is to eliminate at least one idea at a time. The objective is to eliminate an idea that is clearly not a top or winning idea, rather than necessarily the worst idea.

Evaluation by Partitioning: In this strategy, ideas are divided into two groups: top ideas and bottom ideas. In each pass, the goal is to reduce the number of ideas to be evaluated by approximately half.

Moving Window Evaluation: In this approach, only the top idea inside a moving window is selected. For instance, if there are 100 ideas in total, each window may contain five ideas. In the first five ideas, the top idea is selected, then the window advances by five ideas, and the top idea is selected again. Once all the ideas have been evaluated at least once, the process repeats with ideas that have been selected in the previous round. The moving window can also be used to eliminate ideas.

Evaluation using Overlaps: In this approach, an idea moves to the next stage only if more than one evaluation agrees that it is a top idea. Similarly, an idea is eliminated only if multiple evaluations agree that it is a poor idea. These evaluations can be the same evaluator who goes through all ideas and ranks them multiple times, or by independent evaluators.

Evaluation by Binning: In this approach, the ideas are segregated into three bins: a bin for ideas with the potential to be a top idea, a bin for ideas that are clearly not going to be a top idea and should be eliminated, and a bin for ideas that the evaluator is unsure about. The middle bin is then taken up again, and the process is repeated with more time spent per idea in every subsequent round.

1-2-3-Tap Evaluation: In this approach, the evaluator assigns 1, 2, or 3 points or “taps” to an idea based on its perceived level of quality, with the number of taps representing the evaluator's beliefs, not a score. Ideas that do not receive any taps are eliminated, and the process is repeated with the remaining ideas. Ideas that receive 1 tap in two rounds or receive no taps in the second round are also eliminated. Equal amounts of time are allocated for each evaluation round. 

Discussion

Organizations are excited to see a large number of ideas as part of idea contests and innovation challenges. However, the task of identifying the top ideas to take to the next stage of the idea management process is not an easy task. Evaluators are required to scan through hundreds if not thousands of ideas. Some of the methods we have identified could potentially help in simplifying the idea evaluation process. 

Further Reading

To learn how leading Fortune Global 500 companies such as ABB, Bosch, Google, Samsung, and NetApp have used Innomantra's Functional Innovation Methodology to turbocharge their idea management process, schedule a meeting today at calendly.com/innomantra.

Comments

Popular Posts

Camelar: AI Product Ideation for Camel Inspired Cars

By Tojin T. Eapen We used AI tools ( chatGPT and Stable Diffusion ) to generate concept cars ("Camelars") that are inspired by camels, which are known for their exceptional ability to survive and thrive in rugged and challenging environments.  We wanted Camelars to ideally include features and capabilities that would allow them to perform well in conditions such as rough terrain, extreme temperatures, and limited resources. For this, we generated the following description of the Camelar, a bioinspired car that borrows from the appearance and characteristics of the camel. Generate an image of a car inspired by a camel, designed for long distance travel through harsh or remote environments. The car should have a spacious and comfortable interior with amenities like a built-in kitchen and sleeping quarters, as well as storage compartments for supplies and equipment. The exterior should feature a rugged and durable design, with features like high ground clearance, all-terrain ti

Empathy and Confrontation in Idea Generation

By  Tojin T. Eapen Successful innovation often involves two key factors: empathy and confrontation .  Empathy, or the ability to understand and share the feelings of others, is important in both art and science. In art, empathy with human subjects allows artists and writers to create relatable works. In science, empathy with non-human entities and abstract concepts allows investigators to understand them deeply and intuitively. The second key factor in innovation is confrontation, or the clash of ideas , perspectives, or reference frames. While empathy and confrontation may seem contradictory, both are essential for successful innovation, and one often leads to the other. According to MIT professor Edward Roberts , innovation is the combination of invention and exploitation. Theresa Amabile defines innovation as the successful implementation of creative ideas within an organization.  The term innovation can be seen as a portmanteau word that encapsulates its own ingredients: in spira

Generative AI for Bioinspired Product Ideation

By Tojin T. Eapen The design of products, processes, and organizations guided by principles observed in living systems can be referred to as " Bioinspired System Design ." In a series of posts, we delve into the potential of generative artificial intelligence (AI) to generate bioinspired product design concepts as a part of the idea management process. Specifically, we will look at how living organisms can serve as inspiration to redesign common products and human artifacts including bags, cars, bags, pens, tanks, trains, and umbrellas. In each of these articles, we will examine how the unique characteristics and behaviors of a particular living organism can be incorporated into the design of the bioinspired product. Elephantcopter: AI Designed Elephant Inspired Helicopters Camelar: AI Product Ideation for Camel Inspired Cars Koafa: AI Product Ideation for the Koala Inspired Sofas Paradiso: AI Product Ideation for Birds-of-Paradise Inspired T-Shirts Tigoes: AI Product Ideati

The Efficiency-Resilience-Prominence (ERP) Framework

Consider any living organism and its struggle for survival in a changing environment. Three crucial factors are common to all living systems: resource management, especially energy resources; coping with environmental forces such as heat, wind, and currents; and managing relationships with other entities, which can range from friendly to predatory.  These three factors are referred to as survivability concerns. To increase survival, an organism must adapt and manage these concerns, either through biological means like specialized organs, or behavioral means such as action and strategy. Organizations also face these same concerns of resources, forces, and relationships in their quest for survival.  Each living system has three corresponding capability factors: efficiency in managing resources, resilience against environmental forces, and prominence in attracting or evading attention. These three capabilities are collectively known as the ERP factors.