1 Introduction

Today, the absorption of external knowledge has become crucial for firms’ product innovation (Chao-Ton et al. 2006; Robert and Candi 2014; Chuang and Lin 2015; Lichtenthaler 2016). In the era of “open innovation”, scholars, specialists and researchers plead for much more active participation of customers in new product development (NPD) than is seen with traditional market research (Von Hippel and Katz 2002; Chesbrough 2006; Prahalad and Ramaswamy 2004; Sarin and O’Connor 2009; Williamson and Yin 2014). To uphold the pace of innovation—as is necessary in a world of swiftly varying technologies and customer needs—West et al. (2014), Robert and Candi (2014), and Füller et al. (2006b), among others, have proposed integrating customers into value creation and utilising customers’ acquaintance to reinforce a firm’s key competences (Gawer and Cusumano 2014) as well as to comprehend their needs (Dahan and Hauser 2002; Blazevic and Lievens 2008; Bharadwaj et al. 2012). As a consequence, novel approaches are required to ensure the active integration of customers into NPD (Bharadwaj et al. 2012; Cooper 2014). It is, after all, only the customers themselves who are able to evaluate whether they like a new product and whether it fills a previously unmet and quite possibly unrecognised need (Franke et al. 2009; Noble et al. 2012; Robert and Candi 2014).

This situation is reinforced by the increasing amounts of data available to business and the associated data-driven efforts at innovation, by new information and communication technologies, as well as by new business models and organisational forms (Wamba et al. 2013; Bharadwaj and Noble 2015). According to IBM (2013), 90% of the data that exists in the world today was generated in the last 2 years and it is expected the global total of data will reach 35 zettabytes (ZB) by 2020 (Wong 2012; Gantz and Reinsel 2012). This is therefore the era of “big data” (McKinsey 2011; IBM 2013; Chan et al. 2015). A key competitive advantage in today’s rapidly changing business environment is the ability to extract big data to gain helpful business insights (Wong 2012; Tan et al. 2015). Being able to use big data allows firms to achieve outstanding performance against their competitors (Oh et al. 2012). For example, retailers can increase their operating margins by 60% through tapping into ‘hidden values’ in big data (Werdigier 2009).

Although large amounts of both capital and time may be required to build a big data platform and to install the necessary technologies, the long-term benefits provided by big data are vast (Terziovski 2010; Song et al. 2016). Studies show that big data plays a critical role in customer involvement. Many researchers point out that firm can better understand customers’ preferences and needs by leveraging the data available through loyalty cards and social media (Bozarth et al. 1998; Tsai et al. 2013; Wamba and Carter 2014). Big data can be defined as an interactive and large-scale information source resulting from low-cost mass communication (Urban and Hauser 2004; Dahan and Hauser 2002; Tan et al. 2015). It allows customers to better understand new products and provides simplified methods of multi-media rich interaction between customers and managers. A number of big data analytics and techniques can be found in the literature that relate to customer interaction (Yiu 2012; Zikopoulos and Eaton 2011; Tan et al. 2015). However, to the best of our knowledge, there is a lack of empirical studies that shed light on how to enhance customer involvement by using big data in practice. The current study mainly argues that there are huge potential values of big data that remain uncovered in new product development (LaValle et al. 2011; Davenport 2013; Robert and Candi 2014; Tan et al. 2015). In order to fulfil the gap, it leads to the following research questions concerning NPD:

  1. 1.

    How can customer involvement be improved via big data?

  2. 2.

    How can a firm interact with its customers and involve them in the NPD process?

To answer these questions, this study is organised as follows. First, this study provides an overview of the challenges associated with the recent evolution of approaches to customer involvement. Secondly, we point out that by means of big data, customer involvement can be summarised in a three-phase NPD approach which enables customers share their desires and previously unknown needs through trial-and-error loops. Thirdly, we conduct a longitudinal case study to illustrate how big data can be used to involve customers in NPD in practice. Finally, this paper discusses the implications of using big data to support customer involvement in NPD, discusses the limitations of this investigation, and offers some suggestions for future research.

2 Customer involvement

Companies have gradually been forced to reconsider their basic approach to the creation and marketing of new ideas (Urban and Hauser 2004; Cooper 2011. Traditional R&D has been considered costly and vague (Prahalad and Ramaswamy 2004; Steinfeld and Beltoft 2014). Customer involvement has been extensively employed in management rhetoric as an approach to stiffening the feedback loop between expenditure and production cycles (Urban and Hauser 2004; Robert and Candi 2014). Among such perceptions is the assumption that customers are important sources of information and knowledge (Rothwell 1994; Ortt and Duin 2008; Cooper 2014), and it is acknowledged that customer involvement can improve NPD (Cooper and Kleinschmidt 2011).

Tan et al. (2015) recently investigated how some firms are able to determine their customers’ needs and then innovate to meet those needs. Such companies are considered to be much more profitable than others (Cooper and Kleinschmidt 2011; Wong 2012; Chan et al. 2015). These champions of innovation are able to merge their ambitions and key abilities with their customers. Accordingly, a strong market orientation and the capability of managers to acquire customer needs are perceived as significant reasons for the development of new products (IBM 2013; Steinfeld and Beltoft 2014). For example, in the software industry, customer involvement through new techniques and methods has become popular. Bosch-Sijtsema and Bosch (2015) described ‘agile’ software developments, where customers are actively engaged in designing software and pooling resources with the development teams. Previous literature has demonstrated how firms are able to benefit from teaming up with customers or from acquiring customers’ feedback and input (Brown et al. 2002; Narver et al. 2004; Franke et al. 2009; Bharadwaj et al. 2012; Noble et al. 2012; Cooper 2014). Such methods and techniques come from both marketing research and R&D. They centre on three key phases in the NPD process: prospect identification; development; testing and product launch (Van Kleef et al. 2005; Zhan et al. 2016). Many studies have looked principally at the theoretical and early NPD phases (Schaarschmidt and Kilian 2014; Van Kleef et al. 2005). For example, as determined in the PIMS/IMD Brand Innovation Study, growth rates and market share are higher if customers’ assessment of the value of a new product is ascertained in the initial phases of NPD (Kashani et al. 2000). As a result, approaches are required that permit the active involvement of customers in NPD (Füller et al. 2006b; The Economist 2012; Robert and Candi 2014). Only if customers are able to better comprehend a new product will they be able to sensibly evaluate whether or not they like it and whether it fulfils a latent (and possibly hitherto unrecognised) need. Companies that are able to recognise customers’ latent needs and to have this data inform new product features or entire products will be much more likely to develop successful novel products (Sarin and O’Connor 2009; Robert and Candi 2014). As long as managers are able to identify the needs of customers, they will be capable of creating enhanced, customer-centred new products (McKinsey 2009; Noble et al. 2012; Bharadwaj and Dong 2014).

Nonetheless, there are challenges in involving customers in the development of new products and services (Nambisan 2002; Lundkvist and Yakhlef 2004; Sashi 2012; Bowden et al. 2015). As pointed out by Nambisan (2002), one key challenge is simply to get in touch with customers in an effective way. As suggested by Füller et al. (2006b), information related to customers’ needs is often costly for product managers to capture. Customer incentives, company identification and socialising are vital for customers to have their knowledge shared with NPD managers (Robert and Candi 2014). As stated by Janssen and Dankbaar (2008), customer involvement leads to a superior product, but they argue that further insight is necessary, from diverse sources, as a way of acquiring and analysing customer inputs. Nambisan (2002) states that customers’ motivations to contribute includes outcome control and enhanced self-esteem, while Antikainen et al. (2010) point out that community coordination and entertainment are additional factors. According to Nambisan (2002), engaging customers as NPD co-creators can result in greater project uncertainty, as so requires additional evaluation and control. In addition, customers may frequently require extra knowledge regarding the technology and product under assessment, resulting in a likelihood of customer training costs.

In short, the critical issues in customer involvement concern its cost-effectiveness, how to structure customer input, and how to achieve a broad representation of the customer base. This research aims to overcome these challenges by unlocking the power of big data. In the following section, we introduce the concept of big data, which can be used to improve customer involvement and enable managers to capture customers’ explicit and implicit knowledge to support NPD.

3 Big data in new product development

Today, technology has turned the average customer into an incessant generator of both transactional, traditional, structured data as well as more contemporary, unstructured, behavioural data (Wong 2012; Bauer and Leker 2013; Wamba et al. 2015). The magnitude of the data generated, the relentless rapidity at which data is constantly produced, and the diverse richness of the data are transforming NPD and decision making. This is therefore the era of “big data” (Wong 2012). Big data is characterised by its 3V characteristics (volume, velocity and variety) and can be generated through different information systems and technologies, including smartphone applications, online communities, sensor networks, internet clicks and social media platforms (McAfee et al. 2012; Chan et al. 2015). By studying the literature, we have identified three phases that big data can be used to support in NPD: generation of ideas and concepts; design and engineering; and test and launch. Potential roles and tasks that can be transferred to customers are demonstrated in each of these phases.

3.1 Generation of ideas and concepts

The initial phase is centred on the recognition and creation of opportunities, novel ideas and new product concepts (Van Kleef et al. 2005; Cooper 2014). Big data can be engaged in supporting this phase through the collection of huge amounts of external information to offer managers supportive product ideas (Gantz and Reinsel 2012; Tsai et al. 2013). Noteworthy here is the group of inventive customers categorised as ‘lead users’ (Bharadwaj et al. 2012). The information generated can be incorporated in proposals from the firm’s NPD teams (Davenport 2009; Füller et al. 2006b). For instance, Lenovo set up a competition for its customers that involved online services, telematics as well as future PC online assistance systems (Bel 2015). Novel ideas generation by the customers has been endorsed by an interactive multimedia tool for services, as well as assessing ideas generated by others. During the initial phase of NPD process, big data enables the integration of customers and turns them into valuable sources to support companies in ideas generation and evaluation (McAfee et al. 2012; Tsai et al. 2013).

3.2 Design and engineering

In the design and engineering phase, the term ‘co-creator’ (Dahan and Hauser 2002; Shu-Chuan and Kim 2011; Robert and Candi 2014) indicates the customer’s role more precisely. Six web-based approaches have been proposed by Dahan and Hauser (2002) that seek the engagement of the Internet users in an enhanced approach than the conventional market research approaches. For instance, a web-based approach can enable customers to design individual products that will meet their particular needs and wants (Schaarschmidt and Kilian 2014).

With such techniques, the advantage of using big data in customer involvement (assessed against conventional market research) is that customers are not only asked about their needs, opinions and wants. They can, rather, exhibit their creativity and competence by deriving and assessing new product ideas; they can challenge, explain and enhance detailed solutions; they can identify and individualise virtual prototypes, experimenting with and embracing the novel product features (Blazevic and Lievens 2008; Hoyer et al. 2010; Zhang et al. 2011; Chen et al. 2012). This can be achieved by conducting simulations, or by acquiring information from different sources regarding a novel product (Chen et al. 2012; Füller et al. 2006b). For instance, Chow Tai Fook Company (a Chinese company engaged in diversified businesses such as jewellery, property and casinos) instituted an internet-based design and launch competition; primarily, customers assessed Chow Tai Fook’s idea of ‘Forevermark magic’, a novel type of jewellery. Subsequently, an internet-based toolkit enhanced customers’ individual ‘Forevermark magic’ design. Within a timeframe of a single month, thousands of customers engaged in virtual dialogue and stated their personal preferences. The sampled individuals were able to create hundreds of appealing designs, which motivated Chow Tai Fook’s NPD teams in addition to aiding the assessment of customers’ latent needs.

3.3 Test and launch

In the test and launch phase, big data allows companies to transfer individuals from different sources (e.g. web-based communities, websites and platforms) into the roles of end customers or buyers (McKinsey 2011; McAfee et al. 2012; Wong 2012; Wamba et al. 2015). Previous studies have illustrated how customers can represent important resources for a company’s development of new products and services (Payne et al. 2008; Blazevic and Lievens 2008; Hoyer et al. 2010; Ahmad et al. 2013; Cooper 2014). Conventional, manufacture-centric innovation greatly limits the role of the customer (Wong 2012). For instance, previously customers (termed ‘eventual evaluators’) were often used to support companies in fixing bugs (Brown et al. 2002; Fuchs and Schreier 2011). Otherwise, customers were lucky to have any role at all. In contrast, customers can be seen as co-creators or co-developers in a big data environment (Prahalad and Ramaswamy 2004; Robert and Candi 2014). For example, Xuancai Company (a Chinese leading game company), in cooperation with China Telecom (one of the largest telecommunication SOE in China), constructed a customer-friendly online platform (PLAY.CN) to enable customers to compose and download their individual internet mobile java games without any special skills (e.g., programming). Enthusiasts of mobile gaming are acquainted with the novel service, platform testing, as well as downloading their self-designed games for their smartphones (China Tech News 2008). As a consequence, more than a million customers have offered their feedback regarding acceptance, usability, intention to play and willingness to pay. In this way, the customers were able to come up numerous improvement ideas for supporting a company’s NPD.

4 Research methodology

An in-depth case study is presented on the use of big data to improve customer involvement by STE, a young but innovative high-tech company, so as to draw lessons for the effective use of big data to improve customer involvement in NPD. A case study is considered a suitable research approach when exploring emerging complex phenomena (e.g., big data adoption and use) in real-life settings (Dyer and Wilkins 1991; Wamba et al. 2015; Dutta and Bose 2015). Besides, a case study is considered an appropriate approach when answering research questions such as ‘how’ and ‘why’ things are done (Yin 1994). Additionally, the case study approach is recommended for researches where theories are at their formative stage (Eisenhardt 1989; Yin 1994).

4.1 Research settings: a longitudinal case study of STE, an electronics company

Qualitative sampling, unlike quantitative sampling, tends to be purposive rather than random. The choice of informants, episodes and interactions is usually driven by a conceptual question (Miles and Huberman 1994). We followed Wamba et al. (2015) and conducted a longitudinal case study. In this particular case, one of the authors had the opportunity to step into a project started at STE in which the objective was to design and to develop a new wearable electronic headset. STE is a Chinese SME manufacturer founded in 2007. It is an innovator in wearable medical equipment. The company is best known for its wearable electronic headset, which can be used to monitor brain activity. The brain activity data is streamed to a smartphone or stored in the system. The data is then transmitted in real time to a receiver located up to 10 miles away. The company has stated that its main customers are patients in old age; the product can help doctors intervene earlier and avoid complications. However, since this market size is comparatively small and most doctors are not familiar with the product, sales were decreasing year by year. In addition, STE experienced challenges with a lack of understanding and customer empathy on the part of its engineering staff. In order to improve its market performance and gain competitive advantage, the company decided to innovate and to launch new products with improved functionality and a different market focus.

The main objective of the NPD project was to design and develop a new wearable electronic headset with a new virtual reality function for young people to better understand and control their minds. This is a monitoring device in which user will wear it while they are watching videos, searching the Internet or playing games. It provides users with accurate, real-time feedback on the brain activities. In this way, users can have a better understanding of active areas of the brain in different situations, such as relaxation, improved mood and reduced stress. Two main goals for the project were stated by STE:

  1. (a)

    Use big data to better understand customer upcoming trends, expectations and preferences, economically and interactively.

  2. (b)

    Gain insights concerning customer perceptions, acceptance and input through big data and customer involvement.

The company has its own information department for data collection and management in order to help meet both of these objectives. The project had four main stages: customer profile determination, identification of different information sources, customer involvement design, and customer access and participation.

4.2 Data collection

In this study, data collection covered multiple sources of evidence, which allowed us to increase the validity of our constructs (Yin 1994). Initially, a retrospective method (Miller et al. 1997) was applied to become familiar with the initial part of the project, when the concept took shape in 2013. To eliminate distortion, only written reports drawn upon in this wave of data collection. Then, starting in 2014, the subsequent data collection included 6 months of research in real time. The aim was to record how researchers worked in the company (Pettigrew 1990). For this wave of data collection, sources included: semi-structured interviews with key respondents, on-site observations, annual reports, industrial reports, technical or non-technical documents, newsletters, project reports, strategic planning reports, and discussions with the NPD team. One of the authors was actively engaged in project itself, in market research. This meant that every unit of market information was recorded in real time by one of the researchers. In total, 16 semi-structured interviews were conducted, each lasting 1–2 h. The initial interviews were kept broad in scope in an effort to cover a wide range of motivations, decisions and competences. As the research project progressed and the theory was refined, interview questions became more focused, in an effort to ascribe more details to the emerging patterns. STE’s CEO championed the project. The STE NPD team consisted of the CEO, NPD team manager, designers, functional specialists, electronics engineers, information analysts, researchers, and a marketing manager.

5 Findings

In this section, we will present key insights and lessons learned from the in-depth analysis of the longitudinal case study of STE. The company is currently using big data for improved customer involvement in development of a new product. The term ‘big data’ in this case refers typically to the following types of data: (a) traditional enterprise data, (b) social media data, and (c) machine-generated/sensor data (e.g. weblogs, cloud files, smart meters, manufacturing sensors, equipment logs).

5.1 Stage 1: Using big data to determine customer profile

STE used to have little direct feedback from customers. Only recently did STE start to monitor customer comments on social media about its products. The Company has implemented the product innovation strategy of ‘Customer Demand-Orientation’ and analysed customers’ behaviour by machine-style learning using both qualitative and quantitative data. STE Company gathered feedback from their customers as well as partners about their preferences. In order to identify each wearable electronic headset product and generate new ideas, the company collected different source of data such as videos, photos, number of comments and number of followers from the most popular websites (i.e. Amazon, Facebook, eBay) by using web page cleaning, web crawler and HTML parsing technologies. It is worth to mention that all these collected information has vast amounts of data where people produce and share every second and most of the information is unstructured data (i.e. photos, videos or social media) which means it cannot easily be put into tables. Moreover, take Facebook posts as an example, the data quality and accuracy are less controllable. Thus, in order to harvest great values from big data, the trustworthiness of the data is a significant issue that Company STE needs to address. The Company pointed out that data quality can be verified by complete and accurate data which includes values and variables relevant to the purpose of collecting them.

Additionally, STE has developed an application which is highly customisable. It allows its partners and customers to upload their ideas and suggestions to facilitate the company’s NPD and the invention of new features. In this way, potential targets were customers with an interest in becoming involved in developing the new wearable device. Some of these customers were highly innovative and able to offer valuable new product ideas. By visualising the big customer data collected in the past 3 months, early adopters of the high-end wearable device requested functions like Dynamic Streaming Technology, telematics, multi-channel recording, and voice control (see Fig. 1).

Fig. 1
figure 1

Levels of customer involvement for each new product function

Full size image

By making use of social data, the managers face the challenges to extract the useful information from the terabytes of text data. Different from other data sources, data from social media is no second intent opinion, but the data density is extremely low, i.e. the useful information is buried in the unstructured massive data. In order to unravel the hidden information, data scientist need to adopt some data-mining techniques such as frequency analysis, cluster analysis, and sentiment analysis. For example, customers have a discussion of new ideas of certain functions, a multidimensional scaling diagram (MDS) can effectively illustrate the clustering results of different opinion groups in the discussion (Tse et al. 2016). It is a useful approach to uncover information since people will use similar wordings but sentence different structures to express their ideas. Thus, it is useful to use clustering analysis with MDS diagrams to identify major opinion group from Facebook data. Also, username is another useful ‘metadata’ to identify gender of the post owner, so it will be easy to investigate different gender group towards the propose NPD ideas. The STE information department had the job of processing different customers’ suggested attributes of a new product in parallel. In particular, it applies different conventional data techniques to harvest useful information from big data. For example, Apache Mahout for machine learning algorithms in business, Tableau for big data visualisation, Storm for analysing real-time computation system and InfoSphere for big data mining and integration.

5.2 Stage 2: Using big data to identify information sources

STE’s information department is able to analyse customer data captured from different sources. STE Company has set up three research centres and innovation laboratories for acquiring huge data sources from its operators and target customer. Particularly, social media is a very important data source. For example, the main forum of STE on its official website posts (in different formats) more than 10,000 topics/day, including new product information, announcements, feedback and discussions. In this open community, there are 10s of 1000s of posts fed back by customers every week, of which some deep reports of product using came into being. By the way of integrating and analysing information of those posts, STE Company can acquire demands information of customers’ with low costs and high efficiency, providing innovative ideas for research and development of new products.

The Company emphasised the important big data source from social media that can create relations and provide a better understanding of the customers and their product usage and in that way improve the development. Particularly, ‘lead users’ can be differentiated from ‘normal customers’ by the information department through RFM analysis which is a data mining technique quantifying customer value by examining how recency, frequency and monetary a customer purchases (Hughes 1996). The information department analysed the customer information and activities around the STE brand (e.g. platform, communities, apps, and official websites). Figure 2 shows the STE network for customer connection and interaction. STE connects to its customers through a wide range of sources at low cost (e.g. official web forums, mobile apps, website communities). Using the same means, customers can also interact with the company and each other in real-time. The latest product information is updated to these STE sources on a daily basis, partly to attract more customers and partly to gain feedback for further developments. A vast number of these places were cultivated for different interested customers. In this way, the information department can collect a wide range of customer information from different channels and platforms extremely quickly. Data requirements could be different due to different organisations’ needs and problems. Then, a number of data pre-processing techniques, including data cleaning, data integration, data transformation and data reduction, can be applied to remove noise and correct inconsistencies from data sets. After that, data mining techniques can be used to help managers generate lots of useful information.

Fig. 2
figure 2

The STE network for customer involvement

Full size image

5.3 Stage 3: Using big data to improve customer involvement in product design

To come up with a customer involvement process, their individual qualities as well as inspirations should be considered in the design. According to Füller and Matzler (2007), the best solution to customer involvement is dependent on the specific situation: the offered incentives (e.g. monetary compensation, supply of proprietary information, excitement factor, or even just the kudos of being called a ‘co-developer’); the degree of use of multi-media data (e.g. virtual product presentations, short videos, animations); the intensity of interaction (e.g. duration, frequency and number of participants); the applied tools (e.g. open discussion forums, toolkits, virtual stock markets, virtual concept testing or competitions); and the communication style (e.g. anonymity of the interacting parties, informal/formal; uni-, bi- or multi-directional). In particular, STE Company stalls feedback software and sensors into every new product in combination with the advantages of technology and hardware. According to various data transmitted from customers’ smart devices, functional design to products can be made appropriately so that new product with improved features/functions in line with customers’ demands can be launched. The company has grasped the core big data technology (e.g., Spark SQL, and Hadoop Cluster). Therefore, the companies were able to apply data analytics with its big data technology, and react quickly to acquire a large number of loyal customers through adopting reasonable product portfolio, accurate market orientation and perfect function design. These inputs also allow R&D teams to quickly develop a new version of a product, with improved functions and features.

The Company explained that customer involvement can be facilitated through utilising big data to provide new or more precise insights. The insights can be gained in a digital form, such as use tests, in order to understand the customers and adjust the decisions about the products accordingly. Therefore, big data allows users’ behaviours to be examined and thus their demands can be fitted. Since the recruitment of customers was conducted from diverse information sources, it was imperative for the design to align with corporate identity. In addition to this, the customers were led by the design to share their ideas and expertise in a simple and enjoyable approach.

5.4 Stage 4: Using big data to enable customer access and participation

In the programming and testing of the customer involvement platform, customers were accessed (communicated with) by different means. The uses of banners, emails, pop-up windows as well as short articles were considered in communicating with the customers encountered in the Internet and notify them on their responsibility in the NPD project (Füller and Matzler 2007). For example, an inviting pop-up window invited every 50th online user to the STE official forums for 1 month. Email and app subscribers were recruited simultaneously.

Within the involvement process, the company connects with its customers through its own Talend big data platform. It understands customers’ behaviours and needs better by acquiring datasets from 12 processes that run simultaneously and come from sources including third parties, social networking feeds and application programming interfaces (APIs). The customers were asked for help or raising certain questions with the need for apt processing. A wide range of customers enjoyed giving direct feedback and expressed preferences in NPD. At the meanwhile, customer inputs can be analysed to initiate some improvement and to get first results in case the anticipated quantity or quality of customer information is insufficient, the process of collecting it needs to be reconsidered. Customers engaging with the company for the first time should be assessed for background information on their preferences, their willingness to contribute again as well as their expectations regarding further NPD projects. For those customers who have already engaged with the company several times, a relationship can be seen to have been established and company managers should consider creating a specific community for such customers.

The role of big data in the NPD process is directly affected by how data-driven the organisation is overall. In particular, as seen at STE Company, their product and solution, as well as product development is highly dependent on the collection and processing of large customer data sets. In order to handle the problem of difficult data storage, STE Company has set up a data centre of cloud computing specialising in storing and handling big data. By means of reconstructing decision making and analysis system based on big data and establishing product innovation and feedback mechanism based on big data and cloud computing, STE Company improved the competitiveness of products and satisfaction of customers. Furthermore, the Company further explained the important of having data centralised in order to provide the ability to all interested actors within the company to access and process it. To organise and manage successfully big data, organisations should have established innovation ecosystem and built “data-alliances” with stakeholders including partners, suppliers and other actors with common interests.

5.5 Results

In the case of STE, overall, more than 26,000 customers had participated in NPD in some form. Over 13,700 new product ideas were recorded from different sources (e.g. images, text, video and voice mails), more than 127,400 comments were made on the specific functionalities required, and over 3200 visions of future similar devices were gathered. A total of 15,943 customers participated (or around 61.2%) would like to get involved in future NPD projects.

As suggested by Matzler et al. (1996) and Füller and Matzler (2007), a functional/dysfunctional examination was conducted to identify whether the new functions identified were considered exciting or simply basic. Take the function of Dynamic Streaming Technology as an example, according to the customers’ answers toward the question of how they would react if the dynamic streaming technology (high-quality videos in 360\(^{\circ }\) virtual reality) was provided to in the new device and how they would react if the function were not provided, then function/dysfunctional ratios were calculated which evidences whether the new function identified is an excitement or basic factor (Matzler et al. 1996). As Fig. 3 shows, if Dynamic Streaming Technology were provided it would have a significant impact (0.71) on overall satisfaction with the new device, more so than on dissatisfaction if the function were not provided (−0.39). Therefore, the finding shows that the Dynamic Streaming Technology was identified as an excitement factor, and exceeding customer expectations. If the Dynamic Streaming Technology delivered to the new device it brings excitement, otherwise, there is a low dissatisfaction from the customers.

Fig. 3
figure 3

Dissatisfaction and satisfaction level due to absence or presence of Dynamic Streaming Technology

Full size image

To examine whether the customer involvement provided by STE actually allows customers to better understand the value of the new products and to articulate their needs, several questions were asked. In particular, a five-point Likert scale (1 \(=\) very positive; 2 \(=\) positive; 3 \(=\) neither; 4 \(=\) negative; 5 \(=\) very negative) was used to identify the perceptions of customers. The result shows that most of the participated customers highly agreed with the statement that if Dynamic Streaming Technology was supported by the device, its functions and features would meet with satisfaction by customers (mean \(=\) 1.23; SD \(=\) 0.81). The functionality and interactivity of the information department also helped in the articulation of individual needs and wants (mean \(=\) 1.96; SD \(=\) 0.94). Overall, customers stated that they positively and actively made contributions to the NPD (mean \(=\) 2.08; SD \(=\) 0.78).

Engagement with customer-derived big customer data helps STE to understand its customers as well as the market. In this situation, big data supports STE’s customer involvement by revealing the factors that might influence customer loyalty (i.e., how to keep customers coming back again and again). By applying big data analytics, STE can identify optimal investment opportunities across different information sources, and keep optimising its marketing strategies through analysis, measurement and testing. The different information and communication technologies applied offer unstructured, semi-structured, and structured input to the R&D teams. In the case study project, structured and large-scale data sets were gathered during the earlier NPD phases, in order to attain more insight into customer contexts and needs, through dialogues, collaboration and online surveys. For example, STE utilises customer dialogue to shape its NPD through customer data capture, crowd-sourcing and large forums. This structured information was often based on customer stories or dialogues, and customers were able to consciously and actively support the development of new products and functionalities. Semi- and un-structured rich data were captured in the later phases of NPD, when a feature or product had been launched on the market, and customers were able to use the particular product or feature. For instance, STE applies natural language processing (NLP) to unstructured content (captured from apps and social networks) to identify customer satisfaction and preferences. The rich data from a variety sources provided a different type of information to the NPD process, and included real behavioural data based on the click behaviour of customers using a system, for example. In such circumstances, the customer was not actively involved in giving feedback, but information was automatically captured through analysing customers’ online behaviour. Organisations are paying more and more attention to gathering this type of data, to the extent that discussions are arising in social media about ethics and customer privacy. This is an aspect that requires to be taken into consideration when concentrating on capturing customer data for NPD. Structured, semi-structured and unstructured data are common in customer involvement studies in all phases of product development (Sood and Tellis 2005; McKinsey 2009; IBM 2013; Gantz and Reinsel 2012). However, in the case study, the data in the first two phases (generation of idea and concepts; design and engineering) are more connected to feedback, while in the later phase of test and launch larger amounts of data are captured through actual use and customer behaviour.

5.6 Discussion

The in-depth case study indicates how big data is used by STE to improve customer involvement in NPD. In particular, customer involvement has to meet both managers’ expectations and those of the customers participated. That is, it is necessary to ensure that honest and valuable feedback and input is gained from a diverse body of customers; if they are not provided with some motivation, they will stop contributing or be tempted to provide incorrect information (Füller et al. 2006a). From the case study, it is evident that STE, through the use of big data, allows customers not only to embrace inventive products to suggest novel ideas, via trial and error. Apparently, big data enables customers to fiddle around with novel products and new features, which enables them not only to share their tacit knowledge, but also to articulate their explicit needs. Through its efforts in addressing customers’ needs, STE enhances the participation of customer in future company-initiated NPD projects.

Today, customer expectations are high: they want the latest technology and cutting-edge functionality, but at an unprecedented low price, and immediate services. At the same time, they have little brand loyalty and keep comparing the product with others (Cooper 2014). The proposed customer involvement approach can provide companies with the guidance to handle data from various sources and formats, as well as to push intelligence from these data to various channels so as to support NPD. It is meaningful for development of products and services with short product life cycles, notably in the customer electronics industry and social media applications, where demand is driven mainly by lifestyle trends. STE Company indicates that big data in the customer involvement can lead to a greater understanding on how products or users behave, and enable accurate recommendations for existing or new products. Further, they explained that big data compose a key element of some of their products. In this context, the recommended customer involvement, based on big data, has to support a company’s key competencies and be in line with its objectives. On the other hand, it is necessary to balance the expected benefits against potential costs of using big data to interaction with them. Furthermore, the case also highlighted that achieving customer involvement requires considerable coordination and planning through the different NPD phases (Robbins and O’Gorman 2014). Thus, support from top management through a product champion and tight interfacing with the target market is essential components of customer involvement. In addition, the existing organisational culture (i.e. hierarchical corporate culture) can represent a barrier to firms seeking to capitalise on big data to improve customer involvement. The STE Company referred to the significance of data-driven culture and the impact upon the connected operations. Brynjolfsson and McElheran (2016) explain that in a data-driven culture, data holds a central function and promotes a fact-based decision-making process, while data-driven decisions are more informed and effective (McAfee et al. 2012). Interviewees from STE Company argued that a data-driven culture is more important than big data technology, since this is where limitations often occur. However, small or young firms are likely to favour the proposed big data supported customer involvement, given their weak inherent ‘culture’. Another issue is that not all firms (especially small ones) have adequate infrastructure or professionals to analyse the big data.

6 Implications of big data for customer involvement

The benefits of using big data to support customer involvement in NPD are identified as follows: first of all, risk and market uncertainties can be reduced by using big data analytics. Market feedback in a variety formats and from different sources can be acquired in the early development stages (Wong 2012; Wamba et al. 2015). Secondly, by harvesting big customer data allows previously unrecognised customer needs or combinations of needs to be identified (Bozarth et al. 1998; Tsai et al. 2013). Market share will go to “first mover” firms that can respond to customers quickly and meet their needs (Hagel and Brown 2011; Williamson and Yin 2014). Thirdly, big data can be used to generate great ideas from a variety sources. Lead customers (or innovative users) often act as co-creators and support NPD managers in developing ‘winning products’ to the market (Trkman et al. 2012; Bauer and Leker 2013; McKinsey 2011). Fourthly, big data enables a company to contact potential customers in different ways. Fifthly, it also lends itself to customer loyalty and retention; for example, through online participation in NPD, customers gain a better understanding of a new product but also become attached to the product to which they have made a contribution (McKinsey 2009; IBM 2013; Gantz and Reinsel 2012). It is a compelling experience which creates commitment and trust (Füller et al. 2006a). Customer involvement in NPD not only improves a company’s product performance, but also serves as a means of building and enhancing relationships with both potential and existing customers (Sirdeshmukh et al. 2002). Finally, big data can provide NPD teams with a broader basis for their decisions. By applying big data analytics, it is possible to increase the number of test options and to institute parallel testing of product alternatives among a variety of customers; moreover, this can be done repeatedly throughout the different stages of NPD (Terziovski 2010; Davenport 2013).

6.1 Implications for research and practice

This study contributes to the big data literature with the development of a customer involvement approach for NPD by unlocking the power of big data. The approach can be applied through all the phases of NPD. In terms of theoretical contributions, this paper extends the traditional NPD boundaries and provides evidence of the vital role of the customer-centred NPD approach in a data-rich environment. Firms are leveraging big data to embed customer sentiment in product development. This enables firms to move away from product-focused innovation and to turn their attention to innovation around the customer experience. The proposed paradigm-shifting customer involvement approach enables firms to find ways to innovate—unlocking the power of big data to improve customer understanding and make NPD faster and less costly. However, the implementation of the customer involvement approach may put considerable strain on an organisation. Nonetheless, we posit that any stress presented by the introduction of the approach will be more than compensated for by the time and cost reductions achieved in the modification of the NPD process.

This study also contributes to practice. The approach is recommended to NPD managers, as it will allow them to arrange their resources to in order to develop new features and new products in a fast and effective manner. From the examples in the case study, NPD managers can maximise positive outcomes from the approach of customer involvement. In the case of STE, implementing the customer-centred approach allowed it to decrease costs, to increase the speed of NPD and to gain a better understanding of customers’ needs (and interaction with customers), and a change in leadership and team organisation. Compared with traditional NPD approach, STE was able to launch a range of new products in <5 months, at a total cost of $2 million. The company estimate that competitors using traditional design approaches have to invest around $20 million over 12 months to complete a similar set of NPD. Nonetheless, the challenges STE faced in implementing the customer involvement approach were identified such as IT infrastructure, managing relationships with intermediates, and the culture shift from product focus to customer focus.

However, managers must also be aware of several vital issues. First of all, the exploitation of customer information may cause intellectual property problems. Secondly, a company’s existing NPD programme could be disrupted by the implementation of the new approach. Thirdly, the greater degree of customer interaction is almost certain to mean that competitors will get access to information that would for preference remain secret. Finally, certain knowledge and expertise is needed to meet company’s objectives through establishing compelling customer involvement. Therefore, in order to balance potential costs and expected benefits, it is necessary for managers to identify their truly needs from big data and plan the customer involvement approach carefully for specific purpose.

6.2 Limitations and future research

It is worth to mention that the main focus of this study is to investigate the approaches for utilising big data in new product development. Because the real company situation is more than complicated, for different companies, they have different objectives, R&D focus, big data technology, available data and so on. More importantly, the feedback from the industrialists indicate that most of the companies already have their own big data analytics and technologies. Therefore, instead of conducting specific big data analyses, this research explores the approaches that could support organisations tap into new ideas captured from big data to facilitate their customer involvement in NPD. In this way, future empirical studies can be conducted at the organisational level to identify the implications of the approach. In particular, relevant business models, strategies, as well as data analytics need to be developed to support the approach. Although the findings of this research focus on a high-tech industry, we believe it can be generalised to any industry that applies big data and employs R&D in its product development and enables their businesses to be connected to the Internet. Additionally, in this research we pay attention to the approach needed to achieve customer involvement via unlocking the power of big data and we found that the approach can also be generalised and applied to other properties of the service or product (e.g. software companies, hospitals or real-estate companies). So far, the development of a high-level approach for such a complicated phenomenon as customer involvement in a big data environment may highlight some obvious connections while failing to capture others. The approach applied is mainly focused on utilising big data to improve customer involvement in NPD, where different big data analytics were applied to support each of the stages. Therefore, the approach may not work where there is no data or data analytics to support it. For effective and efficient customer involvement, certain big data skills and knowledge are necessary. However, not every company may have these specific skills and knowledge and it may prove hard to get the information from the right customers.

As illustrated in this paper, big data plays an important role in enabling companies to come up with genuinely innovative new products. On the contrary, customers facing leading edge problems may spark a company’s innovation or NPD process. With this study we intended to provide helpful insights into how big data can be used to enhance customer involvement in developing new products. We are hopeful, though, that this broad approach will offer a means to help integrate the wealth of research on big data and NPD in order to advance both research and practice.