Requirements Engineering in Software Startups: A Systematic Mapping Study

Featured Application

The systematic mapping study outcome finds its application in the software industry and research centers. Software engineers working in software companies will find easy access to the state of requirement engineering research conducted in the startup context as disseminated in this research paper as the lessons for the innovation management. Software engineers working in the startups will find the bifurcation of the research domain (using classification & counting) useful to visualize the research trend, leading to innovating their requirement engineering practices. Researchers will find it easy to bridge the research gaps as highlighted by the study.

Abstract

Startups have high failure rates due to their inability to attain a sufficient product/market fit, i.e., delivering a solution that best matches the user needs in the market. Requirement engineering is the activity that could help startup teams identify the value proposition that provides high value to the users and continuously innovate it. The objective of the study is to analyse the state of art of the requirement engineering research in the context of startups, as available in the literature. The analysis of the research area highlights the research trends to achieve two things i.e., (a) predict how much support the startups can get from the literature for enhancing their success rates and (b) identify the research gaps to motivate researchers to conduct future research that could be adoptable in startup contexts. Systematic mapping is conducted on studies extracted from the four bibliographic databases (IEEExplore, ACM, Springerlink and ScienceDirect) and studies extracted by using a forward snowballing approach. Individual studies are coded to yield the classification scheme. Formulated schemes and those already available in literature, were populated with information extracted from the abstracts of the studies. The research is mostly focused on generic requirement engineering and product validation activities. The research is conducted mostly as evaluations (empirical studies) with the outcome of providing theory to the research community. Major underlying motivation of the research is to attain the product/market fit. However, research studies focusing on requirement documentation, prioritization and elicitation are losing focus from 2017, 2018 and 2019, respectively. The literature lacks the studies that reports research solutions which are validated in laboratory settings or in real contexts, experience reports, opinion papers and philosophical papers. The positive side of the finding is that the number of requirement engineering research studies in a startup context have increased in the past five years. At this instant, unfortunately the literature has limited ability to support startups by providing solutions (for instance, research solutions, evidence to support decision makings, best practices, experiences etc.) that are adoptable in their real context. Uniform focus of the researchers across all sub-activities of requirement engineering is required with effort distributed across different research types that supports startups, not only by providing validated solutions but experience reports, opinions, new conceptual frameworks and empirical evidence that can aid their decision making.

1. Introduction

A startup is an organization formed to search for a repeatable and scalable business model (https://steveblank.com/2010/01/25/whats-a-startup-first-principles/). Stated in other words, startups are temporary organisations that are continuously experimenting to identify a business model, which is scalable and repeatable, thereafter they attain higher growth levels and returns. At this stage, startups reach maturity and are no longer startups. In fact, they may become large companies. Thus, the objective of the startups is to grow into big organisations as quickly as possible by attaining higher growth in the markets served by their innovative products or services.

Startups deliver innovative products under extreme resource limitations including funding, intellectual and human resources. Being providers of innovative products either in the same market or a completely new market, the assumptions of the startup team about the ability of the innovative product to capture a good portion of the market must be driven by real market facts learned by continuous interactions (or experimentation) with the customers. Experimentation increases the validated learning about the market, which helps the startup team improve their market assumptions and gain better confidence about them, before releasing the product in the market.

Once the startup team gains better confidence about the market needs, they start with the delivery of the Minimal Viable Product (MVP) to further experiment with the real market by delivering the “minimal functionality” to early adopters. The feedback of the customers provides the team with the necessary feedback (or validated learning) to evolve the product till it provides the solution addressing market needs, i.e., product/market fit is attained. In particular, the objective is to match the actual market needs and provide the market with a product that is needed by the customers and avoiding producing products for which there is no market. The number of purchases of the product increases and it starts capturing good market share, providing more stability to the business model, i.e., the business model is more scalable and repeatable at this stage.

Due to dynamism in the market driven by changing needs, technologies and competitor pressure, business models can never be static, but they need to be continuously innovated to maintain a competitive advantage. The release of the product in the market helps the startup evaluate the product performance in the market and continuously improve the product offering as per customer feedback to gain market share. This helps startups to continuously innovate the value proposition to maintain a competitive advantage.

Requirement engineering is the software engineering sub-activity that tries to identify the real needs of customers which are then documented as requirements after a careful analysis, prioritization and validation. This activity requires a continuous interaction with the customers to understand their pain points (i.e., needs). This continuous interaction helps the startup team to gather enough information about the problem domain as well as build mental models about the solution domain as well. This helps them decide (with a higher level of confidence) the higher priority features that will be able to attract the early adopters and to test the attainment of product/market fit. Poor requirement engineering leads to the delivery of products for which there is insufficient customer demand and hence the startup will fail.

As discussed earlier, the startup teams needs to perform continuous interactions with the customers to achieve product/market fit (validate ideas, validate problem/solution fit, validate product/market fit). They also need to continuously evolve the product by innovating the value proposition to maintain a competitive advantage because of continuously changing customer needs, competitor pressure and technological changes. The number of interactions with the customers will bring a large amount of data (requirement gathering) which must be analysed (requirement analysis), prioritizing the features for the next release (requirement prioritization, initially for MVP and later for the product), documented (requirement documentation), requirement validation (for instance using prototypes) and product validation (for instance, using MVP). Product validations bring useful insights that lead to a new phase of requirement engineering to evolve the product. These requirement engineering activities become more structured as the product captures market and the startup gains growth. The evolutionary requirement engineering process model helps startups to continuously innovate their value proposition to maintain and enhance their market share. In other words, the software should always solve the ever-changing needs of the customers and provide them benefits that exceed their expectations in a cost effective way (to enhance perceived customer value). Otherwise, delivering the product for which there is no market (product provides a solution for which there are no customer needs) leads to startup failures.

The literature does not report startup failures cases except to state that high failure rates among software startups are at least partly attributable to software engineering issues in general and requirement engineering in particular (like the inability to find customers due to a poor product/market fit, etc.) [1,2,3,4,5]. This means that the chances of achieving a good product/market fit and hence enough growth increase if the requirement engineering is carried out effectively. To enhance the startup success rates by focusing on product/market fit issue, it is essential that the startup team should get the following regarding requirement engineering:

(a)

The practices undertaken in big companies that are reported in the context of the startups in the literature that provides lessons and guidelines to the other startups that they could customize as per their unique context.

(b)

The experience reports of the software practitioners that reflect their experiences in the software engineering domain related to enhancing the success rates of startups.

(c)

The evaluation studies (including empirical studies and those solutions that are implemented in real settings) in the startup context that will help other startups to reproduce the reported empirical studies and execute the reported research solutions in their context.

(d)

Validation studies (solutions that are validated in laboratory settings) to help startups modify these solutions and execute them in their specific contexts.

Startups have their own context. The studies reported in the literature which are based generic startup contexts or specific to few startups, must be customized by the startups that find them adoptable in their context. The limited number of studies in the literature limits their streamlined adoption by startups as increases in the number of studies leads to wider options, access to quality metrics (like citations), availability of extension of studies, availability of experimental evaluations of studies, etc. In particular, it provides startups with access to a range of studies which are more structured, better evolved, better evaluated and mature compared with the limited literature.

The objective of this mapping study is to study the state of research in the requirement engineering activity in the startup context by identifying the research trends in terms of focused activities, research type, publication evolution relative to time and underlying research motivation. This help to bifurcate the complex domain of requirement engineering into meaningful information that helps identify that how much research is already being conducted by researchers in different areas of requirement engineering of different types, to predict that how much support the startups can get from the literature for enhancing their success rates. A systematic mapping study is the best choice to achieve the objective of the research as these studies are well suited to provide classification and counting of the broad research areas rather providing empirical evidence based on the synthesis of the literature [6]. The outcome of the research may provide researchers the opportunity to analyse the current trends in this area and identify the research gaps that require urgent attention to provide a reasonable number of studies that could find their real context applicability.

This paper is structured as follows: Section 2 provides a brief background about the systematic mapping process and the guidelines available in the literature. Section 3 presents the systematic mapping protocol as planned and executed in this study. The mapping results are presented in Section 4 leading to the discussions in Section 5. The way the mapping study addressed different types of the threats to ensure result validity is mentioned in Section 6. The paper is finally concluded and some directions for future work are outlined in Section 7.

2. Background

Systematic mapping process aims to highlight the research trends by decomposing a research area into various meaningful pieces of information associated with the objective of the study. In other words, the mapping process gives at an abstract level a view of the research studies undertaken in the area in the form of classification and counting. The classification takes the form of a conceptual schema which is formulated after an analysis of the research studies (within the scope of the mapping study). The information associated with the schema helps the researcher answer the research questions of the mapping study (which are usually broadly formulated) to present the research trends, i.e., present the categories of research and count the classification schema categories [6,7]. The guidelines for conducting the systematic mapping process in software engineering are reported in [6] which are updated based on other mapping guidelines and conducted mapping studies as available in the literature. Updated systematic mapping guidelines are reported in [8]. The guidelines are broadly based on the identification of studies (search, inclusion, and exclusion), categorization and classification schemes and processes and the different ways of visualizing the results to attain a higher level of validity in the mapping study. Overall, the guidelines include the following processes:

• Definition of research questions (defining research scope).
• Conduct search.
• Screening of the papers.
• Keywords using abstracts.
• Data extraction and mapping process.

Research questions are broadly formulated, and bibliographic databases are searched (conducting a search) resulting in a large number of papers. These papers are screened according to the conditions laid down in a mapping protocol (inclusion and exclusion criteria) to filter out the non-relevant ones. The relevant ones are subjected for further analysis to create the classification schema. The information extracted by abstracts (and conclusion) is used to populate the classification schema. The mapping process, thus, consists of three stages, i.e., planning the mapping protocol, executing the protocol, and reporting the findings.

To the best of the authors’ knowledge, the literature lacks mapping studies that highlight the research trends of requirement engineering in a startup context. This mapping study provides the research trends in the requirement engineering for startups to catalyze the future research to bridge the research gaps.

3. Research Method

The systematic mapping study undertaken in this paper is based on the guidelines disseminated in [6] and the updated guidelines as disseminated in [8]. The detailed mapping protocol is mentioned in the next sections and sub sections.

3.1. Research Questions

This mapping study is based on the objective of analysing the conducted requirement engineering research in the context of startups. The analysis of the research area highlights the research trends and suggests the suitable research gaps for future research. This mapping study meets this objective by providing responses based on careful analysis of evidences as available in the literature by providing a meaningful populated classification scheme.

In particular, this classification scheme is reached by considering the individual research questions that meet the research objective.

Overall, this research tries to find an answer to the research question (RQ) formulated as RQ: “What is the state of research in the field of requirement engineering as applied in the context of startups?”. To answer this research question, the paper tries to answer the four sub research questions (RQ 1 to 4) as formulated below.

RQ 1.

What areas in requirement engineering are addressed and how many articles covers the different areas?

RQ 2.

How does the research addressing the different requirement engineering areas can be classified in terms of research type and contribution?

RQ 3.

How did requirement engineering research evolve over time?

RQ 4.

What could be possible underlying research challenges that are addressed by the researchers (at an abstract level)?

Each research question is answered by classifying the research studies (using already available classification schemes and deriving a new classification scheme) and finally populating them with information. The mapping of the research questions to the classification schemes is given in Figure 1.

3.2. Search Process

The search process as per the mapping protocol was conducted in the first week of the April 2020. The systematic mapping protocol include the following steps:

• Formulation of the search string and selection of bibliographic databases (Section 3.3).
• Triggering of the bibliographic database against the formulated search string, resulting in the initial list of studies.
• The initial list of extracted studies from four bibliographic databases was screened through multiple stages against the inclusion and exclusion criteria to limit the number to “most suitable” studies.
• The multiple stages of screening include the following:

  ○

  Screening the studies on the basis of title and then the abstract, resulting in n studies (two stage screening).

  ○

  These n studies are then subjected to a third stage of screening. In this stage, their Google citations are extracted, which are then screened by reading their title and abstract (forward snowballing approach as proposed in [9]. This results in m publications (two stage screening).

  ○

  The total number of studies subjected for analysis to meet the objectives of the mapping study are then n + m.

The search process included all the authors’ efforts to ensure the quality of the study searching process and selection process [10,11]. The well-known papers related to requirement engineering in software startup context as published in leading venues are checked if they are included in the list of 461 papers extracted after execution of search strong over bibliographic databases. The objectivity of the inclusion & exclusion criteria is ensured by testing it over the same bibliographic databases, using the same search string but only for year 2020. The resulting papers selected by the researchers are then compared to see the level of agreements between the selections. The small disagreements were resolved using consensus meetings and some inclusion & exclusion criteria are refined to make them more objective.

3.3. Search String and Bibliographic Databases

The Population, Intervention, Comparison and Outcomes (PICO) framework as suggested in [7] is used to draft the search string. The population is built by startups and intervention is requirement engineering. However, the comparison and outcome were not considered, to maximise the search operation breadth. The mapping protocol as planned for this mapping study defines the use of four bibliographic databases, i.e., IEEExplore, ACM digital library, SpringerLink and ScienceDirect. These bibliographic databases were triggered against the search string to extract research studies published from the years 2015 to 2020 (both end inclusive). The formulated search string is as given below:

(Startup*) AND ((“Requirement Engineering”) OR (“Requirement Management”) OR (“Requirements Engineering”) OR (“Requirements Management”)).

The extracted studies are then subjected to screening against the inclusion and exclusion criteria (Section 3.3), to extract the ones that have the ability to support the mapping study.

3.4. Inclusion and Exclusion Criteria

The studies as available in the literature are selected for the mapping study on the basis of their ability to satisfy the inclusion criteria. The studies that satisfies the exclusion criteria or do not satisfy the inclusion criteria are filtered out from the further analysis. The inclusion and exclusion criteria as formulated for the mapping protocol are mentioned below.

Inclusion criteria

• Research studies written in English language only.
• Research studies with the full text available in the bibliographic database.
• Studies with potential to answer either of the research question(s) as formulated in Section 3.1.

Exclusion criteria

• Studies that are editorials.
• Studies that are keynote addresses (if the full text not available).
• Studies that are proposals for tutorials.
• Studies that are academic theses.
• Studies that highlight the initial draft of work in progress type articles.

3.5. Data Aggregation and Synthesis

The studies were first subjected to the coding to yield conceptual schemas (and schemas already existing in literature, were also used (Figure 1)). The abstracts were analysed deeply (and conclusions if abstract are not clear enough to extract data) to extract the data to answer the formulated research questions. Stated in other words, the schemas were populated with the data to achieve the objectives of the research study. The data extracted from the studies were first added to the data extraction form and then mapped to the conceptual schemas. The structure of data extraction form is given in Appendix A. This step also included the efforts of all the researchers to ensure that the data is correctly extracted to answer the framed research questions. Any disagreements were resolved in the consensus meetings.

4. Results

4.1. Search Results

The initial search resulted in 461 papers, which after two stages of screening are reduced to 16 (n = 16) studies (analysis of title and abstracts, Section 3.2). These 16 studies are then subjected for another two-stage screening (analysis of title and abstracts on studies extracted by applying forward snowballing on primary 16 studies, Section 3.2) resulting in total 04 studies (m = 04 out of 183). Finally, the total number of 20 studies (n + m) forms the basis of the results of this mapping study.

Table 1. Stages of screening of the studies.

Bibliographic Databases	Two Stage Screening		Two Stage Screening		Final Papers (n + m)
	Initial Papers	Final Papers (n)	Google Citations	Selected Citations (m)
IEEE Explore	10	05	78	00	05
ACM	100	04	07	01	05
Springer	235	04	64	03	07
ScienceDirect	116	03	34	00	03

highlights the stages of the screening process in quantitative terms.

4.2. Studies Selected for Review

The 20 studies selected for answering the formulated research questions of this mapping research study are given in

Table 2. Studies extracted for mapping.

Paper Reference	Reference Number	Publication Year	Google Scholar Citations	Selected Google Citation
IEEExplore
Melegati, J.; Goldman, A.; Paulo, S. Requirements engineering in software startups: A grounded theory approach. In Proceedings of the 2016 International Conference on Engineering, Technology and Innovation/IEEE lnternational Technology Management Conference (ICE/ITMC), Trondheim, Norway, 13–15 June 2016.	[12]	2016	21	None.
Chanin, R.; Pompermaier, L.; Fraga, K.; Sales, A.; Prikladnicki, R. Applying Customer Development for Software Requirements in a Startup Development Program. In Proceedings of the 2017 IEEE/ACM 1st International Workshop on Software Engineering for Startups (SoftStart), Buenos Aires, Argentina, 21–21 May 2017; pp. 2–5.	[1]	2017	09	None.
Rafiq, U.; Bajwa, S.S.; Wang, X.; Lunesu, I. Requirements elicitation techniques applied in software startups. In Proceedings of the 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Vienna, Austria, 30 August–1 September 2017; pp. 141–144.	[13]	2017	15	None.
Gralha, C.; Damian, D.; Wasserman, A.; Goulão, M.; Araújo, J. The evolution of requirements practices in software startups. In Proceedings of the 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), Gothenburg, Sweden, 27 May–3 June 2018; pp. 823–833.	[14]	2018	32	None.
Albuga, S.; Odeh, Y. Towards Prioritizing Software Business Requirements in Startups. In Proceedings of the 2018 8th International Conference on Computer Science and Information Technology (CSIT), Amman, Jordan, 11–12 July 2018; pp. 257–265.	[15]	2018	01	None.
ACM Digital Library
Ochoa-Zambrano, J.; Garbajosa, J. An analysis of the bluetooth terminal development pivots from lean startup perspective: Experience and lessons learnt. In Proceedings of the XP2017 Scientific Workshops, Cologne, Germany, 22–26 May 2017; pp. 1–5.	[16]	2017	00	None.
Nascimento, L.M.A.; Travassos, G.H. Software Knowledge Registration Practices at Software Innovation Startups: Results of an Exploratory Study. In Proceedings of the 31st Brazilian Symposium on Software Engineering, Fortaleza, CE, Brazil, 20–22 September 2017; pp. 234–243.	[17]	2017	05	None.
Yin, H.; Pfahl, D. A preliminary study on the suitability of stack overflow for open innovation in requirements engineering. In Proceedings of the 3rd International Conference on Communication and Information Processing; Tokyo, Japan, 24–26 November 2017; pp. 45–49.	[18]	2017	02	01
Chanin, R.; Pompermaier, L.; Sales, A.; Prikladnicki, R. Collaborative practices for software requirements gathering in software startups. In 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), Montreal, QC, Canada, 27 May 2019; pp. 31–32.	[19]	2019	00	None.
SpringerLink
Klotins, E.; Unterkalmsteiner, M.; Gorschek, T. Software engineering knowledge areas in startup companies: A mapping study. In Proceedings of the International Conference of Software Business, Braga, Portugal; 10–12 June 2015; pp. 245–257.	[5]	2015	36	01
Nguyen-Duc, A.; Wang, X.; Abrahamsson, P. What influences the speed of prototyping? An empirical investigation of twenty software startups. In Proceedings of the International Conference on Agile Software Development; Cologne, Germany, 22–26 May 2017; pp. 20–36.	[20]	2017	23	02
Seppänen, P.; Tripathi, N.; Oivo, M.; Liukkunen, K. How are product ideas validated? In International Conference of Software Business; Essen, Germany, 12–13 June 2017; pp. 3–17.	[21]	2017	05	None.
Pompermaier, L.; Prikladnicki, R. Brazilian Startups and the Current Software Engineering Challenges: The Case of Tecnopuc. In Fundamentals of Software Startups; Springer: Cham, Switzerland, 2020; pp. 331–345.	[22]	2020	00	None.
ScienceDirect
Tripathi, N.; Klotins, E.; Prikladnicki, R.; Oivo, M.; Pompermaier, L.B.; Kudakacheril, A.S.; Unterkalmsteiner, M.; Liukkunen, K.; Gorschek, T. An anatomy of requirements engineering in software startups using multi-vocal literature and case survey. Journal of Systems and Software 2018, 146, 130–151.	[23]	2018	13	None.
Melegati, J.; Goldman, A.; Kon, F.; Wang, X. A model of requirements engineering in software startups. Information and Software Technology 2019, 109, 92–107.	[24]	2019	18	None.
Tripathi, N.; Oivo, M.; Liukkunen, K.; Markkula, J. Startup ecosystem effect on minimum viable product development in software startups. Information and Software Technology 2019, 114, 77–91.	[25]	2019	03	None.

(selected from four bibliographic databases),

Table 3. Studies extracted for mapping (Forward Snowballing).

Paper Reference	Reference Number	Parent Reference Number	Publication Year
Gutbrod, M.; Münch, J.; Tichy, M. How do software startups approach experimentation? Empirical results from a qualitative interview study. In Proceedings of the International Conference on Product-Focused Software Process Improvement; Innsbruck, Austria, November 29–December 1, 2017; pp. 297–304.	[26]	[20]	2017
Yin, H.; Pfahl, D. The OIRE Method-Overview and Initial Validation. In Proceedings of the 2018 25th Asia-Pacific Software Engineering Conference (APSEC), Nara, Japan, 4–7 December 2018; pp. 1–10.	[27]	[18]	2018
Melegati, J.; Chanin, R.; Wang, X.; Sales, A.; Prikladnicki, R. Enablers and Inhibitors of Experimentation in Early-Stage Software Startups. In Proceedings of the International Conference on Product-Focused Software Process Improvement; Barcelona, Spain, 27–29 November 2019; pp. 554–569.	[28]	[5]	2019
Nguyen-Duc, A. An Analytical Framework for Planning Minimum Viable Products. In Fundamentals of Software Startups; Springer: Cham, Switzerland, 2020; pp. 81–95.	[29]	[20]	2020

(selected after execution of two stage screening on citations) and

Table 4. Relation between Parent & Child studies.

Parent Reference Number	Child Reference Number
[18]	[27]
[5]	[28]
[20]	[26,29]

(relation between studies mentioned in Table 2 and Table 3).

4.3. Answers to Research Questions

The abstracts of 20 selected studies were subjected to the coding. The codes extracted from the abstracts were merged as well as relabeled to yield a meaningful classification scheme. Finally, the information extracted from the abstracts were used to populate the classification scheme to answer the formulated research questions. Apart from deriving the classification scheme, this mapping study also employs the already available classification scheme for classifying the research studies on basis of types [10] and the contributions [11]. The mapping study yielded responses to the formulated research questions as follows:

RQ 1.

What areas in requirement engineering are addressed and how many articles covers the different areas?

The mapping study resulted into five sub areas of requirement engineering that attracted the researchers’ attention namely requirement elicitation, requirement prioritization, requirement documentation, requirement validation, and product validation. However, few studies were based on requirement engineering in general also. Since in mapping studies, different researchers could arrive at different schemes, so the authors made following assumptions while arriving at the classification scheme.

• A requirement validation activity is considered as the activity undertaken to make a prototype. This prototype is used to verify the requirements and gather more information on basis of user interaction. Also, the prototype is considered as different from the minimal viable product (MVP) by the authors.
• Product validation is the activity undertaken to verify if the solution delivered fulfils the basic needs of the users. In other words, experimentation undertaken by launching the MVP is termed as product validation in this paper.
• The studies reporting MVP, experimentation, software evolution are categorised under the category of product validation. This assumption is held until and unless these studies focuses on individual requirement engineering activities (like documentation, prioritisation etc.) in relation to experimentation, evolution or MVP.
• Few studies focused on multiple activities of requirement engineering (including generic requirement engineering). The contribution of the study is divided equally amongst the multiple focused activities (excluding the activities that are just focused minorly, for instance, activities just described in the studies). For example, a study is focusing mainly on two activities i.e., requirement engineering (complete) and requirement validation. The focus on prioritization is just at introductory levels. The contribution of the study will be counted as ½, individually for both majorly focused activities (sum is unity), excluding the prioritization activity (minor focus doesn’t get contribution).
• Studies reporting knowledge management are categorised in the requirement documentation category. This is because, knowledge management is not a requirement engineering activity, but it is a supporting activity.

Table 5. Research Focus on Requirement Engineering Activities.

Requirement Engineering Activities	Number of Studies (Contribution)
Requirement Elicitation	3.5 (Decimal signifies joint focus)
Requirement Prioritization	01
Requirement Documentation	01
Requirement Validation	2.5 (Decimal signifies joint focus)
Product Validation	04
Requirement Engineering (Focus on Generic Requirement Engineering)	08

shows the focus of the research studies from 2015 to 2020 (end inclusive).

As shown in Table 5, major studies are focused on generic requirement engineering (40%), followed by product validation (20%), requirement elicitation (17.5%) and requirement validation (12.5%). Requirement prioritization and documentation are the least focused activities (5% each).

RQ 2.

How does the research addressing the different requirement engineering areas can be classified in terms of research type and contribution?

The research studies were classified using existing conceptual schemas i.e., on basis of types [10] and the contributions [11] (Figure 1).

Table 6. Number of studies classified on basis of research type.

Research Type	Number of Studies
Evaluation	17
Validation	02
Solution	00
Philosophical	00
Opinion	00
Experience	01

gives the frequency of research classified on basis of type. It shows that the literature provides mostly the evaluation based studies (85%), followed by validation (10%) and experience (5%). This means that the studies are usually Evaluation studies (like empirical research including literature surveys, case studies etc.). Further the evaluation approaches in the form of solutions validated in the industrial real scenarios are also too limited. Validation based studies (solutions validated in laboratory settings) are very limited.

The literature lacks the philosophical, opinion and experience papers, that could otherwise be very helpful to the startup team as the necessary guidelines to be adopted in their working context. Such guidelines are driven from the experiences that the practitioners gained in their companies and thus are useful lessons that could be useful to the startup team members for avoiding mistakes and improving their success rates.

Table 7. Number of studies classified on basis of contribution type.

Contribution Type	Number of Studies
Tool	00
Method	02
Guidelines	00
Conceptual Model/Framework	04
Theory	13
Lesson Learned	01

gives the frequency of research classified on basis of contribution (outcomes) and it specifies that outcome of 65% studies is the theory, followed by 20% conceptual models/frameworks and 10% methods and 5% lesson learned.

RQ 3.

How did requirement engineering research evolve over time?

To answer this research question (referring Figure 1), the evolution of the research focus is considered against two parameters (or conceptual schemas):

• Evolution of research focus in terms of focus areas over time (Figure 2).
• Evolution of research focus in terms of underlying research motivation (abstract level) over time (Figure 3).

Figure 2 shows that the researcher’s focus across different requirement engineering activities is non-uniform. There is much focus on generic requirement engineering in the past five years. Requirement elicitation had been focused in 2017 and 2019 but the quantitively the focus is declining.

Software documentation and requirement prioritization are least focused activity while the former got attraction in 2017 and later got attention in 2018. The less focus on software documentation may be because startups have high pressure to release as early as possible to the markets under limited resources, that leads to neglection of this activity. Requirement prioritization seems to be an informal process in startup contexts where the subjective judgements of the startup team based on their interactions with the customers, is used to identify the higher priority features for next release (or MVP).

Product validation and generic requirement engineering has been much focused in the past two years (2019 and 2020). Requirement validation is again gaining attention of the researchers.

Figure 3 shows that there had been much focus of researchers to undertake research regarding requirement engineering to attain a good product/market fit and growth. The focus had been much higher since 2017 (60% studies) and the total focus being 62.5% since 2016.

The researcher’s attention to studying the existing processes and innovating them, seems to be reducing with the time. Lack of studies reporting the improvements may hinder the ability of startups to adopt the best recommended practice as per their working contexts.

RQ 4.

What could be possible underlying research challenges that are addressed by the researchers (at an abstract level)?

The underlying motivations of the studied research papers are extracted after carefully analysing the abstracts of the studies. Underlying means that after reading abstracts, the researchers derived the motivations, although they were not explicitly mentioned in most of the studies (with fewer exceptions). This resulted into three underlying research challenges that the studied research papers tried to address:

• Reaching a good product/market fit.
• Process improvement (say for example, streamlining, identification of challenges, promoters etc.).
• Studying ongoing processes.

Table 8. Underlying motivation for research (on basis of challenges supporting research or challenges reported).

Underlying Motivation for Research	Number of Studies
Reaching good product/market fit	12.5 (Decimal signifies joint focus)
Process improvement (for example, streamlining, identification of challenges, promoters etc.)	5.5 (Decimal signifies joint focus)
Studying ongoing processes	02

shows the frequency of different studies classified in either of the three underlying objectives. Major focus of studies is to attain a good product/market fit (12.5 studies) followed by process improvement (5.5 studies) and least focus on studying ongoing processes (2 studies).

Figure 4 shows the percentage of individual motivations. The aspect of product/market fit accounts for 62.5% studies, followed by improving processes (27.5%) and studying processes (10%). It means that major concern among researchers is to conduct empirical studies focusing on various requirement engineering activities to attain a good product/market fit as an underlying objective.

5. Discussion

This systematic mapping study has brought various insights into the research trends in the area of requirement engineering as applied to the startup context. These insights are highlighted briefly in Section 4 and are discussed in this section.

In the context of software startups, where the environment is uncertain and dynamic, the software team will have to quickly build, implement and learn by having continuous interactions with the market. The overall objective is to have match between actual needs of the users and the solutions provided by the startup to avoid failures arising because of poor product/market fit. Once product/market fit is attained, the startups should quickly release better software versions to provide solutions to ever changing needs of the customer. The ability of the team to continuously innovate the value proposition helps them to maintain their hold on the market share and provide them the ability to increase it further by attracting new customers. The market is so dynamic because of everchanging customer needs, changing technologies and competitor pressure that startup team should innovate quickly (i.e., release better software versions quickly).

Evolving requirement engineering is key in helping startups implement an innovative and competitive advantage throughout the startup life cycle. Requirement elicitation ensures that the startup team has understood user needs correctly (which will be translated into requirements), prioritization helps them to select the highest priority features (for MVP and later versions), requirement validation to verify if the requirements are understood correctly, product validation to see if a sufficient product/market fit is attained and documentation to document software development related information. These activities must be executed quickly and should be highly accurate to work under an uncertain and fluctuating environment. Requirement documentation also holds high importance because the knowledge they carry will help in future times, which otherwise may be lost with time or at the time when employees leave the startup.

The startup teams customize the requirement engineering process throughout their life cycle by incorporating the practices that they believe are the best for the particular context prevailing at particular interval of time in the startup environment [12]. This customization is possible because of the real experiences of the software team which include (roles varies across startups) founders, software development managers, developers, etc. The customization process will vary with the experience of the team members and incurs high level of uncertainties with their successful implementations. The studies reported in the literature could provide strong empirical evidences about suitability of the reported research (i.e., research solutions, empirical studies, opinions, experience letters, etc.) for the startups that they could find useful for customization and adoption to their working settings. This will provide more objectivity in the decision makings about implementation of a suitable process model to innovate value proposition.

The systematic mapping study resulted in just 20 studies focusing on requirement engineering in startup domains. Studies that report the transfer of the requirement engineering research specific to the big companies to the startups are missing from the literature. This means that literature has limited evidence to provide to startup teams and very limited lessons from the big companies that are reported to work in startup contexts too. Requirement engineering is a main activity that needs continuous interactions with customers to better explore the problem domain. This helps startup teams improve their understanding of the problem domain and validate their requirements with the customers before the software is implemented and released to the market. The mapping study reported that limited work is conducted in requirement engineering (3.5 studies out of 20) which makes it hard for the startups to improve their understanding of the market needs (and hence improve product/market fit).

Requirement prioritization which is an important activity to decide priority of the requirements has yet to attract enough research (only 1 study found out of 20). Requirement documentation is always an informal process in startups it leads to technical debt and increases the dependence of the startups on the tactical knowledge of the employees and their working relations with the startups. Staff turnover leads to loss of documentation which makes it hard to evolve the software further. Requirement documentation is another ignored area (1 study out of 20). Requirement elicitation is not reported by researchers after 2019, prioritization not reported after 2018 and documentation are not reported after 2017 (Figure 2). Product/market fit which is heavily dependent on requirement elicitation had been reported from 2016 to 2020 by 12.5 studies out of 20. This shows an imbalance between studies focusing on requirement elicitation which were not reported after 2019 and those that report practices to improve product/market fit (Figure 3).

Requirement validation that aims to validate the requirements using prototypes to ensure that they are working on correct solution features, had been reported 2.5 times out of 20 identified studies. Product validation which aim to validate the software product with customers (for instance using MVP), had been reported 04 times out of 20 identified studies. The work to streamline the validation activities help to get streamlines validated learnings to improve the product offerings.

Focus on generic requirement engineering (08 studies out of 20) is not enough to handle the various issues specific to individual phases of requirement engineering. However, they may provide startups with empirical evidences about generic requirement engineering which may help them to identify customized individual requirement engineering activities accordingly. The number of studies conducted in different areas of requirement engineering (Table 5) is not enough to provide the startup community with converged consensus about suitability of a particular requirement engineering activity method and guidelines thereof, to be executed under certain circumstances. The high failure rates of software startups provide evidence that literature should furnish enough work that could be evaluated in practice to the support startup community.

Table 6 and Table 7 show that major focus of researchers is to conduct evaluation research in form of case studies, literature surveys etc. rather providing requirement engineering models as evaluated in practice. The startups could be benefitted learning reported practices but limited number of studies (17 out of just 20) limits the size of the knowledge domain. The literature provides limited studies that provides research solutions validated either in laboratory settings or in real life. For example, requirement prioritization method which is proposed and validated in laboratory or real settings, could help other startup adopt the method as per their own context, but such studies are limited.

Table 6 shows good signs that the available studies are building theories and helping the research community to use them to extend the empirical evidences. However, the available evidences do not provide enough information that could converge into information that could provide startups a guide to enhance success rates. Table 8, Figure 2, Figure 3 and Figure 4 show that the researchers consider the importance of the product/market fit and hence this issue had been underlying focus of major studies. This had been a reason that why the product validation activity had been a major focus of research during the past two years. However, compared to the mapping study results reported in [5], particularly those addressing requirement engineering area, the researchers focus had drastically increased in past 5 years. This increase is reflected by the two main metrics i.e., number of published studies and number of requirement engineering activities targeted by the researchers.

However, it is expected that the research will be uniformly focused across all sub-activities of requirement engineering with uniform focus across different research types that supports startups not only by providing validated solutions but experience reports, opinions, new conceptual frameworks and empirical evidences that aid to their decision makings.

6. Threat to Validity

The following types of validities must be considered in systematic mapping studies i.e., descriptive validity, theoretical validity, generalizability, interpretive validity [8]. The mapping study reported in this paper addresses different types of the threats to validities as under:

(a)

Descriptive Validity: The ability of the research study to accurately describe the gathered information ensures descriptive validity. Data extraction form was used (Appendix A) to accurately record the information to be used to populate generated schemas leading to the answering of the research questions. This way, this study handles the descriptive validity.

(b)

Theoretical validity: Researchers in [8] defined this validity as the ability of the researchers to capture what they intended to capture. This is impacted by the bias during study identification, inclusion and exclusion criteria objectivity, data extraction and classification, etc. The drafting of search string, inclusion and exclusion criteria objectivity, checking the accuracy of studies coverage (by checking for the inclusion of the well-known papers in the requirement engineering and startup area as published in leading venues), ensuring that data is extracted and mapped to correct classification is taken into account by all authors of the study. The first author’s work is checked by the other authors of this study to assure theoretical validity. Any disagreements were resolved using consensus meetings and some inclusion and exclusion criteria are refined to make them more objective. The objectivity of the inclusion and exclusion criteria are ensured by testing it over the same bibliographic databases, using the same search string but only for year 2020. The resulting papers selected by the researchers are then compared to see the level of agreements between the selections. The small disagreements were resolved using consensus meetings and some inclusion & exclusion criteria are refined to make them more objective. Further full text reads for study selection further strengthened the ability of the study to handle theoretical validity threat.

(c)

Generalizability: The results of the mapping study are highly generalizable because the mapping study had ensured the coverage of a broad research area for a longer time interval. There could be studies in grey literature (i.e., body of knowledge that is not controlled by the entities whose primary objective is to publish information for commercial gains), which may or may not be indexed, but as their number is quite small so their impact will be very marginal. The researchers believe that the study is quite generalizable.

(d)

Interpretive validity: The ability of the researcher to accurately interpret the gathered data without interpreting using his own perspective is called interpretive validity. In other words, the conclusion drawn should be based on the data gathered and multiple perspectives about the data should result in similar interpretation of the data. The joint involvement of all researcers with expertise in software engineering and empirical research (especially mapping studies and reviews) helped to ensure interpretive validity.

(e)

Repeatability is the ability of the mapping study to be repeated by another researcher which yields similar results. The mapping procedure and the results based on studied research papers were reported in the paper. This ensures the other researchers can reproduce the mapping study provided the conditions remain the same i.e., search string, date of data extraction etc. However, due to issues with the abstracts of few studies and ambiguous use of few terms, the repeatability with the same classification may vary marginally.

7. Conclusions and Future Work

The systematic mapping resulted in the structuring of the requirement engineering area to analyse the state of art in the software startup context. The paper concludes that there is the urgent need for research in almost all sub-activities of the requirement engineering field to enhance success rates. The literature lacks studies that report research solutions which are validated in laboratory settings or in real contexts, experience reports, opinion papers and philosophical papers. Limited empirical evidence does not provide enough information that could be formulated as a heuristic that a startup could use in the context of suitable market scenarios. The methods to achieve higher accuracy in understanding the actual needs of the user optimally, are missing from the literature. At this instant, unfortunately the literature has limited ability to support the startups by providing solutions (for instance, research solutions, evidence to support decision makings, best practices, experiences etc.) that are adoptable in their real context.

The studies reported in the literature are less structured and seem to be initial reports which require a lot of research to gain maturity in this area. The positive side of the findings is that the number of requirement engineering research studies in the startup context have increased in the past 5 years. Uniform focus of the researchers across all sub-activities of requirement engineering is required with effort distributed across different research types is the need of an hour. In future, it is expected that more empirical studies will be conducted, with more empirical studies evaluating these studies, to result in the meaningful empirical guidelines useful for startups. It is also expected that startups could be benefited with the dissemination of best practices, opinions and availability of the tool supported research solutions for the various software development activities undertaken to release highly innovative software solutions.