A Novel Forecasting Methodology for Sustainable Management of Defense Technology

Abstract

A dynamic methodology for sustainable management of defense technology is proposed to overcome the limitations of the static methodology, which involves comparative analysis based on the criterion of the highest technology level and has limitations for time series analysis, because the country with the highest level undergoes technical changes over time. To address these limitations, this study applies a technology growth model for a dynamic analysis of the Delphi result. An effective method using patents is also proposed to verify and adjust the analysis results. First, technology levels of the present and future are examined by the Delphi technique, and the growth curve is extracted based on the technology growth model. Second, the technology growth curve based on patents is extracted using the annual number of unexamined and registered patents related to the technology. Lastly, the statistical significance of the two growth curves is examined using regression analysis. Then the growth curves are adjusted by the rate of increase in patents. This methodology could provide dynamic technology level data to facilitate sustainable management of defense technology. The results could be useful to research institutions, as they establish strategies for securing technologies in defense or private domains.

1. Introduction

Technology forecasting is to find the systematized knowledge applied to manage technology efficiently in the future [1,2]. Using the results of technology forecasting, we can do many important works in management of technology (MOT) such as technological innovation, technology valuation, road-mapping, new product development, etc. [3,4,5,6]. In addition, technology forecasting is important job for R and D (research and development) planning and technology strategies in a company [7,8,9,10,11,12]. The role of technology forecasting has been increased to diverse areas in MOT. New technology fields, as well as the traditional technology domains including information and communication technology (ICT), bio and medicine technologies, are emerging in the areas of technology forecasting. National defense technology is one of the very important issues in technology forecasting. Most companies and nations would like to improve their competitiveness in national defense. Furthermore they want to sustain the technological competitiveness for national defense. In this paper, we propose a forecasting methodology to manage the sustainability in national defense technology.

Most research for technology forecasting were based on qualitative and quantitative approaches such as Delphi and patent analysis [7,13,14,15,16,17,18,19,20,21,22,23,24,25]. The Delphi technology forecasting is to forecast the future technology of target domain by repeated survey from domain expert group [26]. This provides more subjective and qualitative results for technology forecasting because the Delphi methodology is depended upon the experts’ experience and knowledge. In comparison the patent analysis is more objective method for technology forecasting because patent analysis is to analyze patent data by quantitative methods such as statistics and data mining [27,28,29,30]. Basically, the patent analysis uses the retrieved patent documents related to target technology from the patent databases in the world [2]. Next, we transform the patent documents to structured patent data for statistical analysis. In this step, we apply text mining techniques to the document data [2,7,30]. Lastly, we analyze the structured data for technology forecasting and apply the result to MOT. Each method for technology forecasting has its merits and faults. So, in our research, we combine the technology forecasting methods to build a more accurate forecasting model. That is, we reflect all results from Delphi and patent analysis for efficient and effective technology forecasting. We meet another problem in technology forecasting process. It is about the forecasting time. In general we forecast the future technology static at this point, that is, we forecast only one aspect of future technology at static time. However, most technologies have been changing with the times. So, we need to forecast the future technology dynamically. This paper proposes a dynamic methodology for technology forecasting in national defense technology.

The purpose of the National Technology Level Evaluation is to secure basic data for sustainable management of technology by understanding technology levels and trends in the domestic environment and in foreign countries [31,32]. In order to accomplish this, many institutes conduct a technology level evaluation on a regular basis. For the basic data of technology level evaluation, domestic institutes conduct qualitative analyses, such as expert surveys [33] ,and the Delphi technique [13], whereas foreign institutes use quantitative indicators such as patents, journal papers, R and D expense, and research manpower [14]. These qualitative and quantitative data are based only on the country with the highest technology level and are, thus, used to make comparisons from the highest technological level. This static analysis is inadequate since the country’s technology level changes as time goes by, which makes time series analysis impossible [34]. This study introduces dynamic analysis of sustainable management of defense technology to overcome the limitations of static analysis. The technology growth curve is extracted based on the present and future technology levels as determined from the Delphi method. In order to verify the extracted growth curve, another technology growth curve is extracted based on the unexamined and registered patents from 1974 to present. These two growth curves are examined by regression analysis, and then the curves are adjusted according to the rate of increase in patents. This paper introduces a novel methodology for growth curve analysis based on both the Delphi method and survey results and patent data, which is used to verify and adjust the survey-based curve.

2. Related Work

“Technology level” is a compound noun that can be widely defined according to the purpose of the technology or the level. Schmookler (1966) defined technology level as the technical knowledge accumulation related to industry production [31]. Martino (1993) considers technology level as how the technology function is implemented, and this performance is divided into functional parameters and technical parameters for a quantitative presentation [15]. In order to evaluate levels of scientific and industrial technology, the technology level evaluation makes comparisons and evaluations of many agents, such as countries, industries, or firms [35,36]. Many institutes have studied expert surveys; the Delphi technique; and journal, patent, or statistical indicators analysis as methods for technology level evaluation. Existing cases of technology level evaluation apply static analysis when making comparisons among many countries. Since static analysis is focused on a developed country’s highest level of technology as the criterion, it is hard to perform time series analysis and examine technology development phases. This makes it difficult to establish sustainable management of defense technology. Kim (2009) also points out this difficulty [37]. Although there are many institutes conducting technology level evaluation, only a few of them have accumulated significant data for evaluation. This is because the evaluation technology types change over time in accordance with policy objectives. Moreover, there are problems with the method used in technology level evaluation. When using the Delphi technique for technology level evaluation, a developed country of the highest technology level is set as the criterion for evaluation. There are several approaches to deal with this problem. Freeman and Soete (1997) suggested new concepts of technology development phases and growth curve, which were invented by Potthoff and Roy (1964) [38,39]. Bark (2007) proposed a theoretical method for technology level evaluation using the growth curve [40]. The Korea Institute of S and T Evaluation and Planning (KISTEP) published a technical report that includes technology level evaluation for national strategic technologies by using the Delphi technique and dynamic analysis [41]. Ryu (2011) conducted technology level evaluation with dynamic analysis based on the technology growth curve, the Pearl model, and the Gompertz model [42]. The common characteristic of these studies is the data collection process through Delphi or expert surveys. Since Delphi examination lacks accumulated results [43], a verification process is necessary for ensuring reliability of the result. However, the studies cited do not propose a definite solution for the verification process. As shown in existing studies, the dynamic analysis for technology level evaluation is used for the growth curve only. A study conducting growth curve analysis with the Delphi technique and verifying the result with patent analysis has never been published in the academic world. Therefore, patent analysis, which aims for the verification and revision of results extracted, is the most realistic method to solve the problem regarding reliability of results.

Patent analysis has been done for technology analysis, as well as technology level evaluation. Ernst (2003) used patent information to evaluate patent portfolios and the technology level in order to assess a firm’s technological competence [44]. Mogee (1991) studied the several advantages of patent by presenting its applicability for technology analysis and planning since patents have detailed information, comprehensive coverage, standardized technology level, and time series of data [16]. Patent analysis is also known to be useful for technology forecasting with various analytic techniques applied on detailed technical information in the patent data [45,46]. Jun (2012) used the data mining technique to construct a matrix map with patent clustering for technology forecasting [7]. Jun (2015) also applied quantitative patent analysis to construct a technology valuation model for big data marketing technology [17]. Choi (2015) conducted patent analysis to extract the potential growth rate of technology convergence by extracting a diffusion pattern [47]. There are several existing studies that applied the growth curve to patent data in order to understand the evolution of individual technologies and their systems. Anderson (1999) collected US patent data and constructed an S-shaped image of the technology growth curve [48]. Bengisu (2006) conducted a Gompertz or logistic curve research, including a comparison analysis between quantified expert survey results on several technologies and the number of patents of the same technologies [49]. Furthermore, Chang (2015) studied the telematics technology life cycle by collecting patent data and constructing a logistic growth model, and the patent growth of telematics technology was verified by the logistic growth model [18]. As other existing studies on the sustainable management of technology are taken together [50,51], sustainable technology level analysis is expected to perform better since the patent analysis results play a key role in verifying the survey-based analysis.

3. Proposed Method

An understanding of the stages and trends of technology development using time series analysis is essential for sustainable management of defense technology. Therefore, the proposed methodology in this research includes extracting the growth curve of defense technology analysis using the Delphi technique, and validating the process by the patent-based growth curve. A schematic of the proposed methodology is shown in Figure 1.

Figure 1. Procedure of proposed methodology.

Figure 1. Procedure of proposed methodology.

The proposed procedure consists of four curve steps from derive survey based growth curve to correct derived growth curves. This procedure perform key role for sustainable technology forecasting and valuation in our national defense technology.

3.1. Derivation of the Survey-Based Growth Curve

The Delphi results of target technologies, which were examined in the past, are collected. There are critical technologies such as C4I (command, control, communication, computer, and intelligence) and the weapon system as target technologies. These technologies have been analyzed using the Delphi method in 2003, 2007, and 2010. Each Delphi result of the technologies is evaluated relative to the most developed country’s technology level, which is designated as 100. In this paper, we use the Gompertz model for forecasting growth trends of the defense technology because this model showed its better performances than other models in the growth curves from the results of previous researches [2,52,53,54,55]. The Gompertz model, the goodness-of-fit of which has been validated by many advanced studies [56], is used in order to extract the growth curve of the developed country’s technology. The technology level and forecasting level of each technology by year are used for this growth curve extraction:

$$Y\left(t\right)=L{e}^{\mathsf{\beta }{e}^{-\mathsf{\alpha }t}}$$

(1)

In this functional formula, Y(t) is the technology level at time t. L is the theoretical upper limit of technology that can be reached; the upper limit is 100. α is the time required to reach the inflection point which affects technology level. β is the slope of the curve that affects the technology development speed. These parameters of technology growth curve are extracted by fitting the curve, which is done by minimizing the sum of squared errors:

$$SSE={\displaystyle \sum }_{t=1}^T{(Y_t-{\widehat{Y}}_t)}^2$$

(2)

Y_t is the Delphi survey value, and Ŷ_t is the value estimated by the technology growth curve.

3.2. Derivation of the Patent-Based Growth Curve

The annual numbers of patents related to the target technology are collected. This research extracts keywords from the definitions of the core C4I weapon system technologies; the keywords are then used to identify patents from 1974 to 2013. The growth curve is extracted on the basis of the annual accumulated number of patents. The Gompertz model is used again for the patent-based growth curve. As the parameters of the growth curve, which are α and β, are maintained, the upper limit L of patent-based growth curve is adjusted to 100 so as to make it correspond to the upper limit of the survey-based growth curve. The patent-based growth curve is then translated parallel to the X-axis to make this correspond to the upper limit of the survey-based growth curve. This is because the two curves should be cross-correlated by the corresponding technology level. (F: (t, y) → (t − k, y)):

$$Y\left(t\right)=L{e}^{\mathsf{\beta }{e}^{-\mathsf{\alpha }\left(t-k\right)}}$$

(3)

Since Y(t), L, β, α, t are already defined, k can be calculated by using logarithm to solve the exponential equation.

3.3. Verification of the Derived Growth Curves

Both the survey-based and the patent-based growth curves are verified to check for statistical significance. Since it verifies the statistical significance only, simple regression analysis is conducted to simplify the model. In this paper, we consider simple linear regression to avoid the overfitting of nonlinear regression model [57,58,59,60,61,62,63]. Since we already consider the Gomperts model as a growth curve, this has nonlinear characteristic. So, we build a linear model and a nonlinear model for constructing an efficient forecasting model. In order to verify the statistical significance, regression analysis is done as follows:

• Step 1: Identify the regression analysis result of the survey-based growth model.
• Step 2: Identify the regression analysis result of the patent-based growth model.
• Step 3: Identify whether the confidence intervals of the two growth curves overlap.

If there is an overlapping area between the regression coefficients of both the survey-based and the patent-based growth curves, there is similarity between those two curves. It there is no overlapping area, these two curves are completely different.

3.4. Correctly Derived Growth Curves

If there is any similarity between the survey-based and the patent-based growth curves, it is that the survey-based growth curves can be used as the basic data for the sustainable management of defense technology. However, it is impossible to have reliability in the survey-based growth curve if the two curves are completely different. That is, the Delphi survey, which requires the inefficient use of cost and time, has to be conducted all over again. Therefore, this research proposes a new concept of that growth curve that statistically satisfies both the survey-based and the patent-based growth curves. The proposed curve is the weighted sum of the two growth curves:

$$F_{new}=\left(0.25+A\right)\times F_{survey}+\left(0.75-A\right)\times F_{patent}$$

(4)

$$F_{new}=\left(0.25+A\right)\times 100e^{{\mathsf{\beta }}_1{e}^{-{\mathsf{\alpha }}_{1}t}}+\left(0.75-A\right)\times 100e^{{\mathsf{\beta }}_2{e}^{-{\mathsf{\alpha }}_{2}t}}$$

(5)

• A: Rate of increase in the patent-based growth curve in the base year
• α₁, β₁: Parameters of the survey-based growth curve
• α₂, β₂: Parameters of the patent-based growth curve

A, the rate of increase in the patent-based growth curve in the base year, reflects the speed of technology development. The base year refers to the year in which the Delphi survey is conducted. The growth curve attains the high growth rate at the initial stage, but the rate decreases to zero as time goes by. Since the growth rate of the curve decreases with the lapse of time, a low growth rate suggests a mature technology. The patent-based growth curve, extracted on the basis of accumulated quantitative data, presents the precise technology level. That is, the lower rate of increase, the higher the weighted value. Thus, this research uses 0.75 less A as the weight of the patent-based growth curve. A constant weight of 0.25 is applied to account for at least 25% of the survey-based growth curve. It is possible to adjust the constant A in accordance with the features of the target technology. In this paper, we decide the weights by experience through trial and error. If the regression coefficient confidence interval of the proposed curve overlaps with the confidence intervals of both the survey-based and the patent-based growth curve, then the proposed curve statistically represents the two curves. Figure 2 presents the proposed method.

The differences between the proposed method and the old method are as follows.

(1)

The old method is focused on static analysis, which makes a comparison in the present with the country of the highest technology level as the criterion. It makes time series analysis impossible to conduct. On the other hand, the proposed method uses the Delphi result for the growth curve, which allows for conducting time series analysis to understand the technology development stages.

(2)

There are no verification and adjustment processes with patent in old method.

Table 1 shows the difference between the old method and the proposed method in detail.

So this paper contributes to new approaches for sustainable technology forecasting in dynamic analysis, verification of technology level, and technological correlation.

Figure 2. Proposed method diagram.

Figure 2. Proposed method diagram.

Table 1. Old method vs. proposed method.

Method	Survey	Static Analysis	Dynamic Analysis	Verification	Correlation
Old Method	Delphi Technique	Available	N/A	N/A	N/A
Proposed New Method	Delphi Technique Patent survey	Available	Growth Curve	Regression Analysis	Growth Rate Correlation

Table 1. Old method vs. proposed method.

Method	Survey	Static Analysis	Dynamic Analysis	Verification	Correlation
Old Method	Delphi Technique	Available	N/A	N/A	N/A
Proposed New Method	Delphi Technique Patent survey	Available	Growth Curve	Regression Analysis	Growth Rate Correlation

4. Experimental Results and Analysis

4.1. Survey-Based Growth Curve

The Defense Agency for Technology and Quality (DTaQ) in South Korea has conducted a defense science and technology survey related to a system of 27 weapons. This research used the Delphi-expert survey results related to C4I system of 2003, 2010, and 2013. Among the 21 technologies of C4I, six technologies that had been continuously examined in 2003, 2010, and 2013 were selected as the technology levels in each year and future for the experiment. Table 2 presents the technology levels of the six technologies. In 2003, technology levels in 2003 and 2010 were predicted. In 2010, technology levels in 2010 and 2013 were predicted. Then in 2013, technology levels in 2013 and 2018 were predicted.

Table 2. C4I technology level survey results.

Weapon Systems	Technology Name	2003 Survey		2010 Survey		2013 Survey
		2003	2010	2010	2013	2013	2018
C4I	Common Operational Picture Tech.	70	80	75	85	80	90
	Systems Surveillance and Control Tech.	60	80	77	84	79	88
	Data Fusion Tech.	52	63	75	86	77	87
	Situation and Threat Evaluation Tech.	65	75	69	80	73	80
	Information Management and Analysis Tech.	75	82	79	86	81	92
	Systems Interoperability Tech.	66	83	78	86	81	92

Table 2. C4I technology level survey results.

Weapon Systems	Technology Name	2003 Survey		2010 Survey		2013 Survey
		2003	2010	2010	2013	2013	2018
C4I	Common Operational Picture Tech.	70	80	75	85	80	90
	Systems Surveillance and Control Tech.	60	80	77	84	79	88
	Data Fusion Tech.	52	63	75	86	77	87
	Situation and Threat Evaluation Tech.	65	75	69	80	73	80
	Information Management and Analysis Tech.	75	82	79	86	81	92
	Systems Interoperability Tech.	66	83	78	86	81	92

Generally, the DTaQ conducts the defense science and technology survey on a three-year basis. This is a part of the sustainable management of defense technology, and the survey results are used as fundamental data for R and D planning of weapon systems and critical technology. In 2010, 2361 experts participated in 1485 technology examinations. In 2013, there were 707 experts conducting examinations for 1163 technologies.

Table 3 is a list of growth curves extracted with the Gompertz model. The survey examination results were well-reflected in the derived growth curves since all the values of the sum of squared errors (SSE) were close to zero.

Table 3. Derived growth curves (survey-based).

Technology Name	Survey Year	Technology Occurrence Year	Parameter α	Parameter β	Inflection Point (lnβ/α)	Sum of Squared Errors (SSE)
Common Operational Picture Tech.	2003	1964	0.07	1.78	1988	2.07568 × 10−10
	2010	1994	0.19	105.00	2003	1.53091 × 10−6
	2013	1991	0.15	36.69	2003	4.64809 × 10−7
Systems Surveillance and Control Tech.	2003	1983	0.12	8.74	1998	1.71566 × 10−7
	2010	1987	0.13	17.14	2000	8.36200 × 10−8
	2013	1987	0.12	15.07	2001	2.27701 × 10−5
Data Fusion Tech.	2003	1960	0.05	2.15	1995	1.99857 × 10−7
	2010	1996	0.22	227.02	2004	3.51160 × 10−6
	2013	1989	0.13	18.87	2002	4.11392 × 10−6
Situation and Threat Evaluation Tech.	2003	1959	0.06	1.72	1989	5.81908 × 10−9
	2010	1994	0.17	70.93	2004	4.23032 × 10−5
	2013	1971	0.07	3.26	1996	2.22658 × 10−8
Information Management and Analysis Tech.	2003	1948	0.05	1.03	1979	1.62091 × 10−8
	2010	1989	0.15	23.78	2000	2.47212 × 10−6
	2013	1995	0.19	114.73	2005	2.04372 × 10−5
Systems Interoperability Tech.	2003	1980	0.11	6.50	1996	4.25663 × 10−8
	2010	1991	0.17	42.97	2002	2.88339 × 10−5
	2013	1995	0.19	114.73	2005	2.04372 × 10−5

Table 3. Derived growth curves (survey-based).

Technology Name	Survey Year	Technology Occurrence Year	Parameter α	Parameter β	Inflection Point (lnβ/α)	Sum of Squared Errors (SSE)
Common Operational Picture Tech.	2003	1964	0.07	1.78	1988	2.07568 × 10−10
	2010	1994	0.19	105.00	2003	1.53091 × 10−6
	2013	1991	0.15	36.69	2003	4.64809 × 10−7
Systems Surveillance and Control Tech.	2003	1983	0.12	8.74	1998	1.71566 × 10−7
	2010	1987	0.13	17.14	2000	8.36200 × 10−8
	2013	1987	0.12	15.07	2001	2.27701 × 10−5
Data Fusion Tech.	2003	1960	0.05	2.15	1995	1.99857 × 10−7
	2010	1996	0.22	227.02	2004	3.51160 × 10−6
	2013	1989	0.13	18.87	2002	4.11392 × 10−6
Situation and Threat Evaluation Tech.	2003	1959	0.06	1.72	1989	5.81908 × 10−9
	2010	1994	0.17	70.93	2004	4.23032 × 10−5
	2013	1971	0.07	3.26	1996	2.22658 × 10−8
Information Management and Analysis Tech.	2003	1948	0.05	1.03	1979	1.62091 × 10−8
	2010	1989	0.15	23.78	2000	2.47212 × 10−6
	2013	1995	0.19	114.73	2005	2.04372 × 10−5
Systems Interoperability Tech.	2003	1980	0.11	6.50	1996	4.25663 × 10−8
	2010	1991	0.17	42.97	2002	2.88339 × 10−5
	2013	1995	0.19	114.73	2005	2.04372 × 10−5

It is important to note that the growth curves vary depending on the examination of the data, even though the curves are derived from the same technology. It comes from the relative comparison with the most developed country’s technology level as the criterion. That is, the technology level is not examined continuously with absolute comparison, so it is necessary to extract growth curves each time when conducting dynamic analysis.

4.2. Patent-Based Growth Curve

As keywords were extracted from definitions of target technologies, patent data from KIPRIS had been retrieved to ascertain the number of patents per year [64]. The status of retrieved patents are unexamined and registered during 1 January 1974 to 31 December 2013. In addition, the ended patents are excluded for make a comparison between common generations. We used the bibliographic data, abstract, scope of claims in the retrieved patent documents. Table 4 presents target technology, its definitions, and the patent search query.

Table 4. Patent search query.

Target Technology	Definition	Search Query
Common Operational Picture Tech.	Digital information network system. Real-time convergence and automation of all intelligence sources, including international information management system and joint command information system. Shows battlefield at a glance.	(COP + (common × operational × picture) + map) × (draw + drawing + graphic + show + showing + display + illustrate)
Systems Surveillance and Control Tech.	System management technology, conducts monitoring and controlling to satisfy user’s needs.	(C4I + system + computer) × (surveillance + control + management)
Data Fusion Tech.	Credible and implicative information production based on comparison, evaluation, and integration of each system’s various sensors	(data + information + sensor + knowledge) × (fusion + comparison + evaluation + convert)
Situation and Threat Evaluation Tech.	Risk evaluation technology based on current battlefield situation with data convergence	(situation + threat) × (evaluation + algorithm + order)
Information Management and Analysis Tech.	Classification, categorization, disposal of technology by distinguishing important data from useless information. Collection of various data such as weather, topography, target information, and location per a second.	(data + information + sensor + knowledge) × (store + storage + manage + management + analysis + classification + categorize + discard)
Systems Interoperability Tech.	Communication between program and application support technology. Enables connection among distributed system; computing environment; and other hardware, protocol, or communication network.	(hardware + protocol + system) × (interoperability + standard + communication + share + sharing + exchange + operation)

Table 4. Patent search query.

Target Technology	Definition	Search Query
Common Operational Picture Tech.	Digital information network system. Real-time convergence and automation of all intelligence sources, including international information management system and joint command information system. Shows battlefield at a glance.	(COP + (common × operational × picture) + map) × (draw + drawing + graphic + show + showing + display + illustrate)
Systems Surveillance and Control Tech.	System management technology, conducts monitoring and controlling to satisfy user’s needs.	(C4I + system + computer) × (surveillance + control + management)
Data Fusion Tech.	Credible and implicative information production based on comparison, evaluation, and integration of each system’s various sensors	(data + information + sensor + knowledge) × (fusion + comparison + evaluation + convert)
Situation and Threat Evaluation Tech.	Risk evaluation technology based on current battlefield situation with data convergence	(situation + threat) × (evaluation + algorithm + order)
Information Management and Analysis Tech.	Classification, categorization, disposal of technology by distinguishing important data from useless information. Collection of various data such as weather, topography, target information, and location per a second.	(data + information + sensor + knowledge) × (store + storage + manage + management + analysis + classification + categorize + discard)
Systems Interoperability Tech.	Communication between program and application support technology. Enables connection among distributed system; computing environment; and other hardware, protocol, or communication network.	(hardware + protocol + system) × (interoperability + standard + communication + share + sharing + exchange + operation)

We the search query equations for the target technologies by the specialists in national defense. Next, patent search results are shown in Table 5.

Table 5. Patent search results.

Year	Common Operational Picture Tech.	Systems Surveillance and Control Tech.	Data Fusion Tech.	Situation and Threat Evaluation Tech.	Information Management and Analysis Tech.	Systems Interoperability Tech.
1994	18	245	196	14	188	185
1995	37	538	504	50	462	470
1996	52	777	629	65	629	685
1997	80	1235	1031	103	1045	1042
1998	128	1422	1214	147	1265	1318
1999	170	1853	1509	144	1624	1724
2000	214	2046	1783	216	1983	2083
2001	287	2855	2324	314	2771	2818
2002	289	3231	2709	394	3191	3249
2003	413	4515	3750	501	4576	4303
2004	711	6820	5717	764	7077	6631
2005	961	9171	7738	947	9339	8607
2006	1141	9285	7754	1029	9432	8712
2007	1258	8824	7798	1309	9348	9042
2008	1039	7203	6401	1305	7682	7743
2009	893	6245	5641	1162	6693	6704
2010	846	6006	5259	1098	6304	6333
2011	795	5757	5048	1171	6057	6124
2012	655	4259	3772	823	4584	4652
2013	205	1228	1072	215	1335	1293

Table 5. Patent search results.

Year	Common Operational Picture Tech.	Systems Surveillance and Control Tech.	Data Fusion Tech.	Situation and Threat Evaluation Tech.	Information Management and Analysis Tech.	Systems Interoperability Tech.
1994	18	245	196	14	188	185
1995	37	538	504	50	462	470
1996	52	777	629	65	629	685
1997	80	1235	1031	103	1045	1042
1998	128	1422	1214	147	1265	1318
1999	170	1853	1509	144	1624	1724
2000	214	2046	1783	216	1983	2083
2001	287	2855	2324	314	2771	2818
2002	289	3231	2709	394	3191	3249
2003	413	4515	3750	501	4576	4303
2004	711	6820	5717	764	7077	6631
2005	961	9171	7738	947	9339	8607
2006	1141	9285	7754	1029	9432	8712
2007	1258	8824	7798	1309	9348	9042
2008	1039	7203	6401	1305	7682	7743
2009	893	6245	5641	1162	6693	6704
2010	846	6006	5259	1098	6304	6333
2011	795	5757	5048	1171	6057	6124
2012	655	4259	3772	823	4584	4652
2013	205	1228	1072	215	1335	1293

The growth curve was extracted based on the accumulated number of patents per year. Since patents applied in Korea go public 18 months after their application, patents that had been in effect for less than two years were used to construct the growth curve. For example, patents disclosed by 2001 were used to extract the growth curve, and it was compared with the survey-based growth curve. As parameters α and β were maintained, the upper limit L of patent-based growth curve was adjusted to 100. Then patent-based growth curve was translated parallel to X axis to make this correspond to the technology level of the survey-based growth curve. To make it easier to understand the patent trends by technologies, we show the plots of their applied numbers of patents by year in Figure 3.

Figure 3. Plots of retrieved patents.

Figure 3. Plots of retrieved patents.

We know that the technological trends of “Systems Surveillance and Control Tech.”, “Data Fusion Tech.”.

“Information Management and Analysis Tech.” and “Systems Interoperability Tech.” are similar each other.

Table 6 shows the parameters of both the survey-based and the patent-based growth curves:

$$Inclination=-\mathsf{\beta }\mathsf{\alpha }Ln\left(\frac{Y}{L}\right),L=100,Y:TechnologyLevel$$

(6)

Although technologies, years of examination, and technology levels were all the same, the survey-based and the patent-based growth curve showed different speeds of technology development. Therefore, it was necessary to verify whether there were any statistical differences.

Table 6. Survey-based growth curve vs. patent-based growth curve.

Technology Name	Survey Year	Technology Level	α		β		Inclination
			Survey	Patents	Survey	Patents	Survey	Patents
Common Operational Picture Tech.	2003	70	0.07	0.14	1.78	47.11	3.11	164.67
	2010	75	0.19	0.08	105	28.13	430.44	48.55
	2013	80	0.15	0.17	36.69	121.1	98.25	367.51
Systems Surveillance and Control Tech.	2003	60	0.12	0.14	8.74	47.06	32.15	201.93
	2010	77	0.13	0.08	17.14	28.41	44.84	45.74
	2013	79	0.12	0.17	15.07	121.13	33.68	383.47
Data Fusion Tech.	2003	52	0.05	0.18	2.15	66.14	3.66	398.67
	2010	75	0.22	0.09	227.02	28.56	1077.61	55.46
	2013	77	0.13	0.16	18.87	83.4	49.37	268.55
Situation and Threat Evaluation Tech.	2003	65	0.06	0.09	1.72	25.34	2.89	63.86
	2010	69	0.17	0.13	70.93	58.64	308.73	195.18
	2013	73	0.07	0.1	3.26	32.84	5.24	75.45
Information Management and Analysis Tech.	2003	75	0.05	0.15	1.03	48.4	1.11	155.15
	2010	79	0.15	0.13	23.78	59.03	66.42	142.9
	2013	81	0.19	0.17	114.73	118.75	372.07	344.57
Systems Interoperability Tech.	2003	66	0.11	0.14	6.5	43.48	19.61	166.94
	2010	78	0.17	0.1	42.97	30.38	141.57	58.88
	2013	81	0.19	0.16	114.73	83.13	372.07	227.02

Table 6. Survey-based growth curve vs. patent-based growth curve.

Technology Name	Survey Year	Technology Level	α		β		Inclination
			Survey	Patents	Survey	Patents	Survey	Patents
Common Operational Picture Tech.	2003	70	0.07	0.14	1.78	47.11	3.11	164.67
	2010	75	0.19	0.08	105	28.13	430.44	48.55
	2013	80	0.15	0.17	36.69	121.1	98.25	367.51
Systems Surveillance and Control Tech.	2003	60	0.12	0.14	8.74	47.06	32.15	201.93
	2010	77	0.13	0.08	17.14	28.41	44.84	45.74
	2013	79	0.12	0.17	15.07	121.13	33.68	383.47
Data Fusion Tech.	2003	52	0.05	0.18	2.15	66.14	3.66	398.67
	2010	75	0.22	0.09	227.02	28.56	1077.61	55.46
	2013	77	0.13	0.16	18.87	83.4	49.37	268.55
Situation and Threat Evaluation Tech.	2003	65	0.06	0.09	1.72	25.34	2.89	63.86
	2010	69	0.17	0.13	70.93	58.64	308.73	195.18
	2013	73	0.07	0.1	3.26	32.84	5.24	75.45
Information Management and Analysis Tech.	2003	75	0.05	0.15	1.03	48.4	1.11	155.15
	2010	79	0.15	0.13	23.78	59.03	66.42	142.9
	2013	81	0.19	0.17	114.73	118.75	372.07	344.57
Systems Interoperability Tech.	2003	66	0.11	0.14	6.5	43.48	19.61	166.94
	2010	78	0.17	0.1	42.97	30.38	141.57	58.88
	2013	81	0.19	0.16	114.73	83.13	372.07	227.02

4.3. Verification of Growth Curves

In order to verify whether there were any statistical differences between the survey-based and the patent-based growth curves of 2003, 2010, and 2013, it was necessary to perform a regression analysis.

Table 7 and Table 8 presents the summary of regression analysis results.

Table 7. Regression analysis results (95% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.83024	1.01808	1.12528	1.48076	N
	2010	1.4075	1.78188	1.08785	1.33166	N
	2013	1.4065	1.74371	1.42143	1.77499	Y
Systems Surveillance and Control Tech.	2003	1.20453	1.5255	1.21419	1.56421	Y
	2010	1.307	1.64241	1.03883	1.28447	N
	2013	1.35198	1.66048	1.45262	1.80009	Y
Data Fusion Tech.	2003	1.00648	1.13655	1.31166	1.69054	N
	2010	1.4246	1.81283	1.20777	1.47808	Y
	2013	1.39038	1.698	1.46498	1.79974	Y
Situation and Threat Evaluation Tech.	2003	0.86127	1.0216	1.05844	1.31829	N
	2010	1.43861	1.79015	1.39459	1.7073	Y
	2013	1.12966	1.32869	1.36189	1.62798	N
Information Management and Analysis Tech.	2003	0.63017	0.75559	1.08663	1.45505	N
	2010	1.31092	1.66297	1.27867	1.6154	Y
	2013	1.44633	1.80931	1.44932	1.80252	Y
Systems Interoperability Tech.	2003	1.11871	1.43656	1.17763	1.53158	Y
	2010	1.35227	1.71637	1.15153	1.43342	Y
	2013	1.44633	1.80931	1.40929	1.75456	Y

Table 7. Regression analysis results (95% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.83024	1.01808	1.12528	1.48076	N
	2010	1.4075	1.78188	1.08785	1.33166	N
	2013	1.4065	1.74371	1.42143	1.77499	Y
Systems Surveillance and Control Tech.	2003	1.20453	1.5255	1.21419	1.56421	Y
	2010	1.307	1.64241	1.03883	1.28447	N
	2013	1.35198	1.66048	1.45262	1.80009	Y
Data Fusion Tech.	2003	1.00648	1.13655	1.31166	1.69054	N
	2010	1.4246	1.81283	1.20777	1.47808	Y
	2013	1.39038	1.698	1.46498	1.79974	Y
Situation and Threat Evaluation Tech.	2003	0.86127	1.0216	1.05844	1.31829	N
	2010	1.43861	1.79015	1.39459	1.7073	Y
	2013	1.12966	1.32869	1.36189	1.62798	N
Information Management and Analysis Tech.	2003	0.63017	0.75559	1.08663	1.45505	N
	2010	1.31092	1.66297	1.27867	1.6154	Y
	2013	1.44633	1.80931	1.44932	1.80252	Y
Systems Interoperability Tech.	2003	1.11871	1.43656	1.17763	1.53158	Y
	2010	1.35227	1.71637	1.15153	1.43342	Y
	2013	1.44633	1.80931	1.40929	1.75456	Y

The verification result explains the differences in slope between the survey-based and the patent-based growth curves. At the 95% confidence interval, the common operational picture technology had differences in verification results in 2003 and 2010.

Table 8. Regression analysis results (99% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.79962	1.0487	1.06732	1.53872	N
	2010	1.34646	1.84292	1.0481	1.37141	Y
	2013	1.35152	1.79869	1.36379	1.83263	Y
Systems Surveillance and Control Tech.	2003	1.1522	1.57783	1.15713	1.62127	Y
	2010	1.25231	1.69709	0.99877	1.32452	Y
	2013	1.30168	1.71078	1.39596	1.85674	Y
Data Fusion Tech.	2003	0.98528	1.15776	1.24989	1.75232	N
	2010	1.3613	1.87613	1.1637	1.52216	Y
	2013	1.34023	1.74816	1.4104	1.85432	Y
Situation and Threat Evaluation Tech.	2003	0.83513	1.04774	1.01608	1.36065	Y
	2010	1.3813	1.84747	1.3436	1.75828	Y
	2013	1.09721	1.36114	1.31851	1.67137	Y
Information Management and Analysis Tech.	2003	0.60972	0.77604	1.02656	1.51512	N
	2010	1.25352	1.72037	1.22377	1.6703	Y
	2013	1.38715	1.86849	1.39174	1.86011	Y
Systems Interoperability Tech.	2003	1.06689	1.48839	1.11992	1.58928	Y
	2010	1.29291	1.77573	1.10557	1.47938	Y
	2013	1.38715	1.86849	1.353	1.81086	Y

Table 8. Regression analysis results (99% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.79962	1.0487	1.06732	1.53872	N
	2010	1.34646	1.84292	1.0481	1.37141	Y
	2013	1.35152	1.79869	1.36379	1.83263	Y
Systems Surveillance and Control Tech.	2003	1.1522	1.57783	1.15713	1.62127	Y
	2010	1.25231	1.69709	0.99877	1.32452	Y
	2013	1.30168	1.71078	1.39596	1.85674	Y
Data Fusion Tech.	2003	0.98528	1.15776	1.24989	1.75232	N
	2010	1.3613	1.87613	1.1637	1.52216	Y
	2013	1.34023	1.74816	1.4104	1.85432	Y
Situation and Threat Evaluation Tech.	2003	0.83513	1.04774	1.01608	1.36065	Y
	2010	1.3813	1.84747	1.3436	1.75828	Y
	2013	1.09721	1.36114	1.31851	1.67137	Y
Information Management and Analysis Tech.	2003	0.60972	0.77604	1.02656	1.51512	N
	2010	1.25352	1.72037	1.22377	1.6703	Y
	2013	1.38715	1.86849	1.39174	1.86011	Y
Systems Interoperability Tech.	2003	1.06689	1.48839	1.11992	1.58928	Y
	2010	1.29291	1.77573	1.10557	1.47938	Y
	2013	1.38715	1.86849	1.353	1.81086	Y

However, there was no difference in verification result in 2013. According to the characteristics of technology or examination methods, the confidence interval can be adjusted. At the 99% confidence interval, three out of 18 confidence intervals did not overlap.

4.4. Correction of the Growth Curves

The patent-based growth curve’s rate of increase per year, shown in Table 9, was used to extract the proposed curve of Formula (4).

Table 9. Patent growth rate.

Year	Common Operational Picture Tech.	Systems Surveillance and Control Tech.	Data Fusion Tech.	Situation and Threat Evaluation Tech.	Information Management and Analysis Tech.	Systems Interoperability Tech.
1994	-	-	-	-	-	-
1995	2.055556	2.195918	2.571429	3.571429	2.457447	2.540541
1996	0.945455	0.992337	0.898571	1.015625	0.967692	1.045802
1997	0.747664	0.791667	0.775771	0.79845	0.817045	0.777612
1998	0.684492	0.508766	0.514407	0.633621	0.54432	0.553317
1999	0.539683	0.439412	0.422216	0.379947	0.452494	0.465946
2000	0.441237	0.337068	0.350777	0.413002	0.380395	0.384034
2001	0.410587	0.351774	0.338479	0.424899	0.385075	0.375383
2002	0.293103	0.294504	0.294777	0.374169	0.320157	0.314673
2003	0.323922	0.317913	0.315153	0.346234	0.347773	0.317003
2004	0.421209	0.364375	0.365327	0.392197	0.399064	0.370924
2005	0.400584	0.359126	0.362164	0.349189	0.376406	0.351191
2006	0.339583	0.267518	0.266424	0.281224	0.276193	0.263083
2007	0.279493	0.200577	0.211569	0.279224	0.214492	0.216176
2008	0.180413	0.136377	0.14334	0.217609	0.145135	0.152215
2009	0.131362	0.104049	0.110484	0.159134	0.110424	0.114379
2010	0.109999	0.090636	0.092755	0.129726	0.093663	0.096959
2011	0.093124	0.079659	0.081476	0.122464	0.082286	0.085472

Table 9. Patent growth rate.

Year	Common Operational Picture Tech.	Systems Surveillance and Control Tech.	Data Fusion Tech.	Situation and Threat Evaluation Tech.	Information Management and Analysis Tech.	Systems Interoperability Tech.
1994	-	-	-	-	-	-
1995	2.055556	2.195918	2.571429	3.571429	2.457447	2.540541
1996	0.945455	0.992337	0.898571	1.015625	0.967692	1.045802
1997	0.747664	0.791667	0.775771	0.79845	0.817045	0.777612
1998	0.684492	0.508766	0.514407	0.633621	0.54432	0.553317
1999	0.539683	0.439412	0.422216	0.379947	0.452494	0.465946
2000	0.441237	0.337068	0.350777	0.413002	0.380395	0.384034
2001	0.410587	0.351774	0.338479	0.424899	0.385075	0.375383
2002	0.293103	0.294504	0.294777	0.374169	0.320157	0.314673
2003	0.323922	0.317913	0.315153	0.346234	0.347773	0.317003
2004	0.421209	0.364375	0.365327	0.392197	0.399064	0.370924
2005	0.400584	0.359126	0.362164	0.349189	0.376406	0.351191
2006	0.339583	0.267518	0.266424	0.281224	0.276193	0.263083
2007	0.279493	0.200577	0.211569	0.279224	0.214492	0.216176
2008	0.180413	0.136377	0.14334	0.217609	0.145135	0.152215
2009	0.131362	0.104049	0.110484	0.159134	0.110424	0.114379
2010	0.109999	0.090636	0.092755	0.129726	0.093663	0.096959
2011	0.093124	0.079659	0.081476	0.122464	0.082286	0.085472

In consideration of patent disclosure in 18 months after application, patent data of two years before the examination year are used. Consequently, rates of increase in 2001, 2008, and 2011 are used for adjusting the growth curves. Figure 4 shows time series plots of all technologies by the values of patent growth rate.

Figure 4. Plots of patent growth rate.

Figure 4. Plots of patent growth rate.

We find that the time series plots of all technologies except the technology of “Common Operational Picture Tech.” are similar to each other. Table 10 shows the regression analysis results at 95% confidence interval. Regression coefficients of the proposed growth curve had 16 confidence intervals overlapping with both survey-based and patent-based growth curves. The other two confidence intervals did not overlap with any growth curves.

Table 10. Regression analysis results of proposed growth curves (95% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Proposed Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	(Yes/No)	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.83024	1.01808	1.12528	1.48076	0.93152	1.17398	Y
	2010	1.40750	1.78188	1.08785	1.33166	1.23202	1.51885	Y
	2013	1.40650	1.74371	1.42143	1.77499	1.41641	1.76416	Y
Systems Surveillance and Control Tech.	2003	1.20453	1.52550	1.21419	1.56421	1.20789	1.53856	Y
	2010	1.30700	1.64241	1.03883	1.28447	1.15687	1.43591	Y
	2013	1.35198	1.66048	1.45262	1.80009	1.44388	1.78645	Y
Data Fusion Tech.	2003	1.00648	1.13655	1.41756	1.73126	1.13448	1.36212	N
	2010	1.42460	1.81283	1.20777	1.47808	1.24914	1.65367	Y
	2013	1.39038	1.69800	1.46498	1.79974	1.40514	1.84521	Y
Situation and Threat Evaluation Tech.	2003	0.86127	1.02160	1.05844	1.31829	0.89413	1.14930	Y
	2010	1.43861	1.79015	1.39459	1.70730	1.36207	1.79914	Y
	2013	1.12966	1.32869	1.36189	1.62798	1.29215	1.63263	Y
Information Management and Analysis Tech.	2003	0.63017	0.75559	1.08663	1.45505	0.79789	1.00969	N
	2010	1.31092	1.66297	1.27867	1.61540	1.23572	1.68989	Y
	2013	1.44633	1.80931	1.44932	1.80252	1.39136	1.86079	Y
Systems Interoperability Tech.	2003	1.11871	1.43656	1.17763	1.53158	1.08710	1.52584	Y
	2010	1.35227	1.71637	1.15153	1.43342	1.18527	1.59423	Y
	2013	1.44633	1.80931	1.40929	1.75456	1.35612	1.81559	Y

Table 10. Regression analysis results of proposed growth curves (95% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Proposed Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	(Yes/No)	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.83024	1.01808	1.12528	1.48076	0.93152	1.17398	Y
	2010	1.40750	1.78188	1.08785	1.33166	1.23202	1.51885	Y
	2013	1.40650	1.74371	1.42143	1.77499	1.41641	1.76416	Y
Systems Surveillance and Control Tech.	2003	1.20453	1.52550	1.21419	1.56421	1.20789	1.53856	Y
	2010	1.30700	1.64241	1.03883	1.28447	1.15687	1.43591	Y
	2013	1.35198	1.66048	1.45262	1.80009	1.44388	1.78645	Y
Data Fusion Tech.	2003	1.00648	1.13655	1.41756	1.73126	1.13448	1.36212	N
	2010	1.42460	1.81283	1.20777	1.47808	1.24914	1.65367	Y
	2013	1.39038	1.69800	1.46498	1.79974	1.40514	1.84521	Y
Situation and Threat Evaluation Tech.	2003	0.86127	1.02160	1.05844	1.31829	0.89413	1.14930	Y
	2010	1.43861	1.79015	1.39459	1.70730	1.36207	1.79914	Y
	2013	1.12966	1.32869	1.36189	1.62798	1.29215	1.63263	Y
Information Management and Analysis Tech.	2003	0.63017	0.75559	1.08663	1.45505	0.79789	1.00969	N
	2010	1.31092	1.66297	1.27867	1.61540	1.23572	1.68989	Y
	2013	1.44633	1.80931	1.44932	1.80252	1.39136	1.86079	Y
Systems Interoperability Tech.	2003	1.11871	1.43656	1.17763	1.53158	1.08710	1.52584	Y
	2010	1.35227	1.71637	1.15153	1.43342	1.18527	1.59423	Y
	2013	1.44633	1.80931	1.40929	1.75456	1.35612	1.81559	Y

In the 99% confident interval, all growth curves were overlapped. Therefore, the proposed growth curve in this research represents both of the growth curves, statistically.

Therefore, we verified the performance of the proposed methodology in this paper from the result of Table 11. Additionally, this research contributes to the technological forecasting and valuation for national defense technology.

Table 11. Regression analysis results of proposed growth curves (99% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Proposed Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	Upper	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.79962	1.04870	1.06732	1.53872	0.89199	1.21351	Y
	2010	1.34646	1.84292	1.04810	1.37141	1.18525	1.56562	Y
	2013	1.35152	1.79869	1.36379	1.83263	1.35971	1.82086	Y
Systems Surveillance and Control Tech.	2003	1.15220	1.57783	1.15713	1.62127	1.15398	1.59247	Y
	2010	1.25231	1.69709	0.99877	1.32452	1.11137	1.48141	Y
	2013	1.30168	1.71078	1.39596	1.85674	1.38803	1.84230	Y
Data Fusion Tech.	2003	0.98528	1.15776	1.36641	1.78241	1.09737	1.39923	Y
	2010	1.36130	1.87613	1.16370	1.52216	1.24914	1.65367	Y
	2013	1.34023	1.74816	1.41040	1.85432	1.40514	1.84521	Y
Situation and Threat Evaluation Tech.	2003	0.83513	1.04774	1.01608	1.36065	0.89413	1.14930	Y
	2010	1.38130	1.84747	1.34360	1.75828	1.36207	1.79914	Y
	2013	1.09721	1.36114	1.31851	1.67137	1.29215	1.63263	Y
Information Management and Analysis Tech.	2003	0.60972	0.77604	1.02656	1.51512	0.76336	1.04423	Y
	2010	1.25352	1.72037	1.22377	1.67030	1.23572	1.68989	Y
	2013	1.38715	1.86849	1.39174	1.86011	1.39136	1.86079	Y
Systems Interoperability Tech.	2003	1.06689	1.48839	1.11992	1.58928	1.08710	1.52584	Y
	2010	1.29291	1.77573	1.10557	1.47938	1.18527	1.59423	Y
	2013	1.38715	1.86849	1.35300	1.81086	1.35612	1.81559	Y

Table 11. Regression analysis results of proposed growth curves (99% confidence interval).

Technology Name	Survey Year	Survey-Based Growth Curve Regression Coefficients		Patent-Based Growth Curve Regression Coefficients		Proposed Growth Curve Regression Coefficients		Confidence Interval
		95% Confidence Interval		95% Confidence Interval		95% Confidence Interval		Overlap
		Lower	Upper	Lower	Upper	Lower	Upper	(Yes/No)
Common Operational Picture Tech.	2003	0.79962	1.04870	1.06732	1.53872	0.89199	1.21351	Y
	2010	1.34646	1.84292	1.04810	1.37141	1.18525	1.56562	Y
	2013	1.35152	1.79869	1.36379	1.83263	1.35971	1.82086	Y
Systems Surveillance and Control Tech.	2003	1.15220	1.57783	1.15713	1.62127	1.15398	1.59247	Y
	2010	1.25231	1.69709	0.99877	1.32452	1.11137	1.48141	Y
	2013	1.30168	1.71078	1.39596	1.85674	1.38803	1.84230	Y
Data Fusion Tech.	2003	0.98528	1.15776	1.36641	1.78241	1.09737	1.39923	Y
	2010	1.36130	1.87613	1.16370	1.52216	1.24914	1.65367	Y
	2013	1.34023	1.74816	1.41040	1.85432	1.40514	1.84521	Y
Situation and Threat Evaluation Tech.	2003	0.83513	1.04774	1.01608	1.36065	0.89413	1.14930	Y
	2010	1.38130	1.84747	1.34360	1.75828	1.36207	1.79914	Y
	2013	1.09721	1.36114	1.31851	1.67137	1.29215	1.63263	Y
Information Management and Analysis Tech.	2003	0.60972	0.77604	1.02656	1.51512	0.76336	1.04423	Y
	2010	1.25352	1.72037	1.22377	1.67030	1.23572	1.68989	Y
	2013	1.38715	1.86849	1.39174	1.86011	1.39136	1.86079	Y
Systems Interoperability Tech.	2003	1.06689	1.48839	1.11992	1.58928	1.08710	1.52584	Y
	2010	1.29291	1.77573	1.10557	1.47938	1.18527	1.59423	Y
	2013	1.38715	1.86849	1.35300	1.81086	1.35612	1.81559	Y

5. Conclusions

This paper conducts a technology level evaluation for sustainable management of defense technology. In particular, this research proposes an evaluation methodology of dynamic analysis in order to overcome limitations of static analysis. This methodology has achieved decisive results in the following ways. First, it suggests a technology growth curve based on Delphi analysis, and a verification method to rectify the problem of inadequate reliability. Methods of static analysis have been applied to technology level evaluation cases, and it has been difficult for both performing time series analysis and understanding technology development phases. Existing studies have used the Delphi or survey analysis to collect data and forecast technology level with technology growth curve. The growth curves should have been verified because the curves are extracted with the Delphi method’s limited data. However, the existing studies have not proposed any definite solutions for ensuring reliability. Thus, this research proposes patent analysis to verify the growth curve. Keywords are extracted from the definitions of target technologies to construct growth curves, based on the number of patents per year. Then, regression analysis was performed to figure out whether there are any overlapping coefficient confidence intervals of the survey-based and the patent-based curves. As the patent-based statistical verification process is proposed, it is possible to verify the survey-based growth curve with high reliability. Second, an effective method is suggested to adjust technology level evaluation extracted by the Delphi technique. The purpose of technology level evaluation is to establish an appropriate strategy for technology development. Thus, it is highly important to determine how technology develops over time. Although it is possible to know the technology development with the growth curve, the reliability of the growth curve is critical. This research proposes a new growth curve analysis by weighted sum of the survey-based and the patent-based growth curves. The proposed growth curve shows that regression coefficient confidence intervals of both the survey-based and the patent-based growth curves overlap in the 99% confidence level. Thus, the proposed growth curve represents both the growth curves statistically. The methodology suggested in this research can provide dynamic technology level data for sustainable management of technology. Therefore, it is expected to be useful for institutes or research laboratories where the technology securing strategy is studied. Traditional approaches to forecast defense technology were based on static forecasting. This is to forecast sustainable technology at a static point from the time period but a technology trend is constantly changing. Thus, in this paper we need a dynamic approach to forecast the future technology to overcome the problem of static forecasting in sustainability of defense technology. This research contributes to the dynamic forecasting works for sustainable technologies in diverse domains, as well as the defense technology. Though our research has some limitations, this is worthy as a first trial for dynamic technology forecasting in national defense technology. This is an important issue in sustainable technology forecasting. For further research, it is necessary to analyze not only patents but also various quantitative data in order to verify and adjust it in a more precise way.