Comparing the Performance of Neural Networks for Predicting Peak Outflow from Breached Embankments when Back Propagation Algorithms Meet Evolutionary Algorithms

This investigation provides a review of some methods for estimation of peak outflow from breached dams and presents an effective and efficient model for predicting peak outflow based on artificial neural network (ANN). For this reason the case study data on peak outflow discharge were compiled from various sources and reanalyzed using the ANN technique to see if better predictions are possible. By employing two important effective parameters namely, height (Hw) and volume (Vw) of water behind the dam at failure, four scenarios were addressed. To train the models two different algorithms were examined namely, back propagation (BP) and imperialist competitive algorithm (ICA). Among the BP algorithms, Levenberg–Marquardt (LM), resilient back propagation (RP), fletcher–reeves update (CGF), and scaled conjugate gradient (SCG) were ut ilized. Therefore, 20 different ANN models were developed and compared to each other. Results showed that both Hw and Vw parameters are similarly dominant in estimating the peak outflow d ischarge. Among the different training functions, LM was the best, because of higher coefficient of determination (R=0.87) and lower error (RMSE=9616). Comparing the results of ANN and empirical formulas indicated higher ANN performance, so such formulas are better to be replaced with superior ANN model.


Introduction
Dam failure is a catastrophic phenomenon that can lead to large damages to human life and property. Overviewing of historical dam failures shows that overtopping and piping were the major causes of dam failures. Overtopping is mostly dangerous for embankment dams because it washes away or erodes very quickly the dam's materials. In Pip ing, water seeps under the dam and gradually erodes the dam materials. The extension of this phenomenon may lead to dam collapse. The various modes o f b reach fo rmat ion in emban kment dams, and the large nu mber o f factors that influence the o ut flo w ch aract erist ics, are d ifficu lt to d escribe with rigo rous ly p recis e math emat ical fo rmu las. Because of complexity and uncertainty resulting fro m the wide range of values of the effective parameters, it is worthwhile to reduce the mathematical co mplexity of the problem and to present simp le methods to predict the outflow characteristics from breached embankment. Pred iction of peak outflo w is very important because of the emergency action plan preparation and risk assessment. In some fo rmer investigations, case study data were used to develop emp irical formula by relating peak outflow to the height of water behind the dam and/or volume of water behind the dam. So me investigators developed single-variable equations [13], [21], [36][37][38], [42], and some others presented mult i-variable equations [10], [16], [30], [31], [44]. Hagen [17] introduced "dam factor" as the product of height of water and reservoir storage volume at the time of failure, and proposed a formula relat ing the peak-breach outflow to the dam factor. So me investigators applied the dam factor in their p roposed equations [11], [25].
Although applying empirical equations based on statistical regression is simp le in practice, they are unable to estimate the values of peak outflow accurately. It is felt that this is partly due to the complexity of the phenomenon involved and low accuracy of data driven fro m h istorical dam failures [14], and partly because of the limitation of the analytical tool commonly used by most of the earlier investigators namely, tradit ional statistical regression. Nowadays, traditional statistical analysis has been replaced by newly alternative approaches in many cases. Artificial neural networks (ANN) as an alternative approach have advantages over statistical models like their data-driven nature, model-free form of predict ions, and tolerance to data errors [4]. A NN beside its simplicity and generalizing ability has been used widely in hydraulic [3], [5], [7], [24], [40], [41], [45]. Most of the studies have been done by feed forward back propagation (BP) neural network. The standard back propagation algorithm (SBPA) as the training algorithm in BP has some problems including the low training convergence speed and easy entrapment in a local minimu m [19]. During last decades, researchers have tried to overco me these problems and improve the ANN performance. Ramirez et al. [32] proposed the resilient back propagation (RP) training function for network training to predict the rainfall in Sao Pau lo, Brazil. According to their findings, using RP could improve the results. Some other researchers proposed Levenberg-Marquardt algorithm (LMA ). Noori et al. [28] evaluated different t rain ing functions on ANN operation for predicting the monthly stream flow and found that fletcherreeves update (CGF) and scaled conjugate gradient (SCG) models had the best performance in wet and arid periods, respectively. Chau [9] used particle swarm optimization to optimize the network weights and biases for predicting water level in Sh ing Mun River. He co mpared the results with the SBPA results and showed the superiority of h is model. Rogers et al. [33] p roposed genetic algorith m (GA ) instead of SBPA.
Accordingly, the objective of this study is to comp ile previously presented case study data on peak outflow discharge fro m breached embankments, and reanaly ze the resulting database using the technique of neural networks with a v iew towards seeing if better predictions are possible. Hereby, d ifferent training functions consisted of RP, CGF, SCG, and LMA are examined. Besides, a new evolutionary algorith m, imperialist competit ive algorith m (ICA), proposed by Atashpaz-Gargari and Lucas [1] is used to optimize the network weights and biases. Finally, the results are co mpared to result of empirical formu las.

Data Collection And Empirical Formulas
Valuable documented information is available fro m historical embankment failu res. Babb and Mermel [6] summarized over 600 dam incidents throughout the world; however, high quality and detailed information was lacking in most cases. Here the data fro m 93 embankment dam failures (Table 1) are co llected fro m variety of sources [15], [16], [30], [35], [39], [43], [44]. During the decades, several researchers compiled some data of well-docu mented case studies in efforts to develop predictive relat ions for breach peak outflows. A mong them, Kirkpatrick [21] proposed a formula based on analysis of data from 13 failed embankment dams and 6 hypothetical failures: where Q p = peak outflo w (m 3 /s); and H w = height of the water behind the dam at failure (m). USBR [42] developed an equation using case study data fro m 21 failed dams including several concrete arch and gravity dams: Singh and Snorrason [37] used some simu lated dam failures and presented Eq. (3): where S= reservoir storage at normal pool (m 3 ). To evaluate the applicability of peak outflow relationships as a function of reservoir volu me, Evans [13] examined several man -made and natural dam failu res and proposed a relationship described as below equation: (4) in which V w = volu me of the water behind the dam at failu re (m 3 ). MacDonald and Langridge-Monopolis [25] co llected and analyzed data on a number of historical dam failures and developed graphical relat ionships for predicting breach characteristics for erosion type breaches. They also developed a relationship based on dam factor to estimate the peak outflows fro m dam failures: (5) where H w V w = dam factor (m 4 ). Using dam factor, Costa [11] analyzed 31 dam failu res and presented a relationship based on regression analysis of the case studies: Froehlich [16] assembled data fro m 22 embankment dam failures fro m various sources. The data were used to evaluate and compare several empirical equations as well as to obtain a new empirical exp ression for rapidly estimation of peak outflow fro m a breached embankment. The formula was derived based on mult iple regression analysis: Pierce et al. [30] co mpiled a database of 87 embankment failures including experimental data and performed a statistical analysis using mult iple regression technique to predict peak outflow discharge as a function of the height and volume of water behind the dam:

Neural Network Models
An ANN is a 'black bo x' approach wh ich has great capacity in predict ive modeling [22]. It is a proper mathematical structure having an inter-connected assembly of simple processing elements or nodes. A typical network would consist of three layers of neurons namely, input, hidden, and output; in which each neuron acting as an independent computational element. Neural networks are universal approximators [20], and many theoretical and experimental works have shown that a single hidden layer is sufficient for A NNs to approximate any complex nonlinear function [12], [27], [29]. Accordingly, in this study single hidden layer ANNs were used. The tangent-sigmoid and linear functions were chosen as the activation function respectively in the hidden and output layers, and mean square error (MSE) was utilized as performance function. To check the over-fitting problem in the calibration and testing steps, stop training algorith m method was used.
There are different training functions to optimize the network weights and biases in the case of BP algorith m. They can be divided into two different categories; the first one uses heuristic techniques, while the second one uses standard numerical optimizat ion techniques. On the other hand, some other algorith ms are available wh ich use evolutionary optimization techniques (e.g. ICA) for training the network. So me details of ICA are availab le in the literatures [1,2]. The quick review of the above algorith ms is as follow: Heuristic techni ques Heuristic techniques were developed fro m an analysis of the performance of the standard steepest descent algorithm. Gradient descent, gradient descent with mo mentu m, gradient descent which has variable learn ing rate, grad ient descent with mo mentu m which has variable learning rate and RP are the most famous train ing functions which use heuristic techniques to update the network parameters. Because of using sigmoid transfer function in the hidden layer of mu lti-layer ANN with BP algorith m, the performance of the above training functions except RP can be affected [28]. So, in this research just RP is evaluated among the heuristic techniques.

Standard numerical opti mization techni ques
The SBPA adjusts the weights in the steepest descent direction (negative of the gradient) wh ich the performance function is decreasing most rapidly. Although it decreases most rapidly along the negative of the gradient, this does not necessarily produce the fastest convergence. Conjugate gradient algorith ms (CGA ) are one of the fastest optimization techniques. In the CGAs for faster convergence than steepest descent directions a search is performed along conjugate directions. All the CGAs start out by searching in the steepest descent direction on the first iteration [28]. Discussion of CGAs and their application in neural networks are availab le in Hagan and Demuth [18]. The CGAs require that a line search be performed. Charalambous method, which was worked out in the present research, is a hybrid search which was designed to be used in comb ination with a CGA for neural network training. It uses a cubic interpolation together with a type of sectioning [8]. Various algorith ms for CGA are available, e.g. CGF, SCG, Po lak -Ribiere updates (CGP), and Powell-Beale restarts (CGB). Co mparison of these algorith ms on ANN operation for predicting the monthly stream flow has been done by Noori et al. [28]. Results indicated that the CGF and SCG were the best algorith ms with superior performance, so they are emp loyed in the present study.
LMA developed by Levenberg [23] and Marquardt [26] is another type of standard numerical optimization techniques. LMA p rovides a nu merical solution to the problem of minimizing a nonlinear function over a space of parameters of the function. These min imization problems arise especially in least squares curve fitting and nonlinear programming. LMA interpolates between the Gauss -Newton algorith m (GNA) and the method of gradient descent. LMA is mo re robust than GNA, which means that in Imperialist competiti ve algorithm ICA is a new evolutionary algorith m in the evolutionary computation field based on the human's socio-political evolution. The algorithm starts with an init ial random population called countries. Some of the best countries in the population be selected as the imperialists and the rest form the colonies of these imperialists. In an N d imensional optimization problem, a country is a 1 × N array. This array is defined as below: The cost of each country is found by evaluating the cost function f at the variables (p 1 ,p 2 , . . . ,p n ). Then (10) The algorith m starts with N in itial countries and the N imp best of them (countries with minimu m cost) chosen as the imperialists. The remained countries are colonies that each of them belongs to an emp ire. The init ial colonies belong to imperialists in convenience with their powers. To distribute the colonies among imperialists proportionally, the normalized cost of an imperialist is defined as follow: (11) where, c n is the cost of nth imperialist and C n is its normalized cost. Each imperialist that has more cost value, will have less normalized cost value. The power of each imperialist is calculated as below and based on that the colonies distributed among the imperialist countries: On the other hand, the normalized power of an imperialist is assessed by its colonies. Then, the initial nu mber of colonies of an emp ire will be: (13) where, NC n is init ial number of colonies of nth emp ire and N col is the number of all colon ies. To distribute the colonies among imperialist, NC n of the colonies are selected randomly and assigned to their imperialist. The imperialist countries absorb the colonies towards themselves using the absorption policy. The absorption policy shown in Fig. 1, ma kes the main core of this algorithm and causes the countries move towards to their min imu m optima.
The imperialists absorb these colonies towards themselves with respect to their power that described in Eq. (14). The total power of each imperialist is determined by the power of its both parts, the empire power p lus percents of its average colonies power.  (14) where TC n is the total cost of the nth empire and ξ is a positive number which is considered to be less than one. In the absorption policy, the colony moves towards the imperialist by x unit. The d irection of movement is the vector fro m co lony to imperialist, as shown in Fig. 1. In this figure, the distance between the imperialist and colony shown by d and x is a random variable with uniform distribution.
(15) where β is greater than 1 and is near to 2. So, in this investigation the proper choice is β=2.
In ICA, to search different points around the imperialist, a random amount of deviation is added to the direction of colony movement towards the imperialist. In Fig. 1, this deflection angle is shown as θ, wh ich is chosen randomly and with a uniform distribution. While moving toward the imperialist countries, a colony may reach to a better position, so the colony position changes according to position of the imperialist.
In our imp lementation γ is π/4 (Rad). In this algorith m, the imperialistic co mpetition has an important role. During the imperialistic co mpetition, the weak emp ires will lose their power and their colonies. To model this co mpetit ion, firstly we calcu late the probability of possessing all the colonies by each emp ire considering the total cost of empire. (17) where NTC n is the normalized total cost of nth empire. The possession probability of each emp ire is calculated as below: (18) After a while, all the empires except the most powerful one will collapse and all the colonies will be under the control of this unique empire. ICA had a great performance in both convergence rate and better global optima achievement [2,34]. In the p resent research, this algorith m is also employed in ANN modeling as a training algorith m to determine the network's parameters.

Results and Discussion
To evaluate different models, a database containing of 93 field and experimental dataset was used ( Table 1). Out of the total input-output pairs, 85% were used for calibration (train ing and validation) and the remaining 15% were saved for testing. Since it was needed that the testing data be used for evaluating the empirical formulas, these data were chosen randomly fro m the category which had not been used in derivation of those formulas. A mong the Eqs. 1 to 8, some take H w as the independent variable (Eqs. 1 and 2), so me take V w as the independent variable (Eqs. 3 and 4), some take the dam factor as the independent variable (Eqs. 5 and 6), and the others take both of H w and V w as the independent variables (Eqs. 7 and 8). Accordingly, four scenarios were defined here and four sets of ANNs were developed in which the input variables were different. Furthermo re, in o rder to evaluate the various training algorith ms in each scenario, RP, CGF, SCG, LMA, and ICA were examined. So, totally 20 models were developed and compared to each other. To evaluate and compare the results, three statistical measures were utilized namely, coefficient of determination (R2); mean absolute error (MAE); and root mean square error (RM SE). For confidently evaluating the results, each modeling procedure, including calibrat ion and testing steps, was iterated 50 t imes and the average values of the statistical measures were calculated. Therefore the expected values of the statistical measures were presented in this research.

First scenario's Results (Consi dering H w as the Input Variable)
In the first scenario it was assumed that peak outflow discharge (Q p ) just relates to H w . Table 2 shows the quantitative results of the ANN models as well as the emp irical formu las. According to the results, all ANN models have moderate R 2 value and high error. The RMSE for LMA is 14405 in testing step, which shows a high error. The error measures values in Table 2 show a different performance in training and testing steps e.g. the MAE values in testing steps are almost 3 times mo re than training step's values. This ratio for RMSE values is almost equal to 2. Such results may be due to the below reasons: 1. The H w parameter can't be sufficient for a model to predict the peak outflow d ischarge, lonely. In other words, some other parameters are important in peak d ischarge prediction and consequently the models developed based on just H w has low efficiency.
2. The datasets are insufficient, so lower training may be happened; however, lack of datasets is a general problem in ANN application. It is clear that the more data availab le, the more accuracy in ANN modeling.
3. The method of data derivation in the field or laboratory has been inaccurate, and consequently some inaccurate data in the database may have been led to such error. Scale effects in experimental researches as well as estimation of peak outflow discharge fro m water-mark and stage-discharge curves in the field may be the main sources of error in recording the data. Fig. 2 presents the schematic performance of ANN model with ICA train ing function as the best model co mpared with observed values for both calibration and testing steps. As most of the values of peak outflow in training and testing step aren't estimated correctly, the model performance isn't satisfactory. Besides, as can be seen in Table 2, the Kirkpatrick and USBR formu las have a weak performance. The coefficient of determination for Kirkpatrick formu la is very low (near 0.76) and its RMSE is 11732 wh ich shows very high error. The situation is a little better for USBR formula but it is not satisfactory too because of lo w R 2 (=0.74) and high error (RM SE =10656). Schemat ic co mparison between ANN model and emp irical formulas is available in Fig. 3. Th is figure indicates that most of the points are scattered and the predicted values are underestimated or overestimated. This figure confirmed that H w can't be sufficient for effective prediction of Q p fro m b reached embankments.

Second Scenario's Results (Considering V w as the Input Variable )
In the second scenario the same database was used for training and testing steps, but the simulat ion was done with V w instead of H w as the input variable. The quantitative results are presented in Table 3. The coefficient of determination of each ANN model is good in both steps but the error measures are relatively high. A mong the models RP has the highest RMSE value in the testing step and SCG has the lowest one. In this scenario although ICA has a good R 2 value in testing step, it can't be effective because of different R 2 values in train ing and testing steps as well as high RMSE value.
According to Table 3, it can be inferred that SCG has the best performance. The R 2 values of Singh and Snorrason and Evans formulas respectively are 0.78 and 0.79, wh ich means moderate performance co mpared to SCG. Fig. 4 shows the performance of ANN model co mpared to observed values. Although the results are some better than the first scenario, there are some points that aren't predicted satisfactorily. Fig.  5 shows the ANN performance co mpared to emp irical formulas. It is obvious that all the models underestimate the observed values especially extreme values. However the situation for SCG is much better, but not satisfactory.     By co mparing Tables 2 and 3, it is found that R 2 and error values for different models in the second scenario are mo re satisfactory. Therefore the models based on V w parameter may lead to more reliable results due to extensive range of V w parameter in dam breach database (3700 -600 m 3 ).

Third Scenario'S Results (Considering Dam Factor as the Input Vari able)
In the third scenario, dam factor was emp loyed in A NN modeling. Results of the best ANN model is presented in Fig.  6. It is observed that the predicted values are not so good. In some points the prediction values are about two times over than the observed values. The quantitive results are prepared in Table 4. In this scenario ICA has high R 2 value and low RMSE in testing, so the ICA can be a good model but not exactly an effective model due to difference between the model performance in train ing and testing steps. Fig. 6 shows the results of ICA in calibrat ion and testing steps.   Table 4 is also consisted of quantitative results of two emp irical formulas which use dam factor for peak outflow estimation. The coefficient of determination for these formulas is almost good, but their prediction erro r is very high (RM SE ≈ 14000). Therefore, practical using of such emp irical formu las may lead to high error and undesirable effects. The effects may be irrecoverable because of their underestimat ion especially for large dams (Fig. 7). Co mparing Table 2, 3, and 4 indicates that the models in the third scenario have better performance than the models in the first and second scenario.

Fourth Scenari o's Results (Consi dering H w and V w as the Input Variables)
In the last scenario both variables (H w and V w ) are independently used in the ANN modelling. The quantitive results are presented in Table 5. As the results show, all the ANN models could predict peak outflow values very well because of high coefficient of determinations (R 2 > 0.80). Co mparing the results indicates that MAE and RMSE values of ICA model are lower and its R 2 value is higher than the other models' in both training and testing steps.  Fig. 8 shows ANN model performance in calibrat ion and testing steps separately. It is clear that the model have a good so that most of the predictions are approximately coincided with their corresponding observed values. As it is illustrated in Fig. 8, there are appro ximately 5 points in the testing step and one point in the testing step which aren't predicted satisfactorily. Like to the previous scenarios, most of the badly predicted points are corresponding to extreme values (large dams). Accordingly, one can strongly conclude that insufficient data or lack of h istorical data of large breached embankments is the main reason of lower training of the network. The information on nu mber of nodes, number of epochs required to achieve the error goal, and the CPU t ime taken in the case of each training scheme has been presented in Table  6. It is remarkable that the information is the average result of 50 iterat ions for each model. A PC having a Pentiu m IV processor (CPU: Co re i5, 2.53 GHz), was utilized in this study. As a matter of general information, it can be seen that ICA t rains the network with fewer epochs compared to the other BP algorith ms, but in a h igher amount of time. LMA trains the network in a more nu mber of epochs but in a fraction of the time compared with ICA. The other BP algorith ms also had similar performance as LMA, which indicates their acceptable training efficiency. the results of Froehlich and Pierce formu las are presented in Table 5.
These formulas have relatively weak perfo rmance due to low R 2 value and high errors. Schemat ic co mparison of ANN model with empirical formulas is availab le in Fig. 9. It is seen that empirical formulas underestimate the observed values, but ANN model doesn't have such problem. Co mparing Tab le 5 with Tables 2-4 shows that fourth scenario is more realistic and reliab le than other scenarios due to its better results. Thus, it can be concluded that ANN with ICA as the train ing function and both H w and V w as the inputs is the most effective and efficient model. Furthermore, according to the obtained results from all scenarios, it may be concluded that the effect of various parameters (i.e. height and volume of water behind the dam) on the amount of peak outflow discharge is approximately the same.

Conclusions
This investigation focused on evaluating the ANN performance for predict ing peak outflo w fro m breached embankment dams. In o rder to find an effect ive model, four scenarios were defined. In scenarios I and II, it was assumed that Q p related respectively to H w and V w . In scenario III it was assumed that Q p related to the dam factor, and in scenario IV it was assumed that Q p related to the both H w and V w . Also to train the network, two train ing algorith ms were emp loyed: ICA and BP. ICA is a new evolutionary algorithm, in the evolutionary co mputation field, and BP algorith m is a common method of teaching ANNs called mult i-stage dynamic system optimizat ion method. By considering the statistic indices as well as CPU t ime taken, ICA was recognized as the best training algorith m in most of the scenarios. However, ICA takes a little mo re CPU t ime taken in co mparison with the other algorith ms. Also among the different scenarios, scenario IV was the best. In the scenarios I to III all ANN models underestimate the real extreme values, because the values related to large dam cases (i.e. H w >15 m and/or V w >50 mc m) are limited, so ANNs have not effectively been calib rated around the extre me values. Accordingly, enlarg ing the database by adding the values of new breached large dam cases or generating the synthetic extreme data may be effective. Moreover, results showed that the effect of H w and V w on the amount of peak outflow discharge was the same. This investigation indicated that although empirical formulas were simp ly applicable in practice, they lead to unsatisfactory results due to low R 2 value and high error in their estimat ion especially fo r large breached dams. This is because of limited databases which were used in the formulas derivation as well as the low flexib ility of the tradit ional method of regression analysis to data variation.