«Countering the Adversary: Effective Policies or a DIME a Dozen? Stephen M. Shellman Brian Levey Hans H. Leonard Violent Intranational Political ...»
The better the model, the more confident our inferences will be in determining the impact of a specific U.S. action. Third, we determine a treatment variable of interest – perhaps a specific DIME action such as training the military but such treatment variables could be as specific as carrying out a raid in a particular village. We then divide all of our cases by the treatment variable such that we have data on violent attacks that took place without any U.S. military training and data on violent attacks that took place in the presence of U.S. military training. Next, we match those cases on the control variables such as levels of repression, economic characteristics, social characteristics, previous dissident violence, etc. In each instance of matching, we will end up with cases that do not match up and we discard those cases. We use multiple matching algorithms such as nearest neighbor propensity score matching, exact matching, and genetic algorithms. We then choose the algorithm that provides the best matched set of cases and we use various quantitative measures to determine how well these cases match up. Given the technique, however, we do not need the cases to match-up 100% because we then model these data as opposed to doing a simple difference of means test. The model controls for error in the matching process. Once we have a matched set of cases, we run a statistical model on the cases that did not experience the treatment (control group).
Once we have estimated the model we derive the parameter estimates from the untreated control group.
We then apply those estimates to the treated group data to generate ―predicted values.‖ If this was a linear regression model, we would multiply each observation for each variable by its estimated coefficient estimate and add them together to produce the predicted value for that observation. These predicted values represent the ―counterfactual‖ series. If the treatment effect has no impact, the counterfactual series should be equivalent (or statistically insignificant) to the control group series. If the data for the dependent variable are the same across the cases, there is no effect. However, if those data differ given our model, methodology, and our controls, we can attribute that difference to the treatment variable. An example follows.
In addition to the matched case counterfactual methods, we also employ time-series impact assessment methodologies. It is a complementary method and if we draw the same inferences across techniques we have increased confidence in our results. Impact assessment methodologies essentially code the presence of a particular action over time (e.g., military training exercise) and estimates the impact of that action within the confines of a statistical model shown to model the variance in a dependent variable of interest (e.g., violent attacks). One of the useful properties of such models is its ability to track impacts over time and vary the functional form of the relationship to uncover the best fitting curve (See Wood 1988; Shellman & Stewart 2007a). For example, does military training quell violent attacks overtime, does it first increase and then decrease them, do the two variables exhibit a long-run polynomial relationship? The impact assessment models allow for such analysis and the model provides insight into where such uncovered relationships are by chance or are statistically significant. We illustrate both techniques below.
6.0 Military Training in India For our pilot study, we examined the impact of U.S. military training in India. The training exercises included the Malabar naval training exercises, COPE air force training, and 14 army training exercises.
These exercises took place more or less constantly between 2002 and 2006. We can divide these up in the future and examine specific training exercises but for this study we grouped them together. While we examined several dependent variables, we will concentrate on the findings for violent attacks by separatist groups in India. Note that we can isolate specific groups (e.g., JKLF, New People‘s Army, etc.) in the future and/or explore different dependent variables.
Figure 3 The Model-Predicted and Actual Separatist Violent Events: India 1997-2006 Figure 3 plots our negative binomial model predicted values against the actual values for the number of violent events carried out by separatists in India. The two series correlate at.87 indicating that our model does a very good job at explaining the variance of such violent events overtime. After settling on our model, we then split our cases up across the treatment variable: military training exercises carried out between early 2002 and 2006. We matched our observations on each variable in our model (e.g., government repression, human rights abuses, separatist nonviolent and cooperative actions, levels of democracy, economic indicators such as unemployment and inflation, and societal sentiment towards the government and the separatists) across the treatment variable. In this instance a genetic algorithm provided the best results in terms of maximizing ―balance‖ across the cases.
Table 1 Percent Balance Improvement Between Treated and Control Groups for India Military Training Model*
Table 1 displays the balance improvement overall (i.e., distance) and for each independent variable.
Balance (i.e., how evenly matched the cases are across the variables) improved upwards of 98% for some variables and only as little as 45% for others. The overall propensity score (i.e., distance) yielded a high value (94%) indicating that the cases were well matched. If we had obtained perfect matches, there would be no need to perform additional analyses on the data. We could simply perform difference of means between the control and treatment group. However, since we did not achieve perfect balance, the matching procedure can be thought of as a method of pre-processing the data prior to traditional parametric analysis. Once we obtained our matched cases, we applied our core model to the control group and predicted a counterfactual series using the coefficients from this model and data from the treatment group. Figure 4 shows the counterfactual series overlaid on the actual series of separatist violence events.
The figure highlights the 1.92 difference in attacks during the military training period. That is, U.S.
military training exercises in India increase separatist violent events (controlling for other factors) by about 2 violent events per month. Two fewer violent events would have transpired per month if U.S.
military training had not occurred between 2002 & 2006. The effect is statistically significant at the 95% level.
Figure 4 The Counterfactual Series (red-dotted) Overlaid on the Actual Series (black-solid) In addition to the counterfactual analysis, we also performed an impact assessment analysis. The coefficients from the models are displayed in Table 2. In short, we added a dummy variable coded 1 under military training and 0 under no military training to our model and estimated its effect. We found that on average, military training increases attacks by about 2.56. Our impact assessment finding is consistent with our counterfactual finding adding credence to our results.
Table 2 Negative Binomial Impact Assessment Results for Military Training on Separatist Violence, India 1997-2006
average military training might increase attacks by about 2 per month, was this effect in fact constant across the period or did it vary with respect to the duration of military training. We specified a model that would allow us to test these hypotheses by adding a duration variable to the model and testing its functional relationship to separatist attacks. Figure 5A shows the average change in attacks over time during the military training period. As one can see from the graph, military training increased attacks in the short-run and decreased attacks in the long run. In fact, there were fewer attacks at the end of 2006 than there were in 2002 when training. We need to also perform such an analysis with the counterfactual method to see if our finding is further supported by examining the month to month differences in the counter-factual and actual series. We believe we would find supporting evidence. We conclude that while training may increase attacks on average over the period, the frequency of violent attacks was less at the end of the training than it was at the beginning. We would argue that military training is an overall success and that staying the course ultimately provides greater long-term benefits.
Moreover, we also wanted to discuss some findings with respect to our sentiment. Figure 5B reports our findings. Our results show that as sentiment towards the separatists becomes more positive the number of separatists‘ attacks increase. In contrast, as sentiment towards the host government increases, separatist attacks decrease. While many authors have argued that this should be the case, there is not much empirical evidence for the argument. We think our results are important in understanding the ebb and flow of societal sentiment and how such sentiment affects violence on the ground. Given the attention that the U.S. government and U.S. military COIN manuals shed on winning over the populace we feel we have the capabilities to understand the effects of various U.S. actions on such sentiment and indirectly on the intensity of political violence. All in all, we have the capabilities to begin understanding the dynamic relationship we posited in Figure 1 above. Which U.S. actions win the support of the people and how does such support affect the intensity of conflict? These questions are at the heart of future studies.
Finally, note that we could take our model a bit further examining the direct and indirect effects of military training on various separatist and host nation activities. For example, we might find that U.S.
military training positively affects societal sentiment towards the separatists and that as societal sentiment increases towards separatists, separatists are more likely to increase the number of violent attacks and decrease their nonviolent activities. So U.S. military training might have a direct positive effect on separatists high hostility (violence) and a positive indirect effect through societal sentiment. In sum, the total effect of military training on separatist violent activities is greater than the direct effect by itself. It is important to trace these effects and understand which variables amplify relationships and which variables dampen effects. Some effects can be cancelled out while others can be strengthened when examining complex direct and indirect effects. This is also the subject of forthcoming papers.
7.0 Positive U.S. Diplomatic Actions & Military Actions in the Philippines In the second analysis we examined the effects of positive U.S. diplomatic actions and high-level military actions on separatist violence in the Philippines (1997-2006). Separatist violence was measured by events such as armed clashes with government forces, the use of unconventional violence such as car bombings and suicide attacks, and abductions and hostage takings. Our treatment variables were U.S.
high-level positive diplomatic actions and high-level military actions. Diplomacy was operationalized using the following event types: signings of major agreements such as the agreement to cooperate on the War on Terror and promotions of investment by U.S. companies such as Exxon Mobile, military cooperation with the Philippines government such as training exercises, and the sharing of intelligence information with the Philippines. We operationalized military actions by the mobilization of forces, moving fleets into strategic positions, imposing blockades and restrictions of movement, military training and joint forces exercises, and assassinations of terrorist leaders.
Figure 6 highlights the periods of U.S. diplomatic and military actions across the time-series of separatist violence in the Philippines. The green shading highlights diplomatic actions, the yellow highlights military actions, and the purple indicates that both were present.
Figure 6 Separatist Violent Events Philippines (1997-2006) with Highlighted Periods of U.S. Diplomacy and Military Actions Again, the genetic algorithm we employed proved to provide the most percentage increase in the balance across the cases. The overall percent balance improvement score (i.e., distance) improved upwards of 98% indicating that the cases were well matched. See Table 3 for balance statistics across a couple matching procedures.
Table 3 Percent Balance Improvement Between Treated and Control Groups for Philippine Diplomacy Model*
Again, the modeling strategy assumes and factors in error associated with the matching procedure. Once we had the cases matched, we ran the model on the untreated cases and generated our counterfactual series, we then took the average difference to compute the effect of high-level positive diplomatic actions. Figure 7 shows the counterfactual series (dotted red line) overlaid on the actual series of separatist violence events (grey solid line). The figure highlights a 4.2 difference in attacks during the high-level diplomatic actions periods. That is, U.S. positive diplomacy in the Philippines reduced separatist violent events (controlling for other factors) by about 4 violent events per month. Four fewer violent events would have transpired per month if U.S. diplomacy had occurred during those times. The effect is statistically significant at the 95% level.
Figure 7 Average Effects of High-Level U.S. High-Level Positive Diplomatic Actions on Separatist Violent Events: Philippines 1997-2006 We proceeded in much the same way for military actions. Figure 8 shows the counterfactual series (dotted red line) overlaid on the actual series of separatist violent events (grey solid line). The figure depicts a 5.88 difference in attacks during the high-level military actions periods. That is, U.S.