Live Fire Testing

Summary

My first assignment in Northrop Grumman was to support the vulnerability and live fire test systems engineering teams on our aircraft products. This was the time in my career where I switched from being a mechanical to systems engineer. Over the course of two years, I lead a number of live fire tests from requirements generation to completion. Effort required coordination and leadership over testing, analysis, and design groups, as well as intimate contact with customer, Director for Operation Test and Evaluation (DOT&E), and test range to ensure test met objectives. This validated that the vulnerability of the products was within required specification with significant margin.

Data Analytics

All tests were designed and run successfully, but a major challenge arose when we were required to take the immense quantity of data and use them to draw key conclusions. What did the data show? What variables altered mattered most to the overall conclusion?

One major leap forward we made as a team (that I helped spearhead) was the use of statistical hypothetical testing, or p-value. This value helps a user support or reject a null hypothesis (H). In our application, the null hypothesis could be that varying skin thickness (a test variable) of an aircraft has an overall effect on the test outcome (lets say we were shooting the skin to test its robustness). The p-value, which is our evidence against H, has to achieve a certain value to tell the user to support/reject the hypothesis.

After a user creates a hypothesis H, they need to establish a confidence value, or alpha. This alpha value is typically set to 0.05 (or 95% confidence). If the p-value is greater than alpha, then the user must reject the hypothesis. If not, then the user should maintain their original hypothesis. The p-value can be visualized in the normal distribution graph below.

Once the confidence bands (alpha) are determined, it is time to visualize the data and its effect on the hypothesis made. In the graphic below, the example of skin thickness is shown against the slowdown ratio of a kinetic hit though the skin. There are tools that effortlessly allow the user to see the model (the result of all test data from the test just completed), and show what variables contribute the most to the overall dataset solution. As shown below, the steeper the red line (the p-value), the more likely that this factor (skin thickness, for example) has a very significant influence to the overall dataset solution.

Conclusion

This was a method used very lightly in the live fire test community, but with my contribution and efforts, it resulted in an increased usage across the engineering team. An engineer now has the ability to use this method and develop intuitive models as a result of the test to make key conclusions and recommendations in their test report.