«ABSTRACT The purpose of this research study is to perform a comparison and contrast analysis on three F. Davis Cardwell published longitudinal case ...»
6) Since all three cases studied exhibited a positive change in both quality and productivity when XP methods were used, we must conclude that the relationship is bi-directional and that these improvements induce an improvement in schedule performance as well. In essence, XP methods allow us to “have it both better and cheaper; an additional benefit is that this improved efficiency allows us to have it faster as well.” This conclusion is supported by observation from the studies that the products developed using XP methods included more functionality than was originally specified; in other words, the development teams were able to deliver more than was requested, while remaining within the original schedule constraints. In light of these results, the null-hypothesis that there are no significant differences between schedule performance of traditionally developed software projects and projects developed using agile methods is not supported by the evidence presented.
Problems with computer software are referred to as defects or more properly, faults. A fault occurs when a software developer makes a mistake in performing a software activity. For example, a designer may misunderstand a requirement and create a design that does not match the actual needed functionality. This design fault becomes encoded in the software, and can cascade into other faults, such as incorrect documentation. Thus, a single mistake can generate many faults, and faults can occur in any constructed software. (Pfleeger & Atlee, 2010, p. 6) Faults are the result of errors made during the creation or maintenance of software. Failures, on the other hand, are departures from a system’s desired behavior.
An example of a failure that will be familiar to any Microsoft Windows user is the infamous stop error, also known as the “Blue Screen of Death.”
THE PERFORMANCE OF AGILE METHODS:
COMPARISON TO TRADITIONAL DEVELOPMENT METHODSFigure 16. Windows “Blue Screen of Death” at the 2008 Beijing Olympics A fundamental problem in software engineering is the problem of how to handle software faults in complex systems. Modern computing systems are extremely complex; a few lines of a computer program written in an Object Oriented programming language can have an instruction explosion of 100:1 or more when compiled into machine language; some versions of the compilation system used may introduce faults into the software while others may not. Further, in most cases software doesn’t operate as a stand-alone system; modern computer systems are amalgams of system and application software that must work together as a cohesive unit. [For example, on the computer used to write this document, the Windows folder alone contains nearly 100,000 files in 20,000 folders.] Subtle versioning differences in supporting software can cause unintended interaction, triggering seldomexecuted faulty code and causing failures. A particularly insidious type of fault that can be inconspicuous until it causes a failure is the memory leak. A memory leak occurs when memory that was previously allocated by a programmer is not properly de-allocated. When executed, memory that is no longer in use by the software is still considered reserved, and that memory cannot be used by any process until it is marked as available. When it occurs, computer memory “leaks away” or is wasted, causing degraded performance or even out-of-memory failures in extreme cases. (Sahgal,
2012) Additionally, faults may cause failures only intermittently, making them extremely difficult to find.
An example is a design flaw in the Therac-25 software system, in which a variable was programmed th to increment rather than being set to a specific value; on every 256 cycle this variable would overflow, allowing operation of the machine in a potentially deadly configuration. (Leveson, 1995, pp.
35 – 38) Finally, in a perfect world malware such as viruses, worms, and Trojan horses wouldn’t exist.
Ours is not a perfect world; malware can introduce faults or trigger otherwise faulty but unused code in order to produce exploitable failures. The consensus within the software development community is that it is practically impossible to remove all potential faults from software, especially considering the fact that the software’s operating environment is constantly changing; the focus within the community has been on minimizing faults and mitigating risk from faults.
THE PERFORMANCE OF AGILE METHODS:COMPARISON TO TRADITIONAL DEVELOPMENT METHODS
Additionally, understanding and correcting software problems can be confounded by the nature of software itself. Hardware is tangible; a resistor that is out of tolerance can easily be measured and rejected. Software, on the other hand, is intangible. Software is, by its very nature, an abstraction of reality; it isn’t possible to measure whether software is within tolerance, as it would be with a resistor or other physical component. Generally, software is checked for problems by inspection, either visually or by other computer systems in order to verify that the software is correct grammatically and syntactically, or by testing, in order to verify that the software is correct logically, i.e., that it does what it was intended to do.
Generally, faults represent an inside view of the system, as seen by the eyes of the developers. On the other hand, failures represent an outside view of the system—a problem that is seen while exercising the software, either by developers during testing or by users during use. A complicating factor is that software faults may—or may not—correspond to a failure; for example, if faulty code is never executed or a particular state is never entered, then the fault will never cause the system to fail.
(Pfleeger & Atlee, 2010, p. 6) Quality for the cases studied is measured as a function of defects divided by thousands of lines of executable code [KLOEC.] Further, quality is divided into two categories, internally visible quality and externally visible quality. Internally visible quality refers to faults that are found and corrected during development, while externally visible quality refers to failures that are found after the product is released to the customer.
The Sabre-A study experienced a decrease in total fault density, a decrease in internally visible fault density, a decrease in externally viewable fault density, and fewer severe faults when XP methods replaced traditional methods. (Layman, et al., 2004a., p. 8) The Sabre-P study experienced a decrease in total fault density, similar levels of internally visible fault density, and a decrease in externally visible fault density when XP methods were used, compared to published industry averages. (Layman, et al., 2004b., p. 8) The IBM study experienced a decrease in total fault density, a decrease in internally visible fault density, a decrease in externally visible fault density, and similar levels of fault severity when XP methods replaced traditional methods. (Williams, Krebs, et al., 2004, p. 8) Additionally, all three teams reported the perception that XP methods did a better job of providing what the customer actually wanted, as opposed to what the customer originally requisitioned. In light of these results, the null-hypothesis that there are no significant differences in quality of traditionally developed software projects and projects developed using agile methods is not supported by the evidence presented.
SATISFACTION COMPARISONSStakeholder satisfaction for the cases studied is measured through the use of a survey, the Shodan Adherence Survey [see Appendix C]. The survey is used to gather subjective data from the affected stakeholders as to how well the specific project adhered to XP principles, and how well XP functioned
within the project. The survey is divided into six sections:
1. Foundations. Covers automated user tests, customer acceptance tests, test-first design, pair programming, and refactoring. The goal is to ascertain the sturdiness of the XP base used for developing the specific project.
2. Customer Planning. Covers release planning/the planning game, customer access, short releases, and stand-up meetings. The goal is to determine how effective customer interaction and XP planning activities are for developing the specific project.
3. Teaming. Covers continuous integration, coding standards, and collective code ownership. The goal is to determine how effective XP personnel methods are for developing the specific project.
THE PERFORMANCE OF AGILE METHODS:
COMPARISON TO TRADITIONAL DEVELOPMENT METHODS
4. Craftsmanship. Covers sustainable pace, simple design, and metaphor/system of names. The goal is to determine how effectively XP principles support the development effort for the specific project.
5. Introspection. Covers lessons learned, growth, morale, and artifact reduction. The goal is to determine what the respondents learned about the XP process as a result of using XP for developing the specific project.
6. Perspectives. Asks the respondent to describe three XP practices that are a cause for concern, and three XP practices that hold the most promise.
The Sabre-A study expressed a high level of stakeholder satisfaction, and a morale rating of 68.1%.
The calculated average of the Shodan survey scores supplied for the Sabre-A New release is 76.67%.
Anecdotally, the customer base was highly satisfied with the product of the Sabre-A project; one customer remarked that [it was] “one of the most professionally developed products he had ever used.” (Layman, et al., 2004a., pp. 5 – 7) Customer satisfaction and morale ratings are not summarized for the Sabre-P study as they are for the other studies; it is worth noting, however, that the calculated average of the Shodan survey scores supplied is 70.27%, which is comparable to the results reported in the other studies. (Layman, et al., 2004b., pp. 5 – 7) The IBM study expressed a high level of stakeholder satisfaction and a morale improvement of 11% when XP methods replaced traditional methods. (Williams, Krebs, et al., 2004, pp. 5 – 7) The calculated averages of the Shodan survey scores supplied for the IBM study are 56.33% [Old] and 73.67% [New]. The customer for this project reported satisfaction with the performance of the development team. Unfortunately, the XP-EF tool was modified between the IBM study and the Sabre-A study; one result of the change is that morale scores are reported using a relative scale for the IBM study, while they are reported using an absolute scale for the Sabre-A study, confounding a meaningful comparison. If we assume that a 50% morale rating is the midpoint between high morale and low morale, then the Sabre-A team reports an 18.1% improvement when XP methods replaced traditional methods, a finding that is similar to the results of the IBM study. [See Appendix E for a summary of the survey results.] The Sabre-A team felt that daily stand-up meetings, short releases, and the rapid feedback afforded by unit testing were particularly helpful XP practices. Mandatory pairing and collective code ownership were considered problematic XP practices for the team; additionally, some developers noted that automated tests were difficult to write in some instances, due to technological limitations. (Layman, et al., 2004a., pp. 5 – 7) The Sabre-P team felt that continuous integration and simple design were helpful XP practices. Unit testing, pair programming, and refactoring were considered problematic XP practices for the team. (Layman, et al., 2004b., pp. 5 – 7) The IBM team felt that stand-up meetings, customer acceptance tests, and collective ownership were helpful XP practices. Short releases, testfirst design, and refactoring were considered problematic XP practices for the team. (Williams, Krebs, et al., 2004, pp. 5 – 7) In light of these results, the null-hypothesis that there are no significant differences in stakeholder satisfaction for traditionally developed software projects and projects developed using agile methods is not supported by the evidence presented.
Management satisfaction is not covered in the main case studies per se. Examination of management satisfaction in similar cases is illuminative. Decreased fault density, improved quality, improved developer morale, and increased customer satisfaction are all generally beneficial to management.
Additionally, management reports that the agile focus on collaboration positively affects flexibility;
multiple developers are familiar enough with the project to continue developing it, in case of loss of critical personnel. (Ferreira & Cohen, 2008, p. 3) These benefits come with a caveat, however: Agile methods allocate some traditional management authority to other entities, such as developers and customers. Care must be taken to ensure that managers understand and buy into agile methods, in order to avoid “turf wars” and potential resentments. (Williams & Cockburn, 2003, p. 3)
THE PERFORMANCE OF AGILE METHODS:COMPARISON TO TRADITIONAL DEVELOPMENT METHODS