Agile software development is a relatively new approach for software development. This approach has attracted the attention of development companies in the recent few years. Nowadays many leading organizations have adopted Agile in all or some of their projects such as AOL, CNBC, Yahoo!, Google, Microsoft, Siemens, Shopzilla, Rockstar, and many more.
The term Agile was first used in the 1990’s in many published articles. These articles were based on people looking for new approach to software development process. In 2001, Jim Highsmith, Bob martin and many others, who were involved in agile concept, organized a workshop. They exchanged ideas and came up with the manifesto for Agile Software Development. After the active involvement of many authors who got together to understand each other approaches, Agile alliance was formed. This alliance is a non-profit group intended to search for agile methods and approaches as well as sponsors for annual agile conferences
In the recent few years the Agile methods become popular. More and more organizations are moving toward adopting agile software development. This is driven by the constant need of producing better, faster and cost-effective software solutions and at the same time maintaining a high rate of employee job satisfaction.
This article presents an agile software development case study for a government entity in the United Arab Emirates (U.A.E). The challenges faced in this study are compared with challenges reported in other case studies in this domain.
2. AGILE ADOPTION CASE STUDIES
many agile case studies were conducted to assess the merits and challenges of agile adoption. Srinivasan and Lundqvist presented a case study of the agile development experience of a software product development firm ”GameDevCo”. This research identified several challenges faced by the company to adopt the Scrum method. The main challenges are as follows:
Challenge 1: Requirements: One of the challenges that when the firm first adopted Scrum, the product owners were recently appointed, which lead to product backlog consisting of user stories that are in different levels of abstraction and inconsistent with the previous versions of the software. Another challenge was not involving the team in the initial estimation of existing projects. Accordingly, the negative effects of ambiguous requirements contributed to poor quality and schedule overruns.
Challenge 2: Scrum implementation: The company faced many challenges during the implementation phase. This is mainly due to fact that the project managers and many members of the agile team had limited knowledge of agile method. This resulted in making the agile team spending most of the time arguing about what the books say about the process rather than implementing what the process say.
Challenge 3: Organization learning: Originally, the sprint review meeting was designed to support organization learning. Due to the fact that each Scrum master had his own view of what is Scrum, the learning has become nonexistent. Another challenge was that in the beginning of Scrum adoption, focusing in the process took away from the overall learning at the firm.
Another case study conducted by Conboy et. al emphasizes on the key people challenges in agile development and recommendation to overcome each challenge. The reported challenges with suggested recommendations are as follows:
Challenge 1: Developer fear caused by transparency of skill deficiencies. Recommendations:
• Make stand up meetings voluntary for new junior developers.
• Assign dedicated mentor for new staff.
• Pair Weaker developers with those who had more experience.
Challenge 2: The need for developers to be a ”master of all trades”. Recommendations:
• Use pair programming and pair rotation to distribute knowledge and facilitate learning.
• Encourage task self-assignment to allow developer work in different areas.
Challenge 3: Increased reliance on social skills. Recommendations:
• Combine development and training program to provide customized training materials on social skills.
• Using proper documentation to back up communication
Challenge 4: A lack of business knowledge among developers. Recommendations:
• Ask Customer to run training sessions on basic topics within the business domain and on company specific areas.
• Provide small training modules on a frequent basis.
• Recruit staff and graduates with a combination of IT and business Knowledge.
Challenge 5: The need to understand and learn values and principles of Agile, not just the practices. Recommendations:
• Ensure multiple members get agile training or attend agile conferences.
• Agile coaching and championing.
• Ensure cross-team observation/validation of agile practices.
• Assess agility in terms of Agile values not practice adherence.
Challenge 6: Lack of developer motivation to use agile methods. Recommendations:
• Try to have multiple ”bought-in” developers on each team.
• Collect and share successful adoption stories and positive experiences.
Challenge 7: Implications of devolved decision-making. Recommendations:
• Build a sharing and learning environment to empower team decision making.
• Implement a democratic voting system.
• Project manager should play the role of facilitator.
Challenge 8: The need for agile compliant Performance evaluation . Recommendations:
• Performance evaluation needs to consider breadth of skills, not just depth.
• Performance evaluation to apply much higher weighting for mentoring, voluntary contributions etc.
• Perform 360 degree feedback .
Challenge 9: Lack of Agile-specific recruitment policies and suitably trained IT graduates. Recommendations:
• Develop specific recruiting practices tailored for agile methods to hire right people.
• Use team recruiting to find right person working in the team.
• Put newly recruited graduates on agile projects to get hands on experience.
The case study conducted by Lindvall et. al. illustrates the benefits and challenges of adopting agile development in specifically large organizations. Small organization showed interest in Agile to seek alternatives of traditional approaches, which they found bureaucratic and inflexible. The same needs drove large organizations as well. In addition, to increase their productivity, the need to meet deadlines on time, the difficulty to communication between team members as well as the difficulty of decomposing high level requirement into detailed specifications. Large organizations also questioned if agile practices can develop large, complex and safety critical systems. To decide whether to adopt agile development method, four large organizations decided to conduct pilot projects using XP method. These organizations were ABB, DaimlerChrysler, Motorola and Nokia. The results of these projects were rewarding for many reasons such as: high quality code, faster implementation phase, and easiness of learning the XP method. On the other hand, some of the challenges they faced are:
• The integration of each pilot project with the project environment’s existing processes.
• The necessity to tailor XP method to the organization requirement as XP method is not ”one-size-fits-all” software development process. The amount of tailoring varies from one project to another.
• Adding support for cross-team communication, especially in large teams that might be located in different geographical locations.
• Cultural differences between team members from different nationalities.
3. AGILE ADOPTION EXPERIENCE IN THE UAE
This section presents a case study for an agile software development adoption Scrum. The case study is carried out by a government entity in the U.A.E. and presents the challenges and compare them to the challenges faced by other researchers. This article refers to this government entity as entity ’S’ which consists of an IT Department that includes around 200 employees. The organization structure of ’S’ is as shown in Figure 1, ’S’ consists of a System development Section that has over 50 employees. The employees’ roles include head of division, branch managers, project managers, system analysts, developers, technical writers and testers. The System development division teams develop most projects using Oracle or .NET technologies. Entity ’S’ used a customized waterfall development method.
In 2009, a task force was formed to discuss issues and concerns faced during the software development process. The task force included representatives from every branch under the division. A consultant, who is an expert in agile software development, was hired. The consultant performed a thorough analysis of the currently used customized waterfall model; he also conducted a series of meetings with the management and with the task force. The recommendation was to adopt Scrum approach. Accordingly, a training course ”Introduction to Agile Development” on Scrum was offered to 30 team members from different levels and roles. The course was offered by the same consultant. A survey were conducted immediately after finishing the course, there was a consensus among the team members that Scrum would be better option than the current Waterfall method. At the end of the training course all team members were enthusiastic and optimistic, and were looking forward to lead the change in ’S’. In 2010, based on the feedback from the task force, entity ’S’ management decided to adopt agile development method ”Scrum”.
Challenges of Agile Adoption in S
The Agile teams in ”S” have faced new issues during the Scrum adoption phase. To understand such challenges, many meetings and surveys were conducted with every development team and with the upper management of ”S” and were recorded. Following is the of eight challenges in priority order.
Challenge 1: Missing the Agile Master Role Agile master or Agile coach is an essential role during Agile adopting process in any organization. Agile coach is considered a consultant for the team in every step of a project using any Agile method, such as Scrum, that is responsible of providing guidance and helps to succeed in adopting Agile. Entity ”S” management recognized the need to hire a contractor as an agile master. However the position was not filled due to financial constraints.
Challenge 2: The overzealous teams
After attending a course on Agile methods, many of entity “S” teams wanted to adopt Agile methods as soon as possible hoping it will solve all their previous development challenges known for traditional methods. This overzealous team fast adoption of Agile resulted in a decrease in productivity because the development cycle took longer time due to many mistakes in implementation. This decrease in productivity led many team members to be less optimistic and started to lose interest in agile methods.
Challenge 3: The Absent of a Pilot Project
Another challenge is the absent of a pilot project in the transition from the previous traditional method to the scrum method. Conducting a pilot project was a recommended step in the adoption of agile development for the first time. As a part of the plan to adopt agile method, the pilot project is essential to evaluate how ”S” environment will be able to move from the previous heavy-weight method to a new light method. Many organizations went through the same experience of running a pilot project especially those companies that have large projects such as Amazon, Yahoo, Microsoft and Intel. After investing the needed time and resources they have reached to a successful adoption of Agile.
Challenge 4: Scrum Implementation
Although the employees in ”S” were very experienced but yet none of them had any previous experience with agile development methods or Scrum implementation in particular. This is in addition of the absence of the agile master. For the team members, scrum implementation was not easy as it appeared to be during the training session. The team members find themselves, suddenly, in a completely new set up. The experience of traditional methods is completely different than committing to daily meeting, working with time boxes, finishing tasks in small period iteration and documenting the stories (or backlogs) in a different way.
Challenge 5: Current Work Pressure
Although ”S” software development team serves a very large organization of over 30 departments and developed numerous projects through the years, the development projects require continues maintenance and support. In addition, the team was working on a new project with firm deadlines. The work environment was very demanding and team worked under pressure to produce products according to the planned schedule. Scrum adoption process started while every member of the team was engaged in his/her everyday tasks. With such work pressure the daily Scrum meetings were considered waste of time and added extra pressure to the employees. They used to meet weekly and later twice a month and then only when required and usually after working hours. As teams started to skip daily meetings it also affected the learning process of scrum between the team members. That eventually leads to the failure of learning and implementing agile method correctly.
Challenge 6: Upper Management Concerns
The upper management of ”S” had many concerns about the effectiveness and success of the transition to a new method. They were not easily convinced to invest in a new method.
Challenge 7: Governmental bureaucratic System
The traditional method currently in “S” was customized to comply with the governmental system of other department. The new Agile method being introduced, Scrum, is developed in such highly bureaucratic environment. The Agile team has to secure approvals and signatures before moving from one step to another. This was perceived by the team members as unnecessary and more time was taken into account to develop a new project. The scrum method requires much less correspondence, less time in communication between the customer and the team and requires significantly less paper work and approvals as the customer is supposed to be involved in every step.
Challenge 8: Documentation requirements
After years and years of extensive documentation of every step in the traditional method, moving to a new method with minimum documentation requirements was one of the greatest challenges. Every project used to end up with dozens of document such as project charter, project plan, testing plan, SRS, STS, technical documents, user manual, etc. Each of these document contained large number of pages written by every member of the team and consumed hours of the valuable development time. The documentation requirements were driven basically from the previous challenge (the governmental system), upper management, ISO certificate requirements and the traditional development method that is currently used. Although agile development promises sufficient documentation of the projects, it didn’t seem very convincing to the upper management when they end up receiving few documents in comparison with the previous model of documentation. Many attempts were made to try to balance between the upper management requirement regarding documentation and between adopting Scrum method. Agile teams started to increase the number of documents required for documentation and started to customize Scrum as much as possible to conform to all the upper management requirements of documentation norms. This did not work very well and it created extra burden on the agile teams.
4. DISCUSSION OF CASE STUDY
Many of the challenges faced in case study were also discussed by other researchers in the area of software engineering and information technology management. However, many challenges that we encountered in our case study were not introduced by any researchers and are unique to our environment. These challenges are challenges 5, 7 and 8.
Challenge 5 is related to overloading developers and could be simply resolved by planning the agile method during a time that has a minimum work pressure and by not committing to any new project for at least 6 months. This will allow the team to invest more time for the agile adoption process. This investment will pay back later as the efficiency of the team increase with the agile method.
On the other hand Challenges 7 and 8 are more related to the culture of the organization and the change may require change in the mindset of the managers. One main advantage of agile methods is the ability to be customized based on the culture and the environment of the organization. It is unrealistic to resolve Challenge 7 and 8 overnight, but such issues could be resolved gradually by developers and upper mangers through dialog, meetings and revising the policies to what is best for the organization.
This article presented experience of performing a case study for adopting Scrum agile practices in a government entity in the United Arab Emirates (U.A.E). The organization has a long history of following the traditional Waterfall software development. This article identified the challenges faced by the agile teams during the adoption process and compared the findings with results obtained by other researchers in the area of software engineering. One case study may not be enough to build evidence and draw conclusions on the Scrum agile method or on the software development environment in the United Arab Emirate (U.A.E), however many lessons may be learned from such experience
 Greg Smith and Ahmed Sidky, Becoming Agile: ...in an imperfect world, Manning Publications, 2009.
 Mark Kennaley, SDLC 3.0: Beyond a Tacit Understanding of Agile, Fourth Medium Press,2010.
 GatherSpaceTeam, “Agile software development,” Retrieved January 15, 2011, January 2011.
 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas, “Manifesto for agile software development,” 2001.
 John Hunt, Agile Software Construction, Springer, September 2005.
 G. Benefield, “Rolling out agile in a large enterprise,” in 41st Hawaii International Conference on System Science. 2008, IEEE Computer Society.
 Tore Dyb°a and Torgeir Dingsøyr, “Empirical studies of agile software development: A systematic review,” Information and Software Technology, vol. 50, pp. 833–859, August 2008.
 Kieran Conboy, Sharon Coyle, Xiaofeng Wang, and Minna Pikkarainen, “People over process: Key challenges in agile development,” IEEE Software, vol. 28, pp. 48–57, 2011.
 Jayakanth Srinivasan and Kristina Lundqvist, “Using agile methods in software product development: A case study,” Information Technology: New Generations, Third International Conference on, vol. 0, pp. 1415–1420, 2009.
 Mikael Lindvall, Dirk Muthig, Aldo Dagnino, Christina Wallin, Michael Stupperich, David Kiefer, John May, and Tuomo K?hk?nen, “Agile software development in large organizations,” Computer, vol. 37, pp. 26–34, 2004.
 Mike Cohn and Doris Ford, “Introducing an agile process to an organization,” Computer, vol. 36, pp. 74–78, 2003.