Software Engineering
Introduction to Software Processes

Course Map


Goals

  1. What is software?
  2. What is the difference between software engineering and computer science?
  3. What is a system?
  4. What is a software process?
  5. What is a software process model?
  6. What are the attributes of good software?
  7. What are the costs of software engineering?
  8. Professional and Ethical Responsibility
  9. Your Enemy #1: Complexity
  10. Importance of the Customer - Customer Service Maxims
  11. Software Project Fundamentals
  12. Why We Need Software Project Survival Skills?
  13. Generic Software Process Models
  14. Process Iteration
  15. Code-and-Fix
  16. Rational Unified Process (RUP)
  17. Extreme Programming (XP)
  18. Scrum and its Rules
  19. Forming-storming-norming-performing
  20. Situational Leadership Model

What is software?


What is the difference between software engineering and computer science?

 


What is a system?

System Categories


Systems Engineering


The Systems Engineering Process

 

 


Interdisciplinary Involvement

Air traffic control system


What is the difference between software engineering and system engineering?


What is a software process?


What is CASE (Computer-Aided Software Engineering)


What are the attributes of good software?


What are the key challenges facing software engineering?


Professional and Ethical Responsibility


Issues of Professional Responsibility


ACM/IEEE Code of Ethics


Ethical Dilemmas


Key Points


Importance of the Customer - Customer Service Maxims


Process first, tools later!


Your Enemy #1: Complexity

                                Innovate or Die


Waves of Network Computing

From ... to transfers, transactions, content, telemetry to control


Influences on Architecture: Technical, Business, and Social

Software Architecture in Practice, Second Edition


The Warship Vasa

Does Not Compute

Vasa Museum, Stockholm, Sweden


Software Project Fundamentals


Engineering Fundamentals


Project Management Fundamentals


QA Philosophy

All effective QA approaches are based on the reality that mistakes will be made throughout the project. Project success depends on positioning the project team to detect and correct these mistakes quickly and easily.

QA Fundamentals


Murphy's Law

Does Not Compute


Statistics

People always underestimate the time required to develop software (Standish Group CHAOS report,1995)

Causes of software failure (KPMG report – top 3 from list)


Causes of Software Failure: Standish Group CHAOS Report

  1. Incomplete Requirements (13%)
  2. Lack of User Involvement (12%)
  3. Lack of Resources (11%)
  4. Unrealistic Expectations (10%)
  5. Lack of Executive Support (9%)
  6. Changing Requirements (9%)
  7. Lack of Planning (8%)
  8. Didn’t Need Software Any Longer (8%)
  9. Lack of Technical Management (6%)
  10. Technology Illiteracy (4%)

Cone of Uncertainty (The metaphor of filling a jar )

Steve McConnel - SW Project Survival Guide


Stop and Listen


But I don’t have time for SQA…

“If you can’t find the time to do it right, where are you going to find the time to do it over?”
There is no magic bullet, even with good software development practices, things can go bad.

 


We Want Agile Software not Fragile Software


“SQA slows down code development.”

 


"SQA takes too much time…"


“Some ‘process idiot’ is going to tell me how to do my job.”


"We do not have time to make mistakes."

Winner of the "Not My Job" award 2007.


Why We Need Software Project Survival Skills?


Future Plans to Prevent Runaways


Optimist's View of Project Without "Process"


Examples of "Process"


Optimist's View of Effect of Adding "Process"


Hard-Won View of Project Without Process


Project Without Process: "Trashing"


Hard Won View of Effect of Adding "Process"


The Software Process


What is a software process model?


What are the costs of software engineering?


Product development costs

COTS


Generic Software Process Models


Waterfall Model

The salmon lifecycle model. It’s not impossible to back up with the waterfall model, just hard.


Waterfall Model Problems

The main drawback of the waterfall model is the difficulty of accommodating change after the process is underway. One phase has to be complete before moving onto the next phase.


Evolutionary Development

Evolutionary prototyping model. With evolutionary prototyping, you start by designing and implementing the most prominent parts of the program in a prototype and then adding to and refining the prototype until you’re done. The prototype becomes the software that you eventually release.


Component-based Software Engineering

 

Reuse-oriented development


Process Iteration


Incremental Delivery

Incremental development advantages


Spiral Development

Spiral model sectors


Code-and-Fix

The code-and-fix model. Code and fix is an informal model that’s in common use because it’s simple, not because it works well.


Rational Unified Process (RUP)



Workflows

An example of a design workflow (see requirements workflows in the Use Cases module)


Agile Unified Process

Agile methods are a family of development processes, not a single approach to software development.

Some of the principles behind the Agile Manifesto are:

In the RUP there are three disciplines that encompass modeling activities for a single project – Business Modeling, Requirements, and Analysis & Design.  The AUP on the other hand, being a subset of the RUP, combines the three modeling disciplines into a single Model discipline.


The lifecycle for the Enterprise Unified Process (EUP)

All efforts, including modeling, is organized into disciplines in the UP and is performed in an iterative and incremental manner. The AUP is a subset of the RUP and the EUP a superset of the it.  To some extent the UP is serial in the large and iterative in the small.  The six phases of the EUP clearly occur in a serial manner over time, at the beginning of an UP project your focus is on project initiation activities during the Inception phase, once your initial scope is understood your major focus becomes requirements analysis and architecture evolution during the Elaboration phase, then your focus shifts to building your system during the Construction phase, then you deliver your software during the Transition phase, you operate and support your software in the Production phase, and finally you remove it from production during the Retirement phase.  However, on a day-to-day basis you are working in an iterative manner, perhaps doing some modeling, some implementation, some testing, and some management activities.

In the RUP there are three disciplines that encompass modeling activities for a single project – Business Modeling, Requirements, and Analysis & Design – and the EUP adds Enterprise Business Modeling and Enterprise Architecture.  All six disciplines are described in the table below.  For a description of potential artifacts to create during these disciplines visit Agile Models Distilled.


The Modeling Disciplines of the Unified Process

Discipline Purpose
Business Modeling The purpose of this discipline is to model the business context, the scope, of your system.  Common modeling activities include the development of:
  • A context model (often a data flow diagram)  showing how your system fits into its overall environment
  • A high-level business requirements model (often an essential use case model)
  • A glossary defining critical business terms
  • A domain model (often a class diagram or data diagram) depicting major business classes or entities
  • A business process model (often a data flow diagram or activity diagram) depicting a high-level overview of the business process to be supported by your system.  This diagram is one level of detail greater than your context diagram
Requirements The purpose of this discipline is to engineer the requirements for your project, including the identification, modeling, and documentation of those requirements.  The main deliverable of this discipline is the Software Requirements Specification (SRS), also referred to as the Requirements Model, which encompasses the captured requirements. 
Analysis & Design The purpose of this discipline is to evolve a robust architecture for your system based on your requirements, to transform the requirements into a design, and to ensure that implementation environment issues are reflected in your design.  
Model In the AUP this is the combination of the Business Modeling, Requirements, and Analysis & Design discipline.
Enterprise Business Modeling The Enterprise Business Modeling discipline extends business modeling to enable your organization to explicitly realize the benefits of doing so outside the context of a project. The goal of the Enterprise Business Modeling discipline, also sometimes referred to as enterprise requirements modeling, is to enable you to understand and effectively communicate the business that your organization is engaged in. It explores both the business process that your organization engages in as well as the overarching business requirements for the software applications that support it.
Enterprise Architecture

The Enterprise Architecture discipline defines the enterprise architecture of an organization.  It consists of models that define it, reference architectures, prototypes and working models that demonstrate how it works, and frameworks that make it easier to use. An enterprise architecture model is a representation of the structures and processes of an organization; a good one depicts the organization both as it is today and as it is envisioned in the future, and maps the various views representing the architecture to one another.  These views include both business-oriented perspectives as well as technical perspectives.  In many ways enterprise architecture models are a communication bridge between senior business stakeholders and senior IT professionals.


Extreme Programming (XP)


The Rules and Practices of Extreme Programming

 

Planning

 User stories are written.
 Release planning creates the schedule.
 Make frequent small releases.
 The Project Velocity is measured.
 The project is divided into iterations.
 Iteration planning starts each iteration.
 Move people around.
 A stand-up meeting starts each day.
 Fix XP when it breaks.

Designing

 Simplicity.
 Choose a system metaphor.
 Use CRC cards for design sessions.
 Create spike solutions to reduce risk.
 No functionality is added early.
 Refactor whenever and wherever possible.

Coding

 The customer is always available.
 Code must be written to agreed standards.
 Code the unit test first.
 All code is pair programmed.
 Only one pair integrates code at a time.
 Integrate often.
 Use collective code ownership.
 Leave optimization till last.
 No overtime.

Testing

 All code must have unit tests.
 All code must pass all unit tests before it
 can be released.
 When a bug is found tests are created.
 Acceptance tests are run often and the score
 is published.


RUP Phases vs. XP
 
RUP Phase XP Notion
Inception:
  • User Stories
  • Metaphor
  • initial spike.
  • Elaboration
  • More User Stories.
  • Establishing the metaphor, or other architecture.
  • Some spikes, some production code.
  • Calibration of estimation (Load Factor / Velocity)
  • Construction
  • More User Stories
  • Adding iterations onto the existing code, fleshing out the architecture.
  • Refinement of process
  • Refinement of estimation.
  • First delivery to production, at a very early development phase. (If possible)
  • Transition
  • Continued iterations, adding more and more features to production system.
  • If necessary, old system runs in parallel with new system while new system takes over more and more duties.
  • Performance Tuning.
  •  


    How Good is the Fit Between AM and UP?

    Now that we understand the basics of how modeling in the UP works, we can examine how well AM fits in with it.  Luckily many of AM’s principles and practices are arguably a part of the UP already, although perhaps not as explicitly as I would like.  The table below presents an examination of how well each individual AM practice is currently implemented in the UP, if at all, and discusses how to adopt the practice within the scope of the UP.  My experience is that it is relatively straightforward for UP teams to adopt AM practices if they choose to do so.  This is because the UP is very flexible, one of its underlying principles is that you should tailor it to meet your unique needs, making it easy to merge AM practices into the UP.

    The Fit Between UP and AM

    Practice

    Fit

    Active Stakeholder Participation

    AM has a wide definition for project stakeholders, including users, management, operations staff, and support staff to name a few that are compatible with the UP.  The UP clearly includes project stakeholders, such as users and customers, throughout most of it disciplines.  To be successful UP project teams should allow project stakeholders to take on modeling roles such as Business Process Designer and Requirements Specifier as appropriate, there is nothing in the RUP preventing this by the way.  The more active project stakeholders are the less of a need there will be for reviews, management presentations, and other overhead activities that reduce your team’s development velocity.

    Apply Modeling Standards

    The application of modeling standards, in particular the diagrams of the Unified Modeling Language (UML), is a significant part of the UP.  Furthermore the RUP product includes guidelines for the creation of many modeling artifacts, guidelines that your teams should consider adopting and following as appropriate, and explicitly suggests that you tailor the guidelines that they provide for your exact needs.  To remain agile, however, UP teams should recognize that you often need to bend the guidelines and standards – in other words, don’t let them become a straight jacket.  There are detailed UML modeling guidelines posted at this site.

    Apply Patterns Gently

    UP teams are free to apply modeling patterns, the RUP product describes many common modeling patterns, as part of their efforts for any of the modeling disciplines.  This practice enhances the UP with its advice to ease into the application of a pattern, the UP does not make this concept as explicit as it could. 

    Apply the Right Artifact(s)

    One of the strengths of the UP is that provides some advice for when to create each type of model, and recent incarnations of the RUP product includes significant advice for non-UML artifacts such as data models and user interface storyboards (UI flow diagrams).

    Collective Ownership

    AM’s concept of collective ownership can be used to enhance the efforts on UP projects, assuming that the team culture supports the concept of open and honest communication.  The UP supports collective ownership with its strong focus on configuration management issues, it has a discipline dedicated to this task, although its change management processes may potentially get in your way if developers and project stakeholders are unable to distinguish when to formalize change control and when not to.  To be fair, this is a problem regardless of when you apply AM on an UP project, or on any type of project for that matter.  UP teams should turn the configuration management dial up a few notches and allow anyone on the project to access and work on any artifact that they wish, including models and documents.

    Create Several Models in Parallel

    The UP clearly includes this concept, one only has to look at the activity diagrams depicting each discipline to see that several artifacts are potentially being worked on in parallel.  However, this concept could be communicated better because the near-serial flow in its activity diagrams presented for each major modeling activity doesn’t communicate this concept well.  There is a larger issue as well when you consider the lifecycle as a whole.  Because the UP has organized its modeling efforts into separate disciplines, for very good reasons, it isn’t as apparent that not only could you work on several business modeling artifacts in parallel but you could also work on requirements-oriented artifacts, analysis-oriented artifacts, architecture artifacts, and design artifacts too.  UP teams can turn the dial up a few notches by reading between the lines of the discipline activity diagrams and the UP lifecycle diagram and choosing to perform activities from several disciplines simultaneously when it makes sense to do so.

    Create Simple Content

    This practice is a choice made by the modeler(s), albeit one that must be implicitly supported by the rest of the development team.  UP teams will need to adopt modeling guidelines that allow models that are just good enough and the customers of those models (including programmers, project stakeholders, and reviewers) must also be willing to accept simple models.  This is a cultural issue, one that is often difficult for many organizations to adopt.

    Depict Models Simply

    See Create Simple Content.

    Discard Temporary Models

    Modelers on UP teams are free to discard anything that they wish.  As with the simplicity practices your organization’s culture must accept the concept of traveling light, of developing and maintaining just enough models and documents and no more.

    Display Models Publicly

    UP teams are free to follow this practice.  UP teams can turn the communication dial up a notch by following the principle of Open and Honest Communication by making all artifacts available to everyone as well as to publicly display the critical models used by the project team.

    Formalize Contract Models

    The UP includes the concept of integrating with external systems, these systems are typically identified on use case models and the RUP suggests introducing “boundary classes” to implement the interface to these systems. At the time of this writing the RUP appears weak with respect to activities such as legacy system analysis and enterprise application integration (EAI).  The explicit adoption of this practice clearly strengthens the UP’s integration activities and fits in well with it’s concepts of use case realizations – the interaction between systems could be specified with one or more use cases and then the corresponding use case realization would be the formalized contract model.

    Iterate to Another Artifact

    This practice can be easily adopted by an UP team.  As mentioned previously, the unfortunate depiction of UP modeling activities as quasi-serial processes and the division of modeling activities into separate disciplines can hinder the iterative mindset required of agile modelers.  

    Model in Small Increments

    This practice is clearly an aspect of the UP – the UP’s support for iterations implies that you will be incrementally developing your model throughout your project.  UP teams can easily turn the iterative and incremental dial up a few notches by preferring smaller, simpler models that quickly lead to implementation and testing.

    Model With Others

    The UP implicitly includes this practice.  Every modeling discipline clearly includes several roles, each role being filled by one or more people. UP teams can turn the communication dial up a few notches by adopting tools that support team modeling, such as whiteboards and collaborative modeling tools (see the Communication article) over single-user modeling tools.

    Prove it With Code

    The UP explicitly includes this practice.  At the end of every iteration, except perhaps for the ones during the Inception phase, the UP specifically states that you should have a working prototype.  Furthermore, the UP insists that you have a working end-to-end prototype at the end of the Elaboration phase that proves your architecture.

    Reuse Existing Resources

    Reuse is an implicit part of the UP, and reuse management is an explicit part of the Enterprise Unified Process (EUP).  UP teams can turn the reuse dial up a few notches by actively preferring to reuse existing resources instead of building them from scratch, including but not limited to existing models, existing components, open source software (OSS), and existing tools.

    Single Source Information There is no reason why you cannot store information in a single place when following the UP.  Unfortunately, many organizations choose to instantiate the RUP in a documentation-driven manner (which IBM Rational clearly advices you not to do), and as a result they travel very heavy and clearly take a multi-source approach.

    Update Only When it Hurts

    In theory this can be an easy concept for UP teams to adopt as it dramatically reduces the effort expended to keep your artifacts up to date.  However, in practice many organizations prove to have a problem with this concept, particularly if they have a strong “traceability” culture.  Traceability is the ability to relate aspects of project artifacts to one another, the support for which is a strong feature of the UP as it is an important aspect of its Configuration and Change Management discipline.  Furthermore, the RUP product includes tool mentors for working with Rational RequisitePro, a requirements traceability tool, making it appear easy to maintain a traceability matrix between artifacts.  My experience is that organizations with traceability cultures will often choose to update artifacts regularly, even if it isn’t yet painful to have the artifacts out of date, and update the traceability matrix relating everything to one another.  To turn their productivity dial up several notches UP teams should choose to travel light, to loosen up a bit and allow project artifacts to get out of sync with one another, and to maintain a traceability matrix between artifacts only when there is clear benefit to do so AND their project stakeholders understand the issues involved as well as authorize the effort.  A traceability matrix is effectively a document and is therefore a business decision to be made by project stakeholders.

    Use the Simplest Tools

    The RUP product includes tool mentors that make it easier for teams to work with tools sold by Rational Corporation.  However, the reality is that UP teams are welcome to use any development tool that they want and Rational tools compete on their merits just like the products of any other company.  UP teams can turn their productivity dial up several notches by expanding their horizons to include simple tools such as whiteboards, index cards, and Post-It notes in addition to CASE tools.

    Something that is important to understand is that for AM to be successful the culture of your organization must be open to the concepts, values, and principles of agile software development.  The problem is that the UP is often adopted by organizations that either implicitly or explicitly do not accept the values of agile software development.  Their focus is often on following processes and using tools, the RUP product clearly defines many processes and describes how to apply Rational’s tools effectively on software projects, and therefore the RUP is clearly attractive to them.  Unfortunately this goes against the agile value of preferring individuals and interactions over processes and tools.  When the culture of an organization is documentation centric they may find the UP appealing because you can instantiate it in such a way as to result in the creation of significant amounts of documentation (you can also instantiate it to result in very little documentation, remember, the UP is flexible).   If an organization is documentation centric then this aspect of its culture goes against agile software development’s value of preferring working software over comprehensive documentation.  This organization may still successfully adopt, tailor, and instantiate the UP but be unable to follow many of AM’s principles and practices effectively because it does not have an agile culture (see the article When Does(n't) AM Make Sense).  The point is that how well AM and UP will fit together in your organization depends largely on your organization’s culture and not so much on the UP itself.   You can easily use the techniques of AM to improve your UP modeling efforts, but to be effective you will find that you need to overcome cultural challenges within your organization. 


    Classic Mistakes

    People-Related Mistakes

    Process-Related Mistakes

    Product-Related Mistakes

    Technology-Related Mistakes

    1. Undermined motivation

    2. Weak personnel

    3. Uncontrolled problem employees

    4. Heroics

    5. Adding people to a late project

    6. Noisy, crowded offices

    7. Friction between developers and customers

    8. Unrealistic expectations

    9. Lack of effective project sponsorship

    10. Lack of stakeholder buy-in

    11. Lack of user input

    12. Politics placed over substance

    13. Wishful thinking

    14. Overly optimistic schedules

    16. Insufficient risk management

    17. Contractor failure Insufficient planning

    18. Abandonment of planning under pressure

    19. Wasted time during the fuzzy front end

    20. Shortchanged upstream activities

    21. Inadequate design

    22. Shortchanged quality assurance

    23. Insufficient management controls

    24. Premature or too frequent convergence

    25. Omitting necessary tasks from estimates

    26. Planning to catch up later

    27. Code-like-hell programming

    28. Requirements gold-plating

    29. Feature creep

    30. Developer gold-plating

    31. Push me, pull me negotiation

    32. Research-oriented development

    33. Silver-bullet syndrome

    34. Overestimated savings from new tools or methods

    35. Switching tools in the middle of a project

    36. Lack of automated source-code control


    Scrum and its Rules

    Scrum is an iterative, incremental process for developing any product or managing any work. It produces a potentially shippable set of functionality at the end of every iteration. It's attributes are:

    Frequent, First-hand Observations

     

    Frequent, First-hand Observations

     

    Active, Intelligent Management

     


    XP @ Scrum

    Scrum has been employed successfully as a management wrapper for Extreme Programming engineering practices. Scrum provides the agile management mechanisms; Extreme Programming provides the integrated engineering practices.

     

    Benefits of xP@Scrum include:

    1. The agile management and control mechanisms of Scrum are applicable for any type of project, including business initiatives that consist of multiple, simultaneous software development, business development, re-engineering, marketing, support, and implementation projects. xp@Scrum projects fit within the overall management framework of these initiatives.
    2. xP@Scrum projects realize the full benefits of self-organization; teams are iteration (or Sprint) goal directed, rather than story directed.
    3. When Extreme Programming projects are wrapped by Scrum, they becomes scalable and can be run simultaneously by non-colocated teams.
    4. Scrum implements in a day; Extreme Programming can be gradually implemented within the Scrum framework.

    CS3365 Project Development and Milestones

    1. You work as a creative SW development team
    2. Lifecycle Objectives - at least: extended 4-blocker
    3. Lifecycle Architecture - at least: Use case diagrams, component diagrams, interactions diagrams, prototype (at least storyboard)
    4. Initial Operational Capability - at least: operational SW system and javadoc documentation
    5. Product Release - at least: final presentation and demo

    Rate of Change


    Forming-storming-norming-performing

    Bruce Tuckman's 1965 team-development model

    Team Ground Rules

    The progression is:

    1. forming
    2. storming
    3. norming
    4. performing

    Features of each phase:

    Forming - Stage 1


    Storming - Stage 2 Norming - Stage 3

    Performing - Stage 4


    Hersey's and Blanchard's Situational Leadership Model

    Team Ground Rules

    The classic Situational Leadership model of management and leadership style also illustrates the ideal development of a team from

    The aim of the leader or manager is therefore to develop the team through the four stages, and then to move on to another role.