1.    Introduction to s/w Testing and Quality Assurance

1.1.        Project Organization

           The Test discipline is related to other disciplines, as follows (This topic will be elaborated in chapter 3 Testing Process and its activities)

The Requirements discipline captures requirements for the software product, which is one of the primary inputs for identifying what tests to perform.

The Analysis & Design discipline determines the appropriate design for the software product, which is another important input for identifying what tests to perform.

The Development discipline produces builds of the software product that are validated by the Test discipline. Within an iteration,multiple builds will be tested — typically one per test cycle.

The Environment discipline develops and maintains supporting artifacts that are used during Test, such as the Test Guidelines and Test Environment.

The Project Management: Test team receives the heads-up regarding up-coming projects, change requests, time lines for different phases of testing etc from Project management. Test team provides time and resource estimates to it. If there are any issues or support needs that cannot be solved/met with in the test team, project management team facilitates help from other teams.

      1.2.        What is testing/QC?

Testing is a process of evaluating a software system against pre-defined standards with the intention of finding defects/errors. Here Pre-defined standards are the requirements, which specify the results expected of the system.

A defect/error is a difference between expected result and actual result. This difference can be a difference in behavior of the application (also called functionality) or characteristics of the application (also called look and feel).

Requirements are the basis for expected results. Actual results are realized at the time of testing software application/system.

1.3.        What is QA?

Software QA deals with creation,implementation and continuous improvement of processes employed in all phases of System Development Life Cycle – including but not limited to:

·        Planning

·        Analyses

·        Designing

·        Coding/Building

·        Testing

·        Deployment/Implementation and

·        Maintenance.

Here process is a set of methods, procedures and practices that we use to develop and maintain software systems and the associated products (e.g., project plans,design documents, code, test cases, and user manuals).

1.4.        Testing/QC Vs Quality Assurance

As we can see from above description, QA is much broader than Testing. QA encompasses all phases of SDLC, where as testing is a particular phase in SDLC.

So,testing is only a part of QA.

QA focuses on completeness and thoroughness in the methods and practices followed for creating a software product, where as Testing seeks to find out the incompleteness and deficiencies of the same product.

2.    Software Development Life Cycle and role of Testing

2.1.        SDLC

SDLC stands for System Development Life Cycle. A system is an inter-related set of components, with identifiable boundaries, working together for some purpose.

Every software system passes through certain distinct identifiable phases during its life. These phases together make up its life cycle called System Development Life Cycle (SDLC).

Following table out lines the phases in the SDLC and explains the activities that take place in each phase (please note that these are most commonly identified phases. You will find somewhat different wording in other’s writings, don’t worry about it)

                                                            Table 1

2.2.           SDLC Models

This document provides only high-level descriptions of some common SDLCs and is nota comprehensive list of all known SDLCs. The SDLC models included in this document are Waterfall, Staged,Prototyping, and Extreme Programming (XP).

Waterfall Model

TheWaterfall development methodology (also known as the “Traditional Method”)structures a project into distinct phases with defined deliverables in eachphase. This methodology assumes each phase is completed before the next phasein the sequence begins.

Staged Model

TheStaged model is also known as the Incremental model. The Incremental modeldivides the end product into separate builds, where sections of the product arecreated, tested and deployed parallel. This model is adopted when all of therequirements are known (and considered relatively stable) at the start ofdevelopment (with minimal anticipated changes), and when it is desirable forsome of the builds/work packages to be completed as soon as possible. 

2.3.        Testing/QC life cycle in SDLC

Testing or involvement of test team should begin as early in the development cycle as possible. In the initiation phase, project management team may approach test team lead and/or other expert in test team for time and resource estimates for any up coming changes or projects. They incorporate some or all the estimates in the project plan and schedule. Other than that, usually QC/testing process begins as soon as the first draft of the functional requirement specs is ready for review. In other words testing should begin after system’s functional requirements have been created but before they are base lined for coding. Look at the Table 1 given under section 2.1 SDLC. Testing/QC typically begins in 2^nd phase of SDLC (Analyses phase), to be precise, along with 4^th activity (i.e.Review and baseline system’s functional requirement specifications). The following five diagrams depict the Testing/QC life cycle in the context of SDLC. You can also see the testing activities that go parallel to corresponding SDLC activities. Please be advised that this list of QC activities is by no means a comprehensive. You might find some additions/ subtractions to this list in others writing and some jumbling of sequence of these activities.

3.    Testing Process and its activities

When a new project begins or changes come up for an existing project, the project management team seeks input from test team and/or team lead/manager for resourcee stimates and timelines.

Like wise the project management team gathers input from development team and business analysts. Then, reconciles all the inputs andcreate a project plan with defined phases and deliverables in each phase along with time lines i.e. start and end dates.

In this way, test team will have some involvement at the beginning of every project. Other than this early involvement of test team for providing inputs to the project plan, the actual testing efforts (as explained in section 2.3) typically begins when system’s requirements are ready for review.

The following tables lists Testing/QC activities during life cycle along with person(s) responsible for them

3.1.        Requirements Analyses and walkthroughs

Testing is comparing expected behavior with actual behavior and reporting

bugs if they are different. Who will tell you about expected behavior of the application?.REQUIREMENT / FUNCTIONAL SPECIFICATIONS tell you that. Without requirement specs you cannot meaningfully test any application.

Business/Requirement analysts interact with the end user/client and gather the Business Requirements. Business Requirements are converted into Functional

Specifications also called system requirements. Business requirements are what the client wants. Remember business requirements say what client wants, but not how client want them.

Functional specs or System Requirements say how the system should look and work in order to meet the business requirements of the client. In other words System requirements are what should be seen in the application. The clients could only give their business requirements. Business analysts/Requirement specialists have to convert them into functional specs/System Requirements.

Once functional specs are ready the testing process begins. When Functional Specs are ready a copy of it will be distributed to all stakeholders of the project including testers. Testers have to thoroughly read the requirements and prepare a list of questions they have on them. Then testers participate in what is called the “Requirement walk through sessions” in which application designers,developers and requirement specialists also participate. It is basically a meeting of all and all the requirements (here after requirements mean Functional/Requirement specs) are critically discussed to find missing,incomplete, inconsistent and contradictory requirements.

Example for Missing requirements:

We see a ‘Reset’ button below a form that can clear all the values entered in the form. But there is no mention of that button in the requirements, except in the sample screen shot.

Example of incomplete requirements

A requirement says, “User can change his personal details in the application from abc screen.” This requirement dose not say what if the user logs out of the application without saving the changes he made to his personal details etc.

Exampleof inconsistence requirementsis that - at one place requirements say one thing about a particular action andsay some thing different at other place regarding the same or similar kind ofaction.

Lookat the following example requirements:

Onerequirement reads:

‘When a user tries toaccess files belonging to others he/she should get an error message “ You donot have permission for this action.”’

Anotherrequirement reads:

‘When a user tries toaccess files or folders belonging to others he/she should get the message “ Youneed admin privileges for this kind of action”’

Inthe above two requirements the user action is more or less same but themessages are different.

Anotherexample is that user gets warning message when he attempts to delete individualmails from bulk mail folder but don’t get any warning message when “EmptyFolder” link is clicked, which in fact deletes all the mails from bulk mailfolder.

Anotherexample is at some place requirements document says “when user tries toquit the application without saving his/her changes he/she should get themessage window ‘Are you sure you want to quit without saving changes’ with twobuttons labeled OK and Cancel. ”. At some other place requirementsdocument says,  “when user tries toquit the application without saving his/her changes he/she should get themessage window ‘Your changes will be lost if you logout withoutsaving them. Do you want to continue’ with two buttons labeled Yes and No.”

Inabove case, requirements specify two different messages for the

samekind of action by the user. These are called inconsistent requirements.

Exampleof contradictory requirements:

Ifthere are contradictory requirements, implementing one requirementautomatically leads to elimination of some other requirement. At one placerequirements say one thing about a particular function and an opposite thingabout the same function at a different place or requirements say one thingabout a function at one place and one thing about some other function at someother place, whose effect is to go against what was said about the firstfunction.

Forexample Requirements at one place say that role X should not have access tofunctionalities from A1 to A10 and at some other place say functionality A5should be accessible to role X only. These are mutually contradictory.

Oncethe requirement walkthrough sessions are over and all necessary corrections tofunctional/requirements specs are made, the Requirement Specifications aredeclared as finalized/Base lined. A copy of the requirements is given to thetesting and developing team. Testers go ahead with developing test plan, testcases, test scripts etc (elaborated else where in this document). Developers goahead with coding.

3.2.        Test plan creation

Testplan, also called QC plan, is a comprehensive master plan for executing atesting project. It is a blue print on which every thing else is based. Itaddresses many important questions like:

What to test?

How to test?

            When to test?

            Where to test?

            Who should test it? And

What is required to test it?

Answersto these questions will make up the test plan. Test plan is an evolutionarydocument. It keeps changing and evolving as the testing progresses.       

What to test?– This activity identifies the components of the application under test, anddecision on what to test and what not to test.

How to test?  - This activityhas to do with test strategy. It involves making many such decisions as:

Whatare test/QC entry and exit criteria?

Whetherto test manually or use automation tools! Or use a combination of both.

Whattemplates processes and standards should be followed?

Whereto store the QC/Testing artifacts like, test plan, test cases etc.

Howto track defects and what should be the defect management policy?

Howmany levels of testing should be executed and what should be the executionsequence (execution plan)?

Howthe change control policy should be?

Whatif some things take longer than expected or how to meet tight dead lines orover come pressure situations (This calls for risk analyses and contingencyplan)

What methodologies should be used? – Black Box/WhiteBox/Gray Box or a combination of these.

When to test? – An answer to this questionproduces schedule for the testing project with different phases, detailedactivities in each phase along with clearly defined deliverables and time lines(start and end dates).

Where to test? This question is related totest environment (operating systems, Network, databases, tools, applicationsetc.)

How close the test environment should be to the real worldenvironment?

Who is responsible to setup and manage the testenvironment?

Who should test it? – This question relatesto resource planning. Usually QC Team lead or manager is responsible for this.This activity includes but not limited to:

1.   Identifying the number of resources required andrecruiting new resources if needed.

2.   Identifying the training needs and arranging fortraining.

3.   Define roles and responsibilities of test teammembers.

4.   Dividing work among QC team members.

5.   Responsibilities of QC team towards other teams.

6.   QC Team’s communication process with otherteams.

7.   What QC team could expect from other teams as amatter right.

What is required to test it? – QC Projectmanager with the help of team members should identify the needs of the testingteam in terms of test data, environment, tools etc and arrange to get them tothe team in timely manner for successful execution of testing project.

There are two levels of test plan. Unit/Component testplan and Master test plan. It is the responsibility of every test team memberto create a test plan for the piece of work that was assigned to him or her.Such test plans are called unit/component test plans. From the component testplans the QC Project manager, with input from QC Team lead and other teams, createsa master test plan.  Unit test plans canbe very brief. Since all the information might not be available at the time ofcreation of test unit and master test plans, it is not possible to completethem upfront. Some of the sections might have to be left blank or filled withthe acronym TBD (means ‘to be defined’). And some of the sections might have tobe changed as the testing progresses. Here is a sample or template for theQC/Test plan:

Application under Test

XxxxxRelease xx.xx Test Plan.

Version xx

Last Modified Date:

TABLEOF CONTENTS

1.   The Application undertest…………………………………………………………..24

2.   Test PlanObjectives…………………………………………………………………24

3.   Components of the Release. ………………………………………………..……..24

4.   Releasepriorities…………………………………………………………………….24

5.   ReleaseSchedule……………………………………………………………………25

6.   Contact personnel for theRelease………………………………………….……..27

7.   Subject Matter Experts in TestTeam………………………………..…………….27

8.   Release Test PlanMatrix……………………………………………………………28

9.   Test Dataneeded…………………………………………………………….….xxxx

10. TestURLs………………………………………………………………….……..xxxx

…

1.    TheApplication under Test:

·        ApplicationName:

·        Release#:

·        Releaseduration:

·        Testingduration:

2.    TestPlan Objectives:

Toidentify the components of the Release.

Toidentify critical areas and establish priorities  for testing the release components.

Todefine items to be tested and items not to be tested.

Toidentify the contact personnel in each team for each component of the release.

Toplan the testing resources,

Toidentify the activities needed for successfully testing the Release.

3.    ReleaseComponents:

4.    ReleasePriorities:

After having meetings with Developmentand Requirements team it was understood that following components of thecurrent release will have maximum impact for the business client. So they areregarded as top priority areas for testing as well. However, please note thatrelease components not listed below by means should be construed as less orunimportant ones. Every thing is important.

5.    QCProject Schedule:

6.    ResourcePlan

7.    Releasecontact personnel (out side teams):

8.    SubjectMatter Experts in Test Team:

Thislist has been prepared based completely on who worked on what during previousreleases:

9.    ReleaseTest Plan Matrix:

Therelease test plan matrix begins with identification of deliverables expected oftest team during the course of Testing Cycle and proceeds to spell out thetasks that should be accomplished to have the identified deliverables alongwith the due dates for each of them and the personnel responsible. It alsoprovides a non-detailed strategy to perform each of the tasks.

3.3.        Traceability Matrices

One of thedefining characteristics of good QC or testing practices is to provide thetraceability between the requirements and the tests that verified them.Requirements are one end of the testing while defects represent the other end.It is very important to link them together in some form or the other. It isvery important in the light of the fact that defects are after all deviations fromrequirements. A defect can’t be a defect if there are no requirements relatedto it. We find defects while executing tests. Tests are created by groupingtest conditions. Test conditions are derived from requirements. Traceability isa mechanism to establish link between all theses entities. Traceability can beas simple as a table showing requirements, test conditions, test cases/testsand defects.

Whether youdo it using a tool or do it manually using word documents or excel workbooks itdoesn’t matter, there should be a clear mapping between tests created and therequirements the tests have been covering. Given below is one of the widelyused templates to map tests to requirements. It is called RequirementsTraceability Matrix or Requirements Validation Matrix.

It is highlyadvisable to use such matrix even if you are using a test management tool likeTest Director/Quality Center. Due to their tree structures and limited view ofcontent in the tabs, they tend to inhibit your ability to think clearly. Youcannot see requirements and tests side by side. You get to see only one at atime. So, first design you tests in a matrix like this and then update the testmanagement tool. This may seem extra burden, but benefits are worth the effort.

Risk Codes:

1= Critical; 2= High; 3=Medium; 4=Low; 5=Trivial

The firstmust at least describe the name of the business requirement document along withits location and section which it has been picked from with in that document.If available it is highly advisable to put the requirement numbers also.

The secondcolumn should contain similar information about System Requirements.

For exampleone system requirement reads in one section of the requirement document say –“XYZ System Requirements, section – Registration form”

“The application should be protectedwith User ID and password”

Correspondingsystem requirements may look like below

1. There should be a password field in the registration form and it should accept five to eight alphanumeric characters only. Otherwise the system should throw an error - ‘Invalid password format’ with ok button. Upon clicking ok in the error message, the password field should be cleared so that user can enter a different password.
2. There should be a field for the user to enter his choice of user name. System should validate if that user name is available or already taken by some one else. If the user name has already been taken, then the system should display an error message “This user name is not available. Please select a different user name” with an OK button. Upon clicking ok in the error message, the User Name  field should be cleared so that user can enter a different password

In the abovematrix, first column should have “XYZ Functional Requirements, section –Registration form”. Second column should have above two system requirements

Then each systemrequirement should be broken down further into test conditions using black boxtesting techniques (described else where in this document).

Onerequirement can be broken down into any number of test conditions based on whatthe requirement is.

While firstand second columns are filled during analysis phase of testing life cycle,third, fourth and last columns are usually filled during test design phase.Defect IDs columns can only be filled during test execution phase

3.4.        Test Case Design

Test case design, also called test design, is a process of deriving testconditions from system requirements and grouping related test conditions intological units called tests or test cases. A test case is a test which includesone or more test conditions. A test condition is smallest unit into which arequirement can be broken down.

The process of grouping test conditions into test cases should havefollowing characteristics:

1. Each test condition must be covered at least once in a test case. There should not be any missing of test conditions.
2. Avoid coverage of test conditions multiple times. This causes duplication of effort and redundancy.
3. Test conditions grouped into test case should have some commonality or relationship based on some criteria.

Above are the most important features of an efficient test design.

The criteria for grouping test conditions into test cases could be anyone or combination of the following characteristics of the test conditions.

1. Test conditions might be verifying same screen of the application.
2. They might be verifying same functionality of the application.
3. They might require same kind of inputs.
4. They might need same tools or technologies etc.

This characteristic list could grow on and on based on the applicationunder test.

Once a test case is identified for a set of test conditions, a meaningfulname should be given to it. The test case names should be documented in therequirements traceability matrix (specifically in the third column) against thetest requirements or test conditions they cover.

There could be one to one, one to many as well as many to onerelationship between test cases and test conditions. In other words, one testcondition may be covered by more than one test case or just one test case maycover more than one test condition.

3.5.        Test Script Creation

Test script is set of detailed, step by step instructions for executing atest case.  Every test case MUSTNECESSARILY have a test script associated with it. They always go together. Thesteps in the test script explain what to do in order to verify the testconditions included in the test case.

Some organizations that have very tight schedules and close releasedeadlines might some times skip the test script creation altogether. But, it isnot recommended for critical or some what important applications that haveserious implications should some thing go wrong.

Test cases and test scripts are some times used synonymously. Given belowis a suggested structure of test script of a test case. If you are using sometest management tool like Test Director, you don’t need this template.

Test Case Identifier:<<Test Case Name and Number>>

                         ApplicationIdentifier: ………………………………………               

Test Case logistics:

Execution status codes: 1. NotStarted, 2. In Progress, 3. completed.

Test Case Objectives:

1. To verify that …
2. To verify that …

Environment setup:

            Specify any special test environmentsetup needs here: Like purposefully disconnect an interface to verify errormessages, point the application to an empty database etc.

Test Data Setup:

                Specify test data requirementshere i.e. data needed to execute this test case

Script / Execution steps:

.

.

.

3.6.        Test Execution

Needless to say this activity is performed during test execution phase.Every member of the QC team should include an execution plan in their componenttest plans. Execution plan specifies the execution sequence and execution schedulefor the test cases created.

Following are some of the recommendations for test execution phase:

1. Test execution sequence and schedule should confirm to the application’s release schedule, risks and priorities.
2. Usually tests that verify the defining functionality of the application should be executed first. For example, in an email application, you should first test login, read mail and write mail functionalities, before attachment, address book etc.
3. Execution of some tests results in creation of test data for some other tests. Identify such tests and execute them first. This saves time otherwise required to create test data.

For example there are many online applications thatrequire users to register. In such applications executing tests that, say forexample, verify the functionality of adding new users ahead of the tests thatverify the functionality of modifying existing users will save some time.Because, to modify a user details, first of all there should be users. Thefirst category of tests adds the new users. The same user details could be usedlater while executing the tests that verify the functionality of modifying userdetails.

4. Tests that verify the maximum functionality of the application in minimum time should be executed next.
5. Tests that verify the same functionality with different test data several times could be taken up next.
6. Tests that verify cosmetic features of the application should be pushed to last.

3.7.        Defect Management

Defect management activity goes simultaneously with test execution. Asexplained before a defect is a difference between expected state or behavior ofthe application (as stated in functional and business requirements) and itsactual behavior learned while executing tests. Defect management deals with setof methods policies and procedures followed during the life cycle of a defect.This topic will be elaborated more in the chapter dedicated to it.

4.    Categoriesof Testing

5.    Levels ofTesting:

Testing is appliedto different types of targets, in different stages or levels of work effort.These levels are distinguished typically by those roles that are best skilledto design and conduct the tests, and where techniques are most appropriate fortesting at each level. It's important to ensure a balance of focus is retainedacross these different work efforts.

5.1.        Unit Testing

Unit testing focuses on verifying the smallesttestable elements of the software. Typically unit testing is applied tocomponents represented in the implementation model to verify that control flowsand data flows are covered, and that they function as expected. The Implementerperforms unit testing as the unit is developed. The details of unit testing aredescribed in the Implementation discipline.

Unit testing isalso called Module Testing. It is applied at the level ofeach module of the system under test. Module testing is usually done by thedevelopers themselves, before the module is handed over for integration withother modules. Usually the testing at this level is structural (white-box)testing. Module Testing is sometimes called Unit Testing representing thelowest level component available for testing. Note in some situations, unittesting is also used to refer to testing of logical or function units that haveintegrated a number of modules. We consider this to be integration testing.

5.2.        Integration Testing

Integration testing is performed to ensure that thecomponents in the implementation model operate properly when combined toexecute a use case. The target-of-test is a package or a set of packages in theimplementation model. Often the packages being combined come from differentdevelopment organizations. Integration testing exposes incompleteness ormistakes in the package's interface specifications.

In some cases, the assumption by developers is thatother groups such as independent testers will perform integration tests. Thissituation presents risks to the software project and ultimately the softwarequality because:

·        Integrationareas are a common point of software failure.

·        Integrationtests performed by independent testers typically use black-box techniques andare typically dealing with larger software components.

A better approach is to consider integration testingthe responsibility of both developer and independent testers, but make thestrategy of each teams testing effort does not overlap significantly. The exactnature of that overlap is based on the needs of the individual project. Werecommend you foster an environment where developers and independent systemtesters share a single vision of quality.

Integration Testingis an interim level of testing applied between module testing and systemtesting to test the interaction and consistency of an integrated subsystem.Integration testing is applied incrementally as modules are assembled intolarger subsystems. Testing is applied to subsystems which are assembled either:

1. Bottom-up - assembles up from the lowestlevel modules and replaces higher level modules with test harnesses to drivethe units under test.

2. Top-down – assembles down from the highestlevel modules replacing lower level modules with test stubs to simulateinterfaces

3. Mixture of both bottom-up and top-downapproaches.

Integration testingis performed so a partial system level test can be conducted without having towait for all the components to be available, and thus problems can be fixedbefore affecting downstream development on other components. Thus integrationtesting is usually done as soon as identifiable logical subsystems arecomplete. Integration testing is also performed as some tests cannot be carriedout on a fully integrated system, and requires integration with special testharnesses and stubs to properly test a component. It also helps to focustesting to particular components and to isolate the problems that will bediscovered. Usually the testing at this level is a mixture of functional(black-box) and structural (white-box) testing.

5.3.        Interface Testing

Interfaces are thesources of many errors, as often there is a misunderstanding of the waycomponents and subsystems should work together because they are developed bydifferent people. Interface testing focuses specifically on the way componentsand subsystems are linked and work together. Interface testing can be appliedto internal interfaces between subcomponents of a system (e.g. between separatemodules), as well as to external interfaces to other systems (e.g. datainterchanges to other applications).

5.4.        System Testing

Traditionally systemtesting is done when the software is functioning as a whole. An iterativelifecycle allows system testing to occur much earlier—as soon as well-formedsubsets of the use-case behavior are implemented. Usually the target is thesystem's end-to-end functioning elements.

5.5.        Regression Testing

Regression testing is applied after changeshave been made to a system. The operation of the new version of the system iscompared to the previous version to determine if there are any unexpecteddifferences.

To be precise - “Regression testing is a processof comparing two different version of a same software entity to ensure thatonly intended changes have been made in the old version to come up with latestversion”. Here intended changes means, changes documented in an approvedFunctional/Business requirements document.

Regression is applied as changing software,for example to fix known defects or to add new functions, has a very highlikelihood of introducing new defects. Some studies have predicted that forevery 6 lines of code modified a new defect will be added.

Regression testing has become a commonlyapplied form of testing due to the introduction of capture/playback test tools.Testers use these tools to capture the operation of a version of a system undertest, and then the operation can be played back automatically on later versionsof the system and any differences in behavior are reported.

5.6.        User Acceptance Testing

User acceptance testing is the final test actiontaken before deploying the software. The goal of acceptance testing is toverify that the software is ready, and that it can be used by end users toperform those functions and tasks for which the software was built.

There are other notions of acceptance testing, whichare generally characterized by a hand-off from one group or one team toanother. For example, a build acceptance test is the testing done toaccept the hand-over of a new software build from development into independenttesting.

Acceptance testing is the final test action beforedeploying the software. The goal of acceptance testing is to verify that thesoftware is ready and can be used by your end users to perform those functionsand tasks for which the software was built. There are three common strategiesfor implementing an acceptance test. They are:

       Formal acceptance

       Informal acceptanceor alpha test

       Beta test

The strategy you select is often based on thecontractual requirements, organizational and corporate standards, andapplication domain.

Formal Acceptance Testing

Formal acceptance testing is a highly managed processand is often an extension of the system test. The tests are planned anddesigned as carefully as, and in the same detail as, system testing. The testcases chosen should be a subset of those performed in system test. It'simportant not to deviate in any way from the chosen test cases. In manyorganizations, formal acceptance testing is fully automated.

The activities and artifacts are the same as forsystem testing. In some organizations, the development organization (or itsindependent test group), with the representatives of the end-user organization,performs the acceptance test. In other organizations, acceptance testing isperformed completely by the end-user organization or an objective group of peoplechosen by the end-user organization.

The benefits of this form of testing are:

·        Thefunctions and features to be tested are known.

·        Thedetails of the tests are known and can be measured.

·        Thetests can be automated, which permits regression testing.

·        Theprogress of the tests can be measured and monitored.

·        Theacceptability criteria are known.

The disadvantages include:

• Requires significant resources and planning.
• The tests may be a re-implementation of system tests.
• The testing may not uncover subjective defects in the software, since you're only looking for defects you expect to find.

Informal Acceptance Testing

In informal acceptance testing, the test proceduresfor performing the test are not as rigorously defined as for formal acceptance testing.The functions and business tasks to be explored are identified and documented,but there are no particular test cases to follow. The individual testerdetermines what to do. This approach to acceptance testing is not as controlledas formal testing and is more subjective than the formal one.

Informal acceptance testing is most frequentlyperformed by the end-user organization.

The benefits of this form of testing are:

·        Thefunctions and features to be tested are known.

·        Theprogress of the tests can be measured and monitored.

·        Theacceptability criteria are known.

·        Youwill uncover more subjective defects than with formal acceptance testing.

The disadvantages include:

·        Resources,planning, and management resources are required.

·        Youhave no control over what test cases are used.

·        Endusers may conform to the way the system works and not see the defects.

·        Endusers might focus on comparing the new system to a legacy system, rather thanlooking for defects.

·        Resourcesfor acceptance testing are not under the control of the project and could beconstricted.

5.7.        Beta Testing

Betatesting is the least controlled of the three acceptance test strategies. Inbeta testing, the amount of detail, the data, and the approach taken isentirely up to the individual tester. Each tester is responsible for creatinghis or her own environment, selecting his or her data, and determining whatfunctions, features, or tasks to explore. Each tester is responsible foridentifying his or her own criteria for whether to accept the system in itscurrent state or not.

Betatesting is implemented by end users, often with little or no management fromthe development (or other non end-user) organization. Beta testing is the mostsubjective of all acceptance test strategies.

The benefits of this form of testing are:

·        Testingis implemented by end users.

·        Thereare large volumes of potential test resources.

·        Thereis increased customer satisfaction for those who participate.

·        Youuncover more subjective defects than with formal or informal acceptancetesting.

The disadvantages include:

·        Youmight not test all functions or features.

·        Testprogress is difficult to measure.

·        Endusers might conform to the way the system works and not see or report thedefects.

·        Endusers may focus on comparing the new system to a legacy system, rather thanlooking for defects.

·        Resourcesfor acceptance testing are not under the control of the project and could beconstricted.

·        Acceptabilitycriteria are not known.

·        Youneed increased support resources to manage the beta testers.

A comment about sequence and timing oftest levels

Traditionally,unit testing is thought of as being implemented early in the iteration as thefirst stage of testing: all units required to be passed before subsequentstages are conducted. However, in an iterative development process, thisapproach is as a general rule, inappropriate. A better approach is to identifythe unit, integration and system tests that offer most potential for findingerrors, then implement and execute them based on a combination of greatest riskand supporting environment.