Tuesday, September 4, 2007

Test plan

A software project test plan is a document that describes the objectives, scope, approach, and focus of a software testing effort. The process of preparing a test plan is a useful way to think through the efforts needed to validate the acceptability of a software product. The completed document will help people outside the test group understand the 'why' and 'how' of product validation. It should be thorough enough to be useful but not so thorough that no one outside the test group will read it. The following are some of the items that might be included in a test plan, depending on the particular project:

Ø Title

Ø Identification of software including version/release numbers

Ø Revision history of document including authors, dates, approvals

Ø Table of Contents

Ø Purpose of document, intended audience

Ø Objective of testing effort

Ø Software product overview

Ø Relevant related document list, such as requirements, design documents, other test plans, etc.

Ø Relevant standards or legal requirements

Ø Traceability requirements

Ø Relevant naming conventions and identifier conventions

Ø Overall software project organization and personnel/contact-info/responsibilties

Ø Test organization and personnel/contact-info/responsibilities

Ø Assumptions and dependencies

Ø Project risk analysis

Ø Testing priorities and focus

Ø Scope and limitations of testing

Ø Test outline - a decomposition of the test approach by test type, feature, functionality, process, system, module, etc. as applicable

Ø Outline of data input equivalence classes, boundary value analysis, error classes

Ø Test environment - hardware, operating systems, other required software, data configurations, interfaces to other systems

Ø Test environment validity analysis - differences between the test and production systems and their impact on test validity.

Ø Test environment setup and configuration issues

Ø Software migration processes

Ø Software CM processes

Ø Test data setup requirements

Ø Database setup requirements

Ø Outline of system-logging/error-logging/other capabilities, and tools such as screen capture software, that will be used to help describe and report bugs

Ø Discussion of any specialized software or hardware tools that will be used by testers to help track the cause or source of bugs

Ø Test automation - justification and overview

Ø Test tools to be used, including versions, patches, etc.

Ø Test script/test code maintenance processes and version control

Ø Problem tracking and resolution - tools and processes

Ø Project test metrics to be used

Ø Reporting requirements and testing deliverables

Ø Software entrance and exit criteria

Ø Initial sanity testing period and criteria

Ø Test suspension and restart criteria

Ø Personnel allocation

Ø Personnel pre-training needs

Ø Test site/location

Ø Outside test organizations to be utilized and their purpose, responsibilities, deliverables, contact persons, and coordination issues

Ø Relevant proprietary, classified, security, and licensing issues.

Ø Open issues

Ø Appendix - glossary, acronyms, etc.

What steps are needed to develop and run software tests

The following are some of the steps to consider:

Ø Obtain requirements, functional design, and internal design specifications and other necessary documents

Ø Obtain budget and schedule requirements

Ø Determine project-related personnel and their responsibilities, reporting requirements, required standards and processes (such as release processes, change processes, etc.)

Ø Identify application's higher-risk aspects, set priorities, and determine scope and limitations of tests

Ø Determine test approaches and methods - unit, integration, functional, system, load, usability tests, etc.

Ø Determine test environment requirements (hardware, software, communications, etc.)

Ø Determine testware requirements (record/playback tools, coverage analyzers, test tracking, problem/bug tracking, etc.)

Ø Determine test input data requirements

Ø Identify tasks, those responsible for tasks, and labor requirements

Ø Set schedule estimates, timelines, milestones

Ø Determine input equivalence classes, boundary value analyses, error classes

Ø Prepare test plan document and have needed reviews/approvals

Ø Write test cases

Ø Have needed reviews/inspections/approvals of test cases

Ø Prepare test environment and testware, obtain needed user manuals/reference documents/configuration guides/installation guides, set up test tracking processes, set up logging and archiving processes, set up or obtain test input data

Ø Obtain and install software releases

Ø Perform tests

Ø Evaluate and report results

Ø Track problems/bugs and fixes

Ø Retest as needed

Ø Maintain and update test plans, test cases, test environment, and testware through life cycle

Sunday, September 2, 2007

Software Development Life Cycle

Software development life cycle is a process of the problem solving; the software life cycle begins when an application first conceived and ends when it is no longer time.

· Identification definition of the problem to solve through the process of the consulting with the client.

· Formalize the description into specification

· Implement that specification.


Feasability study
A
nalysis
Design
Coding
Testing
Installation & Maintenance


Feasability Study: analyst conducts an initial study of the problem. In this we gathered information from the client. Feasibility report contains application areas, cost estimation, system requirements and time scale for implementation and expected benefits etc.

Analysis: in this process of investigating a business with a view to determine how best to manage the various procedures and information processing tasks that it involves. System analysis contains Business Requirements Specification (BRS) document, Functional Requirements Specification (FRS) document, use cases (user action and system response), system flow charts, data flow diagrams and organization charts. Depends on feasibility SRS document will be prepared by business analyst. Based on SRS document use cases prepare here.

Design: planning the structure of the information system to be implemented. System analysis determines how it should be done. High Level Design and Low Level Design prepared here. Design report contains user interface design, design of output reports, input screens, database tables, files, system security, backups, validation, passwords and test plan.

Coding: coding is nothing but set of programs. Translate the design of the system into code in a given programming language. It contains all the programs, functions, reports that related to the coding.

Testing: Testing is a process of executing a program with an intention of finding the defects.

Installation & Maintenance: it includes file conversion system testing, staff training, corrective maintenance, perfective maintenance and adaptive maintenance.

Ad-hoc Testing

Ad-hoc Testing: (Informal Testing)

A Tester conducts a test with out following the formal methods is called “Ad- hoc testing”.

Testing of Software Application Without using the Test case Techniques.

There r five types in the Ad- hoc Testing.

1) Monkey Testing

2) Buddy Testing

3) Pair Testing

4) Iterative/Re Testing.

5) Exploratory Testing.

1) Monkey testing :

If Tester conducts the test on main functionalities of application build, that can be called as Monkey Testing.

It is also called “Chimpanzee Test” or “Gorilla Testing”.

2) Buddy Testing:

Due to lack of time, test engineer grouped with developers to conduct test on application build as early as possible. This type of testing is known as Buddy Testing.

Buddy=Group of programmers and testers.

3) Pair Testing:

Due to lack of knowledge on project domain some times juniors grouped with senior test engineers to share their knowledge. This type of testing is known as Pair testing.

4) Iterative Testing/Retesting/Data Driven Testing:

The repletion of same test on same build more than one time is known as Iterative Testing.

5) Exploratory Testing:

Some Times Test engineers are conducting testing, depending on available documents, past experience, discussion with others, through connect with internet.

This type of internet is known as “Exploratory Testing”.


Types of Screen Validations

1 Aesthetic Conditions:

  1. Is the general screen background the correct color?
  2. Are the field prompts the correct color?
  3. Are the field backgrounds the correct color?
  4. In read-only mode, are the field prompts the correct color?
  5. In read-only mode, are the field backgrounds the correct color?
  6. Are all the screen prompts specified in the correct screen font?
  7. Is the text in all fields specified in the correct screen font?
  8. Are all the field prompts aligned perfectly on the screen?
  9. Are all the field edit boxes aligned perfectly on the screen?
  10. Are all group boxes aligned correctly on the screen?
  11. Should the screen be resizable?
  12. Should the screen be allowed to minimize?
  13. Are all the field prompts spelt correctly?
  14. Are all character or alphanumeric fields left justified? This is the default unless otherwise specified.
  15. Are all numeric fields right justified? This is the default unless otherwise specified.
  16. Is all the micro-help text spelt correctly on this screen?
  17. Is all the error message text spelt correctly on this screen?
  18. Is all user input captured in UPPER case or lowercase consistently?
  19. Where the database requires a value (other than null) then this should be defaulted into fields. The user must either enter an alternative valid value or leave the default value intact.
  20. Assure that all windows have a consistent look and feel.
  21. Assure that all dialog boxes have a consistent look and feel.

2 Validation Conditions:

  1. Does a failure of validation on every field cause a sensible user error message?
  2. Is the user required to fix entries, which have failed validation tests?
  3. Have any fields got multiple validation rules and if so are all rules being applied?
  4. If the user enters an invalid value and clicks on the OK button (i.e. does not TAB off the field) is the invalid entry identified and highlighted correctly with an error message?
  5. Is validation consistently applied at screen level unless specifically required at field level?
  6. For all numeric fields check whether negative numbers can and should be able to be entered.
  7. For all numeric fields check the minimum and maximum values and also some mid-range values allowable?
  8. For all character/alphanumeric fields check the field to ensure that there is a character limit specified and that this limit is exactly correct for the specified database size?
  9. Do all mandatory fields require user input?

10. If any of the database columns don't allow null values then the corresponding screen fields must be mandatory. (If any field, which initially was mandatory, has become optional then check whether null values are allowed in this field.)

3 Navigation Conditions:

  1. Can the screen be accessed correctly from the menu?
  2. Can the screen be accessed correctly from the toolbar?
  3. Can the screen be accessed correctly by double clicking on a list control on the previous screen?
  4. Can all screens accessible via buttons on this screen be accessed correctly?
  5. Can all screens accessible by double clicking on a list control be accessed correctly?
  6. Is the screen modal? (i.e.) Is the user prevented from accessing other functions when this screen is active and is this correct?
  7. Can a number of instances of this screen be opened at the same time and is this correct?

4 Usability Conditions:

  1. Are all the dropdowns on this screen sorted correctly? Alphabetic sorting is the default unless otherwise specified.
  2. Is all date entry required in the correct format?
  3. Have all pushbuttons on the screen been given appropriate Shortcut keys?
  4. Do the Shortcut keys work correctly?
  5. Have the menu options that apply to your screen got fast keys associated and should they have?
  6. Does the Tab Order specified on the screen go in sequence from Top Left to bottom right? This is the default unless otherwise specified.
  7. Are all read-only fields avoided in the TAB sequence?
  8. Are all disabled fields avoided in the TAB sequence?
  9. Can the cursor be placed in the microhelp text box by clicking on the text box with the mouse?
  10. Can the cursor be placed in read-only fields by clicking in the field with the mouse?
  11. Is the cursor positioned in the first input field or control when the screen is opened?
  12. Is there a default button specified on the screen?
  13. Does the default button work correctly?
  14. When an error message occurs does the focus return to the field in error when the user cancels it?
  15. When the user Alt+Tab's to another application does this have any impact on the screen upon return to the application?
  16. Do all the fields edit boxes indicate the number of characters they will hold by there length? e.g. a 30 character field should be a lot longer

5 Data Integrity Conditions:

  1. Is the data saved when the window is closed by double clicking on the close box?
  2. Check the maximum field lengths to ensure that there are no truncated characters?
  3. Where the database requires a value (other than null) then this should be defaulted into fields. The user must either enter an alternative valid value or leave the default value intact.
  4. Check maximum and minimum field values for numeric fields?
  5. If numeric fields accept negative values can these be stored correctly on the database and does it make sense for the field to accept negative numbers?
  6. If a set of radio buttons represents a fixed set of values such as A, B and C then what happens if a blank value is retrieved from the database? (In some situations rows can be created on the database by other functions, which are not screen based, and thus the required initial values can be incorrect.)
  7. If a particular set of data is saved to the database check that each value gets saved fully to the database. (i.e.) Beware of truncation (of strings) and rounding of numeric values.

6 Modes (Editable Read-only) Conditions:

  1. Are the screen and field colors adjusted correctly for read-only mode?
  2. Should a read-only mode be provided for this screen?
  3. Are all fields and controls disabled in read-only mode?
  4. Can the screen be accessed from the previous screen/menu/toolbar in read-only mode?

5. Can all screens available from this screen be accessed in read-only mode?

  1. Check that no validation is performed in read-only mode.

7 General Conditions:

  1. Assure the existence of the "Help" menu.
  2. Assure that the proper commands and options are in each menu.
  3. Assure that all buttons on all tool bars have a corresponding key commands.
  4. Assure that each menu command has an alternative (hot-key) key sequence, which will invoke it where appropriate.
  5. In drop down list boxes, ensure that the names are not abbreviations / cut short
  6. In drop down list boxes, assure that the list and each entry in the list can be accessed via appropriate key / hot key combinations.
  7. Ensure that duplicate hot keys do not exist on each screen

8. Ensure the proper usage of the escape key (which is to undo any changes that have been made) and generates a caution message "Changes will be lost - Continue yes/no"

  1. Assure that the cancel button functions the same as the escape key.

10. Assure that the Cancel button operates, as a Close button when changes have been made that cannot be undone.

  1. Assure that only command buttons, which are used by a particular window, or in a particular dialog box, are present. – (i.e) make sure they don't work on the screen behind the current screen.
  2. When a command button is used sometimes and not at other times, assures that it is grayed out when it should not be used.
  3. Assure that OK and Cancel buttons are grouped separately from other command buttons.
  4. Assure that command button names are not abbreviations.
  5. Assure that all field labels/names are not technical labels, but rather are names meaningful to system users.
  6. Assure that command buttons are all of similar size and shape, and same font & font size.
  7. Assure that each command button can be accessed via a hot key combination.
  8. Assure that command buttons in the same window/dialog box do not have duplicate hot keys.
  9. Assure that each window/dialog box has a clearly marked default value (command button, or other object) which is invoked when the Enter key is pressed - and NOT the Cancel or Close button
  10. Assure that focus is set to an object/button, which makes sense according to the function of the window/dialog box.
  11. Assure that all option buttons (and radio buttons) names are not abbreviations.
  12. Assure that option button names are not technical labels, but rather are names meaningful to system users.
  13. If hot keys are used to access option buttons, assure that duplicate hot keys do not exist in the same window/dialog box.
  14. Assure that option box names are not abbreviations.
  15. Assure that option boxes, option buttons, and command buttons are logically grouped together in clearly demarcated areas "Group Box"
  16. Assure that the Tab key sequence, which traverses the screens, does so in a logical way.
  17. Assure consistency of mouse actions across windows.
  18. Assure that the color red is not used to highlight active objects (many individuals are red-green color blind).
  19. Assure that the user will have control of the desktop with respect to general color and highlighting (the application should not dictate the desktop background characteristics).
  20. Assure that the screen/window does not have a cluttered appearance
  21. Ctrl + F6 opens next tab within tabbed window
  22. Shift + Ctrl + F6 opens previous tab within tabbed window
  23. Tabbing will open next tab within tabbed window if on last field of current tab
  24. Tabbing will go onto the 'Continue' button if on last field of last tab within tabbed window
  25. Tabbing will go onto the next editable field in the window
  26. Banner style & size & display exact same as existing windows
  27. If 8 or less options in a list box, display all options on open of list box - should be no need to scroll
  28. Errors on continue will cause user to be returned to the tab and the focus should be on the field causing the error. (i.e the tab is opened, highlighting the field with the error on it)
  29. Pressing continue while on the first tab of a tabbed window (assuming all fields filled correctly) will not open all the tabs.
  30. On open of tab focus will be on first editable field
  31. All fonts to be the same
  32. Alt+F4 will close the tabbed window and return you to main screen or previous screen (as appropriate), generating "changes will be lost" message if necessary.
  33. Microhelp text for every enabled field & button
  34. Ensure all fields are disabled in read-only mode
  35. Progress messages on load of tabbed screens
  36. Return operates continue
  37. If retrieve on load of tabbed window fails window should not open

Black box testing Methods

1 Graph-based Testing Methods

  • Black-box methods based on the nature of the relationships (links) among the program objects (nodes), test cases are designed to traverse the entire graph
  • Transaction flow testing (nodes represent steps in some transaction and links represent logical connections between steps that need to be validated)
  • Finite state modeling (nodes represent user observable states of the software and links represent transitions between states)
  • Data flow modeling (nodes are data objects and links are transformations from one data object to another)
  • Timing modeling (nodes are program objects and links are sequential connections between these objects, link weights are required execution times)

2 Equivalence Partitioning

  • Black-box technique that divides the input domain into classes of data from which test cases can be derived
  • An ideal test case uncovers a class of errors that might require many arbitrary test cases to be executed before a general error is observed
  • Equivalence class guidelines:

1. If input condition specifies a range, one valid and two invalid equivalence classes are defined

2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined

3. If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined

4. If an input condition is Boolean, one valid and one invalid equivalence class is defined

3 Boundary Value Analysis

  • Black-box technique that focuses on the boundaries of the input domain rather than its center

· BVA guidelines:

1. If input condition specifies a range bounded by values a and b, test cases should include a and b, values just above and just below a and b

2. If an input condition specifies and number of values, test cases should be exercise the minimum and maximum numbers, as well as values just above and just below the minimum and maximum values

3. Apply guidelines 1 and 2 to output conditions, test cases should be designed to produce the minimum and maxim output reports

4. If internal program data structures have boundaries (e.g. size limitations), be certain to test the boundaries

4 Comparison Testing

  • Black-box testing for safety critical systems in which independently developed implementations of redundant systems are tested for conformance to specifications
  • Often equivalence class partitioning is used to develop a common set of test cases for each implementation

5 Orthogonal Array Testing

  • Black-box technique that enables the design of a reasonably small set of test cases that provide maximum test coverage
  • Focus is on categories of faulty logic likely to be present in the software component (without examining the code)
  • Priorities for assessing tests using an orthogonal array

1. Detect and isolate all single mode faults

2. Detect all double mode faults

3. Multimode faults

6 Specialized Testing

  • Graphical user interfaces
  • Client/server architectures
  • Documentation and help facilities
  • Real-time systems

1. Task testing (test each time dependent task independently)

2. Behavioral testing (simulate system response to external events)

3. Intertask testing (check communications errors among tasks)

4. System testing (check interaction of integrated system software and hardware)

7 Advantages of Black Box Testing

  • More effective on larger units of code than glass box testing
  • Tester needs no knowledge of implementation, including specific programming languages
  • Tester and programmer are independent of each other
  • Tests are done from a user's point of view
  • Will help to expose any ambiguities or inconsistencies in the specifications

· Test cases can be designed as soon as the specifications are complete

8 Disadvantages of Black Box Testing

  • Only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever
  • Without clear and concise specifications, test cases are hard to design
  • There may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried

· May leave many program paths untested

  • Cannot be directed toward specific segments of code which may be very complex (and therefore more error prone)

· Most testing related research has been directed toward glass box testing

Friday, August 31, 2007

WHITE BOX TESTING

Software testing approaches that examine the program structure and derive test data from the program logic. Structural testing is sometimes referred to as clear-box testing since white boxes are considered opaque and do not really permit visibility into the code.

Synonyms for white box testing

  • Glass Box testing
  • Structural testing
  • Clear Box testing
  • Open Box Testing

The purpose of white box testing

Initiate a strategic initiative to build quality throughout the life cycle of a software product or service.

Provide a complementary function to black box testing.

Perform complete coverage at the component level.

Improve quality by optimizing performance.

Practices :

This section outlines some of the general practices comprising white-box testing process. In general, white-box testing practices have the

following considerations:

  1. The allocation of resources to perform class and method analysis and to document and review the same.
  2. Developing a test harness made up of stubs, drivers and test object libraries.
  3. Development and use of standard procedures, naming conventions and libraries.
  4. Establishment and maintenance of regression test suites and procedures.
  5. Allocation of resources to design, document and manage a test history library.
  6. The means to develop or acquire tool support for automation of capture/replay/compare, test suite execution, results verification and documentation capabilities.

1 Code Coverage Analysis

1.1 Basis Path Testing

A testing mechanism proposed by McCabe whose aim is to derive a logical complexity measure of a procedural design and use this as a guide for defining a basic set of execution paths. These are test cases that exercise basic set will execute every statement at least once.

1.1.1 Flow Graph Notation

A notation for representing control flow similar to flow charts and UML activity diagrams.

1.1.2 Cyclomatic Complexity

The cyclomatic complexity gives a quantitative measure of 4the logical complexity. This value gives the number of independent paths in the basis set, and an upper bound for the number of tests to ensure that each statement is executed at least once. An independent path is any path through a program that introduces at least one new set of processing statements or a new condition (i.e., a new edge). Cyclomatic complexity provides upper bound for number of tests required to guarantee coverage of all program statements.

1.2 Control Structure testing

1.2.1 Conditions Testing

Condition testing aims to exercise all logical conditions in a program module. They may define:

· Relational expression: (E1 op E2), where E1 and E2 are arithmetic expressions.

· Simple condition: Boolean variable or relational expression, possibly proceeded by a NOT operator.

· Compound condition: composed of two or more simple conditions, Boolean operators and parentheses.

· Boolean expression : Condition without Relational expressions.

1.2.2 Data Flow Testing

Selects test paths according to the location of definitions and use of variables.

1.2.3 Loop Testing

Loops fundamental to many algorithms. Can define loops as simple, concatenated, nested, and unstructured.


2 Design by Contract (DbC)

DbC is a formal way of using comments to incorporate specification information into the code itself. Basically, the code specification is expressed unambiguously using a formal language that describes the code's implicit contracts. These contracts specify such requirements as:

· Conditions that the client must meet before a method is invoked.

· Conditions that a method must meet after it executes.

· Assertions that a method must satisfy at specific points of its execution

Tools that check DbC contracts at runtime such as JContract [http://www.parasoft.com/products/jtract/index.htm] are used to perform this function.

3 Profiling

Profiling provides a framework for analyzing Java code performance for speed and heap memory use. It identifies routines that are consuming the majority of the CPU time so that problems may be tracked down to improve performance.

These include the use of Microsoft Java Profiler API and Sun’s profiling tools that are bundled with the JDK. Third party tools such as JaViz [http://www.research.ibm.com/journal/sj/391/kazi.html] may also be used to perform this function.

4 Error Handling

Exception and error handling is checked thoroughly are simulating partial and complete fail-over by operating on error causing test vectors. Proper error recovery, notification and logging are checked against references to validate program design.

5 Transactions

Systems that employ transaction, local or distributed, may be validated to ensure that ACID (Atomicity, Consistency, Isolation, Durability). Each of the individual parameters is tested individually against a reference data set.

Transactions are checked thoroughly for partial/complete commits and rollbacks encompassing databases and other XA compliant transaction processors.