Why White box testing when black box testing is there to test conformance to requirements
Logic errors and incorrect assumptions most likely to be made when coding for "special cases". Need to ensure these execution paths are tested.
May find assumptions about execution paths incorrect, and so make design errors. White box testing can find these errors.
Typographical errors are random. Just as likely to be on an obscure logical path as on a mainstream path.
Software Testing Training
Software testing institute
corporate training software testing
Monday, June 29, 2009
Regression Testing
Regression Testing
Testing conducted for the purpose of evaluating whether or not a change to the system (all CM items) has introduced a new failure. Regression testing is
often accomplished through the construction, execution and analysis of product and system tests.
Regression testing is an expensive but necessary activity performed on modified software to provide confidence that changes are correct and do not adversely
affect other system components. Four things can happen when a developer attempts to fix a bug. Three of these things are bad, and one is good:
Because of the high probability that one of the bad outcomes will result from a change to the system, it is necessary to do regression testing.
It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Most industrial testing is done via test
suites; automated sets of procedures designed to exercise all parts of a program and to show defects. While the original suite could be used to test the
modified software, this might be very time-consuming. A regression test selection technique chooses, from an existing test set, the tests that are deemed
necessary to validate modified software.
There are three main groups of test selection approaches in use:
Minimization approaches seek to satisfy structural coverage criteria by identifying a minimal set of tests that must be rerun.
Coverage approaches are also based on coverage criteria, but do not require minimization of the test set. Instead, they seek to select all tests that
exercise changed or affected program components.
Safe attempt instead to select every test that will cause the modified program to produce different output than original program.
An interesting approach to limiting test cases is based on whether we can confine testing to the "vicinity" of the change. (Ex. If I put a new radio in my
car, do I have to do a complete road test to make sure the change was successful?) A new breed of regression test theory tries to identify, through program
flows or reverse engineering, where boundaries can be placed around modules and subsystems. These graphs can determine which tests from the existing suite
may exhibit changed behavior on the new version.
Regression testing has been receiving more attention as corporations focus on fixing the 'Year 2000 Bug'. The goal of most Y2K is to correct the date
handling portions of their system without changing any other behavior. A new 'Y2K' version of the system is compared against a baseline original system. With
the obvious exception of date formats, the performance of the two versions should be identical. This means not only do they do the same things correctly,
they also do the same things incorrectly. A non-Y2K bug in the original software should not have been fixed by the Y2K work.
A frequently asked question about regression testing is 'The developer says this problem is fixed. Why do I need to re-test?’ to which the answer is 'The
same person probably told you it worked in the first place'
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Testing conducted for the purpose of evaluating whether or not a change to the system (all CM items) has introduced a new failure. Regression testing is
often accomplished through the construction, execution and analysis of product and system tests.
Regression testing is an expensive but necessary activity performed on modified software to provide confidence that changes are correct and do not adversely
affect other system components. Four things can happen when a developer attempts to fix a bug. Three of these things are bad, and one is good:
Because of the high probability that one of the bad outcomes will result from a change to the system, it is necessary to do regression testing.
It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Most industrial testing is done via test
suites; automated sets of procedures designed to exercise all parts of a program and to show defects. While the original suite could be used to test the
modified software, this might be very time-consuming. A regression test selection technique chooses, from an existing test set, the tests that are deemed
necessary to validate modified software.
There are three main groups of test selection approaches in use:
Minimization approaches seek to satisfy structural coverage criteria by identifying a minimal set of tests that must be rerun.
Coverage approaches are also based on coverage criteria, but do not require minimization of the test set. Instead, they seek to select all tests that
exercise changed or affected program components.
Safe attempt instead to select every test that will cause the modified program to produce different output than original program.
An interesting approach to limiting test cases is based on whether we can confine testing to the "vicinity" of the change. (Ex. If I put a new radio in my
car, do I have to do a complete road test to make sure the change was successful?) A new breed of regression test theory tries to identify, through program
flows or reverse engineering, where boundaries can be placed around modules and subsystems. These graphs can determine which tests from the existing suite
may exhibit changed behavior on the new version.
Regression testing has been receiving more attention as corporations focus on fixing the 'Year 2000 Bug'. The goal of most Y2K is to correct the date
handling portions of their system without changing any other behavior. A new 'Y2K' version of the system is compared against a baseline original system. With
the obvious exception of date formats, the performance of the two versions should be identical. This means not only do they do the same things correctly,
they also do the same things incorrectly. A non-Y2K bug in the original software should not have been fixed by the Y2K work.
A frequently asked question about regression testing is 'The developer says this problem is fixed. Why do I need to re-test?’ to which the answer is 'The
same person probably told you it worked in the first place'
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Volume Testing
Performance Testing - Volume Testing
Volume Testing: Testing where the system is subjected to large volumes of data. As regards faults that should be found through volume testing are those,
where the behaviour of the software deviates from that expected for a specified volume of data. Thus a bank system will be tested for faults at much larger
volumes of data, than that of small retailer software. A fault which is only manifested on a table with a million records will be of no concern to the retail
software, but will be picked up by the bank testers.
For example, a window often used window object is populated with data, by calling a database object which runs a complex SQL query. Supposing the component
is tested against tables with only 4-5 records. Of course it will return within seconds. Everything seems fine. It is then integrated with the window, and
system tested. Again everything seems ok. It is only when the application is in User Acceptance (or even gone live) and it is tested against 100,000 records,
is it discovered that, the SQL was not properly optimized and the tables not indexed. Thus it should have been tested at the component level.
Volume testing needs two things. Firstly clear expected outcomes of how the software is to behave for a given level of data. Secondly, data, and lots of it.
The expected behaviour at various levels, should be in the specification documentation. Ideally this will say something like "the customers details will be
returned returned on the screen within 3 seconds, from a database with 1 million customer records." This gives the tester a benchmark to base a test case on.
The second requirement for data, needs either real life data, or simulated data. Usually, real life data will come in the form of a customer database, that
has had private information, such as names and account numbers scrambled. Alternatively records can be created from scratch using automated tools or by
adding rules directly on to the database, with SQL.
As with all testing, proper records must be kept showing the inputs, outputs other information, to aid potential debugging and audit purposes
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Volume Testing: Testing where the system is subjected to large volumes of data. As regards faults that should be found through volume testing are those,
where the behaviour of the software deviates from that expected for a specified volume of data. Thus a bank system will be tested for faults at much larger
volumes of data, than that of small retailer software. A fault which is only manifested on a table with a million records will be of no concern to the retail
software, but will be picked up by the bank testers.
For example, a window often used window object is populated with data, by calling a database object which runs a complex SQL query. Supposing the component
is tested against tables with only 4-5 records. Of course it will return within seconds. Everything seems fine. It is then integrated with the window, and
system tested. Again everything seems ok. It is only when the application is in User Acceptance (or even gone live) and it is tested against 100,000 records,
is it discovered that, the SQL was not properly optimized and the tables not indexed. Thus it should have been tested at the component level.
Volume testing needs two things. Firstly clear expected outcomes of how the software is to behave for a given level of data. Secondly, data, and lots of it.
The expected behaviour at various levels, should be in the specification documentation. Ideally this will say something like "the customers details will be
returned returned on the screen within 3 seconds, from a database with 1 million customer records." This gives the tester a benchmark to base a test case on.
The second requirement for data, needs either real life data, or simulated data. Usually, real life data will come in the form of a customer database, that
has had private information, such as names and account numbers scrambled. Alternatively records can be created from scratch using automated tools or by
adding rules directly on to the database, with SQL.
As with all testing, proper records must be kept showing the inputs, outputs other information, to aid potential debugging and audit purposes
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Stress testing 2
Performance Testing - Stress testing 2
Stress testing :is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, mips, interrupts, etc.) needed to process
that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not
expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data). Bugs and
failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load
(incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Stress testing :is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, mips, interrupts, etc.) needed to process
that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not
expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data). Bugs and
failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load
(incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Performance Testing - Load testing
Performance Testing - Load testing
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software
reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean
representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain
without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Load testing: is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software
reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean
representative load," "overload," "high load," etc. In performance testing, load is varied from a minimum (zero) to the maximum level the systemcan sustain
without running out of resources or having, transactions suffer (application-specific) excessive delay.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Integration Testing
Integration Testing
One of the most difficult aspects of software development is the integration and testing of large, untested sub-systems. The integrated system frequently
fails in significant and mysterious ways, and it is difficult to fix it
Integration testing exercises several units that have been combined to form a module, subsystem, or system. Integration testing focuses on the interfaces
between units, to make sure the units work together. The nature of this phase is certainly 'white box', as we must have a certain knowledge of the units to
recognize if we have been successful in fusing them together in the module.
There are three main approaches to integration testing: top-down, bottom-up and 'big bang'. Top-down combines, tests, and debugs top-level routines that
become the test 'harness' or 'scaffolding' for lower-level units. Bottom-up combines and tests low-level units into progressively larger modules and
subsystems. 'Big bang' testing is, unfortunately, the prevalent integration test 'method'. This is waiting for all the module units to be complete before
trying them out together.
Integration tests can rely heavily on stubs or drivers. Stubs stand-in for finished subroutines or sub-systems. A stub might consist of a function header
with no body, or it may read and return test data from a file, return hard-coded values, or obtain data from the tester. Stub creation can be a time
consuming piece of testing.
The cost of drivers and stubs in the top-down and bottom-up testing methods is what drives the use of 'big bang' testing. This approach waits for all the
modules to be constructed and tested independently, and when they are finished, they are integrated all at once. While this approach is very quick, it
frequently reveals more defects than the other methods. These errors have to be fixed and as we have seen, errors that are found 'later' take longer to fix.
In addition, like bottom up, there is really nothing that can be demonstrated until later in the process.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
One of the most difficult aspects of software development is the integration and testing of large, untested sub-systems. The integrated system frequently
fails in significant and mysterious ways, and it is difficult to fix it
Integration testing exercises several units that have been combined to form a module, subsystem, or system. Integration testing focuses on the interfaces
between units, to make sure the units work together. The nature of this phase is certainly 'white box', as we must have a certain knowledge of the units to
recognize if we have been successful in fusing them together in the module.
There are three main approaches to integration testing: top-down, bottom-up and 'big bang'. Top-down combines, tests, and debugs top-level routines that
become the test 'harness' or 'scaffolding' for lower-level units. Bottom-up combines and tests low-level units into progressively larger modules and
subsystems. 'Big bang' testing is, unfortunately, the prevalent integration test 'method'. This is waiting for all the module units to be complete before
trying them out together.
Integration tests can rely heavily on stubs or drivers. Stubs stand-in for finished subroutines or sub-systems. A stub might consist of a function header
with no body, or it may read and return test data from a file, return hard-coded values, or obtain data from the tester. Stub creation can be a time
consuming piece of testing.
The cost of drivers and stubs in the top-down and bottom-up testing methods is what drives the use of 'big bang' testing. This approach waits for all the
modules to be constructed and tested independently, and when they are finished, they are integrated all at once. While this approach is very quick, it
frequently reveals more defects than the other methods. These errors have to be fixed and as we have seen, errors that are found 'later' take longer to fix.
In addition, like bottom up, there is really nothing that can be demonstrated until later in the process.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Functional testing
Functional testing
Functional testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as
black-box testing.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Functional testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as
black-box testing.
Software Testing Training
http://www.qacampus.com
QACampus is Center of Education for anything and everything about software testing.
http://www.crestech.in
http://www.crestechsoftware.com.au
Subscribe to:
Posts (Atom)