MT - 64 - What is Test Coverage !

Test coverage is the measure of the amount or percentage of testing which is covered by a test set. Its gives information about the part of the system which has been covered in testing while running a test suite. Test coverage is defined as a technique which determines whether our test cases are actually covering the application code and how much code is exercised when we run those test cases.

Note : 100% percent test coverage doesn't means that 100% is tested and system is defect free.

Formula of test coverage :
                     

Test Coverage measurement

Test coverage can be best measured with the following  3 things:

-By mapping the requirements to the test cases.
-By seeing the test case status.
-By code coverage analysis.

Benefit of code coverage measurement:

-We can find the gaps easily in requirements, test cases and defects at an early level and code level.

-Time, scope and cost can be kept under control

-Defect prevention is possible at an early stage of project life cycle

-It can assure the quality of test

-It can help identify what portions of the code were actually touched for the release or fix

-We can create additional test cases for complete coverage

-It helps in finding areas of a program not exercised by a set of test cases

-The test coverage analysis can determine the decision points and important path made in the application which helps us to increase the test coverage.


Drawback of code coverage measurement:

- Drawback of code coverage measurement is that it measures coverage of what has been written, i.e. the code itself; it cannot say anything about the software that has not been written.

- If a specified function has not been implemented or a function was omitted from the specification, then structure-based techniques cannot say anything about them it only looks at a structure which is already there.

MT - 63 - Absence of error fallacy !

This is one of the principles of testing given by ISTQB. It says that if a system built is unusable from user point of view then there is no use of testing it and finding faults and fixing them. The system built should be in accordance with the user requirements. In other words we can say that customers are more interested in the usability of the software rather than getting a error-free software. The system developed should help them in completing their tasks effectively and efficiently. For the customers usable software is more important than a error free software. The people and organizations who buy and use it to aid in their day-to-day tasks - are not interested in defects or numbers of defects, except when they are directly affected by the instability of the software.

MT - 62 - Orthogonal Array(OATS) !

Before discussing about orthogonal array let me take an example of a real time testing scenario. Suppose there is a requirement of testing where we need to test 3 parameters with 3 different values in a system. So here to do exhaustive testing and see the response of every possible input we need to have 27 test data sets.

So here comes the concept of Orthogonal arrays.This technique is used when we have to test with huge number data having many permutations and combinations.This technique maximizes the coverage with lesser number of test cases. I'd say this technique guarantees 100% coverage with comparatively lesser no of cases.
Now we map each values with the parameters. To do the mapping we need to identify all the parameters(also called as factors and values. Then make a table with all the possible combinations saying that each row is unique So here we can see that all the permutations and combinations are possible in 9 cases only. We can write our test sets based on the parameters and values.

Orthogonal Array based on parameter and value

Test data sets ↓	Parameter 1	Parameter 2	Parameter 3
1	Value 1	Value 1	Value 3
2	Value 1	Value 2	Value 2
3	Value 1	Value 3	Value 1
4	Value 2	Value 1	Value 2
5	Value 2	Value 2	Value 1
6	Value 2	Value 3	Value 3
7	Value 3	Value 1	Value 1
8	Value 3	Value 2	Value 3
9	Value 3	Value 3	Value 2

Now once the array is created and we do testing there is a possibility of defects. There may be a single mode or multimode faults which we need to confirm seeing the parameters and values. Let's see it one by one :

Single Mode Faults - Single mode faults occur only due to one parameter. For example, in above Orthogonal array if test cases 7, 8 and 9 show error, we can expect that value 3 of parameter 1 is causing the error. Likewise we can detect as well as isolate the error.

Double Mode Fault - Double mode fault is caused by the two specific parameters values interacting together. Such an interaction is a harmful interaction between interacting parameters.

Multimode Faults - If more than two interacting components produce the consistent erroneous output, then it is a multimode fault. Orthogonal array detects the multimode faults.


Some important points :
- Creates an efficient and concise test set with many fewer test cases than testing all combinations
of all variables.

- This technique exercises some of the complex combinations of all the variables.

- It is simpler to generate and less error prone than test sets created by hand.
- This is a type of black box testing techniques.
-Orthogonal arrays can be applied in user interface testing, system testing, regression testing, configuration testing and performance testing
-OAT, is a systematic and statistical approach to pairwise interactions.
- Test cycle time is reduced

MT - 61 - Defect Density !

Defects are considered to be the unexpected behaviour of the system with respect to the client requirements. The purpose of testing is to bring the confidence in the built system by mapping them with requirements. During testing we log lot of defects, some of them gets cancelled and some of them gets approved. From the management's perspective we need to have a track over the defects , its density and occurrence.

Defect Density is defined as the number of confirmed defects in software/module during a specific period of testing\development divided by the size of the software/module. The count may be used to ensure the correct time for release.



Here, size may be the number of lines of code or may be measured using FP(Funtion Points).

Number of defects represents the count of confirmed defects.

The defect density is calculated over a period based on the organisation standards.


Defect density can be affected by : 

- The complexity of code
- Less time to test more code
- Unskilled testers


Advantages 

-  To compare the relative no of defects in various components so that risky components may be identified

-For comparing software/products so that quality of each software/product can be quantified and resources focused towards those with low quality.

-It helps measure the testing effectiveness

-It can be helpful in estimating the testing and rework due to bugs

-It can estimate the remaining defects in the software

-Before the release we can determine whether our testing is sufficient

MT - 60 - Testing Techniques !

There is always a confusion between testing techniques and testing methods. So lets talk about testing techniques here.. There are basically two types of testing techniques : Static and Dynamic.
These two techniques can be further sub-categorized into various types.
Dynamic techniques are subdivided into three more categories: specification-based (black-box, also known as behavioural techniques), structure-based (white-box or structural techniques) and experience-based. Specification-based techniques include both functional and non-functional techniques (i.e. quality characteristics).
static testing techniques do not execute the code being examined and are generally used before any tests are executed on the software.

MT - 59 - Difference between white box and Black box testing

This is a basic question asked in many interviews. Lets see the differences in tabular form :

Black Box testing	White Box testing
Black Box testing is a testing method in which we consider the system as a black box, we give input and check the output. Here the tester need not to have the knowledge of internal structure.	White box testing is a testing method in which the tester needs to know the internal structure and design of the system. The system is considered as white box or glass box.
Test Cases are written based on the basis of the requirements.	Test cases are written on the basis of detailed design.
Programming/Coding knowledge is not required	Programming/Coding knowledge is required
Also known as functional testing, it is applicable to higher levels like acceptance and system testing	Also known as structural testing , it is applicable to lower level of testing like unit testing
There is a  separate team of Testers to do Black box testing	Usually Developers are responsible for White box testing
This type of testing always focuses on what is performing/ carried out	This type of testing always focuses on how it is performing/ carried out
External Testing	Internal testing
Code optimization can’t be done and since the testing is not at internal level so hidden errors can’t be caught at code level	Code Optimization can be done and hidden errors can be discovered.
Black box testing techniques are Equivalence partitioning , BVA , Error guessing , CE graphing, etc	White box testing techniques are path testing , branch testing , condition testing , loop testing , etc
Implementation knowledge is not required	Implementation knowledge is required

MT - 58 - When to stop the testing ?

Testing is a never ending process , we can't stop the testing saying that we have tested every thing and the system is defect free. There is always a possibility for defects ,it may be masked or latent.
There are some industry constraints on which we decide that we should stop or hold the testing :

-> Test execution has been completed with a certain number of passed cases so that test coverage is achieved.

-> There are no known critical bugs.Bug rate falls below a certain level, now testers are not getting any priority 1, 2, or 3 bugs.

-> Coverage of code, functionality, or requirements reaches a specified point.

-> Stop the Testing when deadlines like release deadlines or testing deadlines is very near

-> When the time allocated to the test team is expired and client doesn't wants the date to be extended.

-> When the allocated budget comes to an end.

-> Stop the Testing when the period of beta testing / alpha testing gets over.

MT - 57 - How Incident Management works ?

Incident is a common name given to the term defect , bug , problem , fault or failure. Simply any deviation of the actual results from the expected results or any questionable behavior of the system is named as incident.

Some important things to consider about incidents :

- There are various causes of incidents like mis-configuration or failure of the test environment,corrupted test data, bad tests, invalid expected results and tester mistakes.

- Incident is about the questionable behavior of the system , all the incidents may not be a true defect. We make a record of all the incidents what we observed so that we can follow up on it and take necessary actions if required.

- Incidents can be logged along all the testing phases , development and reviews.

- Most commonly defects are reported against the code or the system itself , but defects can be reported against requirements and design specifications, user and operator guides and tests.

- Incident reports are used to provide programmers, managers and others with detailed information about the behavior observed and the defect. It is to support the analysis of trends in aggregate defect data, either for understanding more about a particular set of problems or tests or for understanding and reporting the overall level of system quality.

- While logging a Incident we can give as much  information as we can. Incidents may be defects or any mis-behave of the system so the details may help the management and development teams in finding out the issues.

- IEEE 829 - Incident Report Template contains :
Test incident report identifier Summary Incident description (inputs, expected results, actual results, anomalies, date and time, procedure step, environment, attempts to repeat, testers and observers) Impact

- Incident report life cycle :

Every Incident goes through a life cycle. All incident reports move through a series of clearly identified states after being reported. After an incident is reported, a peer tester or test manager reviews the report. If successful in the review, the incident report becomes opened, so now the project team must decide whether or not to repair the defect. If the defect is to be repaired, a programmer is assigned to repair it. Once the programmer believes the repairs are complete, the incident report returns to the tester for confirmation testing. If the confirmation test fails, the incident report is re-opened and then re-assigned. Once the tester confirms a good repair, the incident report is closed. No further work remains to be done. In any state other than rejected, deferred or closed, further work is required on the incident prior to the end of this project.  In a rejected, deferred or closed state, the incident report will not be assigned to an owner.

- Standard Incident Procedures :

->Identification - detect or report the incident

->Registration - the incident is registered

->Categorization - the incident is categorized by priority, severity ,SLA etc.

->Prioritization - the incident is prioritized for better utilization of the resources and the Support Staff time

->Diagnosis - reveal the full symptom of the incident in details

->Escalation - should the Support Staff need support from other organizational units

->Investigation and diagnosis - if no existing solution from the past could be found the incident is investigated and root cause found

->Resolution and recovery - once the solution is found the incident is resolved

->Incident closure - this is the last phase where Incidents are closed