First, the analyst should establish that there is indeed attribute data. It can be assumed that assigning a code – that is, classifying a code into a category – is a decision that characterizes the error by an attribute. Either a category is correctly assigned to a defect or it is not. Similarly, the defect is either attributed to the right source or not. These are “yes” or “no” and “good assignment” or “wrong assignment” answers. This part is quite simple. Despite these difficulties, performing an attribute agreement analysis on bug tracking systems is not a waste of time. In fact, it is (or can be) an extremely informative, valuable and necessary exercise. The analysis of the award agreement must be applied with caution and a certain focus. ISO/TR 14468:2010 evaluates a measurement process in which measured characteristics are available as attribute data (including nominal and ordinal data). Of course, this can be avoided a bit by increasing the sample size and, better yet, wait a while before the scenarios are made available to reviewers a second time (maybe one to two weeks.
. . .