Methods and apparatus are described for evaluating a
binary classification system operable to classify each of a plurality of events as a first
event type or a second
event type. At least some of the events of the first
event type are independently verifiable with reference to
verification data. The
binary classification system is susceptible to a first error type in which events of the first event type are classified as the second event type, and a second error type in which events of the second event type are classified as the first event type. Operation of a first configuration of the
binary classification system is evaluated with reference to an objective function. The objective function is derived by expressing a number of errors of the second error type in terms of a number of errors of the first error type with reference to the
verification data, and by assuming relative proportions of the first and second event types within the plurality of events.