Falsifification Assessment Theory
Falsifiability increases empirical knowledge – an example in search assessment
Swets’ ROC analysis was developed and validated for signal detection given a known location. Clinically relevant tasks are often more complicated, e.g., search, multi-class classification, estimation, etc. Many extended ROC methods are proposed for each task, and there is a sincere lack of consensus concerning which extended ROC method is more appropriate. Before discussing how the lack of falsifiability contributes to this lack of consensus, in this section, I present a falsifiable search assessment method.
Just as Swets’ ROC, a falsifiable assessment theory is aimed at understanding “reality”. Thus, in addition to preserving at least some empirical content of Swets’ ROC, e.g., prevalence-independence, a search assessment …show more content…
1. The generation of the core hypothesis is known to be an irrational process 2,14,28. The readers may refer to He et. al. for a set of evidence that leads to the following core hypothesis27. However, there is no logical path in reaching the core hypotheses. The ESS theory starts with the core …show more content…
Because M* is constant across a range of scenarios, it should allow predictions. For example, given M* and LKE ROC measured from an observer, one should be able to predict the same observer’s absolute performance in different search tasks where LKE ROC and M* are claimed constant, including 1) single-response search tasks with one or multiple signals in the signal-present images 2) search tasks with a larger or smaller search size, and 3) single-response or free-response tasks,
Swets’ ROC analysis was developed and validated for signal detection given a known location. Clinically relevant tasks are often more complicated, e.g., search, multi-class classification, estimation, etc. Many extended ROC methods are proposed for each task, and there is a sincere lack of consensus concerning which extended ROC method is more appropriate. Before discussing how the lack of falsifiability contributes to this lack of consensus, in this section, I present a falsifiable search assessment method.
Just as Swets’ ROC, a falsifiable assessment theory is aimed at understanding “reality”. Thus, in addition to preserving at least some empirical content of Swets’ ROC, e.g., prevalence-independence, a search assessment …show more content…
1. The generation of the core hypothesis is known to be an irrational process 2,14,28. The readers may refer to He et. al. for a set of evidence that leads to the following core hypothesis27. However, there is no logical path in reaching the core hypotheses. The ESS theory starts with the core …show more content…
Because M* is constant across a range of scenarios, it should allow predictions. For example, given M* and LKE ROC measured from an observer, one should be able to predict the same observer’s absolute performance in different search tasks where LKE ROC and M* are claimed constant, including 1) single-response search tasks with one or multiple signals in the signal-present images 2) search tasks with a larger or smaller search size, and 3) single-response or free-response tasks,