Global information
| Timestamp | 2008/02/11 15:11:37 GMT | |||||||||||||||||||||||||
| Date and time at which the query was submitted. | ||||||||||||||||||||||||||
| Feature score table | terms.csv | |||||||||||||||||||||||||
| CSV spreadsheet detailing the calculation of the feature support scores. | ||||||||||||||||||||||||||
| Relevant PubMed IDs | positives.txt | |||||||||||||||||||||||||
| List of PubMed IDs of the relevant training examples. Dividing the file into 10 parts yields the cross validation folds. | ||||||||||||||||||||||||||
| Irrelevant PubMed IDs | negatives.txt | |||||||||||||||||||||||||
| List of PubMed IDs of the irrelevant examples (randomly sampled from Medline). Dividing the file into 10 parts yields the cross validation folds. | ||||||||||||||||||||||||||
| Feature score method | scores_bgfreq | |||||||||||||||||||||||||
| Name of the method used to calculate feature scores. Docstring for the method: The prior is 'background frequency' successes, out of 1 total occurrence in each class. | ||||||||||||||||||||||||||
| Number of folds | 10 | |||||||||||||||||||||||||
| Number of partitions into which the relevant and irrelevant data sets were split. | ||||||||||||||||||||||||||
| Prior score | -7.30823284558 | |||||||||||||||||||||||||
| The log ratio of relevant to irrelevant articles in the cross validation data. This prior log ratio is added to log likelihood ratios to obtain posterior article scores. | ||||||||||||||||||||||||||
| Base score | -7.20029168724 | |||||||||||||||||||||||||
| The log likelihood ratio of an empty article (one in which every feature failed to occur). | ||||||||||||||||||||||||||
| Min Document Frequency | 1 | |||||||||||||||||||||||||
| Minimum Document Frequency. In each fold, we exclude features occurring fewer than this many times in the data | ||||||||||||||||||||||||||
| Min Information Gain | 0 | |||||||||||||||||||||||||
| Minimum Information Gain. In each fold, we exclude features with less than this information gain. | ||||||||||||||||||||||||||
| Score threshold | 0.087 | |||||||||||||||||||||||||
| If an article has a score greater than or equal to this value, classify it as relevant. The threshold is either the lowest one >= 0, or may be chosen to obtain break-even, maximum F measure, or maximum utility. | ||||||||||||||||||||||||||
| Average Precision | 0.71690 | |||||||||||||||||||||||||
| Precision averaged over all ranks where an article is retrieved. | ||||||||||||||||||||||||||
| Break-Even (precision=recall) | 0.642 | |||||||||||||||||||||||||
| Shared value at the point where Recall = Precision = F1-measure. Typically the F1-Measure at break-even is slightly lower than the maximum F1-Measure. | ||||||||||||||||||||||||||
| Area under ROC curve (AUC) | 0.99387 | |||||||||||||||||||||||||
| Area under the graph of the true positive rate versus false positive rate. Equals the probability that a randomly selected relevant article will be ranked above a randomly selected irrelevant article. | ||||||||||||||||||||||||||
| Standard Error of AUC | 0.00504 | |||||||||||||||||||||||||
| Standard error of the area under the ROC curve. Calculated using the method of Hanley (1982). | ||||||||||||||||||||||||||
| 11-point precision |
|
|||||||||||||||||||||||||
| Precision at recall equal to 0, 0.1, ... 1.0 | ||||||||||||||||||||||||||
Confusion Matrix at threshold
The columns of the confusion matrix are actual categories of the documents, and the rows are the predicted categories. Hover the mouse over each of the squares for a full description of the quantity, and the formula for calculating it.
| Actual | Totals | Rates | |||
|---|---|---|---|---|---|
| Relevant | Irrelevant | ||||
| Predicted | Relevant' | TP=60 | FP=141 | P'=201 | PPV=0.30 |
| Irrelevant' | FN=7 | TN=99859 | N'=99866 | NPV=0.99993 | |
| Totals | P=67 | N=100000 | 100067 | Prev=0.00067 | |
| Rates | TPR=0.90 | FPR=0.00141 | Acc=0.99852 | ||
Precision, Recall and F measure
| Precision (PPV) π=TP/(TP+FP) | 0.299 (0.231 to 0.375) | |
| Proportion of predicted positives which are true positives. | ||
| Recall (True Positive Rate / Sensitivity) | 0.896 (0.714 to 1.000) | |
| Proportion of positives which were correctly predicted to be positive. | ||
| F1-Measure (α=0.5) (2*ρ*π/(ρ+π)) | 0.448 (0.364 to 0.538) | |
| Harmonic mean of recall and precision at the threshold corresponding to the maximum α-weighted F-Measure. | ||
| F-Measure (α=0.5) (1/(α/π+(1-α)/ρ)) | 0.448 (0.364 to 0.538) | |
| The F measure evaluated using the given alpha. 0 <= α <= 1 controls the weight of precision. When α=0.5, F=F1. | ||
| Maximum possible F1-Measure | 0.715 | |
| This is the F_1 measure that would be achieved if we had set α=0.5 | ||
Utility
Utility is a weighted sum of True and False positives. A false positive has utility -1, and a true positive has utility ur, by default equal to N/P (the assumption being that returning all the articles should result in utility of zero).
Hence, U = (ur * TP - FP)/Umax where Umax = ur * P is the maximium achievable utility. If ur defaults to N/P this reduces to U=(TP/P)-(FP/N).
| Utility (ur=1492.54) | 0.894 (0.713 to 0.999) |
| Maximum possible utility | 0.960 |
Miscellaneous Performance Measures
| Prevalence in cross validation P/(P+N) | 0.00067 | |
| Proportion of training data which was positive. | ||
| False Positive Rate (FPR) FPR=FP/(TN+FP)=1-TNR | 0.00141 (0.00100 to 0.00200) | |
| Proportion of negatives which were incorrectly predicted to be positive. | ||
| Specificity (TNR) TNR=TN/(TN+FP)=1-FPR | 0.99859 (0.99800 to 0.99900) | |
| Proportion of negatives which were correctly predicted to be negative. | ||
| Error Rate (FP+FN)/(P+N)=1-Accuracy | 0.00148 (0.00110 to 0.00210) | |
| Enrichment (= precision/prevalence) | 445.832 (329.901 to 555.889) | |
| Precision over prevalence. This is is how much better this classifiers precision is over a classifier which calls everything positive. | ||
Feature Statistics
| Quantity | Relevant Docs | Irrelevant Docs |
|---|---|---|
| Number of documents | 67 | 100000 |
| Number of distinct features | 261 | 28156 |
| Total feature occurrences | 1053 | 1347794 |
| Terms per document | 15.716 | 13.478 |
| In total, 28160 features are in use, and 14003 features are ignored. | ||
Performance graphs
Document score distributions
Normalised histograms (sum of bar areas normalised to 1), approximating probability distributions for relevant and irrelevant article scores. Good performance is associated with clean separation of the distributions.
Feature score distribution
Normalised histogram approximating the probability distribution for feature feature scores (after training on all available data).
ROC Curve
True Positive Rate versus False Positive Rate. The closer to the top left the curve gets, the better. Worst case is a diagonal line (true positives increasing at the same rate as false positives).
Precision-Recall Curve
Precision as a function of Recall. The recall corresponding to the chosen threshold is marked with a vertical line. Worst case is a horizontal line at the level of prevalence.
F measure versus threshold
Precision, Recall and F-measure as a function of threshold. The chosen threshold is marked with a vertical line.
