Global information
| Timestamp | 2009/11/03 16:39:34 GMT | |||||||||||||||||||||||||
| Date and time at which the query was submitted. | ||||||||||||||||||||||||||
| Feature score table | terms.csv | |||||||||||||||||||||||||
| CSV spreadsheet detailing the calculation of the feature support scores. | ||||||||||||||||||||||||||
| Relevant PubMed IDs | positives.txt | |||||||||||||||||||||||||
| List of PubMed IDs of the relevant training examples. Dividing the file into 10 parts yields the cross validation folds. | ||||||||||||||||||||||||||
| Irrelevant PubMed IDs | negatives.txt | |||||||||||||||||||||||||
| List of PubMed IDs of the irrelevant examples (randomly sampled from Medline). Dividing the file into 10 parts yields the cross validation folds. | ||||||||||||||||||||||||||
| Feature score method | scores_laplace_split | |||||||||||||||||||||||||
| Name of the method used to calculate feature scores. Docstring for the method: For feature probabilities we use a Laplace prior, of 1 success and 1 failure in total, split between the classes according to size. This avoids problems with class skew. | ||||||||||||||||||||||||||
| Number of folds | 10 | |||||||||||||||||||||||||
| Number of partitions into which the relevant and irrelevant data sets were split. | ||||||||||||||||||||||||||
| Prior score | -5.31649133713 | |||||||||||||||||||||||||
| The log ratio of relevant to irrelevant articles in the cross validation data. This prior log ratio is added to log likelihood ratios to obtain posterior article scores. | ||||||||||||||||||||||||||
| Base score | -97.5934352128 | |||||||||||||||||||||||||
| The log likelihood ratio of an empty article (one in which every feature failed to occur). | ||||||||||||||||||||||||||
| Min Document Frequency | 0 | |||||||||||||||||||||||||
| Minimum Document Frequency. In each fold, we select features having at least this many occurrences in the training corpus. | ||||||||||||||||||||||||||
| Min Information Gain | 2e-05 | |||||||||||||||||||||||||
| Minimum Information Gain. In each fold, we select features having at least this relative information gain (information gain divided by entropy of original class variable. | ||||||||||||||||||||||||||
| Random Seed | None | |||||||||||||||||||||||||
| Random seed for shuffling the data. If None, the random seed is set using the system clock. | ||||||||||||||||||||||||||
| Score threshold | 0.461 | |||||||||||||||||||||||||
| If an article has a score greater than or equal to this value, classify it as relevant. The threshold is either the lowest one >= 0, or may be chosen to obtain break-even, maximum F measure, or maximum utility. | ||||||||||||||||||||||||||
| Average Precision | 0.97483 | |||||||||||||||||||||||||
| Precision averaged over all ranks where an article is retrieved. | ||||||||||||||||||||||||||
| Break-Even (precision=recall) | 0.924 | |||||||||||||||||||||||||
| Shared value at the point where Recall = Precision = F1-measure. Typically the F1-Measure at break-even is slightly lower than the maximum F1-Measure. | ||||||||||||||||||||||||||
| Area under ROC curve (AUC) | 0.99862 | |||||||||||||||||||||||||
| Area under the graph of the true positive rate versus false positive rate. Equals the probability that a randomly selected relevant article will be ranked above a randomly selected irrelevant article. | ||||||||||||||||||||||||||
| Standard Error of AUC | 0.00183 | |||||||||||||||||||||||||
| Standard error of the area under the ROC curve. Calculated using the method of Hanley (1982). | ||||||||||||||||||||||||||
| 11-point precision |
|
|||||||||||||||||||||||||
| Precision at recall equal to 0, 0.1, ... 1.0 | ||||||||||||||||||||||||||
Confusion Matrix at threshold
The columns of the confusion matrix are actual categories of the documents, and the rows are the predicted categories. Hover the mouse over each of the squares for a full description of the quantity, and the formula for calculating it.
| Actual | Totals | Rates | |||
|---|---|---|---|---|---|
| Relevant | Irrelevant | ||||
| Predicted | Relevant' | TP=488 | FP=227 | P'=715 | PPV=0.68 |
| Irrelevant' | FN=2 | TN=99773 | N'=99775 | NPV=0.99998 | |
| Totals | P=490 | N=100000 | 100490 | Prev=0.00488 | |
| Rates | TPR=1.00 | FPR=0.00227 | Acc=0.99772 | ||
Precision, Recall and F measure
| Precision (PPV) π=TP/(TP+FP) | 0.683 (0.598 to 0.766) | |
| Proportion of predicted positives which are true positives. | ||
| Recall (True Positive Rate / Sensitivity) | 0.996 (0.980 to 1.000) | |
| Proportion of positives which were correctly predicted to be positive. | ||
| F1-Measure (α=0.5) (2*ρ*π/(ρ+π)) | 0.810 (0.748 to 0.867) | |
| Harmonic mean of recall and precision at the threshold corresponding to the maximum α-weighted F-Measure. | ||
| F-Measure (α=0.5) (1/(α/π+(1-α)/ρ)) | 0.810 (0.748 to 0.867) | |
| The F measure evaluated using the given alpha. 0 <= α <= 1 controls the weight of precision. When α=0.5, F=F1. | ||
| Maximum possible F1-Measure | 0.929 | |
| This is the F_1 measure that would be achieved if we had set α=0.5 | ||
Utility
Utility is a weighted sum of True and False positives. A false positive has utility -1, and a true positive has utility ur, by default equal to N/P (the assumption being that returning all the articles should result in utility of zero).
Hence, U = (ur * TP - FP)/Umax where Umax = ur * P is the maximium achievable utility. If ur defaults to N/P this reduces to U=(TP/P)-(FP/N).
| Utility (ur=204.08) | 0.994 (0.977 to 0.999) |
| Maximum possible utility | 0.995 |
Miscellaneous Performance Measures
| Prevalence in cross validation P/(P+N) | 0.00488 | |
| Proportion of training data which was positive. | ||
| False Positive Rate (FPR) FPR=FP/(TN+FP)=1-TNR | 0.00227 (0.00150 to 0.00330) | |
| Proportion of negatives which were incorrectly predicted to be positive. | ||
| Specificity (TNR) TNR=TN/(TN+FP)=1-FPR | 0.99773 (0.99670 to 0.99850) | |
| Proportion of negatives which were correctly predicted to be negative. | ||
| Error Rate (FP+FN)/(P+N)=1-Accuracy | 0.00228 (0.00149 to 0.00328) | |
| Enrichment (= precision/prevalence) | 139.972 (122.549 to 157.016) | |
| Precision over prevalence. This is is how much better this classifiers precision is over a classifier which calls everything positive. | ||
Feature Statistics
| Quantity | Relevant Docs | Irrelevant Docs |
|---|---|---|
| Number of documents | 490 | 100000 |
| Number of selected, occurring features | 9914 | 34332 |
| Total occurrences of selected features | 58138 | 4899132 |
| Selected features per Medline record | 118.649 | 48.991 |
| Of the considered feature types, 36621 features are selected out of 444080 occurring at least once in training data. The aggressivity of selection is 12.126. The complete database lists 3703762 potential features. | ||
Performance graphs
Document score distributions
Normalised histograms (sum of bar areas normalised to 1), approximating probability distributions for relevant and irrelevant article scores. Good performance is associated with clean separation of the distributions.
Feature score distribution
Normalised histogram approximating the probability distribution for feature feature scores (after training on all available data).
ROC Curve
True Positive Rate versus False Positive Rate. The closer to the top left the curve gets, the better. Worst case is a diagonal line (true positives increasing at the same rate as false positives).
Precision-Recall Curve
Precision as a function of Recall. The recall corresponding to the chosen threshold is marked with a vertical line. Worst case is a horizontal line at the level of prevalence.
F measure versus threshold
Precision, Recall and F-measure as a function of threshold. The chosen threshold is marked with a vertical line.
