July 12, 2007
by Ryan Sasaki, NMR Product Manager, ACD/Labs
Perhaps some readers will believe that I have spent too many posts on the NMRShiftDB debate. I would like to re-iterate that one of my main purposes for creating this blog was to keep the public informed on the world of NMR software. In this particular debate, it has been possible to inform users of NMR software around the world with more information on three different NMR software resources, The NMRShiftDB, ACD/CNMR Predictor, and Modgraph NMRPredict.
So with that I would like to make one final posting on the prediction validation of the NMRShiftDB. The previous presentation of results may lead one to believe that Modgraph has the edge in prediction accuracy.
I have continually asked Modgraph to clear up the overlap issue that we have so heavily discussed here. Again, what is the amount of overlap of identical chemical structures between the NMRShiftDB and Modgraph NMRPredict database? Once that overlap is acknowledged and removed, what is the accuracy of this software package on completely novel chemical shifts?
As a co-author of the original validation of prediction accuracy on the NMRShiftDB dataset, I firmly believe that we have created a best practice for prediction validation where we openly shared our structural overlap and generated a prediction error on completely novel chemical shifts.
Our core belief is that information that is provided to the public should be shared with the best interest of quality science in mind. As a result when we publish prediction accuracy numbers, we strive to produce numbers that best reflect the performance of the prediction algorithms in a way that is relevant to an end user. In our original study we produced an error of 1.59 ppm for the entire NMRShiftDB. In order to produce a number that is a true reflection of the prediction algorithms, we always exclude equal codes from the database of structures when doing an accuracy assessment. In addition to this, we also clearly acknowledged that there was a significant overlap between our prediction database and the NMRShiftDB (57%). As a result we produced a number for a subset of the NMRShiftDB (43%) that excluded all of this overlap (1.74 ppm). This is what we believe was a true prediction validation on novel chemical shifts, and would be similar to what a user could expect to get for novel structures and is therefore a relevant number for users to know.
So if Modgraph is unwilling to share the important details of this study regarding overlap with the public, we can only respond by providing the public with a prediction accuracy value that can fairly be compared to theirs.
As a result, we have produced a prediction accuracy result that DOES NOT exclude identical HOSE codes. It is simply the result that we have generated by downloading the SDF file of the NMRShiftDB and running it through version 10 of ACD/CNMR Predictor, the same version that is available to customers today.
The result of this study is an average deviation of 0.96 ppm.
That is an incredible result, no doubt. But understand that it is not the most relevant result in practice. We have simply provided this new number of 0.96 ppm in order to provide our users and followers a number that they can accurately compare to the number of 1.40 ppm provided by Robien using Modgraph’s NMRPredict.
In our original validation document, we freely shared the news that 57% of the structures in NMRShiftDB were also found in the ACD/Labs CNMR prediction database. We believe that the most important result to an end-user would be the performance of the predictor on the remaining 43% of the structures that are not present in ACD/Labs CNMR prediction database, hence the value of 1.74 ppm.
I should also add that the major reason we conduct these validation studies on independent datasets is to challenge our own technology. We are always very eager to test our predictors on new data because it provides us with an unbiased evaluation of the performance of our software. In addition, many times, these validations help us improve our software. As a matter of fact, during the evaluation of the NMRShiftDB we have identified areas within our software in which prediction can be improved. We have already made changes in preparation for version 11 and have achieved significantly better prediction accuracy numbers than the 0.96 ppm I have presented today. I will however not share these numbers to the public since these improvements will not be available until the release of Version 11 in the fall.
An average deviation of 0.96 ppm on the entire NMRShiftDB database (214,136 chemical shifts).
An average deviation of 1.74 ppm on the subset of shifts in the NMRShiftDB database that are not represented in ACD/CNMR Predictor (92,927 chemical shifts).
If the overlap between the two databases is not acknowledged, prediction accuracy results cannot be trusted. For this reason, I will not post any other prediction accuracy results on this blog, either from ACD/Labs or any other software company or research group, unless the overlap with the independent dataset (NMRShiftDB) is clearly acknowledged.
However, in the end, I agree with Jeff Seymour from Modgraph, Consultants Ltd. when he says:
“Of course customers are really interested in how accurately a prediction program can predict THEIR molecules – not a collection of external data such as NMRShiftDB.”
Go ahead and evaluate the software packages on your own.