Skip To Content

Final Note on the NMRShiftDB Debate- 0.96 ppm

Perhaps some readers will believe that I have spent too many
posts on the NMRShiftDB debate. I would like to re-iterate that one of my main
purposes for creating this blog was to keep the public informed on the world of
NMR software. In this particular debate, it has been possible to inform users
of NMR software around the world with more information on three different NMR
software resources, The NMRShiftDB, ACD/CNMR Predictor, and Modgraph

So with that I would like to make one final posting on the
prediction validation of the NMRShiftDB. The previous presentation of results
may lead one to believe that Modgraph has the edge in prediction accuracy. 

I have continually asked Modgraph to clear up the overlap
issue that we have so heavily discussed here. Again, what is the amount of
overlap of identical chemical structures between the NMRShiftDB and Modgraph
NMRPredict database? Once that overlap is acknowledged and removed, what is the
accuracy of this software package on completely novel chemical shifts? 

As a co-author of the original validation of prediction
accuracy on the NMRShiftDB dataset
, I firmly believe that we have created a
best practice for prediction validation where we openly shared our structural
overlap and generated a prediction error on completely novel chemical shifts. 

Our core belief is that information that is provided to the
public should be shared with the best interest of quality science in mind. As a result
when we publish prediction accuracy numbers, we strive to produce numbers that
best reflect the performance of the prediction algorithms in a way that is
relevant to an end user. In our original study we produced an error of 1.59 ppm
for the entire NMRShiftDB. In order to produce a number that is a true
reflection of the prediction algorithms, we always exclude equal codes from the database of structures when doing an
accuracy assessment. In addition to this, we also clearly acknowledged that
there was a significant overlap between our prediction database and the
NMRShiftDB (57%). As a result we produced a number for a subset of the NMRShiftDB
(43%) that excluded all of this overlap (1.74 ppm). This is what we believe was
a true prediction validation on novel chemical shifts, and would be similar to
what a user could expect to get for novel structures and is therefore a
relevant number for users to know.

So if Modgraph is unwilling to share the important details
of this study regarding overlap with the public, we can only respond by providing the public with
a prediction accuracy value that can fairly be compared to theirs. 

As a result, we have produced a prediction accuracy result
that DOES NOT exclude identical HOSE codes. It is simply the result that we
have generated by downloading the SDF file of the NMRShiftDB and running it
through version 10 of ACD/CNMR Predictor, the same version that is available to
customers today. 

The result of this study is an average deviation of 0.96

That is an incredible result, no doubt. But understand that
it is not the most relevant result in practice. We have simply provided this
new number of 0.96 ppm in order to provide our users and followers a number
that they can accurately compare to the number of 1.40 ppm provided by Robien
using Modgraph’s NMRPredict. 

In our original validation document, we freely shared the
news that 57% of the structures in NMRShiftDB were also found in the ACD/Labs
CNMR prediction database. We believe that the most important result to an
end-user would be the performance of the predictor on the remaining 43% of the
structures that are not present in ACD/Labs CNMR prediction database, hence the
value of 1.74 ppm. 

I should also add that the major reason we conduct these
validation studies on independent datasets is to challenge our own technology.
We are always very eager to test our predictors on new data because it provides
us with an unbiased evaluation of the performance of our software. In addition,
many times, these validations help us improve our software. As a matter of
fact, during the evaluation of the NMRShiftDB we have identified areas within
our software in which prediction can be improved. We have already made changes
in preparation for version 11 and have achieved significantly better prediction
accuracy numbers than the 0.96 ppm I have presented today. I will however not
share these numbers to the public since these improvements will not be
available until the release of Version 11 in the fall. 

For now: 

An average deviation of 0.96 ppm on the entire NMRShiftDB
database (214,136 chemical shifts). 

An average deviation of 1.74 ppm on the subset of shifts in
the NMRShiftDB database that are not represented in ACD/CNMR Predictor (92,927
chemical shifts). 

If the overlap between the two databases is not
acknowledged, prediction accuracy results cannot be trusted. For this reason, I will not post any other
prediction accuracy results on this blog, either from ACD/Labs or any other
software company or research group, unless the overlap with the independent
dataset (NMRShiftDB) is clearly acknowledged.

However, in the end, I agree with Jeff Seymour from
Modgraph, Consultants Ltd. when he says: 

“Of course customers are really interested in how accurately
a prediction program can predict THEIR molecules – not a collection of external
data such as NMRShiftDB.”

Go ahead and evaluate the software packages on your own.  Contact me if you would like to conduct an evaluation of
ACD/CNMR Predictor.