Intensity determination by profile-fitting applied to precession photographs

A profile-fitting algorithm has been applied to the determination of X-ray diffraction intensities from precession photographs. It is assumed that each reflection on the film has the same profile or scaled intensity distribution, over a two-dimensional array of points about its center. The integrated intensity of a reflection is that scale factor which gives the best fit between a scaled model profile and the optical density measurements of the reflection. The systematic error in the calculated intensity was evaluated from the discrepancy between the model and observed profiles and was about 2% for strong reflections (about 1.5 OD units). Profile fitting reduced the likely random error in the intensity, produced by errors in optical density measurements, to half the value given by conventional integration. This gain is especially significant for weak reflections which form the bulk of protein data sets. A comparison between film data processed by this method and diffractometer data for the protein lactate dehydrogenase, (34000 Dalton in asymmetric unit), showed that film data agreed with diffractometer data as well as diffractometer data sets agreed among themselves. Film data for glyceraldehyde-3-phosphate dehydrogenase (144000 Dalton in asymmetric unit) had a reliability index of 6% and significant Bijvoet differences were measured for mercury derivatives of the protein. Methods for the correction for non-linear effects in film data are evaluated for model data. It is shown that these corrections give significant improvement only when the data satisfy two conditions: they extend over a wide range of optical densities (at least 20D units) and they are reliable (better than 4% reliability index). Under these conditions an extended version of the Hamilton, Rollett and Sparks scaling algorithm performs better than the other methods considered.