1. Introduction

Recent decades have seen an astonishing improvement in terms of synchrotron performance, beamline instrumentation and data-processing software. Protein crystallography is now a mainstream research technique and the rate of structure deposition in the Protein Data Bank keeps increasing (Abad-Zapatero, 2012). Nevertheless, despite much automation, data collection remains a nontrivial experiment that requires diligent planning and careful checks, while the brightness and brilliance of third-generation synchrotrons pose a whole additional set of challenges (Chavas et al., 2012; Dauter et al., 2010; Fodje et al., 2012; Gabadinho et al., 2010; González et al., 2008; McCarthy et al., 2009). The availability of ever smaller and brighter beams has boosted the field on the one hand, but it has also increased the requirement of the experimenter to make the correct choices at the beamline. The experiment potentially remains tricky and still requires experience and quick, accurate decisions.

The validity of this truism can be well observed in large research laboratories dedicated to structure solution, such as the Structural Genomics Consortium (SGC). The Oxford site of this international initiative currently deposits three (historically six) novel human protein crystal structures per month, and its productivity depends crucially on frequent visits to the state-of-the-art beamlines at the Diamond Light Source and, before 2009, to the Swiss Light Source. In terms of operational efficiency, while the cornerstone of the whole gene-to-structure process is to work in parallel, we have consistently found that if diffracting crystals have been identified then the most effective route to reducing additional biochemical effort is to approach data collection with extreme diligence in order to obtain the best data possible.

Thus, the output of a structural genomics project is not sustained by indiscriminately collecting vast amounts of data and hoping that their sheer number will eventually lead to the desired number of structures, contrary to the popular sense invoked by the term `high-throughput' that is often used in the same context. As is frequently pointed out, the collection of diffraction data is the last real experiment and errors made at this stage may make structure solution unnecessarily difficult or even impossible. Indeed, by focusing on collecting few but high-quality data sets, we found that not only is the data analysis itself accelerated, but also the average amount of wet laboratory work required per project is greatly reduced; the time consumed by attempting to evaluate large quantities of mediocre data can never be overestimated.

Here, we provide an overview of the methods and routines that we have applied and refined over eight years of data collection at both the Diamond Light Source and the Swiss Light Source. This topic has been discussed extensively in the literature (Dauter, 1999, 2010; Evans, Axford, Waterman et al., 2011; Pflugrath, 2004; Rupp, 2010; Mueller et al., 2007) and much of this will be obvious to experienced synchrotron users; certainly not everything is crucial to every situation, but we have consistently found that shortcuts taken by impatient operators have come back to haunt them and their projects.

2. Homework

By far the most reliable route to good data is to have a good crystal, so effort spent in finding one before coming to the synchrotron generally pays off. Crystal morphology is not a predictor of diffraction quality, and one should therefore always explore all crystals grown in different crystallization conditions. Moreover, there are many ways to generate variant crystals, e.g. crystallization in the presence of different ligands (Vedadi et al., 2006), crystallization of different constructs (Gileadi et al., 2007), seeding (D'Arcy et al., 2007), surface mutagenesis (Derewenda, 2010), lysine methylation (Walter et al., 2006), diversified screening approaches such as the `silver bullets' screen (McPherson & Cudney, 2006) and in situ proteolysis (Wernimont & Edwards, 2009). Post-crystallization procedures such as dehydration or annealing can in some cases lead to impressive improvements in diffraction quality (Heras & Martin, 2005; Sanchez-Weatherby et al., 2009). Crystals should be mounted with great care and adhering to some simple rules will always be beneficial for the quality of a data set without any additional overheads (Pflugrath, 2004). A few points are worth emphasizing.

Modern synchrotrons, in combination with sample-changing robotics, allow the rapid screening of many crystals at the beamline, but at SGC the policy remains that all crystals have to be first characterized for diffraction quality (although not necessarily space group) on our in-house rotating-anode generator to allow them to be prioritized for the synchrotron; unscreened crystals generally receive low priority (Figs. 1a and 1b). We have found this to be crucial as it allows us to focus our efforts and attention during the limited period of a beamtime on the actual collection of data. State-of-the-art rotating-anode generators often allow a realistic assessment of the quality of protein crystals: the diffraction observed for crystals larger than 100 µm on the Rigaku FR-E generator at the SGC (300 µm beam size) is typically between 0.5 and 2 Å poorer in resolution than that observed at the synchrotron, and spectacular improvements are only observed for crystals smaller than 50 µm (Figs. 1c and 1d). Additionally, pre-screening helps to exclude salt crystals, to identify issues with freezing or to identify crystal problems early on. Samples are prioritized taking account of both project importance and in-house screening results, so generally it is clear what kind of data-collection approach will be required for each project and how much time it will require even before the synchrotron visit starts. This procedure ensures that the time at the synchrotron is used as efficiently as possible.

Figure 1 Efficiencies achieved by careful pre-visit preparation. (a) The distribution of diffraction experiments performed by the SGC throughout 2011 illustrates the value of testing crystals in-house: three quarters of crystals could be discarded as inferior, allowing beamtime to be dedicated to the collection of high-quality data sets, so that half led to deposited structures. (b) A well structured list of priorities such as this example from our laboratory, prepared and distributed in advance of the visit, greatly speeds up experiments by enabling decisions to be made rapidly. (c) A comparison of the best resolutions observed for a series of projects with various beams (see legend) illustrates that diffraction can generally be reliably identified from less intense beams and that improvements may actually be quite modest. (d) The very different improvements in resolution observed for crystals from two different projects using a trimmed and microfocused beam (c) can be rationalized by the ratio of the crystal size to the respective beam sizes: if the crystal is already much larger than the beam (ABL2) an even smaller beam will not help much.

3. Mechanical alignment of the beam with the spindle axis

Modern beamlines are remarkable machines, and we have found beamline support and maintenance to be consistently excellent worldwide, not only at our usual `home' synchrotrons. Nevertheless, `drift' is an unavoidable phenomenon for any system where movements of 10 µm or less are important, and since these are the typical experimental dimensions at modern synchrotrons, both the beam and the rotation axis can easily drift from the intersection point marked on the screen. It is therefore crucial that the user understands the beamline and takes responsibility for being alert to whether the beam is in fact hitting the crystal at all times and knowing how to diagnose and correct this both before and during the experiment. Even for the Diamond beamlines, whose stability we are very familiar with, we set aside time at the start of each visit for a start-up routine to check some critical features together with the beamline scientists; if nothing else, this reminds us as users how to control the beamline and diagnose it once we are on our own during the rest of the visit.

On-axis viewing systems (Perrakis et al., 1999) offer the extraordinary benefit that users are able to directly visualize the crystal-to-beam intersection. The beam position can easily be checked by mounting an X-ray sensitive screen to verify that the beam is actually at the position of the crystal, and many beamlines have motorized YAG screens that can be rapidly placed in the beam, even while the sample is on the goniometer. What should also be investigated is how different slit or aperture settings affect both the shape and the position of the beam; even if the viewing software draws outlines of the apparent beam, these will necessarily only be approximations of the actual beam profile. It is advisable to repeatedly review the position of the beam over the course of a visit, especially when working with small samples; when beams drift, this happens on the timescale of hours, and if the beam position is not regularly checked this can easily lead to false-negative results. It is also important to be familiar with how to correct for such drifts when they occur.

Equally important is to assess the sphere of confusion: the alignment of the rotation axis and its rotational accuracy. A large sphere of confusion will cause the sample to `wobble' around the beam, and assessing it requires a well defined point mounted on the goniometer; typically, beamlines have sharp needles mounted on standard bases available and we have also found it useful to carry our own. On a well aligned rotation axis, the tip of the object (needle) should stay in the marked beam centre for an entire 360° oscillation, but mechanical limitations or a slight offset of the axis may prevent this. The latter is easily corrected by the beamline controls, but the former is a feature of the instrument and this rotation-dependent deviation may need to be taken into account when planning a particular diffraction experiment, especially for very small crystals and experiments that require a full 360° rotation (§4).

4. Alignment of sample to beam

The most fundamental aspect of the experiment is that the beam must properly intersect with the sample for the entire oscillation. This is not a problem when the crystals are much larger than the beam, but with small crystals and small beams it is easy to get wrong. When aligning visually, the main obstacle to aligning the sample is parallax owing to the shape of the liquid surrounding the crystal in the loop; the only reliable view of the crystal is perpendicular to the loop, and identifying this orientation should be the first step during the alignment of each sample. The quickest way to do so is to find the `side-on' direction, since it is most readily distinguished; from there, a 90° oscillation brings up the `face-on' view. Any alignment should be confirmed by viewing the loop from the other side, i.e. by applying a 180° rotation.

However, the side-on orientation presents a fundamental problem whenever the crystal does not extend between both surfaces of the liquid in the loop or when it is thinner than the thickness of the loop material itself. In this case, diffraction must be used to locate the crystal by scanning X-rays across a region of interest and measuring the diffraction strength (Aishima et al., 2010; Hilgart et al., 2011). Low doses are recommended for such scans in order to avoid unnecessary X-­ray damage, since only a small fraction of the photons required for an image in a properly measured data set are sufficient to identify the strongest diffraction. While automated analyses are helpful when available, we recommend visual inspection of the diffraction pattern, especially for low-dose exposures. This is easily performed by opening an image in a diffraction-image viewer, selecting an obvious Bragg spot, zooming in to view the actual pixel counts and then finding the image with the highest number of counts in the spot.

Grid scans also offer the opportunity to characterize diffraction throughout the crystal; in our experience, crystals very commonly diffract inhomogeneously, especially those that are poorly ordered and thus diffract to low resolution, regardless of their morphological appearance. Crystals of membrane proteins appear to be particularly prone to this phenomenon (Evans, Axford & Owen, 2011), and even when large crystals can be generated the use of the microfocus beam of I24 remains informative.

Visual alignment encounters two other common obstacles. (i) Loops that are significantly bent away from the rotation axis will present further parallax, and to view these reliably may require manual intervention to straighten the loop or else the use of kappa goniometry where available (see §6.4). (ii) Surface ice obscures the crystal, but also degrades the diffraction pattern with strong powder diffraction rings. We have been able to remove it routinely by blowing it away with a stream of liquid nitrogen using a Cry-Ac cryosurgery device (Brymill, Connecticut, USA; Fig. 2; James Holton, personal communication). Be aware, though, that the cold gas stream may damage the beamline equipment and the use of such a device should be discussed with the beamline scientists beforehand. Crystal `washers', which flood the crystal with a stream of liquid nitrogen on demand, are also becoming more widespread at beamlines (Diamond Light Source, personal communication).

Figure 2 Removal of surface ice with a jet of liquid nitrogen from a BryMill cryosurgery device. The process is efficient and safe, provided that it is performed correctly. (a) In order not to blow away the cold air stream the jet should be at an acute angle to it, and it should not be pointed directly at the crystal but instead swept towards it across the cold air stream. There must be sufficient pressure so the jet does not splutter and large loops should be turned so their edge faces the jet, because the sample drop does not stick well to the loop. (b) A nozzle with a non-uniform edge will produce a divergent jet which may blow the cold stream away and lead to destruction of the sample; a quick remedy is to drill out one of the inserts supplied. (c) Image of the loop before and after deicing.

Use of very small crystals on beamlines with a sphere of confusion of similar magnitude to either the crystal or the beam requires special attention: either angular ranges with the largest offset need to be avoided or else several re-alignments may need to be performed in the course of a 360° data set either side of a particular oscillation range (we use scans of 90°). Care also has to be taken when setting up helical data collection (Evans, Axford, Waterman et al., 2011; Flot et al., 2010; see §6.5): for such scans on thin rods on a microfocus beamline the end points need to be determined with the utmost care and diffraction-based alignment (Flot et al., 2010; Hilgart et al., 2011; Song et al., 2007) is strictly required, since even small parallax may be sufficient to offset the apparent position of the crystal.

Another source of misalignment is more prosaic, namely defective sample mounts, where either the pin is loose in the base or the loop itself is loose in the pin; certainly this is worth investigating before instigating a more wide-ranging diagnosis of potential beamline failures. Incidence of this problem can be avoided by diligent pin management in the home laboratory, but for the inevitable exceptions, if a particularly valuable sample is involved, creative application of Plasticine may temporarily alleviate the problem.

5. Documentation of experiments

The need for rigorous documentation of diffraction experiments is easily overlooked during preparation for synchrotrons, yet it is of the utmost importance, since when it is neglected inconsistent practices will lead over time to a set of widely divergent notes that are difficult to search and are usually significantly incomplete. We have found it invaluable to develop a single documentation convention that has to be followed by every group member. Mostly, this merely requires only a little thought from an experienced synchrotron user in the group, and if the conventions are properly disseminated people are perfectly willing to adopt them.

The most basic aspect concerns files and directories, which can be judiciously named to be self-documenting. The use of arbitrary names is common (`test1', `crystal3', `thing'), but this practice should be strongly discouraged since such naming will at best be informative to one individual, but will mostly mean the experiment cannot be reconstructed by anybody, including the creator after a few weeks. At SGC Oxford we adopted a naming convention that maps onto that of the CCP4 program suite (Winn et al., 2011): each harvested crystal is given a unique crystal ID from a given project (e.g. BBOX1-x023 for project BBOX1) and each data set is given a unique data-set ID (BBOX1-d005), where a data set comprises one or more passes collected consecutively in time. These are combined for image names: BBOX1-d001-x003, BBOX1-d002-x047 etc. All of the data files are stored in a uniformly organized file system, which it is advisable to keep separate from where processing occurs: this greatly facilitates the management of disc space, since it makes it easier to locate the far larger data directories for deletion (after backup). Such practice is not encouraged by the default behaviour of the common data-integration packages, yet is nevertheless easy to perform in all of them.

Rigorous use of database infrastructure, where available, provides even greater benefits. SGC Oxford, which comprises around 80 researchers from diverse backgrounds ranging from cell biology, (bio)chemistry, informatics, engineering to crystallography, has benefited enormously from a central database (BeeHive; Molsoft LLC, La Jolla, California, USA) and an electronic laboratory journal (Contur ELN; Contur Software AB, Sweden). Every step of a project from target selection to structure deposition can be rapidly reconstructed through the query engine, because this has been the only formal experimental record accepted in the laboratory from the start. The consistent utility that this infrastructure has provided, not least for real-time decision-making, prompts us to submit that such systems should be a strict requirement in any large operation, whether industry or academic efforts such as the SGC; even only a subset of functions would provide enormous benefit to smaller research groups. Freely available database systems are available, including a publicly hosted server of PIMS (Morris et al., 2011), and many synchrotrons offer database support systems such as ISPyB, which record all details of a diffraction experiment (Delagenière et al., 2011). However, the usefulness of such a system stands and falls with the willingness of the head of the laboratory to enforce both its use and the consistency of the corresponding naming conventions.

6. Data collection

Even when the crystals have been pre-screened in-house, it is very important to reassess at least several of the best crystals before investing the time to set up and execute data collection. Modern sample changers are extremely reliable and we cannot confirm anecdotal reports that dismounting crystals can lead to loss of diffraction: in more than 650 diffracting projects, crystals that diffracted before being returned to liquid nitrogen, invariably still diffracted just as well when replaced on the goniometer. Indeed, rather than introducing risks, a willingness to remove cryopreserved crystals from the gonio­meter once tested, allows synchrotron time to be used much more efficiently.

Another potential pitfall is presented by the revolutionary speed of modern detectors, especially pixel-array detectors, which makes it possible to collect many more data sets in the allotted beamtime than in the past. This capability is a powerful tool, but we also observe that, at least for unphased structures, meticulous planning of the diffraction experiment cannot be dispensed with and cannot be compensated for by `data-set roulette' i.e. the collection of data as quickly as possible from as many crystals as possible in the hope that one is good enough. Moreover, data processing can become very time-consuming or even impossible when crystals are suboptimal, suffer from radiation damage or are poorly aligned, and ultimately no time is saved.

Finally, it is important to become closely familiar with the necessary software, experimental requirements and strategy options before arriving at the synchrotron, because once there there is always great time pressure, and the ability to generate and understand real-time diagnostics often makes the difference between success and failure. Historical data sets from other projects in the laboratory are an excellent resource for learning the software.

6.1. Initial characterization

Once a sample is properly aligned we record two diffraction images that are 90° apart, since frequently diffraction appears to be good in one crystal orientation but is problematic in the perpendicular direction. For this reason, careful visual inspection of diffraction is important using an image viewer such as ADXV (https://www.scripps.edu/~arvai/adxv.html ) or ALBULA (https://www.dectris.com/software_albula.html ) to identify diffraction pathologies such as split spots, multiple lattices, smeary spots, ice rings or anisotropic diffraction.

Although beamline staff can advise on typical exposure times and transmission settings for their beamline, this will not necessarily allow adequate characterization of a given crystal and, especially when diffraction is weak, higher fluxes should be explored to see whether reflections start to appear at higher resolution or whether there is just an increase in background intensity. For a semi-quantitive assessment, the program iMosflm (Battye et al., 2011) provides the invaluable facility of a graphical view of I/σ(I) per resolution bin when integrating even only a single image (Fig. 3), provided that the program succeeds in indexing the diffraction. This exercise has the added advantage of allowing a visual comparison between the predicted and observed diffraction, which is easy to overlook when relying on nongraphical (scripted) integration.

Figure 3 Evaluation of diffraction resolution using single-image integration in iMosflm: the histogram shows I/σ(I) for each resolution bin, which increases correspondingly when the flux is increased (compare the upper and lower panels). In the absence of decay, the resolution where I/σ(I) ≃ 1 on a single image will generally yield I/σ(I) ≃ 2 when the whole data set is merged.

If attempts at indexing fail, it may be better to search for a better crystal; on the other hand, if there are no better crystals it is certainly worth proceeding with data collection, since frequently data processing does then succeed when all images of the data set are available.

6.4. Overlap strategy: oscillation angle

Another important consideration is the oscillation angle per image, in order to avoid spatial overlap of diffraction spots on the detector. The availability of pixel area detectors makes it feasible to use fine slicing (Pflugrath, 1999) by default, and apart from the potential gains in data quality and signal to noise (Mueller et al., 2012), this also greatly reduces the risk of spot overlap.

However, even fine slicing will not help when a sufficiently long unit-cell axis is oriented perpendicular to the beam: in such cases the geometry ensures that there will be spatial overlap even for still images, with no guarantee that any algorithm can deconvolute the overlapped intensities reliably. This particular problem is easy to spot with an interactive program such as iMosflm, as it displays the percentage of overlaps for a given oscillation angle as long as the mosaicity has been properly estimated.

This long-axis problem can only be circumvented by re­orienting the crystal: this is straightforward with a goniometer that supports κ offsets (e.g. a kappa motor), although many modern beamlines do not have this facility. Alternatively, if one has previous knowledge of how the crystal morphology is related to the unit-cell axes, one can try to mount the crystal in a favourable orientation (§2); some manufacturers (e.g. Hampton Research, California, USA) sell special bendable loops for this purpose. However, in the absence of all of this (and with a bit of practice) it is very feasible to bend nylon loops while they are mounted on the goniometer (Fig. 4; Dauter, 1999). This naturally cannot achieve any angular accuracy, yet the procedure has proved itself to be adequate for a significant number of projects in which reorientation was crucial to success but where kappa goniometry was not available. Non-nylon sample mounts are much less amenable to this procedure.

Figure 4 Crystal reorientation to avoid spot overlap owing to a long unit-cell axis oriented close to the beam: after the long axis (in this example c*, oriented in the plane of the loop) has been rotated into the plane perpendicular to the beam, the cell axis can be brought more parallel to the spindle axis by bending the loop in that plane. (a) Schematic representation of the goniometer showing the crystal in an arbitrary orientation, along with the laboratory reference frame conventions used by iMosflm. (b) Diffraction pattern and crystal view (upper inset) from the arbitrary crystal orientation, showing the close spacing of predictions characteristic of a long axis oriented close to the beam; strategy analysis in iMosflm indicates that data collection will be compromised by overlaps (not shown); it also provides (lower inset) the rotation required to bring the long axis into the iMosflm reference plane XZ (blue highlight). (c) Schematic showing the crystal rotated to place the long axis into the desired YZ plane: since it is perpendicular to the iMosflm reference plane, 90° is added to the angle indicated in (b). (d) Diffraction pattern of the rotated crystal, with the orientation of the long c* axis now clearly identifiable at 68° from the spindle axis; this angle needs to be reduced. (e) Schematic of how to reorient the crystal manually, using a sharp point to bend the loop in the required direction by pushing at where it is attached to the metal pin (care should be taken not to breathe away the cold stream!). Bending works best when pushing the nylon against the edge of the pin; the insert shows the less effective direction when bending towards the centre of the pin. Of course, a kappa goniometer, if present, allows the crystal to be tilted far more conveniently and accurately. (f) Diffraction pattern of the reoriented crystal, showing the long c* axis now significantly closer to the spindle axis, sufficient in this case to avoid spot overlap in the whole data set.

7. Data processing and diagnostics

Given how many things can go wrong, it is absolutely crucial to be aware of whether the data being collected are of useful quality or not. In the best case, poor data allow an accurate diagnosis (e.g. misaligned beam, decaying crystal), allowing one to rerun the experiment and obtain better data; in the worst case one can recognize the futility of the project and proceed to the next one.

It is thus uncontestably best practice to attempt data processing in as close to real time as possible. Many synchrotrons offer not only excellent computing facilities but also automated data processing; the facilities at the Diamond Light Source have become indispensable for much of our work, particularly since the installation of continuous-readout PILATUS detectors (Winter & McAuley, 2011). However, the onus still remains on the user to actually look at the output, know how to interpret it, attempt to correlate it to the actual experiment and attempt data processing manually when automated methods fail.

8. Summary

This article emphasizes data collection from novel poorly characterized crystal systems. Of course, many shortcuts make sense when a system is well known, e.g. multiple protein–ligand complexes (often called `molecular substitution'; Wasserman et al., 2012). However, for difficult projects we have found that one cannot be careful enough. Regardless of how easy it seems on modern facilities, X-ray data collection is actually a complicated experiment, and given how tiny the signal actually is, it is astonishing that we are able to measure what we do. The fact that this can nowadays be easily forgotten is a testament to the huge developments over recent decades.

Nevertheless, crystals that need special care will appear with great regularity; for these, an intimate knowledge of the experiment will remain crucial for many years to come.