The MAX IV imaging concept

Sardana control system

Sardana [10] is a software suite for Supervision, Control and Data Acquisition (SCADA) in scientific installations. It aims to reduce cost and time of design, development and support of the control and data acquisition (DAQ) systems. Sardana development was started at the ALBA synchrotron (Spain) and designed using a large experience from ESRF beamlines. Today, it is used and supported by a larger community which includes several other laboratories (ALBA, DESY, MAX IV, Solaris). Sardana was evaluated at the Max-lab facility for a potential use at the next MAX IV Laboratory. After the completion of the MAX IV Laboratory, it became the preferred platform even though its immaturity in terms of well-developed user friendliness was apparent. It was foreseen that the community would improve this as its use became more widespread. The main desirable characteristics of Sardana are as follows:

• The same programming language (python) is used to implement the framework, resulting in a more accessible application and lower complexity. This was thought to have a positive impact for the user autonomy. Python is very widespread both in the control system and scientific software domains which gives more accessibility to these communities.

• Sardana is fully integrated with Tango [11] and therefore complementing the Tango standard which was already chosen for MAX IV.

• Taurus [12] provided an easy way to build simple or complex graphical panels with easy access methods for displaying data acquisition results and control of the instrumentation, making it versatile for all accelerator and beamline graphical interfaces.

Control with virtual machines

Maintenance of computer hardware in a research facility can be a difficult task because of the widespread distribution of components, sometimes difficult to access. This can lead to issues related to maintaining good reliability and reducing down time caused by computer component failures. In a distributed control system, a controller is present near every sensor or actuator and traditionally requires some computer locally connected via an interface. In the MAX IV control system architecture, the hardware controller manages the hard real-time operation. In combination, all soft real-time operations are delegated to the higher level control computing. Nowadays, more and more connections from devices are made using Ethernet, permitting easier management of the compute resources by relocating the control computing hardware elements into a virtual machine cluster. In this way, only the hardware controller remains distributed, which reduces the distribution of complex computer systems, therefore improving reliability. Other advantages include centralized management with features, such as automatic fail-over of control virtual machines, remote software configuration of machine CPUs, and storage and easy online upgrade of the overall capacity when needed, e.g., adding more dedicated CPU when the software needs more resources.

Integrating new elements

The Sardana/Tango SCADA is designed for straightforward integration of new components. Area X-ray detectors are the most complex elements of the imaging DAQ as they are expected to produce the crucial data. Their integration into the acquisition process has three layers.

1. In the lowest control layer, the specific device control protocol is adapted to Tango by an intermediate LIMA (Library for Image Analysis) system (initially an ESRF development) [13], providing a generic Tango interface for the class of “Detector” (2D or 1D). Other non-generic Tango interfaces can be developed when need to access a specific set of features.

2. Sardana is the second layer in the acquisition work flow which orchestrates acquisitions and data recording. Hardware devices to control are divided into categories (actuators, sensors, counters, etc.) associated with a generic programmable interface. It ensures that data are acquired without including too much specific control logic in the code that executes the sequence. Equipment incompatible with the conventional detector interface defined by LIMA is transparently handled by specific extensions. The role of Sardana is also to make sure data are formatted and stored in a safe space.

3. The last layer of the acquisition process provides raw data visualization and processing. The visualization channel uses Sardana and LIMA interfaces to which any suitable plotting software can be connected. Besides this basic data visualization concept, MAX IV aims to provide a platform for handling (imaging) data with advanced methods and algorithms already during the experiment to maximize effective use of the experimental time. In its most cutting edge variants, it is on-the-fly data handling with feedback to experiment in the form of already interpreted data [6, 7, 14]. The platform for handling imaging data will be running at the central computing infrastructure (see Fig. 1). From the control point of view, the first task is pushing data down in the scheme from the beamlines in Fig. 1. The protocols depend on capabilities of detector systems. Processing HDF5 data with low latencies is always possible today but huge improvement can be achieved using the LIMA interface also for data transfer or streaming protocols supported directly in the detectors. We refer here to current developments as GigaFRoST [6] or trends in high data rate macromolecular crystallography [15]. Another control task is providing a feedback from fast data analysis done at the computing infrastructure to the beamline control system (as indicated in Fig. 1). Finally, the important role of Sardana is also managing the master experimental HDF5/NeXus data file and making links to HDF5 data provided by detectors or processed from the streams.

Timing the acquisition

An obvious asset of imaging in full-field mode (and the near-field diffraction regime) is the simple and robust experimental setup, flexible sample environments but most importantly a uniquely fast acquisition of a large 3D dataset [16]. The timing of such experiments is critical and is up to now only weakly correlated with the dynamics in the sample. One reason is that there is practically no feedback available at these acquisition rates. Our concept of real-time data monitoring at the imaging beamline at MAX IV will enable an informed decision taking process in terms of acquisition control and raw data recording. This step is critical for ensuring that only relevant raw data are saved and enter the offline analysis pipeline. One way to achieve this is through fast tomographic reconstruction and feature separation (image binarization) with consequent basic topological and statistical measurement on selected tomographic slices.

Tomographic and phase reconstruction platform

At synchrotron facilities, the image reconstruction (tomographic and phase) is often provided using one single method that has proven to be robust. There are no enough resources or know-how directly at the facilities to develop or simply implement existing multiple advanced reconstruction methods. We think that a platform based on the recently developed Operation Discretisation Library (ODL) [17] in python will stimulate the development and implementation of modern tomographic and phase reconstruction schemes to reflect the variable experimental settings. In particular, this approach will favor the construction of iterative schemes combining tomography and phase retrieval in the near-field (tomographic microscopy) and far-field (ptychography) approximation. Currently, the library includes the Astra toolbox and the Gridrec forward projector [18]. When completed by most of the modern and also traditional tomographic and phase reconstruction operators and when the support of large data volumes and compatibility with real-time data monitoring is enabled, the ODL will be a valuable tool at the imaging beamlines for fast prototyping to facilitate advances.

Currently, there are three distinct groups of tomographic reconstruction routines implemented at the Lunarc GPU cluster. One being a Log-polar fast Radon transform approach developed at the Mathematics Department at Lund University [20], the second being algebraic methods such as SIRT and the third group being discrete methods such as DART and an energy minimization method [21]. The latter are more computationally intensive and therefore slower. But useful for specific cases in particular when a small number of higher quality projections are available. Contrary, the first method is fast and qualifies for on-the-fly data analysis during experiment making it particularly valuable for processing time resolved acquisition. Performance of all the methods in terms of speed is evaluated in Table 1, taken from [20].

Quantitative analysis

The acquisition of time resolved three-dimensional datasets brings several major challenges. First, the more precise in time and space we are becoming the larger becomes the data size, surpassing easily several TB when following one single dynamic process lasting less then 10 min. Second, new user communities are attracted to synchrotron facilities by the unique capabilities coming up, offered by the higher brightness and faster detectors. These users have typically very limited or non-existent help for data analysis. Some guidance is traditionally offered in the spare time of the facility staff, but in the majority of cases the user is left alone with the task to find ways to access and retrieve the relevant information. Third, it is imperative that quantitative results emerge for these facilities to become a routine tool of use for materials design and structural biology, among others.

One way of dealing with the high demand for feature extraction tools from multi-dimensional images is to introduce showcases comprising typical features re-occurring in various scientific fields. One example is cellular systems. A lot of effort has been put into the development of quantitative analysis tools of complex cellular systems in the past [23], yet no reliable single tool exists today that would give a confidence level better than a few percents in the quantification of the topology and texture of this type of images. In the case of liquid foams, the main reason is that the resolution of dynamic tomographic microscopy is insufficient to resolve the films between individual cells, yet the analysis imperatively expects each single cell uniquely labeled to distinguish the boundaries between them. The new direction which we think has a potential to help in this is to refine the outcome of basic image processing by modeling the system using Monte Carlo methods. We used Cellular Potts Model [24], in the framework of the open source software CompuCell3D [25] one can reconstruct the correct foam from the tomographic images and to obtain a realistic description of the foam topology in 3D. Foam behavior and structure are determined by physical boundaries; in particular, the surface tension is the main contribution to the total entropy of a foam structure. Starting from a rough initial guess, the individual cell wall positions predicted by simple image processing in a liquid foam are refined using the Cellular Potts Model in Fig. 2 by minimizing the surface tension. Bringing physical laws into the image processing workflow is an attractive opportunity that is likely to help in gaining confidence in the quantification platforms applied to complex multi-dimensional data. For a number of selected shapes, basic but robust quantitative image analysis workflows should be developed to guide the users of imaging facilities toward an efficient and reliable quantification of acquired images.