-
Notifications
You must be signed in to change notification settings - Fork 4
EMC reconstruction
With the reduced data, we can proceed with the EMC reconstruction, which consists of two parts:
The high-resolution reconstruction applies the local update scheme of the EMC algorithm to speed up the reconstruction based on the results from the low-resolution reconstruction.
When the computational resource is not a concern, the high-resolution reconstruction can be skipped by using a
small res_cutoff (but no smaller than res_max) in Data reduction and only implementing the low-resolution reconstruction with the reduced data.
Here we apply the standard EMC algorithm to the low-resolution intensity reconstruction.
We start by updating the parameters in the [low-res-emc] section of config.ini:
[low-res-emc]
iter_data_block = 5
prob_dir = [reduced-data-dir]/Data/high-prob
prob_orien_file = [work-dir]/aux/prob-orien.bin
reduced_data_id_file = [work-dir]/reduce-data/reduced-data_id.dat
start_phi_file = [work-dir]/aux/start-phi.dat
start_intens_file = [work-dir]/aux/start_intensity.binSince the data size is generally several hundred GB or more, the data frames are separated into iter_data_block blocks and read in sequentially in each EMC iteration to save memory.
The directory prob_dir stores the output files of the reconstruction.
The file prob_orien_file records the probable orientations for each data frame, and reduced_data_id_file stores the original data frame order in reduce-data, before the rearrangement.
The files start_phi_file and start_intens_file are the initial models for the reconstruction.
Next, we move to the directory aux to generate start_phi_file and sym-op.dat, which stores the symmetry operators of the crystal lattice.
The operations in this paragraph can be skipped if the reduced data downloaded from CXIDB is used.
By executing the command
python init-phi.py
the initial values of the scale factors, φk, are estimated with the average peak value in
each data frame.
Two additional files start-phi-A.dat and start-phi-B.dat are also generated by this command.
These files store the scale factors for the two independent halves of the data frames, which will be used in Resolution estimation
The file sym-op.dat is generated with the command
python make-sym-op.py
for the tetragonal crystal lattice. This file should be created by the user for general crystal lattices.
The EMC reconstruction is started by executing the commands
make
mpirun -np [nproc] ./emc ../config.ini [iter] > run.log &
Here [nproc] should be the same as the value specified in config.ini, and [iter] is the number of EMC iterations.
The 3D intensity model is initialized by placing 3D Gaussian of random height at each Bragg position, whose values are stored in start_intens_file.
In the n-th EMC iteration, our program creates two directories in prob_dir: iter_flag-[2n − 1] and iter_flag-[2n], which store the outputs from the updates of the intensity model and the scale factors, φk, respectively.
In order to resume a previous reconstruction, the user has to replace the files, start_phi_file and start_intens_file, by total-phi.dat and finish_intensity.bin output in the last iteration of the previous reconstruction.
The output files from the previous reconstruction have to be moved elsewhere to avoid being overwritten.
After the reconstruction reaches convergence, we execute the command
python move-recon-files.py ../config.ini
to create the directory low-res-recon in prob_dir, and move start_phi_file, start_intens_file and the output files of the reconstruction there.
Finally, we move to the directory rej-frames and execute the command
python rej-frames.py ../config.ini
to exclude the frames that contain no or multiple crystals.
This command creates an updated start_phi_file, where the excluded frames will have the scale factors, φk, set as zero.
Here the local update scheme of the EMC algorithm is implemented to extend the reconstruction to high resolution based on the converged models and probability distribution given by the low-resolution reconstruction.
We first choose the resolution cutoff that will be used in the high-resolution EMC reconstruction.
This value is specified by the parameter high_res_cutoff in config.ini, and should be larger than res_max but smaller than res_cutoff.
We then move to the directory setup-local, and execute the command
python setup-quat.py ../config.ini >> run.log &
to generate the rotation samples that will be used and the file that stores the mapping between this rotation sampling and that used in the low-resolution reconstruction.
The angular resolution of the new rotation samples is chosen to not to miss any Bragg peaks within the resolution high_res_cutoff.
By executing the command
python setup-intens.py ../config.ini >> run.log &
we generate the initial 3D intensity model for the high-resolution reconstruction, which is stored in the file start_intens_file.
Finally, the mapping between Bragg peaks and detector pixels for the new rotation samples is generated using the command
python setup-Ematrix.py ../config.ini >> run.log &
This mapping is stored in the files local_r2peak_file and local_peak2r_file, as
specified in config.ini.
After moving to the directory local-update we start the high-resolution reconstruction using the commands
make
mpirun -np [nproc] ./emc ../config.ini [iter] > run.log &
The parameter [nproc] should be the same as the value specified in config.ini, and [iter] is the number of EMC iterations.
The output files of the n-th EMC iteration are stored in the directory prob_dir/iter_flag-[n].
When the reconstruction reaches convergence, we execute the command
python move-recon-files.py ../config.ini
to create the directory high-res-recon in prob_dir, and move start_phi_file,
start_intens_file and the output files of the reconstruction there.