Add doc

[antolinos]

Hi @marcus-oscarsson

I propose:

LIMS integration

Integrating a Laboratory Information Management System (LIMS) is an essential part of data acquisition software. It enables the retrieval of sample and data acquisition information for use during data collection, as well as real-time monitoring of experimental results.
MXCuBE can be integrated with any LIMS, currently supporting ISPyB, ICAT, or both via the AbstractLims interface.

Background

The ISPyB LIMS system has for a very long time been the LIMS standard in the MXCuBE community. There was for that reason, little need for an abstraction that enabled the use of other systems. Several sites within the community have started to use other LIMS systems, creating a need for a more general abstraction within MXCuBE.

The current AbstractLims abstraction is heavily based on the interface of the previous ISPyBClient, which was used for communication with ISPyB. Much of the API defined in ISPyBClient was kept the same so that the new AbstractLims, in an initial phase, can be used with minimal changes.

The aim of the current implementation of AbstractLims was to facilitate the usage of other LIMS systems. It also allows the use of other LIMS system in parallel with ISPyB. It's understood that the AbstractLims will need to evolve with the introduction of new LIMS systems and further development of existing solutions.

Development status

Internal data model

The data structures passed to the LIMS system via AbstractLims are still, as with ISPyBClient, the dictionary data structure used by the collect routines based on AbstractCollect and AbstractMultiCollect. A deliberate choice to leave room for discussion on the data model and make the transition into AbstractLims easier. There is a longer-running discussion in the community on the topic of defining these data structures as data classes and, more recently, as PyDantic models. The dicitonary structurs have for that purpouse been doucmented in the new AbstractLims class.

There is an attempt to define the data structures retrieved from the LIMS system in model/lims_session.py. This data model currently defines information related to the experiment or session, which is assumed to be very similar across any synchrotron. Work is needed to further develop it so that it includes the different data entities, such as sample.

Abstract LIMS

The AbstractLIMS class contains the methods that must be implemented for any LIMS to function properly. The following list is the result of merging, cleaning and refactoring the methods found in HardwareObjects/ISPyBClient.py and HardwareObjects/ISPyBRestClient.py:

    def get_user_name(self)
    def get_full_user_name(self)
    def login(self, login_id: str, password: str, create_session: bool)
    def get_proposals_by_user(self, login_id  str)
    def get_samples(self)
    def store_robot_action(self, proposal_id  str)
    def store_beamline_setup(self, session_id  str, bl_config_dict  dict)
    def store_image(self, image_dict  dict)
    def store_energy_scan(self, energyscan_dict  dict)
    def store_xfe_spectrum(self, xfespectrum_dict  dict)
    def store_workflow(self, workflow_dict  dict)
    def store_data_collection(self, datacollection_dict: dict, beamline_config_dict: dict, event: StoreEvent = "CREATE")
    def update_bl_sample(self, bl_sample  str)

ISPyB

ISPyBAbstractLIMS

The former ISPyBClient has been refactored and renamed to ISPyBAbstractLIMS. It uses the ISPyBDataAdapter class, a data adapter that handles all calls to ISPyB via SOAP web services. It improves the readability and maintainability of the code by reducing its complexity.

Depending on the facility, a user can be logged in as either a user or a proposal. In the previous implementation, this logic was combined, increasing complexity. It has now been separated into two distinct classes that inherit from ISPyBAbstractLIMS: UserISPyBLims and ProposalTypeISPyBLims.

A facility using the ISPyB LIMS would need to configure the lims.xml hardware object either with ProposalTypeISPyBLims or UserISPyBLims. Example:

<object class="ProposalTypeISPyBLims">
  <object role="ldapServer" href="/ldapconnection" />
  <ws_root>
    https://ispyb.server.fr/ispyb/ispyb-ws/ispybWS/
  </ws_root>
  <ws_username>********</ws_username>
  <ws_password>********</ws_password>
  <object role="session" href="/session" />
  <authServerType>ldap</authServerType>
</object>

ICAT

"Similar to ISPyB, MXCube can now use ICAT out of the box with the ICATLIMS class. Note that ICATLIMS can only work with user authentication this is done in purpose to improve security.

The way a user selects their session is more flexible than in ISPyB, and it also manages beamline changes (in case of breakdown), making the configuration slightly more complex.

<object class="ICATLIMS">
  <ws_root>https://icatplus.server.com</ws_root>
  <loginType>user</loginType>
  <object role="session" href="/session" />
  <queue_urls>ingester:61613</queue_urls>
  <before_offset_days>20</before_offset_days>
  <after_offset_days>20</after_offset_days>
  <compatible_beamlines>ID30A-1</compatible_beamlines>
  <!-- the list of sessions will be filtered by the override_beamline_name, this is needed when
  HWR.beamline.session.beamline_name does not match with the name of the instrument in the catalog -->
  <override_beamline_name>ID30A-1</override_beamline_name>

  <data_portal_url>https://data2.esrf.fr/investigation/{id}/datasets</data_portal_url>
  <logbook_url>https://data2.esrf.fr/investigation/{id}/logbook</logbook_url>
  <user_portal_url>
    https://smis.esrf.fr/manageAForm.do?action=view&amp;expSessionVO.pk={id}</user_portal_url>

</object>

ESRFLIMS

On top of the mentioned configuration we have developed a LIMS that combines both. It means that sample information can be retrieve from either ISPyB or ICAT and the results are sent to both LIMS. The authorization is based in ICAT.
The configuration lims.xml relies on the previous hardware object for ICAT (drac.xml) and ISPyB (ispyb.xml):

<object class="ESRFLIMS">
  <object role="drac" href="/drac" />
  <object role="ispyb" href="/ispyb" />
</object>