CopyXnat

A utility for downloading or copying projects between XNAT servers.

This is a prerelease. It may not work as intended, and may lead to data loss. Please use caution when running on production servers

Quick start

Please make sure your XNAT archive and PostgreSQL database are fully backed up before running CopyXnat on a real XNAT server. CopyXnat may modify and overwrite your data.

Install and run using pip

Python 2.7 or 3.6+ is required, and pip should be installed.

It is recommended that you do not modify your system Python installation. Use a tool such as Homebrew (macOS) to install the latest python version without affecting your system python. Then, create a dedicated virtual python environment using virtualenv and virtualenvwrapper or similar tools.

Once you have a suitable python environment set up, install with pip:

    pip install copyxnat --upgrade
    copyxnat --help

You can use pip's --user flag to install dependencies in the user space instead of globally.

Install and run using Docker

If you have Docker installed you can run CopyXnat in a container. Note that interactive mode (-it) is generally required in order to enter server passwords where prompted.

    docker run --rm -it tomdoel/copyxnat --help

Example use cases

Show project contents

• Print a list of projects, subjects etc on an XNAT server

  copyxnat show -s "https://YOUR-SERVER-URL" -u "YOUR-USER-NAME"
  

• Show differences between two projects

  copyxnat compare --src-host "https://SOURCE-SERVER-URL" --src-user "XNAT-USER-SOURCE" --project "PROJECT-NAME-SRC:PROJECT-NAME-DST" --dst-host "https://DEST-SERVER-URL" --dst-user "XNAT-USER-DEST"
  

Download projects

• Download XNAT projects onto your computer

  copyxnat export --src-host "https://YOUR-SERVER-URL" --src-user "YOUR-USER-NAME" --project "PROJECT1,PROJECT2"
  

• Download all projects you have access to onto your computer

  copyxnat export --src-host "https://YOUR-SERVER-URL" --src-user "YOUR-XNAT-USER-NAME"
  

Copy projects from one XNAT server to another

• Duplicate a project on a destination server

  copyxnat copy --src-host "https://SOURCE-SERVER-URL" --src-user "XNAT-USER-SOURCE" --project "PROJECT-NAME" --dst-host "https://DEST-SERVER-URL" --dst-user "XNAT-USER-DEST"
  

• Duplicate a project and rename

  copyxnat copy --src-host "https://SOURCE-SERVER-URL" --src-user "XNAT-USER-SOURCE"  --project "PROJECT-NAME:NEW-PROJECT-NAME" --dst-host "https://DEST-SERVER-URL" --dst-user "XNAT-USER-DEST"
  

• Duplicate a project on the same server (with a new name)

  copyxnat copy --src-host "https://SERVER-URL" --src-user "XNAT-USER-SOURCE"  --project "PROJECT-NAME:NEW-PROJECT-NAME" --dst-host "https://SERVER-URL" --dst-user "XNAT-USER-DEST"
  

Maintenance

• Rebuild catalog for all experiments

    copyxnat rebuild-catalog -s "https://YOUR-SERVER-URL" -u "YOUR-USER-NAME"
  

• Reset OHIF viewer for all experiments

    copyxnat ohif -s "https://YOUR-SERVER-URL" -u "YOUR-USER-NAME" 
  

Command-line parameters

The parameters vary depending which command you are running. To see help for a particular command, run with the command name and --help, for example:

copyxnat copy --help

Parameter	Short form	Default	Description	Notes
--help	-h	-	Shows general help
command --help	command -h	-	Shows detailed help for this command
--version		-	Shows version number
command		-	Command to run. One of copy, export, show, compare, check-datatypes, ohif, rebuild-catalog	See above for examples of each command.
--src-host hostname	-s	-	hostname of source XNAT server
--src-user username	-u	-	XNAT username for source server
--project project_string	-p	all projects	list of project IDs to process	Use a colon if you want to rename project IDs between source and destination servers, eg srcId:dstId. To process multiple projects, separate the IDs with commas, eg proj1,proj2. Do not include any spaces. You can do both, e.g. prj1,prj2:dst2,prj3
--dst-host hostname	-d	-	hostname of destination XNAT server	Only for some commands
--dst-user username	-w	-	XNAT username for destination server	Only for some commands
--fix-scan-types	-f	False	Fix undefined scan types on the copy	If the source server does not correctly specify the scan type, CopyXnat will attempt to determine the scan type and set it on the destination server
--transfer-mode mode	-t	file	Specifies how files will be copied or downloaded. One of file, zip, meta, rsync	file to copy one file at a time. zip to zip up each resource collection as a zip file (generally faster). meta to transfer metadata only, not resource files. rsync to transfer files using rsync (experimental)
--limit-subjects limit	-l	no limit	Process a maximum number of subjects specified by the integer limit.	During a copy, the limit will exclude subjects which already exist on the destination and have not changed.
--skip-existing	-n	False	Only copy new experiments; do not synchronise existing experiments	Speeds up copying process when some experiments have already been copied to the destination server. However will not copy over new scans and resources that have been added to existing experiments.
--verbose	-v	False	Output additional information	For debugging and for providing additional information about what the command is doing.
--insecure	-k	False	Do not verify server certificates	Do not use in production as is vulnerable to MITM. For testing or working with self-signed certificates within a secure private network.
--dry-run	-y	False	Do not make changes on the destination server	Useful for seeing what changes a command may make before running it for real. May cause errors in some commands because the expected changes on the destination server didn't happen.
--overwrite-existing	-o	False	Overwrite existing data on the destination server	When copying, will modify and overwrite existing data
--ohif-rebuild	-g	False	When copying, trigger an OHIF viewer rebuild after each experiment copy.	The OHIF Viewer does not always detect new data being added to existing experiments. This will cause a session rebuild when data are copied, but this will take additional time.
--rsync-src-user	-i	-	ssh source server username for copy via rsync	Experimental. Only used for copy using --transfer-mode rsync
--rsync-dst-user	-w	-	ssh destination server username for copy via rsync	Experimental. Only used for copy using --transfer-mode rsync

Supported XNAT data levels

CopyXnat can process the following XNAT data hierarchies.

• Projects
  • Project Resources
  • Subjects
    • Subject Resources
    • Experiments
      • Experiment Resources
      • Scans
        • Scan Resources
      • Assessors
        • Assessor In-Resources (copy not supported)
        • Assessor Out-Resources
      • Reconstructions (copy not supported)
        • Reconstruction In-Resources
        • Reconstruction Out-Resources

Current limitations

• Reconstructions can be downloaded but cannot be copied to another server
• Assessor In-Resources can be downloaded but cannot be copied to another server
• Support for eCat files is limited and untested

Using CopyXnat in your own code

Importing via pip

You can import and run the module with pip and then import from your code

    from copyxnat.xnat.run_command import run_command
    from copyxnat.commands.show_command import ShowCommand

    result = run_command(command=ShowCommand,
                         src_host=src_host,
                         src_user=src_user,
                         src_pw=src_pw,
                         project_filter=None)

Local development: running the command-line utility directly

You can run directly from the source code. This is primarily useful if you are modifying the code.

   pip install -r requirements-dev.txt
   python copyxnat --help

Links

• Documentation
• Source code
• PyPi
• Docker Hub

Authors

• Tom Doel

Copyright

Copyright 2021 Tom Doel

License

This project is released under the MIT License. Please see the license file for details.