cloner

A tool to clone efficiently all the repos in an organization

Install

In PyPi it's called wr-cloner, cloner was already taken :sad:

pip install wr-cloner

Usage

Note: If using cloner after cloning the project and not a PyPi package, add "python" at the start. There's an example of that in the examples section.

Usage: cloner [OPTIONS] GITHUB_ORGANIZATION

  A tool to clone efficiently all the repos in an organization.

Options:
  --version                       Show the version and exit.
  --token TEXT                    GitHub token to read private repos. This
                                  parameter is needed when cloning from a
                                  GitHub Enterprise server.
  --ghe TEXT                      GitHub Enterprise URL. It needs the
                                  GITHUB_ORGANIZATION parameter to clone repos
                                  from there and the TOKEN option as well.
  --threads INTEGER               Number of threads and processes to use. For
                                  maximum threads and processes on the system,
                                  use '--max-threads'  [default: 4]
  --max-threads                   If declared, uses the maximum available
                                  threads and processes in the system. As per
                                  physical cores on the system cpu.
  --logging [ERROR|WARNING|INFO|DEBUG]
                                  Logging level  [default: INFO]
  --path TEXT                     Sets a path where to clone the repositories
                                  (eg: ./another/path/)
  --git-options TEXT              Add options to the clone command (eg: --git-
                                  options "--depth 1"). By default, clones
                                  quietly (--quiet).
  --ignore-archived               If declared, will ignore archived repos when
                                  cloning.
  --ignore-template               If declared, will ignore template repos when
                                  cloning.
  --ignore-fork                   If declared, will ignore fork repos when
                                  cloning.
  --help                          Show this message and exit.

Examples

# For github.com with 8 threads
cloner --threads 8 GITHUB_ORGANIZATION

# For github.com with the maximum threads on the system running
cloner --max-threads GITHUB_ORGANIZATION

# For GHE, default threads
cloner --ghe GHE_URL --token SUPER_SECURE_TOKEN GITHUB_ORGANIZATION

# Cloning with options
cloner --git-options "--depth 1" GITHUB_ORGANIZATION

Examples if cloning the repo

# Install dependencies with
make install
# Adjust PYTHONPATH
export PYTHONPATH=$PYTHONPATH:.
# Use it
python cloner GITHUB_ORGANIZATION

Contributing

Check the CONTRIBUTING.md file.

Security

Follow the instructions in the SECURITY.md file.

License

MIT

Known Issues

• Multithreading doesn't work to clone repos, since the os.system call is 1 for each PID. The splitting is done with multithreading, the cloning with multiprocessing. Same amount of threads and processes.
• Windows usage and support
• If seeing RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8) either adjust your git config, as per this StackOverflow issue. If still persists, either decrease the number of threads using the --threads option or adjust the depth of cloning with --git-options "--depth 10" (10 or any other small number).

Star History