Skip to content

Commit

Permalink
chore: clarify default location in help output
Browse files Browse the repository at this point in the history
  • Loading branch information
Justintime50 committed Sep 15, 2022
1 parent 449e3ed commit 9f38a0a
Show file tree
Hide file tree
Showing 13 changed files with 158 additions and 94 deletions.
16 changes: 15 additions & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,18 @@
name: build

on: [push, pull_request]
on:
push:
paths:
- '.github/workflows/build.yml'
- '**/*.py'
branches:
- "**"
tags:
- "!**"
pull_request:
paths:
- '.github/workflows/build.yml'
- '**/*.py'

jobs:
sh-checker:
Expand All @@ -25,6 +37,8 @@ jobs:
run: make install
- name: Check format
run: make format-check
- name: Scan for security vulnerabilities
run: make scan
test:
runs-on: ubuntu-latest
strategy:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,6 @@ jobs:
make install
make build
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@master
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_API_TOKEN }}
142 changes: 73 additions & 69 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,138 +1,142 @@
# CHANGELOG

## 4.5.1 (2022-09-15)

- Clarifies default location of github-archive in help output

## v4.5.0 (2022-02-22)

* Switches from the `threading` package to the `concurrent.futures` package allowing us to return values from each individual thread (in this case, the names of failed git assets)
* Fixes a long-standing bug where git repos that failed to clone would remain in the archive on Windows. This meant that recloning was impossible due to the repo existing and pulling was not possible because the repo wasn't yet fully initialized. The tool now has a proper cleanup step after everything has run that is compatible with both Unix and Windows environments (closes #38)
* Adds `--log_level` to allow the user to specify a custom log level
* Small adjustment to `include` and `exclude` help verbage used to be more clear they are optional filters
- Switches from the `threading` package to the `concurrent.futures` package allowing us to return values from each individual thread (in this case, the names of failed git assets)
- Fixes a long-standing bug where git repos that failed to clone would remain in the archive on Windows. This meant that recloning was impossible due to the repo existing and pulling was not possible because the repo wasn't yet fully initialized. The tool now has a proper cleanup step after everything has run that is compatible with both Unix and Windows environments (closes #38)
- Adds `--log_level` to allow the user to specify a custom log level
- Small adjustment to `include` and `exclude` help verbage used to be more clear they are optional filters

## v4.4.0 (2022-02-18)

* Adds an `--include` and `--exclude` CLI flag that accepts a comma-separated list of repo names to either include or exclude. If neither are passed, no filtering will occur (closes #43)
- Adds an `--include` and `--exclude` CLI flag that accepts a comma-separated list of repo names to either include or exclude. If neither are passed, no filtering will occur (closes #43)

## v4.3.0 (2021-12-16)

* Adds the ability to specify a custom base_url for GitHub (useful for enterprise GitHub users with a custom hostname, closes #41)
- Adds the ability to specify a custom base_url for GitHub (useful for enterprise GitHub users with a custom hostname, closes #41)

## v4.2.2 (2021-11-29)

* Adds `mypy` and fixes typing errors
- Adds `mypy` and fixes typing errors

## v4.2.1 (2021-11-25)

* Bumps `woodchips` to use the new implementation (also fixes a bug where we were creating a new `woodchips.Logger` class each time we called the logger instead of reusing the same logger instance)
* Adds missing `__all__` variable for importing
* Added Python type hinting
- Bumps `woodchips` to use the new implementation (also fixes a bug where we were creating a new `woodchips.Logger` class each time we called the logger instead of reusing the same logger instance)
- Adds missing `__all__` variable for importing
- Added Python type hinting

## v4.2.0 (2021-11-13)

* Uses `woodchips` for logging and removes internal logging logic from the package
* Refactors git commands to not change directories but instead run commands with the `-C` flag to invoke it in the directory we want
- Uses `woodchips` for logging and removes internal logging logic from the package
- Refactors git commands to not change directories but instead run commands with the `-C` flag to invoke it in the directory we want

## v4.1.1 (2021-11-02)

* Fixes a bug that wouldn't allow for gist cloning/pulling because of a bad "forks" check on a gist GitHub object
* Adds a missing check to ensure that at least one CLI argument was passed (closes #40)
* No longer invoke a shell while using the subprocess module. Git operations should now be more stable across operating systems
- Fixes a bug that wouldn't allow for gist cloning/pulling because of a bad "forks" check on a gist GitHub object
- Adds a missing check to ensure that at least one CLI argument was passed (closes #40)
- No longer invoke a shell while using the subprocess module. Git operations should now be more stable across operating systems

## v4.1.0 (2021-09-19)

* Adds a new `--https` flag which will authenticate via HTTPS instead of the default `SSH`
* Fixes a bug where using the `--stars` flag would not properly run due to a missing parameter. This parameter wasn't actually being used anymore and has been removed. Tests were beefed up for this function to protect against this happening again
- Adds a new `--https` flag which will authenticate via HTTPS instead of the default `SSH`
- Fixes a bug where using the `--stars` flag would not properly run due to a missing parameter. This parameter wasn't actually being used anymore and has been removed. Tests were beefed up for this function to protect against this happening again

## v4.0.0 (2021-08-24)

### Breaking Changes

* Reworks the entire app config to use CLI flags solely instead of a mix of CLI flags and env variables, additionally, most flags have changed names and functionality. See the README for new usage instructions or run `github-archive --help` (closes #30)
* Repos or gists that fail to clone or pull will now be completely removed so that they can be retried from scratch on the next run of the tool. This was an especially important change for bad clones as the tool would previously leave an empty initialized git folder even if the clone failed which would not possess the actual git repo yet. In this state, you could not properly pull new changes because the content of the repo hadn't properly been cloned yet
* Bumps required Python version from 3.6 to 3.7
- Reworks the entire app config to use CLI flags solely instead of a mix of CLI flags and env variables, additionally, most flags have changed names and functionality. See the README for new usage instructions or run `github-archive --help` (closes #30)
- Repos or gists that fail to clone or pull will now be completely removed so that they can be retried from scratch on the next run of the tool. This was an especially important change for bad clones as the tool would previously leave an empty initialized git folder even if the clone failed which would not possess the actual git repo yet. In this state, you could not properly pull new changes because the content of the repo hadn't properly been cloned yet
- Bumps required Python version from 3.6 to 3.7

### Features

* Adds a new `--users` flag which can be used to clone or pull git assets for a list of comma separated users (closes #20)
* Adds a new `--threads` flag which can specify the number of concurrent threads to run at once, default is `10` (closes #22)
* Adds a new `--view` flag which allows you to "dry run" the application, seeing the entire list of repos and gists based on the input provided (closes #25)
* Adds a new `--stars` flag which you can pass a comma separated list of users to and GitHub Archive will retrieve all of their starred repos which you can then view, clone, or pull (closes #26)
* Adds a new `--forks` flag which will include forks for whatever lists and operations you provide, default is `False` (closes #17))
- Adds a new `--users` flag which can be used to clone or pull git assets for a list of comma separated users (closes #20)
- Adds a new `--threads` flag which can specify the number of concurrent threads to run at once, default is `10` (closes #22)
- Adds a new `--view` flag which allows you to "dry run" the application, seeing the entire list of repos and gists based on the input provided (closes #25)
- Adds a new `--stars` flag which you can pass a comma separated list of users to and GitHub Archive will retrieve all of their starred repos which you can then view, clone, or pull (closes #26)
- Adds a new `--forks` flag which will include forks for whatever lists and operations you provide, default is `False` (closes #17))

### Fixes

* Removed verbose logging of skipped actions and "Already up to date" messages. Added additional logging related to API calls
* Added proper validation and type checking of variables and environment on startup
* Various code refactors, bug fixes, and optimizations
* Bumped the default git operation timeout from `180 seconds` to `300 seconds` to assist with cloning or pulling larger repos (closes #22)
* Removes `mock` library in favor of builtin `unittest.mock` library
- Removed verbose logging of skipped actions and "Already up to date" messages. Added additional logging related to API calls
- Added proper validation and type checking of variables and environment on startup
- Various code refactors, bug fixes, and optimizations
- Bumped the default git operation timeout from `180 seconds` to `300 seconds` to assist with cloning or pulling larger repos (closes #22)
- Removes `mock` library in favor of builtin `unittest.mock` library

## v3.1.1 (2021-07-24)

* Removed `branch` flag and functionality as it was causing issues and inconsistencies when cloning/pulling and branches didn't match up. This became especially prevelant when repos started changing from `master` to `main`
- Removed `branch` flag and functionality as it was causing issues and inconsistencies when cloning/pulling and branches didn't match up. This became especially prevelant when repos started changing from `master` to `main`

## v3.1.0 (2020-12-16)

* Changed all classmethods to staticmethods
* Corrected a bug where org names may not have had whitespace stripped properly
* CLI arguments now have an explicit default of `False`, this shouldn't change behavior from previous versions
* CLI default argument for `branch` has been changed from `master` to `None` and is handled via logic now. If no branch is specified, the default repo branch will be used instead of blindly assuming that `master` is the default branch (closes #18)
* Revamped the entire test suite to use conftest, simplified boilerplate, etc
- Changed all classmethods to staticmethods
- Corrected a bug where org names may not have had whitespace stripped properly
- CLI arguments now have an explicit default of `False`, this shouldn't change behavior from previous versions
- CLI default argument for `branch` has been changed from `master` to `None` and is handled via logic now. If no branch is specified, the default repo branch will be used instead of blindly assuming that `master` is the default branch (closes #18)
- Revamped the entire test suite to use conftest, simplified boilerplate, etc

## v3.0.1 (2020-10-01)

* Fixed broken entrypoint after shifting code around
- Fixed broken entrypoint after shifting code around

## v3.0.0 (2020-10-01)

* Refactored the entire codebase to be more pythonic, simpler, DRY, and documented (closes #15)
* Better error handling by raising errors where applicable and switching from a homegrown logger to the built-in Python logger (closes #12)
* Added unit tests and test coverage (closes #14)
* Added various additional configuration options
* Automated releasing on PyPi via Travis
* Various bug fixes throughout
* Better documentation on exactly what is possible with this tool
* Added a Makefile
* Adjusted most of the command and option names to be more uniform and explicit
- Refactored the entire codebase to be more pythonic, simpler, DRY, and documented (closes #15)
- Better error handling by raising errors where applicable and switching from a homegrown logger to the built-in Python logger (closes #12)
- Added unit tests and test coverage (closes #14)
- Added various additional configuration options
- Automated releasing on PyPi via Travis
- Various bug fixes throughout
- Better documentation on exactly what is possible with this tool
- Added a Makefile
- Adjusted most of the command and option names to be more uniform and explicit

## v2.1.2 (2020-08-14)

* Fixed the script's entrypoint (PyPi installs work again)
- Fixed the script's entrypoint (PyPi installs work again)

## v2.1.1 (2020-07-14)

* Fixed the long argument names which had underscores intead of hyphens
* Fixed a bug where threads were not waiting at the end of the script before printing the completion message
- Fixed the long argument names which had underscores intead of hyphens
- Fixed a bug where threads were not waiting at the end of the script before printing the completion message

## v2.1.0 (2020-07-01)

* Replaced `requirements.txt` with `setup.py`
* Removed Launch Agent
* Set the default git pull behavior as `--ff-only` to avoid git message (closes #9)
* Increased default timeout for git operations from `120` seconds to `180` seconds
* Replaced the majority of environment variables with command line args making customization easier as "feature flags"
* Various bug fixes and minor improvements
* Removed log deletion functionality
* Published to Pypi
- Replaced `requirements.txt` with `setup.py`
- Removed Launch Agent
- Set the default git pull behavior as `--ff-only` to avoid git message (closes #9)
- Increased default timeout for git operations from `120` seconds to `180` seconds
- Replaced the majority of environment variables with command line args making customization easier as "feature flags"
- Various bug fixes and minor improvements
- Removed log deletion functionality
- Published to Pypi

## v2.0.1 (2020-05-30)

* Fixed the Python program line in the launch agent plist
- Fixed the Python program line in the launch agent plist

## v2.0.0 (2020-05-18)

* Rewrote the script in Python
* Added concurrency to clone/pull changes incredibly fast for large amounts of repos and gists
* Added even more custimization and control
* Uniform log naming, better logging details
* Added changelog
- Rewrote the script in Python
- Added concurrency to clone/pull changes incredibly fast for large amounts of repos and gists
- Added even more custimization and control
- Uniform log naming, better logging details
- Added changelog

## v1.1.0 (2020)

* Added more customization options
* Allowed organization and gists to be cloned/pulled
* Allowed private repos to be cloned/pulled
* Bug fixes
- Added more customization options
- Allowed organization and gists to be cloned/pulled
- Allowed private repos to be cloned/pulled
- Bug fixes

## v1.0.0 (2019)

* Wrote the script in Bash with Python as dependency
* Added some customization options
- Wrote the script in Bash with Python as dependency
- Added some customization options
6 changes: 5 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,12 @@ lint:
mypy:
$(VIRTUAL_BIN)/mypy $(PROJECT_NAME)/ $(TEST_DIR)/

## scan - Scans the project for security vulnerabilities
scan:
$(VIRTUAL_BIN)/bandit -r $(PROJECT_NAME)/

## test - Test the project
test:
$(VIRTUAL_BIN)/pytest

.PHONY: help build coverage clean black black-check format format-check install isort isort-check lint mypy test
.PHONY: help build coverage clean black black-check format format-check install isort isort-check lint mypy scan test
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ A powerful tool to concurrently clone or pull user and org repos and gists to cr

GitHub Archive is a powerful tool to concurrently clone or pull repositories or gists from GitHub with incredible flexibility. It's the perfect tool for spinning up a new dev environment, keeping a local copy of your GitHub instance, or quickly pulling in projects from your favorite users and organizations.

The power of GitHub Archive comes in its configuration. Maybe you only want to clone or pull your personal public repos or maybe you want to go all out and include private repos from you and all organizations you belong to along with your gists.
The power of GitHub Archive comes in its configuration. Maybe you only want to clone or pull your personal public repos or maybe you want to go all out and include private repos from you and all organizations you belong to along with your gists. GitHub Archive can do it all.

## Install

Expand All @@ -25,7 +25,7 @@ pip3 install github-archive

# Install locally
make install
```
```

## Usage

Expand Down Expand Up @@ -53,7 +53,7 @@ Options:
-e EXCLUDE, --exclude EXCLUDE
Pass a comma separated list of repos to filter what is excluded from the Archive.
-l LOCATION, --location LOCATION
The location where you want your GitHub Archive to be stored.
The location where you want your GitHub Archive to be stored. By default, this is /Users/USERNAME/github-archive
-ht, --https Use HTTPS URLs instead of SSH.
-to TIMEOUT, --timeout TIMEOUT
The number of seconds before a git operation times out.
Expand Down
5 changes: 4 additions & 1 deletion github_archive/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
from github_archive.archive import GithubArchive
from github_archive.archive import (
GithubArchive,
)


__all__ = [
'GithubArchive',
Expand Down
29 changes: 22 additions & 7 deletions github_archive/archive.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,27 @@
import os
import shutil
import stat
import subprocess
from concurrent.futures import ALL_COMPLETED, ThreadPoolExecutor, wait
from datetime import datetime
from typing import List, Optional, Union
import subprocess # nosec
from concurrent.futures import (
ALL_COMPLETED,
ThreadPoolExecutor,
wait,
)
from datetime import (
datetime,
)
from typing import (
List,
Optional,
Union,
)

import woodchips
from github import Gist, Github, Repository
from github import (
Gist,
Github,
Repository,
)

from github_archive.constants import (
DEFAULT_BASE_URL,
Expand All @@ -17,6 +31,7 @@
DEFAULT_TIMEOUT,
)


CLONE_OPERATION = 'clone'
PULL_OPERATION = 'pull'

Expand Down Expand Up @@ -370,7 +385,7 @@ def archive_repo(self, repo: Repository.Repository, repo_path: str, operation: s
git_command = commands[operation]

try:
subprocess.run(
subprocess.run( # nosec
git_command,
stdout=subprocess.DEVNULL,
stdin=subprocess.DEVNULL,
Expand Down Expand Up @@ -409,7 +424,7 @@ def archive_gist(self, gist: Gist.Gist, gist_path: str, operation: str) -> Optio
git_command = commands[operation]

try:
subprocess.run(
subprocess.run( # nosec
git_command,
stdout=subprocess.DEVNULL,
stdin=subprocess.DEVNULL,
Expand Down
12 changes: 9 additions & 3 deletions github_archive/cli.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
import argparse
from typing import get_args
from typing import (
get_args,
)

from github_archive import GithubArchive
from github_archive import (
GithubArchive,
)
from github_archive.constants import (
DEFAULT_BASE_URL,
DEFAULT_LOCATION,
Expand Down Expand Up @@ -116,7 +120,9 @@ def __init__(self):
type=str,
required=False,
default=DEFAULT_LOCATION,
help='The location where you want your GitHub Archive to be stored.',
help=(
f'The location where you want your GitHub Archive to be stored. By default, this is: {DEFAULT_LOCATION}'
),
)
parser.add_argument(
'-ht',
Expand Down
5 changes: 4 additions & 1 deletion github_archive/constants.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
import os

from typing_extensions import Literal
from typing_extensions import (
Literal,
)


DEFAULT_BASE_URL = 'https://api.github.com'
DEFAULT_LOCATION = os.path.expanduser(os.path.join('~', 'github-archive'))
Expand Down
Loading

0 comments on commit 9f38a0a

Please sign in to comment.