diff --git a/.gitignore b/.gitignore index fd3b4fd..8422016 100644 --- a/.gitignore +++ b/.gitignore @@ -1,20 +1,23 @@ # misc plan.txt +notes.txt .dgupdaterignore .env +*.exe # Virtual Environments venv/ # Tests tests/ -main.py +dgupdater.py dgupdaterconf.json -# Builds +# Builds & release dgupdater.egg-info/ dist/ build/ +dgupdater_release/ # Pycache *.pyc diff --git a/README.md b/README.md index a072614..65f2827 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,61 @@ -You need to have a mongodb database preferrably on cloud. You can use mongodb atlas for the same, MongoDB have a serverless db cluster which is very cheap (0.1$ / 1M reads). they also have a good free tier. After configuring, get the connection url and save it. +# DGUPDATER -You need to get two connection string from mongo db, one with write access and one with read access. -the one with write access will be used by the developer to write data to the database and the one with read access will be used by the client to read data from the database. both the connection string should have access to the same cluser and databases. \ No newline at end of file +## Requirements for using dgupdater + +* You need to have a mongodb database preferrably on cloud. You can use mongodb atlas for the same, MongoDB has a serverless db cluster which is very cheap (0.1$ / 1M reads). they also have a good free tier. +After configuring, get the connection string and save it. + +* You need to create two database users in the database, one with only read access and one with only write access. +The user with write access will be used by the developer to write data to the database and the user with read access will be used by the client to read data from the database. + +* When initializing the application will ask two connection strings, one with write access and one with read access. + So replace the username and password in the connection string with the username and password of the user with write and read access respectively. + + + + +## Initialization +You have open a terminal and navigate into the directory which you want to initialize for auto updation. Then run the following command: + +```bash +dgupdater init +``` + +* This will ask you for the application name, application version, mongodb connection string with read access, mongodb connection string with write access. After providing the required information, it will create a file named 'dgupdaterconf.json' in the current directory. This file will be used to store the configuration of the application. + +* The mongodbconnection string with write access will be stored in a different place as follows +```bash +Windows: C:/Users//AppData/local/DarkGlance/dgupdater/dgupdaterconf.json +Linux: /home//.config/DarkGlance/dgupdater/dgupdaterconf.json +Mac: Users//Library/Application Support/DarkGlance/dgupdater/dgupdaterconf.json +``` + +* Then it will check if the application is already registered in the database or not. If it is not registered, it will register the application in the database. or it will ask to overwrite the application details. + +* It will also create a file named .dgupdaterignore in the current directory. +this file works just like the .gitignore file. you can add the files and directories which you want to ignore while updating the application. + +* If you want to change the mongodb connection string, you can do it by running 'dgupdater init' again and providing the new connection string. + +* After initializing the application, you can continue building the application and when you are ready to update the application, you can commit the changes. + +## Commiting the changes + +* After making the changes in the application, you can commit the changes by running the following command: + +```bash +dgupdater commit +``` + +* This will ask for the new versio number of the application. +* After providing the version number, it will ask confirmation to commit the changes. If you confirm, it will commit the changes. + +* In this process the following things will happen: + + 1. The files and directories mentioned in the .dgupdaterignore file will be ignored. + 2. It will create a folder named 'dgupdater_release' in the directory + 3. It will create another folder named 'chunks' in the dgupdater_release folder. + 4. It will then create a new dgupdaterconf.json file in the dgupdater_release folder with the updated data. + 5. It will then divide the files into smaller chunks and save them in the chunks folder. + 6. The commiting process is complete. \ No newline at end of file diff --git a/dgupdater/cli.py b/dgupdater/cli.py index dd5483f..c1d99b4 100644 --- a/dgupdater/cli.py +++ b/dgupdater/cli.py @@ -1,13 +1,14 @@ from click import group from .cli_commands.init.init import init +from .cli_commands.commit.commit import commit @group() def cli(): pass cli.add_command(init) -# cli.add_command() +cli.add_command(commit) # cli.add_command() # cli.add_command() # cli.add_command() diff --git a/dgupdater/cli_commands/commit/commit.py b/dgupdater/cli_commands/commit/commit.py index e69de29..2d87929 100644 --- a/dgupdater/cli_commands/commit/commit.py +++ b/dgupdater/cli_commands/commit/commit.py @@ -0,0 +1,23 @@ +from click import command, option, prompt, echo + +from .func.check_configuration_files_exists import check_configuration_files_exists +from .func.update_version import update_version +from .func.commit_changes import commit_changes + +@command() +@option('--version', '-v', required = True, prompt = 'New version number', help='New version number of the application') +def commit(version: str) -> None: + + check_configuration_files_exists() + + if prompt("Are you sure you want to commit the changes? (y/n): ").lower() in ["y", "yes"]: + echo("Committing...") + + update_version(version) + commit_changes() + + echo("Committed successfully.") + return + + echo("Commit aborted.") + diff --git a/dgupdater/cli_commands/commit/func/__init__.py b/dgupdater/cli_commands/commit/func/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/dgupdater/cli_commands/commit/func/check_configuration_files_exists.py b/dgupdater/cli_commands/commit/func/check_configuration_files_exists.py new file mode 100644 index 0000000..5ac78df --- /dev/null +++ b/dgupdater/cli_commands/commit/func/check_configuration_files_exists.py @@ -0,0 +1,21 @@ +from os.path import exists +from click import echo +from json import load +from ...init.init import dgupdaterconf_json + + +def check_configuration_files_exists() -> bool: + if not exists('dgupdaterconf.json'): + echo("\ndgupdaterconf.json not found. Run 'dgupdater init' first. \nAlso check if you are in the correct directory.") + return False + + with open('dgupdaterconf.json') as f: + dgupdaterconf_json_ = load(f) + + if (dgupdaterconf_json_.keys() != dgupdaterconf_json.keys()): + echo("dgupdaterconf.json is not valid. \nTry 'dgupdater init' again.") + return False + + return True + + diff --git a/dgupdater/cli_commands/commit/func/commit_changes.py b/dgupdater/cli_commands/commit/func/commit_changes.py new file mode 100644 index 0000000..56e1d6f --- /dev/null +++ b/dgupdater/cli_commands/commit/func/commit_changes.py @@ -0,0 +1,52 @@ +from os import makedirs, getcwd +from json import load, dump, dumps +from shutil import rmtree + +from .find_files import find_files +from .create_chunks import create_chunks + +def commit_changes() -> None: + + try: + rmtree('dgupdater_release/chunks') # Removing the chunks directory and its contents if it exists + except FileNotFoundError as _: + pass + + makedirs('dgupdater_release/chunks', exist_ok = True) + + release_json = {} + + release_files = find_files() + # print(release_files) + + for file in release_files: + + file_path = f'{getcwd()}{file}' + try: + with open(file_path, 'r') as f: + # print(file_path) + release_json[file] = f.read() + except UnicodeDecodeError as _: + with open(file_path, 'rb') as f: + content = str(f.read()) + release_json[file] = content + # print(content[:10]) + + release_json_str = dumps(release_json) + + with open('dgupdaterconf.json') as f: + dgupdaterconf_json = load(f) + + no_of_chunks = create_chunks(release_json_str, dgupdaterconf_json['app_name']) #creating chunks as well as getting the number of chunks + + dgupdaterconf_json['no_of_files'] = len(release_files) + dgupdaterconf_json['update_ready'] = False + dgupdaterconf_json['no_of_chunks'] = no_of_chunks + dgupdaterconf_json['files_in_latest_version'] = release_files + + with open(f'{getcwd()}/dgupdater_release/dgupdaterconf.json', 'w') as f: + dump(dgupdaterconf_json, f, indent = 4) + + +if __name__ == "__main__": + commit_changes() \ No newline at end of file diff --git a/dgupdater/cli_commands/commit/func/create_chunks.py b/dgupdater/cli_commands/commit/func/create_chunks.py new file mode 100644 index 0000000..27f954b --- /dev/null +++ b/dgupdater/cli_commands/commit/func/create_chunks.py @@ -0,0 +1,37 @@ +from json import dump +from math import ceil +from tqdm import tqdm + + +def create_chunks(release_json_str: str, app_name: str) -> int: + + max_chunk_size = 1_000_000 # 10 Lakh (or) 1 Million + + release_json_size = len(release_json_str) + + no_of_chunks = ceil(release_json_size / max_chunk_size) + + start = 0 + + with tqdm(total = no_of_chunks, desc = "Creating Chunks", ncols = 110, unit='chunks') as pbar: + + for i in range(no_of_chunks): + chunk_part = i + 1 + chunk_name = f'{app_name}_part{chunk_part}' + + chunk = release_json_str[start:start + max_chunk_size] + chunk = { + 'obj_type': 'chunk', + '_id': chunk_name, + 'chunk_data': chunk + } + + with open(f'dgupdater_release/chunks/{chunk_name}.json', 'w') as f: + dump(chunk, f, indent = 4) + + start += max_chunk_size + pbar.update(1) + + return no_of_chunks + + diff --git a/dgupdater/cli_commands/init/func/find_files.py b/dgupdater/cli_commands/commit/func/find_files.py similarity index 97% rename from dgupdater/cli_commands/init/func/find_files.py rename to dgupdater/cli_commands/commit/func/find_files.py index 0984c51..12a8442 100644 --- a/dgupdater/cli_commands/init/func/find_files.py +++ b/dgupdater/cli_commands/commit/func/find_files.py @@ -39,7 +39,7 @@ def get_ignore_list() -> list: try: with open('.dgupdaterignore', 'r') as file: return [line.split('\n')[0] for line in file.readlines()] - except FileNotFoundError: + except FileNotFoundError as _: return [] if __name__ == "__main__": diff --git a/dgupdater/cli_commands/commit/func/update_version.py b/dgupdater/cli_commands/commit/func/update_version.py new file mode 100644 index 0000000..ed82ed2 --- /dev/null +++ b/dgupdater/cli_commands/commit/func/update_version.py @@ -0,0 +1,10 @@ +from json import load, dump + +def update_version(version: str) -> None: + with open('dgupdaterconf.json') as f: + dgupdaterconf_json = load(f) + + dgupdaterconf_json['version'] = version + + with open('dgupdaterconf.json', 'w') as f: + dump(dgupdaterconf_json, f, indent = 4) diff --git a/dgupdater/cli_commands/init/func/check_app_exists.py b/dgupdater/cli_commands/init/func/check_app_exists.py index f8bbb85..67386a4 100644 --- a/dgupdater/cli_commands/init/func/check_app_exists.py +++ b/dgupdater/cli_commands/init/func/check_app_exists.py @@ -7,7 +7,7 @@ def check_app_exists(name: str, mongostr: str) -> bool: with MongoClient(mongostr) as client: db = client['DGUPDATER'] collections = [collection.lower() for collection in db.list_collection_names()] - except Exception as e: + except Exception as _: echo("Some error occured. Please try again.") if name.lower() in collections: diff --git a/dgupdater/cli_commands/init/func/check_mongo_string.py b/dgupdater/cli_commands/init/func/check_mongo_string.py index b90c88a..4eb29f8 100644 --- a/dgupdater/cli_commands/init/func/check_mongo_string.py +++ b/dgupdater/cli_commands/init/func/check_mongo_string.py @@ -18,7 +18,7 @@ def check_mongo_string(ctx, param, value: str) -> str|None: echo("Verified !!\n") return value + '*VERIFIEDONCE*' # *VERIFIEDONCE* is added to the end of the string to detect second execution of callback of click.options() - except ConnectionFailure as e: + except ConnectionFailure as _: print() raise BadParameter("Enter a Valid MongoDB Connection String") except Exception as e: diff --git a/dgupdater/cli_commands/init/func/create_configuration_files.py b/dgupdater/cli_commands/init/func/create_configuration_files.py index eee793e..adbc8a6 100644 --- a/dgupdater/cli_commands/init/func/create_configuration_files.py +++ b/dgupdater/cli_commands/init/func/create_configuration_files.py @@ -21,7 +21,7 @@ def create_configuration_files(data: dict, app_name: str, mongodbstrd: str) -> N if exists(file): with open(file, 'r') as f: dgupdaterconf_json = load(f) - except JSONDecodeError as e: + except JSONDecodeError as _: pass dgupdaterconf_json['mongodbstrds'][app_name] = mongodbstrd @@ -29,6 +29,9 @@ def create_configuration_files(data: dict, app_name: str, mongodbstrd: str) -> N with open(file, "w") as f: dump(dgupdaterconf_json, f, indent = 4) + with open('.dgupdaterignore', 'w') as f: + f.write('') + if __name__ == "__main__": diff --git a/dgupdater/cli_commands/init/func/create_entry_in_mongodb.py b/dgupdater/cli_commands/init/func/create_entry_in_mongodb.py index 2ed8a26..f6c7121 100644 --- a/dgupdater/cli_commands/init/func/create_entry_in_mongodb.py +++ b/dgupdater/cli_commands/init/func/create_entry_in_mongodb.py @@ -13,7 +13,7 @@ def create_entry_in_mongodb(dgupdaterconf_json: dict[str:any], mongodbstrd: str, collection.insert_one(dgupdaterconf_json) - except Exception as e: + except Exception as _: raise UsageError("Error while creating entry in MongoDB: ") diff --git a/dgupdater/cli_commands/init/init.py b/dgupdater/cli_commands/init/init.py index 86ae912..561511a 100644 --- a/dgupdater/cli_commands/init/init.py +++ b/dgupdater/cli_commands/init/init.py @@ -2,7 +2,6 @@ from .func.check_mongo_string import check_mongo_string from .func.check_app_exists import check_app_exists -from .func.find_files import find_files from .func.create_entry_in_mongodb import create_entry_in_mongodb from .func.create_configuration_files import create_configuration_files @@ -27,6 +26,7 @@ #template dgupdaterconf_json = { + "obj_type": "config", "_id": "", "app_name": "", "version": "", @@ -36,11 +36,10 @@ @command() -@option("--name", "-n", required = True, prompt = arguments["name"]["prompt"], help = arguments["name"]["help"]) -@option("--version", "-v", required = True, prompt = arguments["version"]["prompt"], help = arguments["version"]["help"]) +@option("--name", "-n", required = True, prompt = arguments["name"]["prompt"], help = arguments["name"]["help"]) @option("--mongodbstrd", "-md", callback = check_mongo_string, required = True, prompt = arguments["mongodb_connection_string_write"]["prompt"], help = arguments["mongodb_connection_string_write"]["help"]) @option("--mongodbstrc", "-mc", callback = check_mongo_string, required = True, prompt = arguments["mongodb_connection_string_read"]["prompt"], help = arguments["mongodb_connection_string_read"]["help"]) -def init(name: str, version: str, mongodbstrd: str, mongodbstrc: str) -> None: +def init(name: str, mongodbstrd: str, mongodbstrc: str) -> None: # print(name, version, mongodbstrd, mongodbstrc) echo("\nInitializing this directory for autoupdation...") @@ -55,9 +54,9 @@ def init(name: str, version: str, mongodbstrd: str, mongodbstrc: str) -> None: dgupdaterconf_json["_id"] = name + '_config' dgupdaterconf_json["app_name"] = name - dgupdaterconf_json["version"] = version + dgupdaterconf_json["version"] = "Version will be updated after publishing the changes." dgupdaterconf_json["mongodb_connection_string_client"] = mongodbstrc - dgupdaterconf_json["files_in_latest_version"] = find_files() + dgupdaterconf_json["files_in_latest_version"] = 'Files will be listed after publishing the changes.' create_entry_in_mongodb(dgupdaterconf_json, mongodbstrd, over_write) diff --git a/setup.py b/setup.py index e13954f..0d6aa4a 100644 --- a/setup.py +++ b/setup.py @@ -3,7 +3,7 @@ setup( name="dgupdater", - version="0.2.5", + version="0.3.0", author="DarkGlance", author_email="darkglance.developer@gmail.com", description="A CLI based auto updation assistant tool for python applications", @@ -16,7 +16,7 @@ 'Development Status :: 2 - Pre-Alpha', "Programming Language :: Python :: 3.12", "License :: OSI Approved :: MIT License", - "Operating System :: Microsoft :: Windows", + "Operating System :: Platform Independent", 'Topic :: Utilities' ], python_requires='>=3.6',