From 3e7025a6cbb27249d604f8053da05423ed69e55b Mon Sep 17 00:00:00 2001 From: Enotias Date: Tue, 5 Mar 2024 17:21:48 +0100 Subject: [PATCH 01/12] Update `Publish your own post` (#27) * Update index.md Correction of typos and adding clarity * Fix typo * Fix typo * Add disclaimer * Add markdown example to add images to articles * Fix typo, tks to @ZynoXelek Co-authored-by: ZynoXelek <116194197+ZynoXelek@users.noreply.github.com> --------- Co-authored-by: ClementMabileau Co-authored-by: ZynoXelek <116194197+ZynoXelek@users.noreply.github.com> --- src/content/posts/publish-your-own-post/index.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/content/posts/publish-your-own-post/index.md b/src/content/posts/publish-your-own-post/index.md index 3acd898..0e61201 100644 --- a/src/content/posts/publish-your-own-post/index.md +++ b/src/content/posts/publish-your-own-post/index.md @@ -23,6 +23,9 @@ I **STRONGLY** recommend you to **read every NOTE** at each section's beginning, Also as stated later, if you're struggling with something, you can contact me on Discord: `ctmbl` or open an [Issue](https://github.com/iScsc/blog.iscsc.fr/issues) on the GitHub repository. +> **DISCLAIMER**: This article has been written for `git`/GitHub **beginners**, to publish through the GitHub's web interface. +> If you're used to `git` cloning and GitHub's forking and PR mechanisms you can create the Pull Request as you're used to! + ## 1- Write your post in markdown > **NOTE**: if you're already used to Markdown you can skip to section 2 :slight_smile: @@ -38,6 +41,7 @@ My list: ``` - next paragraph by letting an **empty line** - [hyperlinks](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/Web_mechanics/What_are_hyperlinks) with `[some blabla](http://blabla.com)` + - images with `![](image.png)` - inline `code` with \`inline code\` - code section with: ``` @@ -46,7 +50,8 @@ import pwn \# python code you got it `` ` -Note: remove the whitespace : `` ` -> ``` I wrote it that way because it would be interpreted as code section otherwise... +Note: remove the whitespace : `` ` -> ``` I wrote it that way because it would be +interpreted as code section otherwise... ``` An online markdown editor to get used to it: https://stackedit.io/ There is also a VSCode extension to render markdown in VSCode: `Markdown All in One`. @@ -107,9 +112,9 @@ OK, let's wrap up a bit, here you should already have: Now let's create the associated **Pull Request** to finally share your article with the world! #### 3.4.A- Create PR directly... -> **NOTE**: if you don't end up on the same webpage than me, skip this **subsection** and go to `OR from the repo's page`. +> **NOTE**: if you don't end up on the same webpage than me, skip this **subsection** and go to `OR from the repo's page` (subsection 3.4.B). -> **NOTE 2**: if you're interested in why we're doing this, `git` mechanisms (branch, ...) and `GitHub` ones (repos, Pull Requests, Forks), I should right soon a blog post on the [blog](https://iscsc.fr) +> **NOTE 2**: if you're interested in why we're doing this, `git` mechanisms (branch, ...) and `GitHub` ones (repos, Pull Requests, Forks), I should write soon an article on the [blog](https://iscsc.fr) Now that your files are uploaded you should see: ![](4a1-compare-across-forks.png) @@ -130,7 +135,7 @@ Finally the head banner should look something like: - click `Compare & pull request` ### 3.5- Write a good Pull Request -The hardest part is done, now just fulfill the title and and description of the Pull Request and click `Create Pull Request` +The hardest part is done, now just fulfill the title and description of the Pull Request and click `Create Pull Request` ![](5-pr-title-description.png) ## 4- Review From 27272450d480f31d131368545f483d7f2cfaeeff Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Thu, 14 Mar 2024 23:29:29 +0100 Subject: [PATCH 02/12] Remove useless empty default layout folders (#34) --- src/archetypes/default.md | 6 ------ src/data/.placeholder | 0 src/layouts/.placeholder | 0 src/resources/.placeholder | 0 src/static/.placeholder | 0 5 files changed, 6 deletions(-) delete mode 100644 src/archetypes/default.md delete mode 100644 src/data/.placeholder delete mode 100644 src/layouts/.placeholder delete mode 100644 src/resources/.placeholder delete mode 100644 src/static/.placeholder diff --git a/src/archetypes/default.md b/src/archetypes/default.md deleted file mode 100644 index 00e77bd..0000000 --- a/src/archetypes/default.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: "{{ replace .Name "-" " " | title }}" -date: {{ .Date }} -draft: true ---- - diff --git a/src/data/.placeholder b/src/data/.placeholder deleted file mode 100644 index e69de29..0000000 diff --git a/src/layouts/.placeholder b/src/layouts/.placeholder deleted file mode 100644 index e69de29..0000000 diff --git a/src/resources/.placeholder b/src/resources/.placeholder deleted file mode 100644 index e69de29..0000000 diff --git a/src/static/.placeholder b/src/static/.placeholder deleted file mode 100644 index e69de29..0000000 From db4558614c3c1a38a42ba56894fa9418f2d29218 Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Thu, 14 Mar 2024 23:31:15 +0100 Subject: [PATCH 03/12] Change theme to `ctmbl/poison` and update to newest changes (#33) --- .gitmodules | 2 +- src/themes/poison | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gitmodules b/.gitmodules index 44b656a..186bb79 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +1,3 @@ [submodule "src/themes/poison"] path = src/themes/poison - url = https://github.com/lukeorth/poison.git + url = https://github.com/ctmbl/poison.git diff --git a/src/themes/poison b/src/themes/poison index 62eecf7..8dffa65 160000 --- a/src/themes/poison +++ b/src/themes/poison @@ -1 +1 @@ -Subproject commit 62eecf75eb5bc31a75322f5373754e1d1999761b +Subproject commit 8dffa65ed3a73b60a5097340761ac73b8774cf73 From c5774c14a097646f2b81caaf0fc50b048b9080a4 Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Mon, 18 Mar 2024 09:15:29 +0100 Subject: [PATCH 04/12] Improve `scripts/new_article.py`: leaf bundle, tests, comments (#26) * Handle leaf bundle articles in new_article.py script * Check that response code is 200 when sending notification * Minor comment change * Wrap new_article.py's code in a function * Pass new artciles paths in parameters * Make content base location configurable in new_article.py * Add tests for new_article script * Update .gitignore * Add pytest workflow --- .github/workflows/pytest.yml | 33 ++++++++++ .gitignore | 4 ++ scripts/new_article.py | 60 ++++++++++++------ scripts/test_new_article.py | 68 +++++++++++++++++++++ scripts/test_resources/article_1.md | 9 +++ scripts/test_resources/leaf_bundle/index.md | 9 +++ 6 files changed, 163 insertions(+), 20 deletions(-) create mode 100644 .github/workflows/pytest.yml create mode 100644 scripts/test_new_article.py create mode 100644 scripts/test_resources/article_1.md create mode 100644 scripts/test_resources/leaf_bundle/index.md diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml new file mode 100644 index 0000000..43a6132 --- /dev/null +++ b/.github/workflows/pytest.yml @@ -0,0 +1,33 @@ +name: Pytest + +on: + # Runs on pull requests to check that the website is building without errors + pull_request: + + # Only run if the push to main + push: + branches: + - main + + # Allows you to run this workflow manually from the Actions tab + workflow_dispatch: + +jobs: + # Build job + build: + runs-on: ubuntu-latest + steps: + # Checkout repo + - name: 🛒 Checkout + uses: actions/checkout@v3 + + # Install pytest + - name: 🛠️ Install pytest + run: | + python3 -m pip install pytest pytest-mock + + # Run tests + - name: 🚀 Run pytest + run: | + cd ./scripts/ + pytest diff --git a/.gitignore b/.gitignore index d6e0359..2ab7933 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ +# env files .env.prod .env.dev @@ -10,3 +11,6 @@ certbot/* # hugo build src/public build/blog/* + +# python +**/__pycache__ diff --git a/scripts/new_article.py b/scripts/new_article.py index 9adda8f..14365af 100644 --- a/scripts/new_article.py +++ b/scripts/new_article.py @@ -1,25 +1,45 @@ +import os import sys import re import requests import yaml -for file_path in sys.argv[1:]: - # Check that this is an article file - if re.match("^src/content/posts/.+\.md$", file_path): - # Read YAML Header - with open(file_path, "r") as f: - raw_txt = f.read() - data = yaml.safe_load(raw_txt.split("---")[1]) - - # Get rid of python objects, only keep basic types - for key in data: - if type(data[key]) not in [int, str, float, bool]: - data[key] = str(data[key]) - - # Add URL info - file_name = file_path.split("/")[-1][:-3] - data["url"] = f"https://iscsc.fr/posts/{file_name}" - - # Finally send Data - requests.post("http://iscsc.fr:8001/new-blog", json=data) - print(file_path, file_name, data) +ARTICLE_FILE_BASE_PATH = "src/content/posts/" + +def main(files_paths): + for file_path in files_paths: + # Check that this is an article file + if re.match(f"^{ARTICLE_FILE_BASE_PATH}.+\.md$", file_path): + ## Read YAML Header + with open(file_path, "r") as f: + raw_txt = f.read() + data = yaml.safe_load(raw_txt.split("---")[1]) + + ## Get rid of python objects, only keep basic types + for key in data: + if type(data[key]) not in [int, str, float, bool]: + data[key] = str(data[key]) + + # we have to deal with both possibilities of new article: + # - an article as a .md file which URL is the name + # - a leaf bundle article (https://gohugo.io/content-management/page-bundles/#leaf-bundles): + # it's an article which name is the folder's name and body is in a index.md in this directory + dirname, basename = os.path.split(file_path) + if basename == "index.md": + # leaf bundle: name is directory name + file_name = os.path.basename(dirname) + else: + # direct article file: name is file name + file_name = basename[:-3] # get rid of the `.md` + + ## Add URL info: + data["url"] = f"https://iscsc.fr/posts/{file_name}" + + ## Finally send Data + req = requests.post("http://iscsc.fr:8001/new-blog", json=data) + print(file_path, file_name, data) + assert(req.status_code == 200) + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/scripts/test_new_article.py b/scripts/test_new_article.py new file mode 100644 index 0000000..baae7fa --- /dev/null +++ b/scripts/test_new_article.py @@ -0,0 +1,68 @@ +import pytest + +import new_article + +### DISCLAIMER: +# Whereas other extensions are allowed by HUGO: +# "The extension can be .html, .json or any valid MIME type" +# We only accept Markdown articles and so only parse these +### + + +@pytest.fixture +def mock_requests_post(mocker): + mock_post = mocker.MagicMock() + fake_response = mocker.Mock() + + fake_response.status_code = 200 + mock_post.return_value = fake_response + + mocker.patch("requests.post", mock_post) + mocker.patch("new_article.ARTICLE_FILE_BASE_PATH", "test_resources/") + + yield mock_post + + +def test_new_article_file(mock_requests_post): + new_article.main(["test_resources/article_1.md"]) + + mock_requests_post.assert_called_once_with( + 'http://iscsc.fr:8001/new-blog', + json={ + 'title': 'article title', + 'summary': 'article summary', + 'date': '2024-02-19 10:52:09+01:00', + 'lastUpdate': '2024-02-19 10:52:09+01:00', + 'tags': "['some', 'tags']", + 'author': 'ctmbl', + 'draft': False, + 'url': 'https://iscsc.fr/posts/article_1' + } + ) + +def test_new_leaf_bundle_article(mock_requests_post): + new_article.main(["test_resources/leaf_bundle/index.md"]) + + mock_requests_post.assert_called_once_with( + 'http://iscsc.fr:8001/new-blog', + json={ + 'title': 'leaf bundle title', + 'summary': 'leaf bundle summary', + 'date': '2024-02-19 10:52:09+01:00', + 'lastUpdate': '2024-02-19 10:52:09+01:00', + 'tags': "['leaf', 'bundle']", + 'author': 'ctmbl', + 'draft': False, + 'url': 'https://iscsc.fr/posts/leaf_bundle' + } + ) + +def test_new_branch_bundle(): + # not yet implemented + # https://gohugo.io/content-management/page-bundles/#branch-bundles + pass + +def test_headless_bundle(): + # not yet implemented + # https://gohugo.io/content-management/page-bundles/#headless-bundle + pass \ No newline at end of file diff --git a/scripts/test_resources/article_1.md b/scripts/test_resources/article_1.md new file mode 100644 index 0000000..07ed3b8 --- /dev/null +++ b/scripts/test_resources/article_1.md @@ -0,0 +1,9 @@ +--- +title: "article title" +summary: "article summary" +date: 2024-02-19T10:52:09+01:00 +lastUpdate: 2024-02-19T10:52:09+01:00 +tags: ["some","tags"] +author: ctmbl +draft: false +--- \ No newline at end of file diff --git a/scripts/test_resources/leaf_bundle/index.md b/scripts/test_resources/leaf_bundle/index.md new file mode 100644 index 0000000..a71c101 --- /dev/null +++ b/scripts/test_resources/leaf_bundle/index.md @@ -0,0 +1,9 @@ +--- +title: "leaf bundle title" +summary: "leaf bundle summary" +date: 2024-02-19T10:52:09+01:00 +lastUpdate: 2024-02-19T10:52:09+01:00 +tags: ["leaf","bundle"] +author: ctmbl +draft: false +--- \ No newline at end of file From 78c053816b73ed5a13e20644d46af0630cfcd186 Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Thu, 21 Mar 2024 23:33:25 +0100 Subject: [PATCH 05/12] Deploy on manually triggered workflows (#35) * Trigger deployment on manually triggered workflows * Skip notify job when event is workflow_dispatch: unsupported --- .github/workflows/build_and_deploy.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build_and_deploy.yml b/.github/workflows/build_and_deploy.yml index 1095fed..acd0ee6 100644 --- a/.github/workflows/build_and_deploy.yml +++ b/.github/workflows/build_and_deploy.yml @@ -39,10 +39,10 @@ jobs: path: ./build/blog # Deployment job: heavily inspired from https://swharden.com/blog/2022-03-20-github-actions-hugo/ - # /!\ only triggers on push events + # /!\ only triggers on push events and manually triggered deploy: needs: [build] - if: ${{ github.event_name == 'push' }} + if: ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' }} runs-on: ubuntu-latest steps: - name: 🛠️ Setup build directory @@ -73,8 +73,10 @@ jobs: rsync --archive --stats --verbose --delete ./build/blog/* ${{ secrets.CI_USER_NAME }}@iscsc.fr:${{ secrets.STATIC_WEBSITE_PATH}} # Finally notify of the new article (if any) on the iScsc discord server + # action jitterbit/get-changed-files@v1 doesn't support 'workflow_dispatch' events: https://github.com/jitterbit/get-changed-files/issues/38 notify: needs: [deploy] + if: ${{ github.event_name != 'workflow_dispatch' }} runs-on: ubuntu-latest steps: # Checkout repo, no need to checkout submodule From 846eb0b7c2d0616a2982eb2c2f4d0432066b6da4 Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Sun, 24 Mar 2024 20:34:39 +0100 Subject: [PATCH 06/12] Add author to post info and make author a taxonomy (#37) * Update to ctmbl/poison latest changes: add author info * Make author a taxonomy for the website --- src/config.toml | 1 + src/themes/poison | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/config.toml b/src/config.toml index 36c5741..8866b2b 100644 --- a/src/config.toml +++ b/src/config.toml @@ -103,4 +103,5 @@ pluralizelisttitles = false # removes the automatically appended "s" on sideba [taxonomies] series = 'series' tags = 'tags' + author = 'author' diff --git a/src/themes/poison b/src/themes/poison index 8dffa65..6903687 160000 --- a/src/themes/poison +++ b/src/themes/poison @@ -1 +1 @@ -Subproject commit 8dffa65ed3a73b60a5097340761ac73b8774cf73 +Subproject commit 690368796c75b1567592c9047b0e32f2ae95871d From f04fc87d7c9b2d54169b5725d38c3ad5474e0ac1 Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Mon, 25 Mar 2024 09:40:50 +0100 Subject: [PATCH 07/12] Bump `poison`: Fix author and series URL construction (#38) --- src/themes/poison | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/themes/poison b/src/themes/poison index 6903687..f20c309 160000 --- a/src/themes/poison +++ b/src/themes/poison @@ -1 +1 @@ -Subproject commit 690368796c75b1567592c9047b0e32f2ae95871d +Subproject commit f20c309bdad9e9a98ca1cbf729f416be4c1e42be From 60f7f5ed63f1bed1ad599d4bcdf2d468ad2d89f9 Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Wed, 27 Mar 2024 12:05:37 +0100 Subject: [PATCH 08/12] Fix `/author/` title case (#39) * Change HUGO image repo to floryn90 a klakegg's repo fork * Bump HUGO image to 0.123.7 * Set capitalizelisttitles to false to respect list titles' case (in particular for nicknames) * Fix: change --baseUrl to --baseURL for hugo 0.123.7 * Fix: replace deprecated --verbose by newer --logLevel info --- docker-compose.yml | 4 ++-- src/config.toml | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docker-compose.yml b/docker-compose.yml index d0e24c0..85235eb 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,8 +2,8 @@ version: "3.8" services: builder: - image: klakegg/hugo:0.111.3 - command: --verbose --baseUrl="https://iscsc.fr" --buildFuture + image: floryn90/hugo:0.123.7 + command: --logLevel info --baseURL="https://iscsc.fr" --buildFuture environment: - HUGO_DESTINATION=/build/blog # For maximum backward compatibility with Hugo modules: diff --git a/src/config.toml b/src/config.toml index 8866b2b..c0268c2 100644 --- a/src/config.toml +++ b/src/config.toml @@ -9,6 +9,7 @@ enableEmoji = true # copy paste from https://themes.gohugo.io/themes/poison/#example-config : paginate = 10 pluralizelisttitles = false # removes the automatically appended "s" on sidebar entries +capitalizelisttitles = false # respect list title case; ex with author: ctmbl stays ctmbl not Ctmbl # NOTE: If using Disqus as commenting engine, uncomment and configure this line # disqusShortname = "yourDisqusShortname" From e8335b3571b19a43ef2989d11d36ef536cc660ab Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Fri, 29 Mar 2024 22:39:42 +0100 Subject: [PATCH 09/12] Improve articles: summary, tags and some code syntax highlight (#40) * Improve xlitoni's post summary * Fix wrong turtyo's post summary * Fix turtyo's md code blocks language syntax highlight * Homogenize pots tags --- .../posts/reversing-alien-gibberish-xlitoni.md | 2 +- src/content/posts/web3py-solidity-write-up.md | 14 ++++++-------- src/content/posts/wu_fcsc_2022_a_l_envers.md | 2 +- 3 files changed, 8 insertions(+), 10 deletions(-) diff --git a/src/content/posts/reversing-alien-gibberish-xlitoni.md b/src/content/posts/reversing-alien-gibberish-xlitoni.md index 6faad4d..94e734c 100644 --- a/src/content/posts/reversing-alien-gibberish-xlitoni.md +++ b/src/content/posts/reversing-alien-gibberish-xlitoni.md @@ -1,6 +1,6 @@ --- title: "Write-up of Alien Saboteur (Reversing) CTF HTB Apocalypse 2023" -summary: "Reversing alien gibberish" +summary: "Reversing alien gibberish: a write-up of a Reverse challenge from basic binary analysis to final keygen script and exploit" date: 2023-04-25T17:52:09-02:00 lastUpdate: 2023-04-25T17:52:09-02:00 tags: ["reverse","write-up","Supwn"] diff --git a/src/content/posts/web3py-solidity-write-up.md b/src/content/posts/web3py-solidity-write-up.md index 368df8e..8abae23 100644 --- a/src/content/posts/web3py-solidity-write-up.md +++ b/src/content/posts/web3py-solidity-write-up.md @@ -1,9 +1,9 @@ --- title: "Write-up of The Art of Deception (Blockchain) CTF HTB Apocalypse 2023" -summary: "Simple tutorial to Discord bots using `discord.py`" +summary: "Introduction to Web3 security: an explanation of the logic put behind flagging a Web3 challenge, written in web3py and solidity." date: 2023-03-28T15:08:45-02:00 lastUpdate: 2023-03-28T15:08:45-02:00 -tags: ["web3","solidity","write-up","Supwn"] +tags: ["introduction", "web3", "solidity", "write-up", "Supwn"] author: Turtyo draft: false --- @@ -35,7 +35,7 @@ And the `Get flag` tells us the challenge isn't solved yet. We also have some files that we downloaded at the start of the challenge, let's check what's inside of them: `Setup.sol` -```rust +```solidity // SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.18; @@ -57,7 +57,7 @@ contract Setup { We can for now see with the function `isSolved` that we need to verify `TARGET.lastEntrant() == "Pandora"` `FortifiedPerimeter.sol` -```rust +```solidity // SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.18; @@ -98,7 +98,7 @@ Here we see multiple interesting things. First, we understand what was this "las This `name` function is defined as an `external` function in the `Entrant` interface. In Solidity, the `external` keyword means that the function is called from outside the contract. To read more about function types, you can check the doc [here](https://docs.soliditylang.org/en/v0.8.19/types.html#function-types). Here, it is left to the person interacting with the `enter` function to implement it. In itself, this is not a vulnerability. But the vulnerability comes in the following two lines: -```rust +```solidity require(_isAuthorized(_entrant.name()), "Intruder detected"); lastEntrant = _entrant.name(); ``` @@ -108,7 +108,7 @@ The interesting thing to note here is that the `name` function is called twice, ***"What if we gave a name in the authorized list the first time the function is called and the name Pandora the second time ?"*** I started by writing a solidity file for this (`fake_entrant.sol`) -```rust +```solidity // SPDX-License-Identifier: UNLICENSED pragma solidity ^0.8.18; @@ -203,5 +203,3 @@ Also if you just started working with web3 (as I did before this CTF), the diffe I hope this WU was clear, thank you for reading through it. Turtyo for the Supwn team - -","summary":"An explanation of the logic put behind flagging this challenge, written in web3py and solidity.","createdAt":{"$date":{"$numberLong":"1680016125730"}},"updatedAt":{"$date":{"$numberLong":"1680016125730"}},"__v":{"$numberInt":"0"}} \ No newline at end of file diff --git a/src/content/posts/wu_fcsc_2022_a_l_envers.md b/src/content/posts/wu_fcsc_2022_a_l_envers.md index 25d2ffd..5bbbc85 100644 --- a/src/content/posts/wu_fcsc_2022_a_l_envers.md +++ b/src/content/posts/wu_fcsc_2022_a_l_envers.md @@ -4,7 +4,7 @@ summary: "Initiation à Pwntools" date: 2024-02-14T20:00:00+01:00 author: "Thomas Roberge" draft: false -tags: ["write-up","FCSC","programming","pwntools","FR"] +tags: ["introduction", "programming", "pwntools", "write-up", "FR"] --- > On peut retrouver ce challenge sur [Hackropole](https://hackropole.fr/fr/challenges/misc/fcsc2022-misc-a-l-envers/) From 0bd69d31dac28ddd2e40253af0aaaf68e18a576d Mon Sep 17 00:00:00 2001 From: ZynoXelek <116194197+ZynoXelek@users.noreply.github.com> Date: Sat, 30 Mar 2024 12:06:50 +0100 Subject: [PATCH 10/12] Add blog post : Online Game Article (#30) * Online Game Article (Draft) * Move Article.md to the correct folder Move Article.md to src/content/posts/ * Renamed document + Spelling error * Add the main text of the missing parts. * Add the basic explanation of the server. * Finished writing the first version of the article * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Highlight code as python * Improved some spelling and sentences * Add header * Update Header Tags * Small addition of vocabulary * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Update src/content/posts/multiplayer_online_game.md Co-authored-by: ClementMabileau * Add table of contents and change tags * Improve TOC * Update Table Of Contents --------- Co-authored-by: ClementMabileau --- src/content/posts/multiplayer_online_game.md | 559 +++++++++++++++++++ 1 file changed, 559 insertions(+) create mode 100644 src/content/posts/multiplayer_online_game.md diff --git a/src/content/posts/multiplayer_online_game.md b/src/content/posts/multiplayer_online_game.md new file mode 100644 index 0000000..d42d5b0 --- /dev/null +++ b/src/content/posts/multiplayer_online_game.md @@ -0,0 +1,559 @@ +--- +title: "Trying to make an online multiplayer minigame" +summary: "A simple article to explain how we made a multiplayer online game using python and what we learnt while doing it, from the very basic use of sockets, to the different communication protocols and a bit of optimization." +date: 2024-03-11T16:39:09-02:00 +lastUpdate: 2023-03-26T20:18:09-02:00 +tags: ["iscsc","python","network"] +author: Zyno +draft: false +--- + +## Table of Contents +- [A little introduction](#a-little-introduction) +- [First Step : Successfully sending a simple message to another computer in our LAN](#first-step--successfully-sending-a-simple-message-to-another-computer-in-our-lan) + - [The server-side](#the-server-side-will-look-like-this-) + - [The client-side](#on-the-client-side-it-will-be-this-) +- [Simple online implementation to play a basic game](#simple-online-implementation-to-play-a-basic-game) +- [First improvement of the connection](#first-improvement-of-the-connection) + - [Client-side improvements](#client-side-improvements) + - [Server-side improvements](#server-side-improvements) +- [But, how to reduce ping?](#but-how-to-reduce-ping) +- [The road to UDP connection](#the-road-to-udp-connection) + - [What is UDP and why would we want to use that?](#what-is-udp-and-why-would-we-want-to-use-that) + - [Using UDP sockets instead of TCP sockets](#using-udp-sockets-instead-of-tcp-sockets-) +- [Now : A quite stable game to play](#now--a-quite-stable-game-to-play) + - [The UDP client-side](#the-udp-client-side-now-looks-like-this-) + - [The UDP server-side](#on-the-server-side-the-code-for-udp-is-now-designed-like-this-) +- [Future Improvements to do...](#future-improvements-to-do) + +## A little introduction + +This project, called Haunted Chronicles, started when we wanted to introduce ourselves to online multiplayer games and the code behind it. + +Naturally, we decided to code using python because it was simpler to begin with - everyone knew how to code in Python - and because we just wanted to discover the notion, not to code an AAA game. + +So, we began with a little documentation and we discovered the magic of **sockets**! + +For those who don't know anything about them, it is basically a glass bottle in which you put your message, and that you then throw away in the approximate direction of your friend, hoping for them to receive it. (Here is the very looong documentation of python : https://docs.python.org/3/library/socket.html) + +## First Step : Successfully sending a simple message to another computer in our LAN +The first step here to understand how all this socket-stuff works is to try to make a simple 'email' system. The goal here is to make a python script able to send a predefined message to another computer. + +To do so, we need to define a server script and a client script. +The server will initialize a socket which will listen to incoming messages, while the client will initialize a socket to send messages to the server. +We wrote this code thanks to the [python documentation's example](https://docs.python.org/3/library/socketserver.html#socketserver-tcpserver-example) + +### The server-side will look like this : +```py +import socketserver + +class MyTCPHandler(socketserver.BaseRequestHandler): + """ + The request handler class for our server. + + It is instantiated once per connection to the server, and must + override the handle() method to implement communication to the + client. + """ + + def handle(self): + # self.request is the TCP socket connected to the client + self.data = self.request.recv(1024).strip() + + in_ip = self.client_address[0] + + print("{} wrote:".format(in_ip)) + in_data = str(self.data,'utf-16') + print(in_data) + + out = "Hello client, you correctly sent your message : '" + in_data + "' to the server." + + print(">>> ",out,"\n") + self.request.sendall(bytes(out,'utf-16')) + + + +# ----------------------- Main ----------------------- + +if __name__ == "__main__": + HOST, PORT = str(IP), 9998 + socketserver.TCPServer.allow_reuse_address = True + # Create the server, bound to the given IP on port 9998 + with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server: + print("HOST = ",IP,"\nPORT = ",PORT,"\n") + # Activate the server; this will keep running until you + # interrupt the program with Ctrl-C + server.serve_forever() +``` + +Ok so it may seem a big difficult to understand, but not everything here is important to really understand, and you will see further that what we did was in fact easier to understand in the end. + +But to give some explanation, sockets are 'objects' in Python, so they are custom classes. Here, the class we define (`MyTCPHandler`) is the way the socket must react to incoming messages, not the socket itself which is already coded. It is defined in the `handle(self)` method. What it does here is that it reads the incoming message with `self.data = self.request.recv(1024).strip()`. The data is stored in bytes here. The client address is automatically stored in `self.client_address` as the name is explicit enough. We then just print the client ip and data in the server terminal to be able to check that we correctly received the message (after converting the bytes to str with the `utf-16` convention). + +And then, we just send it back to the client with a little confirmation message after converting it back to bytes with the lines : +```py +out = "Hello client, you correctly sent your message : '" + in_data + "' to the server." +self.request.sendall(bytes(out,'utf-16')) +``` + +So, now that we defined the way we want our socket to react to messages, we just need to initialize it! To do so, we use a [context manager](https://realpython.com/python-with-statement/), which is for example the `open folder` form in Python `with open(...) as f:`. + +Here, it is the initialization of our socket : +```py +with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server: + print("HOST = ",IP,"\nPORT = ",PORT,"\n") + # Activate the server; this will keep running until you + # interrupt the program with Ctrl-C + server.serve_forever() +``` +What happens here is we initialize a new socket with the given address `(HOST, PORT)` where HOST is the IP of our server (it is the IP of the computer that will run this program). To obtain it, you can either use some websites, your terminal, or use the next Python lines : +```py +import socket + +def extractingIP(): + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + s.connect(("8.8.8.8", 80)) + ip = s.getsockname()[0] + s.close() + return(ip) + +IP = extractingIP() +``` + +Once the socket is initialized, we verify that it has the correct address `(IP, PORT)` by printing it, and then we use the `serve_forever()` method which makes the server wait for new messages indefinitely and, when he receives one, executes the code we defined in the `handle(self)` method under the `MyTCPHandler` class. Once a message has been processed, it waits for another one to arrive. + +The only way to make it stop **for now** is to use ctrl+C in the terminal to shut down the process, but we can obviously implement a better way to shut down the server through the `handle(self)` method for instance (example : if the server receives `"STOP"`, the socket closes itself and the server code terminates). + +Don't mind the +`socketserver.TCPServer.allow_reuse_address = True`, this line is not mandatory but it allows the server to be closed and reopened with the exact same `(IP, PORT)`. If you don't write this line, the only thing that will change is the fact that when closing your server, you will have to wait about 30 seconds to be able to open a new server with the same address. + +### On the client-side, it will be this : + +```py +from socket import * + +SERVER_IP = "localhost" +SERVER_PORT = 9998 + +def send(msg="Hello server!"): + """Sends the given message to the server. + """ + + with socket(AF_INET, SOCK_STREAM) as sock: + # send data + sock.connect((SERVER_IP, SERVER_PORT)) + sock.sendall(bytes(msg, "utf-16")) + + # receive answer + answer = str(sock.recv(1024*2), "utf-16") + + return answer + +while True: + msg = input("What message do you want to send?") + print(send(msg)) +``` + +As you can see, we use the same kind of code to initialize a client socket. This time, we just don't need to define our own handler as it was the case for the server, and we can simply use the basic `socket` library instead of the `socketserver`. Just be aware that the socket object is `socket.socket` and that its parameters are `socket.AF_INET` and `socket.SOCK_STREAM`, but you can avoid mistakes by importing everything from socket as we did here with `from socket import *`. + +Then, we simply collect a message from the user in the terminal, and we send it to the server through the `sock.sendall()` method in our `send()` function. We then wait for the server to answer with `sock.recv()` and we print it. + +In this code, we first defined the server ip and port. Here, `"localhost"` is the best way to send the message to yourself without searching for your own ip. This way, you can just execute the server code in an instance of your terminal, and this code in another and try to send yourself some messages! + +Then, you can setup the server on another computer and try to communicate with it by changing this IP to the correct one. + +## Simple online implementation to play a basic game +Well, to make a simple game, you must implement a visual interface and rules in order to make this a bit more interactive, but the online part is in fact almost done! +We used pygame in order to make a small map where squares - which are players - will be able to move. + +The only 'new' thing we need to do is to formalize these messages to make the server understand client's actions. +To do so, we decided that the clients would only send their inputs to the server, and that the server would compute the players' new positions and send them back to the clients. This will implement a semi anti-cheat as players won't be able to directly send their positions to the server, and thus try to teleport. However, it will increase the amount of calculations required by the server and may cause some more lags in case of huge computations due to some game rules later. + +We thus decided to implement some basic formalized messages to communicate with the server which are also defined in the [README.md](https://github.com/iScsc/Haunted-Chronicles) on the github page of the project (There are more messages than the ones described here since the game evolved, but here are the first one we used) : + +* **Connection** : The client sends `CONNECT END` to the server, which sends back `CONNECTED END` if the connection succeeded. +* **Clients' inputs** : The client sends `INPUT END` to the server where `` can be either `L` for left, `R` for right, `U` for up, `D` for down or `.` if there are no inputs. The server computes the new position and sends back the new state of the game with `STATE END` where `` is the list of player structures, which store the player's name, color and position. +* **Disconnection** : The clients sends `DISCONNECT END` and receives `DISCONNECTED END` if the server has correctly destroyed the client's player structure. The client then closes. + +The `` fields are to replace with the correct values. For instance, for the connection : `CONNECT Zyno END`. The `END` field allows the server to easily recognize the end of a formatted message, and the conformity of it. The formatted messages begin with a `` field which allows the server to recognize the rule that must be applied on this kind of incoming message. + +## First improvement of the connection +Yet, this is not optimized at all. In fact, what we do here is we create a new socket, send a message, then automatically destroy this socket (when exiting the `with` indent), and then start all over from the beginning. +Obviously, this is not the way it should be, and we can improve this by creating a socket at first, and then keeping it open as long as the client is connected. + +### Client-side improvements + +Instead of this : +```py +def send(input="INPUT " + USERNAME + " . END"): + """Send a normalized request to the server and listen for the normalized answer. + + Args: + input (str): Normalized request to send to the server. Defaults to "INPUT . END". + + Returns: + str: the normalized answer from the server. + """ + + global PING + + with socket(AF_INET, SOCK_STREAM) as sock: + t = time.time() + + # send data + sock.connect((SERVER_IP, SERVER_PORT)) + sock.sendall(bytes(input, "utf-16")) + + + # receive answer + answer = str(sock.recv(1024*2), "utf-16") + + PING = int((time.time() - t) * 1000) + + return answer +``` +We now write this : +```py +SOCKET = None + +PING = None # To display the approximate ping with the server + +# +# +# More code for display and things like that... +# +# + +def send(input="INPUT " + USERNAME + " . END"): + """Send a normalized request to the server and listen for the normalized answer. + + Args: + input (str): Normalized request to send to the server. Defaults to "INPUT . END". + + Returns: + str: the normalized answer from the server. + """ + + global PING + global SOCKET + + # Initialization, when the socket has not been created yet + if (SOCKET == None and input[0:7] == "CONNECT"): + SOCKET = socket(AF_INET, SOCK_STREAM) + SOCKET.settimeout(SOCKET_TIMEOUT) + SOCKET.connect((SERVER_IP, SERVER_PORT)) + + + # Usual behavior + if SOCKET != None: + t = time.time() + + # send data + try: + SOCKET.sendall(bytes(input, "utf-16")) + + # receive answer + answer = str(SOCKET.recv(1024*2), "utf-16") + + PING = int((time.time() - t) * 1000) + + return answer + except: + exitError("Loss connection with the remote server.") +``` + +So, as you can see, we stopped using the `with` magic formula and we now initialize our socket in a GLOBAL variable named `SOCKET`. In fact, we detect the first need of defining our socket when the formalized `CONNECT` message is used, and that's why we don't initialize it before, in case the message would be wrong, but also to be able in some peculiar cases to disconnect and reconnect with a new socket. + +The rest of the code is really the same function as we used before. We send a formalized message to the server and then wait for the answer. This answer is interpreted to display the correct players at the correct positions, and even the approximate ping with the server which we simply compute with the time the answer took to come back, starting just before we sent our own message. + +It is in another function but we detect the disconnection when the client presses the escape button or closes the window, and we then executes this important code : +```py + SOCKET.close() + SOCKET = None +``` +It is only two lines but it is really important to not forget to close the initialized sockets when they are not needed anymore. And putting SOCKET to None allows us to reconnect to another server if wanted. + +Obviously, we also improved the server-side on the exact same basis. +Another key library we had to use was the threading library which allows to emulate threads in python. It was mandatory to display images with pygame while keeping sending messages to the server and receiving answers from it. + +### Server-side improvements + +For the server-side, our main became this : +```py +from socket import * +from threading import * + +def main(): + global MAINSOCKET + global LOCK + + # Initialization + if MAINSOCKET == None: + MAINSOCKET = socket(AF_INET, SOCK_STREAM) + MAINSOCKET.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) + MAINSOCKET.bind((IP, PORT)) + MAINSOCKET.listen(BACKLOG) + + if LOCK == None: + LOCK = Lock() + + print("Server opened with :\n - ip = " + str(IP) + "\n - port = " + str(PORT)) + + listener_new = Thread(target=listen_new) + manager_server = Thread(target=manage_server) + listener_old = Thread(target=listen_old) + + listener_new.start() + manager_server.start() + listener_old.start() + +if __name__ == "__main__": + main() +``` +Here we initialize the `MAINSOCKET` which is a socket from the basic `socket` library, and which will only be responsible for the connection attempts. We use the `socket.bind(address)` method to make it listens for incoming messages at the given `(IP, PORT)`. We then configure the maximum number of connection attempts the socket can queue with the `socket.listen(BACKLOG)` method. In our case, we used `BACKLOG = 1` and it is good to know that this parameter should usually be between 1 and 5 (usually 5 is the system-dependant maximum possible value). So our server will only accept a single connection each time we use the further explained `socket.accept()` method. + +We can see that we also defined our threads and started them. + +I won't show the complete code of these threads but to resume, the `listen_new` thread is the one that use the `MAINSOCKET` to accept new connections. The `listen_old` is the one that uses the newly created sockets to listen for already connected users. And the `manage_server` thread is a thread that allows the server to take commands from its terminal in order to change some parameters or close the server. + +So, let's start with the new way to accept connections. In the `listen_new` function, we wrote : +```py +sock, addr = MAINSOCKET.accept() +in_ip = addr[0] + +data = sock.recv(1024).strip() + +print("{} wrote:".format(in_ip)) +in_data = str(data,'utf-16') +print(in_data) + +out = processRequest(in_ip ,in_data) +message = out.split(' ') + +if message[0]=="CONNECTED": + LOCK.acquire() + username = message[1] + dicoSocket[username] = (sock, addr) + LOCK.release() + +print(">>> ",out,"\n") +try: + sock.sendall(bytes(out,'utf-16')) +``` + +Ok so, the `MAINSOCKET.accept()` method listens for new connections that use the `SOCKET.connect((SERVER_IP, SERVER_PORT))` we saw on the client-side. When a connection succeeds, this method creates another socket, binds it to the client's socket and returns both the newly created socket and the address of the client. After this, when the client uses the `SOCKET.sendall()` method, the client will in fact send the data to this newly created socket. + +Then, we receive the connection data from the client and try to connect it to the server through our `processRequest(ip, data)` function. If it succeeds, we will send back a message of the type `CONNECTED END`. If it the case, we use our `LOCK` object (class from the `threading` library) which will temporary block the other threads with `LOCK.acquire()` while we store the newly created socket in a global list. Then, the `LOCK.release()` function will resume the threads. + + +Then, to process the data sent to the already connected clients, we use our `listen_old` function, in which we wrote : +```py +for elt in waitingDisconnectionList: + username, sock, addr = elt[0], elt[1], elt[2] + dicoSocket.pop(username) + + # deco remaining player with same ip if needed. + for username in dicoJoueur: + if dicoJoueur[username].ip == addr[0]: + dicoJoueur.pop(username) + break + + sock.close() +waitingDisconnectionList = [] + + +LOCK.acquire() +for username in dicoSocket: + sock = dicoSocket[username][0] + addr = dicoSocket[username][1] + + data = sock.recv(1024).strip() + + in_ip = addr[0] + + print("{} wrote:".format(in_ip)) + in_data = str(data,'utf-16') + print(in_data) + + out = processRequest(in_ip ,in_data) + message = out.split(" ") + + if message[0]=="DISCONNECTED": + username = message[1] + waitingDisconnectionList.append((username, sock, addr)) + + print(">>> ",out,"\n") + try: + sock.sendall(bytes(out,'utf-16')) + except: + waitingDisconnectionList.append((username, sock, addr)) +LOCK.release() +``` + +The first loop is made to disconnect clients that sent the `DISCONNECT END` message. The second loop which is in the `LOCK.acquire()` state process data from the already connected clients, thanks to the socket dictionary we used to store the newly created sockets. This code would crash if new clients connected and if the dictionary changed during the for loop, so that's why we lock the other threads. + +Yet, the code is very similar to the client side here, but we first listen for data, and then send our answer back. + +## But, how to reduce ping? +Yet, when several players connect (more than 3 in average), clients start to suffer from increasing ping, which end up creating seconds of latency for players' movements. But how does this happen? +It seems the server is overcrowded! In fact, we assume that we were DDOSing our own server by sending way too many messages at the same time... + +A first thing we could do is to reduce the frequency of communications with the server to reduce the ping. Indeed it works, but it also make movements less smooth, and ask to change the way we designed the game. Whatever the solution we develop next, this is a good thing to do when possible, because it will greatly help the server and reduce its charge. + +But we will now look into another issue we had with this code, and that I didn't talk much about when explaining sockets : its communicating protocol. + +## The road to UDP connection +### What is UDP and why would we want to use that? +The thing is, from the very beginning of this project, we learnt how to use sockets with Python, but only using the TCP protocol, which is very **NOT** optimal for video games. + +For those who don't know, the TCP protocol means that your communications look like this : +- You establish a communication with an IP sending something like : "I want to talk with you." +- You wait for an answer that says : "Ok, let's talk." +- You send the message you first wanted to send : "Hello I am Zyno and happy to meet you!" +- You wait for the receiver to send back to you : "I have correctly received your message." + +And this is a very simplified vision of it, because the TCP Protocol also runs several tests to assure there has been no loss during the communication. And it even make the frequency of communication vary if it thinks that the server is overwhelmed by many messages. +To resume, when you want to make a game, in which losing a single frame of a 60-FPS game is not a problem at all, and you use a way of communicating with the server that may make your client wait before it is allowed to send messages, you are definitely not using the good communication protocol. + +On the other hand, let me introduce you to the UDP protocol. This amazing communication protocol basically makes your communications look like this : +- You send the only message you wanted to send : "Hello I am Zyno and happy to meet you!" + +And that's it! So, obviously, you may lose some messages in the process, and you won't know it. You don't have TCP's errors detection and correction algorithms either. But as I said earlier, we do not really suffer from a lack of a message every 5 ms in a video game. + +### Using UDP sockets instead of TCP sockets : + +Now, let's go back to our code. Using UDP sockets in Python isn't really that big of a deal. In fact, it's almost the same! + +Look at this : +- This is TCP : +```py +sock = socket(AF_INET, SOCK_STREAM) +``` +- And this is UDP : +```py +sock = socket(AF_INET, SOCK_DGRAM) +``` +`SOCK_STREAM` means TCP, and `SOCK_DGRAM` UDP and *voilà*! + +Ok, it is not **that** simple. The way you previously used this socket has changed a bit as well. Let's see which are the affected functions : + +Previously, we were using `server_sock.listen(BACKLOG)` to make a previously bound socket listen to connections, and then `sock, addr = server_sock.accept()` to make it accept connections. On the client side, we used `client_sock.connect((SERVER_IP, SERVER_PORT))` in order to connect our client socket to the listening socket. The server then generated a new socket - named sock here - and the client was now communicating with the server through its dedicated and newly created socket, using the `sock.sendall()` and `sock.recv()` functions. + +With UDP sockets, these functions have changed a little : + +There is no `sock.listen()`, `sock.accept()` nor `sock.connect()` anymore. However, the `sock.bind((IP, PORT))` function still exists and shall be used to make a socket work as a server which listen at a given IP and PORT. + +However, everything else now works with only two functions : +```py +sock.sendto(bytes(message_to_send, "utf-8"), (SERVER_IP, SERVER_PORT)) +``` +And : +```py +data, addr = sock.recvfrom(MESSAGES_LENGTH) + +message_received = str(data.strip(), "utf-8") # Converting the received bytes into an str +``` +Where `message_to_send`, which is here a string, is the data to send. It is firstly converted into bytes with the utf-8 encoding. You can also send any kind of data as long as you send it using bytes. The address you send the data to is given by the second parameter : `(SERVER_IP, SERVER_PORT)`. +To receive data, the `recvfrom()` function takes the maximum length of the message you want to receive (in bytes). It returns both the `data` in bytes, and the address `addr = (IP, PORT)` from which the data has been sent. In our case, we convert it back into a string using the utf-8 decoding. (We did change our encoding because as our formatted messages gained new keywords, and thus length, we wanted to reduce their sizes and we decided to lose the utf-16 characters to do so.) + +And that's it! Now, it's time for you to think about how you will use these two simple functions in order to make what you want! + +## Now : A quite stable game to play +### The UDP client-side now looks like this : + +Here is the initialization of the socket on the client-side : +```py +SOCKET = socket(AF_INET, SOCK_DGRAM) +SOCKET.settimeout(SOCKET_TIMEOUT) +``` +`SOCKET_TIMEOUT` being 0.5s in our case. + +The data is sent to the server using : +```py +SOCKET.sendto(bytes(input, "utf-8"), (SERVER_IP, SERVER_PORT)) +``` + +The data is received using : +```py +data, addr = SOCKET.recvfrom(MESSAGES_LENGTH) +``` + +When we exit the game, the client finishes by closing the socket using the usual : +```py +SOCKET.close() +``` + +### On the server-side, the code for UDP is now designed like this : + +The initialization is : +```py +MAINSOCKET = socket(AF_INET, SOCK_DGRAM) +MAINSOCKET.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) # Allows for the server to be reopened with the same ip immediately after being closed. +MAINSOCKET.settimeout(TIMEOUT) +MAINSOCKET.bind((HOST, PORT)) +``` + +The server receives the data from clients with : +```py +data, addr = MAINSOCKET.recvfrom(MESSAGES_LENGTH) +``` + +and send back data with : +```py +MAINSOCKET.sendto(bytes(out,'utf-8'), addr) +``` + +However, we give each client a dedicated socket (linked to a given port) to talk to : +```py +if message[0]=="CONNECTED": # Detect connection + sock = socket(AF_INET, SOCK_DGRAM) + sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) + sock.settimeout(TIMEOUT) + + # port attribution + port = availablePorts[0] # use a free port for this new client + out = message[0] + " " + str(port) + for s in message[1:]: + out += (" " + s) + # Add the information of the new port in the connection message + # out = CONNECTED WALLS STATE END + + sock.bind((HOST, port)) + availablePorts.remove(port) + + username = message[1] + dicoSocket[addr] = (sock, username) # Keep the information of the link between sockets and players +``` + +When a client has its own dedicated socket, it receives the information of the new port in the connection confirmation, and changes the port it sends messages to using the command : +```py +SERVER_PORT = int(portStr) +``` +With `portStr` being the extract of the connection message (the second word of the answer). + +After that, clients sends their messages to their own dedicated socket. We detect on the server side which sockets have received data using the lines : +```py +sockets = [MAINSOCKET] + [dicoSocket[addr][0] for addr in dicoSocket] + +if sockets != []: + inSockets, _, _ = select.select(sockets, [], [], TIMEOUT) +``` +Because the select module allows to efficiently (low level) keep only sockets that have received data. + +Once sockets that have received data has been selected, a for loop on them to apply the usual reception and answering code allows for each client to send their inputs and receive the new state of the game. + +Finally, don't forget to close every socket before completely closing the server, including the MAINSOCKET. +When a client disconnects, its socket can be closed as well and its port can be add back in the available ports list: +```py +availablePorts.append(port) +sock.close() +``` + +## Future Improvements to do... +To keep on improving the performances of the online system, we worked on a thread based system in which both clients and the server would have one thread to listen for messages, and one thread to send their messages. In this scenario, the server sends automatically every few milliseconds the current state of the game to every clients connected to the server, while each client sends their input continuously. + +Another way to improve the ping that we did not implement yet is to make clients stop sending permanently all their inputs. Instead, clients would only send their new inputs when the player changes input. This way, the server would receive way less messages and it would instead store the last input made by each player, and assume it is their current input as long as they do not send another one. +This would work by sending a rack of several messages when changing input to be sure the server has correctly received it, and by asking for a confirmation. +In this case, the server would make the state of the game update every X ms, with the stored inputs of every player, and automatically send back to everyone the new state of the game. + +Finally, another way to make the game look a lot smoother would be to let clients assume and compute the next frames of the game without waiting for the actual computations of the server. This could help make the game look smoother even when the connection is not stable, and it is what is done in most online games nowadays. From bd17ae150e85cedb076bdcd78c1472680f9cf966 Mon Sep 17 00:00:00 2001 From: ZynoXelek <116194197+ZynoXelek@users.noreply.github.com> Date: Sun, 7 Apr 2024 14:39:58 +0200 Subject: [PATCH 11/12] Update online game article (#41) * Shorten ToC for ease of reading * Add a link to the repository * Add multiple permalinks to the actual code to help understanding how it works in the 'real' code * Last Update Date --- src/content/posts/multiplayer_online_game.md | 43 +++++++++----------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/src/content/posts/multiplayer_online_game.md b/src/content/posts/multiplayer_online_game.md index d42d5b0..e3dd5a2 100644 --- a/src/content/posts/multiplayer_online_game.md +++ b/src/content/posts/multiplayer_online_game.md @@ -1,8 +1,8 @@ --- title: "Trying to make an online multiplayer minigame" summary: "A simple article to explain how we made a multiplayer online game using python and what we learnt while doing it, from the very basic use of sockets, to the different communication protocols and a bit of optimization." -date: 2024-03-11T16:39:09-02:00 -lastUpdate: 2023-03-26T20:18:09-02:00 +date: 2024-03-11T18:39:09+0200 +lastUpdate: 2024-04-07T14:24:12+0200 tags: ["iscsc","python","network"] author: Zyno draft: false @@ -11,26 +11,18 @@ draft: false ## Table of Contents - [A little introduction](#a-little-introduction) - [First Step : Successfully sending a simple message to another computer in our LAN](#first-step--successfully-sending-a-simple-message-to-another-computer-in-our-lan) - - [The server-side](#the-server-side-will-look-like-this-) - - [The client-side](#on-the-client-side-it-will-be-this-) - [Simple online implementation to play a basic game](#simple-online-implementation-to-play-a-basic-game) - [First improvement of the connection](#first-improvement-of-the-connection) - - [Client-side improvements](#client-side-improvements) - - [Server-side improvements](#server-side-improvements) - [But, how to reduce ping?](#but-how-to-reduce-ping) - [The road to UDP connection](#the-road-to-udp-connection) - - [What is UDP and why would we want to use that?](#what-is-udp-and-why-would-we-want-to-use-that) - - [Using UDP sockets instead of TCP sockets](#using-udp-sockets-instead-of-tcp-sockets-) - [Now : A quite stable game to play](#now--a-quite-stable-game-to-play) - - [The UDP client-side](#the-udp-client-side-now-looks-like-this-) - - [The UDP server-side](#on-the-server-side-the-code-for-udp-is-now-designed-like-this-) - [Future Improvements to do...](#future-improvements-to-do) ## A little introduction This project, called Haunted Chronicles, started when we wanted to introduce ourselves to online multiplayer games and the code behind it. -Naturally, we decided to code using python because it was simpler to begin with - everyone knew how to code in Python - and because we just wanted to discover the notion, not to code an AAA game. +Naturally, we decided to code using python because it was simpler to begin with - everyone knew how to code in Python - and because we just wanted to discover the notion, not to code an AAA game. You can find it on [our github](https://github.com/iScsc/Haunted-Chronicles)! So, we began with a little documentation and we discovered the magic of **sockets**! @@ -461,31 +453,34 @@ And that's it! Now, it's time for you to think about how you will use these two ## Now : A quite stable game to play ### The UDP client-side now looks like this : -Here is the initialization of the socket on the client-side : +[Here](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/client.py#L451) is the initialization of the socket on the client-side : ```py SOCKET = socket(AF_INET, SOCK_DGRAM) SOCKET.settimeout(SOCKET_TIMEOUT) ``` `SOCKET_TIMEOUT` being 0.5s in our case. -The data is sent to the server using : +The [data is then sent to the server](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/client.py#L469) using : ```py SOCKET.sendto(bytes(input, "utf-8"), (SERVER_IP, SERVER_PORT)) ``` -The data is received using : +The [data from the server is received](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/client.py#L483) using : ```py data, addr = SOCKET.recvfrom(MESSAGES_LENGTH) ``` -When we exit the game, the client finishes by closing the socket using the usual : +When we exit the game, the client finishes by [closing the socket](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/client.py#L636) using the usual : ```py SOCKET.close() ``` +Don't forget these lines **every time** the process terminates! + +You can find the whole code [here](https://github.com/iScsc/Haunted-Chronicles/blob/main/client.py) but there are lots of aspects that were not discussed here because this article is focused on the online part only (A huge part of the client code is dedicated to the display of the game). ### On the server-side, the code for UDP is now designed like this : -The initialization is : +The [initialization](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/server.py#L1072) is : ```py MAINSOCKET = socket(AF_INET, SOCK_DGRAM) MAINSOCKET.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) # Allows for the server to be reopened with the same ip immediately after being closed. @@ -493,17 +488,17 @@ MAINSOCKET.settimeout(TIMEOUT) MAINSOCKET.bind((HOST, PORT)) ``` -The server receives the data from clients with : +The server [receives the data from clients](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/server.py#L854) with : ```py data, addr = MAINSOCKET.recvfrom(MESSAGES_LENGTH) ``` -and send back data with : +and [send back data](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/server.py#L894) with : ```py MAINSOCKET.sendto(bytes(out,'utf-8'), addr) ``` -However, we give each client a dedicated socket (linked to a given port) to talk to : +However, we give each client a [dedicated socket](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/server.py#L869) (linked to a given port) to talk to : ```py if message[0]=="CONNECTED": # Detect connection sock = socket(AF_INET, SOCK_DGRAM) @@ -525,13 +520,13 @@ if message[0]=="CONNECTED": # Detect connection dicoSocket[addr] = (sock, username) # Keep the information of the link between sockets and players ``` -When a client has its own dedicated socket, it receives the information of the new port in the connection confirmation, and changes the port it sends messages to using the command : +When a client has its own dedicated socket, it receives the information of the new port in the connection confirmation, and [changes the port it sends messages to](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/client.py#L337) using the command : ```py SERVER_PORT = int(portStr) ``` With `portStr` being the extract of the connection message (the second word of the answer). -After that, clients sends their messages to their own dedicated socket. We detect on the server side which sockets have received data using the lines : +After that, clients sends their messages to their own dedicated socket. We [detect on the server side which sockets have received data](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/server.py#L847) using the lines : ```py sockets = [MAINSOCKET] + [dicoSocket[addr][0] for addr in dicoSocket] @@ -542,13 +537,15 @@ Because the select module allows to efficiently (low level) keep only sockets th Once sockets that have received data has been selected, a for loop on them to apply the usual reception and answering code allows for each client to send their inputs and receive the new state of the game. -Finally, don't forget to close every socket before completely closing the server, including the MAINSOCKET. -When a client disconnects, its socket can be closed as well and its port can be add back in the available ports list: +Finally, don't forget to close **every socket** before completely closing the server, including the MAINSOCKET. +When a client disconnects, its [socket can be closed](https://github.com/iScsc/Haunted-Chronicles/blob/eff735a3cd78b7b9f908d8238945b58e7827e43b/server.py#L967) as well and its port can be add back in the available ports list: ```py availablePorts.append(port) sock.close() ``` +Same as before, you can find the whole server code [here](https://github.com/iScsc/Haunted-Chronicles/blob/main/server.py) but a huge part of the code is dedicated to shadow computations and messages processing. These main aspects were not explained here since it was not the original goal of the article. + ## Future Improvements to do... To keep on improving the performances of the online system, we worked on a thread based system in which both clients and the server would have one thread to listen for messages, and one thread to send their messages. In this scenario, the server sends automatically every few milliseconds the current state of the game to every clients connected to the server, while each client sends their input continuously. From 9f1ba5339438086461f55f43e0bee1c78bba38d3 Mon Sep 17 00:00:00 2001 From: ClementMabileau Date: Fri, 12 Apr 2024 20:46:43 +0200 Subject: [PATCH 12/12] Add Minishell Write Up - THCon 2024 (#42) * Add Minishell write up article from THCon 2024 * Update src/content/posts/minishell_wu_pwn.md Co-authored-by: ZynoXelek <116194197+ZynoXelek@users.noreply.github.com> * Add specific links to the code source * Add a conclusion --------- Co-authored-by: ZynoXelek <116194197+ZynoXelek@users.noreply.github.com> --- src/content/posts/minishell_wu_pwn.md | 139 ++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 src/content/posts/minishell_wu_pwn.md diff --git a/src/content/posts/minishell_wu_pwn.md b/src/content/posts/minishell_wu_pwn.md new file mode 100644 index 0000000..f0b23ec --- /dev/null +++ b/src/content/posts/minishell_wu_pwn.md @@ -0,0 +1,139 @@ +--- +title: "Minishell (pwn) Write-Up CTF ThCon 2024" +summary: "Good introduction to basic heap buffer overflow through a custom vulnerable minimalistic shell in C" +date: 2024-04-07T12:32:53+0200 +lastUpdate: 2024-04-07T12:32:53+0200 +tags: ["pwn", "introduction", "write-up", "Supwn"] +author: ctmbl +draft: false +--- + +> **IMPORTANT**: You can also find this WU (and others), with **the source code** [on my GitHub](https://github.com/ctmbl/ctf-write-ups/tree/main/THCon-2024) + +## Basics + +First of all we don't have binaries associated with the challenge so I add to compile them: +``` +gcc log.c -o log +gcc minishell.c -lcrypto -o minishell +``` + +Once this is done we can start reading the source code! + +> **Note**: +> Contrary to what I'm used to say and do, here there is no need to inspect the binary with `file`, `checksec`, `strings`, `ldd`, `ltrace` and `strace` because we compiled it ourself! +> We can not ensure that it has been compiled the same way in remote, still, it can be useful to experiment a bit. + +## Source code inspection + +> Please find the source code [on my GitHub](https://github.com/ctmbl/ctf-write-ups/blob/main/THCon-2024/pwn/Minishell) + +So let's read the code! +[`log.c`](https://github.com/ctmbl/ctf-write-ups/blob/main/THCon-2024/pwn/Minishell/log.c) is really simple, just a `main` function, it's a logging tool, it will write its arguments passed in command line to a log file, that's all. + +[`minishell.c`](https://github.com/ctmbl/ctf-write-ups/blob/main/THCon-2024/pwn/Minishell/minishell.c) is really something else: 269 lines of code. +When reading `C` code I always start looking globally at the function names and then I deep into the `main` function first. +Here it helped a lot, in `main` we quickly note that there is a bunch of variables initialization, some memory allocation and then a `while(1)`! +This is the infinite loop allowing the shell to always wait for user instructions. + +We understand that the user is prompted for a string, which is then verified (some characters are forbidden in `commandAllowed` maybe there is something here) and parsed with `strtok`. +Then a bunch of `if else` identify which function to execute given the user command. At that point I could have started looking into each and every function to look for vulnerabilities, but I didn't. +I wanted to first finish the reading of `main` and I chose really well. + +So we continue reading `main` to the last `else` (in case the command doesn't match any predefined strings), and there we have some really interesting stuff! +```C + }else { + char *log = malloc(256 * sizeof(char)); + strcpy(log, "./log Error with command:"); + + + strcpy(arg, cmd); + strcat(log, arg); + system(log); + + printf("Unknow command, this event has been reported\n"); + } +``` +Some `strcpy`, a `strcat` and above all a `system` call! + +Of course it instantly caught my eye: if we were able to control the `log` variable, we could inject some commands here. +Unfortunately a predefined string is written in `log` and even if we control `cmd` it is just appended to `./log Error with command:` (remember `./log` is the second binary compiled at the beginning) by `strcat` and because special characters like `;` or `&&` are forbidden we cannot inject a 2nd command to `log` 😢 + +> **Note**: +> However I noticed first that at the beginning of the `while` loop `buffer` is copied into `cmd` **before** verifying it with `commandAllowed`. +> So I tested an exploit where I injected some forbidden command `aa; /bin/sh` which won't be executed **but will be written in `cmd` anyway**. +> And then I inject a second one `a` which is allowed but unrecognized: the idea was that it didn't totally overwrite `cmd` which then would be something like `a\n; /bin/sh` and be appended to `log` then executed. +> Unfortunately, `strcpy` (or other reason) adds a `\x00` between the "new" injection `a` and the "remaining" one in `cmd`, so it ends the string and even if the payload is there in the stack it isn't copied in `log` and wouldn't have been executed by `system` anyway. +> So close! + +So the real vulnerability is still here lying under our eyes: simply `arg` is not the same size as `cmd`, then when copying a long `cmd` into `arg` it overflows. +```C + char* buffer = malloc(256 * sizeof(char)); + char* cmd = malloc(256 * sizeof(char)); + char *arg = malloc(32 * sizeof(char)); +``` +Because `arg` is in the heap the question is then: what do we overflow? +And the answer is "if it's the first prompt, probably `log` which is alloc'd after `arg`", and finally we control `log`!!! + +## Exploitation + +Now `arg` is 32 bytes long, and because we're in the heap we will first overwrite the chunk header before overwriting `log`'s content. +To determine exactly the padding needed for our payload, either we know the heap chunk header size, or we use `gdb` (which is often really useful) but even simpler: a smart payload such as: `AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFFGGGGGGGG` (generated with a `for` loop in python to avoid silly mistakes...) will easily do the job. +We inject it and see: +``` +$ ./minishell +spaceshell> AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFFGGGGGGGG +sh: line 1: GGGGGGGG: command not found +sh: line 2: AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFFGGGGGGGG: command not found +sh: line 3: AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDE: command not found +Unknow command, this event has been reported +spaceshell> +``` +Victory! `GGGGGGGG` is executed as a command (I confirmed it with `ltrace ./minishell` and saw the execution of `system` with our payload and the result). +We then infer that an heap chunk header was 16 bytes long because our payload padding is `AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFF` which is 48 bytes, minus the 32 of `arg` we get 16 bytes for the header. + +The final payload is of course: `AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFF/bin/sh` and like that we get our shell 😉 + +Locally: +``` +$ ./minishell +spaceshell> AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFF/bin/sh +sh-5.2$ whoami +ctmbl +``` +Remotely (I could have used `/bin/sh` too of course): +``` +$ nc 20.19.241.70 3001 +spaceshell> AAAAAAAABBBBBBBBCCCCCCCCDDDDDDDDEEEEEEEEFFFFFFFFcat /home/ctf/flag.txt +THCON{G00d_0ld_0v3rfl0w}Unknow command, this event has been reported +``` +🎉🎉🎉🎉 + +A good old overflow for sure 🙂, but a good reminder and a nice introduction to heap overflow overall 😉 + +## Conclusion + +**To sum up**, here are the main step of the reasoning while tackling this challenge (and maybe how to tackle other `pwn` challenges): +1. First: **what have I got? what do I want to achieve?** + source code, **no binaries**, a remote access to the executing binary -> we want a **shell on the remote machine** +2. Here we got source code but no binary, we **skip the inspect part** and just **compile the source code** as we can. + We'll have to **assume the possible protection** of the remote binary. + > Note that these two first parts are often forgotten but they are basically driving the rest of the exploit... +3. Dive into the source code, take a **global look** at the code but **quickly focus on main**. + We do not try to understand everything or every line, just **identify the structure of the code** and potentialy flawed lines: arrays, `malloc` and `free`, `printf`, bounds of `for` loops, Time of Check Time of Use (TOCTOU)... +4. We do not take a look at other functions while we have not finish reading main +5. Get a **first idea, try it**, understand why it works, or why it doesn't +6. Find a possible exploitable bug (here a buffer overflow), confirm it by several means (direct execution and with `ltrace` in my case) and rigorously define the needed payload (in my case the size of the padding) +7. Exploit, flag, celebrate :tada: + +## Resources + +> If any doubts you can always contact me on Discord `ctmbl` or issue on my [GitHub](https://github.com/ctmbl/ctf-write-ups/issues) if you need more information or resources 😉 + +Links: +- what is a buffer overflow: https://en.wikipedia.org/wiki/Buffer_overflow#Example +- more about heap structure and exploitation: https://heap-exploitation.dhavalkapil.com/diving_into_glibc_heap/malloc_chunk +- `strcpy`: https://man7.org/linux/man-pages/man3/strcpy.3.html +- `strcat`: https://linux.die.net/man/3/strcat +- `strtok`: https://man7.org/linux/man-pages/man3/strtok.3.html +- `system`: https://man7.org/linux/man-pages/man3/system.3.html