Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ambari-26147: Add Ruff integration to ambari #3844

Open
wants to merge 2 commits into
base: trunk
Choose a base branch
from

Conversation

JiaLiangC
Copy link
Contributor

@JiaLiangC JiaLiangC commented Oct 12, 2024

What changes were proposed in this pull request?

The entire Ambari project has accumulated a large amount of Python code, which is the result of the collective efforts of multiple contributors to make Ambari better.

Language Files Lines Code Comments Blanks
Java 3330 690679 427116 156389 107174
JavaScript 2125 512742 374238 87081 51423
Plain Text 68 500082 0 497409 2673
JSON 758 479988 479689 0 299
Python 1096 218057 175499 9547 33011
XML 1135 156581 130300 20189 6092

During this process, a problem has emerged: everyone has different coding styles, which leads to inconsistencies in the submitted code. This can result in various inconveniences, such as increased understanding costs of the underlying principles and greater difficulty in identifying the root causes of issues. As the codebase grows, these problems become increasingly apparent. Therefore, we are introducing ruff, along with some additional processes, to format the code and alleviate these issues.

What is Ruff?

An extremely fast Python linter and code formatter, written in Rust. Its speed is one of its advantages. More information can be found here: ruff.
(Please fill in changes proposed in this fix)

Ruff rules are derived from the Apache Superset project's rule file.

Ruff integration plan:

  1. Project integration with Ruff, formatting all Python files, ensuring all tests pass, and the cluster runs normally. Completed.
  2. Integration of Ruff detection into Jenkins. Completed.

Notice: This PR's Ruff rules ignore some rule checks because these rules involve significant changes that will be addressed in subsequent PRs. The current PR ensures that all Python indentation and basic formatting conform to standards.


Ruff Usage Instructions in Ambari Code Submission

Install Ruff:

pip install ruff

Format Code with Ruff:

ruff format .

Check Code Format Manually:

ruff check .

Check Code Format with Maven:

mvn exec:exec@ruff-check -Pruff-check

Fix Code Format with Maven:

mvn exec:exec@ruff-format -Pruff-format

How was this patch tested?

manual test CI/CD
After formatting thousands of files with Ruff, I deployed a simple cluster to verify that Ambari runs smoothly.

image

(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)

Please review Ambari Contributing Guide before opening a pull request.

@JiaLiangC
Copy link
Contributor Author

@virajjasani Could you help check this PR?

@virajjasani
Copy link
Contributor

Wow, this is huge and yet it seems worth having. Skimmed through at high level, looks good. I wonder if anyone else is also interested to review this, otherwise let's get this in and see how things go.

Nice work!

@arshadmohammad
Copy link
Contributor

Hi @JiaLiangC,
Which environment are you deploying the cluster for testing? It seems you're having success with the setup.
I recently tried to build and deploy an Ambari cluster on Rocky Linux 8 and encountered two blocking issues:
AMBARI-26183 (Fixed)
AMBARI-26187 (Still open)

@JiaLiangC
Copy link
Contributor Author

@arshadmohammad Good catch! Your deployment failed because of this PR: #3770, which introduced numerous bugs in the frontend.
Although the subsequent PR #3846 fixed many issues, around 60 of them, there are still many bugs remaining. One of them is the blocking issue you encountered, which is a bug caused by the frontend upgrade. I tested using the code from before this PR was merged: AMBARI-25289 Upgrade Jquery and Bootstrap to latest versions #3770.

@JiaLiangC
Copy link
Contributor Author

Hi @JiaLiangC, Which environment are you deploying the cluster for testing? It seems you're having success with the setup. I recently tried to build and deploy an Ambari cluster on Rocky Linux 8 and encountered two blocking issues: AMBARI-26183 (Fixed) AMBARI-26187 (Still open)

@arshadmohammad Already resolved by this pr #3849.

@arshadmohammad
Copy link
Contributor

Thanks @JiaLiangC, I will review this patch

@JiaLiangC
Copy link
Contributor Author

@arshadmohammad I have added the usage of Ruff involved in this PR to the description. Additionally, all code changes in this PR are the result of ruff format.

@vishalsuvagia
Copy link
Contributor

Hi @JiaLiangC , many changes include indentation / spacing between the lines, the current codebase uses two spaces for indentation / spacing instead of the typical four spaces / tabs. Is it possible to configure ruff to consider two spaces as the indentation and spacing size as it is the standard followed. Doing so may help reduce the size of the change as well and have clear diff for the actual changes.

@JiaLiangC
Copy link
Contributor Author

@vishalsuvagia Thank you for the reminder. I've changed the indent-width to 2 and committed it. This indeed makes it easier to review. However, there are still a massive 900 files, which I believe makes manual review very difficult. We can only rely on unit tests and subsequent bug reports after usage. Currently, the unit tests have passed, and I've deployed a complete cluster in my environment without finding any issues so far. Theoretically, code formatting tools like ruff shouldn't change the code semantics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants