docs: Spark operator updates #484
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adding a DOEKS_HOME varaible set to the data-on-eks directory means that each command in the spark on eks docs can be copied and pasted directly. Originally the docs simply said `cd analytics/...` repeastedly and that path was never correct.
I am unifying all the examples to use <S3_BUCKET> as the placeholder in files. I have added documentation to show how to use sed to do the varaiable replaement rather than putting the user through manual efforts.
The benchmark docs for spark operator point to the wrong file names. Unify the Bucket place holder to <S3_BUCKET> Update the benchmark docs with clear instructions
This closes out the Karpenter updates.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This pull request adds much needed details to the spark operator page.
cd ...commands now use an environment variable called DOEKS_HOME that ensures copy and paste always works in every example.sedcommands are provided for replacing the S3_BUCKET placeholders in scripts and spark application yaml manifests.Some quality of life changes as well.
Using Docusaurus partials for paragraphs that are repeated throughout examples.
Move large yaml examples out of the single markdown file into their own partials.
Motivation
The spark on EKS examples are currently very terse. Many assumptions were being made on what the user must know ahead of time. Additionally, most examples were simply incorrect (taxi script always needs input, benchmark files didn't exist).
The changes can be previewed here (note that navigation is wonky in this temp deployment).
https://d2gd59uo3ya1kt.cloudfront.net/data-on-eks/docs/blueprints/data-analytics/spark-operator-yunikorn.html
More
website/docsorwebsite/blogsection for this featurepre-commit run -awith this PR. Link for installing pre-commit locallyFor Moderators
Additional Notes