Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HPCC-31972 Update Docs for Cost Tracking improvements #19020

Merged
merged 1 commit into from
Sep 30, 2024
Merged

HPCC-31972 Update Docs for Cost Tracking improvements #19020

merged 1 commit into from
Sep 30, 2024

Conversation

g-pan
Copy link
Member

@g-pan g-pan commented Aug 20, 2024 •
edited
Loading

Type of change:

  • This change is a bug fix (non-breaking change which fixes an issue).
  • This change is a new feature (non-breaking change which adds functionality).
  • This change improves the code (refactor or other change that does not change the functionality)
  • This change fixes warnings (the fix does not alter the functionality or the generated code)
  • This change is a breaking change (fix or feature that will cause existing behavior to change).
  • This change alters the query API (existing queries will have to be recompiled)

Checklist:

  • My code follows the code style of this project.
    • My code does not create any new warnings from compiler, build system, or lint.
  • The commit message is properly formatted and free of typos.
    • The commit message title makes sense in a changelog, by itself.
    • The commit is signed.
  • My change requires a change to the documentation.
    • I have updated the documentation accordingly, or...
    • I have created a JIRA ticket to update the documentation.
    • Any new interfaces or exported functions are appropriately commented.
  • I have read the CONTRIBUTORS document.
  • The change has been fully tested:
    • I have added tests to cover my changes.
    • All new and existing tests passed.
    • I have checked that this change does not introduce memory leaks.
    • I have used Valgrind or similar tools to check for potential issues.
  • I have given due consideration to all of the following potential concerns:
    • Scalability
    • Performance
    • Security
    • Thread-safety
    • Cloud-compatibility
    • Premature optimization
    • Existing deployed queries will not be broken
    • This change fixes the problem, not just the symptom
    • The target branch of this pull request is appropriate for such a change.
  • There are no similar instances of the same problem that should be addressed
    • I have addressed them here
    • I have raised JIRA issues to address them separately
  • This is a user interface / front-end modification
    • I have tested my changes in multiple modern browsers
    • The component(s) render as expected

Smoketest:

  • Send notifications about my Pull Request position in Smoketest queue.
  • Test my draft Pull Request.

Testing:

Successful Unit Tests : https://github.com/g-pan/github-action-dev-build/actions/runs/10474453301

@g-pan g-pan requested review from JamesDeFabia and shamser August 20, 2024 21:21
@github-actions GitHub Actions
Copy link

Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-31972

Jirabot Action Result:
Workflow Transition: Merge Pending
Updated PR

JamesDeFabia
JamesDeFabia requested changes Aug 20, 2024
View reviewed changes
Copy link
Contributor

@JamesDeFabia JamesDeFabia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments inline

docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml Outdated
<para>File access costs are the costs of reading and writing to the
files. Many storage planes do have a separate charge for data
operations. The cost of reading and writing to the file will be
operations. The cost of reading and writing to the file are
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SV agreement. Either "costs are included" or "costs is included"

docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml Outdated
columns in ECL Watch, by clicking at the top of the column.</para>

<para>New columns have been added to the workunits and the logical files pages in ECL Watch.
These columns may be sorted by any cost columns, just like the other
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use "can" instead of "may"

@g-pan g-pan requested a review from JamesDeFabia August 21, 2024 13:25
@g-pan
Copy link
Member Author

g-pan commented Aug 21, 2024

@shamser please verify the information going into the doc is currently accurate.

JamesDeFabia
JamesDeFabia requested changes Aug 21, 2024
View reviewed changes
Copy link
Contributor

@JamesDeFabia JamesDeFabia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one minor comment inline

docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml Outdated
In addition, file operation costs executed by workunits are provided in the workunit’s metrics. </para>
<para>
As the cost tracking matures, the cost calculations also improve significantly providing more accurate cost tracking.
For example, data that was accessed from page cache doesn’t incur any file access cost. HPCC Systems cost tracking now detects data that has been returned from the page cache and adjust cost calculations appropriately resulting in more accurate cost calculations. </para>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/b "adjusts" for SV agreement

shamser
shamser requested changes Aug 22, 2024
View reviewed changes
docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml

<listitem>
<para>File Access Cost</para>
</listitem>

<listitem>
<para>File Cost at Rest</para>
Copy link
Contributor

@shamser shamser Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Line 886) Storage Cost is listed as a separate item in the list of costs above. Storage cost isn't a separate cost item. Storage Cost = File Access Cost + File Cost At Rest. Also, the Execution cost is not listed. The Types of costs tracked are:

  • Execute Cost
  • Compile Cost
  • File Access Cost
  • File Cost At Rest

docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml Outdated
@@ -988,31 +1007,40 @@ thor: []
</sect3>

<sect3 id="FileAccessCosts">
<title>File Access Cost</title>
<title>File Access Costs</title>
<para> For logical files, the costs calculated are based on two types of file access. </para>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should probably be "...there are two types of file costs:". There is just one type of file access cost (File Access Cost) but there are 2 storage related costs (File Access Cost and File Cost at Rest)

docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml Outdated
@@ -988,31 +1007,40 @@ thor: []
</sect3>

<sect3 id="FileAccessCosts">
<title>File Access Cost</title>
<title>File Access Costs</title>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The title should be something like "File Cost". File Access Costs doesn't include Storage At Rest cost.

Also, the paragraph on line 1003-1007, is no longer correct. The two types of storage costs has been separated into 2 different fields and can now be viewed as separate fields.

docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml Outdated
included in the file access cost value. Any other cost associated with
file operations (such as delete or copy) will not be tracked or
included as part of file access cost at this time.</para>

<para>The file access cost displays as part of the cost field on the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are now two separate fields for the file costs now. The file access costs isn't displayed as part of the cost field. File access cost is shown as a field. The At rest is shown as separate field.

docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml Outdated
hThor jobs.</para>
</sect3>
<para>The File Cost at Rest field is shown in the Logical File summary page. It
is the cost of storing the file without accessing the data. Only the storage costs associated with housing the file in the cloud. This value has been added to better differentiate between file storage costs. </para>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"File Cost at Rest field" is the file storage costs. I think the last line in the previous paragraph should be removed. File Costs At Rest and File Access cost are both shown in the logical file's summary page.

@g-pan
Copy link
Member Author

g-pan commented Sep 4, 2024

provided PDF file for review offline.

@g-pan g-pan requested a review from shamser September 4, 2024 16:36
shamser
shamser approved these changes Sep 6, 2024
View reviewed changes
Copy link
Contributor

@shamser shamser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@g-pan g-pan requested a review from JamesDeFabia September 24, 2024 00:47
JamesDeFabia
JamesDeFabia approved these changes Sep 24, 2024
View reviewed changes
Copy link
Contributor

@JamesDeFabia JamesDeFabia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good from my POV

@ghalliday
Copy link
Member

@g-pan please squash

@ghalliday ghalliday merged commit f53825f into hpcc-systems:candidate-9.8.x Sep 30, 2024
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shamser shamser shamser approved these changes

@JamesDeFabia JamesDeFabia JamesDeFabia approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@g-pan @ghalliday @JamesDeFabia @shamser