Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HPCC-31972 Update Docs for Cost Tracking improvements #19020

Merged
merged 1 commit into from
Sep 30, 2024
Merged

HPCC-31972 Update Docs for Cost Tracking improvements #19020

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 83 additions & 49 deletions docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
<chapter id="CustomConfig">
<title>Customizing Configurations</title>
Expand Down Expand Up @@ -865,37 +865,51 @@ thor: []
using cloud native HPCC Systems platform instances in a cloud setting
where some planning and configuration can help reduce expenses.</para>

<para>Two new columns have been added to the workunits page in ECL Watch.
The columns may be sorted by either cost column, just like the other
columns in ECL Watch, by clicking at the top of the column.</para>
<para>New columns have been added to the workunits and the logical files
pages in ECL Watch. These columns can be sorted by any cost columns, just
like the other columns in ECL Watch, by clicking at the top of the column.
In addition, file operation costs executed by workunits are provided in
the workunit’s metrics.</para>

<para>As the cost tracking matures, the cost calculations also improve
significantly providing more accurate cost tracking. For example, data
that was accessed from page cache doesn’t incur any file access cost. HPCC
Systems cost tracking now detects data that has been returned from the
page cache and adjusts cost calculations appropriately resulting in more
accurate cost calculations.</para>

<sect2 id="TypesOfCosts">
<title>Types of Costs</title>

<para>There are three types of costs that are tracked.</para>
<para>There are several types of costs that are tracked.</para>

<para><itemizedlist>
<listitem>
<para>Execution Cost</para>
</listitem>

<listitem>
<para>Storage Cost</para>
<para>Compile Cost</para>
</listitem>

<listitem>
<para>File Access Cost</para>
</listitem>

<listitem>
<para>File Cost at Rest</para>
Copy link
Contributor

@shamser shamser Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Line 886) Storage Cost is listed as a separate item in the list of costs above. Storage cost isn't a separate cost item. Storage Cost = File Access Cost + File Cost At Rest. Also, the Execution cost is not listed. The Types of costs tracked are:

  • Execute Cost
  • Compile Cost
  • File Access Cost
  • File Cost At Rest

</listitem>
</itemizedlist></para>

<variablelist>
<varlistentry>
<term>NOTE:</term>

<listitem>
<para>All cost values calculated and displayed are approximate.
There are many variables that can result in inaccuracies. These
cost values are only intended to be used as a guide.</para>
<para>While reporting accuracy has significantly improved, all
cost values calculated and displayed remain approximate. There are
many variables that can result in inaccuracies. These cost values
are only intended to be used as a guide.</para>
</listitem>
</varlistentry>
</variablelist>
Expand Down Expand Up @@ -962,12 +976,22 @@ thor: []
</sect4>
</sect3>

<sect3 id="Compile_Costs">
<title>Compile Cost</title>

<para>Compile costs are the costs of compiling the ECL code. The cost
of compiling the ECL code is included as a column in the workunit
list. In the workunit summary page there is a compile cost
field.</para>
</sect3>

<sect3 id="StorageCOsts">
<title>Storage Cost</title>
<title>Storage Costs</title>

<para>This is the cost of hosting the data in the storage plane. It
does not include the costs of data operations such as read or write
costs.</para>
<para>This is the cost of hosting the data in the storage plane or the
<emphasis>File Cost At Rest</emphasis> value and the cost of
reading/writing to the storage, is the <emphasis>File Access
Cost</emphasis> value.</para>

<para><variablelist>
<varlistentry>
Expand All @@ -981,37 +1005,49 @@ thor: []
</varlistentry>
</variablelist></para>

<para>The storage costs cannot be viewed as a separate value in ECL
Watch. They can only be viewed as part of a cost field in a logical
file’s summary page. That cost field includes other file related
costs.</para>
</sect3>

<sect3 id="FileAccessCosts">
<title>File Access Cost</title>
<para>For logical files, the costs calculated are based on two types
of file access.</para>

<para>File access costs are the costs of reading and writing to the
files. Many storage planes do have a separate charge for data
operations. The cost of reading and writing to the file will be
included in the file access cost value. Any other cost associated with
file operations (such as delete or copy) will not be tracked or
included as part of file access cost at this time.</para>

<para>The file access cost displays as part of the cost field on the
Logical File’s summary page in ECL Watch.</para>

<para>The costs incurred by a workunit for accessing logical files is
also recorded in the workunit’s statistics and attributes. The
read/write cost is recorded at the activity record and cumulated at
the graph, the subgraph and the workflow scope level. The total file
access cost for a workunit is recorded with the workunit and displayed
in the summary page.</para>

<para>The new cost field is shown in the Logical File summary page. It
is the combined cost of storing and accessing the data.</para>
<itemizedlist>
<listitem>
<para>File Access Cost </para>
</listitem>

<para>The cost information is currently only generated for Thor and
hThor jobs.</para>
<listitem>
<para>File Cost at Rest </para>
</listitem>
</itemizedlist>

<sect4 id="FileAccessCost">
<title>File Access Cost </title>

<para>File access costs are the costs of reading and writing to the
files. Many storage planes do have a separate charge for data
operations. The cost of reading and writing to the file is included
in the file access cost value. Any other cost associated with file
operations (such as delete or copy) will not be tracked or included
as part of file access cost at this time.</para>

<para>The costs incurred by a workunit for accessing logical files
is also recorded in the workunit’s statistics and attributes. The
read/write cost is recorded at the activity record and cumulated at
the graph, the subgraph, and the workflow scope level. Cost Tracking
detects data that has been returned from the page cache and adjusts
cost calculations appropriately to produce more accurate cost
calculations. These file access costs for a workunit are recorded
with the workunit and displayed in the summary page and in the
workunit’s metrics.</para>

<sect5 id="FileCostAtRest">
<title>File Cost at Rest</title>

<para>The File Cost at Rest field is shown in the Logical File
summary page. It is the cost of storing the file without accessing
the data. Only the storage costs associated with housing the file
in the cloud. This value has been added to better differentiate
between file storage costs.</para>
</sect5>
</sect4>
</sect3>
</sect2>

Expand Down Expand Up @@ -1074,9 +1110,7 @@ thor: []

<para><informaltable colsep="1" frame="all" rowsep="1">
<tgroup cols="2">
<colspec align="left" colwidth="122.40pt" />

<colspec />
<colspec align="left" colwidth="122.40pt"/>

<tbody>
<row>
Expand Down Expand Up @@ -1154,9 +1188,9 @@ thor: []

<para><informaltable colsep="1" frame="all" rowsep="1">
<tgroup cols="2">
<colspec align="left" colwidth="122.40pt" />
<colspec align="left" colwidth="122.40pt"/>

<colspec />
<colspec/>

<tbody>
<row>
Expand Down Expand Up @@ -1326,7 +1360,7 @@ thor: []
ldapAdminSecretKey: "additionaltmountname" # use alternate secrets creds
</programlisting>

<para></para>
<para/>
</sect3>
</sect2>

Expand Down Expand Up @@ -1401,7 +1435,7 @@ thor: []
ldapAdminVaultId: "my-authn-vault"
</programlisting>

<para></para>
<para/>
</sect3>
</sect2>
</sect1>
Expand Down
Loading