Skip to content

Commit

Permalink
Merge pull request #19020 from g-pan/H31972-CostTrk
Browse files Browse the repository at this point in the history 
HPCC-31972 Update Docs for Cost Tracking improvements

Reviewed-By: Jim DeFabia <[email protected]>
Reviewed-By: Shamser Ahmed <[email protected]>
Merged-by: Gavin Halliday <[email protected]>
  • Loading branch information
@ghalliday
ghalliday authored Sep 30, 2024
2 parents 37c7f00 + e00fdd1 commit f53825f
Showing 1 changed file with 83 additions and 49 deletions.
132 changes: 83 additions & 49 deletions docs/EN_US/ContainerizedHPCC/ContainerizedMods/CustomConfig.xml
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd">
<chapter id="CustomConfig">
<title>Customizing Configurations</title>
Expand Down Expand Up @@ -865,37 +865,51 @@ thor: []
using cloud native HPCC Systems platform instances in a cloud setting
where some planning and configuration can help reduce expenses.</para>

<para>Two new columns have been added to the workunits page in ECL Watch.
The columns may be sorted by either cost column, just like the other
columns in ECL Watch, by clicking at the top of the column.</para>
<para>New columns have been added to the workunits and the logical files
pages in ECL Watch. These columns can be sorted by any cost columns, just
like the other columns in ECL Watch, by clicking at the top of the column.
In addition, file operation costs executed by workunits are provided in
the workunit’s metrics.</para>

<para>As the cost tracking matures, the cost calculations also improve
significantly providing more accurate cost tracking. For example, data
that was accessed from page cache doesn’t incur any file access cost. HPCC
Systems cost tracking now detects data that has been returned from the
page cache and adjusts cost calculations appropriately resulting in more
accurate cost calculations.</para>

<sect2 id="TypesOfCosts">
<title>Types of Costs</title>

<para>There are three types of costs that are tracked.</para>
<para>There are several types of costs that are tracked.</para>

<para><itemizedlist>
<listitem>
<para>Execution Cost</para>
</listitem>

<listitem>
<para>Storage Cost</para>
<para>Compile Cost</para>
</listitem>

<listitem>
<para>File Access Cost</para>
</listitem>

<listitem>
<para>File Cost at Rest</para>
</listitem>
</itemizedlist></para>

<variablelist>
<varlistentry>
<term>NOTE:</term>

<listitem>
<para>All cost values calculated and displayed are approximate.
There are many variables that can result in inaccuracies. These
cost values are only intended to be used as a guide.</para>
<para>While reporting accuracy has significantly improved, all
cost values calculated and displayed remain approximate. There are
many variables that can result in inaccuracies. These cost values
are only intended to be used as a guide.</para>
</listitem>
</varlistentry>
</variablelist>
Expand Down Expand Up @@ -962,12 +976,22 @@ thor: []
</sect4>
</sect3>

<sect3 id="Compile_Costs">
<title>Compile Cost</title>

<para>Compile costs are the costs of compiling the ECL code. The cost
of compiling the ECL code is included as a column in the workunit
list. In the workunit summary page there is a compile cost
field.</para>
</sect3>

<sect3 id="StorageCOsts">
<title>Storage Cost</title>
<title>Storage Costs</title>

<para>This is the cost of hosting the data in the storage plane. It
does not include the costs of data operations such as read or write
costs.</para>
<para>This is the cost of hosting the data in the storage plane or the
<emphasis>File Cost At Rest</emphasis> value and the cost of
reading/writing to the storage, is the <emphasis>File Access
Cost</emphasis> value.</para>

<para><variablelist>
<varlistentry>
Expand All @@ -981,37 +1005,49 @@ thor: []
</varlistentry>
</variablelist></para>

<para>The storage costs cannot be viewed as a separate value in ECL
Watch. They can only be viewed as part of a cost field in a logical
file’s summary page. That cost field includes other file related
costs.</para>
</sect3>

<sect3 id="FileAccessCosts">
<title>File Access Cost</title>
<para>For logical files, the costs calculated are based on two types
of file access.</para>

<para>File access costs are the costs of reading and writing to the
files. Many storage planes do have a separate charge for data
operations. The cost of reading and writing to the file will be
included in the file access cost value. Any other cost associated with
file operations (such as delete or copy) will not be tracked or
included as part of file access cost at this time.</para>

<para>The file access cost displays as part of the cost field on the
Logical File’s summary page in ECL Watch.</para>

<para>The costs incurred by a workunit for accessing logical files is
also recorded in the workunit’s statistics and attributes. The
read/write cost is recorded at the activity record and cumulated at
the graph, the subgraph and the workflow scope level. The total file
access cost for a workunit is recorded with the workunit and displayed
in the summary page.</para>

<para>The new cost field is shown in the Logical File summary page. It
is the combined cost of storing and accessing the data.</para>
<itemizedlist>
<listitem>
<para>File Access Cost </para>
</listitem>

<para>The cost information is currently only generated for Thor and
hThor jobs.</para>
<listitem>
<para>File Cost at Rest </para>
</listitem>
</itemizedlist>

<sect4 id="FileAccessCost">
<title>File Access Cost </title>

<para>File access costs are the costs of reading and writing to the
files. Many storage planes do have a separate charge for data
operations. The cost of reading and writing to the file is included
in the file access cost value. Any other cost associated with file
operations (such as delete or copy) will not be tracked or included
as part of file access cost at this time.</para>

<para>The costs incurred by a workunit for accessing logical files
is also recorded in the workunit’s statistics and attributes. The
read/write cost is recorded at the activity record and cumulated at
the graph, the subgraph, and the workflow scope level. Cost Tracking
detects data that has been returned from the page cache and adjusts
cost calculations appropriately to produce more accurate cost
calculations. These file access costs for a workunit are recorded
with the workunit and displayed in the summary page and in the
workunit’s metrics.</para>

<sect5 id="FileCostAtRest">
<title>File Cost at Rest</title>

<para>The File Cost at Rest field is shown in the Logical File
summary page. It is the cost of storing the file without accessing
the data. Only the storage costs associated with housing the file
in the cloud. This value has been added to better differentiate
between file storage costs.</para>
</sect5>
</sect4>
</sect3>
</sect2>

Expand Down Expand Up @@ -1074,9 +1110,7 @@ thor: []

<para><informaltable colsep="1" frame="all" rowsep="1">
<tgroup cols="2">
<colspec align="left" colwidth="122.40pt" />

<colspec />
<colspec align="left" colwidth="122.40pt"/>

<tbody>
<row>
Expand Down Expand Up @@ -1154,9 +1188,9 @@ thor: []

<para><informaltable colsep="1" frame="all" rowsep="1">
<tgroup cols="2">
<colspec align="left" colwidth="122.40pt" />
<colspec align="left" colwidth="122.40pt"/>

<colspec />
<colspec/>

<tbody>
<row>
Expand Down Expand Up @@ -1326,7 +1360,7 @@ thor: []
ldapAdminSecretKey: "additionaltmountname" # use alternate secrets creds
</programlisting>

<para></para>
<para/>
</sect3>
</sect2>

Expand Down Expand Up @@ -1401,7 +1435,7 @@ thor: []
ldapAdminVaultId: "my-authn-vault"
</programlisting>

<para></para>
<para/>
</sect3>
</sect2>
</sect1>
Expand Down

0 comments on commit f53825f

Please sign in to comment.