docs: DOC-1500: "Too many open files" Troubleshooting (#4967)

* Initial commit for too many open files error * Additional troubleshooting tweaks; fixed incorrect CI/CD reference * Fixed broken links due to updated heading * Adjusted PCG verbiage and added x-ref * Fixed code block indentation * ci: auto-formatting prettier issues * Incorporating suggestions from Ben * Incorporates additional suggestions per Carolina * ci: auto-formatting prettier issues --------- Co-authored-by: achuribooks <[email protected]>
spectrocloud · Dec 11, 2024 · aa93e9c · aa93e9c
1 parent e4f6e34
commit aa93e9c
Show file tree

Hide file tree

Showing 9 changed files with 119 additions and 58 deletions.
diff --git a/docs/docs-content/automation/palette-cli/palette-cli.md b/docs/docs-content/automation/palette-cli/palette-cli.md
@@ -7,7 +7,7 @@ tags: ["palette-cli"]
 ---
 
 The Palette CLI contains various functionalities that you can use to interact with Palette and manage resources. The
-Palette CLI is well suited for Continuous Delivery/Continuous Deployment (CI/CD) pipelines and recommended for
+Palette CLI is well suited for Continuous Integration/Continuous Deployment (CI/CD) pipelines and recommended for
 automation tasks, where Terraform or direct API queries are not ideal.
 
 To get started with the Palette CLI, check out the [Install](install-palette-cli.md) guide.

diff --git a/docs/docs-content/enterprise-version/upgrade/upgrade-notes.md b/docs/docs-content/enterprise-version/upgrade/upgrade-notes.md
@@ -57,7 +57,7 @@ Palette 4.0 includes the following major enhancements that require user interven
 A known issue impacts all self-hosted Palette instances older then 4.4.14. Before upgrading a Palette instance with
 version older than 4.4.14, ensure that you execute a utility script to make all your cluster IDs unique in your
 Persistent Volume Claim (PVC) metadata. For more information, refer to the
-[Troubleshooting Guide](../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping).
+[Troubleshooting Guide](../../troubleshooting/enterprise-install.md#scenario---non-unique-vsphere-cns-mapping).
 
 :::
 

diff --git a/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/airgap.md b/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/airgap.md
@@ -17,7 +17,7 @@ details.
 
 If you are upgrading from a Palette version that is older than 4.4.14, ensure that you have executed the utility script
 to make the CNS mapping unique for the associated PVC. For more information, refer to the
-[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping).
+[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#scenario---non-unique-vsphere-cns-mapping).
 
 :::
 

diff --git a/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/non-airgap.md b/docs/docs-content/enterprise-version/upgrade/upgrade-vmware/non-airgap.md
@@ -16,7 +16,7 @@ version available. Refer to the [Supported Upgrade Paths](../upgrade.md#supporte
 
 If you are upgrading from a Palette version that is older than 4.4.14, ensure that you have executed the utility script
 to make the CNS mapping unique for the associated PVC. For more information, refer to the
-[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping).
+[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#scenario---non-unique-vsphere-cns-mapping).
 
 :::
 

diff --git a/docs/docs-content/release-notes/known-issues.md b/docs/docs-content/release-notes/known-issues.md
diff --git a/docs/docs-content/troubleshooting/enterprise-install.md b/docs/docs-content/troubleshooting/enterprise-install.md
@@ -10,7 +10,7 @@ tags: ["troubleshooting", "self-hosted", "palette", "vertex"]
 
 Refer to the following sections to troubleshoot errors encountered when installing an Enterprise Cluster.
 
-## Scenario - Self-linking Error
+## Scenario - Self-Linking Error
 
 When installing an Enterprise Cluster, you may encounter an error stating that the enterprise cluster is unable to
 self-link. Self-linking is the process of Palette or VerteX becoming aware of the Kubernetes cluster it is installed on.
@@ -78,7 +78,7 @@ following steps to restart the management pod.
    pod "mgmt-f7f97f4fd-lds69" deleted
    ```
 
-## Non-unique vSphere CNS Mapping
+## Scenario - Non-Unique vSphere CNS Mapping
 
 In Palette and VerteX releases 4.4.8 and earlier, Persistent Volume Claims (PVCs) metadata do not use a unique
 identifier for self-hosted Palette clusters. This causes incorrect Cloud Native Storage (CNS) mappings in vSphere,
@@ -156,3 +156,57 @@ automatically resolve this issue. If you have self-hosted instances of Palette i
 
    Events:  <none>
    ```
+
+## Scenario - "Too Many Open Files" in Cluster
+
+When viewing logs for Enterprise or [Private Cloud Gateway](../clusters/pcg/pcg.md) clusters, you may encounter a "too
+many open files" error, which prevents logs from tailing after a certain point. To resolve this issue, you must increase
+the maximum number of file descriptors for each node on your cluster.
+
+### Debug Steps
+
+Repeat the following process for each node in your cluster.
+
+1. Log in to a node in your cluster.
+
+   ```bash
+   ssh -i <key-name> <spectro@hostname>
+   ```
+
+2. Switch to `sudo` mode using the command that best fits your system and preferences.
+
+   ```bash
+   sudo --login
+   ```
+
+3. Increase the maximum number of file descriptors that the kernel can allocate system-wide.
+
+   ```bash
+   echo "fs.file-max = 1000000" > /etc/sysctl.d/99-maxfiles.conf
+   ```
+
+4. Apply the updated `sysctl` settings. The increased limit is returned.
+
+   ```bash
+   sysctl -p /etc/sysctl.d/99-maxfiles.conf
+   ```
+
+   ```bash hideClipboard
+   fs.file-max = 1000000
+   ```
+
+5. Restart the `kubelet` and `containerd` services.
+
+   ```bash
+   systemctl restart kubelet containerd
+   ```
+
+6. Confirm that the change was applied.
+
+   ```bash
+   sysctl fs.file-max
+   ```
+
+   ```bash hideClipboard
+   fs.file-max = 1000000
+   ```
diff --git a/docs/docs-content/vertex/upgrade/upgrade-notes.md b/docs/docs-content/vertex/upgrade/upgrade-notes.md
@@ -27,4 +27,4 @@ troubleshooting guide for resolution steps.
 A known issue impacts all self-hosted Palette instances older then 4.4.14. Before upgrading an Palette instance with
 version older than 4.4.14, ensure that you execute a utility script to make all your cluster IDs unique in your
 Persistent Volume Claim (PVC) metadata. For more information, refer to the
-[Troubleshooting Guide](../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping).
+[Troubleshooting Guide](../../troubleshooting/enterprise-install.md#scenario---non-unique-vsphere-cns-mapping).
diff --git a/docs/docs-content/vertex/upgrade/upgrade-vmware/airgap.md b/docs/docs-content/vertex/upgrade/upgrade-vmware/airgap.md
@@ -17,7 +17,7 @@ section for details.
 
 If you are upgrading from a Palette VerteX version that is older than 4.4.14, ensure that you have executed the utility
 script to make the CNS mapping unique for the associated PVC. For more information, refer to the
-[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping).
+[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#scenario---non-unique-vsphere-cns-mapping).
 
 :::
 

diff --git a/docs/docs-content/vertex/upgrade/upgrade-vmware/non-airgap.md b/docs/docs-content/vertex/upgrade/upgrade-vmware/non-airgap.md
@@ -17,7 +17,7 @@ for details.
 
 If you are upgrading from a Palette VerteX version that is older than 4.4.14, ensure that you have executed the utility
 script to make the CNS mapping unique for the associated PVC. For more information, refer to the
-[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#non-unique-vsphere-cns-mapping).
+[Troubleshooting guide](../../../troubleshooting/enterprise-install.md#scenario---non-unique-vsphere-cns-mapping).
 
 :::