Use otelcol-config for config manipulations (#103)

* Use otelcol-config for config manipulations * Remove write_sumologic_extension * Install otelcol-config in test-install-script job (#111) Signed-off-by: Eric Chlebek <[email protected]> * Remove the concept of systemd from install script Signed-off-by: Eric Chlebek <[email protected]> * Remove systemd concepts Signed-off-by: Eric Chlebek <[email protected]> * Remove unused code Signed-off-by: Eric Chlebek <[email protected]> * Move much of the script to otelcol-config * Deprecate all systemd configuration manipulation * Deprecate several flags on Linux * Preserve Mac OS functionality Signed-off-by: Eric Chlebek <[email protected]> * Fix infinite loop in script Remove unnecessary tests Signed-off-by: Eric Chlebek <[email protected]> * Return 2 on incorrect usage Signed-off-by: Eric Chlebek <[email protected]> * Hardcode app version for now Also fix some test bugs Signed-off-by: Eric Chlebek <[email protected]> * Don't try to get latest release in install script tests Signed-off-by: Eric Chlebek <[email protected]> * Install package repositories with install.sh Signed-off-by: Eric Chlebek <[email protected]> * Fix debian package uninstallation Signed-off-by: Eric Chlebek <[email protected]> * Add -y to package removal calls Signed-off-by: Eric Chlebek <[email protected]> * Test speculative change Signed-off-by: Eric Chlebek <[email protected]> * Support --purge for apt-get Signed-off-by: Eric Chlebek <[email protected]> * Use --purge in script tests Signed-off-by: Eric Chlebek <[email protected]> * Remove autoconfirm tests on linux Signed-off-by: Eric Chlebek <[email protected]> * Don't do user checks Signed-off-by: Eric Chlebek <[email protected]> * Fixes for darwin and windows Signed-off-by: Eric Chlebek <[email protected]> * Fix add-tag routine Signed-off-by: Eric Chlebek <[email protected]> * Don't show usage for no args when token env set Signed-off-by: Eric Chlebek <[email protected]> * use --quiet for calling package managers Signed-off-by: Eric Chlebek <[email protected]> * Fix crash when otelcol-config is not yet installed Signed-off-by: Eric Chlebek <[email protected]> * Use more --quiet Signed-off-by: Eric Chlebek <[email protected]> * Use otelcol-config for darwin-specific subroutine Signed-off-by: Eric Chlebek <[email protected]> * Replace common.yaml with 00-otelcol-config-settings.yaml Signed-off-by: Eric Chlebek <[email protected]> * Complain when installation token not provided Signed-off-by: Eric Chlebek <[email protected]> * Use otelcol-config for configuration in install.sh Use `otelcol-config` in `install.sh` to perform configuration on both Linux and macOS. The `--download-only` flag will now only work for macOS as it is not applicable to other platforms. The `--skip-config` flag has been removed as it is no longer applicable to installations. The `--skip-token` flag has been deprecated and install.sh will now exit with an error if no token has been provided. Add `DARWIN_PKG_URL` for overriding the URL used to download macOS packages in `install.sh`. Move test-install-script job to build_packages workflow Ownership and permissions of files should no longer be changed by `install.sh`. Added `conf.d-available` directory to Linux & macOS packages. Added `opamp.d` directory to Linux & macOS packages. The `ci-builds` repository is now used in CI for install script tests. Set timeout for test-install-script to 15 mins. Print launchdaemon state while waiting for start. In cases where we detect an existing installation of the collector made with the previous install script (without a package manager), we clean it up, backup its config, and setup the collector using the package manager. Add .op directory to .gitignore Use service wrapper on linux & macOS to change which flags are used to start the `otelcol-sumo` service depending on the existance of the `sumologic-remote.yaml` file. Co-authored-by: Cyril Cressent <[email protected]> Co-authored-by: Eric Chlebek <[email protected]> Signed-off-by: Justin Kolberg <[email protected]> --------- Signed-off-by: Eric Chlebek <[email protected]> Signed-off-by: Justin Kolberg <[email protected]> Co-authored-by: Eric Chlebek <[email protected]> Co-authored-by: Eric Chlebek <[email protected]>
SumoLogic · Nov 6, 2024 · 563a140 · 563a140
1 parent ae4886d
commit 563a140
Show file tree

Hide file tree

Showing 40 changed files with 1,486 additions and 2,740 deletions.
diff --git a/.github/workflows/_reusable_build_package.yml b/.github/workflows/_reusable_build_package.yml
@@ -77,18 +77,6 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
-      - name: Download packagecloud-go tool
-        run: |
-          baseURL="https://github.com/amdprophet/packagecloud-go/releases/download"
-          version="0.1.5"
-          file="packagecloud-go_${version}_linux_amd64.tar.gz"
-          curl -Lo /tmp/packagecloud-go.tar.gz $baseURL/$version/$file
-
-      - name: Install packagecloud-go tool
-        run: |
-          tar -C /tmp -zxf /tmp/packagecloud-go.tar.gz
-          sudo mv /tmp/packagecloud /usr/local/bin
-
       - name: Workflow URL for sumologic-otel-collector
         if: ${{ !inputs.use_release_artifacts && inputs.workflow_id != '' }}
         run: |
@@ -235,6 +223,13 @@ jobs:
           target: publish-package
           packagecloud-token: ${{ secrets.PACKAGECLOUD_TOKEN }}
 
+      - name: Wait for Packagecloud packages to be indexed
+        if: runner.os == 'Linux'
+        uses: ./ci/github-actions/make
+        with:
+          target: wait-for-packagecloud-indexing
+          packagecloud-token: ${{ secrets.PACKAGECLOUD_TOKEN }}
+
   test_package:
     runs-on: ${{ inputs.runs_on }}
     name: Test (CMake)

diff --git a/.github/workflows/build_packages.yml b/.github/workflows/build_packages.yml
@@ -331,3 +331,74 @@ jobs:
             This release packages [${{ env.OTC_APP_VERSION }}](https://github.com/SumoLogic/sumologic-otel-collector/releases/tag/${{ env.OTC_APP_VERSION }}).
 
             The changelog below is for the package itself, rather than the Sumo Logic Distribution for OpenTelemetry Collector.
+
+  test-install-script:
+    name: Test Install Script
+    runs-on: ${{ matrix.runs_on }}
+    timeout-minutes: 60
+    needs:
+      - determine_version
+      - build_packages
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - arch_os: linux_amd64
+            runs_on: ubuntu-20.04
+          - arch_os: darwin_amd64
+            runs_on: macos-latest
+          - arch_os: windows_amd64
+            runs_on: windows-2022
+    env:
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      GH_CI_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      OTC_VERSION: ${{ needs.determine_version.outputs.otc_version }}
+      OTC_BUILD_NUMBER: ${{ github.run_number }}
+      PACKAGECLOUD_MASTER_TOKEN: ${{ secrets.PACKAGECLOUD_MASTER_TOKEN }}
+      PACKAGECLOUD_REPO: ci-builds
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Check if test related files changed
+        id: changed-files
+        uses: tj-actions/changed-files@v44
+        with:
+          files: |
+            install-script/**/*
+            .github/**
+
+      - name: Setup go
+        if: steps.changed-files.outputs.any_changed == 'true'
+        uses: WillAbides/setup-go-faster@v1
+        with:
+          go-version: stable
+
+      - name: Download macOS package and use it for install.sh
+        if: ${{ steps.changed-files.outputs.any_changed == 'true' && runner.os == 'macOS' }}
+        uses: actions/download-artifact@v4
+        with:
+          path: artifacts/
+          pattern: otelcol-sumo_*-intel.pkg
+
+      - name: Show packages
+        if: ${{ steps.changed-files.outputs.any_changed == 'true' && runner.os == 'macOS' }}
+        run: |
+          ls -l artifacts/
+          ls -l artifacts/**/*
+
+      - name: Set DARWIN_PKG_URL
+        if: ${{ steps.changed-files.outputs.any_changed == 'true' && runner.os == 'macOS' }}
+        run: |
+          fp="$(readlink -f artifacts/otelcol-sumo_*-intel.pkg/otelcol-sumo_*-intel.pkg)"
+          echo DARWIN_PKG_URL="file://${fp}" >> $GITHUB_ENV
+
+      - name: Run install script tests (*nix)
+        if: steps.changed-files.outputs.any_changed == 'true' && runner.os != 'Windows'
+        working-directory: install-script/test
+        run: make test
+
+      - name: Run install script tests (Windows)
+        shell: powershell
+        if: steps.changed-files.outputs.any_changed == 'true' && runner.os == 'Windows'
+        working-directory: install-script/test
+        run: make test
diff --git a/.github/workflows/test-install-script.yml b/.github/workflows/test-install-script.yml
diff --git a/.gitignore b/.gitignore
@@ -18,3 +18,6 @@ wix/packages
 msi/wix/packages
 msi/SumoLogic.wixext/packages
 install-script/test/sumologic_scripts_tests.test
+
+# 1Password
+.op/
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -3,7 +3,7 @@ cmake_minimum_required(VERSION 3.24.1 FATAL_ERROR)
 
 # Required and optional programs. Attempts to find required and optional
 # programs used to build the packages.
-find_program(PACKAGECLOUD_PROGRAM packagecloud REQUIRED)
+find_program(PACKAGECLOUD_PROGRAM packagecloud)
 
 # Set version information
 include("${CMAKE_SOURCE_DIR}/version.cmake")

diff --git a/assets/.keep b/assets/.keep
diff --git a/assets/conf.d/ephemeral.yaml b/assets/conf.d/ephemeral.yaml
@@ -0,0 +1,3 @@
+extensions:
+  sumologic:
+    ephemeral: true
diff --git a/assets/productbuild/uninstall.sh b/assets/productbuild/uninstall.sh
@@ -43,19 +43,19 @@ collector_files=(
   "/etc/otelcol-sumo/sumologic.yaml"
   "/etc/otelcol-sumo/conf.d"
   "/etc/otelcol-sumo/conf.d-available"
+  "/etc/otelcol-sumo/conf.d-available/ephemeral.yaml"
+  "/etc/otelcol-sumo/conf.d-available/hostmetrics.yaml"
   "/etc/otelcol-sumo"
+  "/etc/otelcol-sumo/opamp.d"
   "/usr/local/bin/otelcol-config"
   "/usr/local/bin/otelcol-sumo"
+  "/usr/local/share/otelcol-sumo/otelcol-sumo.sh"
+  "/usr/local/share/otelcol-sumo"
   "/var/lib/otelcol-sumo/file_storage"
   "/var/lib/otelcol-sumo"
   "/var/log/otelcol-sumo"
 )
 
-# A list of files & directories to remove for hostmetrics
-hostmetrics_files=(
-  "/etc/otelcol-sumo/conf.d-available/hostmetrics.yaml"
-)
-
 function package_is_registered() {
   package_id="$1"
   pkgutil --pkg-info "$package_id"
@@ -114,7 +114,6 @@ function uninstall_package() {
 
 stop_service "${service_plist_file}"
 uninstall_package "com.sumologic.otelcol-sumo" "${collector_files[@]}"
-uninstall_package "com.sumologic.otelcol-sumo-hostmetrics" "${hostmetrics_files[@]}"
 
 # remove the directory that this script belongs to
 SCRIPT_DIR="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"

diff --git a/assets/services/launchd/com.sumologic.otelcol-sumo.plist b/assets/services/launchd/com.sumologic.otelcol-sumo.plist
@@ -6,12 +6,13 @@
     <string>otelcol-sumo</string>
     <key>ProgramArguments</key>
     <array>
-      <string>/usr/local/bin/otelcol-sumo</string>
-      <string>--config</string>
-      <string>/etc/otelcol-sumo/sumologic.yaml</string>
-      <string>--config</string>
-      <string>glob:/etc/otelcol-sumo/conf.d/*.yaml</string>
+      <string>/usr/local/share/otelcol-sumo/otelcol-sumo.sh</string>
     </array>
+    <key>EnvironmentVariables</key>
+    <dict>
+        <key>SUMOLOGIC_INSTALLATION_TOKEN</key>
+        <string></string>
+    </dict>
     <!-- Service user -->
     <key>UserName</key>
     <string>_otelcol-sumo</string>

diff --git a/assets/services/systemd/otelcol-sumo.service b/assets/services/systemd/otelcol-sumo.service
@@ -2,7 +2,7 @@
 Description=Sumo Logic Distribution for OpenTelemetry Collector
 
 [Service]
-ExecStart=/usr/local/bin/otelcol-sumo --config /etc/otelcol-sumo/sumologic.yaml --config "glob:/etc/otelcol-sumo/conf.d/*.yaml"
+ExecStart=/usr/share/otelcol-sumo/otelcol-sumo.sh
 User=otelcol-sumo
 Group=otelcol-sumo
 MemoryHigh=2000M

diff --git a/ci/verify_installer.sh b/ci/verify_installer.sh
@@ -37,6 +37,7 @@ system_files=(
   "usr"
   "usr/local"
   "usr/local/bin"
+  "usr/local/share"
   "var"
   "var/lib"
   "var/log"
@@ -48,23 +49,23 @@ expected_collector_files=(
   "etc/otelcol-sumo/conf.d"
   "etc/otelcol-sumo/conf.d/common.yaml"
   "etc/otelcol-sumo/conf.d-available"
+  "etc/otelcol-sumo/conf.d-available/ephemeral.yaml"
+  "etc/otelcol-sumo/conf.d-available/hostmetrics.yaml"
   "etc/otelcol-sumo/conf.d-available/examples"
+  "etc/otelcol-sumo/opamp.d"
   "etc/otelcol-sumo/sumologic.yaml"
   "Library/Application Support/otelcol-sumo"
   "Library/Application Support/otelcol-sumo/uninstall.sh"
   "Library/LaunchDaemons/com.sumologic.otelcol-sumo.plist"
   "usr/local/bin/otelcol-config"
   "usr/local/bin/otelcol-sumo"
+  "usr/local/share/otelcol-sumo"
+  "usr/local/share/otelcol-sumo/otelcol-sumo.sh"
   "var/lib/otelcol-sumo"
   "var/lib/otelcol-sumo/file_storage"
   "var/log/otelcol-sumo"
 )
 
-# a list of files that the hostmetrics package should install
-expected_hostmetrics_files=(
-  "etc/otelcol-sumo/conf.d-available/hostmetrics.yaml"
-)
-
 function install_package() {
   mpkg="$1"
   mpkg_basename="$2"
@@ -184,15 +185,9 @@ while IFS=  read -r -d $'\0'; do
   collector_pkg+=("$REPLY")
 done < <(find . -name "*-otelcol-sumo.pkg" -type d -print0)
 
-# create an array of hostmetrics packages (only one is expected)
-hostmetrics_pkg=()
-while IFS=  read -r -d $'\0'; do
-  hostmetrics_pkg+=("$REPLY")
-done < <(find . -name "*-otelcol-sumo-hostmetrics.pkg" -type d -print0)
-
 # verify that the expected number of sub-packages were found
 pkg_count="${#all_pkgs[@]}"
-expected_pkg_count=2
+expected_pkg_count=1
 
 if [ "$pkg_count" -ne $expected_pkg_count ]; then
   echo "error: ${expected_pkg_count} sub-packages were expected but found ${pkg_count}"
@@ -205,12 +200,6 @@ if [ "${#collector_pkg[@]}" -gt 1 ]; then
   exit 1
 fi
 
-# only one hostmetrics sub-package should exist
-if [ "${#hostmetrics_pkg[@]}" -gt 1 ]; then
-  echo "error: more than one hostmetrics sub-package was found"
-  exit 1
-fi
-
 # get a list of files installed by the collector sub-package excluding system
 # files
 collector_pkg_name="$(echo "${collector_pkg[0]}" | cut -d/ -f2-)"
@@ -243,41 +232,6 @@ for f in "${all_collector_files[@]}"; do
   collector_files+=("$collector_file")
 done
 
-cd "$expanded_dir" || exit
-
-# get a list of files installed by the hostmetrics sub-package excluding both
-# system files and collector files
-hostmetrics_pkg_name="$(echo "${hostmetrics_pkg[0]}" | cut -d/ -f2-)"
-cd "${hostmetrics_pkg_name}/Payload" || exit
-all_hostmetrics_files=()
-while IFS=  read -r -d $'\0'; do
-  all_hostmetrics_files+=("$REPLY")
-done < <(find . ! -name '.' -print0)
-
-hostmetrics_files=()
-
-for f in "${all_hostmetrics_files[@]}"; do
-  hostmetrics_file="$(echo "$f" | cut -d/ -f2-)"
-
-  # shellcheck disable=SC2076
-  if [[ " ${system_files[*]} " =~ " ${hostmetrics_file} " ]]; then
-    continue
-  fi
-
-  # shellcheck disable=SC2076
-  if [[ " ${collector_files[*]} " =~ " ${hostmetrics_file} " ]]; then
-    continue
-  fi
-
-  # shellcheck disable=SC2076
-  if [[ ! " ${expected_collector_files[*]} " =~ " ${collector_file} " ]]; then
-    echo "error: unexpected file installed by hostmetrics sub-package: ${hostmetrics_file}"
-    exit 1
-  fi
-
-  hostmetrics_files+=("$hostmetrics_file")
-done
-
 cd "$exec_dir" || exit
 
 install_package "$mpkg" "$mpkg_basename"
@@ -287,7 +241,6 @@ echo "##########################################################################
 echo "Verifying installation: ${mpkg}"
 echo "################################################################################"
 verify_installation "com.sumologic.otelcol-sumo" "${expected_collector_files[@]}"
-verify_installation "com.sumologic.otelcol-sumo-hostmetrics" "${expected_hostmetrics_files[@]}"
 
 echo
 echo "################################################################################"
@@ -305,6 +258,5 @@ echo "##########################################################################
 echo "Verifying uninstallation: ${mpkg}"
 echo "################################################################################"
 verify_uninstallation "com.sumologic.otelcol-sumo" "${expected_collector_files[@]}"
-verify_uninstallation "com.sumologic.otelcol-sumo-hostmetrics" "${expected_hostmetrics_files[@]}"
 
 echo "Success!"