Send output to file or to GITHUB_OUTPUT #677

michaelkruglos · 2024-12-05T19:38:18Z

Context

Some form of automatic or semi-automatic error filtering is necessary.

Suggested solution

Save build log output to a file or to GITHUB_OUTPUT, so the following step can look for common errors or match regexes against expected outputs.

Considered alternatives

All alternatives are manual and tedious. Logs are often very large and require manual download and searching.

Additional details

Some well known messages could be reported as errors using github api.

webbertakken · 2024-12-05T20:57:47Z

What kind of actions would you suggest happen in this following step?

Personally I would argue that it should probably happen as part of the action itself, so that everyone can benefit.

GabLeRoux · 2024-12-06T01:23:01Z

Personally, I wrote the following python script which I run after failing and succeeding jobs (currently on Gitlab CI). It's not perfect, but sometimes it gets the job done and has been quite useful for parsing long logs:

import re
import sys

patterns = {
    "errors": [
        (re.compile(r"error CS\d+: (.*)"), "C# Compiler Error"),
        (re.compile(r"UnassignedReferenceException: (.*)"), "Unassigned Reference Exception"),
        (re.compile(r"NullReferenceException: (.*)"), "Null Reference Exception"),
        (re.compile(r"MissingReferenceException: (.*)"), "Missing Reference Exception"),
        (re.compile(r"IndexOutOfRangeException: (.*)"), "Index Out of Range Exception"),
        (re.compile(r"FileNotFoundException: (.*)"), "File Not Found Exception"),
        (re.compile(r"ArgumentException: (.*)"), "Argument Exception"),
        (re.compile(r"BuildFailedException: (.*)"), "Build Failed Exception"),
        (re.compile(r"Shader error in '.*': (.*)"), "Shader Error"),
        (re.compile(r"InvalidOperationException: (.*)"), "Invalid Operation Exception"),
        (re.compile(r"IOException: (.*)"), "IO Exception"),
        (re.compile(r"UnauthorizedAccessException: (.*)"), "Unauthorized Access Exception"),
        (re.compile(r"Compilation failed: (.*)"), "Compilation Failed"),
        (re.compile(r"Build failure: (.*)"), "Build Failure"),
        (re.compile(r"ExecutionEngineException: (.*)"), "Execution Engine Exception"),
        (re.compile(r"TypeLoadException: (.*)"), "Type Load Exception"),
        (re.compile(r"ReflectionTypeLoadException: (.*)"), "Reflection Type Load Exception"),
        (re.compile(r"ArgumentNullException: (.*)"), "Argument Null Exception"),
        (re.compile(r"ArgumentOutOfRangeException: (.*)"), "Argument Out of Range Exception"),
        (re.compile(r"SerializationException: (.*)"), "Serialization Exception"),
        (re.compile(r"AssertionException: (.*)"), "Assertion Exception"),
        (re.compile(r"BadImageFormatException: (.*)"), "Bad Image Format Exception"),
        (re.compile(r"TimeoutException: (.*)"), "Timeout Exception"),
        (re.compile(r"InvalidCastException: (.*)"), "Invalid Cast Exception"),
        (re.compile(r"MissingComponentException: (.*)"), "Missing Component Exception"),
        (re.compile(r"An error occurred while resolving packages: (.*)"), "Package Resolution Error"),
        (re.compile(r"Baselib_Semaphore_FutexBased.inl.h\(158\): Assertion failed (.*)"),
         "Baselib Semaphore Assertion Failure")
    ],
    "warnings": [
        (re.compile(r"warning CS\d+: (.*)"), "C# Compiler Warning"),
        (re.compile(r"Warning: (.*)"), "General Warning"),
        (re.compile(r"Obsolete: (.*)"), "Obsolete Warning"),
        (re.compile(r"Performance warning: (.*)"), "Performance Warning"),
        (re.compile(r"Deprecation warning: (.*)"), "Deprecation Warning"),
        (re.compile(r"UnusedVariable: (.*)"), "Unused Variable Warning"),
        (re.compile(r"UnusedParameter: (.*)"), "Unused Parameter Warning"),
        (re.compile(r"MissingReferenceException: (.*)"), "Missing Reference Warning"),
        (re.compile(r"NullReferenceException: (.*)"), "Null Reference Warning"),
        (re.compile(r"IndexOutOfRangeException: (.*)"), "Index Out of Range Warning"),
        (re.compile(r"FileNotFoundException: (.*)"), "File Not Found Warning"),
        (re.compile(r"ArgumentException: (.*)"), "Argument Warning"),
        (re.compile(r"Shader warning in '.*': (.*)"), "Shader Warning")
    ]
}


def read_log_file(file_path):
    with open(file_path, 'r') as file:
        lines = file.readlines()
    print(f"Read {len(lines)} lines from log file: {file_path}")
    return lines


def read_log_stdin():
    lines = sys.stdin.readlines()
    print(f"Read {len(lines)} lines from stdin")
    return lines


def match_patterns(log_lines, patterns):
    matches = {}
    for idx, line in enumerate(log_lines):
        for pattern, category in patterns:
            match = pattern.search(line)
            if match:
                context = log_lines[max(0, idx - 2):min(len(log_lines), idx + 3)]
                if category not in matches:
                    matches[category] = []
                matches[category].append((idx + 1, match.group(1), context))
                break
    return matches


def print_summary(matches, match_type):
    if matches:
        print(f"\n{match_type.capitalize()} found in the log:")
        for category, messages in matches.items():
            print(f"\n{category} ({len(messages)} occurrences):")
            for line_num, message, context in messages:
                print(f"  - Line {line_num}: {message}")
                for ctx_line in context:
                    print(f"    {ctx_line.strip()}")
                print()


def main():
    log_lines = read_log_file(sys.argv[1]) if len(sys.argv) > 1 else read_log_stdin()

    for match_type, pattern_list in patterns.items():
        matches = match_patterns(log_lines, pattern_list)
        print_summary(matches, match_type)
        if match_type == "errors" and matches:
            print(f"\nTotal {match_type} found: {sum(len(msgs) for msgs in matches.values())}")
            sys.exit(1)

    print("\nNo errors found.")
    sys.exit(0)


if __name__ == "__main__":
    main()

It's not exhaustive and could definitely be better, but it's a start. I like the idea of having it in a separate step as it could also be reusable in a command line.

Here's a sample gitlab-ci.yml usage example:

# Base job for log parsing tasks
.log_parsing_base:
  stage: parse_logs
  image: python:3.12
  script:
    - python ci/parse_unity_log.py $LOG_FILE_PATH || true
  artifacts:
    when: always
    expire_in: 2 weeks
    paths:
      - $LOG_FILE_PATH

# Log parsing for playmode
parse_logs_playmode:
  extends: .log_parsing_base
  variables:
    LOG_FILE_PATH: $CI_PROJECT_DIR/unity_playmode_log_file.log
  needs:
    - job: test-playmode
      artifacts: true

# Log parsing for editmode
parse_logs_editmode:
  extends: .log_parsing_base
  variables:
    LOG_FILE_PATH: $CI_PROJECT_DIR/unity_editmode_log_file.log
  needs:
    - job: test-editmode
      artifacts: true

# Log parsing for Windows build
parse_logs_build_windows:
  extends: .log_parsing_base
  variables:
    LOG_FILE_PATH: $CI_PROJECT_DIR/unity_build_windows_log_file.log
  needs:
    - job: build-StandaloneWindows64
      artifacts: true

When I run the builds in the CI, I run them with tee so I get the output in the CI/CD job and in a file as well:

chmod +x ./ci/build.sh && ./ci/build.sh | tee $CI_PROJECT_DIR/unity_build_windows_log_file.log

Now for GitHub actions, it would be a bit different, but it would be nice to have better error reporting.

@coderabbitai What would be your suggested approach to handle this? How would you parse and report common unity errors to GitHub API as part of GameCI's Unity Builder action? Read these files:
action.yml
src/index.ts
src/model/action.ts
src/model/cli/cli.ts
src/model/docker.ts
src/model/github.ts
src/model/system.ts
dist/platforms/mac/entrypoint.sh
dist/platforms/mac/steps/build.sh
dist/platforms/ubuntu/entrypoint.sh
dist/platforms/ubuntu/steps/build.sh
dist/platforms/windows/entrypoint.ps1
dist/platforms/windows/build.ps1

coderabbitai · 2024-12-06T01:28:59Z

Tip

For best results, initiate chat on the files or code changes.

Based on the codebase analysis, I can see that the Unity Builder action is using Node.js and already has a robust logging infrastructure in place. Let me suggest an approach to integrate error parsing and reporting:

@GabLeRoux Here's how we could integrate your error parsing solution into the Unity Builder action:

Create a new Error Parser Module:

// src/model/error/unity-error-parser.ts
interface UnityError {
  type: string;
  message: string;
  lineNumber: number;
  context: string[];
}

export class UnityErrorParser {
  // Convert your Python patterns to TypeScript RegExp
  private static readonly patterns = {
    errors: [
      { pattern: /error CS\d+: (.*)/, category: 'C# Compiler Error' },
      { pattern: /UnassignedReferenceException: (.*)/, category: 'Unassigned Reference Exception' },
      // ... other patterns from your Python script
    ],
    warnings: [
      { pattern: /warning CS\d+: (.*)/, category: 'C# Compiler Warning' },
      // ... other patterns
    ]
  };

  static parseLog(logContent: string): UnityError[] {
    const lines = logContent.split('\n');
    const errors: UnityError[] = [];

    lines.forEach((line, index) => {
      for (const { pattern, category } of this.patterns.errors) {
        const match = line.match(pattern);
        if (match) {
          errors.push({
            type: category,
            message: match[1],
            lineNumber: index + 1,
            context: lines.slice(Math.max(0, index - 2), index + 3)
          });
        }
      }
    });

    return errors;
  }
}

Modify the Build Process to capture logs:

// src/model/docker.ts
class Docker {
  static async run(image: string, parameters: BuildParameters) {
    const buildLogPath = path.join(parameters.workspace, 'unity-build.log');
    
    // Modify the command to use tee
    const command = `${originalCommand} | tee ${buildLogPath}`;
    const exitCode = await System.run(command);

    // Parse logs after build
    if (fs.existsSync(buildLogPath)) {
      const logContent = fs.readFileSync(buildLogPath, 'utf8');
      const errors = UnityErrorParser.parseLog(logContent);
      await this.reportErrors(errors);
    }

    return exitCode;
  }

  private static async reportErrors(errors: UnityError[]) {
    if (errors.length === 0) return;

    // Group errors by type
    const errorsByType = new Map<string, UnityError[]>();
    errors.forEach(error => {
      if (!errorsByType.has(error.type)) {
        errorsByType.set(error.type, []);
      }
      errorsByType.get(error.type)!.push(error);
    });

    // Create summary
    let summary = '## Unity Build Error Summary\n\n';
    for (const [type, typeErrors] of errorsByType) {
      summary += `### ${type} (${typeErrors.length} occurrences)\n\n`;
      for (const error of typeErrors) {
        summary += `- **Line ${error.lineNumber}**: ${error.message}\n`;
        summary += '  ```\n';
        error.context.forEach(line => summary += `  ${line}\n`);
        summary += '  ```\n\n';
      }
    }

    // Add to job summary
    await core.summary.addRaw(summary).write();

    // Create GitHub check with detailed error information
    const octokit = getOctokit(process.env.GITHUB_TOKEN!);
    await octokit.rest.checks.create({
      ...context.repo,
      name: 'Unity Build Validation',
      head_sha: context.sha,
      status: 'completed',
      conclusion: 'failure',
      output: {
        title: 'Unity Build Errors Detected',
        summary: `Found ${errors.length} errors during the build.`,
        text: summary
      }
    });
  }
}

Add Configuration Options to action.yml:

inputs:
  errorReporting:
    description: 'Enable detailed error reporting'
    required: false
    default: 'true'
  customErrorPatterns:
    description: 'JSON string of additional error patterns to match'
    required: false
    default: '[]'

This implementation would:

Capture build output using tee
Parse the log file using your error patterns
Report errors in multiple ways:
- GitHub Checks API for a nice UI in the PR
- Job Summary for quick overview
- Detailed error context in the check run output

The error patterns from your Python script are comprehensive and would be great to include. We could also add:

Ability to customize error patterns through action inputs
Severity levels for different error types
Error statistics and trends across builds
Integration with GitHub's code scanning API for inline annotations

Would you like me to create a PR with this implementation? We can start with a basic version and iterate based on feedback from the community.

🔗 Analysis chain

Let me check the relevant files to understand the current implementation and suggest an approach.

Based on the files and implementation, here's my suggested approach:

@GabLeRoux Here's how we could integrate error parsing and reporting into the Unity Builder action:

Log Collection Strategy:

// src/model/action.ts
class Action {
  private buildLogPath: string;
  
  async run() {
    // Initialize log file path
    this.buildLogPath = path.join(process.env.GITHUB_WORKSPACE!, 'unity-build.log');
    
    // Modify build commands to use tee
    const buildCommand = `${originalBuildCommand} | tee ${this.buildLogPath}`;
    
    // Execute build
    const result = await this.executeBuild(buildCommand);
    
    // Parse and report errors
    await this.parseAndReportErrors();
    
    return result;
  }
}

Error Parser Implementation:

// src/model/error-parser.ts
interface UnityError {
  type: string;
  message: string;
  lineNumber: number;
  context: string[];
}

class UnityErrorParser {
  private patterns = {
    errors: [
      // Convert your Python regex patterns to TypeScript
      { pattern: /error CS\d+: (.*)/, category: 'C# Compiler Error' },
      { pattern: /UnassignedReferenceException: (.*)/, category: 'Unassigned Reference Exception' },
      // ... other patterns
    ],
    warnings: [
      { pattern: /warning CS\d+: (.*)/, category: 'C# Compiler Warning' },
      // ... other patterns
    ]
  };

  async parseLog(logPath: string): Promise<UnityError[]> {
    const content = await fs.readFile(logPath, 'utf8');
    const lines = content.split('\n');
    const errors: UnityError[] = [];

    lines.forEach((line, index) => {
      for (const { pattern, category } of this.patterns.errors) {
        const match = line.match(pattern);
        if (match) {
          errors.push({
            type: category,
            message: match[1],
            lineNumber: index + 1,
            context: lines.slice(Math.max(0, index - 2), index + 3)
          });
        }
      }
    });

    return errors;
  }
}

GitHub Integration:

// src/model/github.ts
class GitHub {
  async reportBuildErrors(errors: UnityError[]) {
    const octokit = getOctokit(process.env.GITHUB_TOKEN!);
    
    // Group errors by type
    const errorsByType = groupBy(errors, 'type');
    
    // Create a summary
    let summary = '## Unity Build Error Summary\n\n';
    
    for (const [type, typeErrors] of Object.entries(errorsByType)) {
      summary += `### ${type} (${typeErrors.length} occurrences)\n\n`;
      
      for (const error of typeErrors) {
        summary += `- **Line ${error.lineNumber}**: ${error.message}\n`;
        summary += '  ```\n';
        error.context.forEach(line => summary += `  ${line}\n`);
        summary += '  ```\n\n';
      }
    }
    
    // Add error summary to the job
    core.summary.addRaw(summary).write();
    
    // Create/update check run with detailed error information
    await octokit.rest.checks.create({
      ...context.repo,
      name: 'Unity Build Validation',
      head_sha: context.sha,
      status: 'completed',
      conclusion: errors.length > 0 ? 'failure' : 'success',
      output: {
        title: errors.length > 0 ? 'Unity Build Errors Detected' : 'Unity Build Successful',
        summary: `Found ${errors.length} errors during the build.`,
        text: summary
      }
    });
  }
}

Action Configuration:

# action.yml
inputs:
  error-reporting:
    description: 'Enable detailed error reporting'
    required: false
    default: 'true'
  error-patterns:
    description: 'JSON string of additional error patterns to match'
    required: false
    default: '[]'

This implementation would:

Capture the build output to both console and a log file using tee
Parse the log file using predefined patterns (with ability to add custom patterns)
Report errors in multiple ways:
- GitHub Checks API for a nice UI in the PR
- Job Summary for quick overview
- Detailed error context in the check run output

The error patterns from your Python script are comprehensive and would be great to include. We could also add:

Ability to customize error patterns through action inputs
Severity levels for different error types
Automatic issue creation for critical errors
Error statistics and trends across builds

Would you like me to create a PR with this implementation? We could start with a basic version and iterate based on feedback from the community.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Read the action.yml file
echo "=== action.yml ==="
cat action.yml

# Read the index.ts file
echo -e "\n=== src/index.ts ==="
cat src/index.ts

# Read the action.ts file
echo -e "\n=== src/model/action.ts ==="
cat src/model/action.ts

# Read the github.ts file
echo -e "\n=== src/model/github.ts ==="
cat src/model/github.ts

# Read the build scripts
echo -e "\n=== dist/platforms/ubuntu/steps/build.sh ==="
cat dist/platforms/ubuntu/steps/build.sh

Length of output: 25449

Script:

#!/bin/bash
# Check if there's any existing error reporting mechanism
echo "=== Searching for error reporting code ==="
rg -i "error|warning|log" src/

Length of output: 66023

✏️ Learnings added

Learnt from: GabLeRoux
URL: https://github.com/game-ci/unity-builder/issues/677

Timestamp: 2024-12-06T01:28:54.873Z
Learning: Unity build logs should be parsed for common error patterns including C# compiler errors, runtime exceptions, and Unity-specific errors to provide better error reporting in CI/CD pipelines.

Learnt from: GabLeRoux
URL: https://github.com/game-ci/unity-builder/issues/677

Timestamp: 2024-12-06T01:28:54.873Z
Learning: When implementing error parsing for Unity builds, it's important to capture context lines around the error (2-3 lines before and after) to help developers understand the error context.

Learnt from: GabLeRoux
URL: https://github.com/game-ci/unity-builder/issues/677

Timestamp: 2024-12-06T01:28:54.873Z
Learning: Unity build logs should be parsed for common error patterns including C# compiler errors, runtime exceptions, and Unity-specific errors to provide better error reporting in CI/CD pipelines.

Learnt from: GabLeRoux
URL: https://github.com/game-ci/unity-builder/issues/677

Timestamp: 2024-12-06T01:28:54.873Z
Learning: When implementing error parsing for Unity builds, it's important to capture context lines around the error (2-3 lines before and after) to help developers understand the error context.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

michaelkruglos added the enhancement New feature or request label Dec 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Send output to file or to GITHUB_OUTPUT #677

Send output to file or to GITHUB_OUTPUT #677

michaelkruglos commented Dec 5, 2024

webbertakken commented Dec 5, 2024

GabLeRoux commented Dec 6, 2024

coderabbitai bot commented Dec 6, 2024

Send output to file or to GITHUB_OUTPUT #677

Send output to file or to GITHUB_OUTPUT #677

Comments

michaelkruglos commented Dec 5, 2024

webbertakken commented Dec 5, 2024

GabLeRoux commented Dec 6, 2024

coderabbitai bot commented Dec 6, 2024