Moon bit test agent

.github/workflows/coverage_and_test.yml

@@ -0,0 +1,98 @@
+name: coverage_and_test
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, edited]
+    branches:
+      - main  
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - uses: actions/checkout@v4
+        with:  
+          python-version: 3.9  
+
+      - name: install dependencies  
+        run: |  
+          pip install -r scripts/requirements.txt
+
+      - name: install Moonbit CLI
+        run: |
+          curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash
+          echo "$HOME/.moon/bin" >> $GITHUB_PATH
+
+      - name: initial moon test
+        run: moon test --enable-coverage
+
+      - name: initial coverage report
+        run: |
+          moon coverage report -f summary > coverage_summary.txt
+
+          cat coverage_summary.txt >> "$GITHUB_STEP_SUMMARY"
+
+          moon coverage report -f coveralls
+
+      - name: coverage improvement
+        run: python scripts/main.py --api_key ${{ secrets.api_key }}
+
+      - name: get code changes
+        id: get_code_changes
+        run: |
+            git diff > changes.txt   
+            test_code=$(cat changes.txt)   
+            echo "test_code=$test_code" >> $GITHUB_ENV  
+
+      - name: push comments
+        uses: peter-evans/create-or-update-comment@v1
+        env:
+            GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+            commit-sha: ${{ github.event.pull_request.head.sha }}
+            body: |
+              Here are the test code changes:  
+              ```  
+              ${{ env.test_code }}  
+              ```
+            reaction-type: rocket
+
+  typo-check:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    env:
+      FORCE_COLOR: 1
+      TYPOS_VERSION: v1.19.0
+    steps:
+      - name: download typos
+        run: curl -LsSf https://github.com/crate-ci/typos/releases/download/$TYPOS_VERSION/typos-$TYPOS_VERSION-x86_64-unknown-linux-musl.tar.gz | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - name: check typos
+        run: typos
+
+  license-header-check:
+    runs-on: ubuntu-latest
+    env:
+      HAWKEYE_VERSION: v5.5.1
+    steps:
+      - uses: actions/checkout@v4
+      - name: Download HawkEye
+        run: curl --proto '=https' --tlsv1.2 -LsSf https://github.com/korandoru/hawkeye/releases/download/$HAWKEYE_VERSION/hawkeye-installer.sh | sh
+      - name: Check License Header
+        run: hawkeye check
+
+
+
+
+
+
+
+
+
+

scripts/gettest.py

@@ -0,0 +1,113 @@
+from zhipuai_model import ChatZhipuAI
+from langchain_core.output_parsers import StrOutputParser
+from langchain_core.prompts import ChatPromptTemplate
+import os
+
+
+def generate_test_code(moonbit, path, api_key):
+    filename = os.path.basename(path)
+    test_prompt = ChatPromptTemplate.from_messages(
+    [
+        ("system", """
+            As a MoonBit language engineer, your task is to write a series of test cases to verify the correctness of a project.
+            I will provide the following information:
+            1. **Filename**: This helps you understand the context of the MoonBit Code and guide you in using the correct data structures for generating test cases.
+            2. **MoonBit Code**: This is the MoonBit language code that the test case is supposed to test.
+
+            Please carefully read this information and generate correct test cases for the MoonBit Code with your knowledge of MoonBit language.
+
+            **Input Format:**
+            The filename is <filename>
+            The MoonBit code is <moonbit>
+
+            **Output Format:**
+            ```moonbit
+            test {{
+                assert_eq!(moonbit_code)
+            }}
+            ```
+
+            Note that your output should only contain the code for the test cases, without any analysis, explanations, or any other statements.
+            Also, ensure that you are generating test cases for the MoonBit language, and do not confuse it with any other language.
+        """),
+        ("user", "The filename is \"{filename}\"\nThe MoonBit code is\n{moonbit}")
+    ]
+)
+
+
+    test_llm = ChatZhipuAI(
+        api_key=api_key, model="glm-4-9b:772570335:v5:iwfb27vl", temperature=0.7, max_tokens=4095
+    )
+
+    test_retriever_chain = test_prompt | test_llm | StrOutputParser()
+    test_code_output = test_retriever_chain.invoke(
+        {"filename": filename, "moonbit": moonbit}
+    )
+    test_code = test_code_output.replace("```moonbit\n", "").rstrip(
+        "```"
+    )
+    return test_code
+
+
+def rethink_test_code(moonbit_code, test_moonbit_code, file_path, api_key):
+    filename = os.path.basename(file_path)
+    rethink_prompt = ChatPromptTemplate.from_template(
+        """You are a professional MoonBit language engineer. Now, you need to help me analyze and correct a test case. 
+        I will provide the following information:
+        1. **Test Case Filename**: This helps you understand the context of the test case.
+        2. **Test Case Code**: This is the current test case code, which may contain errors leading to test failures.
+        3. **MoonBit Code**: This is the actual MoonBit code that the test case is supposed to test.
+
+        Please carefully read this information, analyze the cause of the error, and generate a corrected test case code. 
+        Ensure that your output code passes the test and is logically correct.
+
+        **Input Format:**
+        Filename: <Filename>
+        Test Case Code:<test case code>
+        MoonBit Code:<MoonBit code>
+
+        **Output Format:**
+        ```moonbit  
+           test {{  
+           assert_eq!(moonbit_code)    
+         }}
+        ```
+
+        If there are issues with the values in the test case, 
+        you can remove the assertion values in the assert statement like:
+
+        ```test_moonbit_code
+        test "to_string" {{
+         let arr = [1, 2, 3]
+         let str = arr.to_string()
+         assert_eq!(str,"[1, 2, 3]")
+          }}
+        ```
+
+        ```output
+         test "to_string" {{
+         let arr = [1, 2, 3]
+         let str = arr.to_string()
+         assert_eq!(str)
+          }}
+        ```
+        Now, please generate the corrected test case code based on the following input information:
+        MoonBit Code:{moonbit_code}
+        Filename: {filename}
+        Test Case Code:{test_moonbit_code}  
+
+        Please note, your output should only contain the corrected moonbit language test case code, without any additional analysis.
+                """
+    )
+    rethink_llm = ChatZhipuAI(
+        api_key=api_key, model="glm-4-9b:772570335:v5:iwfb27vl", temperature=0.7, max_tokens=4095
+    )
+
+    rethink_retriever_chain = rethink_prompt | rethink_llm | StrOutputParser()
+    test_code_output = rethink_retriever_chain.invoke(
+        {"moonbit_code": moonbit_code, "filename": filename, "test_moonbit_code": test_moonbit_code}
+    )
+    test_code = test_code_output.replace("```moonbit\n", "").rstrip(
+        "```"
+    )
+    return test_code

scripts/main.py

@@ -0,0 +1,38 @@
+from readcoverage import get_coverage_from_summary
+from testagent import testagent
+import argparse
+import subprocess
+
+
+prev_coverage = get_coverage_from_summary("coverage_summary.txt")
+max_iterations = 5
+iteration = 0
+coverage_improved = True
+parser = argparse.ArgumentParser(description="to load API_KEY。")
+parser.add_argument(
+    "--api_key",
+    type=str,
+    help="API_KEY",
+)
+args = parser.parse_args()
+zhipuai_api_key = args.api_key
+new_coverage = prev_coverage
+while coverage_improved and iteration < max_iterations:
+    iteration += 1
+    testagent(zhipuai_api_key)
+    subprocess.run(["moon", "test", "--enable-coverage"])
+    subprocess.run(["moon", "coverage", "report", "-f", "coveralls"])
+    subprocess.run(
+        ["moon", "coverage", "report", "-f", "summary"],
+        stdout=open("coverage_summary.txt", "w"),
+    )
+    new_coverage = get_coverage_from_summary("coverage_summary.txt")
+
+    if new_coverage > prev_coverage:
+        prev_coverage = new_coverage
+        print(f"Coverage improved to {new_coverage}%")
+    else:
+        coverage_improved = False
+        print("Coverage did not improve. Stopping loop.")
+
+print(f"Final coverage: {new_coverage}%")

scripts/readcoverage.py

@@ -0,0 +1,58 @@
+from zhipuai_model import ChatZhipuAI
+from langchain_core.output_parsers import StrOutputParser
+from langchain_core.prompts import ChatPromptTemplate
+
+
+def get_line_content(moonbit_code, index):
+    lines = moonbit_code.strip().split("\n")
+
+    if 0 <= index < len(lines):
+        return lines[index]
+    else:
+        return "Index out of range"
+
+
+def get_coverage_from_summary(file_path):
+    with open(file_path, "r") as file:
+        lines = file.readlines()
+        last_line = lines[-1].strip()
+
+        parts = last_line.split(":")
+        if len(parts) == 2 and parts[0].strip() == "Total":
+            total_parts = parts[1].strip().split("/")
+            if len(total_parts) == 2:
+                total_passed = int(total_parts[0])
+                total_tests = int(total_parts[1])
+                coverage = total_passed / total_tests
+                return coverage
+    return 0.0
+
+
+def read_coverage(moonbit, index, api_key):
+    uncovered_code = get_line_content(moonbit, index)
+    read_prompt = ChatPromptTemplate.from_template(
+        """
+        The following piece of code from a larger moonbit language codebase (moonbit_code):{moonbit_code}
+
+         You are provided with one line of code which is uncovered in the test(uncovered_code):{uncovered_code}
+
+         Your task is to identify the entire function that this line of code belongs to and return the complete function definition.
+
+          Please ensure that you include all lines of the function from its definition to the end of the function body.
+
+          Your output should follow this format:
+
+          **Output Format:**
+            ```moonbit  
+               <complete function>
+            ```
+        """
+    )
+
+    read_llm = ChatZhipuAI(api_key=api_key, model="glm-4-plus", temperature=0.5)
+
+    read_retriever_chain = read_prompt | read_llm | StrOutputParser()
+    response = read_retriever_chain.invoke(
+        {"moonbit_code": moonbit, "uncovered_code": uncovered_code}
+    )
+    return response

scripts/readme.md

@@ -0,0 +1,58 @@
+# MoonBit Test Agent
+
+This repository includes the scripts and dependencies required for the Moonbit Test Agent. These scripts are utilized for coverage analysis and the generation of test cases within the Moonbit project.
+
+## Script Description
+
+- **Main.py**: The main workflow of our Test Agent. Compared to using a LangChain agent, directly executing LLMs within a fixed task workflow can save more time.
+- **Gettest.py**: Invokes the agent to generate test cases.
+- **Readcoverage.py**: Reads the coverage report generated after testing and identifies uncovered code areas based on the index.
+- **TestAgent.py**: Serves as the workflow for the generate test cases, coordinating the execution of coverage analysis and test case generation.
+- **Writedown.py**: Test the test code and writes the generated test cases.
+- **coverage_and_test.yml**: Defines the complete workflow.
+- **requirements**: Lists the dependencies needed to run the scripts.
+
+### Installation
+
+To use MoonBit_Test_Agent, you need to have an API key for ZhiPuAI.
+
+1. Install MoonBit
+   ```
+   curl -fsSL https://cli.moonbitlang.com/install/unix.sh | bash
+   ```
+2.Install the required dependencies:
+   ```
+   pip install -r requirements.txt
+   ```
+3.Add API_KEY in the Github Secret
+
+## Usage
+
+The Test Agent will work in the Github Actioon.
+Our workflow includes "Test", "Typo Check", and "License Header Check".
+
+### 1. Test Job
+- **Steps**:
+  1. **Install Dependencies**: Installs required Python packages.
+  2. **Install Moonbit CLI**: Installs the Moonbit command-line interface.
+  3. **Initial Moon Test**: Runs tests with coverage enabled.
+  4. **Initial Coverage Report**: Generates and displays a summary of test coverage.
+  5. **Coverage Improvement**: Iteratively runs tests to improve coverage.
+  6. **Get code changes**: Captures any code changes made in the pull request and stores them for later use..
+  7. **Push Comments**: Uses a GitHub Action to create or update a comment on the pull request with the changes to the test code.
+
+### 2. Typo Check Job
+- **Steps**:
+  1. **Download Typos**: Installs the Typos tool.
+  2. **Checkout Repository**: Clones the repository.
+  3. **Check Typos**: Scans the codebase for common typos.
+
+### 3. License Header Check Job
+- **Steps**:
+  1. **Checkout Repository**: Clones the repository.
+  2. **Download HawkEye**: Installs the HawkEye tool.
+  3. **Check License Header**: Ensures all files have the correct license header.
+  2. **Download HawkEye**: Installs the HawkEye tool.
+  3. **Check License Header**: Ensures all files have the correct license header.
+
+

scripts/requirements.txt

@@ -0,0 +1,4 @@
+langchain==0.3.1  
+argparse==1.4.0  
+langchain_community==0.3.1  
+pyjwt==2.9.0