workflows/commit/test/prompt

Overview

This README provides information about the workflow used for generating commit messages based on commit differences. The workflow involves recording prompts, defining test cases, and running scripts to process these cases.

Dependency Installation

Before you can run the tests for commit message generation, you need to install the necessary dependencies. This involves setting up both Node.js and Python environments. Follow the steps below to install the dependencies.

Node.js Dependency

To install the promptfoo tool using npx, run the following command:

npx promptfoo@latest

This command will ensure that you are using the latest version of promptfoo.

Python Dependencies

The test script requires certain Python packages to be installed. To install these dependencies, navigate to the directory containing the requirements-test.txt file and run:

pip install -r commit/test/prompt/requirements-test.txt

This will install all the Python packages listed in the requirements-test.txt file, which are necessary for the test script to function correctly.

After completing these steps, you should have all the required dependencies installed and be ready to proceed with the test execution as described in the Getting Started section of this README.

Getting Started

To get started with the commit message generation workflow:

1. Ensure you have npx and promptfoo installed and configured on your system.
2. Navigate to the directory containing the workflow files.
3. Review and update the test cases in commit_message_tests.yaml as needed.
4. Run the test execution command to generate commit messages.
5. Use the test results viewing command to analyze the results.

For a quick way to execute the current test cases, you can use the provided script file ./run.sh. This script will automate the process of installation and test execution for you. Simply run the following command in your terminal:

./run.sh

This will ensure that all necessary dependencies are installed, and then it will proceed to run the tests as defined in your test cases file.

For any additional help or to report issues, please refer to the project's issue tracker or contact the repository administrator.

File Structure

• Prompt Record File: ./commit/diffCommitMessagePrompt.txt

  • This file contains the prompts that are used to generate commit messages.

• Test Cases File: ./commit/test/prompt/commit_message_tests.yaml

  • This YAML file holds the test cases for the commit message generation tests. Each test case includes the commit_url which points to the specific commit differences to be used for generating the commit message.

• Test Script: ./commit/test/prompt/commit_message_run.py

  • This is the Python script that processes the test cases. It fetches the commit differences from the provided commit_url and generates the commit messages accordingly.

• Test Configuration File: ./commit/test/prompt/commit_message_prompt_config.yaml

  • This YAML file provides the overall description of the tests and configurations needed for the test execution.

Test Execution

To execute the tests for commit message generation, use the following command:

npx promptfoo eval -c commit/test/prompt/commit_message_prompt_config.yaml

This command will run the tests as defined in the commit_message_prompt_config.yaml file.

Viewing Test Results

After running the tests, you can display the results using the command below:

npx promptfoo view

This command will present the outcomes of the tests, allowing you to review the generated commit messages and their respective test cases.