Skip to main content

Add Test Suite

Use the Add Test Suite form to create a new test configuration in RoostGPT and prepare it for generation.

Form overview​

The Add Test Suite form is organized into the same sections shown in the left sidebar. Use these sections to configure the overall suite and the specific runtime or framework settings needed to generate tests.

Test Suite​

This section is the main configuration area for all suite types.

  • Name — A unique name for the test suite.
  • Test Type — Choose the type of suite you want to generate. Available options include:
    • Unit Test
    • API Test
    • Integration Test
    • Functional Test
    • UI Test
  • Code Language — Select the language for the test suite. Visible only when Test Type is set to Unit Test. Supported languages: Java, Python.
  • Test Framework — Select the framework used for the suite.
    • Unit Test
      • Java: JUnit6, JUnit5, JUnit4
      • Python: Pytest, Python Unit Test
    • API Test: Postman, Rest Assured, Karate, Pytest
    • Integration Test: Artillery, Rest Assured, Karate
    • UI Test: Playwright
  • Java / Python Version — Choose the version for the selected unit test language. See Supported Versions for the Java and Python options.
  • Java Build Tool — When Java is selected, choose Maven or Gradle and then select a matching version.
  • Maven Version or Gradle Version — Choose the build tool version that matches your project. See Supported Versions for the available options.
  • Share Test Suite with Organization — Toggle on to make the suite reusable by your organization.

Gen AI Models​

Configure the AI provider that RoostGPT uses to generate tests.

  • Supported providers include:
    • OpenAI
    • Claude AI
    • Azure AI
    • Bedrock AI
    • DeepSeek AI
    • Gemini AI

Configuration Steps​

  • Select Provider — Click the icon of your preferred AI provider.
  • Authentication Method — Choose one of the following:
    • Manual Key — Click the Key icon to enter your API token directly.
    • Existing Connector — Click the Connector icon to select a pre-configured integration.
  • Model Settings — Configure the model parameters:
    • Base URL — Required if using a custom OpenAI endpoint.
    • Model Selection — Choose the specific version (e.g., GPT-4o, Claude 3.5 Sonnet) from the dropdown.
    • AI Temperature — Adjust the randomness of the output (default: 0.2). Lower values result in more deterministic, consistent code.
    • Verify — Click to test the connection and ensure your credentials are valid.
  • Azure AI Specific Configuration —
    • Azure Model Type — Choose the model category:
      • Foundation Model — Use standard Azure-hosted base models.
      • Inference Endpoint Model — Use a specific deployed endpoint.
    • Azure Host — Enter your instance URL (e.g., https://roost-gpt.openai.azure.com).
    • Azure Inference Endpoint — (Inference Endpoint Model only) Enter the full URL provided by the Azure AI Model Catalog.
    • Azure Deployment Name — The specific name assigned to your model deployment in the Azure Portal (e.g., roostgpt-4-32k).
    • Azure Key — Your secure API key. Toggle the eye icon to mask/unmask the input.
    • Use Assistant (Beta) — Enable to utilize the OpenAI/Azure Assistant API for enhanced context handling.
  • Bedrock AI Specific Configuration —
    • AWS Credentials — Click Add AWS Credentials to open the credentials dialog. Two input methods are supported:
      • File — Select the File radio button, then click Choose Files to upload your AWS credentials file.
      • Access Key/Token — Select the Access Key/Token radio button and fill in the following fields:
        • AWS Access Key ID — Your IAM access key identifier.
        • AWS Secret Access Key — Your IAM secret key. Toggle the eye icon to mask/unmask the input.
        • AWS Session Token — (Optional) Required when using temporary credentials from AWS STS.
      • Click Save to store the credentials.
    • Bedrock AI Region — Select the AWS region where your Bedrock models are deployed (e.g., ap-south-1).
    • Get Models — Click this button after selecting a region to fetch the available models. Alternatively, select a pre-configured connector from the Connector Name dropdown (e.g., bedrock-ireland). Use the key icon to manage credentials or the info icon to view connector details.
    • Bedrock AI Models — After models are fetched, choose the model category:
      • Inference Profiles — Cross-region inference profiles that route requests across AWS regions for higher availability. Example models: eu.anthropic.claude-opus-4-7, global.anthropic.claude-sonnet-4-6, global.anthropic.claude-opus-4-6-v1.
      • Foundation Models — Base models hosted directly in the selected region. Example models: mistral.voxtral-mini-3b-2507, mistral.magistral-small-2509, mistral.ministral-3-8b-instruct.
    • Bedrock AI Model Name — Select the specific model from the searchable dropdown. Use the search box to filter by name.
    • Verify — Click to validate the selected model and credentials before saving.
    • Add Knowledge Base IDs — Toggle on to attach one or more AWS Bedrock Knowledge Base IDs to the test suite, enabling retrieval-augmented generation (RAG) during test generation. When enabled, available Knowledge Base IDs are populated in a list for selection.

Code Repositories​

This section configures the source code repository that RoostGPT will analyze.

  • Code Repo — Select the Git platform: GitHub, GitLab, Bitbucket, or Azure DevOps.

GitHub​

  • GitHub Type — Choose the hosting type:
    • Cloud — Standard GitHub.com.
    • Enterprise (self hosted) — A self-hosted GitHub Enterprise instance.
Source Repository​

Authenticate using one of the following methods:

  • GitHub Token — Enter your GitHub personal access token directly.
  • Connector Name — Select a pre-configured GitHub connector from the dropdown (e.g., aa-github). Use the key icon to manage credentials or the info icon to view connector details.

Once authenticated, configure the repository details:

  • Repository Name — Search for and select the repository (e.g., roostdev/openai-java). A Valid Repository badge confirms the repository is accessible.
  • Branch Name — Search for and select the branch (e.g., main). A Branch Verified badge confirms the branch exists.
  • Project Path — Enter the path within the repository to use as the project root (e.g., /). A Path Verified badge confirms the path is valid.
  • Packages — Specify the package filter pattern (e.g., * to include all packages).

Azure DevOps​

Source Repository​
  • Organization Name — Enter your Azure DevOps organization name.
  • Token — Enter your Azure DevOps Personal Access Token (PAT). Toggle the eye icon to mask/unmask the input. Click the Azure DevOps icon to open the portal or the connector icon to use a pre-configured integration. Click Verify to validate access.
tip

When creating a PAT, use the following settings:

  • Organization: All
  • Scope: User Profile (Read) and Code (Full)

GitLab​

  • GitLab Type — Choose the hosting type:
    • Cloud — Standard GitLab.com.
    • Self-managed — A self-hosted GitLab instance.
Source Repository​

Cloud:

  • GitLab Token — Enter your GitLab personal access token. Toggle the eye icon to mask/unmask the input. Click the GitLab icon to open the portal or the connector icon to use a pre-configured integration. Click Verify to validate access.

Self-managed:

  • Server URL — Enter the URL of your self-managed GitLab instance (e.g., https://192.xxx.x.xxx/). Click the connector icon to use a pre-configured integration.
  • GitLab Token — Enter your GitLab personal access token. Toggle the eye icon to mask/unmask the input. Click Verify to validate access.

Bitbucket​

  • Bitbucket Type — Choose the hosting type:
    • Cloud — Standard Bitbucket.org.
    • Server (self hosted) — A self-hosted Bitbucket Server instance.
Source Repository​

Cloud:

  • Credential Type — Choose the authentication method:
    • App Password — Authenticate using a Bitbucket app password.
    • Access Token — Authenticate using a workspace access token.
  • Username — Enter your Bitbucket username. Click the connector icon to use a pre-configured integration.
  • App Password / Workspace Access Token — Enter the credential matching the selected type. Toggle the eye icon to mask/unmask the input. Click Verify to validate access.

Server (self hosted):

  • Server URL — Enter the URL of your Bitbucket Server instance (e.g., https://192.xxx.x.xx). Click the connector icon to use a pre-configured integration.
  • Username — Enter your Bitbucket Server username.
  • Workspace Access Token — Enter your access token. Toggle the eye icon to mask/unmask the input. Click Verify to validate access.

Integration​

This section links the test suite to an external issue tracker. Connect Jira or Azure DevOps to associate generated tests with existing issues and track progress.

Jira​

  • Email — Enter the email address associated with your Jira account (e.g., your.name@yourcompany). Click the connector icon to use a pre-configured integration instead.
  • Host Name — Enter your Jira instance URL (e.g., yourdomain.atlassian.net).
  • Access Token — Enter your Jira API token. Toggle the eye icon to mask/unmask the input. Click Verify to validate the credentials.
  • Jira ID(s) — (Optional) Enter one or more Jira issue IDs to associate with this test suite. Click the info icon for formatting guidance.

Azure DevOps​

  • Organization Name — Enter your Azure DevOps organization name. Click the connector icon to use a pre-configured integration instead.
  • Access Token — Enter your Azure DevOps personal access token (PAT). Toggle the eye icon to mask/unmask the input. Click the info icon for guidance, then click Verify to validate the credentials.

API Options​

note

This section is visible only when Test Type is set to API Test.

Configure the Swagger/OpenAPI specification that RoostGPT will use to generate API tests.

  • Type — Currently supports Swagger (OpenAPI spec).
  • Swagger Source — Choose how to provide the specification file:
    • File — Upload a local Swagger file directly. Click Choose Files to select the file from your machine.
    • URL — Enter the raw content URL of the Swagger spec (e.g., a raw GitHub URL). Click + to add multiple URLs.
    • Git Path — Search for the Swagger file within the connected source repository using the Search swagger git file dropdown. Click + to add multiple paths.

Logs Setup​

note

This section is visible only when Test Type is set to API Test.

Configure where RoostGPT sends test execution logs. Select one of the following types:

  • None — No external log destination. Logs are not forwarded (default).
  • Elastic Search — Stream logs to an Elasticsearch instance. Choose a credential type:
    • Username/Password —
      • Elastic Search Endpoint — Enter the Elasticsearch endpoint URL.
      • Elastic Search Username — Enter the username for authentication.
      • Elastic Search Password — Enter the password. Toggle the eye icon to mask/unmask the input. Click Verify to validate the connection.
    • API Key —
      • Elastic Search Endpoint — Enter the Elasticsearch endpoint URL.
      • API Key — Enter the API key for authentication. Toggle the eye icon to mask/unmask the input. Click Verify to validate the connection.
  • Log File — Upload a log configuration file. Click Choose Files to select the file from your machine.

Filters​

note

This section is visible only when Test Type is set to API Test.

Use these filters to define the boundaries of your test generation:

  • HTTP Scope — Select the HTTP methods you want RoostGPT to include in the test suite: GET, POST, PUT, PATCH, DELETE.
  • API Scope (Regex) — Define the specific endpoints for which you want to generate tests or scripts. To include everything, use .*.

UI Test Specification​

note

This section is visible only when Test Type is set to UI Test.

Configure the target application and authentication details that RoostGPT will use to generate UI tests.

  • Base URL — Enter the base URL of the application under test (e.g., https://app.roost.ai).
  • Allowed URLs — Enter one or more HTTPS URLs that the UI test is permitted to navigate to. Click + to add multiple entries.
  • Login Type — Select the login mechanism used by the application (e.g., STANDARD).
  • Login URL — Enter the URL of the login page (e.g., https://app.roost.ai/login).
  • Login Username/Email — Enter the username or email address used to log in.
  • Login Password/Token — Enter the login password or authentication token. Toggle the eye icon to mask/unmask the input.
  • Login Scenario File — Upload a file describing the login flow. Click Choose Files to select it from your machine.
  • User Scenario File — Upload a file describing the user interaction scenarios for test generation. Click Choose Files to select it from your machine.
  • UI-Test Environment Variables — Click Add UI-Test Env Vars to define environment variables available during UI test execution.

Advanced​

Use this section to configure advanced options for test generation behavior, environment settings, and execution preferences.

  • Skip Methods With Existing Test — Toggle on to skip generating tests for methods that already have existing test coverage.
  • Environment Variables — Click Add Env Vars to define key-value environment variables available during test generation.
  • Execution Environment Variables — Click Add Execution Env Vars to define variables available during test execution.
  • Maximum Depth — Check Traverse to all sub-directories to scan the full directory tree under the project path, rather than only the top-level directory.
  • Vulnerability Testing — Check Check for Vulnerability to enable security vulnerability scanning as part of test generation.
  • Trigger Events — Select the repository events that should automatically trigger test generation:
    • Create PR — Trigger on pull request creation.
    • Push — Trigger on code push.
    • Upload file with additional information — Attach a supplementary file to provide extra context for generation.
  • Additional Prompt — Enter any free-text instructions to guide the AI during test generation.
  • User Language — Select the language for generated test comments and output (default: English).
  • Timeout (in hrs) — Set the maximum time allowed for test generation to complete (default: 1 hour).
  • Iterations — Set the number of generation attempts the AI will make (default: 1).
  • Functions to Test — Specify a comma-separated list of function names to limit generation to those functions only.
  • Custom Annotations — Add custom annotation tags to apply to generated tests. Click + to add multiple entries.

Notifications​

This section controls notifications for the test suite. Select a notification channel to receive alerts when test generation completes or execution events occur. Supported types: Email, Slack, and Teams.

Email​

  • Send notification to all project members — Check Send to all members to notify every member of the project automatically.
  • Receiver Emails — Enter one or more comma-separated email addresses to receive alerts.

Slack​

  • Slack API Token — Enter the API token received after installing the RoostAI app to your Slack workspace. Toggle the eye icon to mask/unmask the input. Click the connector icon to use a pre-configured integration, Verify Token to validate the token, or Add to Slack to install the RoostAI app directly.

Teams​

  • Tenant ID — Enter your Microsoft Teams tenant ID. Click the connector icon to use a pre-configured integration.

Supported Versions​

These options are available when creating a Unit Test in the Add Test form.

Java versions​

  • Java SE 21 (Default)
  • Java SE 20
  • Java SE 19
  • Java SE 18
  • Java SE 17
  • Java SE 16

Maven versions​

  • Maven 3.9.12 (Default)
  • Maven 3.9.11
  • Maven 3.9.10
  • Maven 3.9.9
  • Maven 3.9.8
  • Maven 3.9.7
  • Maven 3.9.6
  • Maven 3.8.9
  • Maven 3.8.8
  • Maven 3.8.7
  • Maven 3.8.6
  • Maven 3.8.2
  • Maven 3.8.1
  • Maven 3.8.0
  • Maven 3.6.9
  • Maven 3.6.8
  • Maven 3.6.7
  • Maven 3.6.6
  • Maven 3.6.5
  • Maven 3.6.4
  • Maven 3.6.3
  • Maven 3.6.2
  • Maven 3.6.1
  • Maven 3.6.0

Gradle versions​

  • Gradle 9.1.0 (Default)
  • Gradle 9.0.0
  • Gradle 8.14
  • Gradle 8.13
  • Gradle 8.12
  • Gradle 8.11
  • Gradle 8.10
  • Gradle 8.9
  • Gradle 8.8
  • Gradle 8.7
  • Gradle 8.6
  • Gradle 8.5
  • Gradle 8.4
  • Gradle 8.3
  • Gradle 7.6.4
  • Gradle 7.6.3

Python versions​

  • Python 3 Latest (Default)
  • Python 3.12
  • Python 3.11
  • Python 3.10
  • Python 3.9
  • Python 3.8
  • Python 3.7
  • Python 3.6