Lesson: Automating Selenium Script Generation in Python and GitHub Push Using GPT API and Google Colab

Introduction

In this lesson, we will automate the process of generating a Selenium test script in Python based on a webpage’s DOM structure using Playwright, OpenAI’s GPT API, and Google Colab. We’ll also automate pushing the generated Python script to a GitHub repository using a GitHub API key.

Learning Objectives

By the end of this lesson, you will be able to:

  1. Extract the DOM structure of a webpage using Playwright.
  2. Generate a Python Selenium test script based on this DOM structure using OpenAI’s GPT API.
  3. Push the generated Python script to a GitHub repository automatically from Google Colab.

Step 1: Setting Up Google Colab for Playwright and GitHub Operations

Start by opening a Google Colab notebook, which allows you to execute Python code in a streamlined cloud environment.

Required Libraries

Ensure the following libraries are available in your Colab environment:

  • playwright for DOM extraction.
  • openai for AI-driven script generation.
  • nest_asyncio to handle asynchronous calls.

Install these packages with:

!pip install playwright openai nest_asyncio
!playwright install

Step 2: Extracting DOM Structure Using Playwright

The following Python script uses Playwright to fetch the DOM structure of a webpage. This structure will then serve as input for the GPT API to generate a Python Selenium script.

import os
import nest_asyncio
import asyncio
from playwright.async_api import async_playwright
import openai

# Allow nested asyncio calls in Colab
nest_asyncio.apply()

# Detect if running in Google Colab and set API key accordingly
try:
    from google.colab import userdata
    IN_COLAB = True
except ImportError:
    IN_COLAB = False

if IN_COLAB:
    os.environ["OPENAI_API_KEY"] = userdata.get('OPENAI_API_KEY')
else:
    os.environ["OPENAI_API_KEY"] = os.getenv('OPENAI_API_KEY')

# Initialize the OpenAI client
openai.api_key = os.getenv("OPENAI_API_KEY")

# Async function to retrieve the entire body structure dynamically using Playwright
async def fetch_body_structure(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)
        body_structure = await page.evaluate("document.body.outerHTML")
        await browser.close()
    return body_structure

# Function to generate Selenium test script using OpenAI's ChatGPT
def generate_selenium_script(body_html):
    prompt = f"""
    Given the following HTML body structure:

    ```html
    {body_html}
    ```

    Generate Python Selenium code (only the code, no comments or explanations) that:

    1. Navigates to the webpage: https://practiceautomatedtesting.com/webelements/Checkboxes.
    2. Selects each checkbox by clicking on the associated `label` or `span` element. Avoid using the `input` element's ID or directly interacting with the checkbox `input`.
    3. Verifies whether selecting each checkbox displays a "😊" or "😠" using spans.
    4. Follows best practices, including waiting for elements to load, validating interactions, and checking expected outcomes.
    5. Uses the pytest framework to structure tests.
    
    Return only the Python code enclosed within ```python and ```.
    """

    response = openai.Completion.create(
        model="gpt-4",
        prompt=prompt,
        max_tokens=500
    )

    script_code = response.choices[0].text.strip().split("```python")[1].split("```")[0].strip()
    print("Generated script:\n", script_code)
    return script_code

# Step 1: Fetch body structure from the live webpage
url = "https://practiceautomatedtesting.com/webelements/Checkboxes"
body_html = asyncio.run(fetch_body_structure(url))

# Step 2: Generate the Selenium script using the body HTML structure
script_code = generate_selenium_script(body_html)

# Step 3: Save the Selenium script to a file
script_file_path = "/content/python_test_script.py"
with open(script_file_path, "w") as script_file:
    script_file.write(script_code)
print(f"Selenium script saved to {script_file_path}")

Step 3: Pushing the Generated Script to GitHub

Once we have the generated Python Selenium script, we’ll save it to a file and push it to GitHub.

Creating a GitHub API Key

  1. Go to GitHub > Settings > Developer settings > Personal access tokens.
  2. Click Generate new token and select the required permissions (e.g., repo).
  3. Copy the generated token and save it securely for use in your Colab environment.

Automating GitHub Operations in Colab

The following code automates GitHub operations in Colab, including cloning the repository, committing the script, and pushing changes.

import os
import shutil
from google.colab import userdata

# GitHub credentials and repository information
GITHUB_TOKEN = userdata.get('GITHUB_TOKEN')  # Ensure your GitHub token is set in Colab
GITHUB_USERNAME = "your_github_username"
REPO_NAME = "your_repo_name"
BRANCH_NAME = "main"

# Define the repository path and URL
repo_path = f"/content/{REPO_NAME}"
repo_url = f"https://{GITHUB_USERNAME}:{GITHUB_TOKEN}@github.com/{GITHUB_USERNAME}/{REPO_NAME}.git"

# Configure Git
!git config --global user.email "your-email@example.com"
!git config --global user.name "Your Name"

# Step 1: Clone or Pull the Repository
if os.path.exists(repo_path):
    print(f"Repository already exists at {repo_path}. Attempting to pull latest changes.")
    pull_result = !git -C {repo_path} pull origin {BRANCH_NAME}
    if "fatal" in pull_result[0].lower():
        print("Repository pull failed. Re-cloning the repository.")
        shutil.rmtree(repo_path)
        !git clone -b {BRANCH_NAME} {repo_url} {repo_path}
else:
    print(f"Cloning repository into {repo_path}.")
    clone_result = !git clone -b {BRANCH_NAME} {repo_url} {repo_path}
    if "fatal" in clone_result[0].lower():
        print("Repository clone failed. Check credentials or repository URL.")
        raise SystemExit("Failed to clone the repository.")

# Verify that .git exists in the repo_path
if not os.path.exists(os.path.join(repo_path, ".git")):
    print("Repository initialization failed. Retrying with git init.")
    !git -C {repo_path} init
    !git -C {repo_path} remote add origin {repo_url}
    !git -C {repo_path} fetch
    !git -C {repo_path} checkout {BRANCH_NAME}

# Step 2: Copy the Selenium Script into the Repository
script_file_path = "/content/python_test_script.py"  # Path to the generated Selenium script
destination_path = f"{repo_path}/tests/selenium_script.py"

# Ensure the destination directory exists
os.makedirs(os.path.dirname(destination_path), exist_ok=True)

# Copy the generated Selenium script into the cloned repository
shutil.copyfile(script_file_path, destination_path)
print(f"Copied {script_file_path} to {destination_path}")

# Step 3: Commit and Push the Changes to GitHub
!git -C {repo_path} add .
!git -C {repo_path} commit -m "Add or update generated Selenium test script"
!git -C {repo_path} push origin {BRANCH_NAME}

print("Selenium script pushed to GitHub successfully.")

Summary

By following this lesson, you now know how to automate DOM extraction using Playwright, generate a Python Selenium script with GPT-4, and push it to GitHub, all within Google Colab. This setup helps streamline your test automation workflow and improve efficiency.


Try It Yourself: Automating Selenium Script Generation in Google Colab

Instructions

1. Open the Colab Notebook

2. Set Up Environment and Install Libraries

  • Run the first cell to install any required libraries (playwright, openai, and nest_asyncio) that are necessary for extracting DOM elements and automating code generation.

3. Enter API Keys

  • Ensure you have your OpenAI API Key and GitHub API Key ready. These keys are necessary for connecting to OpenAI for script generation and pushing your results to GitHub. Add your OpenAI key in the designated cell. or in the notebook access key

4. Fetch the DOM Structure

  • Run the provided code cell to fetch the DOM structure from a specified webpage using Playwright. The notebook contains code to access elements dynamically.

5. Generate the Selenium Script

  • Use OpenAI’s GPT-4 model in the following cell to generate a Python Selenium test script based on the extracted DOM structure. Follow the prompts to customize the output.

6. Save and Push to GitHub

  • Input your GitHub repository details, such as username and repository name. Run the final cell to push the generated Selenium script to your GitHub repository.

7. Check Results

  • After running all cells, check your GitHub repository to confirm that the generated script has been successfully uploaded.

Key Points

  • Personal Access Tokens: Ensure that both your OpenAI and GitHub API keys are kept private.
  • Running Cells in Order: Complete each cell sequentially to ensure dependencies and environment settings are correctly configured.

This Colab notebook enables you to perform end-to-end automation of Selenium test generation and GitHub updates, helping you streamline your test workflow with minimal manual intervention.