Lesson: Automating Selenium Script Generation in Python and GitHub Push Using GPT API and Google Colab
Introduction
In this lesson, we will automate the process of generating a Selenium test script in Python based on a webpage’s DOM structure using Playwright, OpenAI’s GPT API, and Google Colab. We’ll also automate pushing the generated Python script to a GitHub repository using a GitHub API key.
Learning Objectives
By the end of this lesson, you will be able to:
- Extract the DOM structure of a webpage using Playwright.
- Generate a Python Selenium test script based on this DOM structure using OpenAI’s GPT API.
- Push the generated Python script to a GitHub repository automatically from Google Colab.
Step 1: Setting Up Google Colab for Playwright and GitHub Operations
Start by opening a Google Colab notebook, which allows you to execute Python code in a streamlined cloud environment.
Required Libraries
Ensure the following libraries are available in your Colab environment:
playwright
for DOM extraction.openai
for AI-driven script generation.nest_asyncio
to handle asynchronous calls.
Install these packages with:
!pip install playwright openai nest_asyncio
!playwright install
Step 2: Extracting DOM Structure Using Playwright
The following Python script uses Playwright to fetch the DOM structure of a webpage. This structure will then serve as input for the GPT API to generate a Python Selenium script.
import os
import nest_asyncio
import asyncio
from playwright.async_api import async_playwright
import openai
# Allow nested asyncio calls in Colab
nest_asyncio.apply()
# Detect if running in Google Colab and set API key accordingly
try:
from google.colab import userdata
IN_COLAB = True
except ImportError:
IN_COLAB = False
if IN_COLAB:
os.environ["OPENAI_API_KEY"] = userdata.get('OPENAI_API_KEY')
else:
os.environ["OPENAI_API_KEY"] = os.getenv('OPENAI_API_KEY')
# Initialize the OpenAI client
openai.api_key = os.getenv("OPENAI_API_KEY")
# Async function to retrieve the entire body structure dynamically using Playwright
async def fetch_body_structure(url):
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto(url)
body_structure = await page.evaluate("document.body.outerHTML")
await browser.close()
return body_structure
# Function to generate Selenium test script using OpenAI's ChatGPT
def generate_selenium_script(body_html):
prompt = f"""
Given the following HTML body structure:
```html
{body_html}
```
Generate Python Selenium code (only the code, no comments or explanations) that:
1. Navigates to the webpage: https://practiceautomatedtesting.com/webelements/Checkboxes.
2. Selects each checkbox by clicking on the associated `label` or `span` element. Avoid using the `input` element's ID or directly interacting with the checkbox `input`.
3. Verifies whether selecting each checkbox displays a "😊" or "😠" using spans.
4. Follows best practices, including waiting for elements to load, validating interactions, and checking expected outcomes.
5. Uses the pytest framework to structure tests.
Return only the Python code enclosed within ```python and ```.
"""
response = openai.Completion.create(
model="gpt-4",
prompt=prompt,
max_tokens=500
)
script_code = response.choices[0].text.strip().split("```python")[1].split("```")[0].strip()
print("Generated script:\n", script_code)
return script_code
# Step 1: Fetch body structure from the live webpage
url = "https://practiceautomatedtesting.com/webelements/Checkboxes"
body_html = asyncio.run(fetch_body_structure(url))
# Step 2: Generate the Selenium script using the body HTML structure
script_code = generate_selenium_script(body_html)
# Step 3: Save the Selenium script to a file
script_file_path = "/content/python_test_script.py"
with open(script_file_path, "w") as script_file:
script_file.write(script_code)
print(f"Selenium script saved to {script_file_path}")
Step 3: Pushing the Generated Script to GitHub
Once we have the generated Python Selenium script, we’ll save it to a file and push it to GitHub.
Creating a GitHub API Key
- Go to GitHub > Settings > Developer settings > Personal access tokens.
- Click Generate new token and select the required permissions (e.g.,
repo
). - Copy the generated token and save it securely for use in your Colab environment.
Automating GitHub Operations in Colab
The following code automates GitHub operations in Colab, including cloning the repository, committing the script, and pushing changes.
import os
import shutil
from google.colab import userdata
# GitHub credentials and repository information
GITHUB_TOKEN = userdata.get('GITHUB_TOKEN') # Ensure your GitHub token is set in Colab
GITHUB_USERNAME = "your_github_username"
REPO_NAME = "your_repo_name"
BRANCH_NAME = "main"
# Define the repository path and URL
repo_path = f"/content/{REPO_NAME}"
repo_url = f"https://{GITHUB_USERNAME}:{GITHUB_TOKEN}@github.com/{GITHUB_USERNAME}/{REPO_NAME}.git"
# Configure Git
!git config --global user.email "your-email@example.com"
!git config --global user.name "Your Name"
# Step 1: Clone or Pull the Repository
if os.path.exists(repo_path):
print(f"Repository already exists at {repo_path}. Attempting to pull latest changes.")
pull_result = !git -C {repo_path} pull origin {BRANCH_NAME}
if "fatal" in pull_result[0].lower():
print("Repository pull failed. Re-cloning the repository.")
shutil.rmtree(repo_path)
!git clone -b {BRANCH_NAME} {repo_url} {repo_path}
else:
print(f"Cloning repository into {repo_path}.")
clone_result = !git clone -b {BRANCH_NAME} {repo_url} {repo_path}
if "fatal" in clone_result[0].lower():
print("Repository clone failed. Check credentials or repository URL.")
raise SystemExit("Failed to clone the repository.")
# Verify that .git exists in the repo_path
if not os.path.exists(os.path.join(repo_path, ".git")):
print("Repository initialization failed. Retrying with git init.")
!git -C {repo_path} init
!git -C {repo_path} remote add origin {repo_url}
!git -C {repo_path} fetch
!git -C {repo_path} checkout {BRANCH_NAME}
# Step 2: Copy the Selenium Script into the Repository
script_file_path = "/content/python_test_script.py" # Path to the generated Selenium script
destination_path = f"{repo_path}/tests/selenium_script.py"
# Ensure the destination directory exists
os.makedirs(os.path.dirname(destination_path), exist_ok=True)
# Copy the generated Selenium script into the cloned repository
shutil.copyfile(script_file_path, destination_path)
print(f"Copied {script_file_path} to {destination_path}")
# Step 3: Commit and Push the Changes to GitHub
!git -C {repo_path} add .
!git -C {repo_path} commit -m "Add or update generated Selenium test script"
!git -C {repo_path} push origin {BRANCH_NAME}
print("Selenium script pushed to GitHub successfully.")
Summary
By following this lesson, you now know how to automate DOM extraction using Playwright, generate a Python Selenium script with GPT-4, and push it to GitHub, all within Google Colab. This setup helps streamline your test automation workflow and improve efficiency.
Try It Yourself: Automating Selenium Script Generation in Google Colab
Instructions
1. Open the Colab Notebook
- Go to this Google Colab notebook link to access the pre-configured Colab notebook.
2. Set Up Environment and Install Libraries
- Run the first cell to install any required libraries (
playwright
,openai
, andnest_asyncio
) that are necessary for extracting DOM elements and automating code generation.
3. Enter API Keys
- Ensure you have your OpenAI API Key and GitHub API Key ready. These keys are necessary for connecting to OpenAI for script generation and pushing your results to GitHub. Add your OpenAI key in the designated cell. or in the notebook access key
4. Fetch the DOM Structure
- Run the provided code cell to fetch the DOM structure from a specified webpage using Playwright. The notebook contains code to access elements dynamically.
5. Generate the Selenium Script
- Use OpenAI’s GPT-4 model in the following cell to generate a Python Selenium test script based on the extracted DOM structure. Follow the prompts to customize the output.
6. Save and Push to GitHub
- Input your GitHub repository details, such as username and repository name. Run the final cell to push the generated Selenium script to your GitHub repository.
7. Check Results
- After running all cells, check your GitHub repository to confirm that the generated script has been successfully uploaded.
Key Points
- Personal Access Tokens: Ensure that both your OpenAI and GitHub API keys are kept private.
- Running Cells in Order: Complete each cell sequentially to ensure dependencies and environment settings are correctly configured.
This Colab notebook enables you to perform end-to-end automation of Selenium test generation and GitHub updates, helping you streamline your test workflow with minimal manual intervention.