webscraping and ai

Web Scraping and AI

A collection of modern web scraping techniques and implementations that combine traditional scraping with AI capabilities. The repository showcases different approaches from traditional BeautifulSoup to LLM-powered solutions, with concrete examples, performance metrics, and cost analysis.

Content

• CODECON 2024 #ZILINA Slides / Code

Installation Guide

1. Create environment:

conda create -n webscrape python=3.12

2. Activate environment:

conda activate webscrape

3. Install uv:

pip install uv

4. Install dependencies:

uv pip install -r requirements.txt

5. Install Playwright:

playwright install

Environment Setup

Create a .env file with:

LITELLM_API_KEY=
LITELLM_URL=

---