DSAI 103 – Product Search Engine

This is my first data science project, built for the DSAI 103 course. It demonstrates how to create a simple product search engine by scraping e‑commerce sites and comparing different methods of data collection.

1. Introduction

The goal is to pull product titles, prices, and review counts from eBay using three approaches—static HTML parsing, an API, and browser automation—then clean, combine, and visualize the results.

2. BeautifulSoup + SerpAPI

• Use requests + BeautifulSoup to parse eBay’s static search results.
• Call SerpAPI (a structured Google Search API) to fetch clean JSON listings.
• Collect titles, prices, and review counts into pandas DataFrames.

3. Selenium

• Automate a Chrome browser with Selenium + ChromeDriver to load dynamic, JavaScript-rendered pages.
• Extract product info that appears only after full page load.
• Note: This may not run inside Streamlit due to browser automation security restrictions.

4. Cleaning & Integration

• Merge all scraped data into a single DataFrame.
• Remove duplicates, normalize price formats, and unify structure.

5. Visualization & Comparison

• Generate KDE heatmaps to compare price distributions.
• Use Network Graphs (NetworkX) to explore relationships.
• Create 3D scatter plots for multi-feature insights.

6. Streamlit Deployment

The app includes:

• A search bar for real-time product queries
• Tabs for each scraping method
• Interactive charts and comparisons
• CSV download for combined results

---

💻 How to Run the Project

1. Download the repository as a ZIP file from GitHub and extract it.
2. Open a terminal and navigate to the extracted folder
3. Install the required Python libraries: "pip install -r requirements.txt"
4. streamlit run main.py