Flow Scrape
Build browser automation visually. Run it, schedule it, and inspect every step.
Overview & Context
Flow Scrape provides a visual DAG editor for designing multi-step browser scraping pipelines, with scheduled execution, execution-phase telemetry, and AI-assisted extraction.
The Problem
Writing headless browser scripts manually requires managing browser lifecycles, selector retries, cron schedules, credential encryption, and error recovery in isolation.
The Architectural Approach
Constructed an interactive DAG flow builder using @xyflow/react. Workflows compile into topological execution plans executed by Puppeteer. Integrated Gemini 1.5 Flash for unstructured-to-structured HTML extraction, encrypted per-user credential storage, phase-by-phase execution logging, cron scheduling, and credit-based usage tracking with PostgreSQL and Prisma.
Key Technical Highlights
- Interactive DAG flow builder with @xyflow/react connection validation
- Puppeteer execution engine with navigation, clicks, form-filling, and AI extraction
- Cron-scheduled runs, encrypted credentials, execution phase logs, and credit billing
GitLoom
Ask a codebase questions. Follow the answer back to the files.