Data Engineer – Web Scraping & ETL

At TechBiz Global, we are providing recruitment service to our TOP clients from our portfolio. We are currently seeking a Data Engineer – Web Scraping & ETL to join one of our clients' teams. If you're looking for an exciting opportunity to grow in a innovative environment, this could be the perfect fit for you.

Location: Remote
Reports To: Senior ETL Specialist
Employment Type: Full-Time Employee
Working Hours: European timezone with some overlap to USA

Role Overview
We are looking for an Data Engineer – Web Scraping & ETL to join our data infrastructure team. This role operates primarily within a European timezone, with some overlap with US Central Time to support collaboration with our Houston-based team. You will help manage and monitor our existing fleet of web scrapers, build new data collection pipelines for regulatory and energy market sources, and contribute to the broader evolution of our ETL architecture.

Key Responsibilities
Overnight Operations & Data Quality
• Monitor and manage overnight scraper and ingestion runs, triaging failures and applying fixes in real time to minimize data gaps before US market open
• Verify data completeness and quality across all automated feeds, flagging anomalies and coordinating with the Houston team on persistent issues
• Maintain run logs, error documentation, and escalation notes for seamless async handoffs
New & Expanded Data Collection
Build and maintain scrapers, parsers, and ingestion pipelines across a growing set of energy market data domains, including but not limited to:
• Pipeline operator portals, electronic bulletin boards, and related filings (notices, maintenance, capacity, gas quality, customer indices)
• Government and regulatory agency databases at the federal, state, and provincial level across North America
• International energy data sources covering European, Canadian, and Mexican supply, demand, and power markets
• Emissions and environmental reporting systems
• Financial and corporate filings, including public company disclosures and production reporting
• Geospatial and mapping data related to production, infrastructure, and market geography
ETL Architecture & Enhancement
• Contribute to the design and build-out of our broader ETL infrastructure, including scheduling, orchestration, and error handling
• Write transformation logic to clean, normalize, and load raw data into PostgreSQL staging and production tables
• Optimize existing pipelines for performance, reliability, and cost efficiency
• Help build monitoring dashboards and alerting for pipeline health and data freshness
• Document data lineage, schema changes, and pipeline dependencies
Technical Environment
• Languages: Python (primary), SQL, Bash scripting
• Database: PostgreSQL, Mongo and Snowflake
• Infrastructure: AWS (EC2, S3, Lambda), Docker
• Scraping: Selenium, Playwright, BeautifulSoup, Scrapy, or similar frameworks
• Orchestration: cron, Airflow, or equivalent scheduling tools
• Version Control: Git / GitHub
• Communication: Slack, with async handoffs to US-based team

Required Qualifications
• 3+ years of experience building and maintaining ETL pipelines or data engineering systems
• Strong Python skills with experience in web scraping, data parsing, and automation
• Proficiency in SQL and experience working with relational databases (PostgreSQL preferred)
• Experience with headless browsers, anti-bot mitigation, and scraping resilience patterns
• Strong debugging instincts and ability to triage pipeline failures quickly
• Clear written communication in English for async collaboration and documentation
Preferred Qualifications
• Experience with energy, commodities, or financial data pipelines
• Familiarity with FERC-regulated pipeline data, EIA reporting, or utility/regulatory filings
• Experience with PostgreSQL-specific features (partitioning, materialized views, JSONB, pg_cron, logical replication)
• Familiarity with infrastructure-as-code (Terraform, CloudFormation) or containerized deployments
• Prior experience working on a distributed team across time zones
What We Offer
• Competitive compensation and benefits with room to grow as the team scales
• Direct impact on the data infrastructure behind a leading natural gas intelligence platform
• A small, senior team where your work is visible and valued from day one
• Flexible remote work with clear async workflows
• Exposure to the North American energy markets and commodity data at scale