ETL Pipeline: Dow Jones Sentiment
A complete PySpark ETL pipeline for sentiment analysis of Dow Jones news, with visualizations and multi-format storage.
What it does: Schema-validated extraction, PySpark feature engineering and loading to Parquet, CSV, SQLite and JSON, with pytest tests.
Overview
An ETL pipeline that processes 4,101 Dow Jones news records (2008–2016) using PySpark with Adaptive Query Execution. The extraction stage validates the CSV schema and runs automatic quality checks. The transformation applies feature engineering: null cleanup, date conversion, sentiment indicators, text statistics, headline aggregation and temporal features.
The multi-format load writes to Parquet (Snappy compression), CSV, SQLite with indexes, and JSON. Visualizations include class distribution, sentiment over time, word clouds, top unigrams/bigrams and an interactive Plotly dashboard. The full pipeline runs in under 60 seconds with less than 4 GB of RAM, includes a pytest unit and integration test suite, and is dockerized for reproducibility.
Stack
Interested in this kind of work?
I'm looking for remote work as a Full Stack Engineer. If you'd like to talk about this project or your team, get in touch.