04 / Big Data Analytics
Log Analytics at 10TB/day
PySpark log analysis for major Vietnamese news sites, processing 10TB of daily traffic with Airflow-orchestrated workflows.
- Client
- VCCorp
- Role
- Data Analyst
- Year
- 2021 - 2022
- Category
- Big Data Analytics
[ The project ]
Used PySpark to analyse user notification logs across Kenh14, Soha, Pega and other VCCorp news properties.
Built repeatable Airflow DAGs that gave analysts on-demand, scheduled access to reports without engineering involvement.
Tuned Spark jobs to gracefully handle bursty 10TB/day workloads on shared cluster resources.