Data Engineering is the discipline of designing, building, and maintaining systems for collecting, storing, and processing data at scale. It enables businesses to leverage data for analytics, machine learning, and decision-making.
Week-1. Introduction to Data Engineering
Role of Data Engineers, ETL vs ELT, Data Pipelines
Week-2. Relational Databases & SQL
MySQL, PostgreSQL, Query Optimization
Week-3. NoSQL Databases
MongoDB, Cassandra, Redis, Use Cases
Week-4. Data Warehousing & Data Lakes
Snowflake, BigQuery, Amazon Redshift, Delta Lake
Week-5. Introduction to Big Data
Hadoop Ecosystem, HDFS, MapReduce
Week-6. Apache Spark for Data Engineering
RDDs, DataFrames, Spark SQL
Week-7. Data Streaming with Apache Kafka
Kafka Topics, Producers & Consumers, Streaming Use Cases
Week-8. Batch Processing vs Streaming
Airflow, Dagster, Prefect for Workflow Orchestration
Week-9. Building Data Pipelines
ETL Tools, Data Ingestion Strategies
Week-10. Data Engineering on AWS
S3, Glue, Redshift, Lambda
Week-11. Data Engineering on GCP & Azure
BigQuery, Dataflow, Azure Synapse
Week-12. Serverless Data Pipelines
FaaS, Dataflow, Event-Driven Pipelines
Week-13. Data Governance & Compliance
GDPR, HIPAA, Data Quality & Lineage
Week-14. Data Security & Privacy
Encryption, Access Control, Role-Based Permissions
Week-15. Capstone Project – Part 1
Designing a Scalable Data Pipeline
Week-16. Capstone Project – Part 2
Implementing, Testing, and Deploying the Pipeline