How to turn a 1000-line messy SQL into a modular, & easy-to-maintain data pipeline?
1. Introduction
If you’ve been in the data space long enough, you would have come across really long SQL scripts tha...
Similar Articles (10 found)
🔍 66.1% similar
You’ve done it!
You’ve built your first data pipeline. Maybe you’re a junior data engineer. Maybe you’re a data analyst shipping your first pipeline i...
🔍 62.7% similar
SQL Anti-Patterns You Should Avoid
Introduction
Today, I will be talking about some of the common and high impact SQL anti-patterns I have seen from e...
🔍 62.5% similar
Should Data Pipelines in Python be Function based or Object-Oriented (OOP)?
- 1. Introduction
- 2. Data transformations as functions lead to maintaina...
🔍 62.0% similar
How to add tests to your data pipelines
Introduction
Testing data pipelines are different from testing other applications, like a website backend. If ...
🔍 61.1% similar
How to unit test sql transforms in dbt
Introduction
With the recent advancements in data warehouses and tools like dbt
most transformations(T of ELT) ...
🔍 60.7% similar
Data Engineering Interview Preparation Series #3: SQL
- 1. Introduction
- 2. Step-by-step process to solve any SQL interview question
- 3. Lead the co...
🔍 60.5% similar
Data Engineering Best Practices - #1. Data flow & Code
- 1. Introduction
- 2. Sample project
- 3. Best practices
- 3.1. Use standard patterns that pro...
🔍 60.4% similar
End-to-end data engineering project - batch edition
- Objective
- Setup
- Components
- Choosing tools & frameworks
- Future work & improvements
- Conc...
🔍 59.8% similar
How to test PySpark code with pytest
- 1. Introduction
- 2. Ensure the code’s logic is working as expected with tests
- 3. Conclusion
- 4. Further Rea...
🔍 59.6% similar
Data Engineering Projects
1. Introduction
Whether you are new to data engineering or have been in the data field for a few years, one of the most chal...