How to Scale Your Data Pipelines
- 1. Introduction
- 2. What is scaling & why do we need it?
- 3. Types of scaling
- 4. Choose your scaling strategy
- 5. Conclusion
- 6. Further reading
- 7. Reference...
Similar Articles (10 found)
🔍 73.0% similar
How to choose the right tools for your data pipeline
1. Introduction
If you are building data pipelines from the ground up, the number of available da...
🔍 68.5% similar
Data Engineering Projects
1. Introduction
Whether you are new to data engineering or have been in the data field for a few years, one of the most chal...
🔍 68.2% similar
You’ve done it!
You’ve built your first data pipeline. Maybe you’re a junior data engineer. Maybe you’re a data analyst shipping your first pipeline i...
🔍 68.1% similar
End-to-end data engineering project - batch edition
- Objective
- Setup
- Components
- Choosing tools & frameworks
- Future work & improvements
- Conc...
🔍 67.3% similar
What are the Key Parts of Data Engineering?
1. Introduction
If you are trying to break into (or land a new) data engineering job, you will inevitably ...
🔍 67.2% similar
Why use Apache Airflow (or any orchestrator)?
- 1. Introduction
- 2. Features crucial to building and maintaining data pipelines
- 3. Conclusion
- 4. ...
🔍 67.1% similar
Hi, fellow future and current Data Leaders; Ben here 👋
I recently was skimming the data engineering subreddit and came across a post with a title that...
🔍 67.0% similar
How to add tests to your data pipelines
Introduction
Testing data pipelines are different from testing other applications, like a website backend. If ...
🔍 66.7% similar
How to gather requirements to re-engineer a legacy data pipeline
Introduction
As data engineers, you will have to re-engineer legacy data pipelines. W...
🔍 65.8% similar
Data Engineering Best Practices - #1. Data flow & Code
- 1. Introduction
- 2. Sample project
- 3. Best practices
- 3.1. Use standard patterns that pro...