Showing 20 of 636 URL(s)
(Page 15 of 32)
How to Scale Your Data Pipelines
How to Scale Your Data Pipelines
- 1. Introduction
- 2. What is scaling & why do we need it?
- 3. Types of scaling
- 4. Choose your scaling strategy
- 5. Conclusion
- 6. Further reading
- 7. References
1. Introduction
Choosing tools/frameworks to scale your data pipelines can be confusing. If you ha...
💡 Top Recommendations:
Understand & Deliver on Your Data Engineering Task
Understand & Deliver on Your Data Engineering Task
- 1. Introduction
- 2. Understanding your data engineering task
- 3. Delivering your data engineering task
- 4. Conclusion
- 5. Further reading
1. Introduction
Congratulations! You are given a quick overview of the business and data architecture and...
💡 Top Recommendations:
What is a staging area?
What is a staging area?
- 1. Introduction
- 2. What is a staging area
- 3. The advantages of having a staging area
- 5. Conclusion
- 6. Further reading
1. Introduction
Working with data pipelines, you might have noticed a staging
area in most data pipelines. If you work in the data space and have qu...
💡 Top Recommendations:
What is a Data Warehouse?
What is a Data Warehouse?
- 1. Introduction
- 2. Business requirements: dashboards and analytics
- 3. What is a data warehouse
- 4. OLTP vs OLAP based data warehouses
- 5. Conclusion
- 6. Further reading
- 7. References
1. Introduction
If you are a student, analyst, engineer, or anyone in the data s...
💡 Top Recommendations:
10 Skills to Ace Your Data Engineering Interviews
10 Skills to Ace Your Data Engineering Interviews
Introduction
Are you a student, analyst, engineer, or someone preparing for a data engineering interview and overwhelmed by all the tools and concepts? Then this post is for you. In this post, we go over the most common tools and concepts you need to...
💡 Top Recommendations:
Whats the difference between ETL & ELT?
Whats the difference between ETL & ELT?
- 1. Introduction
- 2. E-T-L definition
- 3. Differences between ETL & ELT
- 4. Conclusion
- 5. Further reading
1. Introduction
If you are a student, analyst, engineer, or anyone working with data pipelines, you would have heard of ETL and ELT architecture. If...
💡 Top Recommendations:
How to add tests to your data pipelines
How to add tests to your data pipelines
Introduction
Testing data pipelines are different from testing other applications, like a website backend. If you
Have inherited a data pipeline that has no tests
Have to start adding new features to a data pipeline that doesn’t have any tests
Then this post i...
💡 Top Recommendations:
6 Key Concepts, to Master Window Functions
6 Key Concepts, to Master Window Functions
- Introduction
- Prerequisites
- 6 Key Concepts
- Efficiency Considerations
- Conclusion
- Further reading
- References
Introduction
If work with data, window functions can significantly level up your SQL skills. If you have ever thought
window functions ar...
💡 Top Recommendations:
What are Common Table Expressions(CTEs) and when to use them?
What are Common Table Expressions(CTEs) and when to use them?
- Introduction
- Setup
- Common Table Expressions (CTEs)
- Performance comparison
- Tear down
- Conclusion
- References
Introduction
If you are a student, analyst, engineer, or anyone in the data space and are
Wondering what CTEs are?
Try...
💡 Top Recommendations:
How to improve at SQL as a data engineer
How to improve at SQL as a data engineer
- 1. Introduction
- 2. SQL skills
- 3. Practice
- 4. Conclusion
- 5. Further reading
- 6. References
1. Introduction
SQL is the bread and butter of data engineering. Mastering SQL and understanding what can be done with it can make you a better data engineer....
💡 Top Recommendations:
6 Responsibilities of a Data Engineer
6 Responsibilities of a Data Engineer
Introduction
Data engineering is a relatively new field, and as such, there is a huge variance in the actual job responsibilities across different companies. If you are a student, analyst, engineer, or new to the data space and
Unclear with data engineers’ job r...
💡 Top Recommendations:
How to choose the right tools for your data pipeline
How to choose the right tools for your data pipeline
1. Introduction
If you are building data pipelines from the ground up, the number of available data engineering tools to choose from can be overwhelming. If you are thinking
Most of the tools seem to be doing the same/similar thing, which one shou...
💡 Top Recommendations:
Setting up end-to-end tests for cloud data pipelines
Setting up end-to-end tests for cloud data pipelines
- 1. Introduction
- 2. Setting up services locally
- 3. Writing an end-to-end data pipeline test
- 4. Conclusion
- 5. Further reading
- 6. References
1. Introduction
Data pipelines can have multiple software components. This makes testing all of t...
💡 Top Recommendations:
Automating data testing with CI pipelines, using Github Actions
Automating data testing with CI pipelines, using Github Actions
- 1. Introduction
- 2. CI
- 3. Sample project: Data testing with Github Actions
- 4. Conclusion
- 5. Further reading
1. Introduction
Automated testing is crucial for ensuring that your code is bug-free and avoiding regressions. If you a...
💡 Top Recommendations:
What is the difference between a data lake and a data warehouse?
What is the difference between a data lake and a data warehouse?
- Introduction
- Data lakes and data warehouses
- Criteria to choose lake and warehouse tools
- Conclusion
- Further reading
- References
Introduction
With the data ecosystem growing fast, new terms are coming up every week. Some of th...
💡 Top Recommendations:
End-to-end data engineering project - batch edition
End-to-end data engineering project - batch edition
- Objective
- Setup
- Components
- Choosing tools & frameworks
- Future work & improvements
- Conclusion
- Further reading
- References
Objective
It can be difficult to know where to begin when starting a data engineering side project. If you have ...
💡 Top Recommendations:
5 Steps to land a high paying data engineering job
5 Steps to land a high paying data engineering job
1. Introduction
The data industry is booming! & data engineering salaries are skyrocketing. But landing a new job is not an easy task. If you are
Thinking about getting a data engineering job
A data analyst but are doing data engineering work
A data...
💡 Top Recommendations:
Setting up a local development environment for python data projects using Docker
Setting up a local development environment for python data projects using Docker
- 1. Introduction
- 2. Set up
- 3. Reproducibility
- 4. Developer ergonomics
- 5. Conclusion
- 6. Further reading
- 7. References
1. Introduction
Data systems usually involve multiple systems, which makes local developm...
💡 Top Recommendations:
Data Pipeline Design Patterns - #1. Data flow patterns
Data Pipeline Design Patterns - #1. Data flow patterns
- 1. Introduction
- 2. Source & Sink
- 3. Data pipeline patterns
- 4. Conclusion
- 5. Further reading
- 6. References
1. Introduction
Data pipelines can become flakey over time if the data pipeline design foundations are not solid. If you are
Wo...
💡 Top Recommendations:
How to gather requirements for your data project
How to gather requirements for your data project
1. Introduction
Data engineers are often caught off guard by undefined end-user assumptions. As a data engineer, if you feel
Requirements gathering is terrible!
Scope creep kills your ability to deliver on time
Disappointed that you do not get specifi...
💡 Top Recommendations: