Showing 20 of 732 URL(s)
(Page 20 of 37)
What is a Data Warehouse?
What is a Data Warehouse?
- 1. Introduction
- 2. Business requirements: dashboards and analytics
- 3. What is a data warehouse
- 4. OLTP vs OLAP based data warehouses
- 5. Conclusion
- 6. Further reading
- 7. References
1. Introduction
If you are a student, analyst, engineer, or anyone in the data s...
💡 Top Recommendations:
10 Skills to Ace Your Data Engineering Interviews
10 Skills to Ace Your Data Engineering Interviews
Introduction
Are you a student, analyst, engineer, or someone preparing for a data engineering interview and overwhelmed by all the tools and concepts? Then this post is for you. In this post, we go over the most common tools and concepts you need to...
💡 Top Recommendations:
Whats the difference between ETL & ELT?
Whats the difference between ETL & ELT?
- 1. Introduction
- 2. E-T-L definition
- 3. Differences between ETL & ELT
- 4. Conclusion
- 5. Further reading
1. Introduction
If you are a student, analyst, engineer, or anyone working with data pipelines, you would have heard of ETL and ELT architecture. If...
💡 Top Recommendations:
How to add tests to your data pipelines
How to add tests to your data pipelines
Introduction
Testing data pipelines are different from testing other applications, like a website backend. If you
Have inherited a data pipeline that has no tests
Have to start adding new features to a data pipeline that doesn’t have any tests
Then this post i...
💡 Top Recommendations:
6 Key Concepts, to Master Window Functions
6 Key Concepts, to Master Window Functions
- Introduction
- Prerequisites
- 6 Key Concepts
- Efficiency Considerations
- Conclusion
- Further reading
- References
Introduction
If work with data, window functions can significantly level up your SQL skills. If you have ever thought
window functions ar...
💡 Top Recommendations:
What are Common Table Expressions(CTEs) and when to use them?
What are Common Table Expressions(CTEs) and when to use them?
- Introduction
- Setup
- Common Table Expressions (CTEs)
- Performance comparison
- Tear down
- Conclusion
- References
Introduction
If you are a student, analyst, engineer, or anyone in the data space and are
Wondering what CTEs are?
Try...
💡 Top Recommendations:
How to improve at SQL as a data engineer
How to improve at SQL as a data engineer
- 1. Introduction
- 2. SQL skills
- 3. Practice
- 4. Conclusion
- 5. Further reading
- 6. References
1. Introduction
SQL is the bread and butter of data engineering. Mastering SQL and understanding what can be done with it can make you a better data engineer....
💡 Top Recommendations:
6 Responsibilities of a Data Engineer
6 Responsibilities of a Data Engineer
Introduction
Data engineering is a relatively new field, and as such, there is a huge variance in the actual job responsibilities across different companies. If you are a student, analyst, engineer, or new to the data space and
Unclear with data engineers’ job r...
💡 Top Recommendations:
How to choose the right tools for your data pipeline
How to choose the right tools for your data pipeline
1. Introduction
If you are building data pipelines from the ground up, the number of available data engineering tools to choose from can be overwhelming. If you are thinking
Most of the tools seem to be doing the same/similar thing, which one shou...
💡 Top Recommendations:
Setting up end-to-end tests for cloud data pipelines
Setting up end-to-end tests for cloud data pipelines
- 1. Introduction
- 2. Setting up services locally
- 3. Writing an end-to-end data pipeline test
- 4. Conclusion
- 5. Further reading
- 6. References
1. Introduction
Data pipelines can have multiple software components. This makes testing all of t...
💡 Top Recommendations:
Automating data testing with CI pipelines, using Github Actions
Automating data testing with CI pipelines, using Github Actions
- 1. Introduction
- 2. CI
- 3. Sample project: Data testing with Github Actions
- 4. Conclusion
- 5. Further reading
1. Introduction
Automated testing is crucial for ensuring that your code is bug-free and avoiding regressions. If you a...
💡 Top Recommendations:
What is the difference between a data lake and a data warehouse?
What is the difference between a data lake and a data warehouse?
- Introduction
- Data lakes and data warehouses
- Criteria to choose lake and warehouse tools
- Conclusion
- Further reading
- References
Introduction
With the data ecosystem growing fast, new terms are coming up every week. Some of th...
💡 Top Recommendations:
End-to-end data engineering project - batch edition
End-to-end data engineering project - batch edition
- Objective
- Setup
- Components
- Choosing tools & frameworks
- Future work & improvements
- Conclusion
- Further reading
- References
Objective
It can be difficult to know where to begin when starting a data engineering side project. If you have ...
💡 Top Recommendations:
5 Steps to land a high paying data engineering job
5 Steps to land a high paying data engineering job
1. Introduction
The data industry is booming! & data engineering salaries are skyrocketing. But landing a new job is not an easy task. If you are
Thinking about getting a data engineering job
A data analyst but are doing data engineering work
A data...
💡 Top Recommendations:
Setting up a local development environment for python data projects using Docker
Setting up a local development environment for python data projects using Docker
- 1. Introduction
- 2. Set up
- 3. Reproducibility
- 4. Developer ergonomics
- 5. Conclusion
- 6. Further reading
- 7. References
1. Introduction
Data systems usually involve multiple systems, which makes local developm...
💡 Top Recommendations:
Data Pipeline Design Patterns - #1. Data flow patterns
Data Pipeline Design Patterns - #1. Data flow patterns
- 1. Introduction
- 2. Source & Sink
- 3. Data pipeline patterns
- 4. Conclusion
- 5. Further reading
- 6. References
1. Introduction
Data pipelines can become flakey over time if the data pipeline design foundations are not solid. If you are
Wo...
💡 Top Recommendations:
How to gather requirements for your data project
How to gather requirements for your data project
1. Introduction
Data engineers are often caught off guard by undefined end-user assumptions. As a data engineer, if you feel
Requirements gathering is terrible!
Scope creep kills your ability to deliver on time
Disappointed that you do not get specifi...
💡 Top Recommendations:
Data Pipeline Design Patterns - #2. Coding patterns in Python
Data Pipeline Design Patterns - #2. Coding patterns in Python
- Introduction
- Sample project
- Code design patterns
- Python helpers
- Misc
- Conclusion
- Further reading
- References
Introduction
Using the appropriate code design pattern can make your code easy to read, extensible, and seamless to...
💡 Top Recommendations:
Change Data Capture, with Debezium
Change Data Capture, with Debezium
Introduction
Change data capture is a pattern where every change to a row in a table is captured and sent to downstream systems. If you have wondered
How to ingest data from multiple databases into your data warehouse?
How to make data available for analytical quer...
💡 Top Recommendations:
How to become a valuable data engineer
How to become a valuable data engineer
1. Introduction
So you are a new data engineer (or looking for a DE job) and want to better yourself as a data engineer. However, when you look at job postings or company tech stack, you are overwhelmed by the sheer amount of tools you have to learn! You feel o...
💡 Top Recommendations: