How to quickly set up a local Spark development environment?
- 1. Introduction
- 2. Setup
- 3. Use VSCode devcontainers to set up Spark environment
- 4. Conclusion
- 5. Read these
1. Introduction
Sett...
Similar Articles (10 found)
🔍 77.1% similar
Docker Fundamentals for Data Engineers
1. Introduction
Docker can be overwhelming to start with. Most data projects use Docker to set up the data infr...
🔍 76.1% similar
Setting up a local development environment for python data projects using Docker
- 1. Introduction
- 2. Set up
- 3. Reproducibility
- 4. Developer erg...
🔍 70.0% similar
Build Data Engineering Projects, with Free Template
- 1. Introduction
- 2. Run Data Pipeline
- 3. Architecture and services in this template
- 4. CI/C...
🔍 68.2% similar
How to test PySpark code with pytest
- 1. Introduction
- 2. Ensure the code’s logic is working as expected with tests
- 3. Conclusion
- 4. Further Rea...
🔍 66.5% similar
Visual Studio Code (VSCode) extensions for data engineers
- 1. Introduction
- 2. Python environment setup
- 3. VSCode Primer
- 4. Extensions overview
...
🔍 66.1% similar
Data Engineering Projects
1. Introduction
Whether you are new to data engineering or have been in the data field for a few years, one of the most chal...
🔍 62.2% similar
Data Engineering Project for Beginners - Batch edition
- 1. Introduction
- 2. Objective
- 3. Run Data Pipeline
- 4. Architecture
- 5. Code walkthrough...
🔍 60.8% similar
Why use Apache Airflow (or any orchestrator)?
- 1. Introduction
- 2. Features crucial to building and maintaining data pipelines
- 3. Conclusion
- 4. ...
🔍 60.4% similar
How to set up a dbt data-ops workflow, using dbt cloud and Snowflake
- Introduction
- Pre-requisites
- Setting up the data-ops pipeline
- Conclusion a...
🔍 60.1% similar
End-to-end data engineering project - batch edition
- Objective
- Setup
- Components
- Choosing tools & frameworks
- Future work & improvements
- Conc...