Elektrine lite

← Feed

@vi21@lemmy.ml

2026-11-02 21:52 UTC

A data pipeline commonly has three phases: extract, transform, and load (ETL). The Unix philosophy of “do one thing well” can apply here by making each program handle only one ETL phase; for example, a program that only extracts data. This strategy works well whether orchestration is done with a shell script or with a Python-based de facto tool such as Apache Airflow. In contrast to mixing ETL within a single program, Airflow and shell scripts using GNU Parallel can handle retries, concurrency, logging, etc. Data pipeline engineering is not a rare case. Still, other kinds of applications, such as GUI, TUI, and web apps, are rarely built on shell scripts; however, shell scripts may work behind the scenes for many apps. Moreover, “do one thing well” still applies, but the unit may be a function or module instead.

Replies (1)

  • @arty@feddit.org 2026-11-03 11:55

    In contrast to mixing ETL within a single program, Airflow and shell scripts using GNU Parallel can handle retries, concurrency, logging, etc Am I reading your message wrong? It appears to say that a single program cannot handle retries, concurrency, logging, etc.

    Open ##5472208