2026-05-04 22:35 UTC
@jonocarroll
not @tylermorganwall but the scale is more about team members / organization structure and tools you need to connect than volume of data.
dbt with duckdb will be different than dbt with spark. I ran half TB data with targets and duckdb without (big) issue (but it did not need to run every few minutes).
Replies (1)
-
@noamross@ecoevo.social 2026-05-04 22:43
@defuneste @jonocarroll @tylermorganwall I think an interesting thing to think about is pipelines where intermediate steps may or may not be "virtual"? Is a dbt step a materialized table or a view? Are multiple steps able to be combined into a single, efficient query via the SQL engine? Targets and R have some challenges taking advantage of some of these situations because of the core object-and-serialization model.