2026-05-04 22:17 UTC
Replies (2)
-
@noamross@ecoevo.social 2026-05-04 22:33
@jonocarroll @tylermorganwall Yes, almost all my projects are like this, but some projects reach a scale where it doesn't make sense to treat it all as one graph. Often the data engineering of assembling a database or lake is one project, and then pipelines that generate things from that are another. In some cases we use target's own multiple stores/projects framework within one repo, in some cases they are separate repos.
-
@defuneste@fosstodon.org 2026-05-04 22:35
@jonocarroll not @tylermorganwall but the scale is more about team members / organization structure and tools you need to connect than volume of data. dbt with duckdb will be different than dbt with spark. I ran half TB data with targets and duckdb without (big) issue (but it did not need to run every few minutes).