Datasets

[martindurant]

Most of the project tools projspec currently handles are concerned with (code) dependencies, build and run commands. But many things done at the project scope are data-dependent, so it makes sense to declare dataset use too. We currently mention two ways to do this (intake and frictionlessdata), but these are far from common and probably there are many ways to do this.

Question: do many people declare their data use at all, or just include a URL/query/etc in the code?

Examples:

• https://docs.getdbt.com/reference/dbt_project.yml
• https://docs.prefect.io/v3/how-to-guides/deployments/prefect-yaml
• pydantic, sqlalchemy, django ORMs
• raw SQL?

[martindurant]

Some work is done here:

• DataPackage
• DVC
• Intake