diff --git a/docs/examples/ibis.mdx b/docs/examples/ibis.mdx new file mode 100644 index 000000000..33a0580eb --- /dev/null +++ b/docs/examples/ibis.mdx @@ -0,0 +1,108 @@ +--- +description: query and visualize Overture data with Ibis and DuckDB +title: Ibis +--- + +[Ibis](https://ibis-project.org/) is a Python dataframe library that provides a unified interface to many query backends. With its default [DuckDB](https://duckdb.org/) backend, you can query Overture's GeoParquet files directly from S3 — with filter and projection pushdown so you only download the data you need. + +:::info +This example requires `duckdb>=1.1.1` for GeoParquet support. See the [Ibis blog post](https://ibis-project.org/posts/ibis-overturemaps/index.html) for an extended walkthrough including visualization with Lonboard. +::: + +
+ + + **Installation requirements** + + + +```bash +pip install 'duckdb>=1.1.1' +pip install 'ibis-framework[duckdb,geospatial]' +``` + +
+ +## Query Overture data with Ibis + +### Read and filter data + +Use `ibis.read_parquet()` to point Ibis at an Overture release on S3. Ibis spins up a DuckDB connection automatically. Here we query the `base/infrastructure` type and filter to power infrastructure within a bounding box around Washington, D.C.: + +```python +import ibis +from ibis import _ + +t = ibis.read_parquet( + "s3://overturemaps-us-west-2/release/2024-09-18.0/theme=base/type=infrastructure/*", + table_name="infra", +) + +# Filter and project — DuckDB pushes these down to S3, so only matching data is downloaded +expr = t.filter( + _.bbox.xmin > -77.119795, + _.bbox.xmax < -76.909366, + _.bbox.ymin > 38.791631, + _.bbox.ymax < 38.995968, + _.subtype == "power", +).select(["names", "geometry", "sources", "class"]) +``` + +:::tip +Ibis uses lazy evaluation — `expr` is just an expression tree and no data is fetched until you execute it. DuckDB pushes the filters and column projections down to the parquet reader, minimizing data transfer. +::: + +### Save results locally + +```python +expr.get_backend().to_parquet(expr, "infra-power-dc.geoparquet") +``` + +### Explore interactively + +Load the saved file and turn on interactive mode to preview results inline: + +```python +ibis.options.interactive = True + +power_dc = ibis.read_parquet("infra-power-dc.geoparquet") + +# Rename 'class' — reserved word that causes issues with the deferred operator +power_dc = power_dc.rename(infra_class="class") + +# Count by infrastructure class +power_dc.infra_class.value_counts().order_by(ibis.desc("infra_class_count")) +``` + +Filter to a specific class: + +```python +power_lines = power_dc.filter(_.infra_class == "power_line") +power_lines["names", "geometry", "infra_class"] +``` + +### Visualize with Lonboard + +Convert to a GeoDataFrame to visualize with [Lonboard](https://developmentseed.org/lonboard/latest/): + +```python +import geopandas as gpd +import lonboard +from lonboard.basemap import CartoBasemap + +gdf = gpd.GeoDataFrame(power_lines.to_pandas(), geometry="geometry", crs="EPSG:4326") + +lonboard.viz( + gdf, + map_kwargs={ + "basemap_style": CartoBasemap.Positron, + "view_state": {"longitude": -77.01, "latitude": 38.9, "zoom": 10}, + }, +) +``` + +## Next steps + +- Full walkthrough with maps: [Ibis blog — Exploring GeoParquet Overture Maps with Ibis, DuckDB, and Lonboard](https://ibis-project.org/posts/ibis-overturemaps/index.html) +- Reading parquet files with Ibis: [Ibis how-to guide](https://ibis-project.org/how-to/input-output/duckdb-parquet) +- Lonboard visualization: [Lonboard + Overture example](https://developmentseed.org/lonboard/latest/examples/overture-maps/) diff --git a/docs/examples/index.mdx b/docs/examples/index.mdx index 363b5adba..18dab3109 100644 --- a/docs/examples/index.mdx +++ b/docs/examples/index.mdx @@ -5,22 +5,27 @@ title: Examples sidebar_label: Overview description: Overture Maps data examples pagination_label: Examples +sidebar_position: 0 --- +import { useCurrentSidebarCategory } from '@docusaurus/plugin-content-docs/client'; + +export function ExamplesList() { + const category = useCurrentSidebarCategory(); + return ( + + ); +} # Examples Work with Overture Maps data using the tools and platforms you already know. These examples show how to access, query, visualize, and analyze Overture data across a range of environments. -- **[CARTO](/examples/CARTO)** -- **[Fused](/examples/fused)** -- **[Kepler.gl](/examples/kepler-gl)** -- **[Lonboard](/examples/lonboard)** -- **[Build a Map](/examples/build-a-map)** -- **[Pandas](/examples/pandas)** -- **[Overture Tiles](/examples/overture-tiles)** -- **[QGIS](/examples/QGIS)** -- **[Rapid](/examples/rapid)** -- **[Spark](/examples/spark)** -- **[Wherobots](/examples/wherobots)** + Have an example you'd like to share? We welcome contributions: [open a pull request](https://github.com/OvertureMaps/docs/pulls) or [start a discussion](https://github.com/OvertureMaps/data/discussions) on GitHub. diff --git a/docs/getting-data/data-mirrors/index.mdx b/docs/getting-data/data-mirrors/index.mdx index 9a513a79b..62a2477d6 100644 --- a/docs/getting-data/data-mirrors/index.mdx +++ b/docs/getting-data/data-mirrors/index.mdx @@ -2,6 +2,7 @@ title: Data Mirrors sidebar_label: Overview pagination_label: Data Mirrors Overview +sidebar_position: 0 --- # Data Mirrors diff --git a/docs/guides/index.mdx b/docs/guides/index.mdx index eadbcf949..04b0cdf1c 100644 --- a/docs/guides/index.mdx +++ b/docs/guides/index.mdx @@ -18,4 +18,4 @@ import useBaseUrl from '@docusaurus/useBaseUrl'; - **[Buildings](./buildings)** - **[Divisions](./divisions)** - **[Places](./places)** -- **[Transportation](./transportation)** \ No newline at end of file +- **[Transportation](./transportation)** diff --git a/sidebars.js b/sidebars.js index 02b6cb930..a8d4b9849 100644 --- a/sidebars.js +++ b/sidebars.js @@ -16,12 +16,7 @@ const sidebars = { label: 'Data Mirrors', collapsed: true, items: [ - 'getting-data/data-mirrors/index', - 'getting-data/data-mirrors/bigquery', - 'getting-data/data-mirrors/databricks', - 'getting-data/data-mirrors/fused', - 'getting-data/data-mirrors/snowflake', - 'getting-data/data-mirrors/wherobots', + { type: 'autogenerated', dirName: 'getting-data/data-mirrors' }, ], }, ], @@ -31,7 +26,7 @@ const sidebars = { label: 'Schema Reference', collapsed: true, items: [ - 'schema/index', + 'schema/index', // Ensure index is first { type: 'autogenerated', dirName: 'schema/reference' }, ], }, @@ -40,21 +35,7 @@ const sidebars = { label: 'Examples', collapsed: true, items: [ - 'examples/index', - 'examples/CARTO', - 'examples/fused', - 'examples/kepler-gl', - 'examples/lonboard', - 'examples/build-a-map', - 'examples/pandas', - 'examples/overture-tiles', - 'examples/QGIS', - 'examples/rapid', - 'examples/spark', - 'examples/sedona', - 'examples/wherobots', - 'examples/athena-aws', - 'examples/synapse-azure', + { type: 'autogenerated', dirName: 'examples' }, ], }, { @@ -135,7 +116,7 @@ const sidebars = { 'attribution', 'release-calendar', 'support', - + ], }; -module.exports = sidebars; \ No newline at end of file +module.exports = sidebars;