Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions docs/examples/ibis.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
---
description: query and visualize Overture data with Ibis and DuckDB
title: Ibis
---

[Ibis](https://ibis-project.org/) is a Python dataframe library that provides a unified interface to many query backends. With its default [DuckDB](https://duckdb.org/) backend, you can query Overture's GeoParquet files directly from S3 — with filter and projection pushdown so you only download the data you need.

:::info
This example requires `duckdb>=1.1.1` for GeoParquet support. See the [Ibis blog post](https://ibis-project.org/posts/ibis-overturemaps/index.html) for an extended walkthrough including visualization with Lonboard.
:::
Comment thread
lowlydba marked this conversation as resolved.

<details>

<summary>
**Installation requirements**

</summary>

```bash
pip install 'duckdb>=1.1.1'
pip install 'ibis-framework[duckdb,geospatial]'
```

</details>

## Query Overture data with Ibis

### Read and filter data

Use `ibis.read_parquet()` to point Ibis at an Overture release on S3. Ibis spins up a DuckDB connection automatically. Here we query the `base/infrastructure` type and filter to power infrastructure within a bounding box around Washington, D.C.:

```python
import ibis
from ibis import _

t = ibis.read_parquet(
"s3://overturemaps-us-west-2/release/2024-09-18.0/theme=base/type=infrastructure/*",
table_name="infra",
)

# Filter and project — DuckDB pushes these down to S3, so only matching data is downloaded
expr = t.filter(
_.bbox.xmin > -77.119795,
_.bbox.xmax < -76.909366,
_.bbox.ymin > 38.791631,
_.bbox.ymax < 38.995968,
_.subtype == "power",
).select(["names", "geometry", "sources", "class"])
```

:::tip
Ibis uses lazy evaluation — `expr` is just an expression tree and no data is fetched until you execute it. DuckDB pushes the filters and column projections down to the parquet reader, minimizing data transfer.
:::

### Save results locally

```python
expr.get_backend().to_parquet(expr, "infra-power-dc.geoparquet")
```

### Explore interactively

Load the saved file and turn on interactive mode to preview results inline:

```python
ibis.options.interactive = True

power_dc = ibis.read_parquet("infra-power-dc.geoparquet")

# Rename 'class' — reserved word that causes issues with the deferred operator
power_dc = power_dc.rename(infra_class="class")

# Count by infrastructure class
power_dc.infra_class.value_counts().order_by(ibis.desc("infra_class_count"))
```

Filter to a specific class:

```python
power_lines = power_dc.filter(_.infra_class == "power_line")
power_lines["names", "geometry", "infra_class"]
```

### Visualize with Lonboard

Convert to a GeoDataFrame to visualize with [Lonboard](https://developmentseed.org/lonboard/latest/):

```python
import geopandas as gpd
import lonboard
from lonboard.basemap import CartoBasemap

gdf = gpd.GeoDataFrame(power_lines.to_pandas(), geometry="geometry", crs="EPSG:4326")

lonboard.viz(
gdf,
map_kwargs={
"basemap_style": CartoBasemap.Positron,
"view_state": {"longitude": -77.01, "latitude": 38.9, "zoom": 10},
},
)
```

## Next steps

- Full walkthrough with maps: [Ibis blog — Exploring GeoParquet Overture Maps with Ibis, DuckDB, and Lonboard](https://ibis-project.org/posts/ibis-overturemaps/index.html)
- Reading parquet files with Ibis: [Ibis how-to guide](https://ibis-project.org/how-to/input-output/duckdb-parquet)
- Lonboard visualization: [Lonboard + Overture example](https://developmentseed.org/lonboard/latest/examples/overture-maps/)
27 changes: 16 additions & 11 deletions docs/examples/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,27 @@ title: Examples
sidebar_label: Overview
description: Overture Maps data examples
pagination_label: Examples
sidebar_position: 0
---
import { useCurrentSidebarCategory } from '@docusaurus/plugin-content-docs/client';

export function ExamplesList() {
const category = useCurrentSidebarCategory();
return (
<ul>
{category.items.map((item) => (
<li key={item.href}>
<strong><a href={item.href}>{item.label}</a></strong>
Comment thread
lowlydba marked this conversation as resolved.
</li>
))}
</ul>
);
}

# Examples

Work with Overture Maps data using the tools and platforms you already know. These examples show how to access, query, visualize, and analyze Overture data across a range of environments.

- **[CARTO](/examples/CARTO)**
- **[Fused](/examples/fused)**
- **[Kepler.gl](/examples/kepler-gl)**
- **[Lonboard](/examples/lonboard)**
- **[Build a Map](/examples/build-a-map)**
- **[Pandas](/examples/pandas)**
- **[Overture Tiles](/examples/overture-tiles)**
- **[QGIS](/examples/QGIS)**
- **[Rapid](/examples/rapid)**
- **[Spark](/examples/spark)**
- **[Wherobots](/examples/wherobots)**
<ExamplesList />

Have an example you'd like to share? We welcome contributions: [open a pull request](https://github.com/OvertureMaps/docs/pulls) or [start a discussion](https://github.com/OvertureMaps/data/discussions) on GitHub.
1 change: 1 addition & 0 deletions docs/getting-data/data-mirrors/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
title: Data Mirrors
sidebar_label: Overview
pagination_label: Data Mirrors Overview
sidebar_position: 0
---

# Data Mirrors
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ import useBaseUrl from '@docusaurus/useBaseUrl';
- **[Buildings](./buildings)**
- **[Divisions](./divisions)**
- **[Places](./places)**
- **[Transportation](./transportation)**
- **[Transportation](./transportation)**
29 changes: 5 additions & 24 deletions sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,7 @@ const sidebars = {
label: 'Data Mirrors',
collapsed: true,
items: [
'getting-data/data-mirrors/index',
'getting-data/data-mirrors/bigquery',
'getting-data/data-mirrors/databricks',
'getting-data/data-mirrors/fused',
'getting-data/data-mirrors/snowflake',
'getting-data/data-mirrors/wherobots',
{ type: 'autogenerated', dirName: 'getting-data/data-mirrors' },
],
},
],
Expand All @@ -31,7 +26,7 @@ const sidebars = {
label: 'Schema Reference',
collapsed: true,
items: [
'schema/index',
'schema/index', // Ensure index is first
{ type: 'autogenerated', dirName: 'schema/reference' },
],
},
Expand All @@ -40,21 +35,7 @@ const sidebars = {
label: 'Examples',
collapsed: true,
items: [
'examples/index',
'examples/CARTO',
'examples/fused',
'examples/kepler-gl',
'examples/lonboard',
'examples/build-a-map',
'examples/pandas',
'examples/overture-tiles',
'examples/QGIS',
'examples/rapid',
'examples/spark',
'examples/sedona',
'examples/wherobots',
'examples/athena-aws',
'examples/synapse-azure',
{ type: 'autogenerated', dirName: 'examples' },
],
},
{
Expand Down Expand Up @@ -135,7 +116,7 @@ const sidebars = {
'attribution',
'release-calendar',
'support',

],
};
module.exports = sidebars;
module.exports = sidebars;
Loading