Skip to content

fix(doc): remove repetitions and copy-paste leftover #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lfoppiano merged 2 commits into from
Feb 12, 2026
Merged

fix(doc): remove repetitions and copy-paste leftover #16

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a62f7b4
fix(doc): remove repetitions and copy-paste leftover
lfoppiano Feb 12, 2026
0f91b1c
fix(doc): move warnings above
lfoppiano Feb 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -790,17 +790,18 @@ The program then writes that one record into a local Parquet file, does a second

### Bonus: download a full crawl index and query with DuckDB

If you want to run many of these queries, and you have a lot of disk space, you'll want to download the 300 gigabyte index and query it repeatedly. Run
All of these scripts run the same SQL query and should return the same record (written as a parquet file).
In case you want to run many of these queries, and you have a lot of disk space, you'll want to download the 300 gigabyte index and query it repeatedly.

> [!IMPORTANT]
> If you happen to be using the Common Crawl Foundation development server, we've already downloaded these files, and you can run ```make duck_ccf_local_files```

To download the crawl index, there are two options: if you have access to the CCF AWS buckets, run:

```shell
mkdir -p 'crawl=CC-MAIN-2024-22/subset=warc'
aws s3 sync s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2024-22/subset=warc/ 'crawl=CC-MAIN-2024-22/subset=warc'
```

> [!IMPORTANT]
> If you happen to be using the Common Crawl Foundation development server, we've already downloaded these files, and you can run ```make duck_ccf_local_files```

If, by any other chance, you don't have access through the AWS CLI:

```shell
Expand All @@ -822,7 +823,7 @@ rm cc-index-table.paths
cd -
```

The structure should be something like this:
In both ways, the file structure should be something like this:
```shell
tree my_data
my_data
Expand All @@ -835,10 +836,8 @@ my_data

Then, you can run `make duck_local_files LOCAL_DIR=/path/to/the/downloaded/data` to run the same query as above, but this time using your local copy of the index files.

> [!IMPORTANT]
> If you happen to be using the Common Crawl Foundation development server, we've already downloaded these files, and you can run ```make duck_ccf_local_files```
Both `make duck_ccf_local_files` and `make duck_local_files LOCAL_DIR=/path/to/the/downloaded/data` run the same SQL query and should return the same record (written as a parquet file).

All of these scripts run the same SQL query and should return the same record (written as a parquet file).

## Bonus 2: combine some steps

Expand Down