Add scientific data and geospatial publishing guides & case study#2243
Add scientific data and geospatial publishing guides & case study#2243
Conversation
🚀 Build Preview on IPFS ready
|
|
This is great as a specific how-to. Is there another, complementary place we can write about all the ways geospatial users can benefit from IPFS? From live meeting:
|
Co-authored-by: Volker Mische <volker.mische@gmail.com>
Co-authored-by: Mosh <1306020+mishmosh@users.noreply.github.com>
Co-authored-by: Mosh <1306020+mishmosh@users.noreply.github.com>
| --raw-leaves \ | ||
| --chunker=size-1048576 \ | ||
| --cid-version=1 \ |
There was a problem hiding this comment.
Once Kubo 0.40 ships these could be removed and replaced by one-time ipfs config profile apply unixfs-v1-2025 or setting Import.* values one-by-one
This comment was marked as outdated.
This comment was marked as outdated.
There was a problem hiding this comment.
Pull request overview
Adds new Scientific Data documentation content to the IPFS docs site, including a hands-on guide for publishing geospatial Zarr datasets and supporting context via a landscape overview and an ORCESTRA case study. Updates the VuePress sidebar to surface the new pages and case study.
Changes:
- Add a new “Publish Geospatial Zarr Data with IPFS” how-to guide.
- Add a new “Scientific Data and IPFS Landscape Guide” overview page.
- Add a new ORCESTRA case study and update VuePress navigation (including sidebar re-organization and case study list).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 20 comments.
| File | Description |
|---|---|
| docs/how-to/scientific-data/publish-geospatial-zarr-data.md | New step-by-step publishing guide (Zarr + IPFS), including discovery/access patterns. |
| docs/how-to/scientific-data/landscape-guide.md | New overview of scientific data formats, architectural patterns, and ecosystem tooling. |
| docs/case-studies/orcestra.md | New case study describing ORCESTRA’s use of IPFS for scientific data distribution. |
| docs/.vuepress/config.js | Adds the new Scientific Data pages to the How-to sidebar and adds ORCESTRA to case studies; also reorganizes peer-related sidebar entries. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
In the rest of the docs we mostly use sentence casing.
1bfb8d4 to
ff5c629
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 12 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ds = xr.open_dataset(filename) | ||
| # Example: targeting ~1 MB chunks with float32 data | ||
| ds.to_zarr('output.zarr', encoding={ | ||
| 'var_name': {'chunks': (1, 512, 512)} |
There was a problem hiding this comment.
The chunking example uses undefined placeholders (filename, var_name), which will error if readers copy/paste. Consider making these explicit string placeholders (e.g., "path/to/file" / "variable_name") or adding a short comment that they must be replaced.
| ds = xr.open_dataset(filename) | |
| # Example: targeting ~1 MB chunks with float32 data | |
| ds.to_zarr('output.zarr', encoding={ | |
| 'var_name': {'chunks': (1, 512, 512)} | |
| filename = "path/to/your/file.nc" # Replace with the path to your dataset | |
| ds = xr.open_dataset(filename) | |
| # Example: targeting ~1 MB chunks with float32 data | |
| ds.to_zarr('output.zarr', encoding={ | |
| 'variable_name': {'chunks': (1, 512, 512)} # Replace with the name of your variable |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
lkluft
left a comment
There was a problem hiding this comment.
Thanks for adding this really nice article about ORCESTRA!
I made a couple of minor suggestions, but I do like the overall story very much! 👍
vmx
left a comment
There was a problem hiding this comment.
The only major thing is the comment about using some sample data to follow along the steps of the publishing guide.
In all Zarr docs it would make sense that you refer to Zarr v3 as e.g. the metadata is different in v2.
| ```python | ||
| import xarray as xr | ||
|
|
||
| ds = xr.open_dataset(filename) |
There was a problem hiding this comment.
In the MFS section you introduce "HALO flight datasets". I think it would be good if you could introduce them right here at the beginning and then make all steps working with that dataset, so one can follow along locally.
When I started to look into Zarr, I had a hard time figuring out what it exactly looked like on disk. Most resources I found where online notebooks. So such a start to finish walk through with actual data would've been useful. E.g. it would've been clearer to me that a "somedata.zarr" is actually a directory and not a single file.
Also mentioning that this is about Zarr v3 would make sense as Zarr v2 still seems to be quite common.
|
|
||
| # Each mutation produces a new root CID — a lightweight versioned snapshot | ||
| ipfs files stat --hash /datasets/halo | ||
| # bafybeihqixf5ew7mfr74bzb74qiw2mgtnytabnpzjnf5xeejzq4p2ocygu |
There was a problem hiding this comment.
Maybe add a final ls to show that all directories are there.
Co-authored-by: Lukas Kluft <lukas.kluft@gmail.com>
Co-authored-by: Lukas Kluft <lukas.kluft@gmail.com>
Co-authored-by: Lukas Kluft <lukas.kluft@gmail.com>
What
Adds new documentation focused on scientific/geospatial data publishing with IPFS (Zarr + tooling), plus an ORCESTRA case study and related VuePress navigation updates, with a small quickstart retrieval enhancement.
Changes:
Added “Scientific data and IPFS landscape guide” and “Publish geospatial Zarr data with IPFS” how-to pages.
Added ORCESTRA case study and updated VuePress sidebar/navigation to include the new section + case study.
Extended retrieval quickstart with a Python/ipfsspec verified retrieval example and updated spellcheck ignore list.