# 🗂️ Provisioning the datasets

Datasets are never committed to this repository and never downloaded by test code (AGENTS.md sec. 3). Every unit gate runs offline against synthetic fixtures (A26/D12b); only the WPs marked `[DATA]` need a real tree, and a tier acceptance run needs one that passes `lucid-data check`.

This file says where each dataset comes from, what has to be on disk before a run starts, and — since [WP-099d](https://borda.github.io/lucid-YOLO/work_packages/wp-099d/) — how the two dataset formats lay out on disk and what their annotations actually contain. None of it is restated from memory: the provisioning sections follow what `lucid_yolo.data.check` validates, and the **Dataset formats** reference below follows `lucid_yolo.data.layout`, `.coco`, `.yolo` and `.dota` — those modules are the specification, and every claim below cites the one it came from.

## 🖼️ COCO 2017 (R12)

Automated, because the archives are served from a stable public host (`images.cocodataset.org`) at fixed URLs:

```bash
lucid-data download --data_root /data/coco                       # val + annotations, ~1 GB
lucid-data download --data_root /data/coco --splits '[train,val]'  # adds the 18 GB train archive
lucid-data check    --data_root /data/coco
```

Transfers stream to a `.part` file and are renamed on completion, so a killed run resumes rather than leaving a truncated archive, and an already-extracted split is skipped without touching the network. A `.part` carries a sidecar naming the URL and the expected total size; a partial that disagrees with it, or that has no sidecar at all, is discarded and the transfer restarts rather than resuming into a file of unknown provenance. The official archives publish no authoritative SHA-256 digests, so none are hard-coded; the computed digest of every archive is printed, and an expected one may be enforced per archive with `--sha256`.

`--sha256` enforces on what a run downloads. A split that is already extracted is skipped before any transfer, and the `.zip` it would hash is gone (removed after extraction unless `--keep_archives` was set), so a digest supplied for it prints `checksum not verified: … already extracted` instead of being checked — `--force true` re-downloads and re-verifies.

### The keypoint tier needs a second annotation file, and must be told so

Three of the four tiers read `annotations/instances_{split}2017.json`. The keypoint tier does not: it trains and scores on **`annotations/person_keypoints_{split}2017.json`**, the `person`-only 17-point human-pose annotations.

Nothing extra has to be downloaded. `person_keypoints_train2017.json` and `person_keypoints_val2017.json` ship inside `annotations_trainval2017.zip` — the same archive `lucid-data download` already fetches — and land beside the `instances_` files in `annotations/`.

What *is* required is naming them. `lucid_yolo.data.layout` resolves only the `instances_` spelling, in every one of its `CANDIDATES` rows, so the keypoint config carries two overrides that are **mandatory rather than optional**, unlike every other tier's `data:` block:

```yaml
data:
  data_root: /data/coco
  train_ann_file: /data/coco/annotations/person_keypoints_train2017.json
  val_ann_file: /data/coco/annotations/person_keypoints_val2017.json
```

Those two lines are already in `pose_nano_smoke.yaml` as placeholders; point them at your own root before launching. Leaving them at the placeholder paths, or dropping them, leaves the layout probe to resolve the `instances_` files, whose annotations carry no `keypoints` field — `parse_coco_keypoints` rejects that by name, reporting the file and image rather than raising a bare `KeyError` (L-29). It fails at dataset construction, not silently, but it fails after you have queued the run.

`lucid-data check` validates the layout, which is the `instances_` question; it does not verify that the keypoint annotation files are present or readable. Confirm both paths exist before spending GPU hours on the tier.

## 🛰️ DOTA-v1.0 (R18)

**Manual, and it stays manual.** The distribution has no equivalent of `images.cocodataset.org`: the dataset page offers Google Drive and Baidu Drive folders, which are interactive pages rather than archive URLs. A folder link cannot be resolved to a stable file URL without rendering JavaScript, the Drive large-file path adds a confirmation interstitial, and Baidu Drive requires an account. A downloader written against any of that is a scraper of someone else's UI, which breaks silently and off-repository. So `lucid-data download` covers COCO only, and DOTA provisioning is an operator step.

### Where

Start at the dataset page of the official DOTA site, maintained by the authors of R18:

- <https://captain-whu.github.io/DOTA/dataset.html>

Under **Data Download**, take the **DOTA-v1.0** section — the page also carries DOTA-v1.5 and DOTA-v2.0 sections, and those are different label sets. Each version section offers the same three items, on either host:

| Item | Contains | Needed here |
| -- | -- | -- |
| Training set | images plus their `labelTxt` annotations | yes |
| Validation set | images plus their `labelTxt` annotations | yes |
| Testing images | images only | no |

The testing images are not needed and not downloaded: R18 states that ground truth is released for the training and validation sets *"but not for the testing set. For testing, we are currently building an evaluation server."* The tier is scored on val ([WP-063](https://borda.github.io/lucid-YOLO/work_packages/wp-063/)), so nothing here depends on the withheld annotations.

Two cautions from the page as it stands:

- The v1.5 section links the **same** Drive folders as the v1.0 section. Whichever link is followed, take the DOTA-v1.0 label archive, not a v1.5 one: v1.5 adds a sixteenth category (`container crane`), and `lucid_yolo.data.dota` reads the fifteen of R18 in their published order, where the index *is* the class id. A v1.5 label file fails the read with the offending file and line named rather than being silently dropped.
- The Drive folder listings are rendered client-side, so the archive filenames inside them are not quoted here — they have not been read, and a filename copied from memory is the kind of detail that sends an operator to the wrong file. Take what the folder offers for the version and item named above.

### What has to be on disk

Unpack so that each split directory holds an `images/` and a `labelTxt/` directory, with one label file per image sharing its stem:

```text
<dota_root>/
    train/
        images/P0000.png ...
        labelTxt/P0000.txt ...
    val/
        images/P0003.png ...
        labelTxt/P0003.txt ...
```

The archives do not necessarily unpack into this shape; moving directories after extraction is expected. `lucid-data check --data_root <dota_root> --dataset dota` is what decides whether the result is usable: it verifies both directories exist per split, that images and label files pair one-to-one in both directions, and that every object line parses, naming the file and line of the first that does not.

A complete train-plus-val provisioning from the official distribution reports:

```text
  PASS train — 1411 images, 98990 instances, 15 classes
  PASS val — 458 images, 28853 instances, 15 classes
  NOTE totals across ['train', 'val']: 1869 images, 127843 instances, 15 classes
```

Those are measured, not published — R18 gives whole-dataset totals only, and 2,806 minus 1,869 is exactly the 937 images of the withheld testing third. Compare against them, and pass them to make the comparison a gate:

```bash
lucid-data check --data_root <dota_root> --dataset dota \
  --expected_images 1869 --expected_instances 127843
```

**What it does not check by default, and why.** The totals across the checked splits are reported, not required. The published 2,806 images / 188,282 instances / 15 classes describe DOTA-v1.0 *whole*, and R18 splits it as "half of the original images as the training set, 1/6 as validation set, and 1/3 as the testing set" while releasing ground truth for the first two only — so an annotated root holds about two thirds of that and can never sum to it. Requiring those totals by default is what the check did until [WP-097](https://borda.github.io/lucid-YOLO/work_packages/wp-097/), which made it fail on every correct download. The line to read is now `NOTE totals across ...`; once the counts for a given provisioning are known they can be asserted with `--expected_images`, `--expected_instances` and `--expected_classes`, and are then enforced exactly as before.

### Then build the tiles

The tier trains on 1024 px tiles, never on the original tree — DOTA images run to several thousand pixels a side in PNG, which has no random-access region decode, so cropping per sample would decode a whole image to yield one window ([WP-094](https://borda.github.io/lucid-YOLO/work_packages/wp-094/)):

```bash
lucid-data build-tiles --root <dota_root> --out <tiles_root> --splits train,val --overlap 512 --workers 8
```

Tiles are PNG by default, matching the lossless source. `--suffix .jpg --quality 92` trades that for decode speed, and the trade is worth weighing: the loader decodes every tile once per epoch and mosaic pulls four source tiles per training sample, so a 64-image step at 1024 px is around 250 decodes and PNG is where a run starves before the GPU does. Annotations are unaffected by the choice.

The tiles are a build artifact and are never committed. The output is a COCO container with quadrilateral rings, in the layout `obb_nano_smoke.yaml` names, so training and evaluation read it through the same dataset class as COCO.

Overlap is a real choice, not a default to accept unread. Pixels are amplified by `(patch / (patch - overlap))²`: 512 px of overlap is R18's own crop stride and costs 4.0x, while the 200 px of A21 costs about 1.55x. Larger overlap means fewer objects severed by a tile edge and a longer epoch.

## 🧩 Dataset formats

This is a reader's map, not a restatement: every claim below cites the module that enforces it, and none of it substitutes for reading that module when something is unclear.

Two words matter here and the code keeps them apart. *Layout* is where a split's files sit on disk. `lucid_yolo.data.layout` decides it from directory and file names alone: `detect_layout`'s two probes test only `is_dir()`, `is_file()` and (for a YOLO root) whether `data.yaml` exists — they never open an annotation to make the call (`_probe_coco`/`_probe_yolo`, `lucid_yolo/data/layout.py`). *Format* is what one annotation actually encodes once a reader has been chosen — field count, units, coordinate range, what an empty file means — and that is a property of `lucid_yolo.data.coco`, `.yolo` and `.dota`, not of the layout that pointed a reader at the root.

`lucid_yolo.data.layout` keeps each format's directory convention in its own table — `CANDIDATES` for COCO, `YOLO_CANDIDATES` for YOLO — rather than as rows of one merged table, because a YOLO split's annotations are a *directory*, not a file: the "both halves exist" predicate that lets `CANDIDATES` fall back safely would have to become an existence check that accepts either kind of thing, and a root satisfying both conventions would then resolve to whichever table came first — handing a labels directory to the COCO reader or an `instances_*.json` to the YOLO one (`lucid_yolo/data/layout.py`).

| Task | COCO format | YOLO format |
| -- | -- | -- |
| detect | `bbox`: pixel `[x, y, w, h]`, converted to `xyxy` on read | five normalized fields: `cls cx cy w h` |
| segment | first ring of `segmentation`: pixel `(P, 2)` polygon, `P >= 3` | not expressible — the format carries no rings |
| obb | four-point `segmentation` ring, plus four non-schema keys (A53) | nine normalized fields: `cls x1 y1 x2 y2 x3 y3 x4 y4` |

The structural difference is one file against many: COCO keeps a split's whole annotation set — and its class list — in a single JSON beside the images, while YOLO writes one text file per image and keeps the class list in a separate `data.yaml`. The two trees below are the same `val` split of the same two images in each format.

### COCO

#### Layout

```text
data_root/
├── annotations/
│   └── instances_val.json      one file per split: "images", "categories", "annotations"
└── val/                        images only — the JSON names them and holds every box
    ├── 000000000139.jpg
    └── 000000000285.jpg
```

Class ids come from the JSON's own `categories`, remapped to contiguous labels in sorted-id order rather than used raw (`_build_category_maps`, `lucid_yolo/data/coco.py`). An image with no annotations is still listed under `"images"`, so the split knows it exists.

`CANDIDATES`, tried in this order for a split (`val` below), the first row whose directory *and* file both exist winning (`resolve_split`, `lucid_yolo/data/layout.py`):

| Convention | Images directory | Annotation file |
| -- | -- | -- |
| COCO 2017 | `val2017/` | `annotations/instances_val2017.json` |
| plain COCO / tiled | `val/` | `annotations/instances_val.json` |
| images-subdirectory | `images/val/` | `annotations/instances_val.json` |
| per-split export | `val/` | `val/_annotations.coco.json` |

`resolve_split` never raises on a miss: when nothing matches, it returns the first row unchanged, leaving the "no such file" to the reader that actually opens the path (`lucid_yolo/data/layout.py`).

DOTA-v1.0's own `images/` + `labelTxt/` layout, documented above under **DOTA-v1.0 (R18)**, is not a row of this table — it is read directly by `lucid_yolo.data.dota`, not through `lucid_yolo.data.layout`, and only becomes a COCO-layout root (one of the rows above) after `lucid-data build-tiles` writes it out.

#### Format

**Detect.** An annotation's `bbox` is `[x, y, w, h]` in pixels, top-left corner plus width and height; the reader converts it to this project's `xyxy` convention and clamps it to the image bounds (`_xywh_to_xyxy`, `lucid_yolo/data/coco.py`). On a 100x50px image, `"bbox": [10, 5, 20, 10]` reads as `xyxy = [10, 5, 30, 15]` (`x2 = x + w = 30`, `y2 = y + h = 15`).

**Segment.** `segmentation` is a list of flat polygon rings; only the *first* ring of an ordinary (list-typed) annotation is kept, reshaped to `(P, 2)`, and only when it has at least three points (`_parse_ring`, `lucid_yolo/data/coco.py`). Any `iscrowd=1` annotation, and any `segmentation` value that is *present but unusable* — a run-length dict, an empty list, or a shorter ring — is always skipped entirely, in every mode: neither is ever a real countable instance. The same object as above, `"segmentation": [[10, 5, 30, 5, 30, 15, 10, 15]]`, parses to the ring `[[10, 5], [30, 5], [30, 15], [10, 15]]` — the envelope of that ring is the same box `bbox` gives above, though the (non-oriented) reader does not derive one from the other here; it does on the oriented reading below.

A `segmentation` key that is **absent entirely** is different ([WP-121b](https://borda.github.io/lucid-YOLO/work_packages/wp-121b/)): it is fatal only for an oriented reading, which has no other source for the rotated box; a plain or keypoints reading keeps the box instead — when any instance in an image has no `segmentation` key at all, that image's `Targets.polygons` comes back empty (`[]`) rather than one instance being silently dropped, matching `Targets.polygons`'s "0 (no masks) or N" contract (`lucid_yolo/data/targets.py`). Requesting `mask_targets=True` against such an image still raises in `instance_mask_targets` (`lucid_yolo/ptl/seg_targets.py`), the same way it does for a genuinely maskless dataset — this reader relaxation does not weaken that guard.

**Obb.** Under `oriented=True` the `segmentation` ring is read as a rotated box's quadrilateral rather than a free polygon: it must be exactly four points, fitted to a canonical long-edge box `(cx, cy, w, h, theta)` (`lucid_yolo/data/rotated_geom.py`), and `boxes` is recomputed as the **envelope of that same ring** rather than read from `bbox` — so `boxes[i]` and `rboxes[i]` describe one object by construction (`_oriented_targets`, `lucid_yolo/data/coco.py`). An annotation with no usable ring is skipped entirely under `oriented=True`, since there is no other source for the rotated box. The module's own worked example: the ring `[[3, 2], [7, 2], [7, 4], [3, 4]]` yields `boxes = [[3.0, 2.0, 7.0, 4.0]]` and `rboxes[0] = [5.0, 3.0, 4.0, 2.0, 0.0]` (`_oriented_targets`'s own doctest, `lucid_yolo/data/coco.py`).

A53 adds four keys the COCO schema has none for, written by `lucid-data build-tiles` and read back by name — never by a standard COCO reader, which sees an ordinary detection set and ignores all four:

| Key | Scope | Written at | Read by this project's reader |
| -- | -- | -- | -- |
| `difficult` | per annotation | `annotation_records` (`lucid_yolo/data/tiles.py`) | yes — forwarded onto `Targets.difficult`; absent reads as `False` (`_parse_annotation`, `lucid_yolo/data/coco.py`) |
| `visible_fraction` | per annotation | `annotation_records` (`lucid_yolo/data/tiles.py`) | no — not read by `CocoDetectionDataset` |
| `source_image` | per image | `_image_record` (`lucid_yolo/data/tiles.py`) | no |
| `window` | per image | `_image_record` (`lucid_yolo/data/tiles.py`), a `(x0, y0, x1, y1)` pixel box (`tile_file_name`, `lucid_yolo/data/tiles.py`) | no |

### YOLO

#### Layout

```text
data_root/
├── data.yaml                   names + one line per split; outranks the table below
└── val/
    ├── images/
    │   ├── 000000000139.jpg
    │   └── 000000000285.jpg
    └── labels/                 one .txt per image, paired by stem
        ├── 000000000139.txt
        └── 000000000285.txt    an empty file is a background image, not a missing one
```

A class id is an index into `data.yaml`'s `names` list — the position *is* the label, so nothing is remapped (`YoloDataConfig`, `lucid_yolo/data/yolo.py`). The pairing is by stem and nothing else: the reader opens `labels_dir / f"{image_path.stem}.txt"`, and a missing file raises naming both paths rather than reading as empty, because a tree with no image manifest cannot tell a background image from a truncated export — `allow_missing_labels=True` is the opt-in that says the absences are deliberate (`_load_targets`, `lucid_yolo/data/yolo.py`).

`YOLO_CANDIDATES`, the same rule over two rows, both halves directories so both are tested with `is_dir()` (`resolve_yolo_split`, `lucid_yolo/data/layout.py`):

| Convention | Images directory | Labels directory |
| -- | -- | -- |
| per-split export | `val/images/` | `val/labels/` |
| split-subdirectory | `images/val/` | `labels/val/` |

Same rule as COCO's table: `resolve_yolo_split` returns the first row unchanged on a miss rather than raising.

A YOLO root's own `data.yaml` outranks this table for any split it names. `resolve_split_dirs` is what `YoloDetectionDataset.from_root` actually calls to get a split's directories: a split the file names resolves through `YoloDataConfig.images_dir` — the literal reading against the file's own directory first, then the same entry with leading `..` components dropped, since the published export writes `../train/images` for a directory that actually sits beside the yaml (`YoloDataConfig.images_dir`, `lucid_yolo/data/yolo.py`) — and only a split the file leaves unnamed falls back to `YOLO_CANDIDATES` above (A58; `resolve_split_dirs`, `from_root`, `lucid_yolo/data/yolo.py`).

#### Format

**Detect.** Each non-blank label line is five whitespace-separated fields, class index first, the rest `cx cy w h` normalized by the image's own width/height (`lucid_yolo/data/yolo.py`). At `width=100, height=40`, the row `1 0.5 0.5 0.5 0.25` denormalizes to `xyxy = [25.0, 15.0, 75.0, 25.0]` — `cx=50, w=50 -> x1=25, x2=75`; `cy=20, h=10 -> y1=15, y2=25` (`load_yolo_targets`'s own doctest, `lucid_yolo/data/yolo.py`).

**Obb.** The oriented row is exactly nine fields — class index, then four normalized `(x, y)` corners — and a ten-field row (R18's own label lines append a trailing `difficult` flag the normalized variant has no published spelling for) is rejected naming the file and line, never assumed away (A56; `_parse_row`, `lucid_yolo/data/yolo.py`). At `width=10, height=10`, the row `0 0.1 0.1 0.5 0.1 0.5 0.3 0.1 0.3` denormalizes to the quadrilateral `[[1, 1], [5, 1], [5, 3], [1, 3]]`, giving `boxes = [[1.0, 1.0, 5.0, 3.0]]` and `rboxes[0] = [3.0, 2.0, 4.0, 2.0, 0.0]` (`_oriented_targets`'s own doctest in `lucid_yolo/data/yolo.py`, distinct from the COCO reader's function of the same name).

**Segment.** Not expressible. A label line carries five or nine normalized numbers and never a polygon ring — the format has no rings to rasterize a mask from (`lucid_yolo/data/yolo.py`). `DetectionDataModule` refuses `mask_targets=True` on a YOLO root rather than rasterizing empty masks and reporting a plausible detection number off a segmentation head trained on nothing: "`mask_targets=True is unavailable on a YOLO root: the format carries no per-instance polygon rings, so there is nothing to rasterise masks from`" (`_check_layout_support`, `lucid_yolo/ptl/datamodule.py`).

### Resolving which format a bare root uses

`lucid-data check` is the other caller. It named `--dataset coco` as its default until [WP-099c](https://borda.github.io/lucid-YOLO/work_packages/wp-099c/), which is a pre-flight answering a different question than the run it precedes: a COCO-shaped verdict on a YOLO root reports a missing `train2017` for a tree that trains fine. With `--dataset` unstated it now probes, so the check dispatches exactly as `fit` does; `--dataset dota` still has to be named, because no run reads a DOTA root directly and no probe covers it. Both of the probe's undecidable states come back as report lines and exit 1 rather than a traceback (`_infer_dataset`, `lucid_yolo/data/check.py`).

`detect_layout` is the one caller with no reader of its own to ask — `DetectionDataModule` is handed a bare `data_root` and has to decide which reader it calls for, which is the question neither table above answers on its own. It probes both layout tables in full — by directory and file existence only, per the *layout*/*format* distinction above — and raises if neither is satisfied, naming every path tried, or if **both** are, since across the two tables the loser is a different label space and a different image set, not another spelling of the same reader (A63; `detect_layout`, `lucid_yolo/data/layout.py`).

## ☁️ Where a provisioned tree belongs on a hosted runtime

A hosted runtime is ephemeral and its local disk goes with it, so the two halves of a run belong in different places.

**The dataset and the tiles stay on the runtime's local disk** (`/content/...`), not on a mounted Drive. Every epoch reads every tile, and Drive is a network filesystem mounted through FUSE — putting the training set there turns each sample fetch into a round trip and starves the loader workers the epoch-recycling path exists to keep busy.

**Checkpoints and logs go the other way**, onto storage that outlives the runtime. That is `--trainer.default_root_dir`, and it belongs to the launch rather than to provisioning: docs/TRAINING.md carries it, along with the four tiers' commands.

## 📜 License terms, and what they constrain

DOTA is not permissively licensed. The dataset page states:

> All images and their associated annotations in DOTA can be used for academic purposes only, but any commercial use is prohibited.

and separately that use of the Google Earth imagery within it must respect the Google Earth terms of use. This project's dependency policy — permissive licenses only, never AGPL or a commercial license — is about code that ships inside the wheel, and DOTA is neither shipped nor vendored. What the terms do constrain is what may be derived from it: releases publish no trained weights (D14), datasets are never committed (AGENTS.md sec. 3), and the oriented tier's outputs are the reported metrics and the frozen goldens. Anyone reusing this repository for anything other than academic work should read the terms before provisioning DOTA at all.

COCO 2017 carries its own terms, published on `cocodataset.org` and not restated here — the annotations and the images they describe are covered separately, and the wording on that page is the authority. Neither dataset is redistributed by this repository.